Well, this is a good one
Can access Managed metadata service in Central admin -> Manage service apps but not from the site itself.
Try this
Set the app pool for the site temporarily to run as the Farm account
IISRESET
Load site term store manager, it should now load
now …
Switch the site app pool back to SPAppPool or whatever it was
IISRESET
Load site term store manager, it should still load
My explanation is, Black magic and voodoo 🙂