Well, this is a good one
Can access Managed metadata service in Central admin -> Manage service apps but not from the site itself.
Set the app pool for the site temporarily to run as the Farm account
Load site term store manager, it should now load
Switch the site app pool back to SPAppPool or whatever it was
Load site term store manager, it should still load
My explanation is, Black magic and voodoo 🙂