Access control on taxonomy causes EntityMalformedException

I have a site that has Domain Access setup and has custom access control on vocabularies so that terms in a certain vocabulary can only appear on certain domains.

If you go to a term page where the term belongs to the vocabulary thats not set for the domain you're using, you get an exception: EntityMalformedException: Missing bundle property on entity of type taxonomy_term. in entity_extract_ids().

This is caused by the globaldirect_init assuming taxonomy_term_load will return a valid term object.