Deny access to term page?

If a user is denied access to a term (let's say "members only"), then that user can still access the term's taxonomy view page (ie. /taxonomy/term/XX), which is generally ok, because it's empty.

I was wondering if there was a way to deny access to that page too? It looks a bit dodgy having an empty page there for anonymous visitors...

If there isn't already a way to do this, consider it a feature request.