A topic that makes people cry, especially if they have used Taxonomy_Access.module previously.
I've just been looking over the various functions that the taxonomy_access.module does and wondered if it would be possible to simplify this. rather than jumping in and developing a new module to test this I wondered if people could point out problems beforehand.
Here's the idea.
Everytime a node is about to be rendered for viewing, the new module would check the associated taxonomy term ids via the hook_nodeapi() function.
If the current viewing users role doesn't match the permission table for any of the taxonomies associated with the node in question, redirect the user to the Drupal 403 error page.
Special case
If the node path matches that of the 404 or 403 error nodes defined in the main admin->settings then always return true to prevent server 500 errors due to 403 caused by ErrorDocument not being available to Apache.
Patches
I think the above idea would work, and be much lighter to implement than the current taxonomy_access.module
It would still need the taxonomy.module patch in order to not return taxonomies that a user role should not be seeing.
I don't see the need to store view/create/update permissions for every node. Which is what it looks like taxonomy_access.module is doing everytime a node is touched (through the taxonomy_access_nodeapi() call).