User-role based access in drupal 6.10

I am trying to build an intranet portal with drupal 6.10 which has varous sections. Now, i am creating these sections with the help of "Taxonomy". e.g. a vocabulary called "Finance" will have terms like "capex projects", "opex projects", "purchase orders" etc. and another vocabulary term like "IT" would have terms like "Network diagrams", "SOPs", "Knowledge Base" etc.

When "user A" from IT logs in, he sees only articles belonging to the vocabulary "IT" and not "Finance". The opposite holds true for "User X" from Finance. I could group the users into specific user groups and try to assign "permissions", but it does not let me assign permissions to vocabulary groups!

Now i see there is possibly a dirty way to do this by adding "Access rules' and denying a user access to page where pattern matches taxonomy/term/X.... I haven't tried to be sure to know that this works, but i realize it would become difficult when user count goes up.

Any pointers?