I just wanted to point you guys to this little modification I decided to use for my site.
Because I still want my authors to see the "unallowed" terms, so the hierarchy of the select doesn't get destroyed, I needed the options to be disabled instead of unset. Drupal 6' form api doesn't allow this, so I decided to use Javascript.

I hope this helps someone.
In term_permissions.module, instead of doing this:
unset($terms->option[$tid]);

Do this:
drupal_add_js('$("#edit-taxonomy-' . $vid . '>option[value=\''. $tid . '\']").attr("disabled","disabled");', 'inline', 'footer');