remove a term from content, only works with administer taxonomy permission

I wrote a rule to load taxonomy terms, add a term to content and remove a term from content. And it works ok by user 1 (or with users with the administer role), it also works to add taxonomy terms to content. But, when the rule is triggered by a non-admin saving content, the remove a term part of the rule does not work. So, my content ends up with the new term that was added, and also the old term that should have been deleted.

If I give this non admin user a role that has administer taxonomy permission, then the taxonomy term IS removed when the rule is triggered.

This seems like a bug because:
adding the term works without the permission
and
I dont want the user to be able to mess with the vocabulary.