Removing term from Node does not remove node from tax queue

I have a taxonomy queue based on a vocabulary with two terms (term1 and term2).
I create a node, select term1, save it, and add it to the subqueue for term1.
Now I edit the node again and select term2.
The node however stays in the subqueue for term1.
I can only remove it by going to the subqueue management page.

If I change the term I expect the node to be removed from its subqueue.