Pager jumping between taxonomy terms

Thank you for a great module! I have a problem though:
I have a bunch of nodes that I want to group using taxonomy terms. They are of different node types and the only thing in common is the taxonomy term used. The Custom Pager works great as long as each node only has one taxonomy term but my problem is that I want to use multiple terms on some of the nodes.
The result is that one term seems to work as the default and as soon as I get to one of the nodes with multiple terms the pager "jumps" and starts counting the nodes with this term.

Example:
Node 1: term1
Node 2: term 2
Node 3: term 1, term 2
Node 4: term 1
Node 5: term 2

Menu item 1: term 1
Menu item 2: term 2

When I click on "Menu item 1", the pager works like it should and goes through nodes 1, 3 and 4
When I click on "Menu item 2", the pager goes through nodes 2, 3 but instead of continuing with 5, it shows 4 and stays with that taxonomy term.

How can I get around this problem? Thankful for any ideas!