Grouping by taxonomy terms when a content type use more than one taxonomy vocabulary (D6)

So I have a content type that uses 2 vocabularies and I need to group these (in a view) by one taxonomy vocabulary (the other vocab is used by a different view):

group1 (vocab1): apples
node:1
node:2

group2 (vocab1): oranges
node:1
node:3

group3 (vocab1): bannanas
node:3
node:6

but I can't seem to narrow down the taxonomy to one vocab, so instead of the above we get:

group1 (vocab2): PC
node:1
node:2

group2 (vocab2): oranges
node:1
node:3

group3 (vocab1): bannanas
node:3
node:6

is there any way to narrow down the vocabulary the nodes are grouped by?