Help with Different Views for Different Content Types and Categories

This has been absolutely driving me nuts for ages. I've searched around a bunch and have never come up with anything so I apologize if this is covered somewhere else.

Let's say I have two content types, video and blog. Each of the content types has different taxonomies to categorize the nodes. The video content type has a taxonomy called "video categories" and the blog content type has a taxonomy called "blog categories". Now I want to list all the nodes for each different category using views. Views has the default taxonomy/term page view which will output all of the categories in the same format. So when I click on a term on either the a video node, or a blog node, I'm taken to the views page for whatever term I clicked on and what is shown is what I had set up that view to show, node title, teaster, etc. But what if I want to have different fields show up in the views for different taxonomies?

So for example, the view for the video terms would show the node title, the video rating from fivestar, and the number of comments. The view for the blog terms would show the node title, the teaser, and the number of comments. Those are just off the cuff examples, I don't have any problems creating views that have different fields, but what I can't figure out is how to make the different taxonomies use different views.

I'm thinking this is a URL and arguments issue, and I've been struggling with arguments a bit using views as a whole, so if anyone can help me out on this I would be forever grateful! If I'm not entirely clear in this writeup please feel free to ask for clarification.

Thanks in advance!

G