Too many queries... advice on tracking down

Yes you can run Devel on a live site (unless permissions are explicitly granted for other roles to view them, the stats it outputs are limited to the admin seeing them only). However personally I would only leave it on long enough to collect sample data and then disable it (I don't know about the current version, though I've had instances in the past where it had some issues with the jQuery connected with filefield uploads... could be fixed now but I don't know).

A good resource for performance info with Drupal is http://2bits.com/contents/articles
There's also a number of performance related videos at http://dc2009.drupalcon.org

Drupal does have a ton of queries (especially some aspects can add up such as pagers and aliases can produce a rather prodigious quantity of queries, and also updating the caches can add many queries... albeit very tiny ones). I believe this is improved quite a bit in Drupal 7 (once it's released) though with some care it can be worked with in D6.

I don't think the number of queries though is likely upping your CPU use "that" much (I believe the quantity of queries affects disk access more than CPU time, though I'm not an expert in that area). Perhaps "heavy" queries if you have some are hitting the CPU hard, but probably not just the quantity of queries over all. You can see very quickly in Devel how much time is spent on queries versus PHP processing. If you don't have any real bad queries than PHP is more likely where your high CPU usage may be coming from (make sure you're using an Op code cache such as APC, and also check for performance draining PHP mistakes). You can probably gain a lot of performance switching to Memcache for caching (see Memcache or Cache Router module). Also consider Boost module for your anonymous visitors to steal back a huge chunk of performance for use towards your logged in users. Most likely you can get everything you need on a good VPS (though I'd stay away from shared hosting).

I don't know of a list of worst performing modules... one I know off hand that can perform badly is Custom Pagers. The new Previous/Next API from 2bits is a better replacement for that if needed. Various other module performance issues are mentioned on the 2bits site.

Hope this helps.

Be sure you have enabled caching.

If most of your traffic is anonymous, install Boost.

Forgot... one other module that's slow on a large site is Tracker. Drupal.org actually has switched to a contrib module to replace it, Tracker 2. Node access modules can have a slow-down effect on things, and also the forum module has a number of slow spots on sites with large quantities of nodes (it's possible to cache the forum issues with a tiny tweak, let me know if that's an issue you need help with). I haven't used it but I hear Tagadelic is hard on CPU (tag clouds I think), and also Faceted Search is hard on your system (for a large site it's better to switch to Solr). From 2bits I just learned that Admin Menu adds a lot of overhead (of course only to users who have access to it)... I believe version 3 though is much better. Also XML sitemap and core Statistics module are slow.

Lots of good info here. Appreciate it. Haven't actually seen any slowness to speak of.
I think the hosting co I'm with may be complaining about mysql queries just because they want to keep me on an expensive server.
Hard to be sure w/o more research though.

We just don't really have high traffic most of the time. About 2/3 of visitors are anon. so Boost looks attractive. And maybe we don't need Tracker... we just moved to Drupal so I think most users probably don't even know what it is yet.

I think I can get away with some Devel use briefly, but I've found that I have to turn off Ad module and Reftagger module when I do... they don't seem to get along.

Got some Devel results on my front page. But I don't really know what they mean.

Devel reports...
3727 queries in 1133.69 milliseconds. Page execution: 2808.96
Memory used at: devel_init()=1.71 MB, devel_shutdown()=33.71 MB.

I have some queries that apparently run 10 times.

Sample:
0.26  10   user_load SELECT * FROM users u WHERE uid = 36

0.29  10   user_load SELECT r.rid, r.name FROM role r INNER JOIN users_roles ur ON ur.rid = r.rid WHERE ur.uid = 36

0.43  10   profile_load_profile SELECT f.name, f.type, v.value FROM profile_fields f INNER JOIN profile_values v ON f.fid = v.fid WHERE uid = 36

0.21  10   signature_forum_user SELECT signature FROM users_signature WHERE uid = 36

Thought this might be the Who's Online block so I disabled on front page and loaded again. No change... so I really don't know why so many user_load's are occurring.

So any insight on what this data means?

(Edit: might help to mention what's on this front page... there's a main article section, nothing fancy. There's a sidebar block that is fed by a View, filtered by content type. There are two Ad module adblocks displaying one ad each.
One Quicktab block w/two tabs.... each showing diff. content types. A couple menu blocks. A taxonomy block (of the Advanced Taxonomy module type). An archive block fed by a View.)