Come together with the global Drupal community in Rotterdam, 28 Sept – 1 Oct 2026. Sessions, contribution, connection, and Early Bird savings until 8 June.
By beekerstudios on
Needless to say despite my having upgraded and ironing out kinks with issues, my Drupal installation is now out of date since yesterday. I upgraded my previous installation 5.2 to 5.10, over this last week. Under very little load the site seemed marginally slower, now that the upgrade site is live to the world, it's bogged down with hits.
The frontpage of the site is taking an excruciating long period of time to load... to be exact: 12273.23 ms (painful I know)
Here is what I have done
1. Increased php memory from 32MB to 64MB, the server has plenty, so this didn't seem like a huge leap for me, I even went higher, but things seem to get even worse with more memory (perhaps because of swapping)
2. I have gone into my logs, and tried to resolve any outstanding errors that may be occurring.
3. Disabled modules I know I am not using right now, Forum, Statistics, and a few others.
There are 2 functions of my site I think are big killers here, and that is the calendaring (calendar, cck, date), and the image gallery (image, imageapi, imagecache)
Here is a rundown of devel on the frontpage of the site (almost every other line, if not more, gets marked red btw):
40.49 1 drupal_lookup_path SELECT COUNT(pid) FROM url_alias
17.06 1 drupal_lookup_path SELECT src FROM url_alias WHERE dst = 'frontpage'
5.25 1 module_list SELECT name, filename, throttle FROM system WHERE type = 'module' AND status = 1 ORDER BY weight ASC, filename ASC
39.74 1 user_roles SELECT * FROM role ORDER BY name
36.85 1 ipauth_get_ip_authenticators SELECT roles, id FROM ip_authenticator WHERE (ip1 = '2185337590') OR (ip1 <= '2185337590' AND ip2 >= '2185337590')
147.59 1 cache_get SELECT data, created, headers, expire FROM cache_menu WHERE cid = '1:en'
21.26 1 sections_in_section SELECT sid, name, path, status, visibility, theme, weight FROM sections_data WHERE status = 1 ORDER BY weight
11.23 1 drupal_lookup_path SELECT dst FROM url_alias WHERE src = 'frontpage'
39.09 1 cache_get SELECT data, created, headers, expire FROM cache_views WHERE cid = 'calendar_views'
38.27 2 cache_get SELECT data, created, headers, expire FROM cache_views WHERE cid = 'views_urls'
3.77 1 _views_load_view SELECT v.* FROM view_view v WHERE v.name = 'frontpage'
24.07 1 _views_load_view SELECT * FROM view_sort vs WHERE vid = 13 ORDER BY position ASC
21.13 1 _views_load_view SELECT * FROM view_argument WHERE vid = 13 ORDER BY position ASC
9.7 1 _views_load_view SELECT * FROM view_tablefield WHERE vid = 13 ORDER BY position ASC
11.84 1 _views_load_view SELECT * FROM view_filter WHERE vid = 13 ORDER BY position ASC
4.27 1 cache_get SELECT data, created, headers, expire FROM cache_views WHERE cid = 'views_tables:en'
23.93 1 _views_load_view SELECT * FROM view_exposed_filter WHERE vid = 13 ORDER BY position ASC
21.39 2 cache_get SELECT data, created, headers, expire FROM cache_views WHERE cid = 'views_urls'
12.43 1 _views_alpha_pager_get_placement SELECT placement from views_alpha_pager WHERE vid = 13
28.17 1 cache_get SELECT data, created, headers, expire FROM cache_views WHERE cid = 'views_query:frontpage'
17.47 1 cache_get SELECT data, created, headers, expire FROM cache_views WHERE cid = 'views_style_plugins:en'
33.87 1 node_access_view_all_nodes SELECT COUNT(*) FROM node_access WHERE nid = 0 AND ((gid = 0 AND realm = 'all') OR (gid = 2 AND realm = 'nodeaccess_rid') OR (gid = 7 AND realm = 'nodeaccess_rid') OR (gid = 6 AND realm = 'nodeaccess_rid') OR (gid = 1 AND realm = 'nodeaccess_uid') OR (gid = 1 AND realm = 'nodeaccess_author')) AND grant_view >= 1
21.73 1 pager_query SELECT count( DISTINCT(node.nid)) FROM node node WHERE (node.promote = '1') AND (node.status = '1')
8.99 1 pager_query SELECT DISTINCT(node.nid), node.sticky AS node_sticky, node.created AS node_created_created FROM node node WHERE (node.promote = '1') AND (node.status = '1') ORDER BY node_sticky DESC, node_created_created DESC LIMIT 0, 10
36.74 1 list_themes SELECT * FROM system WHERE type = 'theme'
28.12 1 node_load SELECT n.nid, n.vid, n.type, n.status, n.created, n.changed, n.comment, n.promote, n.sticky, r.timestamp AS revision_timestamp, r.title, r.body, r.teaser, r.log, r.format, u.uid, u.name, u.picture, u.data FROM node n INNER JOIN users u ON u.uid = n.uid INNER JOIN node_revisions r ON r.vid = n.vid WHERE n.nid = 1
19.82 1 _node_types_build SELECT nt.type, nt.* FROM node_type nt ORDER BY nt.type ASC
19.02 1 cache_get SELECT data, created, headers, expire FROM cache_content WHERE cid = 'content:1:1'
11.88 1 comment_nodeapi SELECT last_comment_timestamp, last_comment_name, comment_count FROM node_comment_statistics WHERE nid = 1
0.9 1 path_nodeapi SELECT dst FROM url_alias WHERE src = 'node/1'
4.42 1 taxonomy_node_get_terms SELECT t.* FROM term_node r INNER JOIN term_data t ON r.tid = t.tid INNER JOIN vocabulary v ON t.vid = v.vid WHERE r.nid = 1 ORDER BY v.weight, t.weight, t.name
0.6 1 cache_get SELECT data, created, headers, expire FROM cache_filter WHERE cid = '2:8b0a8069232b2ab9982f26958392fcc9'
30.62 1 filter_format_allowcache SELECT cache FROM filter_formats WHERE format = 2
14.43 1 filter_list_format SELECT * FROM filters WHERE format = 2 ORDER BY weight ASC
22.27 1 image_get_random SELECT DISTINCT(n.nid), RAND() AS rand FROM node n WHERE n.type = 'image' AND n.status = 1 ORDER BY rand LIMIT 0, 1
5.08 1 node_load SELECT n.nid, n.vid, n.type, n.status, n.created, n.changed, n.comment, n.promote, n.sticky, r.timestamp AS revision_timestamp, r.title, r.body, r.teaser, r.log, r.format, u.uid, u.name, u.picture, u.data FROM node n INNER JOIN users u ON u.uid = n.uid INNER JOIN node_revisions r ON r.vid = n.vid WHERE n.nid = '471'
4.17 1 image_load SELECT i.image_size, f.filepath FROM image i INNER JOIN files f ON i.fid = f.fid WHERE i.nid = 471
1.61 1 cache_get SELECT data, created, headers, expire FROM cache_content WHERE cid = 'content:471:471'
27.7 1 comment_nodeapi SELECT last_comment_timestamp, last_comment_name, comment_count FROM node_comment_statistics WHERE nid = 471
24.54 1 path_nodeapi SELECT dst FROM url_alias WHERE src = 'node/471'
4.66 1 taxonomy_node_get_terms SELECT t.* FROM term_node r INNER JOIN term_data t ON r.tid = t.tid INNER JOIN vocabulary v ON t.vid = v.vid WHERE r.nid = 471 ORDER BY v.weight, t.weight, t.name
0.85 1 taxonomy_get_vocabularies SELECT v.vid, v.*, n.type FROM vocabulary v LEFT JOIN vocabulary_node_types n ON v.vid = n.vid WHERE n.type = 'image' ORDER BY v.weight, v.name
0.8 1 drupal_lookup_path SELECT dst FROM url_alias WHERE src = 'files/imagecache/square50/files/images/gti_08_31_Henna_and_Shopping.thumbnail.jpg'
20.61 1 cache_get SELECT data, created, headers, expire FROM cache_content WHERE cid = 'content_type_info'
3.99 1 drupal_lookup_path SELECT dst FROM url_alias WHERE src = 'user/1'
15.83 1 drupal_lookup_path SELECT dst FROM url_alias WHERE src = 'node/1'
0.73 1 user_load SELECT * FROM users u WHERE uid = 1
0.87 1 user_load SELECT r.rid, r.name FROM role r INNER JOIN users_roles ur ON ur.rid = r.rid WHERE ur.uid = 1
5.84 1 block_list SELECT DISTINCT b.* FROM blocks b LEFT JOIN blocks_roles r ON b.module = r.module AND b.delta = r.delta WHERE b.theme = 'coe' AND b.status = 1 AND (r.rid IN (2,7,6) OR r.rid IS NULL) ORDER BY b.region, b.weight, b.module
17.57 1 drupal_lookup_path SELECT dst FROM url_alias WHERE src = 'frontpage/edit'
10.91 1 drupal_lookup_path SELECT dst FROM url_alias WHERE src = 'frontpage/clone'
4.27 1 drupal_lookup_path SELECT dst FROM url_alias WHERE src = 'frontpage/export'
21.54 1 drupal_lookup_path SELECT dst FROM url_alias WHERE src = 'frontpage/devel/load'
19.14 1 drupal_lookup_path SELECT dst FROM url_alias WHERE src = 'frontpage/devel/queries'
65.94 1 drupal_lookup_path SELECT dst FROM url_alias WHERE src = 'frontpage/devel/items'
15.76 1 system_region_list SELECT * FROM system WHERE type = 'theme' AND name = 'coe'
11.05 1 drupal_lookup_path SELECT dst FROM url_alias WHERE src = 'node/4'
16.63 1 drupal_lookup_path SELECT dst FROM url_alias WHERE src = 'node/352'
23.62 1 drupal_lookup_path SELECT dst FROM url_alias WHERE src = 'node/8'
19 1 drupal_lookup_path SELECT dst FROM url_alias WHERE src = 'node/45'
36.08 1 drupal_lookup_path SELECT dst FROM url_alias WHERE src = 'node/46'
20.03 1 drupal_lookup_path SELECT dst FROM url_alias WHERE src = 'node/351'
3.1 1 block_block SELECT body, format FROM boxes WHERE bid = 10
3.7 1 cache_get SELECT data, created, headers, expire FROM cache_filter WHERE cid = '3:14ab718c32aa45bffc5d5f6cb8d16cbc'
1.62 1 _views_load_view SELECT v.* FROM view_view v WHERE v.name = 'news_ticker'
13.24 1 _views_load_view SELECT * FROM view_sort vs WHERE vid = 1 ORDER BY position ASC
24.47 1 _views_load_view SELECT * FROM view_argument WHERE vid = 1 ORDER BY position ASC
22.14 1 _views_load_view SELECT * FROM view_tablefield WHERE vid = 1 ORDER BY position ASC
14.87 1 _views_load_view SELECT * FROM view_filter WHERE vid = 1 ORDER BY position ASC
4.34 1 _views_load_view SELECT * FROM view_exposed_filter WHERE vid = 1 ORDER BY position ASC
18.7 1 _views_alpha_pager_get_placement SELECT placement from views_alpha_pager WHERE vid = 1
17.85 1 cache_get SELECT data, created, headers, expire FROM cache_views WHERE cid = 'views_query:news_ticker'
25.65 1 views_build_view SELECT DISTINCT(node.nid), node.created AS node_created_created, node.title AS node_title, node.changed AS node_changed, node.created AS node_created FROM node node WHERE (node.type IN ('news')) AND (node.status = '1') ORDER BY node_created_created DESC LIMIT 0, 3
3.59 1 drupal_lookup_path SELECT dst FROM url_alias WHERE src = 'node/883'
5.73 1 drupal_lookup_path SELECT dst FROM url_alias WHERE src = 'node/870'
10.41 1 drupal_lookup_path SELECT dst FROM url_alias WHERE src = 'node/868'
10.82 1 drupal_lookup_path SELECT dst FROM url_alias WHERE src = 'about/news'
0.87 1 _views_load_view SELECT v.* FROM view_view v WHERE v.name = 'events_ticker'
26.38 1 _views_load_view SELECT * FROM view_sort vs WHERE vid = 5 ORDER BY position ASC
16.1 1 _views_load_view SELECT * FROM view_argument WHERE vid = 5 ORDER BY position ASC
50.25 1 _views_load_view SELECT * FROM view_tablefield WHERE vid = 5 ORDER BY position ASC
23.19 1 _views_load_view SELECT * FROM view_filter WHERE vid = 5 ORDER BY position ASC
3.7 1 _views_load_view SELECT * FROM view_exposed_filter WHERE vid = 5 ORDER BY position ASC
0.54 1 _views_alpha_pager_get_placement SELECT placement from views_alpha_pager WHERE vid = 5
3.97 1 cache_get SELECT data, created, headers, expire FROM cache_views WHERE cid = 'date_browser_views'
20.29 1 views_build_view SELECT DISTINCT(node.nid), node_data_field_start_of_event.field_start_of_event_value AS node_data_field_start_of_event_field_start_of_event_value, node.title AS node_title, node.changed AS node_changed, node_data_field_start_of_event.field_start_of_event_rrule AS node_data_field_start_of_event_field_start_of_event_rrule, node_data_field_start_of_event.delta AS node_data_field_start_of_event_delta FROM node node LEFT JOIN content_field_start_of_event content_field_start_of_event ON node.vid = content_field_start_of_event.vid LEFT JOIN content_field_start_of_event node_data_field_start_of_event ON node.vid = node_data_field_start_of_event.vid WHERE (node.type IN ('coe_events')) AND (node.status = '1') AND ( UNIX_TIMESTAMP(REPLACE(content_field_start_of_event.field_start_of_event_value, 'T', ' ')) >= 1223535600 && UNIX_TIMESTAMP(REPLACE(content_field_start_of_event.field_start_of_event_value, 'T', ' ')) <= 1286693999) ORDER BY node_data_field_start_of_event_field_start_of_event_value ASC LIMIT 0, 3
12.94 1 drupal_lookup_path SELECT dst FROM url_alias WHERE src = 'node/872'
6.5 1 cache_get SELECT data, created, headers, expire FROM cache WHERE cid = 'date_timezone_offsets:America/Los_Angeles'
46.05 1 drupal_lookup_path SELECT dst FROM url_alias WHERE src = 'node/892'
37.43 1 drupal_lookup_path SELECT dst FROM url_alias WHERE src = 'node/893'
72.35 1 drupal_lookup_path SELECT dst FROM url_alias WHERE src = 'about/events'
93.27 1 cache_get SELECT data, created, headers, expire FROM cache_views WHERE cid = 'views_arguments:en'
49.12 1 drupal_lookup_path SELECT dst FROM url_alias WHERE src = 'blog/1'
14.73 1 drupal_lookup_path SELECT dst FROM url_alias WHERE src = 'node/add'
93.6 1 drupal_lookup_path SELECT dst FROM url_alias WHERE src = 'devel/queries'
11.89 1 drupal_lookup_path SELECT dst FROM url_alias WHERE src = 'image/recent'
129.62 1 drupal_lookup_path SELECT dst FROM url_alias WHERE src = 'tracker'
15.87 1 drupal_lookup_path SELECT dst FROM url_alias WHERE src = 'admin'
46.36 1 drupal_lookup_path SELECT dst FROM url_alias WHERE src = 'logout'
8.21 1 sess_write SELECT sid FROM sessions WHERE sid = 'id5r9ejnpscpteb5g721c57dv5'
13.94 1 sess_write UPDATE sessions SET uid = 1, cache = 0, hostname = '130.65.154.246', session = 'hit_once|b:1;form|a:9:{s:32:\"04f4e67b4ac261dee52b48d5334320c8\";a:2:{s:9:\"timestamp\";i:1223566477;s:4:\"args\";a:1:{i:0;s:14:\"system_modules\";}}s:32:\"ce8c40ac49fe7bd3c5c688129cddca4b\";a:2:{s:9:\"timestamp\";i:1223570977;s:4:\"args\";a:1:{i:0;s:14:\"system_modules\";}}s:32:\"c595c0eb1f6bf7a3c06de14e4ffd6960\";a:2:{s:9:\"timestamp\";i:1223571008;s:4:\"args\";a:1:{i:0;s:14:\"system_modules\";}}s:32:\"68c01dfc2723cd51de7e0420700a5ea0\";a:2:{s:9:\"timestamp\";i:1223571515;s:4:\"args\";a:1:{i:0;s:14:\"system_modules\";}}s:32:\"7f764e316c634c0560303d6fc44a7b67\";a:2:{s:9:\"timestamp\";i:1223571848;s:4:\"args\";a:1:{i:0;s:14:\"system_modules\";}}s:32:\"971acc9123e8f2013c6902e5006ca48e\";a:2:{s:9:\"timestamp\";i:1223571905;s:4:\"args\";a:1:{i:0;s:14:\"system_modules\";}}s:32:\"c9716653bb1cfb57aa4a2dd95051d6a2\";a:2:{s:9:\"timestamp\";i:1223571938;s:4:\"args\";a:1:{i:0;s:14:\"system_modules\";}}s:32:\"cbbcae1a76f04babf0ca3c68b9c317e9\";a:2:{s:9:\"timestamp\";i:1223573877;s:4:\"args\";a:1:{i:0;s:14:\"system_modules\";}}s:32:\"e9e6c362c27b6ef6b5eeabbe0e443b19\";a:2:{s:9:\"timestamp\";i:1223573994;s:4:\"args\";a:1:{i:0;s:14:\"system_modules\";}}}watchdog_overview_filter|s:3:\"all\";', timestamp = 1223574471 WHERE sid = 'id5r9ejnpscpteb5g721c57dv5'
0.39 1 sess_write UPDATE users SET access = 1223574471 WHERE uid = 1
Comments
All those queries are taking
All those queries are taking much longer than they should. But do they add up to the 12273.23ms? Maybe PHP is also very slow? How may concurrent website visitors does the site/server have? Is the server simply overloaded? If anything there have been one or two performance *improvements* between 5.2 and 5.10.
gpk
----
www.alexoria.co.uk
gpk
----
www.alexoria.co.uk
Static HTML loads QUICK
Thanks for the quick response.
Yes those queries are taking WAY too long, 12 seconds, and a hit just a minute ago took almost 50 seconds. WOW! Something is amuck here! No extraordinary traffic spike, and there is no reason for it, and I can see on analytics that we aren't breaking records. Max I see is about 500 hits a day, today there is maybe 50 or 60 if that, I am talking about unique visitors I can see in google analytics, but maybe I am reading it wrong.
- Static HTML loads very quick on the same exact server
- 2 off-shoot drupal installations running 5.10, and 6.4 both run pretty quick at the same time on the same server
This one installation with these modules installed runs way slower, and starts to peg the server. It's also the main site, and is the main point of entry for all sites within it (including the offshoots, and static HTML sites).
Is there a particular module or configuration of the installation that is killing me?
Here are my installed and enabled modules
Drupal Core 5.10 for starters
Drupal Core Blog
Drupal Core Comment
Drupal Core Menu
Drupal Core Search
Drupal Core Taxonomy
CCK Field Permissions 5.x-1.x-dev
CCK Content 5.x-1.9
CCK Option Widgets 5.x-1.9
CCK Text 5.x-1.9
Calendar 5.x-2.3
Calendar iCal 5.x-2.3
Date 5.x-2.3
Date API 5.x-2.3
Date PHP4 5.x-2.3
Date Timezone 5.x-2.3
Devel 5.x-1.x-dev
Image 5.x-2.0-alpha3
Image Attach 5.x-2.0-alpha3
Imge Gallery 5.x-2.0-alpha3
Image Import 5.x-2.0-alpha3
ImageAPI 5.x-1.2
ImageAPI GD2 5.x-1.2
ImageCache 5.x-1.2
ImageCache UI 5.x-1.2
Notify 5.x-2.x-dev
Custom Pagers 5.x-1.x-dev
Google Analytics 5.x-1.x-dev
IP Authenticator 5.x-1.2
Nodeaccess 5.x-1.2
Pathauto 5.x-2.x-dev
Sections 5.x-1.x-dev
Secure Pages 5.x-1.x-dev
Token 5.x-1.x-dev
Tagadelic 5.x-1.x-dev
Taxonomy redirect 5.x-1.x-dev
Video 5.x-1.x-dev
Daterange 5.x-1.x-dev
Views Alpha Pager 5.x-1.x-dev
Views 5.x-1.6
Views RSS 5.x-1.6
Views UI 5.x-1.6
I couldn't deal with my
I couldn't deal with my server going up and down anymore, I had to revert to the previous installation.
If anyone has any insight into the problem, please let me know. If I am not giving enough info please let me know.
I'd like to be a good "web citizen" and have this upgraded, at least to the latest 5.x version, but it makes it really hard when the upgrade path is so unreliable.
Thanks in advance!
Well that's a fair amount of
Well that's a fair amount of modules but doesn't explain the change in performance.
is incredible, usually url_alias lookups are blindingly fast. I wonder if the DB is damaged in some way - the problem really seems to be at the MySQL level if a simple query is taking so long. Might be worth doing check/repair/optimize on the DB, and also check if indexes are missing e.g. from url_alias table, which should have pid as primary key, unique key on src and ordinary index on dst.
Beyond that I think you'd need some MySQL expertise to diagnose the problem with the database. Maybe something went wrong during the upgrade - might be worth trying again?
gpk
----
www.alexoria.co.uk
gpk
----
www.alexoria.co.uk
Does anyone know of a good resource for "testing"
I need a tool that can bang on my site in a sporadic manner. I suspect I can probably write some cron, with wget, to download pages randomly. But does anyone know of a tool out there that can do this for me. Something I can stop mid-way if I see similar or the same behavior occurring.
I am going to upgrade to 5.11, and then do a check/repair/optimize on the mySQL, but I can't let it go live without some good testing.
GPK thanks for the help.
Answering my own question
It's related, but not entirely on topic, I found a good list of "web stress/load" tools available here:
http://www.softwareqatest.com/qatweb1.html#LOAD
Does anyone have any input on which may work better than another?
Thanks!
I realized something
Drupal 5.2 -> Drupal 5.10 introduced some performance hindering additions. However, I never had to have performance caching enabled, so once I did that, the performance of the site skyrocketed.
I used jmeter (apache's load testing tool), and was getting 30-40 hits per minute, and the response times were all over the map (but with only 30-40 hits, still way too slow). With performance caching on, I am getting upwards of 600-1,000 hits per minute. What this means is the response times drop to almost NOTHING (100 ms in some cases). Most if not all of the users of my site are not logged in users, and even so the ones who get "auto" logged in, it's more of a method to prevent some access to some content to the outside world.
Here as a side by side comparison of Drupal with and without performance caching enabled: http://www.engr.sjsu.edu/images/webmaster/projects/drupal/drupal_perf_ca...
Needless to say I am disappointed that Drupal's core and basic rendering of pages is so DANG SLOW, but on the other hand I am very impressed by the intelligent caching and performance that Drupal has included as the core of the CMS, but again very disappointing that I NEED this caching on to get the performance I should really have from the getgo.
P.S. Jmeter while not the prettiest application, is a great tool to bang on a server/application quickly, really quickly!
P.P.S. The server this is running on is a homegrown in-house hosted (sitting on the internet 2 backbone), LAMP box, with a beastly hardware install, so to see my server getting bonked was REALLY surprising.
>Drupal's core and basic
>Drupal's core and basic rendering of pages is so DANG SLOW
Well this site is on 5.10 and no apparent problems.
>performance hindering additions
The only particular thing I'm aware of, as I intimated above, is actually a performance improvement that reduces the amount of processing of blocks that Drupal does (probably not a huge improvement though). I suspect you hit some sort of a "limit" on your server - possibly the code footprint is marginally bigger, or one or two more queries, and suddenly PHP or MySQL doesn't have quite enough RAM and everything starts paging. With normal cacheing on then the memory footprint and number of queries of all anon. page requests is drastically reduced. Probably tweaking the server config would have fixed it but I'm not an expert in this.
TBH using the normal cacheing mode is probably the most sensible "first step".
gpk
----
www.alexoria.co.uk
gpk
----
www.alexoria.co.uk
Probably right
gpk, I suspect you are right.
The box is a Dell poweredge 2950 with Dual Xeon 3.0Ghz Procs and 4GB of ram.
Now the OS config is a little confusing, and the sys admin who set it up no longer works for us (he is a good friend/guy, and I am working with him on this to resolve it).
For security and redundancy the box is setup with xen VM's, and to tell you the truth, there hasn't been much issue until now.
The xen box that the main webserver sits on has 2GB of ram allocated to itself, and it's Running Cent Os 5, I believe with Apache 2.0.52 and PHP 5.1.6.
Php right now is setup with 96MB of memory limit, which is a requirement of imagecache.
Now mysql is a whole other issue, and if I thought or knew how to allocate more memory or resources to it, I suspect that would resolve my query delay issues.
Now this whole xen setup may be the wrong thing to do, in fact I have another box for video streaming, and I made it a requirement that it did not do any xen, mainly because of latency, and the security concerns with the box are not as much of an issue, since it's mainly only a box I log into, and nobody else has a reason to.
I can live with performance caching on, but things still seem to be kind of spotty, especially with the logged in interface for my colleague and me, and in the future we hope to allow people to login for other purposes, it's a valid request.
I would be interested in hearing from people who are running Drupal on non-shared hosted sites. Co-lo's or self admined boxes, to see what kind of special tweaks were needed to make Drupal spring to it's potential.
I am willing to admit that I have quite a few modules, and that some (calendar, video, image, imagecache) are pretty resource intensive, and if I didn't have them in place things might be running a little quicker, but some are a necessary evil, and it would be almost impossible at this point to drop them off.
Thanks for the discussion, and help.
P.S. I have been live with 5.11 with performance caching on for the last 3 or so hours and haven't seen much that has raised a red flag with me.
Date PHP4 with Use PHP default timezone set to "FALSE" kills
Final culprit appears to be the Date PHP4 module, which is required since I am still running PHP 5.1.2 (for one reason or another).
Once I changed the Date PHP4 Use PHP default timezone to "TRUE", things seemed to dramatically calm down. The server also stopped spawning what seemed like infinite instances of httpd. I am right around 21 right now, but after an hour or 2 before things started to climb into the 42 range, and even with 2 vcpu's the CPU was pegged with little to no idle cpu time.
I am glad I was able to do a little bit of tuning of mysql, apache, etc, but boy am I am irritated.
http://n0tablog.wordpress.com
http://n0tablog.wordpress.com/2007/11/19/drupal-caching/ might be useful, especially the "Accelerating PHP apps" link. If Apache is spawning like crazy then it's probably trying to handle too many simultaneous connections -> too much RAM required -> everything starts paging.
Not familiar with Date PHP4 so quite why that has such a dramatic impact I don't know!
Good luck ...
gpk
----
www.alexoria.co.uk
gpk
----
www.alexoria.co.uk