stumped on performance issues
Hi all - I have been unsuccessfully wrestling with this problem all day so I need to ask my Drupal brethren for a sanity check.
Basically, my site's performance is terrible. At best there is a several-second load time, but very often the site is completely unreachable. The server load (per "uptime") is very high, sometimes as high as 6 or 7, and rarely much below 3. After various attempts at tuning and upgrading my memory allocation, performance is still bad, so I want to describe the situation and ask whether folks think my server is too weak, or whether something else is up.
My site gets about 2500 unique visitors and 9000 page views each day. According to "who's online," during the high points there are typically 60 anonymous users and 25 registered users. (I use drupal caching). There is a decent amount of nodes - maybe 1-2 thousand? - and I have a not-excessive number of blocks employed.
I had a VPS with 256Mb of RAM (that's 256 guaranteed - I share "burstable" ram with other VPS on the server). This also had a mailserver on it, so I thought maybe 256 guaranteed wasn't enough and upgraded that to 512 MB of RAM today. But it didn't really seem to help at all.
Now 512M isn't exactly a powerhouse server. But considering that i use caching and that I typically top out at 25 registered users on at a given time, I just don't understand why my server should be at a crawl like that.
Other info: while my site was never blazingly fast, the performance has degraded severely in the past week (this happened in the past occasionally, but always magically fixed itself.)
Also very mysterious, cron.php takes a really long time to run. Perhaps this is just symptomatic of the loaded server.... but what's weird is that "cron run completed" messages show up several times per hour in my logs, even though I have it set to run only once every four hours! I changed the name of cron.php to ensure that someone else wasn't invoking it, but even after that, it seems to run by itself, if the logs are to be believed.
In addition to doubling my guaranteed RAM, I have attempted to do various tuning things such as setting the right number of apache processes, etc. (I am pretty new at that kind of stuff, so I tried to just hit the big ticket items). But nothing seems to make a difference. I tried installing the devel module, and sure enough some queries take a really long time, but it seems random which ones, so this could again be the result of an overloaded server in general.
So considering all this -- my site's size, the server, the ineffectiveness of tuning attempts, the weird cron behavior -- does anyone have ideas about the problem? Do I just need to throw more server at it, or is cron running amock, or is something else weird going on?
Any thoughts would be appreciated as I really don't know what to do next.
Thanks very much!!!
Rich
have you taken a look at the
have you taken a look at the mysql optimizations mentioned in the handbook pages ?
drupal not being a file based system relies heavily on your database server.
how many contrib modules are you running ?
missed that page
Hi VM - I thought I had, but I now realize that I was reading the "main" optimization page (which basically tells you to install devel) and I missed the dedicated mysql page. I will check it out and try all that stuff, thanks!
Regarding contrib modules, besides devel I am running:
- gsitemap
- image
- img_assist
- pathauto and path
- print
- subscriptions
- taxonomy_block
- tinymce
So 8 in total, if I am counting right...
OK, I'm off to check out that more detailed mysql page... thank you!
rich
Make sure you have the
Make sure you have the latest pathauto module installed. One of the older versions had a bug that really slowed things down.
Have you
tried one of those PHP accelerators/instruction caches? I haven't myself, but they seem to make a difference to others.
It sounds like you are hitting your CPU quite hard which makes me suspect that PHP is the bottleneck. Have you run top to see which processes are working the hardest?
Also, 512MB is still possibly on the low side for a site like that but try all the other tuning parameters before upgrading that further. Maybe your host could let you try out some more and see how it runs with more memory. The DB will appreciate more memory the most, so if the DB is the bottleneck more memory will help.
--
Anton
New to Drupal? | Forum posting tips | Troubleshooting FAQ
no accelerator...
Hi Anton - I have not tried one of those accelerators. I've run top (incessantly!) and it's always the httpd processes that are the big CPU hogs.
Thanks for the reply!
Rich
httpd opts
If httpd is the bottleneck, make sure you have enough memory and aren't swapping out to disk. You should take a look at your max connections in httpd.conf and make sure it isn't set too high. For example, say you run "top" and notice that each httpd process is taking up 10MB, and you have 1GB of RAM. That means that you can do 100 Max Connections before you'll swap (1000/10 = 100).
Turning the KeepAlive down to something like 2 or 3 seconds will also help free up some of the httpd processes.
You can also play around with letting Apache instead of Drupal handle 404 errors within your .htaccess file:
# This overrides the Drupal 404 handler for files that should never be handled by Drupal# Note: If you're using imagecache module you can't 404 images or it'll break the module
<FilesMatch "\.(jpg|png|gif|s?html|css|js|cgi|ico|swf|flv)$">
ErrorDocument 404 /404.html
</FilesMatch>
And one other thing, you can use mod_expires to cache static content on the browser side to cut down the number of requests apache needs to handle.
<IfModule mod_expires.c>ExpiresActive On
ExpiresByType text/css "access plus 1 day"
ExpiresByType text/javascript "access plus 1 day"
ExpiresByType application/x-javascript "access plus 1 day"
ExpiresByType image/* "access plus 1 month"
</IfModule>
There's plenty of other options as well, such as moving .htaccess files into httpd.conf to apache doesn't need to do directory scans for each request, but this should give you a good start.
***
My Drupal Book:
http://amazon.com/o/ASIN/1590597559
some of the above
Hi Matt - My noob-ish attempts at optimization included the first two (calculating appropriate number of httpd threads, reducing keepalive to 2 seconds).
I have not tried the second two things. They seem like very good things to try, especially caching the static content browser-side. I have many images that don't change much so that's a great idea.
I am still digging into the db stuff as suggested by VeryMisunderstood, and I am finding lots of area for improvement. I will try your stuff after that and report back...
thanks,
Rich
Trying to activate expires/caching...
I tried to activate mod_expires as described above. But I'm still getting the cache-control response header saying everything must be validated/re-downloaded with a default 1978 expiration date:
Server Apache/2.2.3 (Win32) mod_ssl/2.2.3 OpenSSL/0.9.8d mod_perl/2.0.3-rc1 Perl/v5.8.8Expires Sun, 19 Nov 1978 05:00:00 GMT
Last-Modified Tue, 17 Jul 2007 05:12:26 GMT
Cache-Control no-store, no-cache, must-revalidate, post-check=0, pre-check=0
I'm running Drupal 5.1 using php5, Apache 2.x under Windows 2000...
Radi Shourbaji
http://www.NaSPA.com - Now Drupal Powered!
It sounds like your running
It sounds like your running on limited hardware, but that shouldn't be a problem -- assuming you are running drupal out of the box. However, my guess is that you aren't running drupal out of the box.
Here's my advice:
1. Look at what blocks you have enabled. Each block = performance hit -- but not all blocks are made equal. A recent comments is relatively harmless to performance -- in high traffic situtations -- a popular content block spells death.
2. Make sure locale is turned OFF.
3. Using views or CCK? See if you can live without them.
4. If you have custom modules take a good long look at what you are asking php to do every page request.
5. If you have a custom theme, and it has over 15 tpl.php ciles -- than you've probably found your problem right therel.
6. Upgrade to drupal 5.0 beta 2 and take advantage of its aggressive caching -- they make a HUGE difference. Especially with sites that have a lot of logged on users (4.7 doesn't cash anything besides modules and menus -- as far as I know.
--
"I'm not concerned about all hell breaking loose, but that a PART of hell will break loose... it'll be much harder to detect." - George Carlin
--
http://www.nicklewis.org
updates
OK, some updates:
First, that mysql page that I initially hadn't found was a gold mine. My default mySQL settings were totally wrong. After having made some simple config file changes, my site is faster than it has been for a long time. Nothing is for sure until the max load gets hit tomorrow, but right now my site is acting faster than it has for a very long time.
For the benefit of those who read this in the future seeking help on performance, I would emphasize that the db could be the problem even if it doesn't look like it (from "top," etc). A crash course in how to tweak the db is here; an extremely helpful "shortcut" is here.
I will also implement the other non-mysql suggestions to really optimize performance.
And a couple of followup questions:
Nick, thanks for the suggestions and I will make sure I follow them. (Though I will probably wait for gold release on Drupal 5 if performance is as improved as I think it will be... that's really good to know caching gets even better though). I don't know what CCK or views are -- does that mean I am not using them?
Matt, I wanted to ask you a followup on your tip about optimizing apache processes so they don't swap. A typical line from ps will look like this:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMANDapache 7186 1.0 0.2 35008 22484 ? S 21:01 0:01 /usr/sbin/httpd
As I understand it, the "RSS" field (22 megs) is the physical memory and what I should use to calculate my max apache processes. But even though I set the max processes nice and low, I see that the VSZ field, which I understand to include swap space, is at 35 megs. Did I screw up, and apache is still swapping? Or do I misunderstand the ps output?
One other general question for folks: as thrilled as I am with the improvement in page load time, I notice that cron.php still takes a long time to run - about 8 minutes. Is that normal? I don't get a whole lot of new content to index -- there are a lot more reads than writes - so it seems something may still be wacky with cron.php -- any ideas?
OK, I'm going to go back and implement the suggestions that I haven't done yet. I will report back tomorrow during primetime with an update on performance. Thanks for all the great suggestions... you have all salvaged what would have been a frustrating day.
Rich
Cron, optcache, query cache, slow query log
For the cron issue, open each module and look for "_cron(" to identify which modules implement hook_cron. Try disabling these modules one at a time (or commenting out the cron hook) and running cron manually, logging the different run times. You may have a misbehaving cron that is doing evil.
What operating system are you using and how do you install new software? Do you think you'd be up to the task of installing something like APC? It could make a huge difference in processor load.
I didn't see mentioned above whether you've turned on your MySQL query cache?
You might want to let the site run with the slow query log turned on and post any offending queries here. It is also worth the effort to use devel moudule and see if there are any blatant problems like duplicated queries.
Oh, and please tell us which blocks are enabled. Unfortunately, Nick Lewis was not totally correct about the "recent comment blocks", it becomes more expensive linearly in proportion to the number of comments you have on your system. See this issue for more details:
http://drupal.org/node/69111
If your site is forum/comment heavy, it would be a good idea to update to the 4.7-dev (and later to 4.7.5 when it comes out) because of an optimization that was made to comment loading that makes the process 23% faster:
http://drupal.org/drupal-4.7.x-dev
- Robert Douglass
-----
Lullabot | My Drupal book | My Digg stories
I stand corrected.
I stand corrected. :-)
--
"I'm not concerned about all hell breaking loose, but that a PART of hell will break loose... it'll be much harder to detect." - George Carlin
--
http://www.nicklewis.org
Thankfully the improvement patch went in
So while the "recent comments" block still gets slower in relation to how many comments you have, it gets slower much less quickly =)
- Robert Douglass
-----
Lullabot | My Drupal book | My Digg RSS feed
thanks for this
We are about to go live...
performance seems to be a big issue.
of all things:
1. pictures take a long time to send
2. paths/redirect seem to take a long time
we are not using jsaggregation and css optimization. but i am concerned.
I was wondering if anyone knew how to setup a cron to cache pages nightly.
Chris
Static page caching
For the fastest possible caching for Drupal 4.7 (and not even 5.0's improved caching comes close), you could try out the Boost module which caches Drupal pages as static files that can be directly served by Apache without invoking PHP, Drupal or MySQL at all.
While the module is still in a test phase, it has proven solid on a number of sites (and yes, makes a big difference in resource-starved VPS and shared hosting settings).
--
Arto Bendiken -- author of drush, Trace, Boost and Timeline.
update
Hi everyone - Well I am very pleased to say that as of this morning site performance is still quite good considering the fact that we are into "rush hour." I think the DB tweaks were extremely helpful. I've noticed that there are fewer httpd processes, and they use up less memory... this could be because of the browser-side caching Matt suggested or maybe because the DB improvements suggested by VM caused the httpd processes to spend less time waiting around. Not sure. But I am pretty sure that the DB was the primary thing because when I made those changes (most of which are detailed on the db optimization shortcut page above) performance really improved.
Cron is still acting weird... I had it set to go off just once last night, at 4am, and starting at that time I started to get either "cron run completed" or "lsat cron run didn't complete" messages every 20 minutes or so until 6AM, and then they stopped. Very weird.
Will post another message to keep things readable...
answers
Robert and Arto, thanks for the info.
Robert, that's a great suggestion on testing which module is causing cron to get weird. I will try that tonight or tomorrow night (might not be able to tonight).
I did turn on my mysql query cache (which was enabled, but cache size 0 by default, so was effectively disabled) and I think this made a huge difference.
Here are the blocks I am using:
- navigation
- login
- a couple of "custom" ones that don't do any calculation, but just display text
- active forum topics
- aggregator (on front page only)
Even though the site performance has improved substantially, it looks like there is still more room for improvement via APC, upgrading (I do get a lot of comments), and Boost (thanks Arto - I get a lot of anon traffic so this would help a lot). It is excellent to have a list of things that can potentially improve performance even more.
First, though, I want to find out what's going on with cron - i will post back when I figure it out, in case anyone is still interested or benefitting from this info.
Thanks everyone for the suggestions... I will be back soon with more updates.
Have a great day!
Rich
one more thing...
I thought I would just share a sight that brings tears to my eyes:
-bash-3.00$ uptime09:05:39 up 6:28, 0 users, load average: 0.51, 0.52, 0.52
At this time yesterday, those load numbers were in the mid-single digits...
rich
glad to have helped. with
glad to have helped.
with issue of cron.php (grasping at straws here)
have your tried uploading a fresh cron.php file ?
I know sounds goofy, but worth a shot.
More cron suggestions
Are you sure that you know how it is getting called? You can make it non-executable to prove that no module is invoking cron hooks. You could change it to mycron.php and update your crontab so that you know that you are initiating it. You could post your
crontab -loutput here in case you've made a mistake with the definition.- Robert Douglass
-----
Lullabot | My Drupal book | My Digg stories
yes, I am the only one calling it...
Hi Robert - I changed the name of the file a while back on the suspicion that someone/thing else was calling it. But every time it gets called, it's with the new name I gave it.
Well here is some further info, on the subject of cron. I tried your suggestions of disabling a module at a time, and the guilty party was search. Even if the site was fully indexed, the search_cron was taking forever to run (and presumably re-invoking itself somehow????).
So not knowing what else to do I rebuilt the search index. I then ran cron and it appeared to execute a lot faster than before, so perhaps the index had gotten corrupted somehow. I will have to wait for tomorrow to see if the extra "cron run completed" messages are still appearing, but I have a suspicion/hope this may have fixed it.
VM by the time I read your suggestion, I'd tracked down the source, so I think my cron.php is ok...
Thanks guys... I will report back on my findings.
Rich
Accelerators
I've recently found eAccelerator (php-eaccelerator) works FANTASTICALLY and, depending on your system and its setup, is an absolute doddle to install. To give you an idea of how "good" it is... Here are some state from a test site setup:
On top of this, the server has 2 other sites on it getting up to 7000 unique users a day (about 16,000 hits). This was causing us a server load average of about 0.1 (we have a seperate MySQL server). As soon as php-eaccalerator was installed the server load started flat-lining at 0. Yup - zero (or as good as). We were running Redhat EL v4 and I think to install it all I had to do was "up2date php-eaccelerator" and then enable in in the config in /etc/php.d/eaccelerator.ini.
I dont get why php-eaccalerator isn't installed by default!!
update
Cool, I will check out that accelerator...
As an update: the cron did mysteriously run one extra time during the night; something may still be whacky.
And I'm sad to report that my performance, after blazing for two days, suddenly went to hell. It happened right after I submitted a search query. Coincidence? I don't know... for now I am turning off the search module until it is a better time to test things out (eg. not in the middle of the day.)
thanks,
Rich
update time
I would like to present my findings since the last post. (Anyone still reading? ;-)
First, of all, I absolutely had a problem with search. I even tried manually emptying the search tables and rebuilding, but at some point, the index build got all weird, cron kept "repeating itself" (as described above), and server load got notably higher. Once I disabled search, all cron weirdness went away and performance was great.
After poking around the forums, I discovered "trip search." I installed it and not only is it blazingly fast by comparison, but I like it a lot better - it's got better functionality for the nature of my site's content. And of course no more choking on an index build because it uses native mySQL indexing.
So with that, I think my problems were solved.
But since I was having fun I decided to install Boost because I do get a lot of anon traffic, and especially at those times I am occasionally "slashdotted" it's all anon traffic. To log out out and watch those cached pages (even with lots of images) load instantly is truly a joy to behold.
I did discover a little bug with Boost, I think, but I will report that in the proper venue.
With that, my site's performance is now so much better than I ever expected. So I decided to be a wuss and not install a php accelerator... I was reading on this site about eaccelerator casuing apache to coredump, and while I'm sure it's all tractable, my system admin skills are pretty weak so I don't want to mess with it unless I have to (which I don't).
So, we've left it at optimizing mysql cache, optimizing httpd max children, tweaking the htaccess file (all described above), in addition to swapping trip search for regular search and installing boost. And with that my site's performance has improved immeasurably. (Well, you could probably measure it, but you know what i mean).
Thanks so much to everyone who contributed their great ideas. It's helped me a lot and hopefully will help future forum searchers who come across this thread as well.
Rich
Thanks for reporting back
Thanks for reporting back Rich, I was following this discussion and learned a lot out of it.
Also thanks to all the people who took part.
Woohooo! We love happy endings =)
- Robert Douglass
-----
Lullabot | My Drupal book | My Digg RSS feed