Hello,
For about the last month or two, my site has been suffering from numerous crashes/server lockups (as in system frozen - even the console is locked). It's gotten to the point where if the system stays running for two days, I'm doing well.
The environment was SuSE 9.3 running Apache 2, PHP 4 and MySQL. After one particularily bad set of crashes, the database was saved (using mysqldump) and the machine was completely rebuilt. It currently is SuSE 10.0 running Apache 2, PHP 5 and MySQL.
It looks like the source of the problem may be from excessive bot traffic (mostly spam bots). The times when the server locks up is when a lot of traffic happens at once (in the last crash (about a half hour ago), around 150 connections were made in just a few seconds).
It looks like the session table is getting damaged. I'm running a regular series of mysqlcheck and (with mysql stopped) myisamchk.
So, anyway, I'm looking for any suggestions (I've been searching through other threads and found some ideas - so far no luck) on how to keep Drupal running.
Thanks,
-aw
Comments
This sounds like a classic
This sounds like a classic hardware problem and not a drupal specific problem. I would recommend doing a full battery of hardware tests such as memtest86 to see if the box hardware is stable. Unfortunately those tests require you to take the box out of commission for a few days to run the tests completely. For more ideas on how to test, a Linux support group (e.g. a SuSE group) would be more helpful than the Drupal forums.
To fix the bot problem, there are several possible solutions. In her PS to a Drupal consulting list email, Laura Scott mentions how she has battled this problem: http://lists.drupal.org/pipermail/consulting/2006-March/000691.html Maybe you can follow up on that.
Regards,
Greg
--
Knaddison Family Blog
--
Morris Animal Foundation
Agreed, sounds like hardware
Or possibly a driver/kernel issue.
Generally on a Linux machine with solid hardware, kernel and drivers, apps can't do anything to bring the machine down like that. The Suse guys are going to be a much better bet in diagnosing this.
--
Anton
New to Drupal? | Forum posting tips | Troubleshooting FAQ
Example Knowledge Base built using Drupal
I'm thinking you are right
I think you have it nailed down - the corrupt tables were throwing me off.
I will do some testing and see what happens.
Thanks for your help,
-aw
So... uh... (current status)
After several weeks of testing, replacing bits, testing replacing bits and more testing. Things are looking... Well.... pretty much the same.
Have you isolated it to
Have you isolated it to Drupal or just to heavy load? I would recommend trying other system "burn in" tests to see if the system just has problems under load. It would be useful to know what kinds of tests you did and which pieces you replaced.
Also, another test would be to get some reliable kit and then install Drupal on that reliable kit - hopefully you'll notice that it doesn't fall over very often.
--
Growing Venture Solutions
Drupal Implementation and Support in Denver, CO
--
Morris Animal Foundation
link change
Styro - you need to fix your signature links - stick a / on the beginning of the relative URLs and they should work.
Like so:
<a href="/node/35172" rel="nofollow">New to Drupal?</a> | <a href="/forum-posting" rel="nofollow">Forum posting tips</a> | <a href="/node/199" rel="nofollow">Troubleshooting FAQ</a>New to Drupal? | Forum posting tips | Troubleshooting FAQ
--
Growing Venture Solutions
Drupal Implementation and Support in Denver, CO
--
Morris Animal Foundation
OT: Hmmm
That's odd - they worked OK for me (with Firefox, Konqueror and Epiphany on Linux). What browser/OS were you using?
(I've changed them for this post)
--
Anton
New to Drupal? | Forum posting tips | Troubleshooting FAQ
Example Knowledge Base built using Drupal
I was using Firefox - and
I was using Firefox - and now both versions work (the original one I objected and the new version).
In the 4.7 update of Drupal.org I think that these relative paths had some potential problems. Maybe that was fixed at the same time.
Either way, problem solved, let's rejoice.
--
Growing Venture Solutions
Drupal Implementation and Support in Denver, CO
--
Morris Animal Foundation
Lockups not a hardware problem
Actually, I don't think this is a hardware problem.
My site, http://www.writingup.com, has had these exact same problems for months.
Below is the progression of servers I've had this site on, each of which had been thoroughly tested (the server) and had had other sites running on it smoothly for months previously:
1. Site started on a dedicated server with the database on the same box (crashes happened)
2. I moved the site to a Xeon 3.2 with 2gb ram (crashes happened)
3. I moved the database off to another box so the Xeon 3.2 just had drupal running on it (crashes happened)
4. I moved the site to a load balanced environment with a dual opteron with 4 GB ram for one of the webservers, and a few other lower hardware webservers. The database is also on a dual opteron with 4 GB ram. Crashes are still happening.
The only thing I've come up with that causes this is a spike in traffic (which is usually caused by bad bots).
When the spike happens, because of drupals high memory usage (on my drupal site it takes 12 Mb with every page load, 35 Mb when loading a "tags" page), the server runs out of memory and starts using the swap. If the spike goes back down, the server can recover. If the load continues at the high rate, eventually the swap runs out and it crashes the whole machine. I've had it run through 4GB of ram and 4GB of swap a few times now.
Today we're going to implement a bot trap to see if it will stop it from happening in the future. Other than that, we weren't able to solve the problem except by putting $14,000 worth of hardware on the site.
John
My blog
John - I don't think you can
John - I don't think you can say definitively that lockups are not a hardware problem - in many cases (certainly most, I'd say >90%) they are situations of "I had this old machine sitting around and I installed Linux and it crashed" and guess what, that old system has a bunk PSU and disk - hardware problems. In the case of Drupal, it's on 100,000+ sites in the world and the two of you (and a few others) are the only who have this problem so it's not like Drupal has an inherrent "server killing" bug in it.
Your server description isn't really useful without knowing a few things like hits per day, any performance improvements you've attempted on the configuration/databases, what your maintenance schedule is and such. From a quick glance I'd say 1) your site gets a lot of traffic - even more if you have lots of bots visiting 2) by using the pushbutton theme that makes me guess that you are using defaults for table types in MySQL and haven't done much if any performance tuning. Those two things together could easily make a server fall over after it fills the RAM.
I recommend in your situation reading up on the performance hints in the handbook and also that you use the devel module to identify the most performance threatening modules and then set those modules to be throttled.
--
Growing Venture Solutions
Drupal Implementation and Support in Denver, CO
--
Morris Animal Foundation
Was having same problem with crashing under heavy load
Autowitch,
We were having the same problem shortly after our multi-site Drupal installation launched - server crashing nearly daily. We have a WIMP installation, and after checking for server software and hardware issues, we discovered that within our own network (we're part of a government network) another agency had their Google Search Appliance spidering our main site VERY agressively - 74,000 page requests in 2 weeks, or 2.5 days of page generation time. Yeow! When their spider would hit the "recent popular content" pages produced by CCK (if I'm not mistaken) it would try to spider all the pager pages therein and it would crash the system.
Our fix: Temporarily, I've banned the offending bots by IP (at admin/logs/visitors), until I can get those bots dialed down. Additionally, I've disabled the popular content pages (at admin/views). And enabled Drupal's caching system, and institututed MySQL query caching at 20MB (helpful article at http://www.databasejournal.com/features/mysql/article.php/3110171) and installed the PHP accelerator eAccelerator (http://eaccelerator.net/ or see http://www.sitebuddy.com/PHP/Accelerators/eAccelerator_windows_binaries_... if you're on Windows like me).
Since then, page generation times for anonymous users have gone from 2000 ms to 200 ms, and for authenticated users, in general it's down to 1000 ms. To gather the page gen info for anonymous users, I've been using www.site24x7.com . For authenticated users, the Devel module works excellent.
We haven't crashed since, and I've tested our capacity by spidering our site myself using the Xenu link checker.
Hope this proves helpful to somebody.
Michael Samuelson
Idaho Commission for Libraries
Dotted i Design
mlsamuelson