Hello,
I am having a lot of trouble running a very large Drupal website. On a regular basis - more that 4 or 5 times a day - my server dies on me. More specifically MySQL dies. I am quite frustrated, especially when GoDaddy says that Drupal cannot handle such a large website, but I think to myself that drupal.org is pretty darn big, so why the heck not? Save the Blocks, NONE of the data will ever change, people are not adding to the data (like comments) and I only occasionally change the contents of the Blocks.
Facts about the size:
- 646,262 records in the node table and 152 megabytes,
- 646,264 records in the node_revisions table at 8.7 gigabytes,
- 9.2 Gigs after clearing cache,
- Up to 23 Gigs when cache block, filter, form, menu page and update have filled up for a few weeks.
- Here is a snapshot of the database: http://openjurist.org/DRUPAL-DATABASE-HELP.png
On a Dedicated server at GoDaddy with the following configuration:
- PHP 5.2.9 - PHP Info: http://openjurist.org/test-server-settings/php/superinfo.php
- MySQL database 5.0.77
- PHP Memory limit of 128 MB
- Apache/2.2.11 (Unix)
- 1 Gb Ram
- Intel Celeron 2ghz processor
- 120 GB Hard Drive
- Cent OS 5 server
- Plesk Version 8.6
I am running the following modules:
- Boost 6.x-1.0-alpha
- Color
- Database Logging
- Help
- Menu
- PHP filter
- Update Status
- Pathauto
- Taxonomy Menu
- Teaser by Content Type
- Token
- Token Actions
- Taxonomy VTN
I am having trouble with a couple of modules:
- Boost - http://drupal.org/node/410730
- Taxonomy VTN - http://drupal.org/node/389780#comment-1439762
I have a few questions:
- What can I do to make the website perform better, e.g. not crash and be stable?
- Do you suggest using some different/additional caching program? eAccelerator?
- I don't know what exactly is in the node revisions table - is that where the data is stored or is the data in the node table? I don't need to maintain the old revisions of the data, I just need the current version of it. Does node revisions have multiple copies of the node data in old revisions? Can I dump it safely?
- When I remove a module, the tables seem to stay there. Are there any tables that I can safely discard given the modules I am running? See database snapshot: http://openjurist.org/DRUPAL-DATABASE-HELP.png
- The watchdog table can get pretty huge - mostly with page not found errors for CSS and JS files. Are my users not seeing the website like I am - are they not getting the CSS/JS files:
Count Message
676 sites/default/files/css/186c2cd9cfe00133dbc0f501c33c ...
669 sites/default/files/css/89bd6f888e41519f42215d5319ca ...
658 sites/default/files/js/2c21a3141715d1b074c8e9318fb9f ...
289 sites/default/files/css/b969f1c5470c54e5d704a0d89c90 ...
285 sites/default/files/css/e686e126b531f5c4a5c3c20dc40d ...
267 sites/default/files/js/ef769a50c31812b847c82baee9995 ...
234 sites/default/files/css/bf662c7c4164ecd3f94b44f246e8 ...
Your help is appreciated.
Comments
what i suggest is: disable
what i suggest is:
disable database log and enable syslog(if u need it)
try something like memcache
if the search is used alot try sphinx
dont know about your errors at boost and T.VTN
maybe u need better server
Thanks for your suggestion...
Can you tell me the difference between database log and syslog?
Does Db log go into Watchdog?
Where does syslog show up? Does it take MySQL resources?
We don't have our own internal search - we use Google Search to save the Db.
I have been thinking about getting a better server as well.
check this out
http://2bits.com/articles/drupal-performance-tuning-and-optimization-for...
http://wimleers.com/article/improving-drupals-page-loading-performance
What is new in Drupal 6 is that logging is pluggable and you don't have to use database logging if you don't want to. eg you can save load on your DB by switching off database logging (which can be a bottleneck), and turning on logging to syslog (or some other contrib logging module) instead. from http://drupal.org/node/330152
also see if google searches your site too many times in a week for example and if it does you can change(only lower) that.google it for more info
drupal.org
Well I don't know how your traffic compares to drupal.org, but drupal.org doesn't run on the equivalent of a modern low end laptop. d.o uses multiple web servers, a couple of large DB servers, a reverse proxy etc and a lot of custom DB tuning.
Without knowing what your traffic levels are (especially authenticated traffic), it may be that you have enough hardware. But I doubt it.
No. The node_revisions table contains the nodes content etc. You only have 2 more revision records than node records, so there isn't any room to delete stuff there.
Have you uninstalled the modules after disabling them? Properly written modules have an uninstall routine to remove their tables.
Now that great computers are
Now that great computers are relatively cheap, I have considered getting my own server and co-locating it. But i have never done anything like that and don't really know the really know how to go about it.
We currently have about 50,000 visitors a month. It is impossible to see how my resources are being used up with GoDaddy. It is very frustrating.
Is there a module for re-indexing the Db to make things run smoother?
As far as uninstalling modules, I have unchecked them from the modules page and deleted them from the sites/modules folder. Is that the proper way to uninstall?
tuning
That depends on how much sysadmin abilities you have.
The stats of interest to get an idea of load are peak page views per min or sec, and what fraction of those are authenticated. In saying that though - 50,000 visitors a month isn't that high.
You do have a very high number of nodes though - the table joins might cause MySQL some problems.
MySQL would probably be happier with more RAM, and some tuning.
You can use the OPTIMIZE TABLE command in MySQL, to tidy up some minor things. It's useful when you have deleted a lot of data from a table (a bit like a defrag).
No. After you have disabled them (ie unchecked them), if you want to remove their config/data you need to go to the uninstall tab to run the uninstallers for the disabled modules. If you delete the files before this, Drupal won't know about the uninstall routines and the data/config will hang around.
Most importantly you should
Most importantly you should install a bytecode cache. Either eaccelerator or APC. Both work well. Then you should have a look at why mysql crashes. What do the log files of mysql tell you? Do MySQL tables get corrupt? Which Storage Engine do you use (MyISAM or InnoDB)? Have you enabled the slow query log? Do you have RAM or CPU problems? Is your mysql tuned to the number of visitors on that page? Is the query cache enabled? Do you have enough processes, is your wait time too long? Perhaps you have filesystem problems? There is not enough information in your post, honestly.
Good idea... we are doing the
Good idea... we are doing the InnoDB ... let's see how it works out...
Yeah Watch dog can be a problem.
I had "forgot" to set a cron job and my watchdog table grew to over 4GB and kept crashing. It got so bad that my shared hosting provider had to shut my site down due to it was causing problems to other sites. Watchdog was the problem. I emptied it and it was smooth sailing from there. I lost about $2K in sales while my site was down. :(
Sorry to hear about your
Sorry to hear about your problems.
I am not sure whether you mean just the database/content size is large or whether the traffic volume is sizeable too.
Anyhow, either way here is what you need to do for sure:
1) Enable PHP APC
2) 1GB Ram is quite low, make it at least 4 GB
3) Change all database tables to InnoDB
4) Index all database columns where you have major join queries
For the long term, get a better server:
www.rackspace.com provides some of the best servers that is scalable with perfect support.
Look into having a second server that servers as a pure database server.
You can check out my case study http://drupal.org/node/422260 for more info on how Drupal can be further optimized.
Thanks for your advice. We
Thanks for your advice. We have installed APC, bumped up to 2 gb ram and gotten a 3.0 Ghz Pentium 4.
Your case study was very useful. If these updates don't fix it we will need to go the next step with your case study.
Thanks arnold.
_
Maybe it's just me, but imo "very large website" and "godaddy" just don't belong together. =/
lol agree
lol agree
Pfft, GoDaddy is clueless. Is
Pfft, GoDaddy is clueless. Is that -really- a dedicated server with a celeron processor?
First I would go through all the MySQL and PHP configs and make sure they are streamlined in accordance with those echoed by 2bits and other such dev shops. Secondly (immediately) I would see if they can add 2+ GB to the server.
Node revisions table is the table that holds the body/teaser content for a node. The node table is the 'master' table of nodes, and has no content. Don't dump either.
I've heard of issues of using Plesk, especially when it comes to upgrading PHP, MySQL and other server level applications.
You may want to look into Squid and eAccelerator.
Some modules don't come with uninstall routines (this should be a mandate...) to remove their database tables or variables data.
What we've seen is that MySQL chokes on cache data. Try having it empty cache every hour. Cache should never be GBs worth of data... no wonder MySQL is dying.
One question...
"646,262 records in the node table and 152 megabytes,
646,264 records in the node_revisions table at 8.7 gigabytes"
What kind of site is this?
===========
read my thoughts
When you say empty cache
When you say empty cache every hour, do you mean truncating or deleting all the data every hour? Or can I just run cron?
Here is the URL: http://openjurist.org
Is there something fishy with having 152 and 8.7 gigs?
seems down from here
seems down from here
Everyone here seems to hate GoDaddy
Who else can host this site without costing an arm and a leg. RackSpace is VERY expensive. $4800 for their most basic server -which is much better than the server that server that I have at GoDaddy - but hella expensive.
Which host do y'all suggest?
Hmm.
Hmm.
SliceHost (A+)
HostMySite
Mosso
HostGator
When push comes to shove, short term, I would probably hop into MySQL and do a TRUNCATE _____; on the cache tables.
===========
read my thoughts
THE PLANET SOFTLAYER it
THE PLANET
SOFTLAYER
it depends if u want a managed or unmanaged server
Is your mysql instance hosted
Is your mysql instance hosted on the dedicated server, or does the dedicated server only serve httpd traffic? I think you may have your memory limit on php set a little too high (128mb is a bit excessive, you could probably get away with much less). As noted, if your mysql instance is on your dedicated server, you're way under spec. What's probably happening is your db is querying for data, the db needs to join a few tables, hits swap, and then commits suicide.
Also, keep in mind that innodb requires more resources than myisam. Until you convert over to a bigger, better machine, if you are in fact running mysql with big tables like yours on a server with only 1gb of ram, you should probably keep your tables as myisam. You could end up increasing your crashing by converting without the proper hardware in place.