Drupal eats all server memory
First of all I'm sorry about my english, but i've tried to get help on drupal.org.es (in my language) and I've failed.
Second of all, I'm not sure if it's the rigth place to post my problem, if it doesn't I beg you point me to the rigth place.
So here I go.
What the server is:
Dedicated server.
AMD Sempron 2200+, 256 Mb Ram, 80Gb. Windows 2003 Web Edition SP1.
What the server software is:
IIS 6.0 - PHP 5.0.3 as ISAPI module - MySQL 5.0 - Drupal 4.6.5
What it serves:
2 comunities in drupal (not the same code), 1 big and 1 small in diferent AppPools.
The big one, and the drupal wich eats the memory has ~6500 registered users and serves 14000 page-views each day. The database backup is ~72Mb.
Installed modules are:
adsense, attachment, browscap, controlpanel, feedback, filemanager, gsitemap, hof, members, nodevote, sitemenu, syndication, taxonomy_access, taxonomy_dhtml, tinymce, userpoints y xstatistics.
Themes are:
Frienselectric variation and mobile varition.
Active modules where (when the problem start):
adsense, aggregator, archive, attachment, blog, book, browscap, comment, contact, feedback, filemanager, forum, gsitemap, help, hof, locale (7 languajes), members, menu, node, nodevote, page, poll, profile, queue, search, sitemenu, statistics, story, syndication, taxonomy, taxonomy_access, taxonomy_dhtml, throttle (not configured), tinymce, tracker, upload, userpoints y xstatistics
Modules I've allready deactivated:
xtatistics, archive, browscap, help, agregador, hof, gsitemap
I've also deactivated 5 of the languages, 5 blocks and the access log feature.
I've raised the MySQL cache and max_connections in my.ini (and restarted the server, obiously).
Here is the issue:
Not depending on the user load of the server, the processes wich serve drupal (w3wp.exe) start eating tons of ram (more than 100Mbs each). It seems to be related with a fall in the ram eated by mysqld-nt.exe (by 10Mb or so). This ram consuption effectively blocks the server almost avoiding me to gain remote desktop access. The server gains lag and continue eating memory until it's restarted or the processes ara manually killed.
You can see the live server in:
The big comunity: beta.pesepe.com and beta.pesepe.com/mini
The small one: novedades.pesepe.com and novedades.pesepe.com/mini
So the questions are:
Why it worked so fine until January 13?
What can I do?
It's a documented issue?
The only fix is to change of host?
I'm absolutly hopeless seeing the server down day after day :_(
Thank you very much for your attention. I can't wait for your answers.
PD: I've tried the 4.6 version of devel module to throubleshot this with very few lack (btw how do I measure mem consuption?)
What happened on/before Jan 13?
Apart from what happened on/before Jan 13 (patch? upgrade?), this sounds like 1) a memory leak or 2) a trashing server.
1) In IIS 6 worker threads are kept in a pool and killed of once in a while. When the worker threads keep allocating memory your server will keel over after a while. The memory is released when the worker threads end. I'm not sure about the name, but you should be able to decrease the lifetime of the workers, in order to free memory earlier.
2) What is the CPU usage of the server? If it's too high, new requests will come in, before old ones are handled and finally the server won't be able to do anything usefull anymore except paging.
--
Tips for posting to the forums.
When your problem is solved, please post a follow-up to the thread you started.
Nothing happened
No new modules, no upgrade, no patch....
1) Yeah, I can schedule thread recycling in various ways, at a scheduled time, in memory consuption base, etc... But I've played widely with recycling with no success, since when IIS kills a thread it lets it close in x seconds and when a tread is killed another replaces it consuming another ton of memory (since it's a fresh thread it souldn't be affected by the mem leak)
2) The CPU usage of server is below 50% on average, never getting 100% for more than a second (nor when the issue matter)
I've been thinking in this 90% of time since 13th. The only thing that I can point and remark is that when mysqld-nt.exe relases memory drupal start conuming it in tons.
So can i have a wrong mySQL setting? could it be a temp table being writted to disk?
It's like mySQL doing someting and not responding drupal, the thing that mySQL does free its memory. That's the only clue I have :( (and i'm not experienced with mySQL configuration).
The page was living in that server for more than two weeks before the issue start.
Administering: Win 2003 SP1-IIS 6.0-PHP 5.0.3 ISAPI-MySQL 5.0-Drupal 4.6.5
Webmaster: www.pesepe.com-Beta.pesepe.com-gamers.pesepe.com
One thing I forgot to say
The comunity widely uses the drupal's private download system. I think each download takes about 8-10Mb of RAM while is being served.
Could it be related? In that case could the FileRequest module be a workaround?
Administering: Win 2003 SP1-IIS 6.0-PHP 5.0.3 ISAPI-MySQL 5.0-Drupal 4.6.5
Webmaster: www.pesepe.com-Beta.pesepe.com-gamers.pesepe.com
I don't have any suggestions
But I'm impressed that you were getting any type of good performace with 256M of memory. And you're running a graphical remote desktop? Aside from finding and fixing what is obviously a problem with the mix between the PHP scripts, the database and the web server, perhaps you could consider going up to at least 512M?
- Robert Douglass
-----
My Drupal book: Building Online Communities with Drupal, phpBB and WordPress
Nowadays I don't dream with performance but with stability
I agree 256 is very little RAM today, but we can't afford more yet :( We plan to switch as soon as posible to a bigger server (i think the end of february or maybe the start of march, more realistic in march).
But we need to keep it live and living (if posible growing) in that server until we can switch to a bigger one.
Yeah, i'm running a graphical remote desktop with no performance impact because windows is set to give priority to the serving threads. In the other hand doing something when the memory issue comes is like swimming in mercury, taking minutes to get things under control again (mostly rebooting windows), because windows gives drupal almost all available resources.
You say it's "obviously a problem with the mix between the PHP scripts, the database and the web server". I can't change the server machine rigth now so... I go on disabling modules :_(
Administering: Win 2003 SP1-IIS 6.0-PHP 5.0.3 ISAPI-MySQL 5.0-Drupal 4.6.5
Webmaster: www.pesepe.com-Beta.pesepe.com-gamers.pesepe.com
Like I said, sorry I have no suggestions
But please keep us updated with whatever you try and whether it works.
- Robert Douglass
-----
My Drupal book: Building Online Communities with Drupal, phpBB and WordPress
Thanks for your attention
Of course I will keep posting, I'm very grateful not only for the suggestions but for the attention and patience of reading this thread in "my english" ;) sorry if my coment sounds ungrateful, I think I need to keep improving my english.
Thanks again.
Administering: Win 2003 SP1-IIS 6.0-PHP 5.0.3 ISAPI-MySQL 5.0-Drupal 4.6.5
Webmaster: www.pesepe.com-Beta.pesepe.com-gamers.pesepe.com
No need
No need to be ashamed of your english; you wrote a very good post. Too bad no one could really help (yet). I hope you resolve the issue soon.
--
Tips for posting to the forums.
When your problem is solved, please post a follow-up to the thread you started.
I can't really add any help
I can't really add any help here. I run php4.3.x on a Windows IIS6 server, MySQL4.1 and don't have a problem. I do not run a site as large as yours. Just my small stuff.
The only things I can think of are things you are probably already doing. Checking the event logs, memory, process tracking, etc. Is it posible there was a MS security patch applied about the time the issue started? If so, then that would be a place to look.
-sp
---------
Test site, always start with a test site.
Drupal Best Practices Guide -|- Black Mountain
Before the issue it was an unpached W2k3
When the issue start the first thing I thougt was that must be a DOS attack so I patched the SO completely from SP1 to current. Unfourtnately the IIS logs probed me to be wrong :( (or I think so).
BTW I must say that I'm prety a newbie administering webservers and websites (I started to do it in a serious way last october) so feel free pointing me to the very basis ;)
Administering: Win 2003 SP1-IIS 6.0-PHP 5.0.3 ISAPI-MySQL 5.0-Drupal 4.6.5
Webmaster: www.pesepe.com-Beta.pesepe.com-gamers.pesepe.com
If the windows swap file is
If the windows swap file is on the same disk as MySQL then move it to another disk. This should reduce the cpu usage significantly.
Check to make sure that there is only one mysql.exe service running. On upgrades and experimentation windows will just pile up services and run them all. If you have more than one them you'll have to go the the command line to stop and remove them all then reinstall. This applies to apache.exe also
The server only have one disk
I've check for duplicates as you suggest. In task manager I see a mysql.exe(owned by admin) and a mysqld-nt.exe(owned by SYSTEM) processes. I think this is ok, isn't it?
Administering: Win 2003 SP1-IIS 6.0-PHP 5.0.3 ISAPI-MySQL 5.0-Drupal 4.6.5
Webmaster: www.pesepe.com-Beta.pesepe.com-gamers.pesepe.com
This may be the cause of
This may be the cause of your problem. Having only one disk on a server with heavy calls to a slow or low spaced swap will force everything into a ram memory que (increasing the swap space might help here). There is also the fact that mysql and windows are trying to use randomly available disk space to write files. This happens more when fat32 is used than in NTFS which has a better and faster allocation table.
If you have just the one disk then you may consider splitting the disk into seperate partitions and setting the swap file to this partiion so that it does not collide with mysql when trying to find write space. alternatively since mySQL 5 supports tablespacing you can move the tablespace to the new partion and achieve the same thing. You also may want to run a defragmentation or two of the hard drive.
Also remember that MySQL 5 is multithreaded so each connection will start a new thread and each thread uses more memory and consumes more cpu time. So if you have a lot of modules that hit the database then you may experience system problems with low ram.
Take into account microsofts consideration for running MSSQL (because mysql 5 is more like MSSQL than mySQL 4). They do not recommend that MSSQL be on the same machine as the webserver. They say Windows 2003 requires 256 in ram and MSSQL requires about the same. You may also need about 256 for other applications and the web server. So in all you need around 750 mb in ram to avoid problems.
Also you might understand that a new install of windows always use less space and memory. But the virulent nature of windows increases the need over time.
Thanks for your ideas
Ok, I'll try to go step by step.
When the issue first happened I resized the swap file (becuase allocating more swap is one of the slowest windows operation) and nowadays it's never completely used.
The server disk is NTFS.
Spliting the disk without having phisicall access to the server... ok, it's a idea, but let me try something different first ;) (It's a dangerous step to resize an active volume in windows i think).
About defragmentation I'm running diskeeper in the server in order to avoid fragmentatinon, Diskeeper launchs itself at least twice a day so fragmentation is keeped at a low percentage.
I currently have disabled almost every not core module.
Let me ask, what about disabling IIS logs in order to reduce disk accessses?
It's now clean for me that we need more ram, so we need to change of server as soon as we can afford it.
BTW I'm well experienced with windows and it's degradation over time, but in this case i think windows is not the one to blame about ;)
Administering: Win 2003 SP1-IIS 6.0-PHP 5.0.3 ISAPI-MySQL 5.0-Drupal 4.6.5
Webmaster: www.pesepe.com-Beta.pesepe.com-gamers.pesepe.com
has disabling Drupal modules made a difference?
You had an impressive list of modules at the start. If you're now down to just a few more than core, and if Drupal was the problem (or a rogue module), then you should have seen a major improvement by now. Has there been a decrease in memory usage in line with your decreased active modules?
- Robert Douglass
-----
My Drupal book: Building Online Communities with Drupal, phpBB and WordPress
Current active modules
By now theese are the active modules (note that when I say I've disabeld a module I mean disabling it in drupal, not removing it form modules directory)(not cores modules in bold):
adsense, attachment, blog, book, comment, contact, feedback, filemanager, forum, gsitemap, locale (3 installed languages - 2 active languajes), menu, node, page, poll, profile, queue, search, sitemenu, statistics, story, syndication, taxonomy, taxonomy_access, taxonomy_dhtml, throttle, tinymce, tracker, upload.
(gsitemap, was disabled for a while, but i reenable it after seeing the issue whit it disabled)
Drupal's cache is active (thowing errors, of course ;))
Clean URLs are disabled.
I've not noticed less memory memory usage. Next candidates for disabling are sitemenu, syndication and taxonomy_dhtml.
Administering: Win 2003 SP1-IIS 6.0-PHP 5.0.3 ISAPI-MySQL 5.0-Drupal 4.6.5
Webmaster: www.pesepe.com-Beta.pesepe.com-gamers.pesepe.com
Trashing
The best way to see if you server is indeed trashing (sucking mud), ie spends all it's time swapping pages is to look at the numer of 'hard page faults' per second.
Hard page faults are page faults where a page needs to be retrieved from disk (and may involve another page to be written to disk).
According to MS sustained hard page faults of > 5 /sec are indicative of a memory bottleneck. Process explorer, but also the normal performance monitoring tool can display the number of hard page faults as paging file read and write delta.
Scenario:
When this happens on a busy site, request cannot be handled quickly because the cpu is allmost all the time waiting for a page to get swapped into or out of memory. Hardly any work is done, leading to request pileup, continued memory allocation by the webserver.
See for a nice story Servers must not page.
--
Tips for posting to the forums.
When your problem is solved, please post a follow-up to the thread you started.
I'll try to monitor it. Wich option in performance monitor?
Since my OS is in spanish I need a little bit help in wich counter to activate in performance monitor.
I guess the object is memory, but I don´t know wich counter to choose (nothing like hard page fault ;)). Maybe Pages Writted/s or Pages Readed/s?
I assume if this is the problem the only solution is changing of server, is this correct?
Thaks.
PD: Today I can't monitor the server becouse I'll be out and I don't have a laptop.
Administering: Win 2003 SP1-IIS 6.0-PHP 5.0.3 ISAPI-MySQL 5.0-Drupal 4.6.5
Webmaster: www.pesepe.com-Beta.pesepe.com-gamers.pesepe.com
Guessing the translation...
I left the exact counter out, because my install is in Dutch (and XP)! You indeed need to select memory and then (guessed translation):
Page reads per second - Paginaleesbewerkingen per seconde (NL)
Page writes per second - Pagina's uitvoer per seconde (NL)
If you see this over longer periods of time, you need to put more RAM in. You can also log these performance counters for a day or two.
--
Tips for posting to the forums.
When your problem is solved, please post a follow-up to the thread you started.
Are you running a clean web
Are you running a clean web server or do you have some other items running in the backround like ASP.NET or any VB.dll's
The evening college here is running Drupal under Phalanger. Tehy are still running mySQL 4 but phalanger makes a busy Drupal run like lightning. Something you may want to experiment with at howm for use on your next server ;)
Very clean I think
Althougt Active Server Pages, ASP.NET, FrontPage Server Extensions, Internet Data Connection, Server Side Includes and WebDav ISAPI modules are installed it's Status is Forbbiden. The only ISAPI extension allowed is PHP. I think it's ok, isn't it?
I've never heared about Phalanger, but i'll follow it up and probably give a try in the future ;) thanks for the advice.
Administering: Win 2003 SP1-IIS 6.0-PHP 5.0.3 ISAPI-MySQL 5.0-Drupal 4.6.5
Webmaster: www.pesepe.com-Beta.pesepe.com-gamers.pesepe.com
Performance monitors should
Performance monitors should always be run from another machine so forget using it on the server. Taskmanager also causes lots of cpu overhead but atleast it is accountable overhead.
Yes, disk thrashing is a sure sign that something is wrong. The heaviest database usage modules are any of the taxonomy ones. What you should do is install the dev module and get a look at the taxonomy queries.
How large is your taxonomy and how many do you have? Are you running i18n module for spanish translation?
are you using aliasing in spanish and english so you have a large aliaisng table also?
Taxonomy is minimal I think
I know about the overhead caused by both taskmanager and monitors, but it's all that I have for now :( (suggestions on this will be also welcomed ;))
This is the current taxonomy:
Aplicaciones (editar término)
Emuladores (editar término)
Launchers (editar término)
Juegos (editar término)
Packs (editar término)
Tutoriales (editar término)
-- Tutoriales básicos de Homebrew (editar término)
-- Tutoriales para aplicaciones (editar término)
---- Tutoriales aplicaciones (editar término)
-- Tutoriales para juegos (editar término)
---- Tutoriales juegos (editar término)
-- Tutoriales para launchers (editar término)
---- Tutoriales launchers (editar término)
-- Tutoriales para emuladores (editar término)
---- Tutoriales emuladores (editar término)
-- Tutoriales avanzados de Homebrew (editar término)
And forum taxonomy:
Privado (editar término)
-- Webmasters (editar término)
-- Editores (editar término)
Web Beta.Pesepe (editar término)
-- Sugerencias y Críticas (editar término)
-- Dudas y Preguntas (editar término)
Mundo Homebrew y PSP (editar término)
-- PSP Noticias y Rumores (editar término)
-- PSP Tutoriales (editar término)
-- Usuarios Registrados (editar término)
Off-topic (editar término)
-- Off-topic (editar término)
BetaTesters (editar término)
-- Firmware 2.0 (editar término)
-- Firmware 2.01 a 2.60 (editar término)
Colaboradores Beta.pesepe (editar término)
-- Proyectos (editar término)
Do you think this is a big taxonomy?
I've used devel and haven't found slow taxonomy queries (perhaps I need to do it again browsing all pages instead of a set of it).
No i18n module installed neither aliasing use in this site.
Administering: Win 2003 SP1-IIS 6.0-PHP 5.0.3 ISAPI-MySQL 5.0-Drupal 4.6.5
Webmaster: www.pesepe.com-Beta.pesepe.com-gamers.pesepe.com
The right word is 'thrashing'
The right jargon for this phenomenon is "Thrashing" :-)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
My Drupal-powered Blog: ThoughtfulChaos
My spellng si way off
And now I've learned that trashing has an entirely different meaning...
So far for all those English lessons...
--
Tips for posting to the forums.
When your problem is solved, please post a follow-up to the thread you started.
Yur Englis is prety gud
Your English is pretty good. I often see you helping out in the forums (Thanks!) and I always thought English was your first language.
In any case if a web server is thrashing then the site is being trashed so maybe your mistake was somewhat OK :-)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
My Drupal-powered Blog: ThoughtfulChaos
Could this be related?
Surfing and rechecking pages I ended in this page: http://jeremy.zawodny.com/blog/archives/000173.html.
I not fully understand the things behind what is exposed there, but i tried the following queries:
<code>mysql> SHOW STATUS LIKE "%thread%";
+------------------------+-------+
| Variable_name | Value |
+------------------------+-------+
| Delayed_insert_threads | 0 |
| Slow_launch_threads | 13 |
| Threads_cached | 21 |
| Threads_connected | 2 |
| Threads_created | 157 |
| Threads_running | 1 |
+------------------------+-------+
6 rows in set (0.28 sec)
mysql> SELECT @@global.wait_timeout;+-----------------------+
| @@global.wait_timeout |
+-----------------------+
| 28800 |
+-----------------------+
1 row in set (0.03 sec)
I tuned the threads chached form 11 to 44, but i'm not sure of what i'm doing.
I've noticed when the issue last happened that threads where created and were "slow launch threads".
Could these mysql settings be related with the memory consuption?
Administering: Win 2003 SP1-IIS 6.0-PHP 5.0.3 ISAPI-MySQL 5.0-Drupal 4.6.5
Webmaster: www.pesepe.com-Beta.pesepe.com-gamers.pesepe.com
I've just finished exam time
I'm sorry for the time i have taken to follow up the thread, but you know things get harder in exam time ;)
Finally I achieve server stability with 3 significant steps:
1.- I disabled ISS loging feature (in order to reduce disk accesses and fragmentation).
2.- I scheduled database optimization to be doned each 3 hours.
3.- I rised the thread cache to 66 and lowered the connection timeout to 15 as sugested here "MySQL launch too many new threads"
The server have been up with no issues for more than a week so now I'm in the process of reenabling modules and blocks one by one. If the issue reapears I will keep posting ;)
By the way, I have renamed cron.php because it apears to be a security issue to have a public slow script with a well known name in a server, is not?
Thank you all for your advices and suggestions
Administering: Win 2003 SP1-IIS 6.0-PHP 5.0.3 ISAPI-MySQL 5.0-Drupal 4.6.5
Webmaster: www.pesepe.com-Beta.pesepe.com-gamers.pesepe.com