Support for Drupal 7 is ending on 5 January 2025—it’s time to migrate to Drupal 10! Learn about the many benefits of Drupal 10 and find migration tools in our resource center.
By stephanbauer on
Hi all,
the server recently died a few times (swap exploded and didn't serve any requests) but since I've changed a few numbers in the configuration files everything seems fine (for now).
But the apache2 processes seem to be extremely big, each process takes about 200-300MB and I don't know how I can reduce that or if this is even possible.
So, there are currently 3 sites on this server having 3 separate drupal 6 installations:
- the public site (using memcache 6.x-1.8)
- the development site (only known to a hand full of people)
- and another site (only known to a hand full of people)
And because only a few people know the other two sites I don't think it is a problem having these extra code bases on the same server, but correct me if I'm wrong. I checked the enabled mods and couldn't find any to disable, I also disabled the UIs for some modules (e.g. views) so what else is there I could do?
Thanks a lot for any help,
sb
top
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
21593 www-data 20 0 548m 316m 228m S 0 7.6 28:43.80 apache2
15040 www-data 20 0 505m 307m 262m S 0 7.4 7:29.57 apache2
20553 www-data 20 0 538m 295m 217m S 0 7.1 28:24.97 apache2
31777 www-data 20 0 491m 292m 261m S 0 7.0 11:52.78 apache2
20483 www-data 20 0 521m 290m 229m S 0 7.0 9:46.53 apache2
21598 www-data 20 0 515m 290m 234m S 0 7.0 9:52.10 apache2
20475 www-data 20 0 486m 287m 261m S 0 6.9 11:03.25 apache2
25861 www-data 20 0 486m 287m 261m S 0 6.9 10:10.30 apache2
20809 www-data 20 0 483m 285m 261m S 0 6.9 10:28.34 apache2
15593 www-data 20 0 484m 284m 258m S 0 6.8 3:56.59 apache2
...apache mods enabled
alias
auth_basic
authn_file
authz_default
authz_groupfile
authz_host
authz_user
autoindex
cgi
deflate
dir
env
mime
negotiation
php5
rewrite
setenvif
status
php mods enabled
apc
curl
gd
mcrypt
memcache
mysql
mysqli
pdo
pdo_mysql
suhosin
xapian
APC
Uptime 3 days, 8 hours and 38 minutes
Free: 3.7 MBytes (1.4%)
Used: 252.3 MBytes (98.6%)
Fragmentation: 0%
Cached Files 2314 (220.8 MBytes)
Hits 64020298
Misses 16860
Request Rate (hits, misses) 220.59 cache requests/second
Hit Rate 220.53 cache requests/second
Miss Rate 0.06 cache requests/second
Insert Rate 0.06 cache requests/second
Cache full count 6
memcached
Uptime 5 days, 18 hours and 5 minutes
Free: 25.4 MBytes (19.9%)
Used: 102.6 MBytes (80.1%)
Hits: 7828688 (36.7%)
Misses: 13507506 (63.3%)
memory at devel_shutdown
30 - 40 MB
Comments
what OS? is it 64bit? i've
what OS? is it 64bit? i've seen this before on 64bit virtualised linux installs running Centos.
as a total hack stopgap, you might want to consider a auto_append_file that checks the size of the process and kills itself if the apache process is too fat.
Getting closer
Hi justinrandell, thanks for answering.
We are using Lenny, I think it's the AMD64 port.
Using size of ps (I stole from this site, thanks Nicholas) I get an average memory usage of about 30m. This sounds much more correct to me as we have 70 apache processes running currently without generating any swap on a 4g server.
So could it be that apache just takes the free memory without really using it? MaxSpareServers is set to 70 and I think now this is the reason why the processes are that big.
But then, how do I get good numbers for MaxClients, MinSpareServers and MaxSpareServers? Do I calculate the size of all other processes, add the 256m of APC and a little buffer just to be sure and subtract it from the 4g we have?
I used the following command to calculate the memory used by other pocesses:
ps axo 'pid user size cmd' | grep -v "\(grep\|apache2\)"
which is about 250m (memcached + rsyslogd, mysql is on a different server)
Lets say:
(4g total mem - 250m other processes - 256m APC - 250m buffer) / 30m average apache > 110
So 110 would be a good value for MaxClients I guess but what should MinSpareServers/MaxSpareServers be?
Thanks a lot for any help,
sb
APC is probably the biggest
APC is probably the biggest impact on the large RES numbers for processes, but as that other script shows each Apache process is only using 30-40MB of unique memory space (RES - SHR), since APC memory is shared. Without APC you would likely see smaller values from RES and SHR in TOP, but greater average memory usage per process (since Apache and PHP would be re-parsing every file on each request again).
Your formula for MaxClients looks good, but you may want to give a little bit more room if you haven't monitored the average Apache process size for very long. If you actually reach MaxClients and the memory requirements for Apache are slightly larger, the resulting swap activity could delay response time more than if they requests were simply queued by Apache.
Min/MaxSpareServers are useful to prevent constant creation and destruction of processes, and is more important the more 'bursty' your traffic is. Since your server is only running Apache, and there aren't other processes that would be choked out if Apache got busy, you can set MaxSpareServers to a large value so that the maximum amount of child processes are always running. In that case MinSpareServers would just control how fast Apache creates new children when restarted, and could be set to a moderate value.
Another helpful setting is MaxRequestsPerChild. If an Apache process uses more memory than average, such as by processing a large amount of data, its memory usage will not decrease after the request completes. MaxRequestsPerChild will cause the process to end after serving a certain amount of requests so that its extra memory usage can be reclaimed. There is a cost to killing and recreating processes though, so adjust this setting so that processes have a reasonable lifetime.
Also review your Keep Alive settings: if a process is holding a keep alive session open for a lot more time than needed it will sit idle, unable to serve anyone else until the keep alive closes.
Check Apache config options that deal with DNS/RDNS
We had these same symptoms come about on a non-Drupal system recently however, it was a Linux/Apache/PHP w/APC server w/12GB ram and the culprit was the use of a default Apache setting for TransferLog. Numerous httpd processes would grow over a period of minutes to be several hundred MB in the "RES" column heading shown in "top" for each httpd process. We tried adjusting MaxClients and MaxRequestsPerChild to decrease the lifetime of httpd processes and this helped slightly. But after switching the TransferLog directive to CustomLog with a LogFormat that didn't contain %h and instead used %a to avoid reverse DNS on each request, the problem went away immediately. It's worth noting that this was done in the ssl.conf file as well as the httpd.conf and every reference to %h was replaced with %a and HostnameLookups was also set to Off.
This really helped me out
Thanks gapple, your description really helped me take control of httpd processes which were slowly increasing in number up to 100% of memory for no apparent reason.
Same problem on Drupal 7 Centos 6 64 bit virtual server
Not really sure if the 64 bit Virtual CentoOS has anything to do with this problem. I am experiencing the same out of memory problem, also running APC. 2GB of RAM and nothing running on virtual server. The main "real" server is only hosing a few vm and none are very busy.
Just wanted to report the same problem.
I tried disabling all httpd.conf logging options. Used this guide to help:
http://www.mydigitallife.info/how-to-disable-and-turn-off-apache-httpd-a...
Not sure if it will help isolated the problem to logging by I will monitor and let you know.