After upgrading to Drupal 6.20 from 5.22, I experienced two Apache 2 out of memory errors. The site became inaccessible. I'm not sure if it's anything on my end. The only way I was able to get around the errors was reboot the system from my VPS's dashboard. I did not make any LAMP configuration changes after the upgrade.

I'm using Ubuntu 10.04 LTS 64 bit. Any insight would be appreciated.

--- "RSS" stands for "resident set size," one way to measure the amount of memory used by a process in Linux. It does not refer to RSS feeds. ---

Killed process 18762 (apache2) vsz:187372kB, anon-rss:20172kB, file-rss:380kB
Out of memory: kill process 18740 (apache2) score 46907 or a child

Killed process 18740 (apache2) vsz:187628kB, anon-rss:19900kB, file-rss:576kB
Out of memory: kill process 18739 (apache2) score 46974 or a child

Killed process 18739 (apache2) vsz:187896kB, anon-rss:20400kB, file-rss:0kB
Out of memory: kill process 18723 (apache2) score 47043 or a child

Killed process 18723 (apache2) vsz:188172kB, anon-rss:20680kB, file-rss:660kB
Out of memory: kill process 18735 (apache2) score 47239 or a child

Killed process 18735 (apache2) vsz:188956kB, anon-rss:22036kB, file-rss:420kB
Out of memory: kill process 18758 (apache2) score 47239 or a child

Killed process 18758 (apache2) vsz:188956kB, anon-rss:20692kB, file-rss:400kB
Out of memory: kill process 18732 (apache2) score 47435 or a child

Killed process 18732 (apache2) vsz:189740kB, anon-rss:22000kB, file-rss:420kB
Out of memory: kill process 18745 (apache2) score 47569 or a child
Killed process 18745 (apache2) vsz:190276kB, anon-rss:23052kB, file-rss:508kB
Out of memory: kill process 18809 (apache2) score 47635 or a child
Killed process 18809 (apache2) vsz:190540kB, anon-rss:21476kB, file-rss:624kB
Out of memory: kill process 18784 (apache2) score 47633 or a child
Killed process 18784 (apache2) vsz:190532kB, anon-rss:21748kB, file-rss:624kB
Out of memory: kill process 18760 (apache2) score 47770 or a child
Killed process 18760 (apache2) vsz:191080kB, anon-rss:22212kB, file-rss:412kB
Out of memory: kill process 18786 (apache2) score 47770 or a child
Killed process 18786 (apache2) vsz:191080kB, anon-rss:23148kB, file-rss:308kB
Out of memory: kill process 18747 (apache2) score 47966 or a child
Killed process 18747 (apache2) vsz:191864kB, anon-rss:23864kB, file-rss:352kB
Out of memory: kill process 18748 (apache2) score 47966 or a child
Killed process 18748 (apache2) vsz:191864kB, anon-rss:24024kB, file-rss:320kB
Out of memory: kill process 18746 (apache2) score 47966 or a child
Killed process 18746 (apache2) vsz:191864kB, anon-rss:23296kB, file-rss:380kB
Out of memory: kill process 18767 (apache2) score 48100 or a child
Killed process 18767 (apache2) vsz:192400kB, anon-rss:25076kB, file-rss:612kB

Comments

prokopton’s picture

Here are some possible solutions I've found on my VPS's support page.

Diagnosing and Fixing Memory Issues

When your VPS is running low on physical memory, it may start to "swap thrash." This means it's attempting to use your swap partition heavily instead of real RAM. We recommend you limit your swap partition size to 256 MB; heavy use of swap in a virtualized environment will cause major performance problems.

We strongly recommend running the 32-bit version of your distribution of choice, unless you have a specific reason to run the 64-bit version. The 32-bit version of your distro will use significantly less memory.
Determining Free Memory and Swap Activity

You can use the following command to display memory use on your Linode:

free -m

You can use the following snippet to see a list of your running processes sorted by memory use:

ps -eo pmem,pcpu,rss,vsize,args | sort -k 1 -r | less

To see IO activity on your VPS, you may use the following command (you may need to install the sysstat package under Debian or Ubuntu first):

iostat -d -x 2 5

This will give an extended device utilization report five times at two second intervals. If your VPS is OOMing (running out of memory), Apache, MySQL, and SpamAssassin are the usual suspects.
MySQL Low-Memory Settings

In your MySQL configuration file (typically found in /etc/mysql/my.cnf), change your entries for the various settings shown below to match the recommended values:

key_buffer = 16K
max_allowed_packet = 1M
thread_stack = 64K
table_cache = 4
sort_buffer = 64K
net_buffer_length = 2K

If you don't use InnoDB tables, you should disable InnoDB support by adding the following line:

skip-innodb

Apache 2 Low-Memory Settings

Determine the type of MPM in use by your Apache install by issuing the following command. This will tell you which section to edit in your Apache configuration file.

Debian-based systems:

apache2 -V | grep 'MPM'

Fedora/CentOS systems:

httpd -V | grep 'MPM'

In your Apache 2 configuration file (typically found at /etc/apache2/apache2.conf in Debian and Ubuntu systems, and /etc/httpd/httpd.conf in CentOS and other similar systems), change your entries for the various settings shown below to match the recommended values.

StartServers 1
MinSpareServers 3
MaxSpareServers 6
ServerLimit 24
MaxClients 24
MaxRequestsPerChild 3000

Reducing SpamAssassin Memory Consumption

If you're filtering mail through SpamAssassin in standalone mode and running into load issues, you'll need to investigate switching to something to keep the program persistent in memory as a daemon. We suggest looking at amavisd-newExternal Link.
Troubleshooting Network Issues

If you've added multiple IP addresses to your Linode, you must set up static networking as described in the Linux Static Networking Guide. Please be sure to specify only one gateway. Using multiple gateways frequently causes problems.

If you just added an IP address to your Linode, please be sure to reboot before attempting to use it. This is required to properly route the IP address on our network.

If you've added a private IP address, please be sure to use the network settings shown in the "Remote Access" of the Linode Manager, paying special attention to the subnet mask. Note that private IP addresses do not require a gateway (nor should one be specified).

Valkyri9’s picture

Thank you for your comments @prokopton - I just started experiencing this issue today - most notably MySQL was upgraded over the weekend on my server along with other package updates, so I believe that was part of the problem. Thanks for the detailed configuration recommendations for my Drupal sites.

Cheers!
Valkyri9

Marko B’s picture

Is there some way to kill the process that is OOming or limit all process that this doesnt happen? I have vps and one of my drupal install did that and then 15 sites stoped working. Can something be done for future cases?

Marko

mdupont’s picture

Status: Active » Closed (won't fix)

This is a server configuration issue, not directly related to Drupal. If well configured the LAMP stack should not fill easily the entire memory (unless your server is really underpowered). The underlying OS itself (Linux most of the time) has a mechanism to kill "bad" processes when near an out of memory situation (OOM-killer), yet it often results in a non-functioning system. Yet, even without a OOM-killer, if the system reaches an OOM state it will crash anyway.

Marko B’s picture

How to install this "mechanism to kill "bad" processes when near..." tell me more please?

sridharpandu’s picture

Had the same issue on Rackspace. This post helped me fix it.

Marko B’s picture

"when near an out of memory situation (OOM-killer), yet it often results in a non-functioning system. Yet, even without a OOM-killer, if the system reaches an OOM state it will crash anyway."

So what to do then, you just described like everything is a bad solution :-)

b8x’s picture

This is usefull commands , but the problem still exist test ab.exe -n100 -c20 site/ kills my site, httpd takes to much memory. dont know how to reduce this. 600MB free memory = 0 users, while testing 135MB free.

Maxwell2’s picture

Hee

I am new to all this and have the same problem. I am using VPS and direct admin for my websites. Many times i get on the apache 'out of memory' I tried this useful post, but i am stuck.:(

I cannot get to the file where i change my Apache 2 configuration:
"In your Apache 2 configuration file (typically found at /etc/apache2/apache2.conf in Debian and Ubuntu systems, and /etc/httpd/httpd.conf in CentOS and other similar systems), change your entries for the various settings shown below to match the recommended values.

StartServers 1
MinSpareServers 3
MaxSpareServers 6
ServerLimit 24
MaxClients 24
MaxRequestsPerChild 3000"

And i cannot also change my MySQL to:
'In your MySQL configuration file (typically found in /etc/mysql/my.cnf), change your entries for the various settings shown below to match the recommended values:

key_buffer = 16K
max_allowed_packet = 1M
thread_stack = 64K
table_cache = 4
sort_buffer = 64K
net_buffer_length = 2K"

Thanks for you help!