We have a time-sensitive need for a consultant experienced in large-scale LAMP environments running web services.
Currently we have a serious CPU load problem in our LAMP environment that manifests itself in the form of multiple apache httpd processes taking up anywhere from 4-6% of the CPU, which are (Dual Socket Dual Core AMD Opteron 2214HE). Throughout the day we'll see the load increase on the CPU (not the RAM, mind you) from 30% to 50% to 70%.
We are running three drupal installations and services/xmlrpc on a dedicated server. The three installations are simply dev, qual, and prod for our client's website. Here's the crazy frustrating part: Average visits per day to our site is ~1300. Average users at any given moment of the desktop application is ~200.
So far we've deduced that the problem is coming from the web service side, not drupal's standard website processes. We see a LOT of sleeping processes designated by time_wait, and even after restarting apache there will be 30%+ CPU load with a plethora of time_wait processes abounding. The desktop application making calls to the web service does send spurts of calls in short time and we are working to do more work in-app to reduce the number of calls we have to make.
We've tweaked many of our standard settings including increasing lots of the default buffer and limit variables:
# up from 16M
max_allow_packet = 24M
# Up from 64M
key_buffer_size = 256M
# Up from 100
max_connections=200
# Up from defaults (8M)
innodb_buffer_pool_size = 384M
innodb_additional_mem_pool_size = 20M
# Up from 5 - 5 is a bit too slow for these settings
interactive_timeout = 300
wait_timeout = 300
---------------
These settings were changed at the recommendation of our host company's DBA support. We've also done the following:
* Enabled keepalives with a 2 second limit
* tw_recycle enabled... takes connections that have been sitting and closes them out a little faster than normal.
* within the destkop application that calls the web service using the Services module, decreased the # of calls made to the service.
For the life of us we cannot figure out what is causing our small-needs desktop application to be maxing out our CPU and creating this inexplicable amount of time_wait processes that seem to build and build toward a bottleneck. Note that our 4gb of RAM is fine and we've never used more than 2megs at any given moment.
Our server info:
Dual Socket Dual Core AMD Opteron 2214HE
Memory: 4GB DDR RAM
Redhat Enterprise Linux 4ES
Apache 2.0.52
MySQL 4.1.20
PHP 4.3.9 /no optimizer such as eAccelerator being used currently
Comments
bump
bump
bump
no idea, bump