I'm not sure if the error we're seeing might be related to this issue (http://drupal.org/node/1303384), so I'm logging it as a separate item in case it's not.
We've been running Memcache 6.x-1.10 on a large site for a number of months, and we have noticed that we occasionally receive the following error message in watchdog:
Exception caught in dmemcache_get: Memcache::get() [memcache.get]: Server localhost (tcp 11211) failed with: Failed reading value response body (0)
When this happens, the server slowly starts to become unresponsive and then within 2 minutes it does not respond at all. We then see no server activity for about 30 minutes. After this time has passed, various cron jobs start to become responsive again (we are running Elysia Cron and have various cron activities scheduled throughout the hour). However we notice that the site is still not responsive because Varnish has reached a state where it no longer responds. Varnish at that point is still running on the server, but will not handle a request without restarting the service.
Here is the timeline of the most recent situation that we saw:
1:55pm: simplenews_cron job starts
1:55pm: simplenews_cron job ends
1:55pm: Exception caught in dmemcache_get: Memcache::get() [memcache.get]: Server localhost (tcp 11211) failed with: Failed reading value response body (0); severity: warning
1:55pm: RSS URL access denied (not a real page)
1:57pm: "/user/login" access denied (user is already signed in, therefore cannot access /user/login)
2:00pm: default cron jobs scheduled for :00 never run (is it locked?)
2:01pm: simplenews_scheduler_cron scheduled for :01 never runs (is it locked?)
2:02pm: "/user/login" access denied (another user is already signed in, therefore cannot access /user/login)
2:03pm: simplenews_cron scheduled for :03 never runs (is it locked?)
2:09pm: simplenews_cron scheduled for :09 never runs (is it locked?)
2:12pm: 500.shtml, site not responding
2:16pm: simplenews_cron scheduled for :16 never runs (is it locked?)
2:25pm: 500.shtml, site not responding
2:27pm: ubercart cron jobs scheduled for :21 finally run
2:28pm: default cron jobs scheduled for :00 finally run
So our cron jobs ran smoothly prior to the memcache issue, then we saw no successful logins, page access, or cron runs between 1:55pm and 2:27pm. We restarted Varnish around 2:35pm, at which point traffic returned to the server.
Any thoughts on what might be causing the exception, and how we can potentially mitigate the problem? We do not have stampede protection enabled as is suggested here (http://drupal.org/node/1303384#comment-5307476), and we are not using the memcache lock replacement (http://drupal.org/node/1303384#comment-5361018). Thanks.
Comments
Comment #1
sgdev commentedSorry, forgot to provide the essentials about our configuration:
We are using memcache.inc as our cache_inc, have a total of 6 bins for our memcache_servers (default, pages, blocks, views, content, path), and memcache_persistent is not set to TRUE. The post suggests persistent connections as an option: http://stackoverflow.com/questions/6489236/memcache-error-failed-reading... However I've yet to see the available memory for any of the memcache bins become overly excessive.
Also we are running two identical web servers which are load balanced, and each of them is running an instance of memcache. Might we be better served by instead running memcache on the database server, as recommended here: http://groups.drupal.org/node/191998#comment-653553 It seems as though that thread has examples of some running on the web servers and others running on the db servers, so I'm not sure it makes much difference.
We've used the persistent connections on other servers without issue, so it might make sense to give it a try on this site as well and see if it improves the situation.
Comment #2
damienmckennaDrupal 6 has not been supported for a long time.