memcache locking causes unnaturally high number of sets [#264440]

We rolled cacherouter out on our live cluster that was experiencing a lot of load (mostly slashdot) and noticed that after a couple of minutes of running the total number of sets would jump significantly above the number of gets - which isn't something you expect from a cache.

For example after about 3.5 minutes of running we would get these stats:
get: 20298
set: 44550
gethits: 15557
getmisses: 4741

Or a little later it'd look like this:
get: 63817
set: 184067
gethits: 55877
getmisses: 7940

Usually it would look normal for a minute or so then it would jump up by ~3000 sets in under a second.

This problem is specifically in the spin lock in the memcache engine:

while ($this->memcache->add('lock_' . $this->name, $this->settings['compress'], 0) === FALSE) {
  if (time() - $time >= 3) {
    return FALSE;
  }
}

This causes a few issues:

It makes it very hard to track the performance of memcache (it isn't just used by drupal)
As mentioned in http://drupal.org/node/245341 it cranks up the load on both the web server as it continually asks for a lock AND the memcache server which has to keep responding to these requests

Comment	File	Size	Author
#1	cacherouter.memcache_lock.patch	4.45 KB	triclops

Comments

Comment #1

triclops commented 30 May 2008 at 02:10

Status:

Active

» Needs review

Status	File	Size
new	cacherouter.memcache_lock.patch	4.45 KB

This patch solves this problem by removing the lock entirely and replacing it with an invalidation timestamp.

The only reason the lock is needed is to maintain the lookup table used when flushing memcache. Instead of using a lock this patch will write to memcache the time of the flush. Subsequent gets from memcache will check this timestamp and compare it with their created time. If the invalidation timestamp is after the created time the get fails - otherwise it returns the result as per normal.

This method of flushing assumes servers accessing memcache have synchronized clocks.

This patch was made against HEAD.

Patch developed by IDG Australia.

Comment #2

triclops commented 30 May 2008 at 02:17

Possible extension to improve this might be to cache the invalidation time in another cache, e.g. APC, or to at least make it configurable where this key gets placed (perhaps an special bin just for it :P).

Comment #3

andypost

he/him

Russian

commented 1 June 2008 at 07:06

This is not good idea to remove lock at all. I propose to set different cache-shared settings for tables.
Lookup tables only need for wildcard clearing (cache_pages, cache_filter)
I think to put shared option for all backends so only needed in wildcard clean tables got a lookup

Another idea use $expire as value of lookup table - deletes (sets) goes only on lookup and flushes in backend only on flush (by cron for exmpl)