#1103478: Skip some wildcard fetches attempts to reduce the number of memcache gets for wildcards. To do this, it has to always rely on the variable system to register full bin flushes (currently this only happens if you have minimum cache lifetime enabled).

While in testing this removes a memcache get from normal page caching, and should reduce the number of keys fetched via getMulti a fair bit overall, it means that every node and comment save triggers a variable_set(), which is an issue in itself. If you use something like strongarm module, variable_set() can trigger a full second or two of processing, and in D7 it's a read lock across the entire site for variable cache misses.

I'd like to refactor the current code, so that it allows modules to override the behaviour of cache_clear_all() - we could then ship with a default implementation (in memcache module or another extra module), or people could use http://drupal.org/project/expires instead.

One idea would be to set a variable in the future (say minimum_cache_lifetime * .8) - this would set a timestamp when the cache is going to actually be cleared (as well as the session check). Until that timestamp == time(), if another process clears the cache, we check the existing variable, and if it's still in the future, we don't bother updating it. Once the timestamp is hit, the cache is actually cleared, and the current minimum cache lifetime logic kicks in for the remaining .2 of minimum_cache_lifetime() (so you can still avoid a stampede - and memcache has its own stampede logic anyway).

Say you have 50 comments posted per hour, and a minimum cache lifetime of an hour, this would mean at most two variable_set() per hour instead of 50. It's not quite the same behaviour as the actual minimum cache lifetime, but I don't there's a good way to do this that will actually work for all sites, so making it hookable with the minimum scaffolding in memcache.inc seems like the best bet.

CommentFileSizeAuthor
#2 hook_cache_clear_all.patch1.01 KBcatch
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

catch’s picture

Status: Active » Needs review

Another approach could be this:

Disable cache_clear_all() entirely.

In cron, do SELECT MAX(timestamp) FROM node_revisions;

If the timestamp is greater than the last cache clear, and the last cache clear is older than a certain threshold, clear the page + block caches, otherwise do nothing.

The more I think about this, the more I think it needs to be handled at the level of cache_clear_all() than cache_clear_all('*', $bin, TRUE) - since if I call the latter directly, I do actually want memcache.inc to make an attempt to clear that cache in a timely manner.

With that in mind, here's a start on a patch, adds a new hook- hook_cache_clear_all() - if that hook is implemented, then it allows modules to do their own logic. If it's not implemented, it just clears the bins same as now.

Then we need to decide what (and if) the default solution in memcache.inc should be, could potentially be a couple of choices.

Yet another thing we could do is allow modules to set expires properly on temporary cache items (by bin), so that items in the page cache could be given an explicit expiry, that's a different issue though, although related.

catch’s picture

FileSize
1.01 KB
hefox’s picture

Subscribe: not sure if will be able to use it but might. Starting to look into memcache issues atm; right now the variable_set (memcache_variable_set) will expire the strongarm cache (I believe), which will rebuild the schema, etc... ow. Ideally I'd prefer if memcache wasn't using variables and used something else (#926062: Avoid variable_set() on bin and wildcard flushes mentioned something along those lines), though the other way to address it is look into seeing if strongarm could be doing items differently.

catch’s picture

DamienMcKenna’s picture

Version: 6.x-1.x-dev » 7.x-1.x-dev

Lets push this to D7 first and then backport.

DamienMcKenna’s picture