cache_clear_all and cache_get fail to clear caches when Minimum cache lifetime is on

not fully read this, but may be related to:
http://drupal.org/node/230374
and
http://drupal.org/node/226728

I hate patch core files. Instead, in my_module I have implemented hook_cron and mixed solution http://drupal.org/node/226728 with some additional code. I know, it is not elegant but afaik, there is no official core patch.

function my_module_cron () {
    $core = array('cache', 'cache_block', 'cache_filter', 'cache_page');
    $cache_tables = array_merge(module_invoke_all('flush_caches'), $core);
    $cache_flush = variable_get('cache_flush', 0);    
    foreach ($cache_tables as $table) {
      if ($cache_flush && !variable_get('cache_flush', 0) && variable_get('cache_lifetime', 0)) {
        variable_set('cache_flush', $cache_flush);      
      }      
      cache_clear_all(NULL, $table);
    }
}

Wow, I can't believe how broken the "Minimum cache lifetime" feature is.

I'm not sure I agree with you solution though, ilmaestro. It seems to me that the way it *should* work is that "minimum cache lifetime" means that items stay in the cache for at least X seconds even if they would otherwise expire -- not that it ignores any requests to clear the cache issued within X seconds. (I also don't think it's a good idea have cache clearing code to the cache_get function, but maybe that's just me.)

In other words, eliminate the cache_flush variable altogether and change the query to something like:

db_query("DELETE FROM {%s} WHERE expire != %d AND expire < %d AND created <= %d", 
        $table, CACHE_PERMANENT, time(), time()-variable_get('cache_lifetime', 0));

Edit: I see that, indeed, the documentation says that the minimum lifetime means that Drupal should ignore requests to clear the cache issued within X seconds. If that's the case, then cache_flush probably needs to store a timestamp for each cache table. (Though I think this is a strange approach -- is the idea that we're saving MySQL from being bombarded with frequent requests to empty the cache? I like my way much better :)

Same with 6 and 7 - minimum lifetime doesn't work as expected

This bug still exists in 6.9. For me, minimum cache lifetime is 1 minute and /cron.php runs every 15 minutes. However, table cache_form still has lots of entries that have expired a few *weeks* ago. This shouldn't happen after http://drupal.org/node/226728 cleans up the cache in system_cron(). The site is new so cache_form probably never get flushed.

Perhaps cache_clear_all() should take an array of tables and clear all of them instead of just one table. And cache_get() shouldn't do variable_set('cache_flush', 0) .

See http://lists.drupal.org/pipermail/development/2007-June/024441.html for a very clear explanation of "Minimum cache lifetime" . It's quite confusing. This bug still exists though.

Creating separate cache_flush variables for each cache table as ilmaestro suggested seems to work. I've tried that by replacing all 'cache_flush' in cache.inc with 'cache_flush_'.$table .

Thanks for your comments, ball.in.th. I was poking around in cache.inc for reasons of my own and came to similar conclusions, and was about to post a new bug but found this issue first.

Assuming we are not trying to change the caching model/concept here, there definitely needs to be a 'cache_flush_cache_xxxxx static variable for each cache table to avoid fairly basic problems with cache flushing.

Analysis: Suppose a minimum cache lifetime (i.e. cache flush delay) of say 10 minutes has been set. Suppose a cache flush is requested on cache_page. This sets the cache_flush variable to the current timestamp. Suppose an item is requested from cache_filter 10 minutes and 1 second later. (Or, suppose a cache flush is requested for cache_filter at this time.) Since time() > cache_flush + 10 minutes, old cache data will be flushed from cache_filter. Oh dear, cache_page has not been flushed and there is no longer a cache flush pending! (cache_flush has now been reset to 0).

How did this bug creep in: Drupal 4.7 introduced the cache_lifetime static variable, allowing for cache flushes to be delayed on high traffic sites. Drupal 5 introduced multiple cache tables, but failed to extend the flushing concept to handle the different tables independently.

The bug is present in 5.x, 6.x and 7.x.

I agree that it would be good to renami the cache_lifetime variable. From searching around this issue it is clear that there is a lot of confusion around the "minimum cache lifetime" terminology. On looking at the description on the admin/settings/performance page I was left in doubt a number of times as to whether a min. lifetime of <none> is actually the same as 0, which of course it is. As the post from the dev mailing list at #11 above explains,

the current logic simply adds a delay between when the cache flush request was made, and when it is actually executed

I'd therefore suggest the variable be renamed to cache_flush_delay, which is how it is (accidentally?!) referred to in a comment in http://api.drupal.org/api/function/cache_clear_all/7. Some rewording on the admin/settings/performance page and in the code comments would also help, and I'd be happy to help with this.

Is it too late to fix this bug in D6? It seems to be really easy to fix this bug and doesn't produce any side effects.

Normally this would be fixed in 7.x first and then the fix would be backported to 6.x and 5.x.

So it needs someone to write a patch, and for 7.x this will probably also need to include tests that demonstrate the bug and that the patch corrects it.

So when would this issue be addressed?

One of or our primary clients is suffering because of this issue, page caching does not work as advertised, neither does block caching. This issue may be compounded by how several modules were built in our instance. However, as stated in this discussion the issue has existed for a long time (4.7-5.x). I have spent quite a while researching the issue, in short of modifying the core every time an update occurs fixing it in the head would be the real answer.

Although not contributed to the the answer the following may be useful to others who stumble upon this discussion.

On high volume sites with a large number of anonymous and logged in users the caching system is horribly in antiquate.
I have found several caching schemes being developed in several custom modules. It would be great if the individual efforts were combined to focus on a concerted effort to improve caching across the board in Drupal. Here are a few modules (specifically built for 6) that relate to improving or tweaking Drupal caching.

Management/Utilities
http://drupal.org/project/cache_browser

Tweaks
http://drupal.org/node/370937
http://drupal.org/project/boost

Total Replacements
http://drupal.org/project/cacherouter
http://drupal.org/project/memcache
http://drupal.org/project/authcache

Let's see if this would get this bug fixed! ^ ^

When Minimum cache lifetime is on, only one cache table can be cleared. For example, for the following code in system_cron(), only table 'cache' would be cleared and the rest, i.e., 'cache_block', 'cache_filter', 'cache_page', 'cache_form', 'cache_menu' would *not* be cleared.

  $core = array('cache', 'cache_block', 'cache_filter', 'cache_page', 'cache_form', 'cache_menu');
  $cache_tables = array_merge(module_invoke_all('flush_caches'), $core);
  foreach ($cache_tables as $table) {
    cache_clear_all(NULL, $table);
  }

This patch should fix this bug by separating cache_flush variables for each cache table as ilmaestro suggested.

Still needs to be fixed in 7 first then backported. Re-rolled for 7 with concatenation fixes.

Could use tests but not looked at how feasible that'll be with minimum cache lifetime.

Minus syntax error.

#437952: db table cache_form not grows without getting cleared was a duplicate.

I think this is pretty much ready to be committed.

I'd suggest renaming the variable to cache_flush_delay_$table. I think adding the word 'delay' helps communicate what the variable does, and avoids some (but not enough) of the confusion that exists.

Should we write a test for this?

>I'd suggest renaming the variable to cache_flush_delay_$table
The current cache_lifetime variable is a global parameter that applies to all tables; this is the one that could be renamed to cache_flush_delay.

The new per-table cache_flush... variables (which are only used when the lifetime/delay > 0) store the time at which the flush was first requested. They could be renamed to cache_flush_timer_$table I guess. (A comment in the code says "... start a timer".)

Not sure which you meant?

I've committed the patch to CVS HEAD. We can talk about the semantics and refine the naming, but in the interest of moving this forward, and getting started on the backport to D6 and D5, I've committed it. Thanks for the patch.

Patch against DRUPAL-6

Fixed version - forget about underscore

Straight backport. Looks good.

Adding tag.

Committed to Drupal 6. In terms of follow up patching, I've noticed some misdocumentation on this comment, which is present on D7 as well:

       else if (time() > ($cache_flush + variable_get('cache_lifetime', 0))) {
         // Clear the cache for everyone, cache_flush_delay seconds have
         // passed since the first request to clear the cache.

Look, it is not called cache_flush_delay but cache_lifetime. Also, since this needs to go in Drupal 5 as well, lowering the version.

@Gábor Hojtsy I file an issue #468732: cache_flush_delay should be cache_lifetime

I'm a mysql dummie. Have phpmyadmin. But new to that, too. I am posting here because I'm wondering if this conversation relates to my problem. I got an error message this morning on my dev machine (winxp apache drupal 6.12) saying:

Table ;.\water\cache_form; is marked as crashed and should be repaired query: DELETE FROM cache_form WHERE cid = 'form_form-cb83334e4f2f1664f22841dde59c8329' in C:\apache2triad\htdocs\drupal\includes\cache.inc on line 179.

If it's related, can anyone tell me how to repair/fix? In phpmyadmin, structure display stops at cache_form. I have never done/don't know how to do a manual query to repair or check. Help?

Your problem is unrelated - it's specifically a MySQL problem (perhaps the server died during a table write). You need to go to the cache_form table in phpmyadmin, click on the Operations tab, then at the bottom do Repair table. HTH.

Subscribing.

Is the patch on #27 ( http://drupal.org/node/227228#comment-1503454 ) ready to be ported to 5.x and still pending?

Yes, #27 was committed to D6. The D5 version could also fix #30, or else #468732: cache_flush_delay should be cache_lifetime should also be backported.

Is someone currently working on the D5 backport? If not, I'd like to take a crack at it.

Please go ahead.
Tested the #27 changes on a 5.x site and nothing was broken.

Here the patch for d5 with fix from #468732: cache_flush_delay should be cache_lifetime

Committed to 5.x

Thank you for this great patch but what about the D7.21 users

returning this where it belongs 5.x version