Hi there,

I use Drupal 6 with the Cache Backport module, and all the available cache tables are configured to be stored in Redis. The caching works very well.

I have noticed that the amount of data stored in Redis gradually grows as the cache warms up, which is expected. However, the amount of stored data regularly drops suddenly to almost nothing.

I ran a test by loading up the cache by running a search for the word "the". Usually the amount of data stored in Redis will climb to between 20 and 40 MB on that query. Then, I ran the filter_cron job (I use the Ultimate Cron module and can run the different cron jobs selectively.) The Redis memory immediately dropped to almost nothing. After loading up the cache again to 20 - 40 MB, I ran the system_cron, which instantly resulted in Redis memory dropping even lower than it did on filter_cron. Both of these results are repeatable every time. So obviously both of the system and the filter cron jobs are indiscriminately flushing the Redis cache, which negates the purpose of a cache.

Can I prevent cron jobs from flushing the Redis cache, and if so, how? Is there not effective intelligent cache expiration logic on an individual key basis? Or would the cache system malfunction if I prevented it from being completely flushed by the cron job?

Thanks a lot for the help!

Comments

pounard’s picture

pounard
Primary language French
commented

The problem is that Drupal will pragmatically empty all bins on cache flush, whatever happens. Considering that with Cache Backport, we own the code, we could implement such mecanism, but with Drupal 7 it's not possible without patching core.

That said, we could still implement it into the Redis module, in the redis backend, but this would make its signature being somehow inconsistent with core, so I don't really want to do that. But in theory, you can do it and live with a patched version of the module.

I need to think of this and talk about this with some colleagues of mine using the module, I may do it, but not right now.

About the "how" part, it would be an ugly patch in the clear() method of the cache backend, checking for cron context and returning silent.

pounard’s picture

pounard
Primary language French
commented
Status: Active » Postponed

Does this answers your question?

rahim123’s picture

rahim123 commented

Hi, sorry for the delay. Thanks very much for looking into this. I would appreciate it if you could implement some sort of workaround for Redis.

I don't know if this helps you any, but here is my workaround: I switched to "Ultimate Cron", and disabled the "Filter" and "System" cron functions, which were responsible for flushing the caches. Then, I copied the part of the "System" cron function that does other necessary maintenance tasks, and copied it into a cron hook in a custom module. Then, I enabled that custom task in Ultimate Cron. This way, I don't have to hack the Drupal core.

js’s picture

js commented

Perfect @sb56637. I run ultimate cron which is how I found this tread.

Would you please consider sharing your custom cron hook and save me, and likely others, the time of hunting down what needs to be run. Or, did you just copy all of

function system_cron() {
without 

  $core = array('cache', 'cache_path', 'cache_filter', 'cache_page', 'cache_form', 'cache_menu');
  $cache_tables = array_merge(module_invoke_all('flush_caches'), $core);
  foreach ($cache_tables as $table) {
    cache_clear_all(NULL, $table);
  }

Thanks, Jerry

rahim123’s picture

rahim123 commented

@#4
Hi Jerry, sorry I was rushed when I posted my last comment and I didn't bother to post the entire solution.

I basically copied everything from the system_cron hook except the last lines that refer to cache tables. Notice that I am still on Drupal 6, so mine looks like this:

//implementation of hook_cron
function MYMODULE_cron() {
	// Cleanup the flood.
  db_query('DELETE FROM {flood} WHERE timestamp < %d', time() - 3600);
  // Cleanup the batch table.
  db_query('DELETE FROM {batch} WHERE timestamp < %d', time() - 864000);

  // Remove temporary files that are older than DRUPAL_MAXIMUM_TEMP_FILE_AGE.
  $result = db_query('SELECT * FROM {files} WHERE status = %d and timestamp < %d', FILE_STATUS_TEMPORARY, time() - DRUPAL_MAXIMUM_TEMP_FILE_AGE);
  while ($file = db_fetch_object($result)) {
    if (file_exists($file->filepath)) {
      // If files that exist cannot be deleted, continue so the database remains
      // consistent.
      if (!file_delete($file->filepath)) {
        watchdog('file system', 'Could not delete temporary file "%path" during garbage collection', array('%path' => $file->filepath), 'error');
        continue;
      }
    }
    db_query('DELETE FROM {files} WHERE fid = %d', $file->fid);
  }
}
pounard’s picture

pounard
Primary language French
commented
Status: Postponed » Closed (duplicate)

Cron run will attempt to empty the expired cache entries. In previous Redis module releases actually emptied the bins everytime this happened because of a stupid FIXME I left in the code untouched for monthes. Git version now handles temporary items properly and should work fine, resolving the original issue.

I am closing it now, please re-open if you still experience the problem using the git version (git version is stable, and does not break the API, most patches are preparation for the stable 1.0 release).

j0rd’s picture

j0rd commented

memcache module simply doesn't allow cron to clear it's bins.

This is ok for memcache since memcache has a memory limit (which you expect to get full) and then prune itself.

Memcache hacks in avoiding this call via figuring out which function is calling the cache clear from a debug_stacktrace()..

    // It is not possible to detect a cache_clear_all() call other than looking
    // at the backtrace unless http://drupal.org/node/81461 is added.
    $backtrace = debug_backtrace();
    if ($cid == MEMCACHE_CONTENT_CLEAR || (isset($backtrace[2]) && $backtrace[2]['function'] == 'cache_clear_all' && empty($backtrace[2]['args']))) {
    
     // .... code
     }
     if (empty($cid) || $wildcard === TRUE) {
       // system_cron() flushes all cache bins returned by hook_flush_caches()
       // with cache_clear_all(NULL, $bin); This is for garbage collection with
       // the database cache, but serves no purpose with memcache. So return
       // early here.
       if (!isset($cid)) {
         return;
       }
      // ... code

With Redis module installed, I thinks `drush cc all` needs to be a little more forgiving on what it prunes, or perhaps I just need to be more careful with what commands I run on my production server now that I have this module installed.

I just did my first cache clear and everything is gone, except for my bins I've created myself which specifically don't hook into hook_flush_caches()

rahim123’s picture

rahim123 commented
Issue summary: View changes

Hi there, now on the latest Drupal 7 with the Redis module at version 7.x-3.17. The entire Redis cache gets emptied during common actions like enabling or disabling modules, and during updatedb.

rahim123’s picture

rahim123 commented
Version: 7.x-2.0-beta2 » 7.x-3.17
Status: Closed (duplicate) » Postponed (maintainer needs more info)