Currently Drush Cr or Cache Clear UI does not flush Redis cache

> The problem is the expectation when running drush cr is that the entire cache is flushed at that time.

That's a problem with the expecation :)

The expiration is set to 1 year when it is defined as unlimited.

There is no API for a full cache clear, it is only called per bin. Additionally there's the prefix. We don't know that there are no other prefixes and we can't identify a full cache clear.

Instead, the way it works is that a deleteAll() on a single cache bin sets a cache entry that contains the current time. If an entry is requested that was created before then, it is ignored.

Hello,

I encountered a fatal error after updating a project because "drush cr" did not flush redis cache.

Here is a small patch that allows to do that.

Thanks for the review.

You can also use SiteCommander to flush the Redis cache. It's a Drupal sandbox project for now (still waiting for it to get approved), but you can snag it on GitHub if you want.

https://github.com/IncursusInc/sitecommander/tree/8.x-1.x-dev

Also, confirming that the patch provided in #2765895-4: Currently Drush Cr or Cache Clear UI does not flush Redis cache seems to work fine after I patched my local copy for testing.

Cheers.

Actually, the code doesn't take into consideration that other instances/apps may store something in the same redis instance. So, the implementation just deletes everything, for every instance, which is not something that is expected, or desired.

Setting back to needs work, but if your use case is more simple, with one app/site per redis server, the patch here will work for you.

Hello,

Yes, as I am not familiar with the Redis API and as I needed a quick solution, I don't go too far.

If a maintainer can tell how he/she wants a more sustainable implementation to be done, I will try to make a better patch.

Correct me if I am wrong, but all of the cached Drupal stuff is prefixed anyway. Should be academic to just delete those keys.

Cheers.

In our SiteCommander module, which will be published soon, we clear the keys out using $settings['cache_prefix'], and it only takes a split second to zap thousands of them. FWIW.

Attaching a patch that implements a fairly simple solution to the problem. Not perfect, but about the best I can come up with. Portions of this from our SiteCommander module (again, to be published soon).

You can read the comments in the patch to get the general approach taken. But the main caveat is that as long as they configure a 'base' value (database index from 0-15), all should be well since it deletes via their configured prefixes (or everything if they do not). And even that can be worked around as well if need be.

Cheers.

Also, not sure I submitted that patch correctly. The interface for doing so is not terribly intuitive. But feel free to test/incorporate/edit/etc in any event.

Cheers.

There are some minor Drupal coding standards mistakes, e.g. the 'else' statement should start on a new line instead of hanging at the end of the previous bracket, comments should end in a period, etc.

This might work if you have a few thousand keys, but what if you have a lot mor than this? This doesn't scale and it blocks redis.

I still don't understand the actual problem here. What exactly is the problem you see? Yes, the keys are not deleted, but the only problem with that should be that storage isn't freed up immediately. But eventually, on a sufficiently large site, redis will be full anyway. You should configure redis with some kind of eviction: http://redis.io/topics/lru-cache, e.g. allkeys-lru or volatile-lru. The second one should work fine as well, since all cache entries are set to an expiration of one year by default.

This should *not* affect caching as visible to Drupal. As explained above, we track for each bin the last full flush, and items older than that are discarded.

The only way to truly do this seamlessly is to ensure that the Drupal cache is the only application using the configured database index (0-15). Then, a simple flushDB() call does the trick quickly.

You *could* do something like query to see if there are any other keys other than prefixed keys, and if so, throw a warning and abort the flushDB() call. Otherwise, just call flushDB() and move on. And put it in the docs. :) That might work.

Cheers.

I think too that there is a problem with the end result expected: it seems that some items are not discarded when running drush cr or drupal cache:rebuild all.

Reproducible test case:
- Create a new template in a custom theme (eg: field--text-with-summary.html.twig)
- Run drush cr
- The template is not picked up.

Using:
Drupal 8.2.2
Redis module 8.1.x-dev using the PhpRedis client
Redis PHP extension 3.0.0
Redis 3.2.5 with allkeys-lru

Settings:

// Redis cache
$settings['redis.connection']['host']      = 'redis';  // Docker container.
$settings['cache']['default'] = 'cache.backend.redis';
// Always set the fast backend for bootstrap, discover and config, otherwise
// this gets lost when redis is enabled.
$settings['cache']['bins']['bootstrap'] = 'cache.backend.chainedfast';
$settings['cache']['bins']['discovery'] = 'cache.backend.chainedfast';
$settings['cache']['bins']['config'] = 'cache.backend.chainedfast';
// redis services
$settings['container_yamls'][] = __DIR__ . '/../../modules/contrib/redis/example.services.yml';

Willing to help, can I provide more infos?
Setting to major because side effects are impacting site functionality when doing continuous deployments, but correct if needed.

Most likely what you all had problems with is that the default cache prefix was different for drush and the browser, that's why drush cache clear didn't work: #2846208: Improve default cache prefix

This is fixed now in beta1.

Introducing a command to actually delete all the records could be added but that would be a new feature and running drush redis-flushall isn't easier than running redis-cli flushall IMHO.

The logic in comment #16 seems to be during hook_cache_flush(), flush Redis cache (only when there's a prefix):

 $keys = $redis->keys('*' . $prefix . '*');
 $redis->delete($keys);

This should work in Drupal Admin UI at least.

With the cache prefix in #2846208, I think the approach of hook into hook_cache_flush() is appropriate, as it doesn't break other applications (if they are sharing same Redis instance).

For performance penalties:
- on dev / non-production environments, the ability to flush cache is definitely more important
- on production environment I think it's understandable that flushing Drupal cache could be an expensive operation.

So I think we should merge this patch.

We don't need invent another drush command for this.

Again, for what purpose? The data is not deleted, but functionally, as far as Drupal is concerned, the cache *is* invalidated. If you want to empty it, there is also redis-cli.

The patch doesn't scale if you have a lot of data, keys() should not be used for that and it could also be very slow.

#2840037: Allow dedicated redis connections and/or redis db numbers per feature is another issue that when resolved, would allow for actually deleting data.

IMHO this is a won't fix.

FYI, since KEYS command is not recommended in production, you might be interested in the somewhat related #2851625: New feature: option to use SCAN command instead of KEYS on cache wildcard deletions

Hello,

I am testing Redis 1.0.0-beta1 without the patch.

If I create a new template and run drush cr the template is taken into account.

EDIT: As said by Berdir this is now fixed by the improved cache prefix.

Is this patch no longer relevant for beta1?

https://www.drupal.org/files/issues/redis-n2765895-16.patch

I am having issues with cache flush on beta 1. Does not seem to be working correctly. Like many people, I am using base 0 (the default). However, this falsy check will fail you if you are using the default NULL or 0 database index:

+    // Safety checks
+    if (!$redisDatabaseIndex || !is_numeric($redisDatabaseIndex) || 
+         $redisDatabaseIndex < 0 || $redisDatabaseIndex > 15) return;

Could I take a stab at a new patch for this please?

Adding $settings['cache_prefix'] like described in README.md fixed it for me.

Before I set cache_prefix I had to run redis-cli FLUSHALL after using drush cache:rebuildto get changes active. After adding the cache_prefix I even do not need a cache:rebuild for e.g. views config changes.

I have a relevant use case that needs this behaviour. At the moment it is hypothetical for me, but will become a reality soon.

I have 1 Redis instance, shared with 100 websites, using prefixes to differentiate cache data for those websites.

A website is updated, causing almost all of the cache data to become irrelevant/stale. For example moving from a single language site to a multi-language site would cause all page cache to become invalid.

But now the Redis instance is at max memory usage and starts clearing old data to make room for this new cache data from the updated website.
So now valid cache-data from other websites, that is considered much older, is being deleted...

Running commands on a live webserver to fix this, is not something that can be expected from any developer on your team. Also, the timing in this use case would be very important. So waiting for a developer with access (and knowledge) to the live webserver might be a real issue in this scenario.

An option to flush these caches separately would be required in this scenario.
Ideally a complete cache rebuild would also flush redis cache, or maybe this could be a configuration that is disabled by default? (with a very big warning on possible performance impact)

Please consider reopening this ticket, perhaps marking it as a feature request because of the performance issues.

@weseze: Sounds like a feature request that should go into a new issue?

Yes. The module is specifically designed to not require KEYS or SCAN commands at runtime (except the report), with the downside of leaving data in place. In our case, we actually empty the redis cache on deployment with a flushall. It should be fairly easy to write a deploy script that uses KEYS or SCAN to only delete keys with a given prefix. But not sure about adding it to the project, could also just be a documentation page.

It should be fairly easy to write a deploy script that uses KEYS or SCAN to only delete keys with a given prefix. But not sure about adding it to the project, could also just be a documentation page.

We have been working on this recently and came up with this script -- does that make sense ?

#!/bin/bash

# Usage.
SITE_ALIAS=$1
if [ -z "$SITE_ALIAS" ]; then
  cat <<EOF
Clear all cache entries on a specific site in Redis.

Usage: $0 <site-alias>

Example: $0 mysite
EOF
  exit 0
fi

# Get site's cache prefix.
SITE_CACHE_PREFIX=$(drush "@$SITE_ALIAS" ev 'print \Drupal\Core\Site\Settings::get("cache_prefix")')
if [ -z "$SITE_CACHE_PREFIX" ]; then
  echo "Unable to determine cache prefix. Is \$settings['cache_prefix'] set in settings.php ?"
  exit 1
fi

# Get site's Redis host.
REDIS_HOST=$(drush "@$SITE_ALIAS" ev 'print \Drupal\Core\Site\Settings::get("redis.connection")["host"]')
if [ -z "$REDIS_HOST" ]; then
  echo "Unable to determine Redis host. Is \$settings['redis.connection']['host'] set in settings.php ?"
  exit 1
fi

# Clear cache entries.
# The keys must be wrapped with single quotes (') because of https://stackoverflow.com/a/58879388
PATTERN="$SITE_CACHE_PREFIX:*"
echo "Clearing $PATTERN on Redis host '$REDIS_HOST'"
redis-cli -h "$REDIS_HOST" --scan --pattern "$PATTERN" | sed "s/^/'/;s/$/'/" | xargs redis-cli -h "$REDIS_HOST" DEL

What do you think of that approach?

/**
 * Implements hook_cache_flush().
 */
function xxx_cache_flush() {
  if (!(ClientFactory::hasClient())) {
    return;
  }

  $redis = ClientFactory::getClient();
  if (!is_object($redis)) {
    return;
  }

  // Get the database index to use. If not set, simply abort by returning.
  $redis_connection_settings = Settings::get('redis.connection');
  if (!isset($redis_connection_settings['interface'])) {
    return;
  }

  // Get the cache prefix or prefixes.
  $prefix = Settings::get('cache_prefix');

  // Delete any keys found, handling arrays of prefixes/bins as well.
  if (is_array($prefix)) {
    return;
  }

  $keys = $redis->keys("$prefix*");
  $redis->del($keys);
}

As per #26 comment, "if (!$redisDatabaseIndex" won't work in patch #16, as it's ignoring "0", change it to "!isset($redisDatabaseIndex)" instead.

Hello, I have made a fork and uploaded a commit with the code from DamienMcKenna comment #16, I have modified the first condition, replacing !$redisDatabaseIndex with "!isset($redisDatabaseIndex)" as kenorb commented #33.

For me, in a drupal 9 with the latest version of dockerhub of redis, clearing caches works using
$settings['redis.connection']['base'] = 0;
and
$settings['cache_prefix'] = 'mysite_';

Tested running DBSIZE on redis-cli after and before do drush cr on drupal terminal.

I have made the merge request.

Thanks.

Just tested with #16 patch and MR #21.
With 16 the Redis cache is not flushed. While with the MR it works correctly.