Clearing wildcard URL's

Hi Richard,

Thank you for your feature request, and you are right that its not supported as we speak.

It depends on the configuration of Acquia Cloud's load balancers and the way our Varnish VCL's are setup. I'm putting this on the roadmap here internally and will let you know once it becomes available through Acquia Purge and how it will work.

Thanks for getting it on our radar, hopefully I can get back to you soon!

Niels

Hello,
Sounds like a great feature.
I'm trying to looks for option to clear the varnish cache for indexed view. If I'm clearing the views URL varnish cache post cron run, it does clear it. But all the url's related to this views by facet search are not cleared (e.g.- http://www.example.views-url?f[0]=country_name%3ACosta%20Rica").
It will be great to have a wildcard feature to delete them all together.
Thanks, Yael

Hello,

I am attempting to flush all pages under my /services URL path.

The instructions say:

Enter every new url on a single line. Example: node/[node:nid].
If you want to use wildcard flush, add |wildcard to the end of the line. Example: node/[node:nid]|wildcard

services/|wildcard is this what I should be doing? Am I way off base? what does |wildcard really mean?

Hi @nielsvm,

Just to add my support for this feature request. Has there been any progress with this internally?

Thanks,
Paul.

Would also like to see this. Hey @thicknrich, where are you seeing those instructions?

@nielsvm I'd like to know if there's anything that could be changed in the module to allow this behavior. Looking at the code, there's no clear way for the module to poll for cached paths matching a certain pattern, so implementing the "|wildcard" suffix would have to happen at the load balancer rather than through the module. Is that the case, in your opinion?

If wildcard support could be added at the module level I'd love to help, but I'd need visibility into the API to see if there's a way to get a list of paths matching a particular pattern. Otherwise, let me know where this is in terms of priority. I have a client who's really looking for this functionality.

I'm considering an option that involves enabling the statistics module (yes I know, heresy!) and running a query to find recently accessed paths that match the wildacard pattern and flush them. It's a far less than ideal solution since it will be attempting to flush paths that are already flushed and will likely miss some paths that have dropped out of the statistics table but are still cached in Varnish.

If I can obtain the time / budget to do this, would it be best to write a patch to Acquia Purge (as in would it a potential addition to the module) or a new module?

Either way since this would be a sticking plaster at best, I should mention that the current situation has been seen as an argument against Acquia cloud hosting by a couple of our clients, since there are a number of situations where the only feasible way to flush the related pages is to flush the entire varnish cache from within the Acquia Cloud admin.

Thinking about it we could add a "purged" column to the accesslog table and use it to reduce the number of redundant purges.

There is - albeit super slow - progress on this front. One piece of evidence in this closed dupe:
https://www.drupal.org/node/2429581

After 7.x-1.2 is released, I'll look further to how we can get this facilitated. It barely takes work on AP's side but definitely means a lot of work at the Acquia Cloud side, namely the Varnish implementations for *all* of our customers. Expect this to take longer.

@nielsvm , any news regarding this issue?
The module is already in version 1.3..

A big ++ for this issue.

Unless I can purge wildcard paths, the only way I can make my client's site work is to set a short external cache lifetime.

What VCL changes need to be made in order to support this on Acquia's side? We are going the custom VCL route and can put this in our VCL file on Acquia.

Thanks

I think the changes on Acquia's side via a custom VCL are to use the ban action when the PURGE method is used in sub vcl_recv {}.

Just submitted this with our custom VCL.

if (req.request == "PURGE") {
  if (!req.http.X-Acquia-Purge) {
    error 405 "Not allowed.";
  }
  # Add support for wildcard purging.
  # @see http://twigstechtips.blogspot.com/2014/04/varnish-enabling-wildcard-purging-of.html
  ban("req.http.host == " + req.http.host + " && req.url ~ " + req.url + "$");
  error 200 "Purged.";
  return(lookup);
}

Tested with purging via curl and it works. Gonna play with patching the module now.

@capellic The |wildcard instructions exist in the Content Type specific configuration for Cache Expiration, under "Enter Custom URLs". As expected, |wildcard does not actually work.

More ++ for this feature.

With a decoupled Drupal application it is cumbersome to clear individual parameterized requests from Varnish.

Seems like definite limitation is on Acquia's side.

I've made the following enhancements to allow for wildcard purges:

1. a table called acquia_purge_url_lookup where all lookup URLs gets stored along with URL params.
2. a configuration called acquia_purge_wildcard set to FALSE by dfault.
3. a new function which can be called to do a bulk purge.

Ex:

_acquia_purge_bulk_purge(array('foo/bar','foo/baz'));

Steps to use patch:
1. Run drush updatedb to create the above table.
2. set the wildcard purge configuration to true.

$conf['acquia_purge_wildcard'] = TRUE;

call the function wherever its needed, mostly in hook_node_update() or hook_entity_update().

Please review.

#16 it is much easier to ask Acquia for a custom Varnish config and put https://www.drupal.org/node/2155319#comment-10907958 in it.

Yes, this limitation has been on "Acquia's side" for a long time, but not anymore!

I'm hereby announcing native wild-card support in version 7.x-1.4!

What I mean by native is that this approach will not in anyway, lead to many paths being invalidated at once. Instead, a drush app news/* will now directly let your Acquia load balancer clean all of these paths in one go, without a heavy queue volume:

$ drush ap-purge news/*

# 1 items left...
Purged: http://mysite.com/basepath/news/*                [ok]
Purged: http://a.com/basepath/news/*                         [ok]
Purged: http://b.com/basepath/news/*                         [ok]
Purged: http://c.com/basepath/news/*                         [ok]
Purged: http://e.com/basepath/news/*                         [ok]
Queue processed successfully!                                    [ok]

The only caveat now will be that users who depend on this, will need to setup DrupalFakeCache as pagecache backend and are now also immediately instructed to do so after invoking a single wild-card purge:

 [warning] PAGE CACHE               Enabled (DrupalDatabaseCache)

           Your site invalidated one or more paths with
           wildcard characters in them in the past, e.g.
           "news/*". Because of this, you are now recommended
           to configure DrupalFakeCache as documented on
           https://www.drupal.org/node/797346. If you don't
           do this, outdated cached copies will be returned
           from the page cache for paths you intended to
           clear with a wildcard.

Check out the 7.x-1.4 dev release or wait for it!

After going through this, I believe that we would need DrupalFakeCache to be configured in order to make wild card support work correctly. If we see on this page: https://www.drupal.org/node/797346, it is mentioned that it must not be used in production.

If DrupalFakeCache cannot be used on prod, then how we will be able to use wildcard support on production?

I *think* the requirement to use DrupalFakeCache for this is to prevent pages from being stored in Drupal's internal page cache, and instead exclusively use Varnish now. Without doing so, a wildcard cache purge in Varnish will work, but not for Drupal's page cache? Do I have that right?

nielsvm, with the new functionality, how do we specify a wildcard path to be purged? The expire module has some help text in it that reads:

If you want to match a path with any ending, add "|wildcard" to the end of the line (see function cache_clear_all for details). Example: my/path|wildcard will match my/path, but also my/path/one, my/path/two, etc.