Acquia Dev Cloud and Managed Cloud support

Interesting question but I think you should redirect it at Acquia as they would have to reconfigure Varnish to support http purging.
My main motivation for writing Purge was exactly this issue. The Varnish module works on the administrative socket of Varnish and there is no way to prevent access to Varnish logic that is used for other sites. The Purge method mitigates this by just using http requests. To make this work in the their cloud hosting solutions Acquia would have to add a few lines of code to their Varnish setup. I would like to hear their response and would love to help out if there is any issues with this that can be solved. Would you please post their response back into this issue?

I don't use crawlers myself but Mike Carper has mentioned here how his setup works.
http://drupal.org/node/968366#comment-3695304
In short its Boost and a custom script that uses boost db tables to crawl the urls that need "warming".

I've been looking into this with Acquia as well....I imagine quite a few people will go through exactly the same process, so it would be good to have modules supporting it.

Acquia's already set up to purge items in Varnish via CURL...here's an example to clear 'http://example.com/about':

$ch = curl_init();
$url = 'http://example.com/about';
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Accept-Encoding: gzip','X-Acquia-Purge:myAcquiaSiteName'));
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'PURGE');
curl_exec($ch);
curl_close($ch);

This is pretty close to what Purge does, minus the custom headers. I've posted a patch here: #1154994: Provide ability to add custom headers (for Acquia support etc). Would be great to get some testing and input.

This is great stuff. Thanks for that find. I don't use the dev cloud myself. Is this info available publicly anywhere?

Knowing this, building support for the acquia dev cloud into purge will be straight forward. I will work a basic version for it into head over the coming week, based on this patch and info.

Is the myAcquiaSiteName setting available in the environment? Would be nice to make it self configurable.

Glad it's helpful. I've mainly collected this via the ticketing system and the helpful Acquia folks. I will point them to these issues and see whether they may have additional input - to my mind it's in Acquia's interest to get these modules working seamlessly in their environment.

Let me know if you have specific questions and I'll include them in the conversation. My next point would be to see whether the whole cache can be flushed via CURL.

Oh, and we're using the above code on the Acquia Managed Cloud, not the Dev Cloud, but I'd be surprised if it wasn't the same configuration for that part.

@SqyD, The "myAcquiaSiteName" is available to as an Environment variable: $_SERVER['AH_SITE_NAME']
Thanks for all the great work on this project.

@NicholasH, Managed Cloud and Dev Cloud both have the same Varnish configurations, and the Curl Purge command can only purge individual objects from cache, so you cannot purge an entire domain from Varnish Cache, however we do give our customers the ability to do this via their Hosting Tab, on the Domains page.

Due to the fact that we are using shared load-balancers for a majority of our customers we cannot open up the Varnish administrative socket to the public, however if the customer decides to purchase a dedicated Load-Balancer we can certainly make exceptions since there isn't a risk of cache contamination in this case.

I've just commited basic support for this. See: http://drupalcode.org/project/purge.git/commit/7920785

I did not test it on the Acquia service yet, just my local setup. I will add help, documentation and maybe some specific error handling/logging later but I would like to hear if this is working out for you.

Hey SqyD,

Thanks for that. I only got around to it today to start testing. It did work in staging but threw some fatal PHP errors in prod. I obviously had to roll it back and will investigate further tomorrow.

There's a few syntax parts that confuse me, maybe you could clarify whether this is intentional.

Shouldn't this:

switch ($purge_request_result['http_code']) {
      // First check if everything went ok.
      case 200;
        $purge_log[] = $purge_request_result['purge_url'] . ' on ' . $purge_request_result['proxy_url'] . ' ' . 'Ok';
        $purge_success++;
        break;
      // Notice if the request was not found in the proxy cache
      case 404;
        $purge_log = $purge_request_result['purge_url'] . ' on ' . $purge_request_result['proxy_url'] . ' ' . 'Not Found';
        $purge_success++;
        break;
      // Collect all proxy hosts that are blocking the url requests
      case 405;
        $purge_log = $purge_request_result['purge_url'] . ' on ' . $purge_request_result['proxy_url'] . ' ' . 'Forbidden';
        $purge_blocking[] = parse_url($purge_request_result['url'], PHP_URL_HOST);
        break;
      // Collect all urls and their http error codes
      default;
      $purge_log[] = $purge_request_result['purge_url'] . ' on '
        . $purge_request_result['proxy_url'] . ' ' . $purge_request_result['http_code'];
      break;
    }

be rather like this (note $purge_log being consistently an array and "case" being followed by colons, not semicolons):

switch ($purge_request_result['http_code']) {
      // First check if everything went ok.
      case 200:
        $purge_log[] = $purge_request_result['purge_url'] . ' on ' . $purge_request_result['proxy_url'] . ' ' . 'Ok';
        $purge_success++;
        break;
      // Notice if the request was not found in the proxy cache
      case 404:
        $purge_log[] = $purge_request_result['purge_url'] . ' on ' . $purge_request_result['proxy_url'] . ' ' . 'Not Found';
        $purge_success++;
        break;
      // Collect all proxy hosts that are blocking the url requests
      case 405:
        $purge_log[] = $purge_request_result['purge_url'] . ' on ' . $purge_request_result['proxy_url'] . ' ' . 'Forbidden';
        $purge_blocking[] = parse_url($purge_request_result['url'], PHP_URL_HOST);
        break;
      // Collect all urls and their http error codes
      default:
        $purge_log[] = $purge_request_result['purge_url'] . ' on '
          . $purge_request_result['proxy_url'] . ' ' . $purge_request_result['http_code'];
        break;
    }

Either way, changing that still didn't prevent the error, so I'll need to dig deeper.

Those were completely unintentional and leftovers from a code refactoring some time ago. Thanks for spotting that.
Fatal errors in one env, not the other... Strange. Maybe the response to Purges is different (error codes) between the two?

Ok, the fatal errors were actually fixed by the above changes...I stuffed up with the deployment. It still didn't actually purge (although it said it did) until I also added the "Accept-Encoding: gzip" header from #4.

Works fine for me now, thanks. Would be good to get some feedback from entrepreneur27 as well for DevCloud....

Let me know if you want some details on documentation in regards to Acquia.

Great! I'll commit this patch over the coming week! I've also written a few form validation checks on the proxy settings that should detect misconfiguration of this setting.

Cool, let me know when it's in and I'll let Acquia know, so they can maybe recommend it to clients and/or put it into their documentation library.

I've just committed your patch from #11. Sorry it took longer then expected. The form validation has gotten lost in a messy vm migration (was kinda dirty code anyway, so I'll rewrite it more elegant). I'll also update the documentation etc.

In the meantime I hope you and some more people can give it a spin and report back if it's usefull.

Q: Will this module profit from increased usage stats or are autoupdates disabled in the Acquia Cloud?

I've just added the form validation stuff in this commit. It should check for the acquia env. setting too. Could someone please verity if that works?

Thanks SqyD, works great (including validation). Worth a new release? I'll ask about the auto updates...

Great. I'll roll a 1.2 release when I've updated the documentation with varnish 3.x specifics, so functionality should be what's there now.

It's in the 1.2 release I just rolled. Would love to hear when this is publicized elsewhere...

I've pinged Acquia about this now being in a stable release.

@entrepreneur27
Had a quick peek for the cache warming requirement. I could not find a module immediately, but doing this via wget and an XML sitemap seemed to pop up a few times.

This article (and comments) does discuss this in great detail:
http://www.metaltoad.com/blog/how-drupals-cron-killing-you-your-sleep-si...

A short version was here:
http://www.tutorialarena.com/blog/using-wget-xml-sitemap-spider-drupal-w...

Can't comment much on any of it, just had a quick look.

I think we can separate this out into two:

1: The need to have a warm cache that contains ALL urls that need caching.
For this the xml stiemap approach seems adequate. Not sure how good it scales on site with a LOT of content. Then a more intelligent crawler that can schedule requests, accounts for expire times and makes sure backends don't collapse might be better.
I'll play around with xml sitemap module a bit. Would be nice if it has some cache tables we could reuse as a store for all urls of a site.

2: The need to have a url in cache after it gets updated.
Right now the purge module just purges the url from the cache(s). I plan to have an optional setting to refetch the url once it's purged. This could however create backend performance spikes. Having a more intelligent crawler like in point 1 will help it scale.

No problem continuing to chat a bit here but as I see it now these issues should go their separate ways. 1 as a separate project and 2 as a Purge feature. I'll be glad to initiate both

Yeah, it's definitely a different issue and we'll just chat here.

Would you agree that functionality in point 1 should go into the expire module? Given that mikeytown2 already did this in boost, could this already be in the works?

In regards to point 2, as you already mention, that could be quite dangerous in terms of performance depending on the expire settings. Just to give an example, when I tested purging from varnish with all the expire options enabled, updating a node would sometimes clear hundreds of pages (taxonomy relations etc). We didn't really use those settings in production, but it was interesting nonetheless.

Regenerating all these pages in one hit could really be a problem...it completely depends on the site and its traffic patterns how good or bad this functionality would be.

In general on crawling your own site, I haven't seen any mention on statistics/page impressions/ad serving. I suppose you could identify your crawler in statistics and take it out of the data, but I thought I mention it anyway....would like to get people's thoughts on this.