Making Drupal work with caching servers, such as squid and varnish

subscribe

Looks interesting.

It would be nice if we can develop standard ESIs for Drupal that would flush the appropriate parts of the cache when the underlying entities are changed.

However, Drupal's cache right now does a flush all when a node is changing, so before we can do that we have to make Drupal's cache more granular, and avoid a flush thundering herd.

subscribe

@klavs Any update here, perhaps in a different issue? Edge Side Includes would be an excellent update. Sounds like something for Drupal 8 at this point.

Many of the suggestions here are interesting (ESI could be very useful for Drupal, though I wonder if serving Drupal with nginx + boost module might not have similar results in terms of performance with a lot less hackery), but I'm not entirely convinced the approach in the parent is entirely wise.

Varnish allows for a grace period to serve expired cache objects if you experience failure of your backend(s). Surely, a reasonable TTL plus a well tuned grace period will achieve near-optimal performance for sites with a lot of (basically) static content without the downsides suggested. In high traffic situations, often a few pieces of content will be targeted -- even short TTLs should allow Drupal to serve hundreds or thousands of end user requests with just a few requests made of Drupal, and still be adequately dynamic.

I wonder why you would using a caching proxy and then ask Drupal to do no work at all. Similarly, by sending purge requests, you unnecessarily complicate your Drupal code. Transparent proxies should be, well, transparent. Now your underlying application depends (albeit VERY loosely) on the caching proxy in front of it. Again, that should ring bells that maybe you should be implementing your solution against Drupal itself -- custom caching in Drupal is possible, and can also provide adequate performance and good granularity (since you can write site specific rules with full access to Drupal's guts and state at the time of request).

It seems to be a better way of accomplishing the same end result would be publishing your whole site to static HTML (or parsed XML, or something else -- just something essentially static). I worked on a Plone project which worked like this, where the site was "staged" and then "pushed" to what amounted to static HTML when users updated content. It was a bit of a pain, but the production site was really freakin' fast, and that was using a pretty standard Apache configuration.

Indefinite caching is also not really an adequate failover mechanism -- proxies like Varnish explicitly implement systems like grace periods. I'd rather have a monitoring solution which paged me if httpd failed on my production Drupal site (or some similar failure event occurred) and a grace period to allow me to patch up the problem than the possibility that the wrong content is served indefinitely (and will soon, as a result of my work).

FYI: http://drupal.org/project/varnish

Drupal 7 supports the basic cache-control headers which are necessary to use Varnish as a cache for full pages. Pressflow backports this functionality to version 6.x for those who need it.

I'm definitely interested in working with people to develop/implement ESI solutions.

ESIs are very useful if you want to cache most of your page content but still need some areas to be dynamically served to all (even anonymous) users.

I worked on a high volume drupal 5 site that used varnish with a few ESIs to populate areas of the site that needed to be dynamic for everyone, most notably a breadcrumb that tracked the user's journey and any other blocks that were dependent on this user journey.

My memory is a bit hazy as I worked on this about a year ago, but i seem to remember the main problem was that each ESI would hit Drupal and cause a full bootstrap. This meant that our site with four dynamic areas would have actually caused a greater load on the db than the non-cached site.

To get around this I created a lightweight service framework that only did a partial bootstrap, enough to get the functionality we wanted working. However, four of these would still have caused too many db hits.

The final solution was this. When the lightweight service page was called via the ESI, it could generate a key for the content that is was asked for and check the cache (memcache in this case) for this content. If it couldn't find any (which it probably wouldn't on the first page hit), it would do a partial bootstrap and hit a drupal service which would then generate and preacache all required dynamic content for the entire page, whilst returning the content for the first request. Each successive ESI request would then never hit drupal as content was already available in the cache.

I seem to remember that the key was unique to the situation so that it was possible that the cache was effective for multiple users if they happened to use the site in the same way.

This seemed to do the trick at the time and people were happy with the performance although I left the company shortly after so i don't know how it progressed from there.

Hope this is of use to someone! :-)

I've written a module, Purge, that implements the Purge functionality thought http requests. It should work with both Varnish and Squid. Right now it's still in the waiting line for a review (hint,hint:-) here #938548: SqyD [sqyd] but I hope to have it in contrib soon. It requires the expire module that in turn is a splitoff of the boost code to detect urls that need to be flushed from cache. Having that functionality in core would be good to have so different contrib modules (boost, varnish, purge, etc) can be used when appropriate.

If you want this, please help with http://drupal.org/project/issues/search/drupal?issue_tags=WSCCI