A file based cache utility for Drupal

The people at MayFirst.org have created a utility which lets you create static file copies of a Drupal home page, and the most recent 20 nodes (pieces of content). It's called "Drupal File Cache Generator" (dfcg) You can see the various files for this utility at:

• http://www.benslade.com/drupal/dfcg/
  (php files were given a .txt ending to make them displayable)

You can also see a high volume site that's using this file cache utility at AfterDowningStreet.org

How It Works

Basically, you run a script every so many minutes via cron, it fetches the home page and writes it out to an index.html file, gets the id's of the 20 most recent nodes, saves files for each whole page for a node (ie. displaying a page containing node content plus header, footer, and sidebar blocks).

The utility uses URL rewriting to check if a file cache copy of the page containing the node exists, if yes, that's returned to the client, else Drupal is called to generate the page.

Why Not Drupal Caching?

This script was created after the site AfterDowningStreet.org was having lots of weird errors, slowdowns, and outages. Many of the errors seemed to be load related and might have been race conditions (duplicate cache entry, etc). In fact we ended up turning off Drupal caching in an effort to get the errors to calm down (and weirdly, we kept getting duplicate cache errors).

It's possible that the AfterDowningStreet.org had various options set which kept the changing opening page ("categories" taxonomy block with story posting counts and time since last posting), causing the cache to keep getting flushed. It's also possible that Referer spam was creating lots of garbage cache entries.

In anycse, the people who were running the site (politically, not technically) were going crazy and were close to switching to a Wordpress like blogging solution using an Akami type distributed web caching infrastructure.

After switching to this file cache utility, page views/day went through the roof, all the errors disappeared, and the front page displays blazingly fast. You can see some old page view stats here.

In Conclusion

Obviously, this type of file based caching is only useful in certain situations. You have to setup your site so all users see the same opening page and node content pages, and that's not appropriate for many sites. But if you're a high volume site, it's wonderful to have the opening page display nearly instantly.

This utility probably should be considered in beta mode. There are a few weirdnesses with session id's which are still being worked out (which show up when using curl vs wget). Also, the script probably should be updated to pull all the node id's that are being displayed on the front page, rather than the most recent 20 nodes. If you want to display the uncached version of a page, you have to append an ampersand (&) to the URL. To see if you got a cached page, look for "STATIC Generated:" HTML comment in the HTML source.

Maybe this file based caching utility won't be as necessary if new alternative Drupal caching options are used. But it's going to be hard to get the owners of the ADS site to trust Drupal caching after the problems they've had. I'll have to make sure I understand all the details of Drupal based caching in order to sell them. Specifically, what extra work can occur when a Drupal cached page is accessed? How does Drupal know when to update the cached entry? Could the Referer spam issue mentioned above cause problems for Drupal based caching?

Ben Slade
PublicMailbox at BenSlade dot com
(append 030516 to the subj line to bypass spam filters)
"Everything should be made as simple as possible, but not simpler"
Albert Einstein