HTTPRL Spider uses the excellent HTTP Parallel Request & Threading Library to perform an internal cache seeding spider via Drush. This is not a module, it's a command line call that can be used to effectively rebuild all Entity Cache and Boost cache files for an entire site. By default it will spider all known, front-facing entities in a site with a single call. It also has some hooks to allow for adding additional non entity paths to target (such as views and other menu paths).

Installation

Just run drush dl httprl_spider to get the latest version; it's a drush plugin.

Simplest example

drush cc all
drush hss --nonblocking

This will clear all caches and then issue nonblocking requests for all front-facing entity paths. This has performance implications so make sure that you run this as part of a nightly job and not all the time (especially on larger sites) as this is effectively a self-imposed denial of service attack. A bit safer call is drush hss which issues the calls but sequentially and yet more reasonable is drush hss node to have it only spider nodes on the site.

Another common call

Say you want to request just a single path as to rebuild the cached version of it (like the front page or another critical page). drush hsr node would issue a call against the path "/node" and rebuild it in this example.

History

This project draws on concepts originally pioneered by Drush Entity Cache Loader with the main difference being that while it seems entity cache tables it then makes a http request for the url to the object it just cached. This causes not only the entity cache to be populated but css/js aggregated files, boost static html copies, menu and any other cache system to be populated. This is effectively the same thing as an anonymous user going to the page except they won't have to take the performance hit to be delivered the content.

This project is a spin off of #2087955: Add Drush support for command line HTTPRL calls which provides the same functionality but will be managed in this place as agreed upon by the maintainer.

ELMS

This project has been created as part of the ELMS initiative at Pennsylvania State University.

Project Information

Downloads