The attached patch enhances devel so that its --cache option does more. It caches release XML for 24 hours (by default) and caches tarballs for a year (still need to exclude dev snapshots). The net effect is that a warm cache obviates both network calls for pm-download.

The cache is implemented as files in ~/.drush/cache/download. The contents of the file are release XML or a tarbal unmodified from drupal.org. The names of the files indicate the URL where the file lives. The filectime() of the download tells us if it has gone stale or not. See the new drush_download_file() for details. It wraps the existing _drush_download_file() which changes slightly in this patch (different return value). I think its changes are OK for drush4 but lets discuss.

This has some overlap with the Cache API proposed at #1172044: Cache command files / Add drush cache API. If that gets in, this patch will be enhanced to use its API.

One major goal here is to speed up the unit test suite. It re-downloads the drupal and devel project many times. Unfortunately, the HOME dir for the tests currently gets wiped after each class completes so we get very few cache hits. Needs some thought.

Feedback is appreciated.

CommentFileSizeAuthor
#4 0001-A-more-complete-cache-option-for-pm-download-command.patch13.22 KBmoshe weitzman
0001-A-more-complete-cache-option-for-pm-download-command.patch11.25 KBmoshe weitzman

Comments

jonhattan’s picture

jonhattan
Primary language Spanish
Location Plasencia
commented

I did think on something like this for #1032626: Allow drush pm-update w/o update.module. Unfortunately I haven't had the time to devote to this

greg.1.anderson’s picture

greg.1.anderson commented

This could be very useful for offline Drupal work; with the addition of a cache-refresh command, which could be as simple as drush dl long list of modules if we added a --cache-only or perhaps --destination=FALSE. If one was insightful enough to keep a list of all modules they used for any Drupal project, this would then allow offline dl and up, which would be pretty cool.

Regarding the cache directory, drush should check an environment variable to determine the cache location; if not set, then the current behavior can be used. The test suite could then just set up a cache environment variable, just as it sets up the home dir environment variable, and point it at a location that is not cleared after every run.

Regarding speeding up the test cases, see also #1174500: Speed up the unit tests by caching environment setup.

msonnabaum’s picture

msonnabaum commented

Well, one thing I was planning on adding to #1172044: Cache command files / Add drush cache API was a global option for cache dir. I think we should do that regardless of which approach goes in, since the default is fine for most, but easily overridable with an option.

moshe weitzman’s picture

moshe weitzman commented
StatusFileSize
new 0001-A-more-complete-cache-option-for-pm-download-command.patch13.22 KB

This one does as Greg suggests; the dir for global cache can be set by an environment var and the unit tests take advantage. Now the cache persists across test classes and we are getting cache hits. Note that I changed the phpunit.xml.dist a bit so folks who have customized that need to set UNISH_TMP instead of UNISH_SANDBOX.

I'll let this sit for a bit and work on #1174500: Speed up the unit tests by caching environment setup. Then we'll strategize about which patches should go in.

msonnabaum’s picture

msonnabaum commented

Do we not want a global opt for cache-dir? I figured we would since we also have dump-dir and you might want to set something like this permanently in drushrc.php.

Even though this wouldn't use the cache api, I feel like we can still share the same directory. If we can add cache-dir to drush_directory_cache(), I can bring that into #1172044: Cache command files / Add drush cache API and not call the opt directly.

moshe weitzman’s picture

moshe weitzman commented
Status: Needs review » Fixed

I tweaked so that dev snapshots can be cached for 4 hours and releases for 1 year (assuming --cache is true). We'll have to see if that 4 hour limit gets annoying during offline development. Can be changed.

I've gone ahead and committed this. We can keep talking about the utility of a cache-dir option. Meanwhile, commands can start using drush_directory_cache() as is.

Hopefully we can also push along #1174500: Speed up the unit tests by caching environment setup and #1172044: Cache command files / Add drush cache API which are related, worthwhile patches as well.

I'm leaning towards keeping this for 5.x, but we can discuss it. There are no API breaks for end users.

greg.1.anderson’s picture

greg.1.anderson commented

We'll have to see if that 4 hour limit gets annoying during offline development.

I have not reviewed or tested, but as long as the code will use but never clear cached objects when offline, the time limit should be just fine.

Looking forward to using this feature (offline) soon. :)

moshe weitzman’s picture

moshe weitzman commented
Status: Fixed » Active

Are you saying that we should continue to use an expired file (i.e. more than 4 hour old snapshot) if we are unable to get a new one? I kind of like that fallback. Code does not do that yet.

greg.1.anderson’s picture

greg.1.anderson commented

Yep. I don't mind if a dev snapshot goes away when I'm online; I'll probably run something similar to drush dl $EVERY_MODULE_I_USE just before I go offline to make sure that I have the most recent version of everything before I get on a plane (or something). My big list will have modulename-7.x-1.0-dev in it if I need the dev version. I'll have to see how long $EVERY_MODULE_I_USE takes to download before I decide whether this workflow is good or not. Most of the time the results should already be cached, right? :)

Bonus feature: cache releases for one year only if they are the most recent release (for their respective major version); otherwise, pull back the cache duration to something like one week. Not super important right now, but a year from now I'd probably have a lot of old modules that I didn't need any more if I did the above workflow semi-regularly. I don't think I'd care about that on a sata, but might care on an ssd. I'll probably have my caches always on sata (until I get around to getting an ssd for my laptop).

moshe weitzman’s picture

moshe weitzman commented
Status: Active » Fixed

I committed Greg's fallback suggestion. Hopefully I got it right.

I'll implement your Bonus Feature by wiping my cache dir anytime I get low on space.

greg.1.anderson’s picture

greg.1.anderson commented

Very cool -- I will test this Sunday evening, or hopefully before. :)

+1 for your implementation of the bonus feature. :)

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.