Cache downloaded files

That's a good idea. I'll look into this, along with other things such as unmake.

As an FYI, I saw the utility of such a feature, and set up squid as a proxy server. Using it with drush is then as simple as running:

http_proxy="http://squid.example.com:3128/" /usr/bin/php /path/to/drush/drush.php

@adrinux: ask, and ye shall receive: http://reluctanthacker.rollett.org/node/114

Assuming that using an HTTP proxy cache is an appropriate solution to this issue.

No, I want to implement a cache.

I really think that squid is entirely appropriate for this.

@dmitrig01, can you say why you'd want to implement this in drush make? To me it seems a whole lot simpler both short and long term to leverage squid for this sort of thing.

i use git for this task.

Squid is scary and complicated. This should be fairly simple: 'tar -cf ' . md5(serialize($project)) . '.tar ' . $directory_name.

@dmitrig01 "scary and complicated" ...something is amiss here, why would you scared be scared of squid and me not be? I'm supposed to be the paranoid one around here. acrollet lays out the steps pretty well in his post. What specifically are you not happy with about that process?

Quick update here: I've installed squid and been running it for a few days. The short: it cuts the time required to build Open Atrium about in half, and I'm on a super speedy connection. The setup via macports is even simpler that what acrollet posted on his blog. The configuration for squid ships about ready to go. So the only thing to do is start squid and export the `http_proxy` environment variable (or add it do your `drush` executable).

I think curl does this internally

or rather

Hi,

I'm thinking about adding this to a Drupal development environment virtual machine that I share called Quickstart. http://drupal.org/project/quickstart

I'd hate to waste anyone's development time with crazy caching problems - have yall run into any problems with this? Do you still find it useful?

Thanks,

Mike

Anyone know how to configure squid so it caches the results of updates.drupal.org such as http://updates.drupal.org/release-history/webform/6.x. By default, it does not cache those (per http headers from drupal.org) drush's pm-download and updatecode call those urls very often.

So... how do you handle this issue when you can't install squid? (Like when you're running in a sandbox on a shared hosting environment like Hostgator?) I see the comment about curl doing this internally, but that doesn't tell me much...

Here's a quick and dirty patch to drush_make that caches downloads in the local filesystem. Right now it caches forever, without regard to cacheability etc. Use at your own peril :)

@dmitrig01: let me know if you think this direction is worthwhile? obviously it needs to be much more advanced (checking cacheability, expiry, options for control, etc).

Fix a couple of bugs.

More improvements

subscribing

Hi, the Drupal Quickstart development environment comes with squid configured to cache drush make.

http://drupal.org/project/quickstart

Here's how I configure it on Ubuntu, to work with Git, shutdown quickly, and only cache ftp.drupal.org:

# ################################################################################ Squid caching of ftp.drupal.org

cd ~
sudo apt-get update

# Install caching proxy server
sudo apt-get -y install squid3

echo "http_proxy=\"http://localhost:3128\"" | sudo tee -a /etc/environment > /dev/null

echo "
# fix for git 417 errors
ignore_expect_100 on

# Quickstart
acl drushservers dstdomain ftp.drupal.org
cache allow drushservers
cache deny all

# don't wait to finish requests before shutting down.  Do it now!
shutdown_lifetime 0 seconds 
" | sudo tee -a /etc/squid3/squid.conf

echo "*** REBOOT TO TAKE EFFECT ***"

If you have any suggestions, I'd be excited to hear them.

Mike

In addition to caching tarballs, I am interested in config changes such that squid will ignore the no-cache and similar http headers sent by updates.drupal.org and perform caching of the response for an hour or so. that would really speed up drush unit tests. an example url is http://updates.drupal.org/release-history/views/7.x

While the patch in #22 works fine, it's a naive cache implementation - and could potentially cause issues with caching too much. We need a smarter implementation, but as noted above, it needs to disregard the Cache-Control/Expires headers

nearly committed this cache for drush core #1173776: Optionally cache release XML and tarballs for pm-download and pm-updatecode. once that goes in, we should mark this a dupe of #581348: have drush make use the pm primitives

Oh yes, that's way better. No need to wait until that gets in, it's a dupe already.

#581348: have drush make use the pm primitives

Can anybody confirm that the caching has been implemented? And how to use it?
I don't see any word about cache in these duplicated issues:
#581348: have drush make use the pm primitives
#1267228: Drush Make should use Drush core's native download abilities concurrently

I'm using drush 5.8 and caching doesn't work.

Don't reopen 2 year old issues to ask a support question. Create a new issue if you are observing a defect.