Posted by adrinux on January 28, 2010 at 5:23pm
| Project: | Drush Make |
| Version: | 6.x-2.x-dev |
| Component: | Code |
| Category: | feature request |
| Priority: | normal |
| Assigned: | Unassigned |
| Status: | closed (duplicate) |
Issue Summary
When developing make files one often ends up repeatedly running make as problems are noted and fixed. Each time make runs it downloads files afresh. It would speed up the build process and reduce load on external sources – drupal.org in particular – if downloaded files were cached, and the cache checked before each new build.
Comments
#1
That's a good idea. I'll look into this, along with other things such as unmake.
#2
As an FYI, I saw the utility of such a feature, and set up squid as a proxy server. Using it with drush is then as simple as running:
http_proxy="http://squid.example.com:3128/" /usr/bin/php /path/to/drush/drush.php#3
@acrollet I don't suppose you'd care to document this somewhere? :)
A blog post would be nice...
#4
@adrinux: ask, and ye shall receive: http://reluctanthacker.rollett.org/node/114
#5
Argh! Can't believe it's taken me two months to realize you replied to that!
Will give it a shot.
#6
Squid works nicely, at least for the files retrieved via http.
#7
Assuming that using an HTTP proxy cache is an appropriate solution to this issue.
#8
No, I want to implement a cache.
#9
I really think that squid is entirely appropriate for this.
@dmitrig01, can you say why you'd want to implement this in drush make? To me it seems a whole lot simpler both short and long term to leverage squid for this sort of thing.
#10
i use git for this task.
#11
Squid is scary and complicated. This should be fairly simple: 'tar -cf ' . md5(serialize($project)) . '.tar ' . $directory_name.
#12
@dmitrig01 "scary and complicated" ...something is amiss here, why would you scared be scared of squid and me not be? I'm supposed to be the paranoid one around here. acrollet lays out the steps pretty well in his post. What specifically are you not happy with about that process?
#13
Quick update here: I've installed squid and been running it for a few days. The short: it cuts the time required to build Open Atrium about in half, and I'm on a super speedy connection. The setup via macports is even simpler that what acrollet posted on his blog. The configuration for squid ships about ready to go. So the only thing to do is start squid and export the `http_proxy` environment variable (or add it do your `drush` executable).
#14
I think curl does this internally
#15
or rather
#16
Hi,
I'm thinking about adding this to a Drupal development environment virtual machine that I share called Quickstart. http://drupal.org/project/quickstart
I'd hate to waste anyone's development time with crazy caching problems - have yall run into any problems with this? Do you still find it useful?
Thanks,
Mike
#17
@MichaelCole No I haven't had any problems. And yes it's still useful - particularly on a desktop vm, I find on a production server downloads are fast enough anyway (although there is still the nice advantage of not sucking up so much bandwidth), but my desktop VMware fusion vm still benefits from the speed boost.
#18
Anyone know how to configure squid so it caches the results of updates.drupal.org such as http://updates.drupal.org/release-history/webform/6.x. By default, it does not cache those (per http headers from drupal.org) drush's pm-download and updatecode call those urls very often.
#19
So... how do you handle this issue when you can't install squid? (Like when you're running in a sandbox on a shared hosting environment like Hostgator?) I see the comment about curl doing this internally, but that doesn't tell me much...
#20
Here's a quick and dirty patch to drush_make that caches downloads in the local filesystem. Right now it caches forever, without regard to cacheability etc. Use at your own peril :)
@dmitrig01: let me know if you think this direction is worthwhile? obviously it needs to be much more advanced (checking cacheability, expiry, options for control, etc).
#21
Fix a couple of bugs.
#22
More improvements
#23
subscribing
#24
Hi, the Drupal Quickstart development environment comes with squid configured to cache drush make.
http://drupal.org/project/quickstart
Here's how I configure it on Ubuntu, to work with Git, shutdown quickly, and only cache ftp.drupal.org:
# ################################################################################ Squid caching of ftp.drupal.org
cd ~
sudo apt-get update
# Install caching proxy server
sudo apt-get -y install squid3
echo "http_proxy=\"http://localhost:3128\"" | sudo tee -a /etc/environment > /dev/null
echo "
# fix for git 417 errors
ignore_expect_100 on
# Quickstart
acl drushservers dstdomain ftp.drupal.org
cache allow drushservers
cache deny all
# don't wait to finish requests before shutting down. Do it now!
shutdown_lifetime 0 seconds
" | sudo tee -a /etc/squid3/squid.conf
echo "*** REBOOT TO TAKE EFFECT ***"
If you have any suggestions, I'd be excited to hear them.
Mike
#25
In addition to caching tarballs, I am interested in config changes such that squid will ignore the no-cache and similar http headers sent by updates.drupal.org and perform caching of the response for an hour or so. that would really speed up drush unit tests. an example url is http://updates.drupal.org/release-history/views/7.x
#26
Tested patch from #22 on 4.4 release version of drush, and it seemed a lot snappier than using squid. Nice work @lyricnz !
#28
While the patch in #22 works fine, it's a naive cache implementation - and could potentially cause issues with caching too much. We need a smarter implementation, but as noted above, it needs to disregard the Cache-Control/Expires headers
#29
nearly committed this cache for drush core #1173776: Optionally cache release XML and tarballs for pm-download and pm-updatecode. once that goes in, we should mark this a dupe of #581348: have drush make use the pm primitives
#30
Oh yes, that's way better. No need to wait until that gets in, it's a dupe already.
#31
#581348: have drush make use the pm primitives