Add caching to git package handler [#1076302]

Here's a first stab at adding local caching for the git package handler via the method described here:

This implementation stores the cache in ~/.drush/gitcache. Seems to work and in my very limited testing it appears to shave about 1-2 seconds off the dl time, but code is POC at this point. I'm sure I'm missing some things there. The only bug I've found so far is that git_deploy doesn't seem to like cloned repos that reference the local cache.

Patch attached, but I'm working on this in a branch in my sandbox:

git clone --branch git_cache http://git.drupal.org/sandbox/msonnabaum/1076280.git msonnabaum_drush

Comment	File	Size	Author
#7	1076302-git-caching-7.patch	3.81 KB	msonnabaum
#6	1076302-git-caching-6.patch	3.81 KB	msonnabaum
	drush-gitcache.patch	4.36 KB	msonnabaum

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Comment #1

msonnabaum CreditAttribution: msonnabaum commented 28 February 2011 at 17:45

I should add that to use this, you need to pass the undocumented --cache to `drush dl`. Wasn't sure how best to handle an additional argument to dl that was package-handler specific.

Comment #2

moshe weitzman CreditAttribution: moshe weitzman commented 4 March 2011 at 18:36

We discussed two improvements to this:

add --verbose and --progressive to the git clone commands when drush is in verbose mode
Add documentation for --cache. I kinda think we should just turn this one by default? Feedback?

I will also look into the git_deploy issue.

Comment #3

Damien Tournoud CreditAttribution: Damien Tournoud commented 4 March 2011 at 20:22

Status:

Active

» Needs work

Also, the patch is missing drush_escapeshellarg() mostly everywhere.

Comment #4

greg.1.anderson CreditAttribution: greg.1.anderson commented 4 March 2011 at 21:21

Yes, you should not impode(' ', ...) your command array; drush_shell_exec and variants allow you to put %s replacement strings in your command; these are filled in with escapeshellarg'ed versions of your function arguments. It's preferable to use this facility.

Comment #5

Damien Tournoud CreditAttribution: Damien Tournoud commented 4 March 2011 at 21:25

Git Deploy cannot handle those repositories because Glip, the Git PHP library that it uses doesn't support the alternates object store feature of Git, that allows a git repository to borrow objects from another one.

It is not too much a hassle to implement, that said: https://github.com/damz/glip/commit/79f5472af4b9261d20f51e92f07d4cca01e8...

Comment #6

msonnabaum CreditAttribution: msonnabaum commented 5 March 2011 at 22:46

File	Size
1076302-git-caching-6.patch	3.81 KB

Ok, tried to incorporate all the comments here. Also changed the opt to "gitcache" to be more consistent with the other options.

On the issue of enabling this by default, I'm leaning towards not doing that at first since there are a few gotchas still. Like if you were to delete your ~/.drush/gitcache directory, you won't be able to update one of the projects that was cloned by referencing it as far as I can tell. You can run `git repack -a` on the project to copy the objects local and break the dependency, but the gitcache has to be present for that to work. I'm not seeing that as a huge issue in general, but it could be a big source of WTFs for folks who don't understand how --reference works. Those of us who do can put the gitcache option in our drushrc.

Big thanks to Damien for fixing glip!

Comment #7

msonnabaum CreditAttribution: msonnabaum commented 5 March 2011 at 22:47

File	Size
1076302-git-caching-7.patch	3.81 KB

Forgot patch.

Comment #8

moshe weitzman CreditAttribution: moshe weitzman commented 5 March 2011 at 23:19

I kinda prefer --cache as option name but it’s a minor issue.

+++ commands/pm/package_handler/git_drupalorg.inc
@@ -34,9 +34,27 @@ function package_handler_download_project(&$request, $release) {
+    drush_shell_cd_and_exec($gitcache, 'git remote show'); ¶

trailing white space. one more instance later.

+++ commands/pm/package_handler/git_drupalorg.inc
@@ -144,3 +162,41 @@ function package_handler_post_download($project) {
+/**
+ * Add a project to the cache.
+ *
+ * @param $gitcache The location of the gitcache directory.
+ * @param $request The project array with name, base and full (final) paths.
+ * @param $release The release details array from drupal.org
+ */

Lets document why we do this roundabout business with the temp dir.

Also, lets stick to core code conventions and put the description for @param on the line below.

After those are fixed, feel free to commit to master and 7.x-4.x