Two issues:

1) Debugging shows that the git reference cache is not fully populated; it looks like the --tags parameter limits the fetch from the cloned repo to only copy over the tags, but no commit objects:

Executing: git fetch views --tags

Executing: git clone  --reference H:/drush/.drush/gitcache --verbose --progress "sun@git.drupal.org:project/views.git" H:/drush/drush_tmp_1312745954/views
  Cloning into H:/drush/drush_tmp_1312745954/views...
  remote: Counting objects: 6471, done.
  Receiving objects: 100% (5388/5388), 979.52 KiB | 32 KiB/s, done.
  Resolving deltas: 100% (4333/4333), completed with 639 local objects. <----------------------------
  remote: Compressing objects: 100% (1771/1771), done.
  remote: Total 5388 (delta 4333), reused 4513 (delta 3597)

2) The git reference cache never seems to be updated, so if my cache contains a project from January, then each time I drush pm-clone, all new stuff since January is retrieved from scratch again.

CommentFileSizeAuthor
#6 drush.cache-git.6.patch3.93 KBsun
#1 drush.cache-git.1.patch3.26 KBsun
drush.gitcache-fetch.0.patch1.91 KBsun

Comments

sun’s picture

sun
Primary language German
Location Karlsruhe
commented
StatusFileSize
new drush.cache-git.1.patch3.26 KB

Reconsidered. Attached patch implements separate git reference caches per project, and uses git clone --mirror, which seems to have been specifically designed for this job.

We're using a new cache directory to not have the new cache directories within the previous gitcache directory.

msonnabaum’s picture

msonnabaum commented
Status: Needs review » Reviewed & tested by the community

#1 is great. Solves our two major bottlenecks here in my testing.

If no one else objects I'll commit this to both 4 and 5.

The only downside I see is that projects cloned using the old method will need to be re-cloned, which sucks, but the benefit here is enough that I think it's warranted.

damien tournoud’s picture

damien tournoud commented

The only downside I see is that projects cloned using the old method will need to be re-cloned

That's a BC break. As a consequence, this should not go into 4.x.

msonnabaum’s picture

msonnabaum commented
Status: Reviewed & tested by the community » Needs work

Well, I don't know if it's that clear. What I should have said was project cloned using the old method would need to be re-cloned IF they want to take advantage of the improvements made in this patch.

This puts the new git caches in ~/.drush/cache/git, and doesn't do anything to the old gitcache dir, so the repo's cloned referencing gitcache should be fine, as long as you don't delete it.

Also, sdboyer mentioned that the git caches should really appended with .git since they are bare, and I agree.

moshe weitzman’s picture

moshe weitzman commented

Looks safe to me for 4 and 5. Please use drush_directory_cache() instead of custom path building.

sun’s picture

sun
Primary language German
Location Karlsruhe
commented
Status: Needs work » Needs review
StatusFileSize
new drush.cache-git.6.patch3.93 KB

Many more comments for the non-obvious, and also incorporated feedback from previous comments.

msonnabaum’s picture

msonnabaum commented
Version: » All-versions-4.x-dev
Status: Needs review » Patch (to be ported)

Committed to 5, still need to backport.

Thanks sun!

msonnabaum’s picture

msonnabaum commented
Status: Patch (to be ported) » Fixed

Backported.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.