I had to dig through the source files to figure out how to use the git_drupalorg package handler with a git --reference cache to speed up my development.

After figuring that out, I recognized that drush's cache lives in the user's home directory by default, which is on a different filesystem in my case (slowing down clones from cache), and which also prevents to re-use/share the cache with other local system/network users.

So attached patch is what I came up with; might need some tweaking.

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

sun’s picture

FileSize
3.89 KB

This new clone alias revealed some more instances of drush_delete_dir() that are missing the $force = TRUE argument. Fixed those in attached patch.

EDIT: Those might rather belong into a follow-up patch for #1214018: Bogus filesystem paths, warnings, and stale directories

sun’s picture

The alias could probably have the --dev flag built-in, not sure.

Would also be lovely if there was a way to default --gitusername to the currently logged in user.

---

However, a much larger remaining issue is that - even though the clone seems to come from cache - it is cloned into a temporary directory first, and only afterwards moved to the destination:

H:\test\drupal\sites\all\modules>drush -v clone comment_notify
Initialized Drupal 7.5-dev root directory at H:/test/drupal                                       [notice]

Initialized Drupal site drupal.test at sites/default                                                            [notice]

Executing: git --version
Downloading release history from http://updates.drupal.org/release-history/comment_notify/7.x                   [notice]

http---updates.drupal.org-release-history-comment_notify-7.x retrieved from cache.                              [notice]

Downloading project comment_notify to C:\Users\sun\AppData\Local\Temp/drush_tmp_1311784195 ...                  [notice]

Executing: test $(git rev-parse --is-bare-repository) == "true"
Executing: git init --bare
Executing: git config core.compression 1
Executing: git remote show
Executing: git clone  --reference H:/git/.drush/gitcache --verbose --progress "sun@git.drupal.org:project/comment_notify.git" "C:\\Users\\sun\\AppData\\Local\\Temp/drush_tmp_1311784195/comment_notify"
Executing: git tag -l 7.x-1.0-beta1
Executing: git checkout  7.x-1.0-beta1
Using destination directory H:/test/drupal/sites/all/modules                                      [notice]

Executing: svn info H:/test/drupal/sites/all/modules
Executing: bzr root H:/test/drupal/sites/all/modules
Install location H:/test/drupal/sites/all/modules/comment_notify already exists. Do you want to overwrite it? (y/n): y
Project comment_notify (7.x-1.0-beta1) downloaded to                                                           [success]

H:/test/drupal/sites/all/modules/comment_notify.
Command dispatch complete                                                                                       [notice]

EDIT: To clarify: I don't understand why the repo is cloned into a temp directory first, and only moved afterwards. That looks like a totally superfluous (and slow and heavy) file operation to me...?

sun’s picture

sun’s picture

FileSize
3.96 KB

So #1178036: Exception: bad reference: HEAD in Git->revParse() is caused by drush checking out an odd (possibly bogus?) commit hash by default -- it does not seem to checkout a tag, but its commit hash instead -- or something along those lines...

We can work around that bug by also passing the --dev option, which I think makes sense anyway -- if you drush clone, then you most likely want to work on the currently recommended development branch.

Attached patch performs that change.

sun’s picture

#1235044: halstead/glip 1.0 is outdated, doesn't contain fix for --reference resolves that git_deploy issue with git --reference clones.

sun’s picture

mmm, it's even worse than I outlined in #2 -- drush performs a full separate git clone in the temp directory instead of adding the remote to its gitcache and fetching it:

H:\unleashedmind\test\drupal> drush -d clone cck
...
Shell alias found: clone => pm-download --package-handler=git_drupalorg --gitusername=sun --cache --dev
...
Downloading project cck to C:\Users\sun\AppData\Local\Temp/drush_tmp_1312381479
...
Executing: test $(git rev-parse --is-bare-repository) == "true"
  test: extra argument `rev-parse'
Calling chdir(H:\unleashedmind\test\drupal)

Calling chdir(H:/git/.drush/gitcache)

Executing: git init --bare
  Reinitialized existing Git repository in H:/git/.drush/gitcache/

Calling chdir(H:\unleashedmind\test\drupal)

Calling chdir(H:/git/.drush/gitcache)

Executing: git config core.compression 1

Calling chdir(H:\unleashedmind\test\drupal)

Calling chdir(H:/git/.drush/gitcache)

Executing: git remote show
  backup_migrate
  comment_notify
  oauth
  og_user_roles
  pathauto
  token

Calling chdir(H:\unleashedmind\test\drupal)

Calling chdir(C:\Users\sun\AppData\Local\Temp/drush_tmp_1312381479)

Executing: git clone --bare  --verbose --progress "sun@git.drupal.org:project/cck.git"
...
sun’s picture

Assigned: Unassigned » sun
FileSize
6.08 KB

Alright, attached patch pretty much fixes all issues I've encountered.

I've also seen that #6 happens on purpose in drush_gitcache_add_project() (although the documented reasoning is a bit sparse...)

sun’s picture

umphhhh.... The initial clone is not found, because git remote add points to the temporary directory instead of the bare git repository directory inside of it.

Executing: git clone --bare  --verbose --progress "sun@git.drupal.org:project/cck.git"
  Cloning into bare repository cck.git...
  remote: Counting objects: 13239, done.
  Receiving objects: 100% (13239/13239), 3.78 MiB | 32 KiB/s, done.0% (1/13239)
  Resolving deltas: 100% (9651/9651), done.
  remote: Compressing objects: 100% (4153/4153), done.
  remote: Total 13239 (delta 9651), reused 12428 (delta 8941)

Calling chdir(H:\unleashedmind\test\drupal)

Calling chdir(H:/git/.drush/gitcache)

Executing: git remote add cck H:/git/drush_tmp_1312384407 && git fetch cck --tags
  fatal: 'H:/git/drush_tmp_1312384407' does not appear to be a git repository
  fatal: The remote end hung up unexpectedly

Calling chdir(H:\unleashedmind\test\drupal)

Calling chdir(H:/git/.drush/gitcache)

Executing: git remote set-url cck "sun@git.drupal.org:project/cck.git" && git fetch cck --tags
  warning: no common commits
  From git.drupal.org:project/cck
...

This means that the entire purpose of the separate clone performed in drush_gitcache_add_project() is moot. Instead, the remote is effectively fetched two times from scratch.

sun’s picture

Category: feature » task
FileSize
6.92 KB

Alright, attached patch also fixes that initial gitcache clone path mismatch outlined in #8.

Thus, the only remaining issue is that the final clone is initially performed in a temporary directory, and the files are only moved afterwards into the destination directory.

Due to the new 'home' and 'temp' drushrc options, the massive file move operation is no longer an issue, since it can be performed on the same filesystem by setting the drushrc options.

However, I'd still like to get rid of that move operation, since on Windows, git marks the moved files as "changed" - until I run a git diff, in which case git seems to determine that the files are actually matching the index. Might be better to investigate that in a separate follow-up issue though.

---
Also, but rather minor: the suggested 'clone' alias example still contains my own username. While it's possible that some developers are using a system username that matches their d.o username (as in my case), it might be overkill to implement dynamic placeholder support for --gitusername.

sun’s picture

FileSize
5.4 KB

Follow-up patch in #1214018: Bogus filesystem paths, warnings, and stale directories has been committed.

Incorporated @msonnabaum's suggested alternative fix from https://gist.github.com/1123094

moshe weitzman’s picture

OK, so the bits I am undecided about in the latest patch:

The shell alias 'clone' should probably just map to `git clone` so that folks who confuse their `git` and their `drush` are OK. Thats what we do for pull shell alias. Maybe 'dl-gitdev'?

I think the new options to set tmp and home are a bit dangerous but I'd like feedback on that. I say dangerous because you really change drush's behavior by changing home 'home' (backups, search path for commands, aliases, etc). 'home' and 'tmp' can be changed today using Env variables (see putenv). Thats how the test framework sets them. We would need to mention this env variable technique in the option help.

We could potentially take the hunks that fix reference cache and leave out the drush clone and option stuff

sun’s picture

FileSize
4.6 KB

Replaced 'home' and 'temp' overrides with Windows .bat documentation.

moshe weitzman’s picture

Status: Needs review » Patch (to be ported)

Committed to master (SHA b2cf9ad) without the new options. I think the --reference fixes should backport fine but the shell alias stuff is master only ... Thanks sun.

msonnabaum’s picture

Status: Patch (to be ported) » Fixed

I backported only the changes to the git pm backend. The rest doesn't apply as it builds on changes we made for windows in drush5 that I won't be backporting.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.