The attached patch adds Mercurial (hg) support to drush_make. Can this be added?

I've, also, attached a test.make file that downloads a dummy module from bitbucket.org for testing.

CommentFileSizeAuthor
#22 627012-drush-vcs_option-1.patch1.45 KBsteven jones
#20 627012-drush-vcs_option.patch1010 bytessteven jones
#16 627012-fix-working-copy-doc.patch1.44 KBdrzraf
#12 627012-optionnal-git-cleanup.patch3.65 KBdrzraf
#9 drush_make-627012-9.patch1.83 KBmfer
#9 test.make_.txt733 bytesmfer
#3 test.make_.txt471 bytesmfer
#3 drush_make_hg_bzr_support.patch2.26 KBmfer
test.make_.txt265 bytesmfer
drush_make_hg_support.patch1.7 KBmfer

Comments

mfer’s picture

mfer commented
Assigned: mfer » Unassigned
dmitrig01’s picture

dmitrig01 commented
Status: Needs review » Needs work

is there any command in hg that can export it, ie without the hidden directories hg adds? that would be preferable, but we haven't found one for git (and i'm not sure about bzr either).

mfer’s picture

mfer commented
Title: Add Mercurial (hg) support » Add Mercurial (hg) support and Update bzr support.
Status: Needs work » Needs review
StatusFileSize
new drush_make_hg_bzr_support.patch2.26 KB
new test.make_.txt471 bytes

With regard to bzr, I think you're looking for the bzr export command. The updated patch has a change to "bzr export /local/path /path/to/repo". With this there is no .bzr directory, just the files. The updated test.make file has a config for materialized_view on the fourkitchens bzr server to test this.

The "svn export" equivalent command in hg is "hg archive" but, it only support local repos. If we use that here we would need to do an "hg clone" to a temp directory, a "hg archive" to the files destination, and then clean out the temp space. Unless someone know a better way. Thoughts?

For now I left the setup as "hg clone".

dmitrig01’s picture

dmitrig01 commented
Category: feature » bug
Priority: Normal » Critical
Status: Needs review » Active

thanks about the bzr. I've committed your patch for now, but I'm keeping it open for the following issues:

I'm not sure about exporting to a temporary directory. I know git could use that too.

what about something like this?

drush_shell_exec("(cd %s; hg archive)", $this->project->tmp_path);

something like this for git (maybe, needs testing)

drush_shell_exec("(cd %s; rm -rf git)", $this->project->tmp_path);

Not sure if $this->project->tmp_path is the right variable, but i think something like the above should work.

Thanks!

mfer’s picture

mfer commented

I think I see where you're going with git. Once you've checked out the project delete the .git directory containing the repo information. Isn't there a cleaner way to do this?

With "hg archive" the repo root cannot be it's destination. Maybe we could a hash for the repo name. Something like:

  // Create an md5 hash to use as the repo name
  $hash = md5($this->project->name);

  // Clone the repo using the md5 hash for the name
  drush_shell_exec("hg clone %s %s/%s", $this->project->download['url'], $this->project->tmp_path, $hash);

  // Create an archive of the repo to the modules location.
  drush_shell_exec("hg archive --repository %s/%s %s/%s", $this->project->tmp_path, $hash, $this->project->tmp_path, $this->project->name);

  // Delete the repo.
  drush_shell_exec("rm -Rf %s/$s", $this->project->tmp_path, $hash);

This would feel cleaner to me if the system temp folder were used and if we were using a command that was windows compatible rather than "rm -Rf".

sdboyer’s picture

sdboyer commented

With git, there's a very clean, very network-efficient option that will work sometimes, and a still-fairly-clean but network inefficient version that will work the rest of the time. Warning, pseudo-code!

  // Attempt the clean, but not-always-supported --remote option on git-archive
  drush_shell_exec('git archive --format=tar --remote=%s | tar -xf - -C %s/%s', $this->project->download['url'], $this->project->tmp_path, $this->project->name);

The git-archive --remote option is dependent on the server having enabled the capability to generate remotely-triggered archives. If it fails - which it will on github, for example - then you fall back to an approach pretty much like the one mfer's laid out in #5.

dmitrig01’s picture

dmitrig01 commented

mfer - we are using the system temp dir. something like /tmp/drush_make_XYZ and the site is in __build__. So it should be perfectly possible to do that.

sdboyer - is there really no cleaner solution? I'm not a git wizard myself so I wouldn't know ;-)

sdboyer’s picture

sdboyer commented

Yeah...unfortunately not :( I mean, maybe I've missed something, but I definitely don't know another way.

The only way to reduce network traffic is to do a 'shallow' clone (available in versions >1.5) with --depth=1 . But the reduction isn't very significant...maybe around 10% a reduction in transfer size. Maybe. See http://blogs.gnome.org/simos/2009/04/18/git-clones-vs-shallow-git-clones/ on that.

Really, the --remote option on `git archive` is supposed to be the way you do this kind of thing, and that approach is quite efficient, if it's available. The kicker there is kinda more github than anything else - see http://groups.google.com/group/github/browse_thread/thread/cfcbcb1dc5f41f16 .

mfer’s picture

mfer commented
Title: Add Mercurial (hg) support and Update bzr support. » Clean up .hg and .git files
Status: Active » Needs review
StatusFileSize
new test.make_.txt733 bytes
new drush_make-627012-9.patch1.83 KB

When projects are grabbed from hg and git repos their repos come along with them. The attached patch cleans that up.

For hg/Mercurial the hg archive command is designed to work like cvs export/svn export. But, it does not work on external repos. Following the hg tips and tricks the best method is to just delete the .hg directory. See http://mercurial.selenic.com/wiki/TipsAndTricks#Make_a_clean_copy_of_a_s...

For git the archive command should be used. But, github does not support archiving from their remote repos. So, this patch does the same thing for git as for hg. It removes the .git folder.

The attached test.make file tests against git, hg, and bzr buy pulling from github, bitbucket, and launchpad.

dmitrig01’s picture

dmitrig01 commented
Status: Needs review » Fixed

Done, thanks

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

drzraf’s picture

drzraf commented
StatusFileSize
new 627012-optionnal-git-cleanup.patch3.65 KB

WTF ?!
why would I want to remove such a useful directory ?
.git allows me to track changes, log, ...

I'm perfectly fine with keeping this directory as I don't want to pull the whole repository each time I want to update my modules.

patch in comment #9 is a regression.
Here's a patch which fixes this in the git case,
it changes the drush_make_download_factory() prototype because the code in its current shape does not offer access to global options at git clone time ('drush_make_download_* context).
[ of course, feel free to change the prototype in any another way though ]

To keep your .git directories, add
options[stripvcsdir] = false
in your makefile, the current value (true) is kept as default.

drzraf’s picture

drzraf commented
Version: 6.x-2.x-dev » 6.x-3.x-dev
Priority: Critical » Normal
Status: Closed (fixed) » Active
joestewart’s picture

joestewart commented

drzraf - doesn't --working-copy already provide what you need?

For more options with working-copy, see these patches - #958844: PATCH: Allow [working-copy] in makefile as well as on commandline and #1206340: introduce an options array in the root level of the makefile

drzraf’s picture

drzraf commented

Oups !

yes --working-copy is what I was looking for !

I would rewrite the description
Where possible, retrieve a working copy of projects from their respective repositories.
to something else like:
Preserve (D)VCS (eg: .git) directory for sources fetched from a repository.

sorry for the noise and thanks for the pointer. I'm going to test #1206340: introduce an options array in the root level of the makefile to setup this kind of option in the makefile itself.

drzraf’s picture

drzraf commented
StatusFileSize
new 627012-fix-working-copy-doc.patch1.44 KB

patch
then it can probably be closed again.

helmo’s picture

helmo commented
Project: Drush Make » Drush
Version: 6.x-3.x-dev »
Component: Code » Make
Status: Active » Needs review

[ Powered by #1115636: Issue Macros and Templates - Drush Make]

Drush make is being merged into drush core (discussed in issue:#1310130: Put drush make in drush core)
This means that the issue queue is also moving. The drush queue has a component 'Make' especially for drush_make.

More information will be posted on the drush_make and drush project pages.

NOTE: The patch from #16 already applies to Drush core.

helmo’s picture

helmo commented
Title: Clean up .hg and .git files » Improve description for --working-copy

I'm changing the issue title to reflect the remaining issue.

I would personally suggest to append something like "This preserves VCS directories like .git for the downloaded projects" to the current description.

Where possible, retrieve a working copy of projects from their respective repositories. This preserves VCS directories like .git for the downloaded projects.

drzraf’s picture

drzraf commented

I find the term retrieve confusing.
AFAICT "retrieving using VCS" only depends on [download][type]
while [working-copy] only affects modules which were actually fetched through VCS in that it preserves/keeps the VCS directories.

steven jones’s picture

steven jones commented
StatusFileSize
new 627012-drush-vcs_option.patch1010 bytes

Here's a patch that changes the text to:

Preserves VCS directories, like .git, for projects downloaded using such methods.

drzraf’s picture

drzraf commented

I'm fine with the above one.
[ it only misses the (obvious) part in commands/make/README.txt ]

steven jones’s picture

steven jones commented
StatusFileSize
new 627012-drush-vcs_option-1.patch1.45 KB

Patch that doesn't miss the obvious part.

helmo’s picture

helmo commented
Status: Needs review » Reviewed & tested by the community

Fine by me.

moshe weitzman’s picture

moshe weitzman commented
Status: Reviewed & tested by the community » Fixed

Committed to master.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.