Git version control engine for --gitsync and more

Simplest way:

git ls-files -o | git update-index --add --stdin
git commit -a -m "<commit message here>"

The first line stages untracked files; the second line properly handles any modified or removed files, and then fires off the commit.

This approach will work for now. However, once we get the d.o migration done, one of my first stops will be drush to put together much deeper integration with the d.o git infra. After all, if you're using git for the project you're on, there's no reason NOT to take advantage of all the rich metadata that'll suddenly be available. Which means we'd be looking at an operation that looks more like a merge or rebase of upstream into the code you have locally (which will be utterly uneventful, and not even generate a commit, unless you have local changes), rather committing the changes from upstream.

Just another approach that also work for now:

shell> git add -A
shell> git commit -m "<commit message here>"

From git help add:

-u, --update
Update only files that git already knows about, staging modified content for commit and marking deleted files for removal. This is similar to what "git commit -a" does in preparation for making a commit, except that the update is limited to paths specified on the command line. If no paths are specified, all tracked files in the current directory and its subdirectories are updated.

-A, --all
Update files that git already knows about (same as --update) and add all untracked files that are not ignored by .gitignore mechanism.

Subscribing...

Yes, the command described in #3 will work - assuming a very recent version of git.

Would be nice.

We already have git_drupalorg package handler in HEAD. If some one with the necessary knowledge on git is wanting to work on the version control engine, yhe're two things to do in drush to start up:
* add git as a vc engine in pm_drush_engine_version_control() in commands/pm/pm.drush.inc
* fork any of the existent vcs engines in: commands/pm/version_control

The problem is that --svnsync is a mildly-broken work process, and it's about to get more broken by #781040: Drush is not removing old files when updating modules with svnsync.

The correct process is:

1. update
2. restore files removed by #781040 with svn update or equiv
3. test
4. commit

#2 is not presently required, which is why --svnsync sort of works, but --svnsync ties commit with update. I think it would be better to commit #781040 and work on #908212: drush vcs commands instead.

@jonhattan - the way moshe explained this to me, this has nothing to do with the git package handler. it has to do purely with interacting with a local repository.

i don't know what the 'best' resolution here is with respect to drush, so i'll just focus on the git bits and let someone else decide how to use it. here's a modification to the commands i gave in #1 - if we're executing from the root of the drupal instance, then for each module (devel installed to a standard location, in the below example), do the following:

git add sites/all/modules/devel
git ls-files -o sites/all/modules/devel | git update-index --add --stdin
git commit -m "<commit message here>"

alternately, you can run the first two lines over each updated project, then make a single commit comprising all the changes with the last line. however y'all want it to work.

the approaches using git commit -a would commit everything in the working tree, which would be potentially VERY confusing.

We still want this

Gracias a todos!
Se me está aclarando el panorama! Soy bastante cabeza dura al parecer, gracias por insistir.
A recuperar el tiempo! y perdón!

Thank you all!
I was getting clearer the picture! I'm pretty stubborn apparently, thanks for insisting.
To recover the time! and forgiveness!

PS: First step family!

Yes, I concur; this will help #908212: drush vcs commands, so let's not let perfection stand in the way of improvement.

Here is a decent start. Known issues which would be helpful if others also looked into:

1. I don't think this works with core drupal upgrades, and i'm not entirely clear how that upgrade is supposed to interact with version control engines.
2. Instead of the code by sdboyer #9, I am synching changes just with git add --all sites/all/modules/devel. Seems to work fine.

I made a last minute change in pre_update so hopefully I did not break that.

I'm hoping to get this in for Drush4.

git add --all works quite well - if you have a pretty recent version of git. IIRC -A, --all was added somewhere late in the 1.6 series. The code I posted is safe well back into the 1.5 era, at least.

Before testing the patch here, please update your drush since I made a couple related changes recently. The particular challenge for this patch right now is the drupal core update scenario; get pre_update() to properly determine if the $items_to_test are clean or not. I guess we should model after what bzr and svn are doing there (which is not quite identical I think).

To quickly get to the point where you have an updatecode that shows the problem, apply the latest patch and run these commands (customize to taste). They download an older core and then run updatecode all under the oversight of git.

mkdir tmp
cd tmp
git init
drush dl drupal-7.0-rc2 --drupal-project-rename=docroot
cd docroot
git commit -am "Add RC2"
drush si -y --db-url="mysql://root:@127.0.0.1/core_upc"
drush pm-updatecode -y -d drupal --version-control=git --gitsync

jonhattan is working on improving the status checking commands and should have a patch soon.

After reading through #757790: drush up drupal deletes .svn/.bzr/vcs files within core directories, I am pretty sure the whole $items_to_test stuff can be dropped completely. All we need to do is move the consistency check before we move the core files out of the way. If the specific files we leave behind are consistent before the move, then they will be consistent after the move - there is no benefit in testing them specifically after the move. jonhattan mentioned another case where there was a directory on a shared host the user had no write access to - in this case they should still be able to add it to theie VC ignore file, and the preflight should check clean.

This (supplemental) patch drops the $items_to_test from being generated/called (it doesn't remove the functionality from the engines yet) and moves the order of operations as described. This allows the test script above to complete correctly. It also includes a fix for the git engine option help text.

ok I've seen it clear to remove $items_to_test. Attached patch completely get rid of this. It also fixes minor docs in svn and bzr.

git:

#18 works. This is my testing batch:

mkdir tmp
cd tmp
git init
drush dl drupal-7.0-rc2 --drupal-project-rename=docroot
cd docroot
drush si -y --db-url="mysql://root:root@127.0.0.1/core_upc"
git add .* *
git commit -m "Add RC2"
rm /tmp/log.txt
drush pm-updatecode -y -d drupal --version-control=git --gitsync | tee /tmp/log.txt

I've not tested bzr nor svn after removing $items_to_test. Will do right now.

the patch.

Sorry, *this* is the patch. with new git.inc file.

to test update of core for bzr and svn:

BZR:

mkdir tmp
cd tmp
bzr init
drush dl drupal-7.0-rc2 --drupal-project-rename=docrootbzr
cd docrootbzr
drush si -y --db-url="mysql://root:root@127.0.0.1/core_upc"
bzr add * .*
bzr commit -m "Add RC2"
rm /tmp/log.txt ; drush pm-updatecode -y -d drupal --version-control=bzr --bzrsync 2>&1 | tee /tmp/log.txt

SVN:

mkdir tmp
cd tmp
svnadmin create repo
svn co file:///`pwd`/repo docroot
drush dl drupal-7.0-rc2
mv drupal-7.0-rc2/.* docroot/
mv drupal-7.0-rc2/* docroot/
cd docroot
drush si -y --db-url="mysql://root:root@127.0.0.1/core_upc"
svn add .htaccess *
svn commit -m "Add RC2"
rm /tmp/log.txt ; drush pm-updatecode -y -d drupal --version-control=svn --svnsync 2>&1 | tee /tmp/log.txt

Both tests did work, with that vanilla sites. But @pcambra has found that it doesn't update if there're uncommited changes in /sites/ (for him it was sites/default/settings.php). Previously, sites/ was never in $items_to_test. Should us exclude sites from status check?

My opinion is yes, we should exclude sites from pm-updatecode checks.

I think it is fine to suggest that people simply put any non-version-controlled files in their VC ignore file (rather than trying to rebuild an ignore system ourselves, essentially) - having a clean VC "status" output prior to any change is a good development practice anyway, in my opinion. I think we can easily add a "--skip-preflight" option so that people who cannot or will not use their VC ignore files for whatever reason can still update core/modules (at their own risk of nuking their uncommitted edits, of course).

implemented --skip-preflight per IRC conversation. I have no strong preference.

1. I am seeing gitsync do add for new files and delete for removed ones but I am not seeing any handling for modified files. These need to be added to the git index as well. Without this, they are not included in gitcommit
2. Lets replace $options['package-handler'] = 'cvs'; in examples/example.drushrc.php with 'git_drupalorg' instead. Lets add one for 'version-control'='git'.
3. I think we need a guideline like: don't use --package-handler=git_drupalorg and --version-control=git together. This is very sad. git status thinks there is nothing to add because it is looking at the wrong repos (git.drupal.org). jonhattan suggested a check for git config --get remote.origin.url which prevents errors but does not change the unfortunate guideline. This guideline is so unfortunate we should consider not committing this patch for Drush4.

all three done.
For 3 I've added a new method to VC interface: ->validate($directory). It is a soft validation after the signature succesfully run. In that method we check if $directory is a repo with origin git://git.drupal.org to avoid using git VC.
This method can also be used by other VC engines for example to check availability of --backup-dir or to better fix #1000662: drush dl failing: "Executing: bzr root"..... hangs it.

A bit better text for 'Skip testing if the repository is in a clean state so local changes will be overwritten.' - 'Allow git operations even when the working copy has uncommitted code.'.

I think we can remove the two debug lines like: drush_log(dt('!vcs engine failed to validated.', array('!vcs' => $version_control)), 'debug');

I'm not conceptually clear on the difference between pre_update and validate. It seems like the 'clean working copy' check could be in validate as well?

I think this code is as good as it can be now. We just need to think about how much we want it given the git_drupalorg incompatibility.

Let me see if I understand correctly.

In February, drupal.org will migrate to git.
At that point, you can still dl via wget; in this case, you will be able to use the git version control engine. However, if you use the git package manager (instead of wget), then you cannot use the give version control engine. The only way to fix that would be to make the same directory controlled by two different git repositories, which is impossible.

If I'm not missing something, I think this is still potentially a useful feature, as it is common to dl via wget. You can't switch from the wget package manager to the git package manager without re-downloading the project, so you'd have to do a couple of awkward steps to get into the situation. It should be obvious to people who might try to do this that they would hurt themselves to put one folder under the management of two separate repositories.

The only problem here is that the error reporting might be rather confusing and hard to explain. I still think that we want the feature -- presuming, of course, that I haven't missed some aspect of the problem.

One thing that occurred to me is that the case of using git to manage other git checkouts (via local branches of core, submodules for contrib or whatever) is really very, very different in implementation and workflow to the case of using git to manage a "bunch of files" (which is essentially just the typical svn pattern, but with a different tool).

I think trying to build a single version control engine to handle both scenarios is not a sensible approach, the heuristics and error handling would get very complex and I don't see much upside - rather I am wondering if we should be aiming for 2 git engines, one oriented to regular svn-style file based workflows, and the other to a remote git branching style setup. The former would work only with wget and cvs, whilst the latter would work only with git_drupalorg. The user could choose which one they are using, and/or we could pick up the signature using the "origin git://git.drupal.org" check.

If this sounds like a sensible approach, it seems to suggest that committing this now is the way to go. We can still backport a basic (separate) branching style git engine to 4.x if we like, perhaps use that as a prototype for more radical git-ification of the package and version stuff for 5.x?

The only way to fix that would be to make the same directory controlled by two different git repositories, which is impossible.

I think that could be the part that you are missing here - it actually is possible to have the same directories controlled by two different git repositories. For core, this is actually a simple remote pull-only clone from d.o. with a local branch and a 2nd remote team repository for pushes (a pretty basic way of using git). For contrib it is a bit more complex - you can use git submodules (basically mini git clones/local branches for each project, but the checkout source tracked by a parent repository), or a few other techniques. I dug into this a while ago and wrote up my thoughts at http://groups.drupal.org/node/93449. This kind of approach can give the same benefits as the CVS into SVN approach - you can have 2 remote merge sources (and diff points) - your dev teams version, and the community version - and you can pull in new changes from either as needed. Keeping this all within git also allows you to do lots of cool things, such as see a consolidated log of both sources log entries, rebase your teams changes for cleaner contribution back and so forth. Some of these can get tricky to get your head around, but I am thinking that some drush wrappers around common operations should be able to make this pretty easy.

As you will see from my previous comment though, I think the above should be a separate git engine (different workflow, totally different commands) - and that the current work still has plenty of opportunities to be useful for many site-builders. For people building small sites and/or working on their own, the value of the branching style approach is pretty small (especially weighed against the current learning curve).

I think Owen is on the right track in #31 with the two use cases. What we have built in this issue is a version control engine for managing "a bunch of files" as he puts it. I see this of limited value. In order to use this git.inc, we're going to force users to use wget as their package manager. Thats familiar, but so lame in a git.drupal.org world. Our issue queue will be full of support requests and WTF disappointment. Owen says that this engine will be valuable site builders of small sites but I'm not so sure. They are welcome to use git add and git commit in the meanwhile.

I'm inclined to postpone this issue to drush5, and focus on the other use case. I'm open to discussion though. Many thanks to jonhattan and Owen for polishing this patch.

I'm rolling another RC now without this patch. Could be the final RC.

subscribe

I'm going to mark this 'Won't Fix'. We need to work on a git workflows from scratch. We should do that outside of the version control engine system and maybe outside of the package handler system.

From what I heard at Drupalcon, there is growing momentum for git submodules, despite their shortcomings and complexity. Drush automation can help with the complexity part.

I'm very sorry to open this issue after it's been closed, but I still have to ask if you (moshe mainly) would reconsider adding the functionality in this patch to drush?

If this is not going to happen, I have to ask how I could easily move from my current move from svn to git, while still having a very simple and functional version control syncing for module downloads and updates. So far svn-sync has been working very nicely, and has saved me and my employer loads of time / frustration. It has been especially usefull for updating modules.

What we have is a drushrc.php file for each project, and it usually only has these 2 lines uncommented:

$options['version-control'] = 'svn';
$options['svnsync'] = 1;

We also have update_advanced set to exclude any patched projects, so drush ignores them. This means that for most site updates all we need to do is run drush up -y, and we're done. It's perfectly fine that drush updates the code with wget. Even if drupal.org happens to host it's code on git, we strongly prefer to keep our site fully in our own version control.

Is there any way to achieve the same smooth process in git without complicating our current workflow? So far it seems the best solution for me would be just take this patch and hack my drush on all the 20+ servers it's installed. Or could adding this functionality be reconsidered?

I'm asking this now since our company is in the process of moving from svn to git, and the lack of this patch is hindering our migration for over 10 professional Drupal delevopers.

Have you considered dog? It is specifically designed for tracking patched projects.

My reservations about this still stand.

If you are working from git clones, then after you updatecode, you do a `git commit -a` and all changes are recognized and added to the commit. i think.

If you want this sort of functionality, it would probably be pretty easy to pull code from post_update and post_download (and the functions they call -- sync and commit), and repackage them as a post-update hook in your own commandfile stored at $HOME/.drush.

Thank you for your answers Moshe and Greg :)

Have you considered dog? It is specifically designed for tracking patched projects.

It doesn't seem to have any release yet, and especially nothing production ready. I read the related conversation at http://groups.drupal.org/node/140949 , and this stuff seems advanced and all, but not ready to use, and a bit complex to grasp. I'm also not too familiar with git yet.

If you are working from git clones, then after you updatecode, you do a `git commit -a` and all changes are recognized and added to the commit. i think.

Thanks for this. It might even be enough for our needs. I just today heard that git doesn't have any problems like "missing .svn folders" that mess up a repo. So I guess the actual manual syncing process is not a pain in the ass after all.

Even so, I'm still a bit worried that the new grand git automation plans for Drush end up missing some of the earlier simplicity. But there should be a lot of people whining already, if that was the case, so maybe I'm wrong :)

If you want this sort of functionality, it would probably be pretty easy to pull code from post_update and post_download (and the functions they call -- sync and commit), and repackage them as a post-update hook in your own commandfile stored at $HOME/.drush.

Thanks! I'm not exactly sure on how to actually create this commandfile, would need more handholding with that. Or, maybe I'll just start doing the manual syncing and stop bugging you.

In any case, I'm grateful for your prompt and accurate responses.

See drush topic for help on writing your own commandfiles, and lots of other useful advice.

See also kitten-o-mattic at http://drupal.org/node/392762#comment-2783550. While I do not recommend using this command, it does serve as an interesting example of how to do some post-processing after pm-update, which is exactly what you want to do.

Thanks again for the help. I'll see what I can do with the command files :)

Any chance we could resurrect this?

> If you want this sort of functionality, it would probably be pretty easy to pull code from post_update and post_download (and the functions they call -- sync and commit), and repackage them as a post-update hook in your own commandfile stored at $HOME/.drush.

I am not sure this is doable any more.

I took at look at the kitten-o-matic example, and I have a hook_post_pm_download() implemented, but its only incoming parameter when I do, say 'drush dl PROJECT', is the string of the project name. So it seems that figure out which folder to issue a 'git add' system command on will mean redoing most of the work that's already been done by drush_pm_download().

This issue was marked closed (won't fix) because Drush has moved to Github.

If this feature is still desired, you may copy it to our Github project. For best results, create a Pull Request that has been updated for the master branch. Post a link here to the PR, and please also change the status of this issue to closed (duplicate).

Please ask support questions on Drupal Answers.

https://github.com/drush-ops/drush/issues/83