I know that the PM commands already integrate with various vcs such as svn and cvs, but I thought that it would also be useful to have drush commit, revert and update commands.

The basic concept is that the vcs suite of commands would be wrappers around the corresponding vcs command, but would be done in such a way as to insure that the database would stay in sync with the files. For example, doing a drush revert would not only revert the filesystem, but would also revert the database backup and re-import it into mysql / postgres via sql-sync.

Checking the database into the vcs would not be the only option, though. For example, in postgres, it is possible to create journal files that record the transaction sequences in the database. If you have a transaction number, you can restore a database up to a certain point by importing the last full backup and replaying the appropriate journal entries. If these db dumps are backed up somewhere other than the vcs, then a drush commit would store the files + the transaction sequence number (in some file) in the vcs, and drush revert would rsync the journel files and then import up to the stored transaction sequence number. drush update would work the same way.

I propose a series of vcs commands that would start out by storing db dumps in the repository, but an engine system would be utilized, as in the PM section, so that a different vcs and a different db backup / restore could also be implemented. Seems this would be better off being independent of the PM commands rather than extending them, since repository actions are not always tied to code updates.

Comments

benklocek’s picture

benklocek CreditAttribution: benklocek commented

How would this play with git? In my experience, keeping DB files in a git repo, can create memory issues when pushing and pulling.

greg.1.anderson’s picture

greg.1.anderson CreditAttribution: greg.1.anderson commented

See paragraph 3 in #0. I don't know if mysql supports journaling, so maybe git isn't a good choice for mysql users with large databases. You could still use git for your code development, and make a local file-based svn repository just for managing the deployed files + db.

The point is, db + file synchronization is critical, and this is the weak point of checking in the files of a drupal site. Whatever mechanism you use to back up your db, you also need a way to keep track of where the old db revisions are, so that you can revert the correct one when updating or rolling back your code. An engine-based implementation would allow you to choose how to implement the db saves and restore.

So, you tell me: how would this play with git? ;)

moshe weitzman’s picture

moshe weitzman CreditAttribution: moshe weitzman commented

I don't know that I would use this, but I'm willing to give it a go.

FYI, we added --ordered-dump option on sql-dump so that a DB dump would diff nicely and thus be easy/efficient to put into VCS. I agree with starting with a DB in the VCS strategy. Alas that option is slower (I don't know how much slower) so it could be annoying if frequently used.

That postgres feature sounds really nice. Wake up mysql!

greg.1.anderson’s picture

greg.1.anderson CreditAttribution: greg.1.anderson commented

For reference, the postgres feature is called Point-In-Time recovery. You just need to do a SELECT pg_start_backup('some unique label', true);, archive the "write-ahead log" files (e.g. with rsync or tar), and then SELECT pg_stop_backup();. The manual page quoted above has examples of what you can do with this archive when it's time to recover it.

The point of this feature is to associate a database state with a set of Drupal files. Any technique anyone is currently using to archive a database so that it may later be restored with the corresponding Drupal + contrib module versions could also be adapted to use with this command.

Alternate strategies for potential additional db backup / restore engines are welcome.

Owen Barton’s picture

Owen Barton CreditAttribution: Owen Barton commented

I think this will become more and more important as d.o. switches to git - people will want to start running local branches of projects in much the same way we do with CVS now. Of course if the site is in git then we would need to do a git submodule add, but if not we would need to do a git clone, and we would need a concept of git pull vs git pull --rebase etc so it could get fairly complex. There are a bunch of feature requests with respect to pm-update stuff at http://drupal.org/node/436978#comment-1547080

Note that the current version control system does have the facility to detect what vcs the current directory is running, and load the appropriate engine (which implements the drush_pm_version_control interface). I think we could simply move that out to a separate commandfile (I guess?), and then flush out the interface with some more actions/checks.

MySQL can have binary logs, and so can do point in time recovery (see http://dev.mysql.com/doc/refman/5.1/en/point-in-time-recovery.html), but the logs are separate to the database file and recovery is quite a bit more involved (to the extent that I am not sure it would be feasible).

I think databases are a nice place to start - not always useful for everyone, but even if you don't actively use them in your development workflow it could still be handy for diagnostic purposes. The other thing that comes to mind (non-core, though) is features - it would great if these could merge/commit themselves intelligently, similar to project updates.

greg.1.anderson’s picture

greg.1.anderson CreditAttribution: greg.1.anderson commented

Yes, re-using PM code is a good idea, and was on my mind. I will definitely look into extending the existing version control interface.

Owen Barton’s picture

Owen Barton CreditAttribution: Owen Barton commented

http://drupal.org/project/drush_git probably has some interesting ideas also...

moshe weitzman’s picture

moshe weitzman CreditAttribution: moshe weitzman commented

FYI, we have tried to make sql-dump CVS friendly with --ordered-dump option which improves efficiency of diff and rsync.

luchochs’s picture

luchochs CreditAttribution: luchochs commented

Words welcome?
If so, someone let me know please and I will write a complement comment (I take too long to translate).
Update: better expressed.

greg.1.anderson’s picture

greg.1.anderson CreditAttribution: greg.1.anderson commented

Words are welcome. I won't be getting to this for a while, so take your time.

Owen Barton’s picture

Owen Barton CreditAttribution: Owen Barton commented

Looks like joshk and sdboyer have started on some really nice looking deep git integration Drush commands. I think these might be a good basis for next generation pm-style commands - see http://groups.drupal.org/node/140949

greg.1.anderson’s picture

greg.1.anderson CreditAttribution: greg.1.anderson commented
Status: Active » Postponed

Thanks, Owen, that is definitely a cool project. I needed a two-repository project for a d.o fork not long ago, and did not have the git-foo to set it up, so I'm definitely going to get on the dog sled.

The main goal of this issue (drush vcs commands) is to allow database rollbacks to old backups in a way that also keeps the database in-sync with the code (rolling back the db often requires rolling back the code, & visa-versa, to avoid having your db schema in a different state than the code expects it to be). Tracking this by hand is a bit annoying.

I think that at this point I am ready to start ignoring any vcs other than git when doing Drupal work, so maybe this work (db-to-code-version references) might be better done in dog than drush. Won't have time to work on this for a while, though -- I am severely backlogged! (Are we not all?)

Setting status to 'postponed' until I can decide where to continue work.

sdboyer’s picture

sdboyer CreditAttribution: sdboyer commented

Since Owen pointed to here, let me just throw the link in: http://drupal.org/project/dog :)

so I'm definitely going to get on the dog sled

YAAAAAY!!

greg.1.anderson’s picture

greg.1.anderson CreditAttribution: greg.1.anderson commented
Version: » 8.x-6.x-dev
Status: Postponed » Closed (won't fix)
Issue tags: +Needs migration

This issue was marked closed (won't fix) because Drush has moved to Github.

If this feature is still desired, you may copy it to our Github project. For best results, create a Pull Request that has been updated for the master branch. Post a link here to the PR, and please also change the status of this issue to closed (duplicate).

Please ask support questions on Drupal Answers.