maintaining Drupal sites using version control

arh1 - September 14, 2007 - 18:48

hello all,

i'm sure there has been ample discussion of this in the past, but i'm not finding it in my searches, so i'll pose the question again: what are peoples' takes on maintaining Drupal sites using version control compared to manual updates via tar'd packages?

i (manually) manage a handful of small Drupal sites currently, but will be building a bunch of new sites in the near future. i'm not too handy with CVS yet, but now would be the time for me to dive in if it seems like "the right way". (and btw, how much of a hassle will it be if/when Drupal switches from CVS to SVN?)

any pointers would be appreciated: links, anecdotal experiences, etc. i've read Pro Drupal Development, and also listened to Lullabot's podcast about their new site (discusses their use of SVN ~14:00), but i'm hoping for some "on-the-ground" experience from other developers or administrators.

thanks in advance,

-- andy

SVN + sitecopy...

dgorton - September 17, 2007 - 17:20

Subversion is a great tool, we made the switch to using it a few years ago and wonder how we survived previously.

I listened to the Lullabot podcast as well for reference and it describes our basic process as well. I happen to develop on a Windows platform (sigh), and use XAMPP (www.apachefriends.org/en/xampp.html for apache, mysql, php) and Tortoise SVN (http://tortoisesvn.tigris.org/ a nice GUI interface on SVN).

This helps us keep our house in order and is extremely useful. One other tool we've added to the mix is sitecopy (www.lyra.org/sitecopy/) and a little script that helps us deploy the most recent, stable version of our code. Sitecopy itself is a bit dated, but works fine - you could swap it for a number of other tools (like rsync, for example).

The basic process, however, is that we keep a development server with our combined updates and when we feel that the code is ready for deployment, we run an update command (a small shell script) which uses sitecopy to push the files to the staging server as well as to add a subversion deployment flag (saying that version so-and-so of the files was pushed to the stage server on thus-and-such-date-and-time). If all tests well on the stage, we repeat the command, but this time to the live server. If we ever have problems, we can roll back to the previously released code (using subversion) and push that.

In this way, we use the benefits of subversion to manage our development process as well as the code that gets deployed to the live site.

So - yes - SVN is great, and any version control at all is a wonderful improvement from none.

Drew Gorton
Gorton Studios

integrating Drupal updates into your codebase

arh1 - September 18, 2007 - 23:09

thanks for the reply, Drew. it's always good to hear how other folks are managing their Drupal code -- really helpful stuff.

but i'm also wondering about the updates coming from Drupal (and contrib code maintainers). how do you integrate those into your codebase? when Drupal releases a security update, for example, do you just grab the tarball and manually overwrite Drupal core in your SVN working directory, or do you update your codebase via CVS? (the latter -- mixing a CVS client and SVN server -- seems counterintuitive to me, but the technique is described briefly in Pro Drupal Development...) either way, do you have any comments on how that process works for you?

-- andy

We do it manually...

dgorton - September 19, 2007 - 02:12

It's perhaps a bit inefficient, but we try to keep on top of releases and like to test them before updating. It has happened that contrib updates ('stable' ones) have broken things - sometimes by changing an API that another module is using, sometimes by just having a bug.
Drupal core, on the other hand, is very stable and good and would be just fine to pull in automagically. It's stable enough, though, that such updates are a touch rare. Besides, I actually like reviewing the core changes (when they occur) - it's a good reason to look at core and continue to familiarize yourself with structures, etc.

So - not so fancy - but it works nicely.

Drew Gorton
Gorton Studios

database changes are the difficult part

dkg - September 19, 2007 - 03:28

SVN (or other revision control systems) are really useful for maintaining sets of files and data similar to a filesystem. It's easy to do rapid deployment of your work across many locations with it, and it's also useful to keep track of what you've been doing (and rolling back to a better version, should you need that). here's another developer talking about how he uses SVN for web site maintenance.

However, with projects like drupal, where much of the configuration and deployment is done via a web interface, it's much more difficult to get a smooth rollout. For example: consider adding a new module. You can add the module to a local working directory (on a development web server), and test it to make sure it works. Then (after committing it to your SVN repository) you can roll it out to a staging server to make sure that you've got the deployment technique right. If that works, you roll it out to your live server.

But at each step along the way, after deploying the files via SVN, there are a series of additional steps that need to be taken, usually via the web interface. You have to enable the module, then you have to choose how to configure its access permissions, etc.

I haven't found a way yet to get a revision control system to easily handle these sorts of db changes, which means that these steps have to be done by hand at each step of the deployment process. This is both tedious and error-prone. And even worse, there's no clean way to roll back the db changes necessarily, should you make a mistake or later decide that you made a bad decision.

If drupal had a clean way to script interactions with its UI, it would be another matter: you could just write the things you wanted to do into a script, and after the svn update (which you've already tested elsewhere), run the script.

If someone knows of a project to create such a scriptable interaction, i'd be curious to hear about it. It would make public deployments much less nerve-wracking.

++ agree

dgorton - September 19, 2007 - 04:06

Extremely good points and very true in our experience as well. On particularly snarly upgrades, we keep pretty thorough documentation (nothing fancy - developer-written but still extensive) and follow the process we used on the stage version exactly. But it's time consuming and error prone and must be very carefully managed.

My suspicion is that this is becoming a more and more significant issue as more and more complex Drupal sites age, grow and evolve in the wild. If history can be any guide, I would suspect that improvements to make it easier to code these changes (and therefore version them) will be a someday-future-focus for Drupal core. Pure speculation, but in my experience, any time we've ever thought 'boy, it should would make sense if Drupal did X or behaved like Y', that change has inevitably made it into the core of either one or two releases into the future - and so quickly that we've never even had cause to articulate those thoughts out in the world. Note: I'm not ascribing any foreknowledge of Drupal nor claiming any mystical powers - just that it seems like the development priorities and decision processes have done an excellent job of serving the needs of the community - at least as I have perceived them.

Along these lines, though, I've heard of a process which involves creating a sitename.module and coding all the database and configuration changes possible via that module. Upgrades are done via running update.php, then.
On the downside, this is somewhat more time consuming and a larger barrier to entry than pointing and clicking, but the upside is that you drop errors out of the process and would gain the time back by repeating the process several times (through, say, a dev, stage and prod environment).

We haven't implemented such a strategy with any of our sites yet, but it sounds like a good approach and a very plausible inclusion into our processes at some point in the future. Anyone have further insight into such a process and it's practical ups and downs?

Drew Gorton
Gorton Studios

...

sepeck - September 19, 2007 - 05:09

Look at Auto Pilot and DAST and HostMaster. These are automation methods.

I have a bunch of sites off one code base. These are standalone sites and use some of the same contrib modules.
I have one code bases that holds two sites for a friend and he does most of the management on it.
I have another code base that holds a business and personal site for the same person so they need to be scheduled updates.
I have my site which can be a little more edgy and experimental so is by its lonesome
And finally, I have a few more installs which are experimental in nature and I use to test and blow up.

In general, I download and manually check in modules updates to SVN. Test on the test site and then update the code bases.

-Steven Peck
---------
Test site, always start with a test site.
Drupal Best Practices Guide -|- Black Mountain

SVN/CVS alternative ?

Sree - October 16, 2007 - 09:02

ny other alternative present in drupal for SVN/CVS?

tracking database changes

seanburlington - January 2, 2008 - 23:32

Hi,
this only solves part of the problem - but I found it helpful to at least keep track of database changes which helps to spot missing documentation ...

I run the following script prior to each commit - it creates a sql dump of each table (minus the dump timestamp) - this allows version control to track the changes.

clear_cache just deletes all rows from the cache_tables

#!/bin/bash

mysql mydatabase < clear_cache.sql

for table in $(echo "show tables;" | mysql mydatabase | grep -v Table); do mysqldump --skip-opt mydatabase $table | grep -v 'Dump completed ' > ${table}.sql  ; done

 
 

Drupal is a registered trademark of Dries Buytaert.