By jax on

I was wondering how you guys organize your svn/cvs repository for your drupal installations. Where do you put the contrib modules (if you keep them in your repository)? Do you keep the module code or do you keep only the .tar.gz's? Where do you put your themes/modules? Etc.

The structure I was thinking to implement would be:

/reponame/trunk/themes/mytheme/
/reponame/trunk/modules/mymodule/
/reponame/trunk/vendors/drupal.4.tgz.
/reponame/trunk/vendors/contrib_module.tgz

This way you know which contrib modules are used and which ones are yours. The themes and modules you created can be checked out in
drupal/sites/development.tld when you develop and drupal/sites/example.com for production. I've thought about this a couple of times and every time I come up with another solution.

How do you manage/structure your repository?

Comments

styro’s picture

styro commented

This is the rolls royce gold plated over engineered way not recommended for everyone...

A pain to initially get set up, but sweet once you've got it working and a bit more automated. It probably works best for consultants working on multiple projects.

I fully recommend using svn:externals to develop your modules and themes in separate directories though if you want to reuse them between projects.

drupaldev    (this is where we keep modules and themes we've developed)
   modules
      mymodule1
         branches   (eg branches for each major Drupal version etc)
         trunk
      mymodule2
         branches
         trunk
      ...
   themes
      mytheme1
         branches
         trunk
      mytheme2
         ...
vendor                     (svn vendor drops like from the svn handbook)
   drupal-contrib
      modules
          contribmodule1
             DRUPAL-4-7   (directories for cvs tags and branches)
             HEAD
          contribmodule2
             DRUPAL-4-7
             DRUPAL-5
             HEAD
          ...
   drupal-core-4.6
      current
      4.6.0
      4.6.1
      ...
   drupal-core-4.7
      current
      4.7.0
      4.7.1
      ...
   drupal-core-5
      current
      5.0
      5.1
      ...
project1       (these projects are for each site we make)
   branches
   tags
   trunk
project2
   branches
   tags
   trunk
...

Each project directory is just a copy of a drupal-core vendor drop with various custom modules and themes brought in via the svn:externals property (this is the important part) on the modules and themes directory. That way we can reuse and centrally maintain modules between projects and switch module versions on the fly. New Drupal vendor drops get merged back into the project trees to keep them up to date. We have scripts to deploy branches to all our dev and production instances, and sync databases from production instances back into development ones etc.

--
Anton
New to Drupal? | Forum posting tips | Troubleshooting FAQ

ukdg_phil’s picture

ukdg_phil commented

Interesting setup!

I'm currently in the process of moving over to svn & looking to generally improve my development workflow. This looks like a good way to handle multiple projects.

If you don't mind me asking, how do you go about automating your database sync's? Do you do it with custom scripts to load sql dumps. I suppose what I'm really curious about is schema changes & automating that part, although theirs lots that could go wrong compared to just data updates, so maybe better to keep those manual.

Just wondered how others handled this...

UK Digital Graphics Ltd (ukdg.net)

styro’s picture

styro commented

Our DB syncing scripts only pull data from production into test and dev instances via mysqldump. We don't yet try to automate sending data up to the production instance. So any dev content or schema changes need to be recreated on the production instance.

We use MySQL replication (via an ssh tunnel) to have a production DB replica on our dev server. That way when we do backups or dumps we aren't affecting the production server.

--
Anton
New to Drupal? | Forum posting tips | Troubleshooting FAQ | Project management software knowledge base

ukdg_phil’s picture

ukdg_phil commented

Hi Anton,

Thanks for your responce. That's similar to how I had imagined it might be done. It's probably safer to keep some level of human interaction invloved when updating a live server!

My company has now successfully moved over to svn for all our new projects, & we're slowly importing older projects as they need updating.

Your earlier post has really helped us decide the structure for the repository, and we're adapting it to suit our working style, along with writing scripts to automate adding drupal modules into it & setting the 'svn:externals' properties for sites, linking them to the relevant contrib-modules.

Although it has initially added an extra layer of complexity to our set-up, we're already seeing benifits related to project organisation as well as keeping on top of core & contrib updates. I can see that once we have a few more aspects automated with custom scripts, the time savings will really start to show, and everything should be generally easier to develop with, especially for our less techie designers.

I'd highly recommend svn for anyone reading this thread, perhaps not using quite as complex a repository layout if you only have a few projects, but it will really help developers/companies that handle a large number of client projects.

Thanks again Anton for showing us the light!

Phil
UK Digital Graphics Ltd (ukdg.net)

discursives’s picture

discursives commented

I have been working with some scripting lately as an amateur. Could one of you give me a hint about what that ssh tunnel from live to test for the mysql dump looks like? MySQL in ssh is still a little mysterious and a few pointers would really help me out.

Here's something from the sympal scripts that I was just looking at, trying to wrap my head around what you guys are talking about. This is from the installer, but I see some of the work of getting the mysql loaded in. This is all written in php. Do you guys use php, too, or something else? I wrote my first scripts in bash, and that seems functional, too.

    //We now write a temporary database file to the temp dir.
    //file_put_contents cannot be used, because fclose destroys the tempfile.
    $tmpfname = tempnam("/tmp", "MYSQL_");
    file_put_contents($tmpfname, $content);

    //We now pipe this into mysql. Since we are writing a LOT of data, we use the client mysql for this, and not the database connection with PHP.
    //TODO: rewrite this part to make it use drupals internal database abstraction to write the data to,
    $cmd = escapeshellcmd($sql_binary .' -u '. $variables['mysql_username']->value .' -p'. $variables['mysql_password']->value .' '. $variables['database_name']->value) .' < '. escapeshellcmd($tmpfname);
    report_to_console($cmd, "Executing", TRUE);
    if (exec($cmd)) {
      report_to_console('Inserted database file', "Database", TRUE);
    }
    unlink($tmpfname); // this removes the file
  }
}

Sympal scripts: http://drupal.org/project/sympal_scripts
Sympal scripts blog: http://webschuur.com/taxonomy/term/52
(Credit where credit is due! Thanks Sympal!)

styro’s picture

styro commented

I have been working with some scripting lately as an amateur. Could one of you give me a hint about what that ssh tunnel from live to test for the mysql dump looks like? MySQL in ssh is still a little mysterious and a few pointers would really help me out.

This is all just bash stuff in Debian.

Basically we set up (from the internal dev machine) a tunnel to the web server. The tunnel is on localhost:3307 on the dev server. So from the dev server it looks like the web servers production database is running locally on 3307, and the replication gets set up between the real dev DB (as a slave) on port 3306 and the virtual tunneled DB that appears to be running on 3307 (the master).

Here is /etc/init.d/repltunnel on the devel server:

#!/bin/bash
#
# tunnel
#
# Checks whether port 3307 is listening on localhost.
# Port 3307 is used for mysql replication from DMZ through
# an ssh port forwarding tunnel.
#
# If port 3307 is not open, this script (re)creates the tunnel.
#
# Run as init script. Additionally, running this script from an
# hourly cron job should ensure that the tunnel is kept up most
# of the time.

CHECK=`netstat -A inet -ln|grep "tcp.*127\.0\.0\.1:3307.*LISTEN"`

if [ -z "$CHECK" ]
then
  ssh -f -q -N -L3307:localhost:3306 repl@livewebserver.example.com
fi

That script is also symlinked into /etc/cron.hourly so that the tunnel gets checked every hour and recreated if it is down. The grepping netstat output is probably a bit hackish, but I've never had the need to look for a more 'proper' solution.

You need to set up replication user accounts in both the OS and the DB that can't access more than they need to, and the ssh account needs to be set up using public key auth so you don't need passwords or user intervention to connect.

--
Anton
New to Drupal? | Troubleshooting FAQ
Example knowledge base built with Drupal

mlncn’s picture

mlncn commented

Styro, it's fantastic of you to explain everything so well. I think we'll move more toward your system.

The one actual problem I've been trying to solve, though, is what to do with settings.php -- take it out of version control, use conditional logic on the database settings, or what?

Are you able to use 'localhost' with identical username and password for both dev and test/live with this system?

Many thanks,

ben, Agaric Design Collective :: http://AgaricDesign.com/

benjamin, Agaric

styro’s picture

styro commented

We solve the settings.php issue by creating a multisite setup. Even though we only have 'one' actual site running in each virtual host the version controlled code base has a multisite site directory for each dev/test/production instance - so there is a different settings.php file for each instance.

This way we can do an svn export to any apache virtual host on any server and Drupal will pick up the appropriate settings.php file to use by looking at the host header on the incoming HTTP request.

So yes you can version control the settings.php file if you use a multisite setup.

--
Anton
New to Drupal? | Troubleshooting FAQ
Example knowledge base built with Drupal

mlncn’s picture

mlncn commented

Nice, that just means that any non-database changes to settings.php have to be made across the test and live subdirectories.

~ben

People Who Give a Damn :: http://pwgd.org/ :: Building the infrastructure of a network for everyone
Agaric Design Collective :: http://AgaricDesign.com/ :: Open Source Web Development

benjamin, Agaric

mlncn’s picture

mlncn commented

Agaric's trying to follow this lead, but our understanding of svn:externals is that it's best to use an explicit version control number. (Otherwise, rolling back to a previous version still pulls in the latest version of the directories included by svn:externals.) And it seems like an awful lot of overhead to update the svn:externals property every time a revision is checked in. Is there a way to automate this?

Maybe the main problem is we're trying to bring our various projects into the copy of the drupal-core vendor drop (using svn:externals) for allowing one checkout of an actual multisite setup.

Running around in circles,

~ben

People Who Give a Damn :: http://pwgd.org/ :: Building the infrastructure of a network for everyone
Agaric Design Collective :: http://AgaricDesign.com/ :: Open Source Web Development

benjamin, Agaric

styro’s picture

styro commented

we don't use revision numbers in svn:externals so we don't have that issue. We just use different branches for each module or theme and only update svn:externals when switching branches (eg for a Drupal 4.7 to 5 upgrade).

I can see how using revision numbers for externally developed libraries might be a best practice (in which case wouldn't vendor drops be better anyway?), but it seems excessively redundant when you are developing the svn:externals in parallel and they live in the same repo.

Maybe the main problem is we're trying to bring our various projects into the copy of the drupal-core vendor drop (using svn:externals) for allowing one checkout of an actual multisite setup.

I'm not quite sure exactly what your meant but, we don't bring anything into our vendor drop.

Or if you use multisite for your customer projects it could get complicated to also add multisite for each instance. But for people running their own servers and savvy enough to script deployments, I don't see many advantages with using multisite for different sites - unless maybe you have thousands of very simple ones that can all upgrade at once. With more complicated customer sites, you'd probably want each one to have their own dedicated codebase.

--
Anton
New to Drupal? | Troubleshooting FAQ
Example knowledge base built with Drupal

firebus’s picture

firebus commented

though you shouldn't need to make changes to drupal core or a contrib module, in some cases expediency is the mother of worst-practices.

vendor branches, like you have here, really facilitate these kinds of changes. if you make your own changes in trunk, and commit new versions from vendors in the vendor branch, it's usually easy to merge the vendor changes into trunk without losing your modifications.

if you're *not* going to hack core, then i wonder if keeping all of these vendor branches and using lots of svn:external definitions isn't more trouble than it's worth? for example, instead of keeping all these vendor files in your own source control repo you could keep the compressed tgz files somewhere, and write a deployment script per-project that would uncompress the appropriate drupal and module tars into a deployment location, and check out items from drupaldev as necessary.

if you did want to make changes to core or to a contrib module, i suppose you could always copy a directory from vendor/drupal-contrib/modules into drupaldev and make changes there.

styro’s picture

styro commented

There is an awful lot of stuff now that didn't exist 5 years or so ago when we set up the subversion workflow described above. eg distributed version control, drush, features etc etc.

I wouldn't be setting up something like this from scratch these days. Exactly what I'd do though, well I haven't decided. I'm less involved in the Drupal world these days, and I'm still deciding whether I want to get back up to speed with Drupal 7 and all the new stuff anyway for my last remaining site or move off to (eg) a Python MVC framework instead.