Cross posted from http://www.angrydonuts.com/practical_growth_how_to_implemen

I should’ve written this up last month sometime, as it mostly results from conversations I had with JonBob and jvandyk over above average but not exactly stellar Thai food in Vancouver.

Drupal’s growth rate has reached the point where individuals testing individual bits is no longer keeping up with the other problems that the type of patching we do can cause: the problem, of course, being that one change here can break something seemingly unrelated over there. This kind of problem exists on almost every type of software, but previously I believe the methodical nature of patch committing by Dries and Steven has helped to mollify it somewhat, and the general smallish nature of patches being committed have made it easier to see what might happen.

Drupal’s growth is skyrocketing, though, and major changes have been put in and are coming in the future.

During the google Summer of Code, which is almost but not quite before my time with Drupal, a student wrote a simpletest module, and populated it with some tests. Moshe, at the very least, still works with this somewhat, but it’s been some months before the project’s been touched. And for simpletest to be useful, it needs to be very, very important to the development process.

Now, I don’t feel the need to address the actual tests here. This isn’t to say that it isn’t a huge, major part of the project, but I believe the actual tests are a different problem that needs to be solved in part by getting someone dedicated to maintaining and writing tests. But that’s not the focus of this article. My focus is getting us to the point where we use the module, and I believe that requires automation.

The ideal we discussed — and by the way, this was mostly JonBob’s idea and I am basically documenting and embellishing it here — is a tiered process that can be started with the minimum functionality and expanded upon until everything works well.

To start with, we need to be able to automate a Drupal install. We don’t have to automate it on every environment; in fact, we only really need to automate it on the testbed, which we can hope would run on drupal.org’s hardware. This would involve:

  • Start with a chrooted environment, for safety.
  • Creating a directory under test.drupal.org that is uniquely named; based on a timestamp or a hash or somesuch. This directory is empty.
  • Doing a fresh CVS checkout of Drupal of the appropriate version. This will usually be HEAD but it can be used to test point releases of older versions as well.
  • Apply any patches that we are going to test. If the patch fails, stop.
  • Creating a database. This requires some security because it will require the mysql root password. Alternatively, we use one test database and use prefixes. But to test properly we may need to have both prefixed and non-prefixed databases available. I think that’s actually not a huge concern.
  • Run the mysql script to create all the databases.
  • Run any additional queries necessary. This includes, at the very least, creating an administrative user the easy way, and ensuring the simpletest module (and any modules that require testing) are turned on. It might require simply activating all core modules.
  • Generate some content. I can imagine many tests will require some existing content in the database. devel.module includes scripts to generate basic content.
  • Run the simpletest batch.

Stage two builds upon this with project module’s help. It requires adding 3 states to project issues: patch (needs automated test), patch (passed automated tests), and patch (failed automated tests). A cron job runs incrementally, and it collects all issues attached to core that are set that they need automated testing. The above is run and the output collected. If all of the tests past, the patch is marked as passed, otherwise it is marked as failed. If a patch fails tests, the patch author must determine if the test broke because his code was broken, or because it changed the expected output. If it’s the latter, the patch can be rerolled to include patches to the test.

A test that is marked as passed is now ready for human review. Once it passes human review, it can then be committed. A process also needs to clean up no longer used test-beds, so they must be carefully tracked for garbage collection purposes. The process could do it right away, but it might be nice to leave the testbed for a few hours so that a tester could check out the instance and perhaps see what’s going on.

Security is an important issue here. We need ways to ensure that random PHP isn’t run via the patches. Perhaps only vetted users are allowed to submit patches for automated testing. Perhaps chrooting the testbed is sufficient security.

As near as I can tell, the actual implementation of this procedure — even without any actual simpletests running — isn’t that hard. I’ve already managed to write a short script that does a Drupal install on my system, just to make it easier for me to do the testing process, and I can expand it a little bit to be closer to the process I’ve outlined above. Most of this is straight-forward, brute force beating on Drupal. Then there’s the matter of tests. But they can come later. As we build a library of tests, this can only get better and better. All tests are maintained by force because patches that break existing tests simply won’t be committed until the tests are updated. If the patch was important, updating the test will be important too.

Comments

moshe weitzman’s picture

this is clearly the best way to proceed. make it so, my friend. if we end up needing separate hardware for this, i think boris can find us a sponsor.

greggles’s picture

virtualization (e.g. vmware, xen) can make this testing easier and make it cover more cases (e.g. PHP5.0.x and MySQL4.x can be saved in vmware environments) without requiring lots of extra hardware. It would be great to dovetail this into the project module to show

number of tests
tests passed
red/green/yellow lights on module homepages based upon the last tests that were run for the module.

I'm dreaming here and don't have the time to put in, but we can all dream sometimes, right?
--
Knaddison Family Blog

merlinofchaos’s picture

That's pretty sophisticated. I think it's a lovely idea, and think it's definitely a next step that needs to be kept in mind, but unless there's a vmware expert that wants to show us how to set that up it's going to have to wait.

On the plus side, though, I'd really like to see something like that when this system is mature.

-- Merlin

[Point the finger: Assign Blame!]
[Read my writing: ehalseymiles.com]
[Read my Coding blog: Angry Donuts]

-- Merlin

[Read my writing: ehalseymiles.com]
[Read my Coding blog: Angry Donuts]

merlinofchaos’s picture

As I've thought about this, VMWare or something of that nature could really help address the security concerns that I have, because you could completely isolate the environment and scrub it down when finished.

-- Merlin

[Point the finger: Assign Blame!]
[Read my writing: ehalseymiles.com]
[Read my Coding blog: Angry Donuts]

-- Merlin

[Read my writing: ehalseymiles.com]
[Read my Coding blog: Angry Donuts]

sepeck’s picture

While they are releasing the formerly known as GSX server for free, I am not sure that the license would cover the proposed use. If it did, VMWare is not that hard to leverage and we could probably come up with some folks.

There are Open Source virtual server solutions of varying degrees of complexity, practicality and usability. I believe we may have some experts in the community on one or two of them without driving everyone to the brink of madness.

-Steven Peck
---------
Test site, always start with a test site.
Drupal Best Practices Guide -|- Black Mountain

-Steven Peck
---------
Test site, always start with a test site.
Drupal Best Practices Guide

robertDouglass’s picture

Actually, the client will be setting it up, but for the same reasons; testing and then resetting in a safe, controlled environment. Perhaps I'll glean enough experience with the whole to be able to help out.

- Robert Douglass

-----
My Drupal book: Building Online Communities with Drupal, phpBB and WordPress

Boris Mann’s picture

None of that nasty VMWare business. Xen!

Basically, we can stand up another box at OSU and run Xen on it.

dopry’s picture

Or vserver... its lighter, you'd have access to the vservers's file systems from the host, (easier to apply patches from the host system where the cron job or testing process is running)...

Could set up a group of different apps quickly *mysql 5, 4, apache 1,2, php 4,5 easily as defaul vservers for copying.

Con: you're stuck testing on linux, and if the point of virtualization is to extend the platforms we test on, then

XEN has the most promise(open source), but lacks the ability to run winders so would only extend testings to freebsd and solaris....

VMware Server... Well it'll give us linux, and winders, but leaves the OS X platform out....

So maybe we should just stick to basic LAMP testing for now.

.darrel.

styro’s picture

from AMD and Intel with their respective virtualisation features will (or at least should hehe) allow Xen to run unmodified guest systems like Windows etc.

Also I think Apache/PHP/MySQL differences seem to have a greater affect on Drupal than the OS itself.

But you're probably right about filesystems (I don't know vserver), Xen doesn't like it's filesystems being changed from underneath it. Unless you use GFS (?) apparently - but that sounds pretty tricky to me.

--
Anton
New to Drupal? Please read this
Also: Forum posting tips

merlinofchaos’s picture

Here's a draft of phase 1. Perhaps I should check this in as a project at some point, but not yet; at the moment, this is a toy to play with.

The script checks out Drupal, sets up a database and settings.php, adds a single record for the admin user with a preset password (admin/potrzebie) and generates some content using Jeff Eaton's patched generate-content.php for devel.module.

The script must be edited for a couple of important values before being run. A future version will probably be smart and check.

Anyway, execute it from the command line as php -e install-drupal.php

http://turmoil.logrus.com/files/test-installer.tar.gz

Next steps:
Installing a patch file.
Installing simpletest.
Running a very simple test.
Extracting patches from project issues.
Auto posting followups to project module.

After that:
Xen! If Boris says it, I'm Right There.

-- Merlin

[Point the finger: Assign Blame!]
[Read my writing: ehalseymiles.com]
[Read my Coding blog: Angry Donuts]

-- Merlin

[Read my writing: ehalseymiles.com]
[Read my Coding blog: Angry Donuts]

greggles’s picture

Merlin -

You're talking about automating all of those "next step" items, aren't you?

Where can the script get the patch? I was thinking of using the patch bingo link, but once you get to the page how does the script decide which link is the right patch to be testing? The last link on the page that points to .patch?

--
Knaddison Family Blog

merlinofchaos’s picture

The basic idea is that we extend project module. For all Drupal Core issues, we have 3 new states:

Patch (needs automated testing)
Patch (passed automated testing)
Patch (failed automated testing)

The system would just do a query on the project database and pull the most recent patch on the issue, and apply that.

-- Merlin

[Point the finger: Assign Blame!]
[Read my writing: ehalseymiles.com]
[Read my Coding blog: Angry Donuts]

-- Merlin

[Read my writing: ehalseymiles.com]
[Read my Coding blog: Angry Donuts]

webchick’s picture

For instance, I can upload a patch this minute that passes all unit tests with flying colours, applies cleanly, etc.

Then in an hour someone can commit another patch that totally changes one aspect of the code (let's say function arguments) that my patch touches. Will someone who clicks on that issue still see "Patch (passed automated testing)?" Or will this check run on a cron job or something?

Also, slightly OT: it would be *awesome* if we could at the same time automate a check to see if a patch still applies and if not, instantly sets the status down to "code needs work" with a little automated comment to the issue; this would save lots of time when reviewing patches.

merlinofchaos’s picture

The check will run on cron. Yes, it is completely possible for two patches to collide and cause breakage that this system can't catch. The only hope is that human review will catch that kind of thing.

Alternatively, whoever commits patches could run the auto-checks slightly more manually. Grab a patch; test a patch; commit a patch. Even though it passed automated testing, if it took 2 days to get reviewed and another 2 days to be gotten to, something could easily have changed since it was done. If the system is *easy* enough (cut & paste patch link into a command line...run run run...) it won't be much trouble for the committers to do stuff like that and catch breakage before it happens.

-- Merlin

[Point the finger: Assign Blame!]
[Read my writing: ehalseymiles.com]
[Read my Coding blog: Angry Donuts]

-- Merlin

[Read my writing: ehalseymiles.com]
[Read my Coding blog: Angry Donuts]

dopry’s picture

When change is committed why not just reset all 'automated testing passed' to 'awaiting automated retesting'...

Then test:
1) all items flagged as awaiting automated retesting.
2) all items flagged as ready for automated testing.

or maybe we can just have a test patch button..... :)

greggles’s picture

or maybe we can just have a test patch button..... :)

I think this is going to be a relatively expensive operation - like 5-10 minutes for the automated "setup, test, teardown" phases to execute. Assuming that there are hundreds of bugs that would be "awaiting automated testing" a queue makes more sense to me than a "test right now".

Now, Moore's law and "hardware is cheap" and all - yeah, fine, let's add a button!

--
Knaddison Family Blog

merlinofchaos’s picture

Yea, my feeling was that it wouldn't be a fast process (at least, not in Internet time) and so trying to provide an instant feedback button could be difficult.

-- Merlin

[Point the finger: Assign Blame!]
[Read my writing: ehalseymiles.com]
[Read my Coding blog: Angry Donuts]

-- Merlin

[Read my writing: ehalseymiles.com]
[Read my Coding blog: Angry Donuts]

dopry’s picture

I was just kidding about the button... However there are about 1300 patches on drupal total, including contrib, and 440 against drupal core lying around...

but, Maybe we can have a prioritize button... that moves a patch to a user_requested_testing queue, so things that developers are ready and interested in working on can be given priority. When the user requested queue is empty, it goes back to linearly testing the issue queue.

I don't think this testing process can run from inside drupal as a module. It will need an external component, maybe even a baby daemon like app that runs until the testing queues are clean...

I can envision a MainLoop like...

1) repository changed?

    no , continue.
    yes, update tested status to awaiting retesting
           update test installations to cvs tag they represent    

2) Issues in user_request_queue

     yes, grab issueid
     no, issues in awaiting retesting queue?
         yes, grab issueid
         no, issue in awaiting testing queue
             yes, grab issue id
             no, sleep for a bit  
                    NEXT

3) Patch -p0 < final patch from issue.

4) run tests...

5) update issue status

5) patch -R -p0 < final patch from issue

NEXT

Zack Rosen’s picture

I love this idea. Id be happy to test more patches if I didn't have to apply the patches on my own :)

kbahey’s picture

I am with Boris on this. My VPS is on xen and it has been rock solid and fast.

It requires changes to the kernel, but Linux already have them, so ti is a non-issue.

It is the real successor to other free virtualizers.
--
Drupal development and customization: 2bits.com
Personal: Baheyeldin.com

--
Drupal performance tuning and optimization, hosting, development, and consulting: 2bits.com, Inc. and Twitter at: @2bits
Personal blog: Ba

greggles’s picture

I don't know if anyone has moved any further on this idea, but here is a little shell action that could help:

Set up a p command to apply patches. I use the following:

#!/bin/bash<br>
wget -O - $1 | patch -p0

From http://www.acko.net/blog/handy-drupal-core-development

--
Growing Venture Solutions
Drupal Implementation and Support in Denver, CO

merlinofchaos’s picture

Sadly, I have not; new job has made it impossible for me to work on this. I was actually just thinking about this this week and meant to put word out:

Is there anyone who would be willing to take point on this for awhile? at the time I started this, I didn't foresee getting a 60 hr/week gig plus some other side work to do. =)

-- Merlin

[Point the finger: Assign Blame!]
[Read my writing: ehalseymiles.com]
[Read my Coding blog: Angry Donuts]

-- Merlin

[Read my writing: ehalseymiles.com]
[Read my Coding blog: Angry Donuts]

greggles’s picture

Perhaps the best way to progress right now is simply to get a DEP together that outlines the ideas and tries to nail down how it would work?

Or at least a plan of an idea and then we can worry about progress from there.

--
Growing Venture Solutions
Drupal Implementation and Support in Denver, CO

merlinofchaos’s picture

Yea, that's a really good idea.

-- Merlin

[Point the finger: Assign Blame!]
[Read my writing: ehalseymiles.com]
[Read my Coding blog: Angry Donuts]

-- Merlin

[Read my writing: ehalseymiles.com]
[Read my Coding blog: Angry Donuts]

robertDouglass’s picture

greggles’s picture

I added some more motivation in there, but I'm not sure how to expand the body of the document further to make it more compelling - I'm happy creating text or editing if you have some more direction.

--
Growing Venture Solutions
Drupal Implementation and Support in Denver, CO

allisterbeharry’s picture

Hello,
I've done a SoC 2007 Drupal proposal on automating the whole process of creating a complete Drupal site with a LAMP stack in a self-contained virtual machine image. A formatted PDF version is here:
http://www.abeharry.info/SoC2007_DrupalAST_FullProposal.pdf
Here is the abstract:

The Drupal automated staging toolkit is a proposed set of code
libraries, file schemas and parsers, and code generators, for
automatically creating a Drupal site with specific module code
versions, sample users and data, and a specific LAMP stack
configuration for hosting the Drupal site. The toolkit also has the
ability to stage this generated site on an existing physical server
location, and also as a self-contained virtual machine consisting of a
minimal Linux environment, required LAMP software, and the Drupal
site.

The automated staging toolkit is designed to be part of an automated
unit- and regression testing environment, by providing testers with a
simple, fast way to automatically generate a complete Drupal site
running specific code versions, and using specific LAMP server
configurations. It is also intended for use as part of a performance
and scalability testing environment by providing the ability to
rapidly build and then benchmark the effects of different application,
web and database server configurations on Drupal site performance and
scalability.

This toolkit will be used in the following way:
1. The tester creates or reuses an XML(or other structured) file using
a schema describing the Drupal site code-tree, including modules
installed/enabled/disabled, and the versions of each module to be
used.
2. The tester creates or reuses an XML file using a schema describing
the LAMP stack web server, PHP/application server, and
database server configuration; e.g Apache vs. Lighttpd, mod_php
vs.FastCGI, choice of op-code cache, MySQL vs. PostgreSQL, and so on.
3. The tester creates or reuses an XML file using a schema describing
the sample users and content data the site will contain.
4. The parsers take each file and generate scripts in a lightweight
language (Python or Ruby or PHP-CLI.) These scripts, when executed,
use functions in the code libraries to download Drupal modules,
generate database scripts and datasets, and write server configuration
files.
5. Given a physical server target location, the Drupal modules,
database scripts and server configuration files are deployed to the
designated server location, to produce a new, ready-to-test Drupal
site.

Time and resource permitting, a builder in the toolkit will also use
the generated Drupal site and servers' configuration as input to build
a self-contained virtual machine image in Xen, VMWare, or potentially
the Amazon EC AMI format. This virtual image can also be be used in
testing environments, including advanced performance testing scenarios
such as evaluating clustering, distributed database topologies, and
alternative storage and computing models like Amazon S3 and Elastic
Cloud. The ability to rapidly generate self-contained Drupal virtual
machine images will also be extremely valuable to Drupal consultants
and solution providers for marketing, prototyping and demonstrating
Drupal solutions to potential clients, and large organizations and
ASPs like CivicSpace looking to take advantage of the massive benefits
of virtualization technology from VMware, Xensource, Amazon, and the
like.

Allister.