Last updated May 2, 2013.
Patches against Drupal core are automatically tested against more than 30,000 assertions by http://qa.drupal.org. In addition, many contributed modules provide tests and have automated testing enabled, so are tested in the same way.
Why automated patch testing?
More people submit patches than test them, and more people review patches than commit them (there are only usually two core maintainers), so core committers always operate with a backlog of patches. This works very well in terms of keeping the code quality in Drupal high, but it makes the workload of those reviewing and testing patches quite high.
With the introduction of a testing framework into the Drupal 7 release cycle, the work of testing to see if anything unexpected is broken has been reduced: automated tests catch a high percentage of regressions before the patch makes it into core. However, running all tests on a patch takes several minutes, and you may not discover that a patch breaks until 30-40 minutes into running tests.
Additionally, many patches to Drupal change APIs, might touch many different lines of code, so it's very easy for a patch to go 'stale': the patch might fail to apply, break automated tests, or otherwise cease to work. Additionally, many patches are uploaded and set to "code needs review" which don't apply properly, contain PHP syntax errors, or break tests.
To ensure that the development version of Drupal remains as stable as possible, and to reduce the workload for patch reviewers and core committers, the testing.drupal.org platform downloads, applies and runs automated tests on every relevant patch.
Patches are tested if:
- The project has Enable automated testing setting active. Go to project's Edit page, then to Issues to change the option.
- The issue is set to "code needs review" or "ready to be committed"
- The patch file has a .patch or .diff extension
- The patch name does not end with -do-not-test.patch (this allows backports to be posted without causing failures - see the issue upload description for a more detailed explanation)
- The patch has not already failed testing; you can reset manually if you think something went wrong
Patches are queued for testing when submitted. Under normal circumstances, test results should be returned for any testable patches submitted within one hour.
Patches that pass
If the patch passes testing, a green bar will appear under the file download link to confirm this, showing the number of tests run, and a link to the full results.
Patches that fail
Patches may fail for the following reasons:
- The patch fails to apply. This may be due to changes to core, or it may be rolled incorrectly (see http://drupal.org/patch/create for instructions)
- Drupal fails to install with the patch applied
- The patch contains a PHP syntax error
- The patch causes a failure or exceptions in Drupal's suite of automated tests
In this case, a red bar will appear with a summary of the test results, a link to the full results, and a link to send for re-testing. Additionally if the patch is the most recent upload to the issue, the issue will be marked "needs work".
Patches that are being re-tested
If a patch has been submitted for re-testing, then a yellow bar will appear indicating that the patch is currently queued for re-testing.
Frequently asked questions
How do I find out more about automated testing?
The Testbot project is a centralized place for discussion about testbot problems. The main project page has an FAQ explaining common testbot issues.
What if my patch depends on another patch?
At the moment, the issue should be marked 'postponed', we may add a new status or flag for this later on.
What if I don't want my patch tested automatically?
Name your patch something-do-not-test.patch. For example, an Examples module patch might be named examples.fix_broken_submit_1003046_15-do-not-test.patch.