Automated patch testing (testing.drupal.org)

Since November 2008, patches against Drupal HEAD are tested automatically for various issues by http://testing.drupal.org. This means you may see issues updated by the 'System message' user, saying 'the latest patch on this issue failed testing'.

Why automated patch testing?

More people submit patches than test them, and more people review patches than commit them (there are only usually two core maintainers) so core always operates with a backlog of patches to review. This works very well in terms of keeping the code quality in Drupal high, but it makes the workload of those reviewing and testing patches quite high.

With the introduction of a testing framework into core early in the Drupal 7 release cycle, much of the work of 'clicking around' Drupal to see if anything is broken has been reduced - automated tests catch a very high percentage of regressions before they ever make it into core. However, running all tests on a patch takes several minutes, and you may not discover a patch breaks tests until 25 minutes into running tests.

Additionally, many patches to Drupal change APIs, might touch many different lines of code etc., so it's very easy for a patch to go 'stale' - it might fail to apply, break automated tests, or otherwise cease to work. Additionally, many patches are uploaded and set to CNR which don't apply properly, contain PHP syntax errors, or break tests.

testing.drupal.org

To ensure that the development version of Drupal core (HEAD, currently Drupal 7) remains as stable as possible, and to reduce the workload for patch reviewers and core committers, testing.drupal.org downloads, applies and runs automated tests on every relevant patch against HEAD.

Patches are tested if:

• The issue is set to CNR or RTBC
• The patch file has a .patch or .diff extension
• The patch name does not end with -d6 _D6 -d5 or _D5 (prior to the .patch or .diff extension) -this allows backports to be posted without causing failures - see the issue upload description for a more detailed explanation
• The patch has not already failed testing (you can reset manually if you think something went wrong

Patches are queued for testing when submitted, and also retested every 24 hours on a rotating basis (provided they previously passed all tests, and still meet the criteria listed above) to ensure they continue to work with changes to core. Under normal circumstances, test results should be returned to for any testable patches submitted within one hour.

Patches that pass
If the patch passes testing, a green bar will appear under the file download link to confirm this, showing the number of tests run, and a link to the full results.

Patches that fail
Patches may fail for the following reasons:

• The patch fails to apply. This may be due to changes to core, or it may be rolled incorrectly (see http://drupal.org/patch/create for instructions)
• Drupal HEAD fails to install with the patch applied
• The patch contains a PHP syntax error
• The patch causes a failure or exceptions in Drupal's suite of automated tests

In this case, a red bar will appear with a summary of the test results, a link to the full results, and a link to send for re-testing. Additionally if the patch is the most recent upload to the issue, the issue will be marked Code Needs Work.

Patches that are being re-tested
If a patch has been submitted for re-testing, then a yellow bar will appear indicating that the patch is currently queued for re-testing.

Frequently asked questions

What if my patch depends on another patch?
At the moment, the issue should be marked 'postponed', we may add a new status or flag for this later on.

What if I don't want to send a patch for testing?
You can bypass automated testing by using a .txt extension for your patch file.