Automated testing

Last updated on
15 May 2023

If your project (module, theme, distribution) contains automated test code, you can set up the automated tests to run daily or weekly, and/or to run when new patches are submitted.

Enabling automated testing

  1. Make sure the project has a development release.
  2. From the project page, click "Automated testing".
  3. Click Add test / retest.
  4. Make sure "new test" is selected. Select the Environment, Core version, and Schedule for testing, and save.
  5. Repeat these steps to set up additional schedules.

Set up automated testing for:

  • The maintained branches of your project, usually weekly, against the latest compatible Core version. Sometimes things change in Drupal core that affect contributed projects, so this will test that your project continues to work with the latest version of Drupal core.
  • Patch testing ("Run on commit and for issues"), for each maintained branch. This will make it so when people upload patches or submit merge requests for your project, they will automatically trigger a run of the automated tests.

For more details, see Drupal.org Testing Policy.

Add a composer.json file

If your project has dependencies on other modules or themes, you should definitely add a composer.json file to your project, to list the dependencies for automated testing, because the automated test framework uses the composer.json file to import the dependencies. If you don't have a composer.json:

  • Testing merge requests will not work at all in automated testing
  • Testing patches that contain changes to the dependencies will not work

Why set up automated patch testing?

  • More people submit patches than test them, and more people review patches than commit them. This ensures high code quality, but also results in a high workload for those reviewing and testing patches.

  • Submitted patches might fail to apply properly, contain PHP syntax errors, or cause regressions. Automated tests catch these errors easily.

  • Patches may change APIs or alter many different lines of code. Consequently, a previously valid patch might fail to apply, break automated tests, or otherwise cease to work.

  • Following the introduction of a testing framework during the Drupal 7 release cycle, automated tests catch many regressions before they are committed. However, running all tests on a patch locally takes several hours.

When patches are tested

Patches (and merge requests) are tested if:

  • You have set up automated testing for patches in this branch of your project.
  • The issue is set to "Needs Review" or " Reviewed and Tested By the Community." (For more information on status settings, see status settings for an issue.)
  • The patch file has a .patch or .diff extension, or a merge request is created.
  • The patch has not already failed testing; you can reset manually if you think
    something went wrong.

Patches for different versions

  • Patches can only be tested against the current version of the issue.
  • You should not change the version of an issue to test a patch for a different version. (See the Drupal core backport policy for more information.) Instead, if you wish to upload a patch for a different version, name the patch so that the testbot skips it (see below).

Skipping patches

Patches are skipped if the file name (before the .patch or .diff extension) ends with do-not-test. For example, if you wish to upload a Drupal 7 version of a Drupal core patch when an issue is filed against Drupal 8, you can name the Drupal 7 patch D7-fix_bug_555555-6-do-not-test.patch. It will then be skipped by testbot so that the issue is not marked "needs work" when the Drupal 7 patch fails to apply to Drupal 8.

The testbot queue

Patches are queued for testing when submitted. Under normal circumstances, test results should be returned within one hour for any testable patches submitted.

Patches that pass

If the patch passes testing, a green bar will appear under the file download link to confirm this, showing the number of tests run, and a link to the full results.

Patches that fail

Patches may fail for the following reasons:

  • The patch fails to apply. This may be due to changes to core, or it may be rolled incorrectly. (See Making a Drupal patch with Git for instructions.)
  • Drupal HEAD fails to install with the patch applied
  • The patch contains a PHP syntax error
  • The patch causes a failure or exceptions in Drupal's suite of automated tests

In this case, a red bar will appear with a summary of the test results, a link to the full results, and a link to send for re-testing. Additionally if the patch is the most recent upload to the issue, the issue will be marked "needs work".

Patches that are being re-tested

If a patch has been submitted for re-testing, then a yellow bar will appear indicating that the patch is currently queued for re-testing.

Frequently asked questions

What if my patch depends on another patch?

Mark the issue postponed. (A new status or flag might be added for this in the future.)

What if I don't want my patch tested automatically?

You can bypass automated testing by naming it patchname-do-not-test.patch.

Can I label my patch with "d7" to have it tested against 7.x in an issue for 8.x?

Uploading a patch named (e.g.) patchname-d7.patch will not cause a Drupal core patch to be tested against Drupal 7.x if the issue is filed against 8.x. Please see above regarding patches for different versions. Also see this note about creating patches for multiple Drupal versions for more information.

I created a merge request and set "Status" to "Needs review" but tests are not running?

Are "Add test / retest" links also missing? Check if Automated testing is enabled for the branch, under "Automated testing" on the project page. See also #3358957: Tests disappear in MR after branch update and rebase.

Help improve this page

Page status: No known problems

You can: