Repeat: How many tests will actually fail if setUp() is only executed once for a test case?

This patch cuts down the total time for the full Drupal core test suite to 9 minutes. (from currently 20+ minutes)

API changes

  1. The current setUp() and tearDown() methods are replaced with setUpBeforeClass() and tearDownAfterClass():
    -  function setUp() {
    -    parent::setUp(array('taxonomy', 'node'));
    +  function setUpBeforeClass() {
    +    parent::setUpBeforeClass(array('taxonomy', 'node'));
    
  2. setUpBeforeClass() and tearDownAfterClass() are only invoked once for each test class.
  3. setUp() and tearDown() are however retained, and invoked before and after each test method in a class.
  4. All test methods in a class share the same test environment.
  5. Test methods are executed in the order they are defined.
  6. Test methods need to ensure that they are either executed in the right order, or revert any special conditions that they have changed, so following test methods being executed afterwards still work.

    This especially applies to configuration or context that is set up once for test methods and is shared between them (so called "fixtures").

    Example for stuff that breaks:

    class WhateverTest extends WebTestBase {
      function setUpBeforeClass() {
        parent::setUpBeforeClass();
        $this->admin_user = $this->drupalCreateUser();
        // If any test method logs the user out, following tests will break.
        // Move this into setUp(), so admin_user is logged in for each test method.
        $this->drupalLogin($this->admin_user);
    
        $this->node = $this->drupalCreateNode();
      }
    
      function testMaintenanceAccess() {
        // Maintenance mode will be enabled for all following test methods, which
        // obviously makes most tests fail.
        // Reset or delete the variable to its original/default value at the end of this test.
        variable_set('maintenance_mode', 1);
      }
    
      function testDelete() {
        // This deletes the shared node and thus breaks testResave().
        // Move/relocate this test method after testResave() - or alternatively,
        // take the appropriate measures to restore the expected data at the end of
        // this test (which either means to update $this->node with a new one, or,
        // in case the test requires it, even with the identical ID $this->node->nid).
        node_delete($this->node->nid);
        $this->assertFalse(node_load($this->node->nid), FALSE);
      }
    
      function testResave() {
        $resaved_node = node_load($this->node->nid);
        node_save($resaved_node);
        $this->assertIdentical($this->node->nid, $resaved_node->nid);
      }
    }
    

Help to get this done

@sun will not be able to champion all of the required test changes on his own. But there is a sandbox, so you can help! :)

How to help:

  1. Ask for git write access to the Platform sandbox, if you haven't already. IRC is fastest; but a comment here works, too.
  2. Setup the Platform sandbox for your local D8 git repo/clone.
  3. Checkout @sun's branch into a branch that's specific to you:
    git checkout -b test-once-1411074-[username] platform/test-once-1411074-sun
    git push -u platform test-once-1411074-[username]
    

    This means you're using @sun's branch as the tip to work off from. @sun will review your changes and merge them into the main branch, if appropriate.

  4. To create patches for the testbot, diff against the -base branch:
    git diff platform/test-once-base test-once-1411074-[username]
    
  5. To merge new changes from a test-once branch from another user:
    git merge --no-ff platform/test-once-1411074-[otheruser]
    

    The --no-ff option is key here.

  6. Operations you are not allowed to do:
    • git push platform without specifying a branch name: This would push all of your local branches into the platform sandbox.
    • git push platform [8.x:]test-once-base: Only @sun updates the base branch when needed.
    • git push platform test-once-1411074-sun: You only push to the branch that has your username.
    • git rebase: Rewrites history and makes your branch incompatible / unmergeable.
    • git pull 8.x: Merges in all latest 8.x code into your test-once branch, but all work is based on the -base branch.
    • git merge 8.x: Ditto.
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Status: Needs review » Needs work

The last submitted patch, drupal8.test-setup-once.0.patch, failed testing.

moshe weitzman’s picture

PHPUnit has two methods, setUp() and setUpBeforeClass(). teardown has similar structure. So, a test author can choose to implement either. As a general rule, I agree that one drupal install per class is good.

Nice work so far!

xjm’s picture

#2 sounds like an excellent pattern to me.

sun’s picture

Title: Question: How many tests will actually fail if setUp() is only executed once for a test case? » Setup test environment only once per test class; introduce setUpBeforeClass() and tearDownAfterClass() [PHPUnit]
Status: Needs work » Needs review
Issue tags: +PHPUnit
FileSize
155.21 KB

This makes soooo much sense: http://www.phpunit.de/manual/3.7/en/fixtures.html :-/

So. Introducing setUpBeforeClass() and tearDownAfterClass(). A major leap towards PHPUnit migration.

Awesomeness! :)

Status: Needs review » Needs work

The last submitted patch, drupal8.test-setup-once.4.patch, failed testing.

sun’s picture

Status: Needs work » Needs review
FileSize
179.39 KB

Fixed fixtures.

Status: Needs review » Needs work

The last submitted patch, test.once_.6.patch, failed testing.

sun’s picture

Status: Needs work » Needs review
FileSize
201.84 KB

Fixed more fixtures.

Status: Needs review » Needs work

The last submitted patch, test.once_.8.patch, failed testing.

sun’s picture

Status: Needs work » Needs review
FileSize
211.78 KB

And more.

Status: Needs review » Needs work

The last submitted patch, test.once_.10.patch, failed testing.

sun’s picture

Issue summary: View changes

Updated issue summary.

sun’s picture

Issue summary: View changes

Updated issue summary.

carlescliment’s picture

Take care. The abuse of this method can drive you to coupling between tests and lack of isolation.

BTMash’s picture

Hmm...I both like and the idea and am a little worried by it. The idea that a test class now only has to get setup once for all tests certainly seems pretty awesome if only for the speed bump. But at the same time (and maybe I am mistaken so someone could knock some sense into me), I can't help but think of a class now being one large continuous test. What @carlescliment said in isolating the issue (instead of reading through the setup and test function, you would now be reading through the setup + all test functions prior to and including the test function to debug what the test issue could be) does worry me a bit.

sun’s picture

The downside of shared fixtures are well known. Please read the PHPUnit docs I linked to in #4.

However, this is nothing new in general. If you're afraid of fixtures, then don't use them, and instead create your users/nodes/fields/whatever from scratch for every test method. After all, that's what the setUp() method is for.

The real difference takes effect for the 99.9% of things your test actually does not test, but which you need nevertheless in order to execute the test in the first place. Alas, the extremely optimized testbots (which run with a database on /tmpfs, among many other performance optimizations) spend more than 10 minutes with setting up a new environment for every test method, even though none of our test methods needs a completely new.

Lastly, the shared fixtures between test methods is not concerning at all, given that we're not even able to properly isolate the test from the test runner process. We're currently bumping from one critical bug into the next, and the duct-taping costs plenty of wasted man hours every single week to keep the system running. That's why we want to move away from our homegrown Simpletest and use one of the existing standard testing frameworks in the PHP world instead. This change here is a huge leap towards that.

BTMash’s picture

@sun, Thanks for the clarification. I was reading through it and the line:

It cannot be emphasized enough that sharing fixtures between tests reduces the value of the tests.

Was what brought up part of the doubt. But if we're not able to isolate the tests anyways (and this code moves us towards phpunit which hopefully does help with our test isolation issues and overall will do much more good anyways), then awesome :) I'll make some time and try to post some updates in your sandbox towards this.

sun’s picture

Lacking progress here, let me propose a different plan of attack:

  1. Introduce a new, separate WebRealTest [or whatever name].
  2. Make this base class introduce setUpBeforeClass() and tearDownAfterClass(), as proposed here.
  3. Make this base class use the Testing profile without Node module dependency, as aimed for in #1541298: Remove Node module dependency from Testing profile. Temporarily switch the default profile for WebTest to Minimal profile instead. In other words, make the Testing profile truly mean "nothing."
  4. As a separate follow-up, make this base class implement support for #913086: Allow modules to provide default configuration for running tests
  5. Have it, and start to convert stuff.

I'm fully aware that this will introduce a "second" (duplicate) layer of how people can write tests.

However, I'd rather have this progress, instead of no progress at all.

lucascaro’s picture

@sun what about installing drupal normally in setUpBeforeClass and saving a database snapshot that we can restore on setUp? would that be a good compromise between speed and keeping things working?

moshe weitzman’s picture

we have discussed db restore in the past and Upal even implements that approach. But we are now using the Testing profile a lot more and we need to keep stripping that down so that it does not do menu build and node install and so on. I think we can speed up Testing and use more UnitTestCase such that we don't have to change to DB restore method.

Sylvain Lecoy’s picture

Is there any issues regarding PHPUnit migration ? A [meta] issue or something that records every moves to PHPUnit ?

sun’s picture

@Sylvain Lecoy: There's no official meta issue yet; only #1567500: [meta] Pave the way to replace the testing framework with PHPUnit and possibly rewrite Simpletest. However, that's mainly a temporary dumping ground for collecting relevant input and information. It's definitely missing the latest and greatest results of discussions I had with various people. I will try to create a new and proper meta issue as soon as possible. Meanwhile, all related and forward-thinking issues are tagged with Testing system.

Sylvain Lecoy’s picture

Ok so I created this [meta] issue: #1801176: Deploy a PHPUnit system with a bottom-up approach

I also contacted Moshe Weitzman to ask him if there is any work in progress that I can take over. I am so pissed-off by SimpleTest today that I am making my personal battle to work on replacing it by PHPUnit.

Sylvain Lecoy’s picture

Issue summary: View changes

Added API changes.

sun’s picture

Status: Needs work » Needs review
Issue tags: -API change
FileSize
14.85 KB

Trying once more, with a new approach that hopefully works better:

→ Introduced new SingleEnvInterface to explicitly opt-in to setUpBeforeClass() + tearDownAfterClass().


FWIW, in order to not run into total indentation hell, I also had to adjust the execution flow in TestBase::run(). Will likely split that out into an own issue, so that this is more reviewable.

sun’s picture

Status: Needs review » Needs work

The last submitted patch, 22: test.once_.22.patch, failed testing.

sun’s picture

Status: Needs work » Needs review
FileSize
10.97 KB

#2201783: Simplify execution logic in TestBase::run() has landed + also fixed the undefined $class variable test failures of this patch.

So this is much easier to review now — what do you think of the approach taken? In essence:

  1. For backwards-compatibility, setUpBeforeClass() is not invoked for tests.
  2. If a test class implements the new SingleEnvInterface, then it has to implement setUpBeforeClass() + tearDownAfterClass().
  3. Only through that explicit opt-in, the test execution adapts itself and calls setUpBeforeClass() once before all tests, then proceeds as usual, but Drupal is only installed once for all test methods.
  4. This is accomplished by making the test base classes aware of the different execution paths, so that e.g. WebTestBase::setUpBeforeClass() performs the Drupal installation, and WebTestBase::setUp() turns into a no-op.
tstoeckler’s picture

Now that the test execution process has been revamped a bit I wonder whether we can't get away without the complexity that is added by the conditional logic in this patch.

I.e. couldn't we go back to something like #10 but without converting the whole slew of test classes:
- Make TestBase unconditionally call setUpBeforeClass() and tearDownAfterClass() at the appropriate places.
- Rename WebTestBase::setUp()/tearDown() to setUpBeforeClass() and tearDownAfterClass(). The same for DrupalUnitTestBase and UnitTestBase.
- Add empty setUp() and tearDown() methods to those classes.
- Convert one or two test classes to setUpBeforeClass()/tearDownAfterClass().

??
I may be missing something.

If we stick to this approach we should rename SingleEnvironmentInterface. In general, I'm not sure on the terminology. If we stick to environment the two methods really should be setUpEnvironment() and tearDownEnvironment().

sun’s picture

The old approach requires to convert all tests to the revised test execution flow. That is, because many tests expect that every test method operates in a fresh installation. In turn, all tests would have to be adjusted.

To avoid that, the idea is to allow tests to explicitly opt-in to the different environment setup and execution flow.

Renaming the setUpBeforeClass() and tearDownAfterClass() methods is definitely a no-go, because the names are derived from PHPUnit and meant to be consistent with it. After all, our long-term goal still is to get rid of Simpletest.

tstoeckler’s picture

Hmm... OK, that makes sense then. But the ultimate goal is still to not have this interface and to have all test classes implement this, right? (Even if we might not reach that goal for D8) And for e.g. new tests that get added they should also implement this right off the bat.

Regarding the method names, I didn't know PHPUnit supported that, that's cool. I still think the interface should be renamed to "SingleEnvironmentInterface" and it should have at least one or two more sentences of additional information:
* That implementing the interface leads to a performance increase.
* That all new test classes should implement this so that we can eventually remove the interface (given that that is true)
* That this is compliant with what PHPUnit does.

One nitpick:

+++ b/core/modules/simpletest/lib/Drupal/simpletest/TestBase.php
@@ -801,6 +801,72 @@ public function run(array $methods = array()) {
+        $implements_interface = in_array('Drupal\simpletest\SingleEnvInterface', class_implements($parent_class));

This could use is_subclass_of() I think.

As the logic is a little bit involved this could use a couple more eyes, although I couldn't find any flaws.

Mile23’s picture

I'm not sure why this can't simply be another base class, where setUp() is marked final, and setUpOnce() gets called on subclasses.

sun’s picture

I already investigated the idea of a separate base class (multiple times even), but the major problem with that is that we'd have to duplicate multiple base classes, because TestBase::run() contains the actual test class/method dispatching code, whereas WebTestBase is the subclass that primarily needs the setup-once behavior. In turn, we'd have to duplicate both (or move all code into traits), but I fear that such an approach would make the overall code much more complicated to maintain.

Mile23’s picture

How about adding a flag in getInfo()? This will have the advantage of being easy to backport, too.

webchick’s picture

I admit I have not read this issue, but I did cmd+F for "upstream" at least and didn't find anything mentioned.

I'd really rather not go down the same road we did with SimpleTest of essentially "forking" the library and making it full of Drupalisms, some of them actually useful (like this one sounds). Can we file an upstream patch for this feature?

moshe weitzman’s picture

@webchick setUpBeforeClass() and tearDownAfterClass() are already part of phpunit. This issue brings us *closer* to PHPUnit, not farther.

webchick’s picture

Rock! :) Carry on, then!

Mile23’s picture

@webchick: I'd really rather not go down the same road we did with SimpleTest of essentially "forking" the library and making it full of Drupalisms,

Pretty sure that boat has sailed, unless the goal is to re-implement SimpleTest based on the real framework, in which case this issue needs to be waaaay postponed. https://packagist.org/packages/simpletest/simpletest

@moshe: setUpBeforeClass() and tearDownAfterClass() are already part of phpunit. This issue brings us *closer* to PHPUnit, not farther.

How to make Drupal do modern, performant functional testing using PHPUnit as a framework:

1) Decide that the whole thing can be dependency injected by saying $app = new DrupalKernel('test', $etc, $etc);

2) See: http://symfony.com/doc/current/book/testing.html#your-first-functional-test

Seems obvious to me this is not a goal for Drupal 8 or it would have happened a while back. Like, first. :-)

But as for this issue: Which do people prefer? Add an interface, or make a flag in getInfo()?

jhedstrom’s picture

Status: Needs review » Needs work

Patch no longer applies, but it is probably more work than a simple re-roll at this point since much has changed in the past year.

Version: 8.0.x-dev » 8.1.x-dev

Drupal 8.0.6 was released on April 6 and is the final bugfix release for the Drupal 8.0.x series. Drupal 8.0.x will not receive any further development aside from security fixes. Drupal 8.1.0-rc1 is now available and sites should prepare to update to 8.1.0.

Bug reports should be targeted against the 8.1.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.2.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.1.x-dev » 8.2.x-dev

Drupal 8.1.9 was released on September 7 and is the final bugfix release for the Drupal 8.1.x series. Drupal 8.1.x will not receive any further development aside from security fixes. Drupal 8.2.0-rc1 is now available and sites should prepare to upgrade to 8.2.0.

Bug reports should be targeted against the 8.2.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.6 was released on February 1, 2017 and is the final full bugfix release for the Drupal 8.2.x series. Drupal 8.2.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.3.0 on April 5, 2017. (Drupal 8.3.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.3.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.3.x-dev » 8.4.x-dev

Drupal 8.3.6 was released on August 2, 2017 and is the final full bugfix release for the Drupal 8.3.x series. Drupal 8.3.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.4.0 on October 4, 2017. (Drupal 8.4.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.4.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.4 was released on January 3, 2018 and is the final full bugfix release for the Drupal 8.4.x series. Drupal 8.4.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.5.0 on March 7, 2018. (Drupal 8.5.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.5.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.6 was released on August 1, 2018 and is the final bugfix release for the Drupal 8.5.x series. Drupal 8.5.x will not receive any further development aside from security fixes. Sites should prepare to update to 8.6.0 on September 5, 2018. (Drupal 8.6.0-rc1 is available for testing.)

Bug reports should be targeted against the 8.6.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

apaderno’s picture

Version: 8.6.x-dev » 8.7.x-dev

Version: 8.7.x-dev » 8.8.x-dev

Drupal 8.7.0-alpha1 will be released the week of March 11, 2019, which means new developments and disruptive changes should now be targeted against the 8.8.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.9.x-dev » 9.1.x-dev

Drupal 8.9.0-beta1 was released on March 20, 2020. 8.9.x is the final, long-term support (LTS) minor release of Drupal 8, which means new developments and disruptive changes should now be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.5.x-dev » 10.1.x-dev

Drupal 9.5.0-beta2 and Drupal 10.0.0-beta2 were released on September 29, 2022, which means new developments and disruptive changes should now be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

mondrake’s picture

Component: simpletest.module » phpunit

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

joachim’s picture

Title: Setup test environment only once per test class; introduce setUpBeforeClass() and tearDownAfterClass() [PHPUnit] » Add a flag to set up test environment only once per test class; introduce setUpBeforeClass() and tearDownAfterClass() [PHPUnit]
Issue tags: +sustainability

This should be resurrected - but should be split into separate issues for Kernel / Browser / JS tests.

Mile23’s picture

Chi’s picture

It feels if we had fixed this issue in 2012 the Drupal Association could save tens of thousands of dollars on CI environments.

joachim’s picture

(Posted on wrong issue)

mgifford’s picture

How often are we running these tests? Seems like a good investment of time to cut the execution time in half, even if it's a matter of allowing us to know earlier that the patch works.