In other words: Finish #1373142: Use the Testing profile. Speed up testbot by 50%

A couple of tests have been left out in the original patch, since they rely on a lot of default configuration provided by the Standard profile.

Converting those tests will speed up the testbot for the full Drupal core test suite significantly.

$ grep -lr "\$profile = 'standard'" core/

core/modules/config/lib/Drupal/config/Tests/ConfigImportAllTest.php
core/modules/comment/lib/Drupal/comment/Tests/CommentPreviewTest.php
core/modules/filter/lib/Drupal/filter/Tests/FilterAdminTest.php
core/modules/help/lib/Drupal/help/Tests/HelpTest.php
core/modules/node/lib/Drupal/node/Tests/NodeAccessBaseTableTest.php
core/modules/node/lib/Drupal/node/Tests/NodeTranslationUITest.php
core/modules/rdf/lib/Drupal/rdf/Tests/StandardProfileTest.php
core/modules/search/lib/Drupal/search/Tests/SearchCommentCountToggleTest.php
core/modules/search/lib/Drupal/search/Tests/SearchCommentTest.php
core/modules/system/lib/Drupal/system/Tests/Cache/PageCacheTagsIntegrationTest.php
core/modules/system/lib/Drupal/system/Tests/Common/NoJavaScriptAnonymousTest.php
core/modules/system/lib/Drupal/system/Tests/Menu/BreadcrumbTest.php
core/modules/system/lib/Drupal/system/Tests/Module/ModuleApiTest.php
core/modules/system/lib/Drupal/system/Tests/Theme/EntityFilteringThemeTest.php
core/modules/taxonomy/lib/Drupal/taxonomy/Tests/LegacyTest.php
core/modules/user/lib/Drupal/user/Tests/UserPictureTest.php
core/profiles/standard/lib/Drupal/standard/Tests/StandardTest.php

Of course, StandardTest remains as the only exception.

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

sun’s picture

FileSize
7.81 KB

Or let's even limit it to 'standard'.

Berdir’s picture

Status: Active » Needs review
FileSize
19.34 KB

This removes the standard profile from a bunch of tests, notably the file field tests.

- MenuTest currently has like a million test assertions duplicated on the default tools menu and a custom one. Not sure if there is a point in keeping the tools part, why should that be any different than a custom menu? Dropped that part.
- Not all tests are working yet, most of the partially converted ones are missing filter stuff, will wait with that after filter has been converted to a config entity.

RDF has quite a few tests that use the standard profile, not yet sure how I can change those, most of the remaining ones then actually often test specific standard.module stuff, not sure if they can be updated...

Status: Needs review » Needs work

The last submitted patch, remove-standard-profile-from-tests-1595028-2.patch, failed testing.

Berdir’s picture

Status: Needs work » Needs review
FileSize
14.88 KB

Re-rolled patch and removed changes from classes that don't work yet.

What about making incremental steps and getting this in while it's hot? :)

I've just checked and the difference is huge. "File field widget test" takes 1m4s on HEAD and 25s with this patch. And the difference is even bigger on the testbot because my system is considerably faster. I'm currently running those file tests manually in another issue and it's a pain.

Berdir’s picture

Issue tags: +Test suite performance

Status: Needs review » Needs work

The last submitted patch, remove-standard-profile-from-tests-1595028-4.patch, failed testing.

Berdir’s picture

Fixed the comment permission problem. Didn't notice that because I run exactly the one test that does enable comment ;)

sun’s picture

This looks awesome! Thanks, @Berdir!

I agree we should rather move forward quickly with these test performance optimizations while they're hot + apply. We can keep the issue open after commit to convert some more.

Only one nitpick:

+++ b/core/modules/file/lib/Drupal/file/Tests/FileFieldDisplayTest.php
@@ -11,6 +11,14 @@
+  public static $modules = array('field_ui');

+++ b/core/modules/file/lib/Drupal/file/Tests/FileFieldRSSContentTest.php
@@ -11,6 +11,14 @@
+  public static $modules = array('field_ui');

+++ b/core/modules/file/lib/Drupal/file/Tests/FileFieldTestBase.php
@@ -21,14 +21,13 @@
   public static $modules = array('file', 'file_module_test', 'field_ui');

+++ b/core/modules/file/lib/Drupal/file/Tests/FileFieldWidgetTest.php
@@ -11,6 +11,14 @@
+  public static $modules = array('field_ui', 'comment');

field_ui gets already enabled by FileFieldTestBase, so we can remove it from the extending test classes.

With that, RTBC, please. :)

Berdir’s picture

Sure, can do that.

Those are the remaining standard profile test:

core/modules/system/lib/Drupal/system/Tests/Menu/BreadcrumbTest.php:  protected $profile = 'standard';
core/modules/system/lib/Drupal/system/Tests/Theme/EntityFilteringThemeTest.php:  protected $profile = 'standard';
core/modules/system/lib/Drupal/system/Tests/Module/ModuleApiTest.php:  protected $profile = 'standard';
core/modules/taxonomy/lib/Drupal/taxonomy/Tests/LegacyTest.php:  protected $profile = 'standard';
core/modules/filter/lib/Drupal/filter/Tests/FilterAdminTest.php:  protected $profile = 'standard';
core/modules/comment/lib/Drupal/comment/Tests/CommentPreviewTest.php:  protected $profile = 'standard';
core/modules/text/lib/Drupal/text/Tests/TextTranslationTest.php:  protected $profile = 'standard';
core/modules/text/lib/Drupal/text/Tests/TextSummaryTest.php:  protected $profile = 'standard';
core/modules/menu/lib/Drupal/menu/Tests/MenuNodeTest.php:  protected $profile = 'standard';
core/modules/help/lib/Drupal/help/Tests/HelpTest.php:  protected $profile = 'standard';
core/modules/rdf/lib/Drupal/rdf/Tests/MappingDefinitionTest.php:  protected $profile = 'standard';
core/modules/rdf/lib/Drupal/rdf/Tests/RdfaMarkupTest.php:  protected $profile = 'standard';
core/modules/node/lib/Drupal/node/Tests/NodeRevisionsTest.php:  protected $profile = "standard";
core/modules/node/lib/Drupal/node/Tests/NodeAccessBaseTableTest.php:  protected $profile = 'standard';
core/modules/search/lib/Drupal/search/Tests/SearchCommentTest.php:  protected $profile = 'standard';
core/modules/search/lib/Drupal/search/Tests/SearchCommentCountToggleTest.php:  protected $profile = 'standard';

I'll continue working on these on the patch is in, waiting for the filters API to become stable before tackling them as quite a few are related to that.

I also noticed a number of them that explicitly specify the testing problem, that shouldn't be necessary unless they extend from a test that specifies standard. Will check them too.

core/modules/simpletest/lib/Drupal/simpletest/Tests/InstallationProfileModuleTestsTest.php:  protected $profile = 'testing';
core/modules/system/lib/Drupal/system/Tests/Form/EmailTest.php:  protected $profile = 'testing';
core/modules/system/lib/Drupal/system/Tests/Form/UrlTest.php:  protected $profile = 'testing';
core/modules/system/lib/Drupal/system/Tests/Pager/PagerTest.php:  protected $profile = 'testing';
core/modules/system/lib/Drupal/system/Tests/Theme/RegistryTest.php:  protected $profile = 'testing';
core/modules/image/lib/Drupal/image/Tests/ImageDimensionsTest.php:  protected $profile = 'testing';
core/modules/filter/lib/Drupal/filter/Tests/FilterSettingsTest.php:  protected $profile = 'testing';
sun’s picture

Status: Needs review » Reviewed & tested by the community

Yay, thank you! :)

Let's quickly get those improvements in. Everyone will be happy, even in case of conflicts. :)

Status: Reviewed & tested by the community » Needs work
Issue tags: -Testing system, -Test suite performance

The last submitted patch, remove-standard-profile-from-tests-1595028-9.patch, failed testing.

Berdir’s picture

Status: Needs work » Needs review

#9: remove-standard-profile-from-tests-1595028-9.patch queued for re-testing.

Random test failures.

Status: Needs review » Needs work

The last submitted patch, remove-standard-profile-from-tests-1595028-9.patch, failed testing.

Berdir’s picture

Status: Needs work » Needs review
Issue tags: +Testing system, +Test suite performance

#9: remove-standard-profile-from-tests-1595028-9.patch queued for re-testing.

Another random upgrade path test failure that has nothing to do with this issue. Testbot, this is not funny :( Why don't you fail with my debug patches...

sun’s picture

Status: Needs review » Reviewed & tested by the community

In any case, back to RTBC, whenever it comes green, please ;)

ParisLiakos’s picture

Well this makes testbot 10 minutes faster! yay!! nice cleanup :)
yeah, forget what i just said, berdir explained me how unpredictable the bot testing times are

webchick’s picture

Status: Reviewed & tested by the community » Fixed

Yay for more fasterer tests. :)

Committed and pushed to 8.x.

Berdir’s picture

Status: Fixed » Active

Back to active for the remaining standard tests.

sun’s picture

You mentioned that many of the remaining would be related to the Filter API?

The API for text formats shouldn't actually change that much anymore — the config conversion patch was pretty much final/complete. We're still working on converting filters into plugins, but I don't think that will change the way of how tests create stub/mock/testing formats.

Speaking of, whenever possible, I think we should try to change the test assertions to expect the output of the default plain_text format, instead of a custom filtered_html/full_html format. However, if that means too much work, then let's just create a new one.

Berdir’s picture

Status: Active » Needs review
FileSize
5.82 KB

Some more tests converted.

Some of the standard profile tests also vanished on their own, e.g. some Rdf tests, nice.

There are still a few fails in the changed tests. We also need to make sure that the tests still test test what they are supposed to.

Status: Needs review » Needs work
Issue tags: -Testing system, -Test suite performance

The last submitted patch, more-testing-tests-1595028-20.patch, failed testing.

sun’s picture

Status: Needs work » Needs review
Issue tags: +Testing system, +Test suite performance
sun’s picture

+++ b/core/modules/text/lib/Drupal/text/Tests/TextSummaryTest.php
@@ -13,7 +13,6 @@
 class TextSummaryTest extends WebTestBase {
-  protected $profile = 'standard';

This one needs more adjustments; @see #1235062: text_summary() ignores filter status + #1868772: Convert filters to plugins

That said, due to the bugs that are apparently contained in the functional code, it may pass even without Standard profile config :)

The last submitted patch, more-testing-tests-1595028-20.patch, failed testing.

sun’s picture

Issue summary: View changes

Remaining tests:

$ grep -lr "\$profile = 'standard'" core/
core/modules/comment/lib/Drupal/comment/Tests/CommentPreviewTest.php
core/modules/filter/lib/Drupal/filter/Tests/FilterAdminTest.php
core/modules/help/lib/Drupal/help/Tests/HelpTest.php
core/modules/node/lib/Drupal/node/Tests/NodeAccessBaseTableTest.php
core/modules/node/lib/Drupal/node/Tests/NodeTranslationUITest.php
core/modules/rdf/lib/Drupal/rdf/Tests/StandardProfileTest.php
core/modules/search/lib/Drupal/search/Tests/SearchCommentCountToggleTest.php
core/modules/search/lib/Drupal/search/Tests/SearchCommentTest.php
core/modules/system/lib/Drupal/system/Tests/Cache/PageCacheTagsIntegrationTest.php
core/modules/system/lib/Drupal/system/Tests/Common/NoJavaScriptAnonymousTest.php
core/modules/system/lib/Drupal/system/Tests/Menu/BreadcrumbTest.php
core/modules/system/lib/Drupal/system/Tests/Module/ModuleApiTest.php
core/modules/system/lib/Drupal/system/Tests/Theme/EntityFilteringThemeTest.php
core/modules/taxonomy/lib/Drupal/taxonomy/Tests/LegacyTest.php
core/modules/user/lib/Drupal/user/Tests/UserPictureTest.php
core/profiles/standard/lib/Drupal/standard/Tests/StandardTest.php

Of course, StandardTest remains as the only exception.

sun’s picture

Issue summary: View changes

Sorry, that grep contained false-positive debugging file on my local disk.

sun’s picture

Status: Needs work » Needs review
FileSize
19.94 KB

Re-rolled + blatantly removing the $profile definition from all affected tests... ;)

Got a little hung-up with ModuleApiTest, because I noticed that that this test can be converted from WebTestBase + Standard profile into DUTB... :) — However, we should likely split that into a separate issue.

tim.plunkett’s picture

  1. +++ b/core/modules/help/lib/Drupal/help/Tests/HelpTest.php
    @@ -19,10 +19,7 @@ class HelpTest extends WebTestBase {
    +  public static $modules = array('help');
    

    This is going to test the hook_help of a LOT less modules, right? Shouldn't we just enable everything?

  2. +++ b/core/modules/rdf/lib/Drupal/rdf/Tests/StandardProfileTest.php
    @@ -15,14 +15,7 @@
     class StandardProfileTest extends WebTestBase {
    ...
    -  /**
    -   * The profile used during tests.
    -   *
    -   * This purposefully uses the standard profile.
    -   *
    -   * @var string
    -   */
    -  public $profile = 'standard';
    

    Ummm.

  3. +++ b/core/modules/system/lib/Drupal/system/Tests/Common/NoJavaScriptAnonymousTest.php
    @@ -14,12 +14,12 @@
     class NoJavaScriptAnonymousTest extends WebTestBase {
    ...
    -      'name' => 'No JavaScript for anonymous users in Standard profile',
    -      'description' => 'Tests that anonymous users are not served any JavaScript in the Standard installation profile.',
    +      'name' => 'No JavaScript for anonymous users',
    +      'description' => 'Tests that anonymous users are not served any JavaScript.',
    

    I mean, changing the name is fine, but wasn't this the whole point of the test?

  4. +++ b/core/modules/system/lib/Drupal/system/Tests/Menu/BreadcrumbTest.php
    @@ -19,12 +19,7 @@ class BreadcrumbTest extends MenuTestBase {
    -  /**
    -   * Test paths in the Standard profile.
    -   */
    -  protected $profile = 'standard';
    

    Same here. It seems to be changing the meaning of the test.

  5. +++ b/core/modules/user/lib/Drupal/user/Tests/UserPictureTest.php
    @@ -14,15 +14,7 @@
     class UserPictureTest extends WebTestBase {
    ...
    -   * Using the standard profile to test user picture config provided by the
    -   * standard profile.
    

    Once more.

Status: Needs review » Needs work

The last submitted patch, 27: test.standard.27.patch, failed testing.

sun’s picture

Issue summary: View changes
Parent issue: » #2190805: Benchmark tests

I'm going to create dedicated spin-off issues for each of the offending tests now.

sun’s picture

Version: 8.0.x-dev » 8.1.x-dev

Drupal 8.0.6 was released on April 6 and is the final bugfix release for the Drupal 8.0.x series. Drupal 8.0.x will not receive any further development aside from security fixes. Drupal 8.1.0-rc1 is now available and sites should prepare to update to 8.1.0.

Bug reports should be targeted against the 8.1.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.2.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

  • webchick committed 0072202 on 8.3.x
    Issue #1595028 by Berdir, sun: Convert remaining tests using Standard...

  • webchick committed 0072202 on 8.3.x
    Issue #1595028 by Berdir, sun: Convert remaining tests using Standard...

Version: 8.1.x-dev » 8.2.x-dev

Drupal 8.1.9 was released on September 7 and is the final bugfix release for the Drupal 8.1.x series. Drupal 8.1.x will not receive any further development aside from security fixes. Drupal 8.2.0-rc1 is now available and sites should prepare to upgrade to 8.2.0.

Bug reports should be targeted against the 8.2.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

  • webchick committed 0072202 on 8.4.x
    Issue #1595028 by Berdir, sun: Convert remaining tests using Standard...

  • webchick committed 0072202 on 8.4.x
    Issue #1595028 by Berdir, sun: Convert remaining tests using Standard...

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.6 was released on February 1, 2017 and is the final full bugfix release for the Drupal 8.2.x series. Drupal 8.2.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.3.0 on April 5, 2017. (Drupal 8.3.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.3.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.3.x-dev » 8.4.x-dev

Drupal 8.3.6 was released on August 2, 2017 and is the final full bugfix release for the Drupal 8.3.x series. Drupal 8.3.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.4.0 on October 4, 2017. (Drupal 8.4.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.4.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.4 was released on January 3, 2018 and is the final full bugfix release for the Drupal 8.4.x series. Drupal 8.4.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.5.0 on March 7, 2018. (Drupal 8.5.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.5.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.6 was released on August 1, 2018 and is the final bugfix release for the Drupal 8.5.x series. Drupal 8.5.x will not receive any further development aside from security fixes. Sites should prepare to update to 8.6.0 on September 5, 2018. (Drupal 8.6.0-rc1 is available for testing.)

Bug reports should be targeted against the 8.6.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.6.x-dev » 8.8.x-dev

Drupal 8.6.x will not receive any further development aside from security fixes. Bug reports should be targeted against the 8.8.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.9.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.7 was released on June 3, 2020 and is the final full bugfix release for the Drupal 8.8.x series. Drupal 8.8.x will not receive any further development aside from security fixes. Sites should prepare to update to Drupal 8.9.0 or Drupal 9.0.0 for ongoing support.

Bug reports should be targeted against the 8.9.x-dev branch from now on, and new development or disruptive changes should be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.9.x-dev » 9.2.x-dev

Drupal 8 is end-of-life as of November 17, 2021. There will not be further changes made to Drupal 8. Bugfixes are now made to the 9.3.x and higher branches only. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.2.x-dev » 9.3.x-dev
quietone’s picture

Title: [meta] Convert tests using Standard profile to use Testing profile instead » Convert tests using Standard profile to use Testing profile instead
Version: 9.3.x-dev » 8.0.x-dev
Status: Active » Fixed

This issue was committed to 8.x and then reopened as a Meta to fix the remaining tests which seem to require a bit more work to install necessary configuration. Experience from #bugsmash has shown that issues that are committed and re-opened can be difficult to triage and figure out what needs to be done. So, instead of keeping this open I'm moving the remaining work to a new Meta, a sibling of this one, so this can be closed.

I also noticed that the majority of the child issues have no issue summary. I'll add a template to them when I update them to change the parent issue.

Also, restoring the meta data for the version this was committed which was 8.x

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.