In light of #2188661: Extension System, Part II: ExtensionDiscovery:

I wanted to know where exactly tests, or respectively Drupal core, is terribly slow.

So I quickly hacked up a naive bench() facility for TestBase, which is wrapped around individual aspects of interest and simply prints to stdout (on CLI only).

My local box shows some interesting things already, curious to see what the testbot review log will show.

CommentFileSizeAuthor
#17 interdiff.txt2.49 KBsun
#17 test.bench_.17.patch11.5 KBsun
#13 interdiff.txt535 bytessun
#13 test.bench_.13.patch10.29 KBsun
#12 interdiff.txt1.95 KBsun
#12 test.bench_.12.patch10.06 KBsun
#9 interdiff.txt1.43 KBsun
#9 test.bench_.9.patch8.99 KBsun
#8 TestTime.png11.88 KBmartin107
#5 interdiff.txt1.44 KBsun
#5 test.bench_.5.patch7.95 KBsun
#2 test.bench_.2.patch7.14 KBsun
test.bench_.0.patch7.87 KBsun
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Status: Needs review » Needs work

The last submitted patch, test.bench_.0.patch, failed testing.

sun’s picture

sun
Primary language German
Location Karlsruhe
CreditAttribution: sun commented
Status: Needs work » Needs review
FileSize
test.bench_.2.patch7.14 KB
sun’s picture

sun
Primary language German
Location Karlsruhe
CreditAttribution: sun commented

2: test.bench_.2.patch queued for re-testing.

sun’s picture

sun
Primary language German
Location Karlsruhe
CreditAttribution: sun commented

2: test.bench_.2.patch queued for re-testing.

sun’s picture

sun
Primary language German
Location Karlsruhe
CreditAttribution: sun commented
FileSize
test.bench_.5.patch7.95 KB
interdiff.txt1.44 KB

Added hit list of most expensive tests.

sun’s picture

sun
Primary language German
Location Karlsruhe
CreditAttribution: sun commented

Well, that was most certainly the best idea ever in terms of benchmarking tests :-)

The testbot spends 14 minutes on Drupal\system\Tests\Module\InstallUninstallTest alone! :-(

The very end of the "Review log" (fieldset) holds the hit list of most expensive tests, ranked by time: https://qa.drupal.org/pifr/test/778973

Excerpt:

Most expensive tests:

 839.917s Drupal\system\Tests\Module\InstallUninstallTest
 459.433s Drupal\system\Tests\Module\ModuleApiTest
 402.332s Drupal\menu_ui\Tests\MenuTest
 358.372s Drupal\node\Tests\NodeTranslationUITest
 310.296s Drupal\config\Tests\ConfigImportAllTest
 307.069s Drupal\config_translation\Tests\ConfigTranslationUiTest
 280.740s Drupal\field_ui\Tests\ManageFieldsTest
 280.146s Drupal\views_ui\Tests\DisplayTest
 230.260s Drupal\locale\Tests\LocaleUpdateTest
 221.772s Drupal\toolbar\Tests\ToolbarAdminMenuTest
 214.073s Drupal\search\Tests\SearchCommentTest
 210.247s Drupal\book\Tests\BookTest
 203.317s Drupal\filter\Tests\FilterAdminTest
 186.199s Drupal\shortcut\Tests\ShortcutSetsTest
 180.838s Drupal\views\Tests\Handler\HandlerTest
 178.960s Drupal\system\Tests\Theme\ThemeTest
 177.949s Drupal\user\Tests\UserCancelTest
 165.510s Drupal\shortcut\Tests\ShortcutLinksTest
 165.398s Drupal\update\Tests\UpdateCoreTest
 163.165s Drupal\taxonomy\Tests\TermTest
 162.484s Drupal\views_ui\Tests\ViewEditTest
 161.890s Drupal\rest\Tests\Views\StyleSerializerTest
 155.513s Drupal\field\Tests\FormTest
 155.027s Drupal\views\Tests\Wizard\BasicTest
 148.723s Drupal\image\Tests\ImageAdminStylesTest
 145.652s Drupal\field_ui\Tests\ManageDisplayTest
 145.230s Drupal\system\Tests\Form\FormTest
 138.012s Drupal\comment\Tests\CommentLinksTest
 134.162s Drupal\comment\Tests\CommentPreviewTest
 131.471s Drupal\options\Tests\OptionsFieldUITest
 130.833s Drupal\options\Tests\OptionsWidgetsTest
 129.077s Drupal\views\Tests\Wizard\TaggedWithTest
 127.540s Drupal\system\Tests\Module\DependencyTest
 126.316s Drupal\user\Tests\UserPictureTest
 126.070s Drupal\tracker\Tests\TrackerTest
 125.962s Drupal\views\Tests\Plugin\DisplayTest
 125.582s Drupal\forum\Tests\ForumTest
 124.869s Drupal\file\Tests\FileFieldWidgetTest
 123.864s Drupal\node\Tests\NodeTypeTest
 123.814s Drupal\views_ui\Tests\WizardTest
 123.728s Drupal\block\Tests\Views\DisplayBlockTest
 122.588s Drupal\xmlrpc\Tests\XmlRpcBasicTest
 120.156s Drupal\simpletest\Tests\SimpleTestTest
 119.359s Drupal\xmlrpc\Tests\XmlRpcMessagesTest
 118.420s Drupal\search\Tests\SearchConfigSettingsFormTest
 114.732s Drupal\views_ui\Tests\HandlerTest
 111.264s Drupal\update\Tests\UpdateContribTest
 109.862s Drupal\system\Tests\Update\UpdateScriptTest
 109.459s Drupal\migrate_drupal\Tests\d6\MigrateDrupal6Test
 109.310s Drupal\comment\Tests\CommentPagerTest
 108.963s Drupal\locale\Tests\LocaleTranslationUiTest
 107.973s Drupal\quickedit\Tests\QuickEditLoadingTest
 106.567s Drupal\aggregator\Tests\FeedParserTest
 105.716s Drupal\views\Tests\Plugin\PagerTest
 105.167s Drupal\system\Tests\Routing\RouterTest
 104.619s Drupal\system\Tests\Installer\InstallerTranslationTest
 102.739s Drupal\config\Tests\ConfigImportUITest
 101.537s Drupal\system\Tests\Batch\ProcessingTest
 100.562s Drupal\views_ui\Tests\DisplayTestCRUD
  99.479s Drupal\system\Tests\Menu\BreadcrumbTest
...
sun’s picture

sun
Primary language German
Location Karlsruhe
CreditAttribution: sun commented

Most of the top-ranking tests are tests that are needlessly using the Standard profile, which circles back into:

#1595028: Convert tests using Standard profile to use Testing profile instead

I'm going to create dedicated spin-off issues for each of the offending tests now.

martin107’s picture

martin107 CreditAttribution: martin107 commented
FileSize
TestTime.png11.88 KB

Including a graph to show test time

My brain found it easier to process in minutes so here is the top 10

Most expensive tests:
839.917s = 14.00min Drupal\system\Tests\Module\InstallUninstallTest
459.433s = 7.66min Drupal\system\Tests\Module\ModuleApiTest
402.332s = 6.71min Drupal\menu_ui\Tests\MenuTest
358.372s = 5.97min Drupal\node\Tests\NodeTranslationUITest
310.296s = 5.17min Drupal\config\Tests\ConfigImportAllTest
307.069s = 5.12min Drupal\config_translation\Tests\ConfigTranslationUiTest
280.740s = 4.68min Drupal\field_ui\Tests\ManageFieldsTest
280.146s = 4.67min Drupal\views_ui\Tests\DisplayTest
230.260s = 3.84min Drupal\locale\Tests\LocaleUpdateTest
221.772s = 3.70min Drupal\toolbar\Tests\ToolbarAdminMenuTest

sun’s picture

sun
Primary language German
Location Karlsruhe
CreditAttribution: sun commented
FileSize
test.bench_.9.patch8.99 KB
interdiff.txt1.43 KB

Even better: Let's make all of 'em FAIL.

Added actual constraint/assertion for total test duration time.

Status: Needs review » Needs work

The last submitted patch, 9: test.bench_.9.patch, failed testing.

sun’s picture

sun
Primary language German
Location Karlsruhe
CreditAttribution: sun commented
Related issues: +#2257427: run-tests.sh builds + rebuilds two kernels from scratch for every PHP process

Discovered an additional perf problem on a different end.

sun’s picture

sun
Primary language German
Location Karlsruhe
CreditAttribution: sun commented
Status: Needs work » Needs review
FileSize
test.bench_.12.patch10.06 KB
interdiff.txt1.95 KB

  1. Bumped max allowed duration to 3 minutes.

  2. Make WebTestBase *validate* that a test actually performed any kind of HTTP request.

Based on my recent findings, the second change will reveal *tons* of crappy legacy tests.

sun’s picture

sun
Primary language German
Location Karlsruhe
CreditAttribution: sun commented
FileSize
test.bench_.13.patch10.29 KB
interdiff.txt535 bytes

Fixed HTTP request tracking flag must be reset to FALSE in setUp().

The last submitted patch, 12: test.bench_.12.patch, failed testing.

Status: Needs review » Needs work

The last submitted patch, 13: test.bench_.13.patch, failed testing.

martin107’s picture

martin107 CreditAttribution: martin107 commented

Looking at FieldTestType which is a candidate for conversion

FieldTestType which extends NodeTestBase which extends ViewTestBase which extends WebTestBase.

Quoting from ViewTestBase

* Use this base test class if you need to emulate a full Drupal installation.
* When possible, ViewUnitTestBase should be used instead. Both base classes
* include the same methods.

It looks like this is a common pattern/mistake and that VIewUnitTest should be used in many more cases.

sun’s picture

sun
Primary language German
Location Karlsruhe
CreditAttribution: sun commented
Status: Needs work » Needs review
FileSize
test.bench_.17.patch11.5 KB
interdiff.txt2.49 KB

Fixed reporting for webtest methods that do not perform a request.

sun’s picture

sun
Primary language German
Location Karlsruhe
CreditAttribution: sun commented

Since that web test vs. DUTB situation is only closely related but actually a different problem space:

#2258173: [meta] Various web tests are not performing any HTTP requests

+ first spin-off, since those tests are dominating the report:

#2258177: Convert migrate_drupal tests to KernelTestBase

How to chunk/bundle the other tests should be discussed in the new meta issue. Perhaps by module? Let's discuss over there.

Status: Needs review » Needs work

The last submitted patch, 17: test.bench_.17.patch, failed testing.

sun’s picture

sun
Primary language German
Location Karlsruhe
CreditAttribution: sun commented

Created #2258241: Remove obsolete Drupal\system\Tests\Module\InstallUninstallTest (yes, I'm that bold.)

sun’s picture

sun
Primary language German
Location Karlsruhe
CreditAttribution: sun commented

Same fate: #2258247: Remove useless Drupal\config\Tests\ConfigImportAllTest

Version: 8.0.x-dev » 8.1.x-dev

Drupal 8.0.6 was released on April 6 and is the final bugfix release for the Drupal 8.0.x series. Drupal 8.0.x will not receive any further development aside from security fixes. Drupal 8.1.0-rc1 is now available and sites should prepare to update to 8.1.0.

Bug reports should be targeted against the 8.1.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.2.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.1.x-dev » 8.2.x-dev

Drupal 8.1.9 was released on September 7 and is the final bugfix release for the Drupal 8.1.x series. Drupal 8.1.x will not receive any further development aside from security fixes. Drupal 8.2.0-rc1 is now available and sites should prepare to upgrade to 8.2.0.

Bug reports should be targeted against the 8.2.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.6 was released on February 1, 2017 and is the final full bugfix release for the Drupal 8.2.x series. Drupal 8.2.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.3.0 on April 5, 2017. (Drupal 8.3.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.3.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.3.x-dev » 8.4.x-dev

Drupal 8.3.6 was released on August 2, 2017 and is the final full bugfix release for the Drupal 8.3.x series. Drupal 8.3.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.4.0 on October 4, 2017. (Drupal 8.4.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.4.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.4 was released on January 3, 2018 and is the final full bugfix release for the Drupal 8.4.x series. Drupal 8.4.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.5.0 on March 7, 2018. (Drupal 8.5.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.5.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.6 was released on August 1, 2018 and is the final bugfix release for the Drupal 8.5.x series. Drupal 8.5.x will not receive any further development aside from security fixes. Sites should prepare to update to 8.6.0 on September 5, 2018. (Drupal 8.6.0-rc1 is available for testing.)

Bug reports should be targeted against the 8.6.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.6.x-dev » 8.8.x-dev

Drupal 8.6.x will not receive any further development aside from security fixes. Bug reports should be targeted against the 8.8.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.9.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.7 was released on June 3, 2020 and is the final full bugfix release for the Drupal 8.8.x series. Drupal 8.8.x will not receive any further development aside from security fixes. Sites should prepare to update to Drupal 8.9.0 or Drupal 9.0.0 for ongoing support.

Bug reports should be targeted against the 8.9.x-dev branch from now on, and new development or disruptive changes should be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.9.x-dev » 9.2.x-dev

Drupal 8 is end-of-life as of November 17, 2021. There will not be further changes made to Drupal 8. Bugfixes are now made to the 9.3.x and higher branches only. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.2.x-dev » 9.3.x-dev
quietone’s picture

quietone CreditAttribution: quietone at PreviousNext commented
Project: Drupal core » SimpleTest
Version: 9.3.x-dev » 8.x-3.x-dev
Component: simpletest.module » Code

The simpletest module is obsolete in core. Moving to SimpleTest project.

Berdir’s picture

Berdir
Primary language German
Location Switzerland
CreditAttribution: Berdir at MD Systems GmbH commented

I'd suggest just closing it (and keep in core maybe for historic reasons?). Conceptually, phpunit browser tests work quite the same, but the issue is very old and nothing actionable will come from it anymore I assume.

quietone’s picture

quietone CreditAttribution: quietone at PreviousNext commented
Project: SimpleTest » Drupal core
Version: 8.x-3.x-dev » 9.3.x-dev
Component: Code » simpletest.module
Status: Needs work » Closed (outdated)

Following the advice in #33 and returning to core and closing as outdated.