Closed (fixed)
Project:
GitlabCI Testbed for Drupal core
Component:
Code
Priority:
Normal
Category:
Task
Assigned:
Unassigned
Reporter:
Created:
20 Jul 2023 at 10:41 UTC
Updated:
18 Sep 2023 at 10:24 UTC
Jump to comment: Most recent
Problem/Motivation
Currently, DrupalCI is still way faster than GitlabCI for core.
GitlabCI is on average almost 3 times slower per job.
See some results:
GitLabCI
120s // ⚡️ PHPUnit Unit
1920s // ⚙️️ PHPUnit Kernel
210s // 👷️️️ PHPUnit Build
5220s // 🌐️️ PHPUnit Functional
720s // 🖱️️️ PHPUnit Functional Javascript
440s // 🦉️️️ Nightwatch
Total: 144m
----
DrupalCI
64s // ⚡️ PHPUnit Unit
310s // ⚙️️ PHPUnit Kernel
150s // 👷️️️ PHPUnit Build
1212s // 🌐️️ PHPUnit Functional
390s // 🖱️️️ PHPUnit Functional Javascript
320s // 🦉️️️ Nightwatch
Total: 41min
----
Difference
Job GitLabCI to DrupalCI ratio
⚡️ PHPUnit Unit 1.8 slower
⚙️️ PHPUnit Kernel 6.2 slower
👷️️️ PHPUnit Build 1.4 slower
🌐️️ PHPUnit Functional 4.3 slower
🖱️️️ PHPUnit Functional Javascript 1.8 slower
🦉️️️ Nightwatch 1.3 slower
Average: 2.8 slower
Steps to reproduce
See a test run from a core issue (DrupalCI) vs a run from this project (GitlabCI).
Proposed resolution
Investigate and see what optimizations can be done.
Remaining tasks
User interface changes
API changes
Data model changes
Issue fork gitlab_ci_testbed_for_drupal_core-3375826
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
- 3375826-test-branch
changes, plain diff MR !6
Comments
Comment #2
fjgarlin commentedComment #3
fjgarlin commentedComment #5
fjgarlin commentedI tweaked and triggered many local tests runs and none of the changes made a real impact.
I also triggered a few more tests runs via Gitlab (https://git.drupalcode.org/project/gitlab_ci_testbed_for_drupal_core/-/m...) and the numbers don't change that much from the initially reported in the IS.
I checked caching, artifacts generation, order of operations, and dependencies... I'm happy for other people to try or suggest anything else.
There are some settings that cannot be tried via ".gitlab-ci.yml", like using tmpfs (ramdisk - https://docs.gitlab.com/runner/executors/docker.html#mount-a-directory-i...) that I'd like to try next, but this requires changes to the Gitlab instance.
Comment #6
fjgarlin commentedIteration from https://www.drupal.org/project/gitlab_ci_testbed_for_drupal_core/issues/...
Suggestion:
Comment #7
wim leerstmpfs/ramdisk is crucial for Drupal test performance. I learned this independently for a CI pipeline at my employer — moving from a (fast SSD) file system to ramdisk was the single biggest improvement factor!
(Heck, even a super powerful MySQL server for tests was far slower than SQLite-on-RAMdisk!)
If you're worried about needing too much RAM, I suggest playing with only putting the
sites/simpletest*stuff on a RAMdisk. Past experience it's specifically the need to create and destroy so many DBs for testing Drupal (Kernel + Functional tests) that makes it provide such a big benefit.Comment #8
fjgarlin commentedThanks for validating the approach and for the tips Wim. This is definitely our next thing to try, and it's great to know that this was the biggest factor for improvement in your case as well.
Comment #9
bbralaI'd also do the code though, especially with the loads of cold caches. Although the files might end up in filesystem cache, so the impact might be minimal. The suggestion to minimize the paths by only doing db and the simpletest dir (writing resulst then being pretty free) is a good one.
Comment #10
wim leersThat, plus https://www.php.net/manual/en/intro.opcache.php, mean that filesystem performance
isSHOULD BE far less important for the code itself.Comment #11
bbralaSure, that also.
Git clone and unzip of packages might be slow but that is probably only a small part of the equation.
Comment #13
fjgarlin commentedThe changes suggested in #6 were merged in our Gitlab instance.
Pipeline: https://git.drupalcode.org/project/gitlab_ci_testbed_for_drupal_core/-/m...
* Before: 1h 34min
* After: 28 min
We are on numbers slightly better than DrupalCI after that change, so I think this issue can be closed.
Further optimizations can go into separate issues.
Comment #14
wim leersSLIGHTLY?!!!!!!!!!!
That's a factor of TWO!
TWO TIMES FASTER TESTS would be the biggest Drupal core contribution improvement in … a decade?! 🤩🚀
(The issue summary says core tests on DrupalCI take 41 minutes, but in my experience it always takes more: 47 mins, 48 mins, I can probably find >50 mins test runs too!)
Comment #15
bbralaWill this be standard config for all runners? Or is that a little scary to do? @drumm?
Comment #16
drummYes, it is standard configuration for all runners. We only have one configuration. At some point, we’ll split core & contrib runners, since we’re getting a more-accurate idea of resource usage. Contrib tests usually don’t use all of the over-provisioned instances, so effective speed for contrib should not go down much at all. And we do need to set up staging/testing, but this change was relatively safe, just having Kubernetes use memory instead of disk.
Comment #17
bbralaYeah you'll probably just end up with a few differently tagged runners. Thanks for the ping back:)