WimLeers brought up a good point, that we may be disguising progress by not showing assertion counts.

[04:46:27] Mixologic: oh btw
[04:46:39] Mixologic: I know the "assertions count" vs "test count" thing was discussed a long time ago somewhere
[04:46:47] Mixologic: But thinking about it more, that's actually a regression
[04:46:56] WimLeers: really?
[04:47:09] Mixologic: Because the same number of test failures can hide an increase or decrease in number of failed assertions
[04:47:20] Mixologic: So either not showing progress from patch N to N+1
[04:47:24] Mixologic: Or not showing regressions
[04:48:20] Mixologic: e.g. https://www.drupal.org/node/2476947#comment-10322637 (#133) + #134 (the next)
[04:48:23] https://www.drupal.org/node/2476947 => Convert "title" page element into a block #2476947: Convert "title" page element into a block => 168 comments, 33 IRC mentions
[04:48:31] Mixologic: 10 vs 7 fails in PIFR, 2 vs 2 fails in DrupalCI
[04:48:54] It was definite progress, but DrupalCI's reporting masked that
[04:51:32] WimLeers: Hmm. I see what you're saying there, however I was also seeing that the assertion count isnt actually a fixed number. It appears to have some random variability in it.
[04:51:52] Mixologic: Very rarely, TBH
[04:52:21] Mixologic: The variability is only to the extent that if one assertion fails very badly and triggers a fatal because of the assertion logic, that the rest of the test doesn't run.
[04:52:29] WimLeers: Im pretty sure its every time, but its probably isolated to a few particular tests that do something random... so the "grand total" is random, but not the error to errors.
[04:52:53] ohhhh right
[04:52:59] Mixologic: the total number of assertions seems random
[04:53:04] Mixologic: that's what you mean, right?
[04:53:08] Yes,
[04:53:09] Mixologic: yes, that's definitely true
[04:53:14] Mixologic: But *failed* assertions are not
[04:53:36] Mixologic: So, it'd be great if DrupalCI could say "13K tests, 5 failed, with 53 failed assertions"
[04:53:45] Mixologic: i.e. it can continue to use "tests" as the main unit
[04:53:54] Mixologic: but if it could also tell us the number of failed assertions, that'd be super helpful
[04:54:08] We're still keeping counts of those assertions, and I've got that data in the xml, I just need to figure out a way to expose it to jenkins..

This will be postponed until: #2568137: Expose Assertion Count to Jenkins is done.

Comments

Mixologic created an issue. See original summary.

Mixologic’s picture

Issue summary: View changes
Mixologic’s picture

Issue summary: View changes