Support for Drupal 7 is ending on 5 January 2025—it’s time to migrate to Drupal 10! Learn about the many benefits of Drupal 10 and find migration tools in our resource center.When running tests from the command line, I currently have two options:
a) I can pass --verbose, in which case I get a really long list of mostly irrelevant passing tests that I have to scroll back through to check if anything failed; or
b) I can not pass --verbose, in which case if anything fails I need to re-run the test to find out what it was.
The vast majority of the time when I'm running a test I just want to know that everything passed, or if not then what I need to do to fix it. I propose listing all failed tests and exceptions by default, and adding a --summary-only option to support the current behaviour.
I wonder if this will confuse the d.o test infrastructure?
| Comment | File | Size | Author |
|---|---|---|---|
| #22 | 2200747-nr-bot.txt | 160 bytes | needs-review-queue-bot |
| #1 | drupal-2200747-run-tests-show-errors-1.patch | 2.46 KB | ianthomas_uk |
| #1 | drupal-2200747-run-tests-show-errors-1-will-fail.patch | 3.13 KB | ianthomas_uk |
Comments
Comment #1
ianthomas_ukReroll, plus a version with a deliberate mistake to show this doesn't break d.o's test infrastructure (it shouldn't, as it doesn't change the behaviour of the --verbose flag)
Comment #2
sunI've read the issue summary at least twice in the meantime, but I still do not understand what the problem and goal is... :-/ Can you clarify the summary?
Comment #4
ianthomas_ukBasically I don't find the current default output useful if a test fails, but the verbose output can get very long so isn't easy to read and doesn't give you any extra useful information if all tests pass.
This is an attempt to find a compromise between the two. It's still easy to read, but if a test fails it will tell you what (which is often enough, e.g. if it's a failure you've seen before, or is a test you were working on).
Comment #5
sunNote that we cannot change the default output of run-tests.sh, because that would affect the regular Review log of PIFR and testbots, too:
https://qa.drupal.org/pifr/test/748038#footer (expand the "Review log" fieldset)
The review log is big enough already now — it has to be transferred between the testbot and qa.drupal.org (in addition to the assertion result rows).
In other words, this will need a new flag/argument to run-tests.sh.
Aside from that, we can happily rethink the CLI --verbose output. This flag/option is not used by PIFR/testbots.
We might want to take inspiration from PHPUnit's output in case of a test failure. Our current ASCII table output is certainly very limiting.
That said, given how our (web) tests are designed, it is often helpful to see some level of context before a failing assertion (e.g., 3-6 pass/info messages), so as to better understand what happened before a failing assertion.
Comment #6
ianthomas_ukAh, ok. I thought the testbots were already using --verbose, because they get the assertion-level detail. If they are getting that by other means and log size is an issue then do we even want to be logging the test summary, which runs to several pages?
I'd say the default behaviour should be the one most useful for humans. If test bot wants a different output, then it should be fairly easy to get testbot to pass the extra flag(s).
I didn't want to reduce the verbose output in case the full log is sometimes useful, as there would be no way of getting that. Yes, a few messages before the failure are often useful, but I didn't want to make the default output too noisy. I'm not particularly familiar with PHPUnit's output, so I'm not sure what we'd be getting inspiration from.
Comment #7
cilefen CreditAttribution: cilefen commentedWhat about
--verbose(would be verbose on failures only) and in addition--debug, which would be verbose also on passing tests?Comment #8
texas-bronius CreditAttribution: texas-bronius commentedI have been wanting this for some time, too: Running tests from command line in TDD, it's useful to see "All green!" or any exceptions and errors, but it is burdensome to see a sea of green and have to scroll back to see *whether* anything failed. Maybe if modifying --verbose is too risky, could we consider introducing a flag that means "--errors-and-exceptions" so like "--only-issues" or "--show-issues" might work?
Another idea which I think would also work is a way, from command line, to go back and list previous test outcomes. I guess this would involve storage somewhere: run tests as non-verbose, see a failure, then issue a followup command "show me the last test results details" which would then list either only failures or all, as if it were just run but with --verbose.
-Bronius
Comment #15
texas-bronius CreditAttribution: texas-bronius commentedA related tip that helps me: In mac Terminal (and there must be an equiv in Linux?), after a test is run, you can hit Command+UP_ARROW to jump to the previous prompt in scrollback which, when hit just once, takes you right to where you issued the `run-tests.sh` command and can therefore see the time it took, summary line with Successes and Fails, etc. There's no need to "return" to the original, latest prompt, bc your cursor is really still there-- just hit UP_ARROW to repeat the previous command and run the test again (or whatever your next thing is).
-Bronius
Comment #22
needs-review-queue-bot CreditAttribution: needs-review-queue-bot as a volunteer commentedThe Needs Review Queue Bot tested this issue. It either no longer applies to Drupal core, or fails the Drupal core commit checks. Therefore, this issue status is now "Needs work".
Apart from a re-roll or rebase, this issue may need more work to address feedback in the issue or MR comments. To progress an issue, incorporate this feedback as part of the process of updating the issue. This helps other contributors to know what is outstanding.
Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.