We use Jenkins to use to drush to run simpletest on our projects.
I was not entirely happy with the output of the testresults in Jenkins so I created a patch for test.drush.inc. This adds the --jenkins option to generate jUnit xml result files that are optimized for use with Jenkins.

For me it's a huge improvement in the usability of the testresults so I thought I'd share my patch.

notes:
- no longer counts every single line of simpletest output as a test
- change --xml=... to --jenkins=... to use the new option
- results are written to $dir/testsuite-test_id.xml

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

moshe weitzman’s picture

Status: Needs review » Needs work

We have --xml already. This is an improvement? please submit patches for drush5

JvE’s picture

Version: All-versions-4.x-dev »
FileSize
8.11 KB

It is a huge improvement in usefullness of the xml imho.

The biggest change is the way the results are structured.
The original --xml turns every line of simpletest output into a testcase, creating simpletest-like output.
Tools using jUnit xml are not really suited to simpletest-like output though.
Mine groups them by simpletest testcase, creating more jUnit-like output.

But since the jUnit xml format is not an official standard and many tools choose to parse them in different ways the exact format is always going to be a matter of personal taste.

This patch is optimized for use with Jenkins. I analyzed the Jenkins source code for the way it uses jUnit xml files and worked from there to create the optimum format for simpletest results.
I have not tested it in other jUnit "compatible" tools but I suspect the results will be better than the original --xml option too, ymmv.

moshe weitzman’s picture

Assigned: Unassigned » msonnabaum

Perhaps Mark can review this one.

msonnabaum’s picture

I agree this would be an improvement, but I'm not sure about the --jenkins option. Maybe this should be the default for --xml and we should provide an additional option that's more descriptive that does a test per assertion. I think it's safe to say this is preferable, but I'd rather not remove the ability to match the core test runner.

JvE’s picture

Indeed, that is why I added an option rather than rewriting the xml option. Wouldn't want to break existing use of the --xml option.

Btw, the --jenkins version is just as descriptive, every line of output is still shown, they're just grouped by testcase and fail/pass is determined by test and not per assertion.

If I can find an hour or two somewhere I'll see about making some before and after screenshots.

msonnabaum’s picture

I'm just saying I don't like the name "jenkins" for the option. Really, we're talking ab out jenkin's xUnit plugin, and even what works on hudson, not to mention the fact that bamboo and travis-ci probably consome xUnit XML as well.

If this is truly a better option (would *love* some screenshots), we should make it the default xml option in drush 5 with an additional option to use the current behavior.

JvE’s picture

Like I said, I used the name because I build the xml-writer based on Jenkins' jUnit parsing code and do not know how well the format will perform in other tools.

I have attached some screencaptures that I hope will demonstrate the difference.
I sabotaged my build to create some failures and ran the test on the "Node" testgroup.
The images numbered 1-6 are the result with a patched drush and the ones from a-f are the corresponding originals.

I've also attached the actual xml in the testresults-original.zip and testresults-new.zip for comparison.

teezee’s picture

Like

sdague’s picture

+1 would love to see this in drush 5

lucascaro’s picture

I've been using this patch for several months now and it's really useful. Thanks @JvE !

I'd like to see it merged so let me know how can I help (so far I consider this tested and working). I think replacing the xml output might be too radical and we should add it as a new option. If the name is a problem, maybe we couls use --xunit or even --junit as the option name?

thoughts?

cheers.

batje’s picture

Version: » 7.x-5.8
Status: Needs work » Reviewed & tested by the community

Works wonderfully, would be a great addition.

moshe weitzman’s picture

Status: Reviewed & tested by the community » Needs work

For drush6, lets commit this but change the improved jenkins output to be the default for --xml.

We could potentially go with the most recent patch for drush5, I suppose.

lucascaro’s picture

+1

greg.1.anderson’s picture

Regarding #12, I don't think that we particularly need to worry about backwards compatibility in the 'tests' folder if folks want this in Drush 5. It rather seems that it would be better to have compatibility of option behaviors for the tests at any point in time across all active versions, so that test harnesses could work uniformly if they test multiple versions of Drush.

I don't have any strong need or desire for this to be one way or the other; I'm just saying.

longwave’s picture

Version: 7.x-5.8 » 8.x-6.x-dev
Status: Needs work » Needs review
FileSize
7.69 KB

The changes in this issue drastically increase usability of the test results UI when running SimpleTests through Jenkins.

The attached patch for Drush 6 is based on #2 but removes the alternative output option, simply changing the existing XML output to the improved format.

moshe weitzman’s picture

Status: Needs review » Fixed

I committed this to 6. I think 5 has been out for so long that I'm reluctant to change it.

Perhaps we should have error handling around the final file_put_contents() but that can be a follow-up.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

bulat’s picture

Moshe, this would be an extremely useful extension for 5, really need this for D7.

lucascaro’s picture

@moshe weitzman for 5 we can keep it like the patch and use --jenkins so we get the needed functionality and don't break anything that is already working. I've been using it with --jenkins (with the patch from #2 for a long time now and it feels pretty robust :)

delta’s picture

Issue summary: View changes

drush5-jenkins-xml.patch in #2 works very well, the log are a lot more readable that way, thanks

The patch apply cleanly against drush 7.x-5.4