The way SimpleTest is currently designed, it only logs the results of assertions. There is no information about the overall result of a test case run (i.e. run() method invocation), apart from or the number of passes, failures and exceptions. Information about the results of individual tests (i.e. test*() method invocations) is also not available. It would be nice if this information was either saved in a database table, or exposed via a DrupalWebTestCaseCore member variable/property.

This would make the SimpleTest very useful in larger projects that use continuous integration and rely on automated builds and [batch] testing.

There are two possible approaches:

1. Database-backed

Store the value in a new database table (e.g. simpletest_test_results), with the following columns:

CREATE TABLE simpletest_test_run (
  test_run_id int(11) NOT NULL auto_increment,
  test_id int(11) NOT NULL COMMENT 'References simple_test_id',
  test_class varchar(255) NOT NULL COMMENT 'Test case name',
  test_method varchar(255) NOT NULL COMMENT 'Name of the test executed',
  timestamp int(11) NOT NULL COMMENT 'Start time',
  time int(11) NOT NULL COMMENT 'Test duration',
  status varchar(9) NOT NULL COMMENT 'One of: pass, fail, exception',
  PRIMARY KEY  (test_run_id)
)

Add a test_run_id FK column to the simpletest table, and drop the test_id column.

2. Class member-based

Add the following member variable to the DrupalWebTestCore class:

public $tests = array();

Array keys are the names of test methods and elements are instances of a StdClass, with the following member variables:

$test_method: string;
$timestamp: int;
$time: int;
$status: string // One of: 'pass', 'fail', 'exception'

The solution may also use a combination of the two proposed approaches.

If the maintainers agree to include the feature in 6.x, I can submit a patch.

Comments

boombatower’s picture

boombatower commented
Project: SimpleTest » Drupal core
Version: 6.x-2.7 » 7.x-dev
Component: Code » simpletest.module

This needs to be done in D7 core and backported.

I'm not sure exactly what you looking to accomplish with this, could you explain a bit more or provide an example.

velimir_alic’s picture

velimir_alic commented

The idea is to be able to incorporate a SimpleTest run as part of an automated build. I'm trying to use the CruiseControl for continuous integration. CruiseControl relies on Apache Ant to perform a build - in our case this means to run the tests, copy some files, initialize the database etc. Of course, both CruiseControl and Ant are Java-based and work well with Java-based tools, such as NUnit.

I don't want to write Ant/CruiseControl plugins in Java, so I've set out to create a script that can be called from Ant (based on the run-tests.sh script that comes with SimpleTest) and produces XML output using the same schema as the NUnit Ant task (see org.apache.tools.ant.taskdefs.optional.junit.XMLJUnitResultFormatter.java and org.apache.tools.ant.taskdefs.optional.junit.XMLConstants.java source). Such output can be processed by the CruiseControl and used to determine the status of a build.

An example of NUnit XML output is shown below:

 <testsuite errors="0" failures="1" hostname="xxx" name="org.osce.blah.tests.BlahTest" tests="2" time="0.078" timestamp="2009-03-31T16:13:54">
   <properties>
     ...
   </properties>
   <testcase classname="org.osce.blah.tests.BlahTest" name="testSomething" time="0.015">
     <failure message="expected:&lt;2&gt; but was:&lt;3&gt;" type="junit.framework.AssertionFailedError">junit.framework.AssertionFailedError: expected:&lt;2&gt; but was:&lt;3&gt;
        at org.osce.blah.tests.BlahTest.testSomething(BlahTest.java:8)
        at org.eclipse.ant.internal.ui.antsupport.EclipseDefaultExecutor.executeTargets(EclipseDefaultExecutor.java:32)
        at org.eclipse.ant.internal.ui.antsupport.InternalAntRunner.run(InternalAntRunner.java:423)
        at org.eclipse.ant.internal.ui.antsupport.InternalAntRunner.main(InternalAntRunner.java:137)
    </failure>
   </testcase>
   <testcase classname="org.osce.blah.tests.BlahTest" name="testSomethingElse" time="0.0" />
   <system-out><![CDATA[]]></system-out>
   <system-err><![CDATA[]]></system-err>
 </testsuite>

Apart from the conceptual difference between the NUnit and SimpleTest - SimpleTest tests continue to run after an assertion failure, while NUnit tests fail, the difference is in the level of detail provided by the SimpleTest output. SimpleTest outputs a simple "flat" list of assertion results, without any information about the actual test method that generated them. My solution would track the status of each test method, as well as its start time and execution time.

The rules for determining the overall status of a test [method] is as follows:

1. A test is in error if there is at least one error/exception during its execution.
2. A test fails if at least one assertion fails, and there are no errors/exceptions.
3. A test is successful if all assertions succeed, and there are no errors/exceptions.

joshk’s picture

joshk commented

I'm definitely in favor of picking up and using one of the many existing XML specs for test results/runs. It makes no sense to re-invent the wheel here, and we can more easily integrate with existing platforms that way, which is a major plus.

I've been working w/netaustin and others making a CI environment that uses Hudson to manage builds. He built some additions to run-tests.sh to output jUnit compatible XML so we can get data back into Huston. It could be tuned up a bit (e.g. to use SimpleXML rather than building the XML directly) but generally works well.

If we pick any decent spec that we think is best for Drupal, do the whold scheme and XML output piece right, it should be trivial for anyone to create an XML translator to take our test output into their preferred build system.

boombatower’s picture

boombatower commented

As note this should be done, after #475464: SimpleTest: abstract reporter and generate XML reports.

stephencamilo’s picture

stephencamilo commented
Status: Active » Closed (won't fix)
hestenet’s picture

hestenet
He/Him
Location Portland, OR 🇺🇸
commented
Status: Closed (won't fix) » Active

Reset issue status.

Status: Active » Closed (outdated)

Automatically closed because Drupal 7 security and bugfix support has ended as of 5 January 2025. If the issue verifiably applies to later versions, please reopen with details and update the version.