Posted by sun on August 4, 2010 at 11:24am
9 followers
| Project: | Project Issue File Review |
| Version: | 6.x-2.x-dev |
| Component: | Code |
| Category: | bug report |
| Priority: | major |
| Assigned: | Unassigned |
| Status: | needs review |
Issue Summary
Testing of patches on Mollom's DRUPAL-6--1 branch leads to very odd test results in almost 50% of all test requests currently:
PASSED: [[SimpleTest]]: [MySQL] 0 pass(es).(example: http://qa.drupal.org/pifr/test/76304, but also many others)
When this happens, the patch is reported as "passed", however, with the addition of "zero passes", you do not know whether tests were executed at all or not.
Upon sending the identical patch for re-testing, it most often comes back with proper testing results (i.e., non-zero passes). Interestingly, even the plain branch test reported zero passes once, but similar to patch test results, the branch test result corrected itself after the next branch self-test (i.e., after the next commit).
Note that I've also encountered such odd test results with Drupal core patches against D7 (HEAD).
Comments
#1
It just happened again (over in http://drupal.org/node/872646#comment-3287492), but this time, I know that the patch should not have passed.
Hence, it looks like the result message is technically correct. There are actually no tests executed.
As maintainers may not recognize the "0" but just the green test status, this is a major bug, at minimum.
#2
Is this the wrong project/queue?
#3
The frequency in which this problem occurs increased, http://drupal.org/node/801662#comment-3299862
Only one of those re-tests led to a proper test result. All others returned with Previous result: PASSED: [[SimpleTest]]: [MySQL] 0 pass(es).
#4
This time, getting
Previous result: FAILED: [[SimpleTest]]: [MySQL] 85 pass(es), 23 fail(s), and 0 exception(es).
from test client #33. We are missing ~1.500 assertions. Test results seem to indicate 1 unknown assertion that's interpreted as a pass:
Test name Pass Fail Exception
Access checking (MollomAccessTestCase) [Mollom] 1 0 0
Blacklisting (MollomBlacklistTestCase) [Mollom] 1 0 0
Comment form protection (MollomCommentFormTestCase) [Mollom] 1 0 0
Contact form protection (MollomContactFormTestCase) [Mollom] 1 0 0
Data processing (MollomDataTestCase) [Mollom] 1 0 0
Fallback behavior (MollomFallbackTestCase) [Mollom] 1 0 0
Form administration (MollomFormConfigurationTestCase) [Mollom] 1 0 0
ExpandInstallation and key handling (MollomInstallationTestCase) [Mollom] 72 23 0
Message Group Filename Line Function Status
Post installation warning found. Other mollom.test 708 MollomInstallationTestCase->testInstallationProcess()
...
"Mollom testing mode is still enabled." found Other mollom.test 787 MollomInstallationTestCase->testInstallationProcess()
Call to undefined function _mollom_status() PHP Fatal error mollom.install 26 Unknown
Language detection (MollomLanguageDetectionTestCase) [Mollom] 1 0 0
Reporting functionality (MollomReportTestCase) [Mollom] 1 0 0
Reseller functionality (MollomResellerTestCase) [Mollom] 1 0 0
Server list recovery (MollomServerListRecoveryTestCase) [Mollom] 1 0 0
Server responses (MollomResponseTestCase) [Mollom] 1 0 0
User registration and password protection (MollomUserFormsTestCase) [Mollom] 1 0 0
#5
This appears to be related to #102102: Parse project .info files: present module list and dependency information and the way PIFT instructs pifr to run tests..which is a know contrib testing bug...they will all be fixed together with .info parsing and updated in one shot.
#6
Yeah, this is also nailing me in the project queue, for example:
#69556-51: move "cvs access" tab into project, make it generic
The tests in that patch contain PHP errors if you run them locally. However, the bot said it ran no tests and that everything is happy and green. ;)
What's weird is that this behavior is intermittent. For example, the bot properly ran all the tests here:
#69556-31: move "cvs access" tab into project, make it generic
#69556-33: move "cvs access" tab into project, make it generic
...
#7
Is there anything I could do to help resolve this? It starts to get really annoying -- it almost looks like it happens for /any/ patch sent for a certain branch now. I.e., almost no issues with HEAD, but in 99% of all cases on DRUPAL-6--1
I'd go nuts if I wasn't able to invoke a re-test directly on http://qa.drupal.org/pifr/test/85109 - fortunately, I can, but even doing that directly takes hours for a single patch.
#8
#5 needs completion. All just renaming with like one function addition.
#9
http://qa.drupal.org/pifr/test/158364 (testing from #1069582-19: How to use an alternate stream wrapper when saving media files?) is also doing this.
In looking at the test log, I see:
Tests to be run:
Test run started: Monday, July 18, 2011 - 20:52
Test summary:
-------------
ERROR: No valid tests were specified.].
[13:52:19] Command [/usr/bin/php ./scripts/run-tests.sh --php /usr/bin/php --url 'http://drupaltestbot664-mysql/checkout' --list 2>&1] succeeded
Duration: 0 seconds
Directory: [/var/lib/drupaltestbot/sites/default/files/checkout]
Completion status: [0]
Output: [Available test groups & classes
No valid tests were specified??? Not sure what it means, but that is one possible bit of information that can cause this 0 passes thing. It seems like if no tests are specified, that should count as a fail?
#10
The main issue in #9 can be seen a few rows up:
[13:59:20] Command [/usr/bin/php ./scripts/run-tests.sh --concurrency 8 --php /usr/bin/php --url 'http://drupaltestbot664-mysql/checkout' --file 2>&1] succeededThe '--file' parameter expects an argument, containing the tests to run (defined by file) ... with no file argument, there is no test.
In trying to troubleshoot further, however, I keep getting the argument '--all' instead of '--file' ... so I can't explain 'why' the argument is missing (though from boombatower's comment in #5, it sounds dependency related).
#11
Marked #1156976: 0 tests passing is not green as a duplicate. There's also good information over there.
#12
Happens for core tests too: http://qa.drupal.org/pifr/test/183394 Usually I've seen it coupled with other failures. Like first its says it cannot checkout from git. Then you ask it to retest, then it gets back with this 0 passes... :|
#13
I saw this today with http://qa.drupal.org/pifr/test/183394 - what happened was that the test ruined the testbot (used up the available tmpfs space for the database). The test returned a "0 tests passed" and the testbot never checked again, as mysql had died.
So a probable case for at least some of these is mysql dies (mysql server has gone away) and PIFR misinterprets something about that. We *should* be able to simulate this by just stopping mysqld.
Moving this to the PIFR queue, as this example is certainly PIFR.
Edit: I see this is the exact same test Gabor reported. Hmm. #1260586-12: Consolidate .po file import to one directory. Looks to me like if we just cease to test patches that Gabor posts, we'll be OK. I will try to see if every fill-up of /tmpfs is associated with a particular test.
#14
I see that http://qa.drupal.org/pifr/test/183854, which is #1260586: Consolidate .po file import to one directory did this again. Seems to have broken all the testbots?
#15
So the biggest challenge to this issue is that the ability to determine whether '0 tests passed' is a legitimate response lies within PIFR, while the actual 'No tests found' error message is coming from run-tests.sh. As far as I know, we treat any non-zero response from run-tests.sh as a failure.
The approach in the patch at #1233248: Add check for empty --file argument before running run-tests.sh during simpletest reviews (which should probably be marked as a dupe) was to determine whether there are any actual tests to be run, and if so, execute run-tests.sh ... if not, return the 'no tests found' failure message without actually executing run-tests.sh. The only problem was with the assumption that no tests was a failure ... what we may need to do is have this scenario skip run-tests.sh but still return a 'pass' result, assuming the previous stages had all passed successfully. Alternatively, we can set a flag when the 'no test files found' condition is detected, execute run-tests.sh, and then analyze the run-tests.sh output; overriding the 'no tests found' failure condition and message if no tests were actually expected.
#16
New patch posted to #1233248: Add check for empty --file argument before running run-tests.sh during simpletest reviews which should finally solve this.
#17
Patch referred to in #16 has been committed to 6.x-2.x-dev ... will postpone this for now, and revisit after the next release is deployed to the production testbots.
#18
Seems to be resolved.
#19
Automatically closed -- issue fixed for 2 weeks with no activity.
#20
Could use another revisit ... the file scan isn't the only scenario which can cause this.
#21
Can do a sorta "final last resort protection" ?
Along the lines of this?
if ($test->passes == 0) {$test->success = FALSE;
}
#22
Untested, but this should help. Will validate once I find a test with consistent FATALs.
#23
HEAD just passed with a PDO exception, so adding that check too.
#24
Bah ... Typos. :(
#25