Justin Randell noticed this evening that the Menu item creation/deletion tests had a failure in them. So I spent a few minutes rolling back patches until it stopped failing. It turns out, this patch causes it.

The failure was:

"Menu item was reset" in menu.test line 317 (MenuTestCase->resetMenuItem())

Testing bot, however, did not catch this. :( I marked the patch back for retesting at :30, it passed at :57, and was committed at :18. The only thing committed in this ~20 minute window was http://drupal.org/node/67349#comment-1122071 which is a one-line change to SQLite, nowhere near the menu system.

We need to get to the bottom of this, because if we can't trust testing bot to come back with accurate results, it's not going to actually save core maintainers any time.

CommentFileSizeAuthor
#12 pifr_regex_singular.patch874 bytesboombatower

Comments

damien tournoud’s picture

damien tournoud commented

Ok, my findings so far is that something is weird in the testing kingdom:

(1) the very last test case is never reported (User blocks), because of a bug in the regexp in pifr_review_format_results() --> strike that: I was wrong
(2) there are two different results for "Comment on a node page",
(3) "Menu item Creation / Deletion" is nowhere to be found

dave reid’s picture

dave reid
he/him
Primary language English
Location Nebraska USA
commented

Subscribing as it was my issue that somehow got a false ok from testing bot... :)

webchick’s picture

webchick
she/they
Primary language English
Location Vancouver 🇨🇦
commented

Here's another one it wasn't catching: http://drupal.org/node/338239#comment-1127512. This is not good. :(

dave reid’s picture

dave reid
he/him
Primary language English
Location Nebraska USA
commented

Seems like the test failures are not reported in the testing results, so it looks like they all passed. The user/login patch didn't show the result for the menu item creation/update test.

dave reid’s picture

dave reid
he/him
Primary language English
Location Nebraska USA
commented

Yet another one: http://drupal.org/node/340052#comment-1131402 Gives the green light with only 105 passes. I'm inclined to say the whole test suite did not run. ;)

boombatower’s picture

boombatower commented

Queried the results and found that the following 196 patches seem to be not reporting/running all the tests. All of which were run on slave 8 except: simpletest_tune.patch, 303660_HEAD_input_format_8.patch. Not sure if there is an issue with the slave or PIFR. The fact that slave 8 is reporting them all is almost uncanny.

Exact query for clarification:

SELECT filename FROM pifr_file WHERE pass + fail + exception < 6000 AND message = '' AND status > 3;

+-----------------------------------------------------------+
| javascript_unit_testing.patch                             |
| uploadjs.diff                                             |
| error_class_with_empty_message.patch                      |
| headers_0.patch                                           |
| theme_col.patch                                           |
| menu_translate.patch                                      |
| welcome_intro_change_0.patch                              |
| drupal_cron_cleanup.patch                                 |
| node_types_rebuild.patch                                  |
| node_revision_text.patch                                  |
| install_default_settings_md5.patch                        |
| issue-doxygen-install.patch                               |
| custom_date_00.patch                                      |
| system_perm_move.patch                                    |
| dirindex.patch                                            |
| taxonomy_autocomplete_1.patch                             |
| drupal.run-tests-php-windows.patch                        |
| simpletest_help.patch                                     |
| node_access_10.patch                                      |
| remove-ie-mac-css.patch                                   |
| cron-timing-4.patch                                       |
| language_0.patch                                          |
| cron-timing-131536-26.patch                               |
| javascript_unit_testing_0.patch                           |
| cron-timing-131536.patch                                  |
| language_244162.patch                                     |
| cron-timing-131536-34.patch                               |
| 261284.patch                                              |
| locale_270226.patch                                       |
| 261284_0.patch                                            |
| jquery_fix_D7.patch                                       |
| update_block_cache_mode-235673-11.patch                   |
| locale_32.patch                                           |
| list_0.patch                                              |
| welcome_4_0.patch                                         |
| 115540_1.patch                                            |
| drupal_form_multistep_troubles.patch                      |
| install-notify-update.patch                               |
| 221712-browser-language_0.patch                           |
| 183678.patch                                              |
| trigger-issue-241013_7.x.patch                            |
| issue-theme_comment.patch                                 |
| issue-theme_comment_0.patch                               |
| cgi-headers-175855-D7.patch                               |
| content-length-header-297141-D7.patch                     |
| actions-synchronize-319404-D7_0.patch                     |
| custom_date_01.patch                                      |
| issue-doxygen-install-2.patch                             |
| issue-297141.patch                                        |
| issue-297141_0.patch                                      |
| issue-297141_1.patch                                      |
| tngdb-pdo_pgsql-blob-1223989019.patch                     |
| disabled_checkbox_tests.patch                             |
| issue-261284-15.patch                                     |
| update.sh_.patch                                          |
| mw_212.patch                                              |
| 219584_0.patch                                            |
| drupal-menu-submit-7.x_0.patch                            |
| admin_content_node_11.patch                               |
| table_select_empty.patch                                  |
| drupal.update-themes_0.patch                              |
| show-tables.patch                                         |
| access-callback-harden-D7-319360-7.patch                  |
| menu.module_7.x.mlid_.patch                               |
| drupal.run-tests-php-windows_0.patch                      |
| file_276280.patch                                         |
| simpletest_tune.patch                                     |
| mime_extension_mapping.d7_0.patch                         |
| batch_with_timer-1226226432.patch                         |
| drupal.status-report-export_2.patch                       |
| mime_extension_mapping.d7_1.patch                         |
| 332725-rip-locale-inc-callback_0.patch                    |
| node_types_rebuild_0.patch                                |
| preprocess-rename-file-to-template_file.d7_1.patch        |
| time_limit_174617_7.x_0.patch                             |
| search_36.patch                                           |
| mail.test-299138-13.patch                                 |
| drupal_default_country_0.patch                            |
| js_filter_xss_3.patch                                     |
| dbtng-getFieldTypeMap-1226499767.patch                    |
| 212130_D7_0.patch                                         |
| settings_d6_d7_3.patch                                    |
| dbtng-db_select_temporary-1226655467.patch                |
| issue-57178-9.patch                                       |
| issue-173037-8.patch                                      |
| poll_choice_js-331708.patch                               |
| 247071_show_attachments_in_teaser_3.patch                 |
| system-action-watchdog-202032-4.patch                     |
| 247071_show_attachments_in_teaser_4.patch                 |
| 247071_show_attachments_in_teaser_5.patch                 |
| 220559-language-switcher-with-test.patch                  |
| 165786_aggregator_rename_description_column_to_body.patch |
| d6-body-class-array-04.patch                              |
| comment_approval_4.patch                                  |
| drupal_js-302251.patch                                    |
| cache_get_multiple_1_0.patch                              |
| 335185-remove-no-js-css-D7.patch                          |
| 335185-remove-no-js-css-D7_0.patch                        |
| dirindex_0.patch                                          |
| batch_with_timer-1226896443.patch                         |
| 335237-modules-uninstalled-filter-D7.patch                |
| 320303-modules-uninstalled-menu-D7.patch                  |
| parent_theme_pattern_clobbering.patch                     |
| multiple_load_18.patch                                    |
| 303660_HEAD_input_format_8.patch                          |
| 335035-disabled-controls_0.patch                          |
| preprocess-rename-file-to-template_file-D7.patch          |
| custom_url_rewrite_outbound-2.patch                       |
| issue-334950-6.patch                                      |
| password_checker_fix_3.patch                              |
| password_checker_fix_4.patch                              |
| aggregator_pluggable_5.patch                              |
| 290246_edit_own_comments_permission_0.patch               |
| 290246_edit_own_comments_permission.patch                 |
| multiple_load_21.patch                                    |
| t-docs.patch                                              |
| parent_theme_pattern_clobbering.b-head.patch              |
| 247071_show_attachments_in_teaser_6.patch                 |
| drupal.status-report-export_3.patch                       |
| teaser_undefined_value.patch                              |
| multiple_load2.patch                                      |
| multiple_load_19.patch                                    |
| comment_approval_5.patch                                  |
| dbtng-aggregator-1227258722.patch                         |
| cache_get_multiple_1_0_0.patch                            |
| taxonomy_count_nodes_HEAD_0.patch                         |
| taxonomy_count_nodes.patch                                |
| taxonomy_term_count_nodes.patch                           |
| drupal_nodeapi_signatures.patch                           |
| thai-vowel-7.x-dev.patch                                  |
| 290246_edit_own_comments_permission_1.patch               |
| 27234_help_text_picture_size.patch                        |
| 183678-3.patch                                            |
| 183678-4.patch                                            |
| drupal_310863_auto_locale.patch                           |
| drupal_310863_auto_locale_0.patch                         |
| book.revision-log.patch                                   |
| simpletest_help_2.patch                                   |
| file_30520_7.patch                                        |
| 222855.patch                                              |
| hook_url_rewrite_1.patch                                  |
| file_276280_0.patch                                       |
| 325827-date-default-timezone-set-D7.patch                 |
| subselect_from_1.patch                                    |
| menu-access-callback-load-337907.patch                    |
| dbtng-PDO_CASE_LOWER-1227407521.patch                     |
| 335185-remove-no-js-css-D7_2.patch                        |
| parent_theme_pattern_clobbering.c-head.patch              |
| HEAD_parent_theme_pattern_clobbering.c.patch              |
| parent_theme_pattern_clobbering-c-HEAD.patch              |
| theme-registry-fix-d7.patch                               |
| parent_theme_pattern_clobbering-e-HEAD.patch              |
| recent_content_block_02.patch                             |
| grant-op-278675-1.patch                                   |
| grant-op-278675-6x-2.patch                                |
| node_access_11.patch                                      |
| node_278675.patch                                         |
| multiple_load_28.patch                                    |
| test-menu-callback-registry-337907-4.patch                |
| d7-strict_1.patch                                         |
| multiple_load_25.patch                                    |
| taxonomy_term_count_nodes_1.patch                         |
| assertFieldByXPath-3.patch                                |
| file_create_url-16.patch                                  |
| dbtng-block-1227763112.patch                              |
| blob-locales_source_hash-1227760668_0.patch               |
| blob-menu_router-arguments-1227761227.patch               |
| multiple_load_2_0.patch                                   |
| file.inc__11.patch                                        |
| mw_221.patch                                              |
| drupal_goto-parse_url_1.patch                             |
| t-docs_0.patch                                            |
| 183678-7.patch                                            |
| aggregator_pluggable_7_1.patch                            |
| mw_224.patch                                              |
| drupal_openid-prociders_user_friendly_link_try2.diff      |
| 337926-followup.patch                                     |
| dbtng-menu.inc-1227961848.patch                           |
| 337926-followup_1.patch                                   |
| 296624-fix-do-search_0.patch                              |
| mw_225.patch                                              |
| 340568-fix-distinct-field-sqlite.patch                    |
| anon-sessions-5.patch                                     |
| 340557-drupal-is-frontpage-D7.patch                       |
| hook_help.patch                                           |
| cache_registry_parse_1.patch                              |
| localize.actions.settings_0.patch                         |
| taxonomy_term_count_nodes_3.patch                         |
| page_cache_0.patch                                        |
| 235673-block-caching-fix-D7.patch                         |
| 295983.patch                                              |
| streamwrapper-D7-21.patch                                 |
| transactions_2.patch                                      |
| drupal_web_test_case.php-330582-48.patch                  |
| drupal_web_test_case.php-330582-49.patch                  |
| aggregator_pluggable_9.patch                              |
+-----------------------------------------------------------+
boombatower’s picture

boombatower commented
Status: Active » Needs review

Reset those patches:

UPDATE pifr_file SET status = 1, test_count = 0 WHERE pass + fail + exception < 6000 AND message = '' AND status > 3;
boombatower’s picture

boombatower commented
Assigned: Unassigned » boombatower
Status: Needs review » Fixed

Marking as fixed since no one has reported back any bad results in 7 days.

Feel free to re-open if found.

boombatower’s picture

boombatower commented

Seems to still be occurring.

http://drupal.org/node/345632#comment-1151029

boombatower’s picture

boombatower commented
Status: Fixed » Active
damien tournoud’s picture

damien tournoud commented 
Node preview 18 passes, 1 fail, and 0 exceptions

The issue is that "fail" is singular here, and the regexp can't catch that.

boombatower’s picture

boombatower commented
Status: Active » Needs review
StatusFileSize
new pifr_regex_singular.patch874 bytes

Waiting for confirmation from testing.

boombatower’s picture

boombatower commented
Title: Testing Bot not finding some test failures :( » Correct regex for singular assertions
boombatower’s picture

boombatower commented
Status: Needs review » Fixed

Committed after confirmation that it works.

We should be able to trust the results!

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for two weeks with no activity.