Get update.php running smoothly on 7.devdrupal.org

#1542666: Clean up duplicate files.filepaths blocks this. The temporary workaround is

echo -e 'SELECT concat("DELETE FROM files WHERE fid <> ", f.fid, " AND filepath = \047", f.filepath, "\047;") AS \047\047 FROM files f GROUP BY cast(f.filepath AS BINARY) HAVING count(DISTINCT f.fid) > 1;' | drush @7 sql-cli | drush @7 sql-cli

Getting ready to run this again. Confirmed our only hack is skipping password rehashing in user_update_7000(), this will be removed by #1235638: Increase password security for *.drupal.org using phpass.

I'll be applying #1549390: taxonomy_update_7005() can be faster to test that out.

This completed in 7 hr 44 min. The last comparable runs were 9 hr 57 min and 9 hr 12 min. Not as huge as I'd hoped, but I think we can say #1549390: taxonomy_update_7005() can be faster reduced the time by approximately 20%, with plenty of caveats, these updates have not been running consistently at all. There were some errors, so we can't double check that all the terms are in place and definitively update that issue yet.

The log is attached. There were 3 errors.

Executing user_update_7002 [22999.67 sec, 34.34 MB]                     [notice]
SQLSTATE[42S02]: Base table or view not found: 1146 Table                [error]
'drupal.filter_format' doesn't exist [22999.67 sec, 34.34 MB]

I'm not sure where this is coming from yet. This update function looks like it could not cause that error.

Executing versioncontrol_update_6317 [23000.62 sec, 38.96 MB]           [notice]
WD php: Warning: htmlspecialchars() expects parameter 1 to be string,  [warning]
array given in check_plain() (line 1572 of
/var/www/7.devdrupal.org/htdocs/includes/bootstrap.inc). [23000.62
sec, 38.96 MB]
Cannot add field <em class="placeholder"></em>.<em                       [error]
class="placeholder">versioncontrol_operations</em>: table doesn't
exist. [23000.62 sec, 38.96 MB]

This is trying to run a 6.x update on 7.x. #1568176: [meta] Get latest D6 versioncontrol code deployed will fix it.

Executing role_activity_update_7000 [23000.62 sec, 38.98 MB]            [notice]
SQLSTATE[42S22]: Column not found: 1054 Unknown column 'weight' in       [error]
'order clause' [23000.62 sec, 38.98 MB]

role_activity_update_7000() needs to be run after user_update_7007(), which adds the role.weight column.

The site itself isn't working since drupalorg_update_7003() doesn't fully enable views used on the home page. drupalorg actually has a fix for this in Git, but it hadn't made it into BZR.

Summarize blocker issues.

I updated the issue summary with blockers. Everything is tentatively resolved in some way except

Executing user_update_7002 [22999.67 sec, 34.34 MB]                     [notice]
SQLSTATE[42S02]: Base table or view not found: 1146 Table                [error]
'drupal.filter_format' doesn't exist [22999.67 sec, 34.34 MB]

More runs of user_update_7002() completed when running update.php again. Tracking down the problem will probably need a backtrace next time.

bfroehle - That's exactly what's happening. I filed #1586166: system_update_7013() and user_update_7002() should not call l() to get this in line with other update functions.

I also added drupalorg_update_dependencies() to make sure drupalorg_update_7003() runs after profile_update_7001(), since it ends up using the profile_field table.

#1586166: system_update_7013() and user_update_7002() should not call l() makes it look like a good change for 7. I don't know how to test it.

We are now getting a bit further. The one error is now

Executing system_update_7060 [20970.81 sec, 51.42 MB]                   [notice]
WD system: file module installed. [20970.81 sec, 51.42 MB]                [info]
WD system: file module enabled. [20970.81 sec, 51.42 MB]                  [info]
SQLSTATE[42S02]: Base table or view not found: 1146 Table                [error]
'drupal.profile_field' doesn't exist [20970.81 sec, 51.42 MB]


#0  DatabaseConnection->query called at [/var/www/7.devdrupal.org/htdocs/includes/database/database.inc:2317]
#1  db_query called at [/var/www/7.devdrupal.org/htdocs/modules/profile/profile.module:502]
#2  profile_user_categories called at [/var/www/7.devdrupal.org/htdocs/sites/all/modules/ctools/plugins/content_types/user_context/profile_fields.inc:3]
#3  require_once called at [/var/www/7.devdrupal.org/htdocs/sites/all/modules/ctools/includes/plugins.inc:482]
#4  ctools_plugin_load_includes called at [/var/www/7.devdrupal.org/htdocs/sites/all/modules/ctools/includes/plugins.inc:278]
#5  ctools_get_plugins called at [/var/www/7.devdrupal.org/htdocs/sites/all/modules/ctools/includes/content.inc:110]
#6  ctools_get_content_types called at [/var/www/7.devdrupal.org/htdocs/sites/all/modules/ctools/includes/content.theme.inc:15]
#7  ctools_content_theme called at [/var/www/7.devdrupal.org/htdocs/sites/all/modules/ctools/includes/utility.inc:28]
#8  ctools_passthrough called at [/var/www/7.devdrupal.org/htdocs/sites/all/modules/ctools/ctools.module:426]
#9  ctools_theme called at [/var/www/7.devdrupal.org/htdocs/includes/theme.inc:534]
#10 _theme_process_registry called at [/var/www/7.devdrupal.org/htdocs/includes/theme.inc:686]
#11 _theme_build_registry called at [/var/www/7.devdrupal.org/htdocs/includes/theme.maintenance.inc:91]
#12 _theme_load_offline_registry
#13 call_user_func_array called at [/var/www/7.devdrupal.org/htdocs/includes/theme.inc:276]
#14 theme_get_registry called at [/var/www/7.devdrupal.org/htdocs/includes/common.inc:2374]
#15 l called at [/var/www/7.devdrupal.org/htdocs/modules/field/field.module:379]
#16 field_system_info_alter called at [/var/www/7.devdrupal.org/htdocs/includes/module.inc:1022]
#17 drupal_alter called at [/var/www/7.devdrupal.org/htdocs/modules/system/system.module:2401]
#18 _system_rebuild_module_data called at [/var/www/7.devdrupal.org/htdocs/modules/system/system.module:2429]
#19 system_rebuild_module_data called at [/var/www/7.devdrupal.org/htdocs/modules/field/field.module:426]
#20 field_sync_field_status called at [/var/www/7.devdrupal.org/htdocs/modules/field/field.module:409]
#21 field_modules_enabled
#22 call_user_func_array called at [/var/www/7.devdrupal.org/htdocs/includes/module.inc:823]
#23 module_invoke_all called at [/var/www/7.devdrupal.org/htdocs/includes/module.inc:472]
#24 module_enable called at [/var/www/7.devdrupal.org/htdocs/modules/system/system.install:2636]
#25 system_update_7060 called at [/usr/local/share/drush/commands/core/drupal/update_7.inc:74]

This is an even-more-roundabout case of l() being called, starting the theme system, causing ctools to error out. I'm going to look at a few solutions:

• Disable ctools - This is a bit of a hassle since we have to script disabling and enabling it. I'm not sure if the current approach will work. We already disable apachesolr and a few others with a db query, UPDATE system SET status = 0 WHERE name IN ('apachesolr' …, but updates from those modules are still run. Drush doesn't work since we have the D7 codebase and a D6 database. We could
  • Swap in the D6 BZR checkout for this stage and use drush
  • Otherwise trick drush
  • More-thoroughly disable modules with more SQL
• Patch Drupal core to temporarily disable modules until all core updates are completed. I like this idea since it would make upgrading more straightforward for everyone. I've never actually done the "disable all contrib modules" step myself and it is a bunch of manual work for user. However, I don't think I can do this well & quickly myself.
• I heard from webchick that there is a module which automates disabling modules for updates. I think we looked for a couple minutes and couldn't find it. This probably needs a D6 codebase too.

I'm going to look at getting a D6 codebase involved so we can use drush pm-disable ctools.

(Trying to attach a compressed file, and re-adding tags)

I filed #1587822: Code style: link in field_system_info_alter(), which happens to be a stopgap solution for us. I fully expect to come back to #9.

I'll look into theme_link. I didn't know about that, maybe we don't need it for anything and can save some work for our servers all the time.

Note crashing modules

I went ahead and added $conf['theme_link'] = FALSE; to settings.php to disable it forever. We aren't using it and have link-heavy pages that we can keep fast.

I'm going to leave this issue for the day since we have two stopgaps in place.

This is completing successfully now. I did a bit of cleanup to make sure drupalorg updates work well:
- The drupalorg_user feature wasn't getting the new fields enabled quick enough for the update to populate them.
- The primary links block wasn't shown because blocks hadn't been rehashed at the time.

And I removed some debugging code.

Once this update completes, I'll schedule it to run nightly and we can either call this done, or leave it open to track dependencies from the issue summary.

Scheduled to run nightly at 7:00 Jenkins time, which should be midnight PDT.

Followup issue: #1597828: QA Drupal.org on D7