Cannot upgrade from Drupal 6 to Drupal 7 - meta issue

I have the same issue with a virgin installation of drupal-7.x-dev.tar.gz dated 8/29/2009

confirm error on wamp new install drupal-7.x-dev.tar.gz dated 8/29/2009

shows:
* Warning: filectime(): stat failed for modules/search/search.extender.inc in _registry_parse_files() (line 127 of C:\wamp\www\drupal-7.x-dev\includes\registry.inc).
* Warning: filemtime(): stat failed for modules/search/search.extender.inc in _registry_parse_files() (line 128 of C:\wamp\www\drupal-7.x-dev\includes\registry.inc).
* Warning: file_get_contents(modules/search/search.extender.inc): failed to open stream: No such file or directory in _registry_parse_files() (line 132 of C:\wamp\www\drupal-7.x-dev\includes\registry.inc).

see this cvs message by Dries: http://drupal.org/cvs?file=/modules/search/search.extender.inc

add missing file, fixed

I'm sorry, I should have left it as it was. live and learn.

Methinks this is the very definition of 'critical'. ;)

Hmm.. Not with an upgrade, but ran into this issue trying to generate content with the Devel module and wondered if it might be related -- /batch?op=start&id=7

This may not be close enough.. I do yearn for the good old days of simple SQL errors rather than this PDO stuff which makes no sense!

An error occurred.
Path: /batch?id=7&op=do
Message: PDOException: SQLSTATE[HY000]: General error: 1366 Incorrect integer value: 'FILTER_FORMAT_DEFAULT' for column 'format' at row 1: INSERT INTO {comment} (nid, pid, uid, subject, comment, format, hostname, timestamp, status, thread, name, mail, homepage) VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1, :db_insert_placeholder_2, :db_insert_placeholder_3, :db_insert_placeholder_4, :db_insert_placeholder_5, :db_insert_placeholder_6, :db_insert_placeholder_7, :db_insert_placeholder_8, :db_insert_placeholder_9, :db_insert_placeholder_10, :db_insert_placeholder_11, :db_insert_placeholder_12); Array
(
[:db_insert_placeholder_0] => 6
[:db_insert_placeholder_1] => 0
[:db_insert_placeholder_2] => 0
[:db_insert_placeholder_3] => Verto Interdico Nostrud Jus Dignissim Lobortis Consectetuer
[:db_insert_placeholder_4] => Sagaciter accumsan valde ad. Bene cogo validus. Premo verto vereor at pecus occuro lenis haero roto. Feugiat vulpes ymo humo. Ea laoreet genitus lenis nulla causa aptent pecus.


[:db_insert_placeholder_5] => FILTER_FORMAT_DEFAULT
[:db_insert_placeholder_6] => 74.12.172.225
[:db_insert_placeholder_7] => 1254044787
[:db_insert_placeholder_8] => 1
[:db_insert_placeholder_9] => 01/
[:db_insert_placeholder_10] => devel generate
[:db_insert_placeholder_11] => devel_generate@example.com
[:db_insert_placeholder_12] =>
)
in comment_save() (line 1349 of /home/dm7/modules/comment/comment.module).

Tried updating from 6.14 to 7.x

Ran into a problem with a PDO EXCEPTION "out of range error" running todays HEAD, found part of the problem in
$Id: update.inc,v 1.11 2009/09/29 15:13:54 dries Exp $

First problem is located in the function "update_fix_d7_install_profile()", PDO exception stated 'weight' out of range. Checked the system table defaults of 6.14 to 7.x, totally different.

Changed the default values on my 6.14 system table fields to match 7.x and this part went through.

The next screen came up listing 86 updates.

Click ok and then ran into issues with non-existant table exception error.

If someone can direct me to the api to change a table columns current value to a new value I'll attempt to fix it. with the new DBTNG stuff and database API, it's fairly confusing, especially when clean-ups knock out working functions and you have to look for the new method.

Haven't quite figured out netbeans utility for producing a proper diff so enclosed are the changes I made to get through the first issue with the system table and PDO out of range exception. I've attached a patch (first attempt), probably isn't drupal format but i'm getting there

// $Id: update.inc,v 1.11 2009/09/29 15:13:54 dries Exp $

Line 159

function update_fix_d7_install_profile() {
  $profile = drupal_get_profile();
+++
  // Change 6.x system table field values to 7.x equivelent
  db_change_field('system', 'status', 'status',
    array('type' => 'int', 'length' => 11, 'not null' => TRUE, 'default' => 0));
  db_change_field('system', 'bootstrap', 'bootstrap',
    array('type' => 'int', 'length' => 11, 'not null' => TRUE, 'default' => 0));
  db_change_field('system', 'weight', 'weight',
    array('type' => 'int', 'length' => 11, 'not null' => TRUE, 'default' => 0));
  db_change_field('system', 'schema_version', 'schema_version',
   array('type' => 'int',
     'size' => 'small',
     'length' => 6,
     'not null' => TRUE,
     'default' => -1));

I'm not sure if this is the proper place but it converts a 6.14 table to the 7.x equivalent.

@ctmattice1: I don't see those lines anywhere in the code base.

Damien

Update.php $Id: update.php,v 1.305 2009/09/28 22:16:32 dries Exp $

callls these lines around 266

// We prepare a minimal bootstrap for the update requirements check to avoid
// reaching the PHP memory limit.
require_once DRUPAL_ROOT . '/includes/bootstrap.inc';
require_once DRUPAL_ROOT . '/includes/update.inc';
require_once DRUPAL_ROOT . '/includes/common.inc';
require_once DRUPAL_ROOT . '/includes/entity.inc';
update_prepare_d7_bootstrap();

the problem I was experiencing had to deal with the system tables in 6.14, the weight was valued at tinyint(2). Where I ran into problem was trying to update 6.14 to 7.x. Every time I tried it, it fatal errored with a PDO excpetion. the weight array value I could see was 1000 which would cause the out of range error. If I run a 7.x install to setup the database I don't get this with update.php

If you look in update inc for

function update_fix_d7_install_profile() {
 // some stuff then you'll see the weight hard coded

 // The install profile is always required.
    $file->info['required'] = TRUE;

    $values = array(
      'filename' => $filename,
      'name' => $profile,
      'info' => serialize($info),
      'schema_version' => 0,
      'type' => 'module',
      'status' => 1,
      'owner' => '',
    );

    // Install profile hooks are always executed last by the module system
    $values['weight'] = 1000;

when I added the code to update a 6.14 system table to match 7.x the problem disappeared. The error was only on the weight field but I went ahead and changed all but the type fields lenghts to match. Maybe when I upgraded from 6.12 to 6.14 something broke. But with the 6.14 weight field I had at a value of tinyint(2) the change was necessary to get update.php to the next step which is still broke with missing tables.

Link #278592: Sync 6.x extra updates with HEAD
Some issues summarized #49

@ctmattice1: This issue needs to remain a meta-issue, so I opened a separate bug for this: #601642: Old databases might have a small {system}.weight.

Current issues:

• #583008: IP address blocking does not work on update.php
• #211182: Updates run in unpredictable order
• #585628: Cannot use update.php as anonymous user
• #601642: Old databases might have a small {system}.weight

another issue for d6 and d7 #603498: Book {blocks.cache} upgrade path

Tagging.

some of the changes in this patch have been applied already. rerolling patch including the registry tables in update_fix_d7_requirements() for update.php to finish.

This is an intermediate patch which fixes some of the update functions (but not all). No time to split it into the appropriate issues but hopefully it can help someone getting started with it. This patch worked for me and I was able to upgrade a few dozens of node to the D7 field API :). I haven't fixed taxonomy because it requires to be run before the node update. Make sure to remove the taxonomy record from the system table before attempting any update with this patch.

scor, you're a lifesaver! Even though it requires 2 update.php runs, this managed to upgrade everything on a completely basic D6 to D7 with 15,000 nodes for me. Now I can get #563000: Provide upgrade path to file working. Thanks so much. Of course I'm hoping these will be fixed individually and included soon, but the patch as-is fixes tons of problems.

I would like to commit this, but there are a bunch of commented-out lines. Should these be deleted, or are they just parts that haven't been fixed yet?

I've spent the past couple of hours trying to get the upgrade working properly, starting with scor's patch in #24. I still can't get things working properly due to Taxonomy's updates needing to be run before Node module. Let's not discourage new contributors with calling a task like this "novice".

Patch should take care of the

+          // @todo the last paramter $fields was missing. must be an array of fields IDs.
+          field_sql_storage_field_storage_write('node', $node, FIELD_STORAGE_INSERT, array(1, 2));

My bad from #443422: 'per field' storage engine, I forgot we called that function directly in the 'body as field' update, to let the update run a tad faster.

Applied the patch in 24 #563106-24: Cannot upgrade from Drupal 6 to Drupal 7 - meta issue to todays HEAD. Ran into same error #601642: Old databases might have a small {system}.weight

Changed includes/updates.inc from

if (drupal_get_installed_schema_version('system') < 7000 && !variable_get('update_d7_requirements', FALSE)) {
    // Add the cache_path table.
    $schema['cache_path'] = drupal_get_schema_unprocessed('system', 'cache');
    $schema['cache_path']['description'] = 'Cache table used for path alias lookups.';
    db_create_table('cache_path', $schema['cache_path']);

    // system_update_7042() renames columns, but these are needed to bootstrap.
    // Add empty columns for now.

to this

if (drupal_get_installed_schema_version('system') < 7000 && !variable_get('update_d7_requirements', FALSE)) {
    // Change database system tables to 7.x equivelent structure
    // otherwise update may fail
    db_change_field('system', 'status', 'status',
      array('type' => 'int', 'length' => 11, 'not null' => TRUE, 'default' => 0));
    db_change_field('system', 'bootstrap', 'bootstrap',
      array('type' => 'int', 'length' => 11, 'not null' => TRUE, 'default' => 0));
    db_change_field('system', 'weight', 'weight',
      array('type' => 'int', 'length' => 11, 'not null' => TRUE, 'default' => 0));
    db_change_field('system', 'schema_version', 'schema_version',
      array('type' => 'int',
            'size' => 'small',
            'length' => 6,
            'not null' => TRUE,
            'default' => -1));
    // Add the cache_path table.
    $schema['cache_path'] = drupal_get_schema_unprocessed('system', 'cache');
    $schema['cache_path']['description'] = 'Cache table used for path alias lookups.';
    db_create_table('cache_path', $schema['cache_path']);

    // system_update_7042() renames columns, but these are needed to bootstrap.
    // Add empty columns for now.

This effectively eliminates the PDO exception with weight errors that php was reporting

[08-Nov-2009 19:47:56] PHP Fatal error: Uncaught exception 'PDOException' with message 'SQLSTATE[01000]: Warning: 1264 Out of range value for column 'weight' at row 1' in C:\sites\d6d7\includes\database\database.inc:1786
Stack trace:
#0 C:\sites\d6d7\includes\database\database.inc(1786): PDOStatement->execute(Array)
#1 C:\sites\d6d7\includes\database\database.inc(568): DatabaseStatementBase->execute(Array, Array)
#2 C:\sites\d6d7\includes\database\mysql\query.inc(37): DatabaseConnection->query('INSERT INTO {s...', Array, Array)
#3 C:\sites\d6d7\includes\update.inc(423): InsertQuery_mysql->execute()
#4 C:\sites\d6d7\includes\update.inc(363): update_fix_d7_install_profile()
#5 C:\sites\d6d7\update.php(342): update_fix_d7_requirements()
#6 {main}
thrown in C:\sites\d6d7\includes\database\database.inc on line 1786

Now though I'm getting

[08-Nov-2009 19:56:40] PHP Fatal error: Exception thrown without a stack frame in Unknown on line 0

I've looked at the system tables and last schema_version is 6053.

Tracing the next call is to update_fix_d7_install_profile() which writes the row to the system table the tries to load the schema

Now this is interesting as I have 2 listings as profile being a module, this one earlier in the system table

modules/profile/profile.module profile module 1 0 0 0 0 a:10:{s:4:"name";s:7:"Profile";s:11:"description";... Supports configurable user profiles.

and the one added by update_fix_d7_install_profile() which is the last table entry

profiles/default/default.profile default module 1 0 0 1000 0 a:11:{s:4:"name";s:6:"Drupal";s:11:"description";s...

Both are showing a status of 1, could this be the cause of the latest error

[08-Nov-2009 19:56:40] PHP Fatal error: Exception thrown without a stack frame in Unknown on line 0

Even if not, I don't believe both should be present, based on the function drupal_get_installed_schema_version() I can see where bad data could get passes.

Here's an improved patch that makes it so that Node module can run its updates without tripping over Taxonomy. I'm not sure if it's a good approach, but I got around Taxonomy module's requirements by setting the weight of Node module to "1" in update_fix_d7_requirements(), then setting it back to "0" in node_update_7000(). It's a bit hacky, but it definitely allows node.module to run last after other modules have gotten their Field system adjustments in place.

There's still a couple of TODOs that need to be cleaned up, so I'm moving this to needs work.

Includes ycheds fix from #28.

This patch is fully working, with all TODOs removed. In my testing Upload.module does not upgrade itself properly, but I'd like to just say "screw upload.module" and fix that problem by removing it entirely in #563000: Provide upgrade path to file, which I also have working now when applied in combination with this patch.

subscribe

Just tried with 'clean' Drupal 6.14 install, and somehow this patch requires the upload module to be removed otherwise the ordering of the modules is messed up and node gets executed before taxonomy. Although the weights are applied correctly in the db, drupal_get_installed_schema_version() returns a different ordering depending on whether upload module is present in the code base or not. I find this odd. Anyone care to enlighten me? that makes this patch dependant on #563000: Provide upgrade path to file.

@quicksketch, could you repost your instuctions from #563000-41: Provide upgrade path to file with whether the order of the update is correct or not please (in the collapsed fieldset at the selection step)? I tried with an old Drupal 6 site and it broke everything. I suspect there are other modules causing the same type of problems as upload.

+++ includes/update.inc	9 Nov 2009 04:26:20 -0000
@@ -291,6 +331,14 @@
+    // Set Node module's weight to 1 so that runs after other modules have
+    // properly prepared their Field APIs. Node module sets its weight back
+    // to 0 in node_update_7000().
+    db_update('system')
+      ->fields(array('weight' => 1))
+      ->condition('name', 'node')
+      ->execute();

This is the job of #211182: Updates run in unpredictable order. Please don't introduce more hacks then strictly necessary.

I'm on crack. Are you, too?

Oh dear, I thought modules would at least be updated according to their weight, but it looks like that is not the case. Are we really going to be trapped without an upgrade path until #211182: Updates run in unpredictable order is finished? Perhaps we can implement a basic "updates follow their module weight" for starters so we can get our basic upgrade working. It looks like the circular dependency checking in #211182 is not exactly ready.

Perhaps we can implement a basic "updates follow their module weight" for starters so we can get our basic upgrade working.

That would not really make sense. #211182: Updates run in unpredictable order is basically ready, and the circular dependency checking is not necessary at all.

Well... regardless here's a patch that now makes our upgrade path work by adding an ORDER BY clause to our retrieval of upgrade modules. This should now work except for the error caused by Upload module, but that won't be a problem if you've never enabled the upload module in you sandbox sites. It'd be great if #211182: Updates run in unpredictable order gets done, but at least this will make updates run like everything else in Drupal, by weight then by name.

rerolled #38 with some more fixes for filter_update_7004() and taxonomy_update_7004().

I would absolutely love to commit this patch!

However.

That temporary re-weighting of node module in update_fix_d7_requirements() is a no-go. That function runs by just *touching* update.php, and needs to be non-destructive. Someone should be able to back out D7 files and go back to their D6 site, perhaps with a few extra tables/indexes, but not with a core required module's weight suddenly shifted around, which I'm quite sure would break tons of contrib modules that make the assumption (reasonable or no) that node.module's weight is always 0.

So after a short chat with Webchick and finding that I can think of no solution that doesn't require drastic changes, I'm moving over to #211182: Updates run in unpredictable order in hopes I can get that patch working in tandem with this one (and the upload one) so that we can eventually run an upgrade. This version of the patch is the same as #39 only it doesn't contain the order-by-weight change or the node-weight change. So basically this patch is nothing but upgrade fixes. The hope is that we can commit this patch to *almost* get upgrades working (it's certainly better than what we have now), then have the patch in #211182: Updates run in unpredictable order actually make upgrades works when applied.

Another forgotten table "semaphore"

#627338: Add a cache_bootstrap bin removed the cache_registry table so the system_update_7006() hunk fails. rerolled patch.

Just want to let you guys know that http://drupal.org/project/demo_profile fully works on D7 now:

#628374: Drupal 7 core install profile AWESOMENESSAUCEWITHEXTRACHEESE!

Not sure whether it helps to test the D6->D7 upgrade path, but you may find out. It definitely helps to test regular updates.

add beta blocker tag for the patch here.

This is the patch I used to test the upgrade drush updatedb code

http://drupal.org/files/issues/drupal_7_update_fix.diff

The list of issues are :

1) typo in node.install :
db_upddate to db_update

2) incorrect usage of UpdateQuery fields method
fields(status, 1) to fields(array(status, 1))

3) missing db_change_field
was incorrectly removed in http://drupalcode.org/viewvc/drupal/drupal/modules/node/node.install?r1=...
the 'module' column was not being renamed to the base column

4) a bunch of incorrect modifications of a non-existent 'user' table in user.install
it is now, and has always been 'users'.

5) taxonomy_update_7002 needed to be run before node_update_7006
i fixed this by adding a taxonomy_update_6002_run variable, and calling the update
function directly from node_update_7006.

Looks good, except from the taxonomy_update_7002() workaround. There's a separate issue about the execution order of updates in the queue.

@sun do you know that other issue?

I found an issue with comment module update. The error was "Call to undefined function field_attach_create_bundle()".

field is a new and required module. Is it enabled out of the box?

@mfer I suppose it's #211182: Updates run in unpredictable order

@adrian, all the issues you mention in #48 are already fixed in the patch #44.

rerolling patch after some hunks have already been committed in separate issues. upload.module does not exist anymore. We're still dependent on #211182: Updates run in unpredictable order though.

+++ includes/session.inc
@@ -242,6 +242,11 @@ function drupal_session_commit() {
+  // Expire the keep if possible.
+  if (isset($_SESSION['drupal_keep']) && ($_SESSION['drupal_keep'] > REQUEST_TIME)) {
+    unset($_SESSION['drupal_keep']);
+  }

1) The inline comment is a bit sparse.

2) Additional parenthesis not required in the condition.

3) Should call the reset function.

+++ includes/session.inc
@@ -271,6 +276,25 @@ function drupal_session_started($set = NULL) {
+ /**
+ * Keep the current user's session open for a given length of time.

Wrong PHPDoc indentation.

Could also use a separate, more lengthy PHPDoc description about what this function is for, where the defined value is going to be checked, and when it is intended to be used - and when NOT.

A @see to the reset function wouldn't hurt either.

+++ includes/session.inc
@@ -271,6 +276,25 @@ function drupal_session_started($set = NULL) {
+ * @param $duration The length the session needs to remain open, in seconds.

Wrong @param syntax.

+++ includes/session.inc
@@ -271,6 +276,25 @@ function drupal_session_started($set = NULL) {
+function drupal_session_keep($duration) {
+  $new_timestamp = REQUEST_TIME + $duration;
+  if (!isset($_SESSION['drupal_keep']) || ($new_timestamp > $_SESSION['drupal_keep'])) {
+    $_SESSION['drupal_keep'] = $new_timestamp;
+  }
+}

I don't see how the second condition could ever be not met. Which in turn means that the entire condition is superfluous.

+++ update.php
@@ -395,6 +395,9 @@ else {
+  // Force the session to stay open for 15 minutes.
+  drupal_session_keep(300);

Why not

drupal_session_ttl()

?

And why 15 minutes? The inline comment should explain.

I'm on crack. Are you, too?

Adding #450540: Upgrade instructions recommend enabling stark in D6 to this meta issue to track.

reply to @sun

am i the only person who feels it is patently ridiculous to hold up a MAJORLY CRITICAL piece of functionality like this on something as silly as a 'boil the ocean' acyclic directed graph approach to granular hook weighting?

we're holding up the 4 line solution to the one occurrence of a problem we have solved in similar ways before on the possibility that we can develop a complex replacement algorithm, less than a week before we are meant to release the alpha ?

we should not be adding new hooks at this point.

In case you forgot: Upgrading from D5 to D6 required users to perform very tricky preparations and contrib to invent some nasty hacks to get the upgrade path only remotely right. Upgrading to D7 will have a lot more problems like that, especially when considering the module namespace clashes we purposively introduced as well as renamed modules and the entire CCK upgrade path.

@adrian #58

These 4 lines of code are only for taxonomy / core. We must supply contrib modules a clear solution on their #211182: Updates run in unpredictable order right?

I think I'm going to give the dependent issue until Thursday the 14th to get fixed, and if it isn't, we're going to go with Adrian's solution, and document it as a workaround. I agree with him that this is ridiculous.

Yeah, screw holding back good solutions for perfect solutions.

Back to needs review to see what the bot thinks.

Yes, let's ship with the crappy solution. But someone needs to actually roll a patch for this (mixing #47 and #53), and taking into account the review in #54.

This is utterly ridiculous, but I'm tired fighting.

Here is re-roll with changes:
1) I see no reason in drupal_session_keep()
2) taxonomy_update_7004() adds taxonomy_term_reference not taxonomy_term
3) sequences table used by batch so moved to reqiurements
4) node_update_7006 set $info->old_type = $info->type; else no content types are available after update
5) filter_update_7004() tabs replaced by spaces

Batch totally broken after #629794: Scaling issues with batch API

subscribing

Batch API is fixed again. My apologies. :(

Another round, now update process completes with only one warning

comment.install
Update #7012
    * Failed: SQLSTATE[42S22]: Column not found: 1054 Unknown column 'machine_name' in 'field list'

But after another run update.php it works

Still problem with viewing node body

Subscribed.

Node body was broken because my D6 setup was multilingual and field_multilingual_valid_language() returns only one (default) language for body field... but this for another issue.

comment_update_7005() seems totally useless because:
1) runs before node updates
2) does nothing because original comment_node_type_insert() still holds @todo and uses _comment_body_field_instance_create()

so creating comment fields and bundles in comment_update_7005() should run after node_update_7006()

Suppose we should commit this changes because mostly there's typos and lost code pieces and continue after #211182: Updates run in unpredictable order or after inventing ordered updates :(

I've committed #73 as another interim step on this issue. Also, optimistically tagging for alpha. It would be so amazing to have a working upgrade path for the first alpha! And I really appreciate the work you all have been doing on the alpha blocker patches.

Back to needs work for the next follow-up.

i'm adding #606730: provide a helper function for block delta updates to this list -- it's a major upgrade PITA for module developers who use blocks, and it's tested and ready to go.

Note that I have posted an updated patch #211182: Updates run in unpredictable order which I believe provides a working upgrade path. (I've only lightly tested it, but I can at least successfully update an empty D6 site to D7 without generating any update failures.)

adding #684084: node_update_7006() broken after 'node titles as fields' rollback to the list.

Just to keep this issue updated, I just did a bunch of hammering on the upgrade path over here http://drupal.org/node/211182#comment-2474322 from #114 downward.

added #685416: filter table renames aren't recognized by a later update

More fun #685486: filter_update_7003() doesn't work for contrib modules.

Some more fun: #685572: system_update_7008() is broken.

Another fun one: #685892: Upload -> File module upgrade path is broken

#686128: Body-to-field conversion is too slow

Testing with alpha1. Big problem. The update dies part way through with a white screen of death. How do you tell which update failed? Is there a way to make the updates display as they occur? The number of tables increases by 9. node_revisions is not renamed. The system table does not list a schema change. The url changes to update.php?op=info. There is nothing in watchdog.

I tested running update a few times including without reloading the database. db_create_table will rerun without error. db_add_field goes straight to the WSOD when it tries to add a field that is already there and does not produce a watchdog error.

After a database reload, the WSOD occurs in:
db_insert('sequences')->fields(array('value' => max($max_aid, $max_uid, $max_batch_id)))->execute();

The update adds the sequences table but the table already exists in a different format. I tried an update consisting of reloading the database, dropping sequences, update.php, and it worked.

sequences contains the following names. You check only three of them for the max() value.
menu_mid, drupal_users_uid, drupal_node_nid, drupal_vocabulary_vid, drupal_menu_mid, drupal_term_data_tid, drupal_forms_fid, drupal_form_fields_ffid, drupal_survey_responses_rid, drupal_files_fid, drupal_comments_cid, drupal_node_revisions_vid, path_redirect, boxes_bid

The patch at #211182-140: Updates run in unpredictable order should give you an almost working upgrade path.

I have another really small patch that gets me a working upgrade path with drush head.

1) comment_update_7012 was triggering the error for the taxonomy 'machine_name', through the field api functions.
This is an instance of the exception from the patch at #73 , so I added a call to the update directly.

2) filter_update_7005 was attempting to modify the format field of the column table
this field was removed in comment_update_7013, so this table is no longer necessary.

3) system_update_7027 was not enabling the modules.
This was a tricky one to track down, as the error wasn't occuring in this update, but rather later whenever the field api was being used.

The error being presented was : "Attempt to create a field of unknown type text_long."
What was happening here, is that the update was removing entries from the system table, and then enabling the new modules. These new modules (which define the field types) were never being added to the module list, even after module_list($refresh).

I tracked this down to the fact that the module_enable function doesn't explicitly insert items into the system table, it merely runs an update query. It assumes that all the modules are present in the system table, and the system table is only rebuilt if the 'dependencies' parameter is set, which we don't want in this specific case.

Because the system entries weren't there, the update statement was failing silently, and the modules weren't added to the module list, so all field operations which used these field types would fail.

+++ modules/comment/comment.install
@@ -247,6 +247,8 @@ function comment_update_7011() {
 function comment_update_7012() {
+  taxonomy_update_7002();
+

Can we get some kind of @todo comment here to remove this once proper update ordering is in?

Powered by Dreditor.

added comments to where taxonomy_update_7002 is called directly linking to the ordering issue.

removed the loop around the format column update as only the block custom table was still being updated.

Ive decided to revert back to using the loop for filter_update_7005 , because the block module is optional hence the table might not exist, and even though there's only one item in the array, at least we aren't assigning or typing out the name 'block_custom' twice.

The changes should be reviewed.

4 lines of sensible code. i agree with immediate commit here.

1. @@ -336,7 +336,7 @@ function filter_update_7005() {
      // instances (in Drupal core) to explicitly use the appropriate format.
      // Note that the update of the node body field is handled separately, in
      // node_update_7006().
   -  foreach (array('block_custom', 'comment') as $table) {
   +  foreach (array('block_custom') as $table) {
   

   The format column of the {comment} table is being removed from core, but its data is not. When does its data get the equivalent of filter_update_7005() performed on it if not here?

2. @@ -2246,6 +2246,7 @@ function system_update_7027() {
      $module_list = array('text', 'number', 'list', 'options');
      db_delete('system')
        ->condition('type', 'module')
        ->condition('name', $module_list)
        ->execute();
   +  system_rebuild_module_data();
      module_enable($module_list, FALSE);
   

   This defeats any purpose of calling module_enable() with the FALSE parameter, right? Also defeats the purpose of calling the db_delete() query right above it. I believe that db_delete() query was added in #490074: 6/7 upgrade path broken as an attempt to make the D6->D7 field updates work nicely for sites that have D6 contrib versions of CCK modules lying around... however, doesn't seem like that's effective anyway.

The existing patch at #211182: Updates run in unpredictable order fixes #1 by adding a correct dependency between the filter and comment updates, and #2 by removing the db_delete() altogether. If we want to get a quick patch to commit here, I think it would be best to use the same approach already worked out there.

- foreach (array('block_custom', 'comment') as $table) {
+ foreach (array('block_custom') as $table) {
The format column of the {comment} table is being removed from core, but its data is not. When does its data get the equivalent of filter_update_7005() performed on it if not here?

They were field-ified as far as i can tell. The columns were summarily dropped entirely.

This defeats any purpose of calling module_enable() with the FALSE parameter, right? Also defeats the purpose of calling the db_delete() query right above it. I believe that db_delete() query was added in #490074: 6/7 upgrade path broken as an attempt to make the D6->D7 field updates work nicely for sites that have D6 contrib versions of CCK modules lying around... however, doesn't seem like that's effective anyway.

No, the issue is EXISTING records from BEFORE the upgrade that had schema versions and statuses that weren't in any way related to the new modules. This delete resets the system database for those new modules, so that they can be successfully picked up.

I can't vouch for the logic behind calling it with the false parameter (apart from it seeming like complete overkill to build a dependency tree to enable these 4 modules), but i can validate that is impossible in the current code to enable those modules if they don't have records in the system table first, which is by design.

1) old records for previous versions of those modules (from cck) need to be purged
2) after this is called records for the new modules with the correct status and possibly schema version (although none of these modules have .install files) need to end up in the table.

At the moment the code only handles one of those requirements, and completely ignores the second (main) requirement. you know, actually enabling said modules.

So , going ahead you can either :

a) always rebuild the module table in module_enable, which doesn't seem necessary.
b) accept that almost nobody calls module_enable after directly altering the system table and invalidating all the caches of it, and those people will just need to rebuild the system table.
c) modify the module_enable function to INSERT new records if no existing record was found to update, which would require a lot more queries and require you to re-run the discovery mechanism code in any case.

b in my opinion is the right answer. completely beyond whether

feel free to pain the bike shed and design new hooks in the other issue, but we need to be able to actually test upgrades for now. who cares if these small fixes get replaced by a better system.

They were field-ified as far as i can tell. The columns were summarily dropped entirely.

I think that's true. But the process of fieldiying them means that their text formats were copied to a fields database table. I don't believe there's update code anywhere else that would perform the equivalent of filter_update_7005() on them once they are in that table, meaning that some comment text formats would not wind up being correct.

1) old records for previous versions of those modules (from cck) need to be purged
2) after this is called records for the new modules with the correct status and possibly schema version (although none of these modules have .install files) need to end up in the table.

I agree that's the (ideal) goal, but neither current core nor the patch here does that. The issue is that if you call:

  db_delete('system')
    ->condition('type', 'module')
    ->condition('name', $module_list)
    ->execute();
  system_rebuild_module_data();

then system_rebuild_module_data() just puts back in the database exactly what db_delete() took out (this is because CCK is typically going to be in sites/all/modules or somewhere, which takes precedence over the core module locations)... so you still can't upgrade your site with D6 CCK modules lying around.

That's not for this issue to fix (it might not be fixable without manual intervention), but my point is just that the above two lines of code do nothing when combined together. Instead of adding system_rebuild_module_data(), this patch could remove the db_delete() instead, and it should work just as well.

Tried the patch on a personal D6 site, only 2 errors remain, all the other issues are resolved now!

Failed: SQLSTATE[42000]: Syntax error or access violation: 1170 BLOB/TEXT column 'location' used in key specification without a key length

Failed: SQLSTATE[42S02]: Base table or view not found: 1146 Table 'blog.poll_votes' doesn't exist

Edit:
However, there seems to be no comment/node body field displayed now. Also, I'm recieving the following error in watchdog:

PDOException: SQLSTATE[42S02]: Base table or view not found: 1146 Table 'blog.image_styles' doesn't exist: SELECT image_styles.* FROM {image_styles} image_styles ORDER BY name ASC; Array ( ) in image_styles() (line 409 of .../modules/image/image.module).

Edit 2: Was able to resolve the above error by uninstalling and re-installing image.module. I'm not sure, maybe I've had the "old" image.module installed once on that site.

Failed: SQLSTATE[42000]: Syntax error or access violation: 1170 BLOB/TEXT column 'location' used in key specification without a key length

I cant reproduce this, which update throws this?

This is locale_update_7000(), the "db_change_field('locales_source', 'location', 'location', ... ".

I am not sure what the issue is, but I can't even do that manually through phpmyadmin. That is, changing the field from varchar(255) to TEXT. Always gives me the above error.

Ok, DamZ pointed me to the issue. The problem is that i18n adds an index to that table, and indexes on TEXT columns need a length.

#95 - ok , i get that now.

how about explicitly setting it to the paths we need it to be during install, and making a note that these modules can NOT be overridden, during upgrade at least.

@adrian, yeah, that sounds like it might work - although not totally sure how we prevent it from being overridden, since it seems like that would happen again as soon as someone rebuilds the module data, which could be done by more or less anyone.

In any case, it seems like there should probably be a separate issue for getting the upgrade to work for people with D6 CCK in their installations, since there are at least other workarounds for that? - and this patch isn't going to make the situation any worse if the current technique being used was already broken :)

For now, here is a reroll of the previous patch with minimal fixes for the issues I identified above.

This works 100% for me, and fixes all the issues you came across.

thanks for running with it.

Subscribing

Committed the patch in #101 to help make module upgrades make progress.

Quick question about D7 dev download. I would like to test #101. The download says it was produced January 26, 2010 - 10:04. #104 mentions a commit and is dated January 26, 2010 - 07:33. Do these commits go straight into the dev download?

@peterx: I haven't tried it specifically, but yes, that's how it should work.

RT #105: I guess there might be a delay between a commit (for example #101) and it being available in the tarball (Drupal 7.x-dev tarball).

You should definitely use CVS to work with the dev version. You would always have the latest code.

That being said, I really don't know how fast tarball are generated.

tarballs are every 12 hours afaik - at midday and midnight GMT.

Upgrading my Drupal 6 to Drupal 7 CVS of today, give this errors.:

Some warnings that don't break the site...

    *  Warning: file_put_contents(temporary://file8KmQji): failed to open stream: "DrupalTemporaryStreamWrapper::stream_open" call failed in file_unmanaged_save_data() (line 1522 of /persistent/home/int/sites/drupal7/includes/file.inc).
    * Can't create the file.

And run all updates(113) very fast, at the (111 update) and give lots of time..

Completed 111 of 113.  98%
Updating user module

At linux shell executing TOP command:

27836 www-data  20   0  101m  34m  26m S 20.3  2.0   1:01.91 apache2
27841 www-data  20   0  100m  33m  26m S 11.3  2.0   0:05.01 apache2
1072 mysql     20   0  869m 436m 6620 S  4.7 25.5   6:14.92 mysqld

And where working by between 5 and 10 minuts... and then the apache2 and mysql aren't active working anymore, but the Update page is in the same text..

Completed 111 of 113.  98%
Updating user module

So, I restart the update and show me:

Completed 22 of 24.
Updating user module

and the linux process and same and above... but the process were working for at least 20minuts.. but give the same problem.. The page don't move one, it's stoped at the user module upgrade...

I have 5167 nodes, and 1411 users..
What is the problem with update the user module?

The upgrade process don't give any error, neither timeout error...

I press Esc, and click in show errors:
Give:

he update process was aborted prematurely while running update #7004 in user.module. All errors have been logged. You may need to check the watchdog database table manually.

    * Front page
    * Administration pages

The following updates returned messages
system module
Update #7035

    * Failed: SQLSTATE[23000]: Integrity constraint violation: 1052 Column 'fid' in field list is ambiguous

filter module
Update #7004

    * Failed: SQLSTATE[42000]: Syntax error or access violation: 1091 Can't DROP 'fid'; check that column/key exists

locale module
Update #7000

    * Failed: SQLSTATE[42000]: Syntax error or access violation: 1091 Can't DROP 'source'; check that column/key exists

poll module
Update #7002

    * Failed: SQLSTATE[42S02]: Base table or view not found: 1146 Table 'drupal7.drupal_poll_votes' doesn't exist

user module

# Drupal 6.15 to Drupal 7 CVS Today
# PHP 5.2.10-2ubuntu6.4
# Apache/2.2.12 (Ubuntu)
# Versión del cliente: 5.1.37
# Ubuntu 9.10 Server

user_update_7004 Add the user's pictures to the {file} table and make them managed files.

I remove all Drupal6 files include the modules, files, site folders.. (after I disable all contrib modules)

SELECT COUNT( * )
FROM `drupal_users`
WHERE picture <> ''

= 14

I update the table to drupal_users set picture = '' and now the two update users updates fine..

now I have the others errors:

The following updates returned messages
system module
Update #7035

    * Failed: SQLSTATE[23000]: Integrity constraint violation: 1052 Column 'fid' in field list is ambiguous

filter module
Update #7004

    * Failed: SQLSTATE[42000]: Syntax error or access violation: 1091 Can't DROP 'fid'; check that column/key exists

locale module
Update #7000

    * Failed: SQLSTATE[42000]: Syntax error or access violation: 1091 Can't DROP 'source'; check that column/key exists

poll module
Update #7002

    * Failed: SQLSTATE[42S02]: Base table or view not found: 1146 Table 'drupal7.drupal_poll_votes' doesn't exist

user module

fix the poll_update_7002

Fix the PHPDoc

Can any one give a review of this simple patch in #112, the bot was allready pass this one 6hours ago...

#84 revisited using D7 dev Jan 27 10:05 dev. Crashed inserting into sequences.
Precede $schema['sequences'] = array( with

    $max_sequence = db_query('SELECT MAX(id) FROM {sequences}')->fetchField();
    db_drop_table('sequences');

Change db_insert('sequences')->fields(array('value' => max($max_aid, $max_uid, $max_batch_id)))->execute();
to db_insert('sequences')->fields(array('value' => max($max_aid, $max_uid, $max_batch_id, $max_sequence)))->execute();
Sequence problem disappears.

Got through to 104 updates pending.
Strict warning: Only variables should be passed by reference in update_selection_page() (line 33 of H:\home\example.com\public_html\update.php).
I changed $output = drupal_render(drupal_get_form('update_script_selection_form'));
to

$stuff = drupal_get_form('update_script_selection_form');
$output = drupal_render($stuff);

Update selection gave:
PDOException: SQLSTATE[42S22]: Column not found: 1054 Unknown column 'name' in 'field list': INSERT INTO {queue} (name, data, created) VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1, :db_insert_placeholder_2); Array ( [:db_insert_placeholder_0] => drupal_batch:4146:0 [:db_insert_placeholder_1] => a:2:{i:0;s:13:"update_do_one";i:1;a:2:{i:0;s:6:"system";i:1;s:4:"7000";}} [:db_insert_placeholder_2] => 1264573566 ) in SystemQueue->createItem() (line 192 of H:\home\example.com\public_html\modules\system\system.queue.inc).

Table queue is `nid`, `uid`, `vote`. I manually recreated queue and ran the update by hand. 102 updates ran in an instant. Update 103 is churning away.

User 7004. That continue; has got to go. It makes the code loop if the last user goes through continue because of a missing file. I changed if (!file_exists($user->picture)) { to if (file_exists($user->picture)) { and wrapped the if down to the end of db_update('users'). The change makes users all users, including those with missing files, go through the position update. No more infinite loop in user 7004.

peterx Suggestion

Need a review for this and #112

I haven't been able to test this patch, but looking at the code, it looks good to me.

One question though; if the $user->picture does not exist, is that because we have an invalid path or because we have an empty path? If it happens to be the case that the path is wrong, one could argue that the $user object is in an inconsistent state. Should we consider empty-ing $user->picture when that is the case?

RTBC for #112 (pool_update_7002)

Need Review for #116

@Dries, if the $user->picture does not exist, is that because we have an invalid path. The empty path are allready filter in the SQL query. i think that isn't necessary because we drop 'picture' field

Committed #116. Thanks int!

Committed #112 as well. Thanks!

my patch in #112 was not been committed yet.. (pool_update_7002_02.patch)

Hm. #112 looks to be committed here. Or at least when I try to apply it, it asks me if I want to revert it.

Re-testing upgrade of webchick.net now...

Well that was quick:

"An AJAX HTTP error occurred. HTTP Result Code: 200 Debugging information follows. Path: http://localhost/webchick.net/update.php?id=4&op=do StatusText: OK ResponseText: Fatal error: Call to undefined function field_attach_create_bundle() in /Users/webchick/Sites/webchick.net/modules/comment/comment.install on line 149"

Call Stack:
0.0071 163140 1. {main}() /Users/webchick/Sites/webchick.net/update.php:0
4.0619 15155836 2. _batch_page() /Users/webchick/Sites/webchick.net/update.php:388
4.0633 15157192 3. _batch_do() /Users/webchick/Sites/webchick.net/includes/batch.inc:81
4.0633 15157480 4. _batch_process() /Users/webchick/Sites/webchick.net/includes/batch.inc:163
4.9624 15272576 5. call_user_func_array() /Users/webchick/Sites/webchick.net/includes/batch.inc:286
4.9624 15272576 6. update_do_one() /Users/webchick/Sites/webchick.net/includes/batch.inc:0
4.9625 15274308 7. comment_update_7005() /Users/webchick/Sites/webchick.net/includes/update.inc:689

That's weird, because in
http://drupalcode.org/viewvc/drupal/drupal/modules/poll/poll.install?rev...
and http://drupal.org/project/cvs/3060 don't show the commit.. later I will verify it with in the cvs command..

@webchick error in #125 should be fixed by #211182: Updates run in unpredictable order (the comment update are run before system.install had the chance to install the field module.)

#112 applies cleanly, latest change for poll.install was Dec 4th, and it still contains db_add_field('poll_votes', 'timestamp', $field); which is what #112 fixes.

see also #599122: Do not use module_load_include() in global context which fixes system_update_7008() (see duplicate #685572: system_update_7008() is broken)

RTBC #112 again? :)

LOL. Weird. I must've been tired last night. :P

Really committed to HEAD now. ;)

Today #251792: Implement a locking framework for long operations commited to 6 so updates should be changed

#211182: Updates run in unpredictable order, which adds hook_update_dependencies(), was committed.

Would be good for someone to try the upgrade path now and see how far along we are. I'll try this with webchick.net in the next couple of days.

using todays head and #211182: Updates run in unpredictable order latest patch http://drupal.org/files/issues/211182-updates-180.patch I get the follwoing php error

[03-Feb-2010 15:26:54] PHP Fatal error:  Uncaught exception 'PDOException' with message 'SQLSTATE[42S22]: Column not found: 1054 Unknown column 'value' in 'field list'' in C:\sites\d6d7\includes\database\database.inc:1940
Stack trace:
#0 C:\sites\d6d7\includes\database\database.inc(1940): PDOStatement->execute(Array)
#1 C:\sites\d6d7\includes\database\database.inc(586): DatabaseStatementBase->execute(Array, Array)
#2 C:\sites\d6d7\includes\database\mysql\query.inc(37): DatabaseConnection->query('INSERT  INTO {s...', Array, Array)
#3 C:\sites\d6d7\includes\update.inc(522): InsertQuery_mysql->execute()
#4 C:\sites\d6d7\update.php(378): update_fix_d7_requirements()
#5 {main}
  thrown in C:\sites\d6d7\includes\database\database.inc on line 1940

Initial database does not contain a field table.

Skip that fields table thingy.

Traced progression of update_fix_d7_requirements(). It completes all the way down to

if (db_table_exists('locales_source')) {
      db_add_field('locales_source', 'context', array('type' => 'varchar', 'length' => 255, 'not null' => TRUE, 'default' => '', 'description' => 'The context this string applies to.'));
}

should

'The context this string applies to.'

be doublequoted instead of single quotes due to the period?

Hm. Nope, I can't see that causing a problem. That'd only be if the sentence contained an apostrophe.

Could you elaborate a bit more? At what point do you get an error? At the main update.php page, or after starting the upgrade process, or..? And what is the error?

Found it!

My d6 database has a queue table already.

so

 // Add the queue table.
    $schema['queue'] = array(
      'description' => 'Stores items in queues.',
      'fields' => array(
        'item_id' => array(
          'type' => 'serial',
          'unsigned' => TRUE,
          'not null' => TRUE,
          'description' => 'Primary Key: Unique item ID.',
        ),
        'name' => array(
          'type' => 'varchar',
          'length' => 255,
          'not null' => TRUE,
          'default' => '',
          'description' => 'The queue name.',
        ),
        'data' => array(
          'type' => 'text',
          'not null' => FALSE,
          'size' => 'big',
          'serialize' => TRUE,
          'description' => 'The arbitrary data for the item.',
        ),
        'expire' => array(
          'type' => 'int',
          'not null' => TRUE,
          'default' => 0,
          'description' => 'Timestamp when the claim lease expires on the item.',
        ),
        'created' => array(
          'type' => 'int',
          'not null' => TRUE,
          'default' => 0,
          'description' => 'Timestamp when the item was created.',
        ),
      ),
      'primary key' => array('item_id'),
      'indexes' => array(
        'name_created' => array('name', 'created'),
        'expire' => array('expire'),
      ),
    );
    db_create_table('queue', $schema['queue']);

won't work.... Need to check for the table first, modify if present else write it.

@webchick

"Could you elaborate a bit more? At what point do you get an error? At
the main update.php page, or after starting the upgrade process, or..?
And what is the error? "

WSOD on update.php, error is the same as above

[03-Feb-2010 15:59:31] PHP Fatal error: Uncaught exception 'PDOException' with message 'SQLSTATE[42S22]: Column not found: 1054 Unknown column 'value' in 'field list'' in C:\sites\d6d7\includes\database\database.inc:1940
Stack trace:
#0 C:\sites\d6d7\includes\database\database.inc(1940): PDOStatement->execute(Array)
#1 C:\sites\d6d7\includes\database\database.inc(586): DatabaseStatementBase->execute(Array, Array)
#2 C:\sites\d6d7\includes\database\mysql\query.inc(37): DatabaseConnection->query('INSERT INTO {s...', Array, Array)
#3 C:\sites\d6d7\includes\update.inc(522): InsertQuery_mysql->execute()
#4 C:\sites\d6d7\update.php(378): update_fix_d7_requirements()
#5 {main}
thrown in C:\sites\d6d7\includes\database\database.inc on line 1940

Tracing PDO Exceptions is a ROYAL pain!

@137 Sounds like you already have a {sequences} table as well. Quick check looks like we never deleted the old sequences table from Drupal 5. So any upgrades sites from D5 to D6 to D7 are going to run into this.

On a side note my D6 database also has a sequences table.

This particular database is a copy of a d5 site, brought to my local environment, updated to 6.15 then attempted to update to d7. I've been using this database and found #601642: Old databases might have a small {system}.weight which is now RTBC.

Great sleuthing!

Can we wrap those hunks in a if (!db_table_exists()) check or similar?

db_create_table() auto-checks db_table_exists(), so that's not the problem. The problem is that the D5 and D7 versions of {sequences} are different and I think we'll need to to a drop table if it exists before the db_create_table().

I can but need to wait to cvs refreshes. I have the patch I used but that will chase HEAD when it catches up.

Can work on it in tomorrow morning since what were actually modifying will be update.inc

Dave Reid - February 3, 2010 - 16:45

"db_create_table() auto-checks db_table_exists(), so that's not the problem. The problem is that the D5 and D7 versions of {sequences} are different and I think we'll need to to a drop table if it exists before the db_create_table()."

Was {sequences} used in d6?

Where we're changing the code from my understanding is in a place where if the update fails, d6 will still run from the database

See #114 for a suggestion on sequences in case user modules are using sequences. My first site for conversion went from D5 to D6 some time ago and uses a few add on modules. The last update for the sequences table is after the D5 -> D6 conversion suggesting an add on module used the table. The last update to the table was a long time ago when I replaced some modules with alternatives, suggesting it is probably not used by any module currently installed. A quick scan of the site code shows only the following single use of {sequences}. Perhaps someone could scan all add on modules for {sequences} and notify any module maintainers who access the table direct.

/**
 * Remove auto_increment from {boxes} to allow adding custom blocks with
 * visibility settings.
 */
function system_update_6000() {
  $ret = array();
  switch ($GLOBALS['db_type']) {
    case 'mysql':
    case 'mysqli':
      $max = (int)db_result(db_query('SELECT MAX(bid) FROM {boxes}'));
      $ret[] = update_sql('ALTER TABLE {boxes} CHANGE COLUMN bid bid int NOT NULL');
      $ret[] = update_sql("REPLACE INTO {sequences} VALUES ('{boxes}_bid', $max)");
      break;

#145 sounds sensible to me.

Downloaded from this morning. Loaded database. Deleted sequences table. Ran update.php. Reached update page with 106 pending updates and:
Strict warning: Only variables should be passed by reference in update_selection_page() (line 33 of H:\home\example.com\public_html\update.php).
Continue produced:
PDOException: SQLSTATE[42S22]: Column not found: 1054 Unknown column 'name' in 'field list': INSERT INTO {queue} (name, data, created) VALUES

Back to the start. Reload database. Drop sequences and queue. Create queue using SQL form a D7 test site. update.php. Ran through to completion. Really nice having the extra line of information telling me which update is in progress. Two failed messages.
filter module Update #7004
* Failed: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '3-0' for key 'PRIMARY'
block module Update #7003
* Failed: SQLSTATE[42000]: Syntax error or access violation: 1091 Can't DROP 'list'; check that column/key exists

Logged in as administrator. No theme. No menu entries. I went to admin/appearance and set the admin theme to Garland. I like Seven but Seven was disabled.
Admin by task is empty. Admin by module has many entries.

block_update_7003() starts by dropping index list but there is no index named list. I removed the line. The next line creates index list but the index exceeds 1000 characters when you use fields theme and module as UTF8 so I put module in a separate index. The block updates run to completion. Blocks appear in Garland. When I try to allocate blocks in the blocks page, there are no regions listed.

Added #615634: Update CHANGELOG.txt to explicitly name new core modules and core modules that have been removed. to the list. Its a bit of a blocker for #536854: Review and test the upgrade guide for Drupal 7.
Actually all these issues in this meta issue are a blocker for #536854, but at least some writing could be done while upgrade path gets fixed.

D6: CREATE TABLE IF NOT EXISTS `queue` (
`nid` int(10) unsigned NOT NULL DEFAULT '0',
`uid` int(10) unsigned NOT NULL DEFAULT '0',
`vote` int(3) NOT NULL DEFAULT '0',
PRIMARY KEY (`nid`,`uid`));

D7: CREATE TABLE IF NOT EXISTS `queue` (
`item_id` int(10) unsigned NOT NULL AUTO_INCREMENT COMMENT 'Primary Key: Unique item ID.',
`name` varchar(255) NOT NULL DEFAULT '' COMMENT 'The queue name.',
`data` longtext COMMENT 'The arbitrary data for the item.',
`expire` int(11) NOT NULL DEFAULT '0' COMMENT 'Timestamp when the claim lease expires on the item.',
`created` int(11) NOT NULL DEFAULT '0' COMMENT 'Timestamp when the item was created.',
PRIMARY KEY (`item_id`),
KEY `name_created` (`name`,`created`),
KEY `expire` (`expire`)
)COMMENT='Stores items in queues.';

I cannot find any d6 code or d5 code referring to {queue}.

Applied #601642: Old databases might have a small {system}.weight followup.patch

Renamed the queue and sequences tables to queue_old and sequences_old and added a todo comment

Can now get a D5 -> D6 -> D7 database to this point. I haven't actually gone in yet to the settings.php file to change the free_access setting to allow me into the site after running update.php

Access denied. You are not authorized to access this page. Log in using either an account with the administer software updates permission or the site maintenance account (the account you created during installation). If you cannot log in, you will have to edit settings.php to bypass this access check. To do this:

   1. With a text editor find the settings.php file on your system. From the main Drupal directory that you installed all the files into, go to sites/your_site_name if such directory exists, or else to sites/default which applies otherwise.
   2. There is a line inside your settings.php file that says $update_free_access = FALSE;. Change it to $update_free_access = TRUE;.
   3. As soon as the update.php script is done, you must change the settings.php file back to its original form with $update_free_access = FALSE;.
   4. To avoid having this problem in the future, remember to log in to your website using either an account with the administer software updates permission or the site maintenance account (the account you created during installation) before you backup your database at the beginning of the update process.

patch attached for review. Didn't know if I should create another issue or not.

Tried again, got a new error.

When update.php is finished (batch run throug completely), I got the following fatal error.

Fatal error: Call to undefined function field_info_field() in d7/drupal/modules/node/node.module on line 572

Call Stack:
    0.0002      97408   1. {main}() d7/drupal/update.php:0
    6.3229    4595640   2. _batch_page() d7/drupal/update.php:425
    6.3236    4696968   3. _batch_finished() d7/drupal/includes/batch.inc:90
    6.3251    4748368   4. update_finished() d7/drupal/includes/batch.inc:451
    6.3251    4748760   5. drupal_flush_all_caches() d7/drupal/includes/update.inc:857
   11.1318   12049712   6. node_types_rebuild() d7/drupal/includes/common.inc:6222
   11.1327   12051968   7. node_type_save() d7/drupal/modules/node/node.module:477
   11.1338   12071360   8. node_configure_fields() d7/drupal/modules/node/node.module:538

EDIT: Accessing update.php shows still 53 unfinished tasks, one of them is "7020 - Enable field module.", so the error above is quite obvious. To be exact, it seems that "7018 - Shorten the {system}.type column and modify indexes." is failing. Trying to figure out why now...

Before I reroll, here are my findings

Set access bypass to TRUE in settings.php, started up the update process from where I left off.

Required 111 update steps

it ran through

node
comment
system

picked up

user at step 80
profile at step 100
search at 104
and tracker at 111

Proceeded about another 30-45 second then fataled with

Fatal error: Call to undefined function field_attach_delete_bundle() in C:\sites\d6d7\modules\comment\comment.module on line 321

Checked the database found several content_type_"modules" all with 0 rows
There is no field_ tables.

Possibilities system did not install complete D7 schema or there is a problem with core's field module not enabled.

Any Ideas? Though system was suppose to update first but this may not be what's happening. It could be that the update processes is not installing required tables first.

Ah, found it.
The index 'system_list' seems to be added twice, once in update_fix_d7_requirements() and once in system_update_7018(). The attached patch removes the one in system_updat_7018(), not sure if that's ok or if the other one should be removed.

Berdir: Actually, I think this should now be using hook_update_dependencies()?

It looks to me like the wrong patch file was attached.

So the issue is that update.php cannot be bootstrapped multiple times, and every single operation performed in update_fix_d7_requirements() can lead to an error.

We therefore have to make sure that we can at least get behind a WSOD in case of a previous error.

Additionally, db_index_exists() doesn't seem to be used anywhere yet? How comes that the database-specific methods use an entirely different function signature? ;)

With that patch, I'm still getting to update.php?op=info with the following fatal error on Apache/2.2.11 (Win32) PHP/5.2.6:

Fatal error: Maximum execution time of 240 seconds exceeded in includes\file.inc on line 1647

Call Stack:
    0.0008     197472   1. {main}() update.php:0
  134.9436    6809888   2. drupal_bootstrap() update.php:382
  134.9443    6810784   3. _drupal_bootstrap_full() includes\bootstrap.inc:1559
  134.9834    9307640   4. module_load_all() includes\common.inc:4530
  238.3891    9656520   5. drupal_load() includes\module.inc:24
  238.3891    9656824   6. drupal_get_filename() includes\bootstrap.inc:886
  238.3917    9658512   7. drupal_system_listing() includes\bootstrap.inc:705
  238.9657    9692712   8. file_scan_directory() includes\common.inc:4742
  239.8057    9702928   9. file_scan_directory() includes\file.inc:1654
  239.9816    9705296  10. file_scan_directory() includes\file.inc:1654

That's very possibly a stupid remark, but instead of making sure update_fix_d7_requirements() is idempotent by adding lots of lots of if (!db_column_exists()), if (!db_index_exists()) code
- can't we make sure that it gets called only once in a lifetime ? a variable or something ?
- or cant we use one single tell-tale criteria to determine it has already run and should just return ?

function update_fix_d7_requirements() {
  if (db_table_exists('date_format_type')) {
    return;
  }
  
  ... do the job
}

In my last attempt on todays head
I used

Index: includes/update.inc
===================================================================
RCS file: /cvs/drupal/drupal/includes/update.inc,v
retrieving revision 1.34
diff -u -p -r1.34 update.inc
--- includes/update.inc   4 Feb 2010 17:03:19 -0000   1.34
+++ includes/update.inc   6 Feb 2010 01:54:41 -0000
@@ -483,6 +483,12 @@ function update_fix_d7_requirements() {
     db_create_table('date_formats', $schema['date_formats']);
     db_create_table('date_format_locale', $schema['date_format_locale']);
 
+    // Check for queue table, if found rename it, it has a different
+    // schema than D7 but could be useful for upgrading modules.
+    if (db_table_exists('queue')) {
+      db_rename_table('queue', 'queue_old');
+    }
+
     // Add the queue table.
     $schema['queue'] = array(
       'description' => 'Stores items in queues.',
@@ -528,6 +534,12 @@ function update_fix_d7_requirements() {
     );
     db_create_table('queue', $schema['queue']);
 
+    // Check for sequences table, if found rename it, it has a different
+    // schema than D7 but could be useful for upgrading modules.
+    if (db_table_exists('sequences')) {
+      db_rename_table('sequences', 'sequences_old');
+    }
+
     // Create the sequences table.
     $schema['sequences'] = array(
       'description' => 'Stores IDs.',
Index: modules/system/system.install
===================================================================
RCS file: /cvs/drupal/drupal/modules/system/system.install,v
retrieving revision 1.445
diff -u -p -r1.445 system.install
--- modules/system/system.install   3 Feb 2010 18:16:23 -0000   1.445
+++ modules/system/system.install   6 Feb 2010 02:30:06 -0000
@@ -1610,9 +1610,7 @@ function system_update_6052() {
  * Add a {system} index on type and name.
  */
 function system_update_6053() {
-  $ret = array();
-  db_add_index($ret, 'system', 'type_name', array(array('type', 12), 'name'));
-  return $ret;
+  // Update moved to update_fix_d7_requirements().
 }
 
 /**
@@ -2004,11 +2002,7 @@ function system_update_7017() {
  * Shorten the {system}.type column and modify indexes.
  */
 function system_update_7018() {
-  db_drop_index('system', 'modules');
-  db_drop_index('system', 'type_name');
-  db_change_field('system', 'type', 'type', array('type' => 'varchar', 'length' => 12, 'not null' => TRUE, 'default' => ''));
-  db_add_index('system', 'type_name', array('type', 'name'));
-  db_add_index('system', 'system_list', array('weight', 'name'));
+  // Update moved to update_fix_d7_requirements().
 }
 
 /**

It ran for 4 minutes then fatal error-ed with

[05-Feb-2010 21:36:59] PHP Fatal error:  Class 'views_object' not found in C:\sites\d6d7\sites\all\modules\views\includes\plugins.inc on line 327

[05-Feb-2010 21:36:59] PHP Fatal error:  Exception thrown without a stack frame in Unknown on line 0

The variable does get set when this function completes

@yched: We set a variable already, but naturally, it's only ever set if the code reaches the variable_set() line.

My guess is that the "queue" table only existed, because update.php was bootstrapped more than once.

Regarding the sequences table, my patch uses db_column_exists() to directly check for the old {sequences}.name column of D5. Only if that column exists, the table is renamed.

I grepped throughout core and db_index_exists() indeed doesn't seem to be used anywhere yet - thus, no one realized the wrong function signature.

Additionally, I commented out that ini_set('display_errors', FALSE); because I think we should be a bit more verbose - at least until we're out of alpha. Only after doing so, I was able to see the PDO exception that lead to a WSOD.

@sun: My guess is that the "queue" table only existed, because update.php was bootstrapped more than once.

Nope my D5 -> D6 database has a {queue} table

@catch: Views is a contrib module, please try the same patch without it installed. Also if you could upload your patch files as a .patch extension rather than pasting to comments that makes them easier to review and commit. Seems like it's getting close!

Ok, my problem is at work on a t1 connection I never have a problem accessing drupal.org, when I get home I have ALL kinds of issues connecting to drupal.org over a wireless dsl. have to delete cookies, try several times just to get drupal to post. I originally did have a patch attached but took me 2 hours on and off before it would post. Either get a 503 or WSOD

Reran from where I left off lastnight after pulling out all contrib modules. It fataled with

[06-Feb-2010 08:58:25] PHP Fatal error: Call to undefined function user_permissions_get_modules() in C:\sites\d6d7\modules\user\user.install on line 517

Line 517 is the only call to this function grepping "user_permissions_get_", will dig some more later

actually I made comment there and patched here. can reverse if necessary

Started update from where I left off and it completes.

There is still a problem though. on the completion page I get

* Notice: Undefined index: update_success in update_results_page() (line 159 of C:\sites\d6d7\update.php).
* Warning: reset(): Passed variable is not an array or object in update_results_page() (line 163 of C:\sites\d6d7\update.php).
* Warning: array_pop(): The argument should be an array in update_results_page() (line 163 of C:\sites\d6d7\update.php).

adding #706860: update_fix_d7_block_deltas() helper function broken for non-core modules

adding #706842: Improve comments for the taxonomy upgrade path

#709386: Schema Version reset to -1 when update.module cache is cleared

merged patches from #159 and #168. added condition around db_drop_index('system', 'type_name'); as it was giving me a WSOD at ?op=info (the d6 installation I was trying to upgrade didn't have the index type_name).

Now I'm getting Fatal error: Class 'DrupalQueue' not found on ?op=info page and my registry* tables are empty. maybe related to #708832: Fatal error: Class 'DrupalQueue' not found.

here's a patch i used to test the newly merged drush updatedb command in drush head.

drush is a lot less error tolerant about updates , so it catches more problems.

mostly it was about the order of the updates that weren't quite right :

before filter 7005 can run, you need to run user 7006 and taxonomy 7002 and system 7037
but before you can run user_7006 you need to have run filter_7002

We were also creating the system_list index in duplicate, once in update.inc and then in system 7018.

Looks good apart from the quotes around 7006. Fixed those and committed to HEAD.

Back to active.

Hm, as per my comment at http://drupal.org/node/211182#comment-2488380 I'm not sure any of the hook_update_dependencies() in core actually have the correct dependencies declared - and I'm not sure the ones in this latest patch totally make sense either. However, we left the original ones alone because (at the time) they were sufficient to allow the drupal.org database to be upgraded from Drupal 6 to 7, and the same criteria apply here - if it helps make updates possible, it's an improvement :)

However, I had meant to file a followup issue to go through and actually straighten these out for real. Just did so now: #717834: The dependencies declared in core's hook_update_dependencies() implementations aren't actually correct

Er... So this doesn't look like a meta issue at all. :) From my understanding, a meta-issue is used to track *other* issues and patches are submitted there. If we keep on addressing every single upgrade problem here, the issue is going to get horribly big and it makes it harder for new people (like me!) to get involved easily. I'll try to tackle some of those now anyways.

Basically, we don't need tracking issues anymore now that we have arbitrary tags. So, a meta issue might discuss the overall strategy for upgrading, and not the particular details. My .02

I'm okay with that, but what's happening with that is that we're processing a lot of particular details, it's exactly the problem we're having here. :)

For example, by reading through this issue quickly, I missed the discussion about the sequences table, and filed a separate bug about it: #719730: drop the sequences and queue tables from D5 during the D6 -> D7 upgrade. I hereby encourage people to take the discussion about that issue to the other place.

I will file other issues regarding the upgrade process in separate issue (if i still have time) and keep this issue to general discussions regarding overall strategy.

I had problems closely related with #325169: Move error/exception handler higher up in the bootstrap process during that last issue (blank page in update.php). I fixed this in #719850: errors in update.php during bootstrap yield a blank page, and that may overlap some issues/patches addressed here.

Basically, if you had any error in your error.log that says something about _system_rebuild_theme_data() missing, that issue will fix it.

combined patch from #719850: errors in update.php during bootstrap yield a blank page with {queue} and {sequences} table fixes. I did get the update to process to completely empty the {queue} table (284 entries) but only after changing update.php to not clear the cache.

patch attached but updating is still not completely there yet, now I get a error trying to log into the site. Tried clearing cookies and cache but didn't help.

Excellent, let's kill this issue. I have opened a new meta-discussion issue in #722074: Drupal 6 to Drupal 7 upgrade path meta discussions in case there's any followup. I made it very clear in the new issue that patches and issue-specific discussions should not take place there so I hope it doesn't degenerate the same way as here.

Any one help me to upgrade drupal 6 to drupal 7.

After execute update.php below error generate-
The following updates returned messages
node module
Update #7006

* Failed: DatabaseSchemaObjectExistsException: Table field_data_body already exists. in DatabaseSchema->createTable() (line 630 of /opt/lampp/htdocs/texenergo_demo/includes/database/schema.inc).

I have attach the screen-shot of error message please see it.
http://drupal.org/files/issues/1_1_5.png
http://drupal.org/files/issues/2_1_4.png

please help me.

Thanks & Regards
Vindesh Mohariya

@vindesh: This issue is closed. You need to open a new one or search for an issue that is still opened about the same subject.

The "update system" category no longer exists (that's probably why it got accidentally recategorized to "ajax system" when the comment was posted)...

Putting this back in the correct category for posterity :)