Cannot upgrade from Drupal 6 to Drupal 7 - meta issue
catch - August 29, 2009 - 14:26
| Project: | Drupal |
| Version: | 7.x-dev |
| Component: | update system |
| Category: | bug report |
| Priority: | critical |
| Assigned: | Unassigned |
| Status: | needs review |
| Issue tags: | D7 upgrade path |
Description
This meta-issue aims at fixing the upgrade path from Drupal 6 to Drupal 7.
-
#612920: {url_alias}.source required for upgrade
- #583008: DRUPAL_BOOTSTRAP_ACCESS doesn't exist anymore, but is still an explicit bootstrap step
- #211182: Updates run in unpredictable order
- #585628: Cannot use update.php as anonymous user
- #601642: Old databases might have a small {system}.weight
- #605374: Impossible to move modules from sites/default to sites/all.
- not strictly upgrade path but can hose a site during upgrade process
Please edit this post to add issues at the bottom of this list, and follow-up with a comment so that people can get notified.
#1
I have the same issue with a virgin installation of drupal-7.x-dev.tar.gz dated 8/29/2009
#2
confirm error on wamp new install drupal-7.x-dev.tar.gz dated 8/29/2009
shows:
* Warning: filectime(): stat failed for modules/search/search.extender.inc in _registry_parse_files() (line 127 of C:\wamp\www\drupal-7.x-dev\includes\registry.inc).
* Warning: filemtime(): stat failed for modules/search/search.extender.inc in _registry_parse_files() (line 128 of C:\wamp\www\drupal-7.x-dev\includes\registry.inc).
* Warning: file_get_contents(modules/search/search.extender.inc): failed to open stream: No such file or directory in _registry_parse_files() (line 132 of C:\wamp\www\drupal-7.x-dev\includes\registry.inc).
#3
see this cvs message by Dries: http://drupal.org/cvs?file=/modules/search/search.extender.inc
#4
add missing file, fixed
#5
Not for me, all my tests were done after that file was added.
#6
I'm sorry, I should have left it as it was. live and learn.
#7
Methinks this is the very definition of 'critical'. ;)
#8
Hmm.. Not with an upgrade, but ran into this issue trying to generate content with the Devel module and wondered if it might be related -- /batch?op=start&id=7
This may not be close enough.. I do yearn for the good old days of simple SQL errors rather than this PDO stuff which makes no sense!
An error occurred.
Path: /batch?id=7&op=do
Message: PDOException: SQLSTATE[HY000]: General error: 1366 Incorrect integer value: 'FILTER_FORMAT_DEFAULT' for column 'format' at row 1: INSERT INTO {comment} (nid, pid, uid, subject, comment, format, hostname, timestamp, status, thread, name, mail, homepage) VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1, :db_insert_placeholder_2, :db_insert_placeholder_3, :db_insert_placeholder_4, :db_insert_placeholder_5, :db_insert_placeholder_6, :db_insert_placeholder_7, :db_insert_placeholder_8, :db_insert_placeholder_9, :db_insert_placeholder_10, :db_insert_placeholder_11, :db_insert_placeholder_12); Array
(
[:db_insert_placeholder_0] => 6
[:db_insert_placeholder_1] => 0
[:db_insert_placeholder_2] => 0
[:db_insert_placeholder_3] => Verto Interdico Nostrud Jus Dignissim Lobortis Consectetuer
[:db_insert_placeholder_4] => Sagaciter accumsan valde ad. Bene cogo validus. Premo verto vereor at pecus occuro lenis haero roto. Feugiat vulpes ymo humo. Ea laoreet genitus lenis nulla causa aptent pecus.
[:db_insert_placeholder_5] => FILTER_FORMAT_DEFAULT
[:db_insert_placeholder_6] => 74.12.172.225
[:db_insert_placeholder_7] => 1254044787
[:db_insert_placeholder_8] => 1
[:db_insert_placeholder_9] => 01/
[:db_insert_placeholder_10] => devel generate
[:db_insert_placeholder_11] => devel_generate@example.com
[:db_insert_placeholder_12] =>
)
in comment_save() (line 1349 of /home/dm7/modules/comment/comment.module).
#9
Tried updating from 6.14 to 7.x
Ran into a problem with a PDO EXCEPTION "out of range error" running todays HEAD, found part of the problem in
$Id: update.inc,v 1.11 2009/09/29 15:13:54 dries Exp $
First problem is located in the function "update_fix_d7_install_profile()", PDO exception stated 'weight' out of range. Checked the system table defaults of 6.14 to 7.x, totally different.
Changed the default values on my 6.14 system table fields to match 7.x and this part went through.
The next screen came up listing 86 updates.
Click ok and then ran into issues with non-existant table exception error.
If someone can direct me to the api to change a table columns current value to a new value I'll attempt to fix it. with the new DBTNG stuff and database API, it's fairly confusing, especially when clean-ups knock out working functions and you have to look for the new method.
#10
Haven't quite figured out netbeans utility for producing a proper diff so enclosed are the changes I made to get through the first issue with the system table and PDO out of range exception. I've attached a patch (first attempt), probably isn't drupal format but i'm getting there
// $Id: update.inc,v 1.11 2009/09/29 15:13:54 dries Exp $
Line 159
function update_fix_d7_install_profile() {$profile = drupal_get_profile();
+++
// Change 6.x system table field values to 7.x equivelent
db_change_field('system', 'status', 'status',
array('type' => 'int', 'length' => 11, 'not null' => TRUE, 'default' => 0));
db_change_field('system', 'bootstrap', 'bootstrap',
array('type' => 'int', 'length' => 11, 'not null' => TRUE, 'default' => 0));
db_change_field('system', 'weight', 'weight',
array('type' => 'int', 'length' => 11, 'not null' => TRUE, 'default' => 0));
db_change_field('system', 'schema_version', 'schema_version',
array('type' => 'int',
'size' => 'small',
'length' => 6,
'not null' => TRUE,
'default' => -1));
I'm not sure if this is the proper place but it converts a 6.14 table to the 7.x equivalent.
#11
@ctmattice1: I don't see those lines anywhere in the code base.
#12
Damien
Update.php $Id: update.php,v 1.305 2009/09/28 22:16:32 dries Exp $
callls these lines around 266
// We prepare a minimal bootstrap for the update requirements check to avoid// reaching the PHP memory limit.
require_once DRUPAL_ROOT . '/includes/bootstrap.inc';
require_once DRUPAL_ROOT . '/includes/update.inc';
require_once DRUPAL_ROOT . '/includes/common.inc';
require_once DRUPAL_ROOT . '/includes/entity.inc';
update_prepare_d7_bootstrap();
the problem I was experiencing had to deal with the system tables in 6.14, the weight was valued at tinyint(2). Where I ran into problem was trying to update 6.14 to 7.x. Every time I tried it, it fatal errored with a PDO excpetion. the weight array value I could see was 1000 which would cause the out of range error. If I run a 7.x install to setup the database I don't get this with update.php
If you look in update inc for
function update_fix_d7_install_profile() {
// some stuff then you'll see the weight hard coded
// The install profile is always required.
$file->info['required'] = TRUE;
$values = array(
'filename' => $filename,
'name' => $profile,
'info' => serialize($info),
'schema_version' => 0,
'type' => 'module',
'status' => 1,
'owner' => '',
);
// Install profile hooks are always executed last by the module system
$values['weight'] = 1000;
when I added the code to update a 6.14 system table to match 7.x the problem disappeared. The error was only on the weight field but I went ahead and changed all but the type fields lenghts to match. Maybe when I upgraded from 6.12 to 6.14 something broke. But with the 6.14 weight field I had at a value of tinyint(2) the change was necessary to get update.php to the next step which is still broke with missing tables.
#13
Link #278592: Sync 6.x extra updates with HEAD
Some issues summarized #49
#14
@ctmattice1: This issue needs to remain a meta-issue, so I opened a separate bug for this: #601642: Old databases might have a small {system}.weight.
#15
Current issues:
#16
another issue for d6 and d7 #603498: Book {blocks.cache} upgrade path
#17
Tagging.
#18
Apply #612920: {url_alias}.source required for upgrade and #585628: Cannot use update.php as anonymous user I can get updates to run.
#19
All these patches conflict with each other.
Rolled #612920: {url_alias}.source required for upgrade, #585628: Cannot use update.php as anonymous user and #524710: role_permission required to upgrade from D6 to D7 into one patch here. We can't do proper comparison performance testing of Drupal 6 and 7 (let alone loads of other testing) until we can upgrade from Drupal 6. So hoping for quick commit.
Still remaining here is the out of memory error in system.module - but this at least gets to a point where the updates try to run, even if they blow up when they actually do.
#20
Remaining issues once this patch is applied, don't have time to open separate issues for each one:
An error occurred.Path: http://d7.7/update.php?id=2&op=do
Message:
Fatal error: Allowed memory size of 629145600 bytes exhausted (tried to allocate 2489268833 bytes) in /home/catch/www/7/modules/system/system.module on line 1924
This is from system_update_files_database().
#21
The last submitted patch failed testing.
#22
Now that update.php gets to the end, this is the output we actually get from it - several errors in there, probably most of these were introduced when update.php was broken, so easier to find the issue those were updates were committed and re-open I think. No time for that now, but would be great if someone could take it on.
Tagging 'novice' since this is mainly an issue queue farming task at this point, although it'll be quicker to use cvs annotate to track down where commits happened.
The following updates returned messages
system module
Update #7007
* Inserted into {role_permission} the permissions for role ID 1, Inserted into {role_permission} the permissions for role ID 2, Inserted into {role_permission} the permissions for role ID 3
Update #7013
* Failed: SQLSTATE[42S02]: Base table or view not found: 1146 Table 'd7.cache_path' doesn't exist
block module
Update #7000
* Failed: SQLSTATE[42000]: Syntax error or access violation: 1064 You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '0='weight', 1='-5' WHERE (name = 'block')' at line 1
comment module
Update #7000
* Comment order settings removed.
Update #7001
* Failed: SQLSTATE[42000]: Syntax error or access violation: 1064 You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '0='status', 1='0' WHERE (status = '3')' at line 1
dblog module
filter module
Update #7003
* Failed: SQLSTATE[42000]: Syntax error or access violation: 1064 You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '0='name', 1='filter_html' WHERE (module = 'filter') AND (name = '0')' at line 1
node module
Update #7006
* Failed: SQLSTATE[42S22]: Column not found: 1054 Unknown column 'base' in 'field list'
taxonomy module
Update #7002
* Failed: SQLSTATE[42S02]: Base table or view not found: 1146 Table 'd7.taxonomy_vocabulary' doesn't exist
update module
user module
Update #7000
* User passwords rehashed to improve security
Update #7002
* Failed: SQLSTATE[42000]: Syntax error or access violation: 1064 You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '0='timezone', 1=NULL WHERE (uid = '0')' at line 1
#23
some of the changes in this patch have been applied already. rerolling patch including the registry tables in update_fix_d7_requirements() for update.php to finish.
#24
This is an intermediate patch which fixes some of the update functions (but not all). No time to split it into the appropriate issues but hopefully it can help someone getting started with it. This patch worked for me and I was able to upgrade a few dozens of node to the D7 field API :). I haven't fixed taxonomy because it requires to be run before the node update. Make sure to remove the taxonomy record from the system table before attempting any update with this patch.
#25
scor, you're a lifesaver! Even though it requires 2 update.php runs, this managed to upgrade everything on a completely basic D6 to D7 with 15,000 nodes for me. Now I can get #563000: Remove upload module and provide upgrade path to file working. Thanks so much. Of course I'm hoping these will be fixed individually and included soon, but the patch as-is fixes tons of problems.
#26
I would like to commit this, but there are a bunch of commented-out lines. Should these be deleted, or are they just parts that haven't been fixed yet?
#27
I've spent the past couple of hours trying to get the upgrade working properly, starting with scor's patch in #24. I still can't get things working properly due to Taxonomy's updates needing to be run before Node module. Let's not discourage new contributors with calling a task like this "novice".
#28
Patch should take care of the
+ // @todo the last paramter $fields was missing. must be an array of fields IDs.+ field_sql_storage_field_storage_write('node', $node, FIELD_STORAGE_INSERT, array(1, 2));
My bad from #443422: 'per field' storage engine, I forgot we called that function directly in the 'body as field' update, to let the update run a tad faster.
#29
Applied the patch in 24 #563106-24: Cannot upgrade from Drupal 6 to Drupal 7 - meta issue to todays HEAD. Ran into same error #601642: Old databases might have a small {system}.weight
Changed includes/updates.inc from
if (drupal_get_installed_schema_version('system') < 7000 && !variable_get('update_d7_requirements', FALSE)) {
// Add the cache_path table.
$schema['cache_path'] = drupal_get_schema_unprocessed('system', 'cache');
$schema['cache_path']['description'] = 'Cache table used for path alias lookups.';
db_create_table('cache_path', $schema['cache_path']);
// system_update_7042() renames columns, but these are needed to bootstrap.
// Add empty columns for now.
to this
if (drupal_get_installed_schema_version('system') < 7000 && !variable_get('update_d7_requirements', FALSE)) {
// Change database system tables to 7.x equivelent structure
// otherwise update may fail
db_change_field('system', 'status', 'status',
array('type' => 'int', 'length' => 11, 'not null' => TRUE, 'default' => 0));
db_change_field('system', 'bootstrap', 'bootstrap',
array('type' => 'int', 'length' => 11, 'not null' => TRUE, 'default' => 0));
db_change_field('system', 'weight', 'weight',
array('type' => 'int', 'length' => 11, 'not null' => TRUE, 'default' => 0));
db_change_field('system', 'schema_version', 'schema_version',
array('type' => 'int',
'size' => 'small',
'length' => 6,
'not null' => TRUE,
'default' => -1));
// Add the cache_path table.
$schema['cache_path'] = drupal_get_schema_unprocessed('system', 'cache');
$schema['cache_path']['description'] = 'Cache table used for path alias lookups.';
db_create_table('cache_path', $schema['cache_path']);
// system_update_7042() renames columns, but these are needed to bootstrap.
// Add empty columns for now.
This effectively eliminates the PDO exception with weight errors that php was reporting
[08-Nov-2009 19:47:56] PHP Fatal error: Uncaught exception 'PDOException' with message 'SQLSTATE[01000]: Warning: 1264 Out of range value for column 'weight' at row 1' in C:\sites\d6d7\includes\database\database.inc:1786
Stack trace:
#0 C:\sites\d6d7\includes\database\database.inc(1786): PDOStatement->execute(Array)
#1 C:\sites\d6d7\includes\database\database.inc(568): DatabaseStatementBase->execute(Array, Array)
#2 C:\sites\d6d7\includes\database\mysql\query.inc(37): DatabaseConnection->query('INSERT INTO {s...', Array, Array)
#3 C:\sites\d6d7\includes\update.inc(423): InsertQuery_mysql->execute()
#4 C:\sites\d6d7\includes\update.inc(363): update_fix_d7_install_profile()
#5 C:\sites\d6d7\update.php(342): update_fix_d7_requirements()
#6 {main}
thrown in C:\sites\d6d7\includes\database\database.inc on line 1786
Now though I'm getting
[08-Nov-2009 19:56:40] PHP Fatal error: Exception thrown without a stack frame in Unknown on line 0
I've looked at the system tables and last schema_version is 6053.
Tracing the next call is to update_fix_d7_install_profile() which writes the row to the system table the tries to load the schema
Now this is interesting as I have 2 listings as profile being a module, this one earlier in the system table
modules/profile/profile.module profile module 1 0 0 0 0 a:10:{s:4:"name";s:7:"Profile";s:11:"description";... Supports configurable user profiles.
and the one added by update_fix_d7_install_profile() which is the last table entry
profiles/default/default.profile default module 1 0 0 1000 0 a:11:{s:4:"name";s:6:"Drupal";s:11:"description";s...
Both are showing a status of 1, could this be the cause of the latest error
[08-Nov-2009 19:56:40] PHP Fatal error: Exception thrown without a stack frame in Unknown on line 0
Even if not, I don't believe both should be present, based on the function drupal_get_installed_schema_version() I can see where bad data could get passes.
#30
Here's an improved patch that makes it so that Node module can run its updates without tripping over Taxonomy. I'm not sure if it's a good approach, but I got around Taxonomy module's requirements by setting the weight of Node module to "1" in update_fix_d7_requirements(), then setting it back to "0" in node_update_7000(). It's a bit hacky, but it definitely allows node.module to run last after other modules have gotten their Field system adjustments in place.
There's still a couple of TODOs that need to be cleaned up, so I'm moving this to needs work.
#31
Includes ycheds fix from #28.
#32
This patch is fully working, with all TODOs removed. In my testing Upload.module does not upgrade itself properly, but I'd like to just say "screw upload.module" and fix that problem by removing it entirely in #563000: Remove upload module and provide upgrade path to file, which I also have working now when applied in combination with this patch.
#33
subscribe
#34
Just tried with 'clean' Drupal 6.14 install, and somehow this patch requires the upload module to be removed otherwise the ordering of the modules is messed up and node gets executed before taxonomy. Although the weights are applied correctly in the db, drupal_get_installed_schema_version() returns a different ordering depending on whether upload module is present in the code base or not. I find this odd. Anyone care to enlighten me? that makes this patch dependant on #563000: Remove upload module and provide upgrade path to file.
@quicksketch, could you repost your instuctions from #563000-41: Remove upload module and provide upgrade path to file with whether the order of the update is correct or not please (in the collapsed fieldset at the selection step)? I tried with an old Drupal 6 site and it broke everything. I suspect there are other modules causing the same type of problems as upload.
#35
+++ includes/update.inc 9 Nov 2009 04:26:20 -0000@@ -291,6 +331,14 @@
+ // Set Node module's weight to 1 so that runs after other modules have
+ // properly prepared their Field APIs. Node module sets its weight back
+ // to 0 in node_update_7000().
+ db_update('system')
+ ->fields(array('weight' => 1))
+ ->condition('name', 'node')
+ ->execute();
This is the job of #211182: Updates run in unpredictable order. Please don't introduce more hacks then strictly necessary.
I'm on crack. Are you, too?
#36
Oh dear, I thought modules would at least be updated according to their weight, but it looks like that is not the case. Are we really going to be trapped without an upgrade path until #211182: Updates run in unpredictable order is finished? Perhaps we can implement a basic "updates follow their module weight" for starters so we can get our basic upgrade working. It looks like the circular dependency checking in #211182 is not exactly ready.
#37
That would not really make sense. #211182: Updates run in unpredictable order is basically ready, and the circular dependency checking is not necessary at all.
#38
Well... regardless here's a patch that now makes our upgrade path work by adding an ORDER BY clause to our retrieval of upgrade modules. This should now work except for the error caused by Upload module, but that won't be a problem if you've never enabled the upload module in you sandbox sites. It'd be great if #211182: Updates run in unpredictable order gets done, but at least this will make updates run like everything else in Drupal, by weight then by name.
#39
rerolled #38 with some more fixes for filter_update_7004() and taxonomy_update_7004().
#40
I would absolutely love to commit this patch!
However.
That temporary re-weighting of node module in update_fix_d7_requirements() is a no-go. That function runs by just *touching* update.php, and needs to be non-destructive. Someone should be able to back out D7 files and go back to their D6 site, perhaps with a few extra tables/indexes, but not with a core required module's weight suddenly shifted around, which I'm quite sure would break tons of contrib modules that make the assumption (reasonable or no) that node.module's weight is always 0.
#41
So after a short chat with Webchick and finding that I can think of no solution that doesn't require drastic changes, I'm moving over to #211182: Updates run in unpredictable order in hopes I can get that patch working in tandem with this one (and the upload one) so that we can eventually run an upgrade. This version of the patch is the same as #39 only it doesn't contain the order-by-weight change or the node-weight change. So basically this patch is nothing but upgrade fixes. The hope is that we can commit this patch to *almost* get upgrades working (it's certainly better than what we have now), then have the patch in #211182: Updates run in unpredictable order actually make upgrades works when applied.
#42
Another forgotten table "semaphore"
#43
It's not forgotten, it's in update_fix_d7_requirements() in HEAD already.
#44
#627338: Add a cache_bootstrap bin removed the cache_registry table so the system_update_7006() hunk fails. rerolled patch.
#45
Just want to let you guys know that http://drupal.org/project/demo_profile fully works on D7 now:
#628374: Drupal 7 core install profile AWESOMENESSAUCEWITHEXTRACHEESE!
Not sure whether it helps to test the D6->D7 upgrade path, but you may find out. It definitely helps to test regular updates.