Posted by catch on August 29, 2009 at 2:26pm
| Project: | Drupal core |
| Version: | 7.x-dev |
| Component: | database update system |
| Category: | bug report |
| Priority: | critical |
| Assigned: | Unassigned |
| Status: | closed (duplicate) |
| Issue tags: | beta blocker, D7 upgrade path, webchick's D7 alpha hit list |
Issue Summary
This meta-issue aims at fixing the upgrade path from Drupal 6 to Drupal 7.
- #612920: {url_alias}.source required for upgrade
- #583008: IP address blocking does not work on update.php
- #211182: Updates run in unpredictable order
- #585628: Cannot use update.php as anonymous user
- #601642: Old databases might have a small {system}.weight
- #605374: Dont add nonexisting files to the registry - not strictly upgrade path but can hose a site during upgrade process
- #606730: provide a helper function for block delta updates
- #684084: node_update_7006() broken after 'node titles as fields' rollback
- #685416: filter table renames aren't recognized by a later update
- #685486: filter_update_7003() doesn't work for contrib modules
- #685572: system_update_7008() is broken
- #685892: Upload -> File module upgrade path is broken
- #686128: Body-to-field conversion is too slow
- #615634: Update CHANGELOG.txt to explicitly name new core modules and core modules that have been removed. - not strictly upgrade path, but is holding up upgrade documentation
- #706860: update_fix_d7_block_deltas() helper function broken for non-core modules
Please edit this post to add issues at the bottom of this list, and follow-up with a comment so that people can get notified.
Comments
#1
I have the same issue with a virgin installation of drupal-7.x-dev.tar.gz dated 8/29/2009
#2
confirm error on wamp new install drupal-7.x-dev.tar.gz dated 8/29/2009
shows:
* Warning: filectime(): stat failed for modules/search/search.extender.inc in _registry_parse_files() (line 127 of C:\wamp\www\drupal-7.x-dev\includes\registry.inc).
* Warning: filemtime(): stat failed for modules/search/search.extender.inc in _registry_parse_files() (line 128 of C:\wamp\www\drupal-7.x-dev\includes\registry.inc).
* Warning: file_get_contents(modules/search/search.extender.inc): failed to open stream: No such file or directory in _registry_parse_files() (line 132 of C:\wamp\www\drupal-7.x-dev\includes\registry.inc).
#3
see this cvs message by Dries: http://drupal.org/cvs?file=/modules/search/search.extender.inc
#4
add missing file, fixed
#5
Not for me, all my tests were done after that file was added.
#6
I'm sorry, I should have left it as it was. live and learn.
#7
Methinks this is the very definition of 'critical'. ;)
#8
Hmm.. Not with an upgrade, but ran into this issue trying to generate content with the Devel module and wondered if it might be related -- /batch?op=start&id=7
This may not be close enough.. I do yearn for the good old days of simple SQL errors rather than this PDO stuff which makes no sense!
An error occurred.
Path: /batch?id=7&op=do
Message: PDOException: SQLSTATE[HY000]: General error: 1366 Incorrect integer value: 'FILTER_FORMAT_DEFAULT' for column 'format' at row 1: INSERT INTO {comment} (nid, pid, uid, subject, comment, format, hostname, timestamp, status, thread, name, mail, homepage) VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1, :db_insert_placeholder_2, :db_insert_placeholder_3, :db_insert_placeholder_4, :db_insert_placeholder_5, :db_insert_placeholder_6, :db_insert_placeholder_7, :db_insert_placeholder_8, :db_insert_placeholder_9, :db_insert_placeholder_10, :db_insert_placeholder_11, :db_insert_placeholder_12); Array
(
[:db_insert_placeholder_0] => 6
[:db_insert_placeholder_1] => 0
[:db_insert_placeholder_2] => 0
[:db_insert_placeholder_3] => Verto Interdico Nostrud Jus Dignissim Lobortis Consectetuer
[:db_insert_placeholder_4] => Sagaciter accumsan valde ad. Bene cogo validus. Premo verto vereor at pecus occuro lenis haero roto. Feugiat vulpes ymo humo. Ea laoreet genitus lenis nulla causa aptent pecus.
[:db_insert_placeholder_5] => FILTER_FORMAT_DEFAULT
[:db_insert_placeholder_6] => 74.12.172.225
[:db_insert_placeholder_7] => 1254044787
[:db_insert_placeholder_8] => 1
[:db_insert_placeholder_9] => 01/
[:db_insert_placeholder_10] => devel generate
[:db_insert_placeholder_11] => devel_generate@example.com
[:db_insert_placeholder_12] =>
)
in comment_save() (line 1349 of /home/dm7/modules/comment/comment.module).
#9
Tried updating from 6.14 to 7.x
Ran into a problem with a PDO EXCEPTION "out of range error" running todays HEAD, found part of the problem in
$Id: update.inc,v 1.11 2009/09/29 15:13:54 dries Exp $
First problem is located in the function "update_fix_d7_install_profile()", PDO exception stated 'weight' out of range. Checked the system table defaults of 6.14 to 7.x, totally different.
Changed the default values on my 6.14 system table fields to match 7.x and this part went through.
The next screen came up listing 86 updates.
Click ok and then ran into issues with non-existant table exception error.
If someone can direct me to the api to change a table columns current value to a new value I'll attempt to fix it. with the new DBTNG stuff and database API, it's fairly confusing, especially when clean-ups knock out working functions and you have to look for the new method.
#10
Haven't quite figured out netbeans utility for producing a proper diff so enclosed are the changes I made to get through the first issue with the system table and PDO out of range exception. I've attached a patch (first attempt), probably isn't drupal format but i'm getting there
// $Id: update.inc,v 1.11 2009/09/29 15:13:54 dries Exp $
Line 159
function update_fix_d7_install_profile() {$profile = drupal_get_profile();
+++
// Change 6.x system table field values to 7.x equivelent
db_change_field('system', 'status', 'status',
array('type' => 'int', 'length' => 11, 'not null' => TRUE, 'default' => 0));
db_change_field('system', 'bootstrap', 'bootstrap',
array('type' => 'int', 'length' => 11, 'not null' => TRUE, 'default' => 0));
db_change_field('system', 'weight', 'weight',
array('type' => 'int', 'length' => 11, 'not null' => TRUE, 'default' => 0));
db_change_field('system', 'schema_version', 'schema_version',
array('type' => 'int',
'size' => 'small',
'length' => 6,
'not null' => TRUE,
'default' => -1));
I'm not sure if this is the proper place but it converts a 6.14 table to the 7.x equivalent.
#11
@ctmattice1: I don't see those lines anywhere in the code base.
#12
Damien
Update.php $Id: update.php,v 1.305 2009/09/28 22:16:32 dries Exp $
callls these lines around 266
// We prepare a minimal bootstrap for the update requirements check to avoid// reaching the PHP memory limit.
require_once DRUPAL_ROOT . '/includes/bootstrap.inc';
require_once DRUPAL_ROOT . '/includes/update.inc';
require_once DRUPAL_ROOT . '/includes/common.inc';
require_once DRUPAL_ROOT . '/includes/entity.inc';
update_prepare_d7_bootstrap();
the problem I was experiencing had to deal with the system tables in 6.14, the weight was valued at tinyint(2). Where I ran into problem was trying to update 6.14 to 7.x. Every time I tried it, it fatal errored with a PDO excpetion. the weight array value I could see was 1000 which would cause the out of range error. If I run a 7.x install to setup the database I don't get this with update.php
If you look in update inc for
function update_fix_d7_install_profile() {
// some stuff then you'll see the weight hard coded
// The install profile is always required.
$file->info['required'] = TRUE;
$values = array(
'filename' => $filename,
'name' => $profile,
'info' => serialize($info),
'schema_version' => 0,
'type' => 'module',
'status' => 1,
'owner' => '',
);
// Install profile hooks are always executed last by the module system
$values['weight'] = 1000;
when I added the code to update a 6.14 system table to match 7.x the problem disappeared. The error was only on the weight field but I went ahead and changed all but the type fields lenghts to match. Maybe when I upgraded from 6.12 to 6.14 something broke. But with the 6.14 weight field I had at a value of tinyint(2) the change was necessary to get update.php to the next step which is still broke with missing tables.
#13
Link #278592: Sync 6.x extra updates with HEAD
Some issues summarized #49
#14
@ctmattice1: This issue needs to remain a meta-issue, so I opened a separate bug for this: #601642: Old databases might have a small {system}.weight.
#15
Current issues:
#16
another issue for d6 and d7 #603498: Book {blocks.cache} upgrade path
#17
Tagging.
#18
Apply #612920: {url_alias}.source required for upgrade and #585628: Cannot use update.php as anonymous user I can get updates to run.
#19
All these patches conflict with each other.
Rolled #612920: {url_alias}.source required for upgrade, #585628: Cannot use update.php as anonymous user and #524710: role_permission required to upgrade from D6 to D7 into one patch here. We can't do proper comparison performance testing of Drupal 6 and 7 (let alone loads of other testing) until we can upgrade from Drupal 6. So hoping for quick commit.
Still remaining here is the out of memory error in system.module - but this at least gets to a point where the updates try to run, even if they blow up when they actually do.
#20
Remaining issues once this patch is applied, don't have time to open separate issues for each one:
An error occurred.Path: http://d7.7/update.php?id=2&op=do
Message:
Fatal error: Allowed memory size of 629145600 bytes exhausted (tried to allocate 2489268833 bytes) in /home/catch/www/7/modules/system/system.module on line 1924
This is from system_update_files_database().
#21
The last submitted patch failed testing.
#22
Now that update.php gets to the end, this is the output we actually get from it - several errors in there, probably most of these were introduced when update.php was broken, so easier to find the issue those were updates were committed and re-open I think. No time for that now, but would be great if someone could take it on.
Tagging 'novice' since this is mainly an issue queue farming task at this point, although it'll be quicker to use cvs annotate to track down where commits happened.
The following updates returned messages
system module
Update #7007
* Inserted into {role_permission} the permissions for role ID 1, Inserted into {role_permission} the permissions for role ID 2, Inserted into {role_permission} the permissions for role ID 3
Update #7013
* Failed: SQLSTATE[42S02]: Base table or view not found: 1146 Table 'd7.cache_path' doesn't exist
block module
Update #7000
* Failed: SQLSTATE[42000]: Syntax error or access violation: 1064 You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '0='weight', 1='-5' WHERE (name = 'block')' at line 1
comment module
Update #7000
* Comment order settings removed.
Update #7001
* Failed: SQLSTATE[42000]: Syntax error or access violation: 1064 You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '0='status', 1='0' WHERE (status = '3')' at line 1
dblog module
filter module
Update #7003
* Failed: SQLSTATE[42000]: Syntax error or access violation: 1064 You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '0='name', 1='filter_html' WHERE (module = 'filter') AND (name = '0')' at line 1
node module
Update #7006
* Failed: SQLSTATE[42S22]: Column not found: 1054 Unknown column 'base' in 'field list'
taxonomy module
Update #7002
* Failed: SQLSTATE[42S02]: Base table or view not found: 1146 Table 'd7.taxonomy_vocabulary' doesn't exist
update module
user module
Update #7000
* User passwords rehashed to improve security
Update #7002
* Failed: SQLSTATE[42000]: Syntax error or access violation: 1064 You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '0='timezone', 1=NULL WHERE (uid = '0')' at line 1
#23
some of the changes in this patch have been applied already. rerolling patch including the registry tables in update_fix_d7_requirements() for update.php to finish.
#24
This is an intermediate patch which fixes some of the update functions (but not all). No time to split it into the appropriate issues but hopefully it can help someone getting started with it. This patch worked for me and I was able to upgrade a few dozens of node to the D7 field API :). I haven't fixed taxonomy because it requires to be run before the node update. Make sure to remove the taxonomy record from the system table before attempting any update with this patch.
#25
scor, you're a lifesaver! Even though it requires 2 update.php runs, this managed to upgrade everything on a completely basic D6 to D7 with 15,000 nodes for me. Now I can get #563000: Provide upgrade path to file working. Thanks so much. Of course I'm hoping these will be fixed individually and included soon, but the patch as-is fixes tons of problems.
#26
I would like to commit this, but there are a bunch of commented-out lines. Should these be deleted, or are they just parts that haven't been fixed yet?
#27
I've spent the past couple of hours trying to get the upgrade working properly, starting with scor's patch in #24. I still can't get things working properly due to Taxonomy's updates needing to be run before Node module. Let's not discourage new contributors with calling a task like this "novice".
#28
Patch should take care of the
+ // @todo the last paramter $fields was missing. must be an array of fields IDs.+ field_sql_storage_field_storage_write('node', $node, FIELD_STORAGE_INSERT, array(1, 2));
My bad from #443422: 'per field' storage engine, I forgot we called that function directly in the 'body as field' update, to let the update run a tad faster.
#29
Applied the patch in 24 #563106-24: Cannot upgrade from Drupal 6 to Drupal 7 - meta issue to todays HEAD. Ran into same error #601642: Old databases might have a small {system}.weight
Changed includes/updates.inc from
if (drupal_get_installed_schema_version('system') < 7000 && !variable_get('update_d7_requirements', FALSE)) {
// Add the cache_path table.
$schema['cache_path'] = drupal_get_schema_unprocessed('system', 'cache');
$schema['cache_path']['description'] = 'Cache table used for path alias lookups.';
db_create_table('cache_path', $schema['cache_path']);
// system_update_7042() renames columns, but these are needed to bootstrap.
// Add empty columns for now.
to this
if (drupal_get_installed_schema_version('system') < 7000 && !variable_get('update_d7_requirements', FALSE)) {
// Change database system tables to 7.x equivelent structure
// otherwise update may fail
db_change_field('system', 'status', 'status',
array('type' => 'int', 'length' => 11, 'not null' => TRUE, 'default' => 0));
db_change_field('system', 'bootstrap', 'bootstrap',
array('type' => 'int', 'length' => 11, 'not null' => TRUE, 'default' => 0));
db_change_field('system', 'weight', 'weight',
array('type' => 'int', 'length' => 11, 'not null' => TRUE, 'default' => 0));
db_change_field('system', 'schema_version', 'schema_version',
array('type' => 'int',
'size' => 'small',
'length' => 6,
'not null' => TRUE,
'default' => -1));
// Add the cache_path table.
$schema['cache_path'] = drupal_get_schema_unprocessed('system', 'cache');
$schema['cache_path']['description'] = 'Cache table used for path alias lookups.';
db_create_table('cache_path', $schema['cache_path']);
// system_update_7042() renames columns, but these are needed to bootstrap.
// Add empty columns for now.
This effectively eliminates the PDO exception with weight errors that php was reporting
[08-Nov-2009 19:47:56] PHP Fatal error: Uncaught exception 'PDOException' with message 'SQLSTATE[01000]: Warning: 1264 Out of range value for column 'weight' at row 1' in C:\sites\d6d7\includes\database\database.inc:1786
Stack trace:
#0 C:\sites\d6d7\includes\database\database.inc(1786): PDOStatement->execute(Array)
#1 C:\sites\d6d7\includes\database\database.inc(568): DatabaseStatementBase->execute(Array, Array)
#2 C:\sites\d6d7\includes\database\mysql\query.inc(37): DatabaseConnection->query('INSERT INTO {s...', Array, Array)
#3 C:\sites\d6d7\includes\update.inc(423): InsertQuery_mysql->execute()
#4 C:\sites\d6d7\includes\update.inc(363): update_fix_d7_install_profile()
#5 C:\sites\d6d7\update.php(342): update_fix_d7_requirements()
#6 {main}
thrown in C:\sites\d6d7\includes\database\database.inc on line 1786
Now though I'm getting
[08-Nov-2009 19:56:40] PHP Fatal error: Exception thrown without a stack frame in Unknown on line 0
I've looked at the system tables and last schema_version is 6053.
Tracing the next call is to update_fix_d7_install_profile() which writes the row to the system table the tries to load the schema
Now this is interesting as I have 2 listings as profile being a module, this one earlier in the system table
modules/profile/profile.module profile module 1 0 0 0 0 a:10:{s:4:"name";s:7:"Profile";s:11:"description";... Supports configurable user profiles.
and the one added by update_fix_d7_install_profile() which is the last table entry
profiles/default/default.profile default module 1 0 0 1000 0 a:11:{s:4:"name";s:6:"Drupal";s:11:"description";s...
Both are showing a status of 1, could this be the cause of the latest error
[08-Nov-2009 19:56:40] PHP Fatal error: Exception thrown without a stack frame in Unknown on line 0
Even if not, I don't believe both should be present, based on the function drupal_get_installed_schema_version() I can see where bad data could get passes.
#30
Here's an improved patch that makes it so that Node module can run its updates without tripping over Taxonomy. I'm not sure if it's a good approach, but I got around Taxonomy module's requirements by setting the weight of Node module to "1" in update_fix_d7_requirements(), then setting it back to "0" in node_update_7000(). It's a bit hacky, but it definitely allows node.module to run last after other modules have gotten their Field system adjustments in place.
There's still a couple of TODOs that need to be cleaned up, so I'm moving this to needs work.
#31
Includes ycheds fix from #28.
#32
This patch is fully working, with all TODOs removed. In my testing Upload.module does not upgrade itself properly, but I'd like to just say "screw upload.module" and fix that problem by removing it entirely in #563000: Provide upgrade path to file, which I also have working now when applied in combination with this patch.
#33
subscribe
#34
Just tried with 'clean' Drupal 6.14 install, and somehow this patch requires the upload module to be removed otherwise the ordering of the modules is messed up and node gets executed before taxonomy. Although the weights are applied correctly in the db, drupal_get_installed_schema_version() returns a different ordering depending on whether upload module is present in the code base or not. I find this odd. Anyone care to enlighten me? that makes this patch dependant on #563000: Provide upgrade path to file.
@quicksketch, could you repost your instuctions from #563000-41: Provide upgrade path to file with whether the order of the update is correct or not please (in the collapsed fieldset at the selection step)? I tried with an old Drupal 6 site and it broke everything. I suspect there are other modules causing the same type of problems as upload.
#35
+++ includes/update.inc 9 Nov 2009 04:26:20 -0000@@ -291,6 +331,14 @@
+ // Set Node module's weight to 1 so that runs after other modules have
+ // properly prepared their Field APIs. Node module sets its weight back
+ // to 0 in node_update_7000().
+ db_update('system')
+ ->fields(array('weight' => 1))
+ ->condition('name', 'node')
+ ->execute();
This is the job of #211182: Updates run in unpredictable order. Please don't introduce more hacks then strictly necessary.
I'm on crack. Are you, too?
#36
Oh dear, I thought modules would at least be updated according to their weight, but it looks like that is not the case. Are we really going to be trapped without an upgrade path until #211182: Updates run in unpredictable order is finished? Perhaps we can implement a basic "updates follow their module weight" for starters so we can get our basic upgrade working. It looks like the circular dependency checking in #211182 is not exactly ready.
#37
That would not really make sense. #211182: Updates run in unpredictable order is basically ready, and the circular dependency checking is not necessary at all.
#38
Well... regardless here's a patch that now makes our upgrade path work by adding an ORDER BY clause to our retrieval of upgrade modules. This should now work except for the error caused by Upload module, but that won't be a problem if you've never enabled the upload module in you sandbox sites. It'd be great if #211182: Updates run in unpredictable order gets done, but at least this will make updates run like everything else in Drupal, by weight then by name.
#39
rerolled #38 with some more fixes for filter_update_7004() and taxonomy_update_7004().
#40
I would absolutely love to commit this patch!
However.
That temporary re-weighting of node module in update_fix_d7_requirements() is a no-go. That function runs by just *touching* update.php, and needs to be non-destructive. Someone should be able to back out D7 files and go back to their D6 site, perhaps with a few extra tables/indexes, but not with a core required module's weight suddenly shifted around, which I'm quite sure would break tons of contrib modules that make the assumption (reasonable or no) that node.module's weight is always 0.
#41
So after a short chat with Webchick and finding that I can think of no solution that doesn't require drastic changes, I'm moving over to #211182: Updates run in unpredictable order in hopes I can get that patch working in tandem with this one (and the upload one) so that we can eventually run an upgrade. This version of the patch is the same as #39 only it doesn't contain the order-by-weight change or the node-weight change. So basically this patch is nothing but upgrade fixes. The hope is that we can commit this patch to *almost* get upgrades working (it's certainly better than what we have now), then have the patch in #211182: Updates run in unpredictable order actually make upgrades works when applied.
#42
Another forgotten table "semaphore"
#43
It's not forgotten, it's in update_fix_d7_requirements() in HEAD already.
#44
#627338: Add a cache_bootstrap bin removed the cache_registry table so the system_update_7006() hunk fails. rerolled patch.
#45
Just want to let you guys know that http://drupal.org/project/demo_profile fully works on D7 now:
#628374: Drupal 7 core install profile AWESOMENESSAUCEWITHEXTRACHEESE!
Not sure whether it helps to test the D6->D7 upgrade path, but you may find out. It definitely helps to test regular updates.
#46
add beta blocker tag for the patch here.
#47
This is the patch I used to test the upgrade drush updatedb code
http://drupal.org/files/issues/drupal_7_update_fix.diff
#48
The list of issues are :
1) typo in node.install :
db_upddate to db_update
2) incorrect usage of UpdateQuery fields method
fields(status, 1) to fields(array(status, 1))
3) missing db_change_field
was incorrectly removed in http://drupalcode.org/viewvc/drupal/drupal/modules/node/node.install?r1=...
the 'module' column was not being renamed to the base column
4) a bunch of incorrect modifications of a non-existent 'user' table in user.install
it is now, and has always been 'users'.
5) taxonomy_update_7002 needed to be run before node_update_7006
i fixed this by adding a taxonomy_update_6002_run variable, and calling the update
function directly from node_update_7006.
#49
Looks good, except from the taxonomy_update_7002() workaround. There's a separate issue about the execution order of updates in the queue.
#50
@sun do you know that other issue?
I found an issue with comment module update. The error was "Call to undefined function field_attach_create_bundle()".
field is a new and required module. Is it enabled out of the box?
#51
@mfer I suppose it's #211182: Updates run in unpredictable order
#52
Subscribe
#53
@adrian, all the issues you mention in #48 are already fixed in the patch #44.
rerolling patch after some hunks have already been committed in separate issues. upload.module does not exist anymore. We're still dependent on #211182: Updates run in unpredictable order though.
#54
+++ includes/session.inc@@ -242,6 +242,11 @@ function drupal_session_commit() {
+ // Expire the keep if possible.
+ if (isset($_SESSION['drupal_keep']) && ($_SESSION['drupal_keep'] > REQUEST_TIME)) {
+ unset($_SESSION['drupal_keep']);
+ }
1) The inline comment is a bit sparse.
2) Additional parenthesis not required in the condition.
3) Should call the reset function.
+++ includes/session.inc@@ -271,6 +276,25 @@ function drupal_session_started($set = NULL) {
+ /**
+ * Keep the current user's session open for a given length of time.
Wrong PHPDoc indentation.
Could also use a separate, more lengthy PHPDoc description about what this function is for, where the defined value is going to be checked, and when it is intended to be used - and when NOT.
A @see to the reset function wouldn't hurt either.
+++ includes/session.inc@@ -271,6 +276,25 @@ function drupal_session_started($set = NULL) {
+ * @param $duration The length the session needs to remain open, in seconds.
Wrong @param syntax.
+++ includes/session.inc@@ -271,6 +276,25 @@ function drupal_session_started($set = NULL) {
+function drupal_session_keep($duration) {
+ $new_timestamp = REQUEST_TIME + $duration;
+ if (!isset($_SESSION['drupal_keep']) || ($new_timestamp > $_SESSION['drupal_keep'])) {
+ $_SESSION['drupal_keep'] = $new_timestamp;
+ }
+}
I don't see how the second condition could ever be not met. Which in turn means that the entire condition is superfluous.
+++ update.php@@ -395,6 +395,9 @@ else {
+ // Force the session to stay open for 15 minutes.
+ drupal_session_keep(300);
Why not
drupal_session_ttl()
?
And why 15 minutes? The inline comment should explain.
I'm on crack. Are you, too?
#55
@adrian: I think that the "Updates run in unpredictable order" patch might be able to take care of #48e. If it gets any love...
#57
Adding #450540: Upgrade instructions recommend enabling stark in D6 to this meta issue to track.
#58
reply to @sun
am i the only person who feels it is patently ridiculous to hold up a MAJORLY CRITICAL piece of functionality like this on something as silly as a 'boil the ocean' acyclic directed graph approach to granular hook weighting?
we're holding up the 4 line solution to the one occurrence of a problem we have solved in similar ways before on the possibility that we can develop a complex replacement algorithm, less than a week before we are meant to release the alpha ?
we should not be adding new hooks at this point.
#59
In case you forgot: Upgrading from D5 to D6 required users to perform very tricky preparations and contrib to invent some nasty hacks to get the upgrade path only remotely right. Upgrading to D7 will have a lot more problems like that, especially when considering the module namespace clashes we purposively introduced as well as renamed modules and the entire CCK upgrade path.
#60
@adrian #58
These 4 lines of code are only for taxonomy / core. We must supply contrib modules a clear solution on their #211182: Updates run in unpredictable order right?
#61
@clemens, at the moment, contrib module authors can't even test the most basic of upgrade paths because core's throws errors all over the place. I don't see any harm fixing this critical bug using the existing, albeit a bit crappy, API. Then fix the API in it's own issue, which is also marked as a critical bug, so shouldn't slip through the cracks unless it's reclassified.
For now, let's be honest with the status here, nothing happening until either the update order bug gets fixed, or it ceases to be a blocker here.
#62
#63
I think I'm going to give the dependent issue until Thursday the 14th to get fixed, and if it isn't, we're going to go with Adrian's solution, and document it as a workaround. I agree with him that this is ridiculous.
#64
Yeah, screw holding back good solutions for perfect solutions.
Back to needs review to see what the bot thinks.
#65
Re-test of drupal7_update_work_563106_53.patch from comment #53 was requested by webchick.
#66
Yes, let's ship with the crappy solution. But someone needs to actually roll a patch for this (mixing #47 and #53), and taking into account the review in #54.
This is utterly ridiculous, but I'm tired fighting.
#67
Here is re-roll with changes:
1) I see no reason in drupal_session_keep()
2) taxonomy_update_7004() adds taxonomy_term_reference not taxonomy_term
3) sequences table used by batch so moved to reqiurements
4) node_update_7006 set $info->old_type = $info->type; else no content types are available after update
5) filter_update_7004() tabs replaced by spaces
Batch totally broken after #629794: Scaling issues with batch API
#68
The last submitted patch, 89349_update_fix.diff, failed testing.
#69
subscribing
#70
Batch API is fixed again. My apologies. :(
#71
Another round, now update process completes with only one warning
comment.installUpdate #7012
* Failed: SQLSTATE[42S22]: Column not found: 1054 Unknown column 'machine_name' in 'field list'
But after another run update.php it works
Still problem with viewing node body
#72
Subscribed.
#73
Node body was broken because my D6 setup was multilingual and field_multilingual_valid_language() returns only one (default) language for body field... but this for another issue.
comment_update_7005() seems totally useless because:
1) runs before node updates
2) does nothing because original comment_node_type_insert() still holds @todo and uses _comment_body_field_instance_create()
so creating comment fields and bundles in comment_update_7005() should run after node_update_7006()
Suppose we should commit this changes because mostly there's typos and lost code pieces and continue after #211182: Updates run in unpredictable order or after inventing ordered updates :(
#74
I've committed #73 as another interim step on this issue. Also, optimistically tagging for alpha. It would be so amazing to have a working upgrade path for the first alpha! And I really appreciate the work you all have been doing on the alpha blocker patches.
Back to needs work for the next follow-up.
#75
i'm adding #606730: provide a helper function for block delta updates to this list -- it's a major upgrade PITA for module developers who use blocks, and it's tested and ready to go.
#76
Note that I have posted an updated patch #211182: Updates run in unpredictable order which I believe provides a working upgrade path. (I've only lightly tested it, but I can at least successfully update an empty D6 site to D7 without generating any update failures.)
#77
adding #684084: node_update_7006() broken after 'node titles as fields' rollback to the list.
#78
Just to keep this issue updated, I just did a bunch of hammering on the upgrade path over here http://drupal.org/node/211182#comment-2474322 from #114 downward.
#79
added #685416: filter table renames aren't recognized by a later update
#80
More fun #685486: filter_update_7003() doesn't work for contrib modules.
#81
Some more fun: #685572: system_update_7008() is broken.
#82
Another fun one: #685892: Upload -> File module upgrade path is broken
#83
#686128: Body-to-field conversion is too slow
#84
Testing with alpha1. Big problem. The update dies part way through with a white screen of death. How do you tell which update failed? Is there a way to make the updates display as they occur? The number of tables increases by 9. node_revisions is not renamed. The system table does not list a schema change. The url changes to update.php?op=info. There is nothing in watchdog.
I tested running update a few times including without reloading the database. db_create_table will rerun without error. db_add_field goes straight to the WSOD when it tries to add a field that is already there and does not produce a watchdog error.
After a database reload, the WSOD occurs in:
db_insert('sequences')->fields(array('value' => max($max_aid, $max_uid, $max_batch_id)))->execute();The update adds the sequences table but the table already exists in a different format. I tried an update consisting of reloading the database, dropping sequences, update.php, and it worked.
sequences contains the following names. You check only three of them for the max() value.
menu_mid, drupal_users_uid, drupal_node_nid, drupal_vocabulary_vid, drupal_menu_mid, drupal_term_data_tid, drupal_forms_fid, drupal_form_fields_ffid, drupal_survey_responses_rid, drupal_files_fid, drupal_comments_cid, drupal_node_revisions_vid, path_redirect, boxes_bid
#85
The patch at #211182-140: Updates run in unpredictable order should give you an almost working upgrade path.
#86
I have another really small patch that gets me a working upgrade path with drush head.
1) comment_update_7012 was triggering the error for the taxonomy 'machine_name', through the field api functions.
This is an instance of the exception from the patch at #73 , so I added a call to the update directly.
2) filter_update_7005 was attempting to modify the format field of the column table
this field was removed in comment_update_7013, so this table is no longer necessary.
3) system_update_7027 was not enabling the modules.
This was a tricky one to track down, as the error wasn't occuring in this update, but rather later whenever the field api was being used.
The error being presented was : "Attempt to create a field of unknown type text_long."
What was happening here, is that the update was removing entries from the system table, and then enabling the new modules. These new modules (which define the field types) were never being added to the module list, even after module_list($refresh).
I tracked this down to the fact that the module_enable function doesn't explicitly insert items into the system table, it merely runs an update query. It assumes that all the modules are present in the system table, and the system table is only rebuilt if the 'dependencies' parameter is set, which we don't want in this specific case.
Because the system entries weren't there, the update statement was failing silently, and the modules weren't added to the module list, so all field operations which used these field types would fail.
#87
RTBC per #63, I'm sick of that other issue, this is holding up contrib modules and random site db testers far, far more than dependent updates.
#88
+++ modules/comment/comment.install@@ -247,6 +247,8 @@ function comment_update_7011() {
function comment_update_7012() {
+ taxonomy_update_7002();
+
Can we get some kind of @todo comment here to remove this once proper update ordering is in?
Powered by Dreditor.
#89
added comments to where taxonomy_update_7002 is called directly linking to the ordering issue.
removed the loop around the format column update as only the block custom table was still being updated.
#90
Ive decided to revert back to using the loop for filter_update_7005 , because the block module is optional hence the table might not exist, and even though there's only one item in the array, at least we aren't assigning or typing out the name 'block_custom' twice.
#91
The changes should be reviewed.
#92
4 lines of sensible code. i agree with immediate commit here.
#93
@@ -336,7 +336,7 @@ function filter_update_7005() {// instances (in Drupal core) to explicitly use the appropriate format.
// Note that the update of the node body field is handled separately, in
// node_update_7006().
- foreach (array('block_custom', 'comment') as $table) {
+ foreach (array('block_custom') as $table) {
The format column of the {comment} table is being removed from core, but its data is not. When does its data get the equivalent of filter_update_7005() performed on it if not here?
@@ -2246,6 +2246,7 @@ function system_update_7027() {$module_list = array('text', 'number', 'list', 'options');
db_delete('system')
->condition('type', 'module')
->condition('name', $module_list)
->execute();
+ system_rebuild_module_data();
module_enable($module_list, FALSE);
This defeats any purpose of calling module_enable() with the FALSE parameter, right? Also defeats the purpose of calling the db_delete() query right above it. I believe that db_delete() query was added in #490074: 6/7 upgrade path broken as an attempt to make the D6->D7 field updates work nicely for sites that have D6 contrib versions of CCK modules lying around... however, doesn't seem like that's effective anyway.
The existing patch at #211182: Updates run in unpredictable order fixes #1 by adding a correct dependency between the filter and comment updates, and #2 by removing the db_delete() altogether. If we want to get a quick patch to commit here, I think it would be best to use the same approach already worked out there.
#94
They were field-ified as far as i can tell. The columns were summarily dropped entirely.
No, the issue is EXISTING records from BEFORE the upgrade that had schema versions and statuses that weren't in any way related to the new modules. This delete resets the system database for those new modules, so that they can be successfully picked up.
I can't vouch for the logic behind calling it with the false parameter (apart from it seeming like complete overkill to build a dependency tree to enable these 4 modules), but i can validate that is impossible in the current code to enable those modules if they don't have records in the system table first, which is by design.
1) old records for previous versions of those modules (from cck) need to be purged
2) after this is called records for the new modules with the correct status and possibly schema version (although none of these modules have .install files) need to end up in the table.
At the moment the code only handles one of those requirements, and completely ignores the second (main) requirement. you know, actually enabling said modules.
So , going ahead you can either :
a) always rebuild the module table in module_enable, which doesn't seem necessary.
b) accept that almost nobody calls module_enable after directly altering the system table and invalidating all the caches of it, and those people will just need to rebuild the system table.
c) modify the module_enable function to INSERT new records if no existing record was found to update, which would require a lot more queries and require you to re-run the discovery mechanism code in any case.
b in my opinion is the right answer. completely beyond whether
feel free to pain the bike shed and design new hooks in the other issue, but we need to be able to actually test upgrades for now. who cares if these small fixes get replaced by a better system.
#95
I think that's true. But the process of fieldiying them means that their text formats were copied to a fields database table. I don't believe there's update code anywhere else that would perform the equivalent of filter_update_7005() on them once they are in that table, meaning that some comment text formats would not wind up being correct.
I agree that's the (ideal) goal, but neither current core nor the patch here does that. The issue is that if you call:
<?phpdb_delete('system')
->condition('type', 'module')
->condition('name', $module_list)
->execute();
system_rebuild_module_data();
?>
then system_rebuild_module_data() just puts back in the database exactly what db_delete() took out (this is because CCK is typically going to be in sites/all/modules or somewhere, which takes precedence over the core module locations)... so you still can't upgrade your site with D6 CCK modules lying around.
That's not for this issue to fix (it might not be fixable without manual intervention), but my point is just that the above two lines of code do nothing when combined together. Instead of adding system_rebuild_module_data(), this patch could remove the db_delete() instead, and it should work just as well.
#96
Tried the patch on a personal D6 site, only 2 errors remain, all the other issues are resolved now!
Edit:
However, there seems to be no comment/node body field displayed now. Also, I'm recieving the following error in watchdog:
PDOException: SQLSTATE[42S02]: Base table or view not found: 1146 Table 'blog.image_styles' doesn't exist: SELECT image_styles.* FROM {image_styles} image_styles ORDER BY name ASC; Array ( ) in image_styles() (line 409 of .../modules/image/image.module).
Edit 2: Was able to resolve the above error by uninstalling and re-installing image.module. I'm not sure, maybe I've had the "old" image.module installed once on that site.
#97
I cant reproduce this, which update throws this?
#98
This is locale_update_7000(), the "db_change_field('locales_source', 'location', 'location', ... ".
I am not sure what the issue is, but I can't even do that manually through phpmyadmin. That is, changing the field from varchar(255) to TEXT. Always gives me the above error.
#99
Ok, DamZ pointed me to the issue. The problem is that i18n adds an index to that table, and indexes on TEXT columns need a length.
#100
#95 - ok , i get that now.
how about explicitly setting it to the paths we need it to be during install, and making a note that these modules can NOT be overridden, during upgrade at least.
#101
@adrian, yeah, that sounds like it might work - although not totally sure how we prevent it from being overridden, since it seems like that would happen again as soon as someone rebuilds the module data, which could be done by more or less anyone.
In any case, it seems like there should probably be a separate issue for getting the upgrade to work for people with D6 CCK in their installations, since there are at least other workarounds for that? - and this patch isn't going to make the situation any worse if the current technique being used was already broken :)
For now, here is a reroll of the previous patch with minimal fixes for the issues I identified above.
#102
This works 100% for me, and fixes all the issues you came across.
thanks for running with it.
#103
Subscribing
#104
Committed the patch in #101 to help make module upgrades make progress.
#105
Quick question about D7 dev download. I would like to test #101. The download says it was produced January 26, 2010 - 10:04. #104 mentions a commit and is dated January 26, 2010 - 07:33. Do these commits go straight into the dev download?
#106
@peterx: I haven't tried it specifically, but yes, that's how it should work.
#107
RT #105: I guess there might be a delay between a commit (for example #101) and it being available in the tarball (Drupal 7.x-dev tarball).
You should definitely use CVS to work with the dev version. You would always have the latest code.
That being said, I really don't know how fast tarball are generated.
#108
tarballs are every 12 hours afaik - at midday and midnight GMT.
#109
Upgrading my Drupal 6 to Drupal 7 CVS of today, give this errors.:
Some warnings that don't break the site...
* Warning: file_put_contents(temporary://file8KmQji): failed to open stream: "DrupalTemporaryStreamWrapper::stream_open" call failed in file_unmanaged_save_data() (line 1522 of /persistent/home/int/sites/drupal7/includes/file.inc).* Can't create the file.
And run all updates(113) very fast, at the (111 update) and give lots of time..
Completed 111 of 113. 98%Updating user module
At linux shell executing TOP command:
27836 www-data 20 0 101m 34m 26m S 20.3 2.0 1:01.91 apache227841 www-data 20 0 100m 33m 26m S 11.3 2.0 0:05.01 apache2
1072 mysql 20 0 869m 436m 6620 S 4.7 25.5 6:14.92 mysqld
And where working by between 5 and 10 minuts... and then the apache2 and mysql aren't active working anymore, but the Update page is in the same text..
Completed 111 of 113. 98%Updating user module
So, I restart the update and show me:
Completed 22 of 24.Updating user module
and the linux process and same and above... but the process were working for at least 20minuts.. but give the same problem.. The page don't move one, it's stoped at the user module upgrade...
I have 5167 nodes, and 1411 users..
What is the problem with update the user module?
The upgrade process don't give any error, neither timeout error...
I press Esc, and click in show errors:
Give:
he update process was aborted prematurely while running update #7004 in user.module. All errors have been logged. You may need to check the watchdog database table manually.
* Front page
* Administration pages
The following updates returned messages
system module
Update #7035
* Failed: SQLSTATE[23000]: Integrity constraint violation: 1052 Column 'fid' in field list is ambiguous
filter module
Update #7004
* Failed: SQLSTATE[42000]: Syntax error or access violation: 1091 Can't DROP 'fid'; check that column/key exists
locale module
Update #7000
* Failed: SQLSTATE[42000]: Syntax error or access violation: 1091 Can't DROP 'source'; check that column/key exists
poll module
Update #7002
* Failed: SQLSTATE[42S02]: Base table or view not found: 1146 Table 'drupal7.drupal_poll_votes' doesn't exist
user module
# Drupal 6.15 to Drupal 7 CVS Today
# PHP 5.2.10-2ubuntu6.4
# Apache/2.2.12 (Ubuntu)
# Versión del cliente: 5.1.37
# Ubuntu 9.10 Server
#110
user_update_7004 Add the user's pictures to the {file} table and make them managed files.
I remove all Drupal6 files include the modules, files, site folders.. (after I disable all contrib modules)
SELECT COUNT( * )
FROM `drupal_users`
WHERE picture <> ''
= 14
I update the table to drupal_users set picture = '' and now the two update users updates fine..
now I have the others errors:
The following updates returned messages
system module
Update #7035
* Failed: SQLSTATE[23000]: Integrity constraint violation: 1052 Column 'fid' in field list is ambiguous
filter module
Update #7004
* Failed: SQLSTATE[42000]: Syntax error or access violation: 1091 Can't DROP 'fid'; check that column/key exists
locale module
Update #7000
* Failed: SQLSTATE[42000]: Syntax error or access violation: 1091 Can't DROP 'source'; check that column/key exists
poll module
Update #7002
* Failed: SQLSTATE[42S02]: Base table or view not found: 1146 Table 'drupal7.drupal_poll_votes' doesn't exist
user module
#111
fix the poll_update_7002
#112
Fix the PHPDoc
#113
Can any one give a review of this simple patch in #112, the bot was allready pass this one 6hours ago...
#114
#84 revisited using D7 dev Jan 27 10:05 dev. Crashed inserting into sequences.
Precede
$schema['sequences'] = array(with$max_sequence = db_query('SELECT MAX(id) FROM {sequences}')->fetchField();db_drop_table('sequences');
Change
db_insert('sequences')->fields(array('value' => max($max_aid, $max_uid, $max_batch_id)))->execute();to
db_insert('sequences')->fields(array('value' => max($max_aid, $max_uid, $max_batch_id, $max_sequence)))->execute();Sequence problem disappears.
Got through to 104 updates pending.
Strict warning: Only variables should be passed by reference in update_selection_page() (line 33 of H:\home\example.com\public_html\update.php).I changed
$output = drupal_render(drupal_get_form('update_script_selection_form'));to
$stuff = drupal_get_form('update_script_selection_form');$output = drupal_render($stuff);
Update selection gave:
PDOException: SQLSTATE[42S22]: Column not found: 1054 Unknown column 'name' in 'field list': INSERT INTO {queue} (name, data, created) VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1, :db_insert_placeholder_2); Array ( [:db_insert_placeholder_0] => drupal_batch:4146:0 [:db_insert_placeholder_1] => a:2:{i:0;s:13:"update_do_one";i:1;a:2:{i:0;s:6:"system";i:1;s:4:"7000";}} [:db_insert_placeholder_2] => 1264573566 ) in SystemQueue->createItem() (line 192 of H:\home\example.com\public_html\modules\system\system.queue.inc).Table queue is
`nid`, `uid`, `vote`. I manually recreated queue and ran the update by hand. 102 updates ran in an instant. Update 103 is churning away.#115
User 7004. That
continue;has got to go. It makes the code loop if the last user goes through continue because of a missing file. I changedif (!file_exists($user->picture)) {to if (file_exists($user->picture)) { and wrapped the if down to the end ofdb_update('users'). The change makes users all users, including those with missing files, go through the position update. No more infinite loop in user 7004.#116
peterx Suggestion
Need a review for this and #112
#117
I haven't been able to test this patch, but looking at the code, it looks good to me.
One question though; if the $user->picture does not exist, is that because we have an invalid path or because we have an empty path? If it happens to be the case that the path is wrong, one could argue that the $user object is in an inconsistent state. Should we consider empty-ing $user->picture when that is the case?
#118
RTBC for #112 (pool_update_7002)
Need Review for #116
@Dries, if the $user->picture does not exist, is that because we have an invalid path. The empty path are allready filter in the SQL query. i think that isn't necessary because we drop 'picture' field
#119
#116 looks fine too visually, didn't test it. Suggest we commit both that and #112 to keep this moving.
#120
Committed #116. Thanks int!
#121
#112 needs committing as well.
#122
Committed #112 as well. Thanks!
#123
my patch in #112 was not been committed yet.. (pool_update_7002_02.patch)
#124
Hm. #112 looks to be committed here. Or at least when I try to apply it, it asks me if I want to revert it.
Re-testing upgrade of webchick.net now...
#125
Well that was quick:
"An AJAX HTTP error occurred. HTTP Result Code: 200 Debugging information follows. Path: http://localhost/webchick.net/update.php?id=4&op=do StatusText: OK ResponseText: Fatal error: Call to undefined function field_attach_create_bundle() in /Users/webchick/Sites/webchick.net/modules/comment/comment.install on line 149"
Call Stack:
0.0071 163140 1. {main}() /Users/webchick/Sites/webchick.net/update.php:0
4.0619 15155836 2. _batch_page() /Users/webchick/Sites/webchick.net/update.php:388
4.0633 15157192 3. _batch_do() /Users/webchick/Sites/webchick.net/includes/batch.inc:81
4.0633 15157480 4. _batch_process() /Users/webchick/Sites/webchick.net/includes/batch.inc:163
4.9624 15272576 5. call_user_func_array() /Users/webchick/Sites/webchick.net/includes/batch.inc:286
4.9624 15272576 6. update_do_one() /Users/webchick/Sites/webchick.net/includes/batch.inc:0
4.9625 15274308 7. comment_update_7005() /Users/webchick/Sites/webchick.net/includes/update.inc:689
#126
That's weird, because in
http://drupalcode.org/viewvc/drupal/drupal/modules/poll/poll.install?rev...
and http://drupal.org/project/cvs/3060 don't show the commit.. later I will verify it with in the cvs command..
#127
@webchick error in #125 should be fixed by #211182: Updates run in unpredictable order (the comment update are run before system.install had the chance to install the field module.)
#112 applies cleanly, latest change for poll.install was Dec 4th, and it still contains
db_add_field('poll_votes', 'timestamp', $field);which is what #112 fixes.#128
see also #599122: Do not use module_load_include() in global context which fixes system_update_7008() (see duplicate #685572: system_update_7008() is broken)
#129
RTBC #112 again? :)
#130
LOL. Weird. I must've been tired last night. :P
Really committed to HEAD now. ;)
#131
Today #251792: Implement a locking framework for long operations commited to 6 so updates should be changed
#132
#211182: Updates run in unpredictable order, which adds hook_update_dependencies(), was committed.
Would be good for someone to try the upgrade path now and see how far along we are. I'll try this with webchick.net in the next couple of days.
#133
using todays head and #211182: Updates run in unpredictable order latest patch http://drupal.org/files/issues/211182-updates-180.patch I get the follwoing php error
[03-Feb-2010 15:26:54] PHP Fatal error: Uncaught exception 'PDOException' with message 'SQLSTATE[42S22]: Column not found: 1054 Unknown column 'value' in 'field list'' in C:\sites\d6d7\includes\database\database.inc:1940Stack trace:
#0 C:\sites\d6d7\includes\database\database.inc(1940): PDOStatement->execute(Array)
#1 C:\sites\d6d7\includes\database\database.inc(586): DatabaseStatementBase->execute(Array, Array)
#2 C:\sites\d6d7\includes\database\mysql\query.inc(37): DatabaseConnection->query('INSERT INTO {s...', Array, Array)
#3 C:\sites\d6d7\includes\update.inc(522): InsertQuery_mysql->execute()
#4 C:\sites\d6d7\update.php(378): update_fix_d7_requirements()
#5 {main}
thrown in C:\sites\d6d7\includes\database\database.inc on line 1940
Initial database does not contain a field table.
#134
Skip that fields table thingy.
Traced progression of update_fix_d7_requirements(). It completes all the way down to
if (db_table_exists('locales_source')) {db_add_field('locales_source', 'context', array('type' => 'varchar', 'length' => 255, 'not null' => TRUE, 'default' => '', 'description' => 'The context this string applies to.'));
}
should
'The context this string applies to.'
be doublequoted instead of single quotes due to the period?
#135
Hm. Nope, I can't see that causing a problem. That'd only be if the sentence contained an apostrophe.
Could you elaborate a bit more? At what point do you get an error? At the main update.php page, or after starting the upgrade process, or..? And what is the error?
#136
Found it!
My d6 database has a queue table already.
so
// Add the queue table.$schema['queue'] = array(
'description' => 'Stores items in queues.',
'fields' => array(
'item_id' => array(
'type' => 'serial',
'unsigned' => TRUE,
'not null' => TRUE,
'description' => 'Primary Key: Unique item ID.',
),
'name' => array(
'type' => 'varchar',
'length' => 255,
'not null' => TRUE,
'default' => '',
'description' => 'The queue name.',
),
'data' => array(
'type' => 'text',
'not null' => FALSE,
'size' => 'big',
'serialize' => TRUE,
'description' => 'The arbitrary data for the item.',
),
'expire' => array(
'type' => 'int',
'not null' => TRUE,
'default' => 0,
'description' => 'Timestamp when the claim lease expires on the item.',
),
'created' => array(
'type' => 'int',
'not null' => TRUE,
'default' => 0,
'description' => 'Timestamp when the item was created.',
),
),
'primary key' => array('item_id'),
'indexes' => array(
'name_created' => array('name', 'created'),
'expire' => array('expire'),
),
);
db_create_table('queue', $schema['queue']);
won't work.... Need to check for the table first, modify if present else write it.
#137
@webchick
"Could you elaborate a bit more? At what point do you get an error? At
the main update.php page, or after starting the upgrade process, or..?
And what is the error? "
WSOD on update.php, error is the same as above
[03-Feb-2010 15:59:31] PHP Fatal error: Uncaught exception 'PDOException' with message 'SQLSTATE[42S22]: Column not found: 1054 Unknown column 'value' in 'field list'' in C:\sites\d6d7\includes\database\database.inc:1940
Stack trace:
#0 C:\sites\d6d7\includes\database\database.inc(1940): PDOStatement->execute(Array)
#1 C:\sites\d6d7\includes\database\database.inc(586): DatabaseStatementBase->execute(Array, Array)
#2 C:\sites\d6d7\includes\database\mysql\query.inc(37): DatabaseConnection->query('INSERT INTO {s...', Array, Array)
#3 C:\sites\d6d7\includes\update.inc(522): InsertQuery_mysql->execute()
#4 C:\sites\d6d7\update.php(378): update_fix_d7_requirements()
#5 {main}
thrown in C:\sites\d6d7\includes\database\database.inc on line 1940
Tracing PDO Exceptions is a ROYAL pain!
#138
@137 Sounds like you already have a {sequences} table as well. Quick check looks like we never deleted the old sequences table from Drupal 5. So any upgrades sites from D5 to D6 to D7 are going to run into this.
#139
On a side note my D6 database also has a sequences table.
This particular database is a copy of a d5 site, brought to my local environment, updated to 6.15 then attempted to update to d7. I've been using this database and found #601642: Old databases might have a small {system}.weight which is now RTBC.
#140
Great sleuthing!
Can we wrap those hunks in a if (!db_table_exists()) check or similar?
#141
db_create_table() auto-checks db_table_exists(), so that's not the problem. The problem is that the D5 and D7 versions of {sequences} are different and I think we'll need to to a drop table if it exists before the db_create_table().
#142
I can but need to wait to cvs refreshes. I have the patch I used but that will chase HEAD when it catches up.
Can work on it in tomorrow morning since what were actually modifying will be update.inc
#143
Dave Reid - February 3, 2010 - 16:45
"db_create_table() auto-checks db_table_exists(), so that's not the problem. The problem is that the D5 and D7 versions of {sequences} are different and I think we'll need to to a drop table if it exists before the db_create_table()."
Was {sequences} used in d6?
Where we're changing the code from my understanding is in a place where if the update fails, d6 will still run from the database
#144
See #114 for a suggestion on sequences in case user modules are using sequences. My first site for conversion went from D5 to D6 some time ago and uses a few add on modules. The last update for the sequences table is after the D5 -> D6 conversion suggesting an add on module used the table. The last update to the table was a long time ago when I replaced some modules with alternatives, suggesting it is probably not used by any module currently installed. A quick scan of the site code shows only the following single use of {sequences}. Perhaps someone could scan all add on modules for {sequences} and notify any module maintainers who access the table direct.
/*** Remove auto_increment from {boxes} to allow adding custom blocks with
* visibility settings.
*/
function system_update_6000() {
$ret = array();
switch ($GLOBALS['db_type']) {
case 'mysql':
case 'mysqli':
$max = (int)db_result(db_query('SELECT MAX(bid) FROM {boxes}'));
$ret[] = update_sql('ALTER TABLE {boxes} CHANGE COLUMN bid bid int NOT NULL');
$ret[] = update_sql("REPLACE INTO {sequences} VALUES ('{boxes}_bid', $max)");
break;
#145
The sequences table was explicitly left in Drupal 6, unused by core, so that contrib could transition away from it. I don't think anyone expected to put it back in, or that modules would keep using it.
A handful of Drupal 6 modules are using sequences, I've excluded ones where it only shows up in the .install file in the hope those are upgrade paths away from it:
delete_all/delete_all.module
devel/devel.module
marc/marc.module
phpbb2drupal/phpbb2drupal.module
Note there's an existing issue about this here #198987: Remove {sequences} table.
Given the number of modules (discouting mentions in .install) is so few, I'd suggest just dropping the table and adding a note in the 6/7 upgrade docs that it's been removed/refactored and they need to handle their own upgrade path by selecting MAX(id) from their own tables or whatever - modules have had two years to stop using it.
#146
#145 sounds sensible to me.
#147
Downloaded from this morning. Loaded database. Deleted sequences table. Ran update.php. Reached update page with 106 pending updates and:
Strict warning: Only variables should be passed by reference in update_selection_page() (line 33 of H:\home\example.com\public_html\update.php).
Continue produced:
PDOException: SQLSTATE[42S22]: Column not found: 1054 Unknown column 'name' in 'field list': INSERT INTO {queue} (name, data, created) VALUES
Back to the start. Reload database. Drop sequences and queue. Create queue using SQL form a D7 test site. update.php. Ran through to completion. Really nice having the extra line of information telling me which update is in progress. Two failed messages.
filter module Update #7004
* Failed: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '3-0' for key 'PRIMARY'
block module Update #7003
* Failed: SQLSTATE[42000]: Syntax error or access violation: 1091 Can't DROP 'list'; check that column/key exists
Logged in as administrator. No theme. No menu entries. I went to admin/appearance and set the admin theme to Garland. I like Seven but Seven was disabled.
Admin by task is empty. Admin by module has many entries.
block_update_7003() starts by dropping index list but there is no index named list. I removed the line. The next line creates index list but the index exceeds 1000 characters when you use fields theme and module as UTF8 so I put module in a separate index. The block updates run to completion. Blocks appear in Garland. When I try to allocate blocks in the blocks page, there are no regions listed.
#148
Added #615634: Update CHANGELOG.txt to explicitly name new core modules and core modules that have been removed. to the list. Its a bit of a blocker for #536854: Review and test the upgrade guide for Drupal 7.
Actually all these issues in this meta issue are a blocker for #536854, but at least some writing could be done while upgrade path gets fixed.
#149
There's a D6 backport of queue module which is likely what's causing peterx's problem, that should have the same schema as D7, so we should just do db_table_exists() there before trying to create it.
For sequences, maybe do db_table_exists() too and rename it to sequences_old?
Also marking this back to active since there's no current patch. I think it'd help if we opened individual issues for actual patches and linked from here from now on, otherwise it's easy to end up with multiple conflicting patches for different issues all in various stages in the one issue.
#150
D6: CREATE TABLE IF NOT EXISTS `queue` (
`nid` int(10) unsigned NOT NULL DEFAULT '0',
`uid` int(10) unsigned NOT NULL DEFAULT '0',
`vote` int(3) NOT NULL DEFAULT '0',
PRIMARY KEY (`nid`,`uid`));
D7: CREATE TABLE IF NOT EXISTS `queue` (
`item_id` int(10) unsigned NOT NULL AUTO_INCREMENT COMMENT 'Primary Key: Unique item ID.',
`name` varchar(255) NOT NULL DEFAULT '' COMMENT 'The queue name.',
`data` longtext COMMENT 'The arbitrary data for the item.',
`expire` int(11) NOT NULL DEFAULT '0' COMMENT 'Timestamp when the claim lease expires on the item.',
`created` int(11) NOT NULL DEFAULT '0' COMMENT 'Timestamp when the item was created.',
PRIMARY KEY (`item_id`),
KEY `name_created` (`name`,`created`),
KEY `expire` (`expire`)
)COMMENT='Stores items in queues.';
I cannot find any d6 code or d5 code referring to {queue}.
#151
You must have at least one, I found:
grep -r "{queue}" *6.x/HEAD/ejournal/ejournal.module: $result = db_fetch_object(db_query("SELECT SUM(IF(uid > 0, 1, 0)) AS voted, SUM(vote) AS score FROM {queue} WHERE nid = '%d'", $nid));
#152
Applied #601642: Old databases might have a small {system}.weight followup.patch
Renamed the queue and sequences tables to queue_old and sequences_old and added a todo comment
Can now get a D5 -> D6 -> D7 database to this point. I haven't actually gone in yet to the settings.php file to change the free_access setting to allow me into the site after running update.php
Access denied. You are not authorized to access this page. Log in using either an account with the administer software updates permission or the site maintenance account (the account you created during installation). If you cannot log in, you will have to edit settings.php to bypass this access check. To do this:
1. With a text editor find the settings.php file on your system. From the main Drupal directory that you installed all the files into, go to sites/your_site_name if such directory exists, or else to sites/default which applies otherwise.
2. There is a line inside your settings.php file that says $update_free_access = FALSE;. Change it to $update_free_access = TRUE;.
3. As soon as the update.php script is done, you must change the settings.php file back to its original form with $update_free_access = FALSE;.
4. To avoid having this problem in the future, remember to log in to your website using either an account with the administer software updates permission or the site maintenance account (the account you created during installation) before you backup your database at the beginning of the update process.
patch attached for review. Didn't know if I should create another issue or not.
#153
@cmattice - it looks like your patch includes hunks from the system weight patch which are already in HEAD, so it won't apply.
Only comment issues otherwise - should start with a capital letter, end with a full stop/period. I'd also remove the TODO - if a contrib module salvages data from that old table, it can also take care of dropping it.
#154
Tried again, got a new error.
When update.php is finished (batch run throug completely), I got the following fatal error.
Fatal error: Call to undefined function field_info_field() in d7/drupal/modules/node/node.module on line 572
Call Stack:
0.0002 97408 1. {main}() d7/drupal/update.php:0
6.3229 4595640 2. _batch_page() d7/drupal/update.php:425
6.3236 4696968 3. _batch_finished() d7/drupal/includes/batch.inc:90
6.3251 4748368 4. update_finished() d7/drupal/includes/batch.inc:451
6.3251 4748760 5. drupal_flush_all_caches() d7/drupal/includes/update.inc:857
11.1318 12049712 6. node_types_rebuild() d7/drupal/includes/common.inc:6222
11.1327 12051968 7. node_type_save() d7/drupal/modules/node/node.module:477
11.1338 12071360 8. node_configure_fields() d7/drupal/modules/node/node.module:538
EDIT: Accessing update.php shows still 53 unfinished tasks, one of them is "7020 - Enable field module.", so the error above is quite obvious. To be exact, it seems that "7018 - Shorten the {system}.type column and modify indexes." is failing. Trying to figure out why now...
#155
Before I reroll, here are my findings
Set access bypass to TRUE in settings.php, started up the update process from where I left off.
Required 111 update steps
it ran through
node
comment
system
picked up
user at step 80
profile at step 100
search at 104
and tracker at 111
Proceeded about another 30-45 second then fataled with
Fatal error: Call to undefined function field_attach_delete_bundle() in C:\sites\d6d7\modules\comment\comment.module on line 321
Checked the database found several content_type_"modules" all with 0 rows
There is no field_ tables.
Possibilities system did not install complete D7 schema or there is a problem with core's field module not enabled.
Any Ideas? Though system was suppose to update first but this may not be what's happening. It could be that the update processes is not installing required tables first.
#156
Ah, found it.
The index 'system_list' seems to be added twice, once in update_fix_d7_requirements() and once in system_update_7018(). The attached patch removes the one in system_updat_7018(), not sure if that's ok or if the other one should be removed.
#157
Berdir: Actually, I think this should now be using hook_update_dependencies()?
#158
It looks to me like the wrong patch file was attached.
#159
So the issue is that update.php cannot be bootstrapped multiple times, and every single operation performed in update_fix_d7_requirements() can lead to an error.
We therefore have to make sure that we can at least get behind a WSOD in case of a previous error.
Additionally, db_index_exists() doesn't seem to be used anywhere yet? How comes that the database-specific methods use an entirely different function signature? ;)
#160
With that patch, I'm still getting to update.php?op=info with the following fatal error on Apache/2.2.11 (Win32) PHP/5.2.6:
Fatal error: Maximum execution time of 240 seconds exceeded in includes\file.inc on line 1647
Call Stack:
0.0008 197472 1. {main}() update.php:0
134.9436 6809888 2. drupal_bootstrap() update.php:382
134.9443 6810784 3. _drupal_bootstrap_full() includes\bootstrap.inc:1559
134.9834 9307640 4. module_load_all() includes\common.inc:4530
238.3891 9656520 5. drupal_load() includes\module.inc:24
238.3891 9656824 6. drupal_get_filename() includes\bootstrap.inc:886
238.3917 9658512 7. drupal_system_listing() includes\bootstrap.inc:705
238.9657 9692712 8. file_scan_directory() includes\common.inc:4742
239.8057 9702928 9. file_scan_directory() includes\file.inc:1654
239.9816 9705296 10. file_scan_directory() includes\file.inc:1654
#161
That's very possibly a stupid remark, but instead of making sure update_fix_d7_requirements() is idempotent by adding lots of lots of
if (!db_column_exists()),if (!db_index_exists())code- can't we make sure that it gets called only once in a lifetime ? a variable or something ?
- or cant we use one single tell-tale criteria to determine it has already run and should just return ?
function update_fix_d7_requirements() {
if (db_table_exists('date_format_type')) {
return;
}
... do the job
}
#162
There is a variable, it's possible that it doesn't get set correctly if there's another error though.
#163
In my last attempt on todays head
I used
Index: includes/update.inc
===================================================================
RCS file: /cvs/drupal/drupal/includes/update.inc,v
retrieving revision 1.34
diff -u -p -r1.34 update.inc
--- includes/update.inc 4 Feb 2010 17:03:19 -0000 1.34
+++ includes/update.inc 6 Feb 2010 01:54:41 -0000
@@ -483,6 +483,12 @@ function update_fix_d7_requirements() {
db_create_table('date_formats', $schema['date_formats']);
db_create_table('date_format_locale', $schema['date_format_locale']);
+ // Check for queue table, if found rename it, it has a different
+ // schema than D7 but could be useful for upgrading modules.
+ if (db_table_exists('queue')) {
+ db_rename_table('queue', 'queue_old');
+ }
+
// Add the queue table.
$schema['queue'] = array(
'description' => 'Stores items in queues.',
@@ -528,6 +534,12 @@ function update_fix_d7_requirements() {
);
db_create_table('queue', $schema['queue']);
+ // Check for sequences table, if found rename it, it has a different
+ // schema than D7 but could be useful for upgrading modules.
+ if (db_table_exists('sequences')) {
+ db_rename_table('sequences', 'sequences_old');
+ }
+
// Create the sequences table.
$schema['sequences'] = array(
'description' => 'Stores IDs.',
Index: modules/system/system.install
===================================================================
RCS file: /cvs/drupal/drupal/modules/system/system.install,v
retrieving revision 1.445
diff -u -p -r1.445 system.install
--- modules/system/system.install 3 Feb 2010 18:16:23 -0000 1.445
+++ modules/system/system.install 6 Feb 2010 02:30:06 -0000
@@ -1610,9 +1610,7 @@ function system_update_6052() {
* Add a {system} index on type and name.
*/
function system_update_6053() {
- $ret = array();
- db_add_index($ret, 'system', 'type_name', array(array('type', 12), 'name'));
- return $ret;
+ // Update moved to update_fix_d7_requirements().
}
/**
@@ -2004,11 +2002,7 @@ function system_update_7017() {
* Shorten the {system}.type column and modify indexes.
*/
function system_update_7018() {
- db_drop_index('system', 'modules');
- db_drop_index('system', 'type_name');
- db_change_field('system', 'type', 'type', array('type' => 'varchar', 'length' => 12, 'not null' => TRUE, 'default' => ''));
- db_add_index('system', 'type_name', array('type', 'name'));
- db_add_index('system', 'system_list', array('weight', 'name'));
+ // Update moved to update_fix_d7_requirements().
}
/**
It ran for 4 minutes then fatal error-ed with
[05-Feb-2010 21:36:59] PHP Fatal error: Class 'views_object' not found in C:\sites\d6d7\sites\all\modules\views\includes\plugins.inc on line 327
[05-Feb-2010 21:36:59] PHP Fatal error: Exception thrown without a stack frame in Unknown on line 0
The variable does get set when this function completes
#164
Views is a contrib module, please try the same patch without it installed. Also if you could upload your patch files as a .patch extension rather than pasting to comments that makes them easier to review and commit. Seems like it's getting close!
#165
@yched: We set a variable already, but naturally, it's only ever set if the code reaches the variable_set() line.
My guess is that the "queue" table only existed, because update.php was bootstrapped more than once.
Regarding the sequences table, my patch uses db_column_exists() to directly check for the old {sequences}.name column of D5. Only if that column exists, the table is renamed.
I grepped throughout core and db_index_exists() indeed doesn't seem to be used anywhere yet - thus, no one realized the wrong function signature.
Additionally, I commented out that
ini_set('display_errors', FALSE);because I think we should be a bit more verbose - at least until we're out of alpha. Only after doing so, I was able to see the PDO exception that lead to a WSOD.#166
@sun: My guess is that the "queue" table only existed, because update.php was bootstrapped more than once.
Nope my D5 -> D6 database has a {queue} table
@catch: Views is a contrib module, please try the same patch without it installed. Also if you could upload your patch files as a .patch extension rather than pasting to comments that makes them easier to review and commit. Seems like it's getting close!
Ok, my problem is at work on a t1 connection I never have a problem accessing drupal.org, when I get home I have ALL kinds of issues connecting to drupal.org over a wireless dsl. have to delete cookies, try several times just to get drupal to post. I originally did have a patch attached but took me 2 hours on and off before it would post. Either get a 503 or WSOD
Reran from where I left off lastnight after pulling out all contrib modules. It fataled with
[06-Feb-2010 08:58:25] PHP Fatal error: Call to undefined function user_permissions_get_modules() in C:\sites\d6d7\modules\user\user.install on line 517
Line 517 is the only call to this function grepping "user_permissions_get_", will dig some more later
#167
#607238: Permissions are assumed to be unique among modules, but uniqueness is not enforced caused the user module fatal error, attached one line patch there.
#168
actually I made comment there and patched here. can reverse if necessary
#169
Started update from where I left off and it completes.
There is still a problem though. on the completion page I get
* Notice: Undefined index: update_success in update_results_page() (line 159 of C:\sites\d6d7\update.php).
* Warning: reset(): Passed variable is not an array or object in update_results_page() (line 163 of C:\sites\d6d7\update.php).
* Warning: array_pop(): The argument should be an array in update_results_page() (line 163 of C:\sites\d6d7\update.php).
#170
adding #706860: update_fix_d7_block_deltas() helper function broken for non-core modules
#171
adding #706842: Improve comments for the taxonomy upgrade path
#172
#709386: Schema Version reset to -1 when update.module cache is cleared
#173
The last submitted patch, fixupdate_0.patch, failed testing.
#174
merged patches from #159 and #168. added condition around
db_drop_index('system', 'type_name');as it was giving me a WSOD at ?op=info (the d6 installation I was trying to upgrade didn't have the index type_name).Now I'm getting Fatal error: Class 'DrupalQueue' not found on ?op=info page and my registry* tables are empty. maybe related to #708832: Fatal error: Class 'DrupalQueue' not found.
#175
#176
The last submitted patch, 563106_update_174.patch, failed testing.
#177
here's a patch i used to test the newly merged drush updatedb command in drush head.
drush is a lot less error tolerant about updates , so it catches more problems.
mostly it was about the order of the updates that weren't quite right :
before filter 7005 can run, you need to run user 7006 and taxonomy 7002 and system 7037
but before you can run user_7006 you need to have run filter_7002
We were also creating the system_list index in duplicate, once in update.inc and then in system 7018.
#178
Looks good, let's keep getting these in.
#179
Looks good apart from the quotes around 7006. Fixed those and committed to HEAD.
Back to active.
#180
Hm, as per my comment at http://drupal.org/node/211182#comment-2488380 I'm not sure any of the hook_update_dependencies() in core actually have the correct dependencies declared - and I'm not sure the ones in this latest patch totally make sense either. However, we left the original ones alone because (at the time) they were sufficient to allow the drupal.org database to be upgraded from Drupal 6 to 7, and the same criteria apply here - if it helps make updates possible, it's an improvement :)
However, I had meant to file a followup issue to go through and actually straighten these out for real. Just did so now: #717834: The dependencies declared in core's hook_update_dependencies() implementations aren't actually correct
#181
Er... So this doesn't look like a meta issue at all. :) From my understanding, a meta-issue is used to track *other* issues and patches are submitted there. If we keep on addressing every single upgrade problem here, the issue is going to get horribly big and it makes it harder for new people (like me!) to get involved easily. I'll try to tackle some of those now anyways.
#182
Basically, we don't need tracking issues anymore now that we have arbitrary tags. So, a meta issue might discuss the overall strategy for upgrading, and not the particular details. My .02
#183
I'm okay with that, but what's happening with that is that we're processing a lot of particular details, it's exactly the problem we're having here. :)
For example, by reading through this issue quickly, I missed the discussion about the sequences table, and filed a separate bug about it: #719730: drop the sequences and queue tables from D5 during the D6 -> D7 upgrade. I hereby encourage people to take the discussion about that issue to the other place.
I will file other issues regarding the upgrade process in separate issue (if i still have time) and keep this issue to general discussions regarding overall strategy.
#184
I had problems closely related with #325169: Move error/exception handler higher up in the bootstrap process during that last issue (blank page in update.php). I fixed this in #719850: errors in update.php during bootstrap yield a blank page, and that may overlap some issues/patches addressed here.
Basically, if you had any error in your error.log that says something about _system_rebuild_theme_data() missing, that issue will fix it.
#185
combined patch from #719850: errors in update.php during bootstrap yield a blank page with {queue} and {sequences} table fixes. I did get the update to process to completely empty the {queue} table (284 entries) but only after changing update.php to not clear the cache.
patch attached but updating is still not completely there yet, now I get a error trying to log into the site. Tried clearing cookies and cache but didn't help.
#186
While I think some meta issues are useful, this one is not any more, so agreed with anarcat and moshe. Please keep upgrade fixes in separate, critical, issues tagged with "D7 upgrade path". Closing this one out.
#187
Excellent, let's kill this issue. I have opened a new meta-discussion issue in #722074: Drupal 6 to Drupal 7 upgrade path meta discussions in case there's any followup. I made it very clear in the new issue that patches and issue-specific discussions should not take place there so I hope it doesn't degenerate the same way as here.
#188
Any one help me to upgrade drupal 6 to drupal 7.
After execute update.php below error generate-
The following updates returned messages
node module
Update #7006
* Failed: DatabaseSchemaObjectExistsException: Table field_data_body already exists. in DatabaseSchema->createTable() (line 630 of /opt/lampp/htdocs/texenergo_demo/includes/database/schema.inc).
I have attach the screen-shot of error message please see it.
http://drupal.org/files/issues/1_1_5.png
http://drupal.org/files/issues/2_1_4.png
please help me.
Thanks & Regards
Vindesh Mohariya
#189
@vindesh: This issue is closed. You need to open a new one or search for an issue that is still opened about the same subject.
#190
The "update system" category no longer exists (that's probably why it got accidentally recategorized to "ajax system" when the comment was posted)...
Putting this back in the correct category for posterity :)