Updating CCK Fields and Data from D6 to D7

subscribe

if (!module_exists('flag')) print 'subscribe';

The code lives here for those wanting to take a peek: http://drupalcode.org/viewvc/drupal/contributions/modules/cck/modules/co...

Any update on the migrate module?

I tried migrating my cck fields to D7. Most of them seem to be converted correctly and show up in the structure and the fields are there when editing. But none of them have any data in drupal. The data is there in the mysql database, but the fields are empty when editing a node. Any idea?

Ok, great, I actually got all my CCK data from D6 working on D7. Node_references, lists, integers, everything!

Somehow my upgraded data had field_config_entity_type switched. So node was '2' and comment '1'.

Therefor, I had to change every 'etid' in field_data_field_..... (and field_revision_field_....., just to be sure) to '2'. Maybe I could've just switched the two field_config_entity_type's, but I didn't know what effect that would have.

The nodereferences worked after that, but most other fields didn't. A simple fix was going to structure > node type > manage fields and resave every field that didn't work. (this didn't work when the etid's weren't set to '2')

Not sure what the problem was, but probably some data in field_config or field_config_instance.

Thanks for building the migrate module. Except for what I mentioned above, it works great and helped me a lot! :)

Can anyone confirm that this works with latest D7? Anyone working on the possible bugs reported in #5?

subscribe

I just tried a migration using Amazon module and went down in flames:

    *  Data could not be migrated for field field_asin in bundle amazon_example.
    * exception 'PDOException' with message 'SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '1-2147-0-0-und' for key 'PRIMARY'' in /home/rfay/workspace/d7git/includes/database/database.inc:1961 Stack trace: #0 /home/rfay/workspace/d7git/includes/database/database.inc(1961): PDOStatement->execute(Array) #1 /home/rfay/workspace/d7git/includes/database/database.inc(559): DatabaseStatementBase->execute(Array, Array) #2 /home/rfay/workspace/d7git/includes/database/mysql/query.inc(37): DatabaseConnection->query('INSERT INTO {fi...', Array, Array) #3 /home/rfay/workspace/d7git/sites/all/modules/cck-HEAD/modules/content_migrate/content_migrate.admin.inc(281): InsertQuery_mysql->execute() #4 [internal function]: _content_migrate_batch_process_migrate_data('field_asin', Array) #5 /home/rfay/workspace/d7git/includes/batch.inc(286): call_user_func_array('_content_migrat...', Array) #6 /home/rfay/workspace/d7git/includes/batch.inc(163): _batch_process() #7 /home/rfay/workspace/d7git/includes/batch.inc(81): _batch_do() #8 /home/rfay/workspace/d7git/modules/system/system.admin.inc(2366): _batch_page() #9 [internal function]: system_batch_page() #10 /home/rfay/workspace/d7git/includes/menu.inc(480): call_user_func_array('system_batch_pa...', Array) #11 /home/rfay/workspace/d7git/index.php(22): menu_execute_active_handler() #12 {main}

Tried some others and didn't get the flames. I did get the content types with the fields, but the data from the fields didn't make it over.

Not a very scientific operation. I guess I'll have to try something more rigorous.

This was using content_migrate from CCK HEAD.

I think this will be a super important module for developers going from D6 to D7. After using this module in May (#5), I tried migrating my data again about a week ago.

The trick is still to resave every CCK field under Structure > Content types > etc, after migrating your data. Else the data won't show up in the nodes (there will only be empty CCK blocks).

The changes to the core did cause some new problems for me though. I have a CCK text field with tags, like "car, motor, vehicle, speed, drive, etc, etc". D7 couldn't migrate them because they didn't fit (I think the strings were too long for the targeted MySQL field).

Another problem I had was with a "List (text)" field, "Select list" widget. I couldn't resave the CCK field because of an error when clicking edit:

Fatal error: Call to undefined function list_field_schema() in C:\xampp\htdocs\modules\cck\cck.module on line 117

Not sure if this error is caused by D7 core or broken data..

I just had success with Amazon asin data from 6 to 7 doing with using the process just as it is intended. Didn't have to do anything extra:

1. Run update.php to get to D7
2. Enable asin (the module in Amazon package that provides a CCK field)
3. Run admin/structure/content_migrate
4. Update the content type in question.

I only tried doing the one content type at a time, but when I did it it went swimmingly.

I think the (repeatable) #fail in #8 was doing a content type that had been provided in code by a feature. The feature is not in D7 (because there are no features yet)

Yay for success!

nm

To everybody: If you find a specific, repeatable failure, I think it's important to open an issue against that issue. This is a crucially important module and is key to the upgrade path. (IMO these should be core issues, but it's not happening).

These are my results with the latest D7 release:

• Needed to set every 'etid' in field_data_field_..... to 2 instead of 1 in mysql
• Needed to resave every field (structure) to make the actual data visible
• Can't resave or edit a Text Select List (see error 1 below). Data shows, but doesn't work in Views
• Error migrating text field, too long (see error 2 below)

1.
Fatal error: Call to undefined function list_field_schema() in C:\xampp\htdocs\modules\cck\cck.module on line 117

2.

Data could not be migrated for field field_ctags in bundle video.
exception 'PDOException' with message 'SQLSTATE[HY000]: General error: 1406 Data too long for column 'field_ctags_value' at row 1248' in C:\xampp\htdocs\includes\database\database.inc:1961 Stack trace: #0 C:\xampp\htdocs\includes\database\database.inc(1961): PDOStatement->execute(Array) #1 C:\xampp\htdocs\includes\database\database.inc(559): DatabaseStatementBase->execute(Array, Array) #2 C:\xampp\htdocs\includes\database\mysql\query.inc(37): DatabaseConnection->query('INSERT INTO {fi...', Array, Array) #3 C:\xampp\htdocs\modules\cck\modules\content_migrate\content_migrate.admin.inc(281): InsertQuery_mysql->execute() #4 [internal function]: _content_migrate_batch_process_migrate_data('field_ctags', Array) #5 C:\xampp\htdocs\includes\batch.inc(286): call_user_func_array('_content_migrat...', Array) #6 C:\xampp\htdocs\includes\batch.inc(163): _batch_process() #7 C:\xampp\htdocs\includes\batch.inc(81): _batch_do() #8 C:\xampp\htdocs\modules\system\system.admin.inc(2370): _batch_page() #9 [internal function]: system_batch_page() #10 C:\xampp\htdocs\includes\menu.inc(480): call_user_func_array('system_batch_pa...', Array) #11 C:\xampp\htdocs\index.php(22): menu_execute_active_handler() #12 {main}

@cheek, please provide more explicit details.

How about a feature that contains the CCK type(s) you're updating (http://drupal.org/project/features)?

How about a link to data that can be loaded into it?

Can you recreate this #fail with a simple bit of data (like one node)? Or do specific nodes fail?

Thanks for your work on this.

-Randy

I found there's actually a patch for 1. (although not 100%, but it fixes the big crash) at http://drupal.org/node/906358

I'm not using Features. I have several node types. Only one node type has problems. It's a node that contains meta data for videos. One field, containing tags for the video, causes the error. In my mysql D6 database it's a 'longtext'.

An example value is: "reporter, kevin newspaper, macdonald, Russell Crowe, Ben Affleck, Rachel McAdams, Helen Mirren, Robin Wright, Jason Bateman, Jeff Daniels, Michael Berresse, Harry Lennix"

@Cheek, I suggested features because it's the easiest way to package up a cck type. But an export of the content type into a text file, attached to the issue, would help people to actually try this out.

Can you recreate the fail with *one* node that has a set of values?

Please provide a CCK type export, and a set of values that causes the failure. (Attached as files to the issue, not pasted in)

Thanks!

Sorry, I don't know which video is causing the problem, I have about 1500 videos.

The problem is pretty straightforward though. The text field is set to 'longtext' in the D6 database. The migration module is trying to copy the data into 'varchar(255)' field and it doesn't fit because it's smaller.

Probably line 319:
$field_value['settings']['max_length'] = 255;

I'll run a php script to substr() my fields to 255 and run the upgrade again to test when I have the time..

I upgraded a simple CCK test site and only one of three text fields migrated successfully. Here are the details:

I created a site with Drupal 6.19 and CCK as the only contributed module (except devel to generate content). I created a content type with three text fields (all required), one using a textfield, one using a select list and one using checkboxes. I used devel to generate 50 nodes of that content type.

I upgrade to Drupal 7: disabled cck and devel, removed all Drupal 6 files and dropped in Drupal 7 HEAD files, except for sites/default directory. Added CCK 7.x-2.x-dev and ran update.php. Enabled CCK and Fields and ran update.php again.

Then I tried to migrate the three fields (Admin->Structure->Migrate fields). Only the first field (using checkboxes) showed up in the nodes. I rolled back all three fields and tried again one at a time. Still only the first field showed up in the nodes, but in the create/edit form the second field (using select list) showed up also (with select list populated). The third field with the simple textfield doesn't show up anywhere. This message comes up on the creat/edit form:

* Notice: Undefined index: description in field_multiple_value_form() (line 146 of C:\wamp\www\guinness3\modules\field\field.form.inc).
* Notice: Undefined index: required in field_multiple_value_form() (line 164 of C:\wamp\www\guinness3\modules\field\field.form.inc).
* Notice: Undefined index: required in field_multiple_value_form() (line 191 of C:\wamp\www\guinness3\modules\field\field.form.inc).

Possibly relevant?: the field that did not migrate at all has a Drupal 7 'Field type' of 'text', while the other two have a Drupal 7 'Field type' of 'list_text'.

Same error after "clean d6" > "d7" upgrade, added issue to drupal core http://drupal.org/node/931512

I've just attempted to migrate some fields with no luck. The field itself migrates across ok, but without any data:

When migrating an integer field:

An AJAX HTTP error occurred. HTTP Result Code: 200 Debugging information follows. Path: /batch?id=8115&op=do StatusText: OK ResponseText: Fatal error: Unsupported operand types in /var/aegir/drupal7b/modules/field/field.crud.inc on line 793

When migrating a text field:

Data could not be migrated for field field_name_first in bundle associatecorporate.
exception 'PDOException' with message 'SQLSTATE[42S22]: Column not found: 1054 Unknown column 'old.delta' in 'field list'' in /var/aegir/drupal7b/includes/database/database.inc:1967 Stack trace: #0 /var/aegir/drupal7b/includes/database/database.inc(1967): PDOStatement->execute(Array) #1 /var/aegir/drupal7b/includes/database/database.inc(562): DatabaseStatementBase->execute(Array, Array) #2 /var/aegir/drupal7b/includes/database/mysql/query.inc(37): DatabaseConnection->query('INSERT INTO {fi...', Array, Array) #3 /var/aegir/drupal7b/sites/cope7f.kcud.net/modules/cck/modules/content_migrate/content_migrate.admin.inc(281): InsertQuery_mysql->execute() #4 [internal function]: _content_migrate_batch_process_migrate_data('field_name_firs...', Array) #5 /var/aegir/drupal7b/includes/batch.inc(286): call_user_func_array('_content_migrat...', Array) #6 /var/aegir/drupal7b/includes/batch.inc(163): _batch_process() #7 /var/aegir/drupal7b/includes/batch.inc(81): _batch_do() #8 /var/aegir/drupal7b/modules/system/system.admin.inc(2303): _batch_page() #9 [internal function]: system_batch_page() #10 /var/aegir/drupal7b/includes/menu.inc(480): call_user_func_array('system_batch_pa...', Array) #11 /var/aegir/drupal7b/index.php(22): menu_execute_active_handler() #12 {main}

Migrating long text:

Data could not be migrated for field field_follow_up in bundle case.
exception 'PDOException' with message 'SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '1-798-0-0-und' for key 'PRIMARY'' in /var/aegir/drupal7b/includes/database/database.inc:1967 Stack trace: #0 /var/aegir/drupal7b/includes/database/database.inc(1967): PDOStatement->execute(Array) #1 /var/aegir/drupal7b/includes/database/database.inc(562): DatabaseStatementBase->execute(Array, Array) #2 /var/aegir/drupal7b/includes/database/mysql/query.inc(37): DatabaseConnection->query('INSERT INTO {fi...', Array, Array) #3 /var/aegir/drupal7b/sites/cope7f.kcud.net/modules/cck/modules/content_migrate/content_migrate.admin.inc(281): InsertQuery_mysql->execute() #4 [internal function]: _content_migrate_batch_process_migrate_data('field_follow_up', Array) #5 /var/aegir/drupal7b/includes/batch.inc(286): call_user_func_array('_content_migrat...', Array) #6 /var/aegir/drupal7b/includes/batch.inc(163): _batch_process() #7 /var/aegir/drupal7b/includes/batch.inc(81): _batch_do() #8 /var/aegir/drupal7b/modules/system/system.admin.inc(2303): _batch_page() #9 [internal function]: system_batch_page() #10 /var/aegir/drupal7b/includes/menu.inc(480): call_user_func_array('system_batch_pa...', Array) #11 /var/aegir/drupal7b/index.php(22): menu_execute_active_handler() #12 {main}

Boolean:

Data could not be migrated for field field_invoice_paid in bundle associatecorporate.
exception 'PDOException' with message 'SQLSTATE[42S22]: Column not found: 1054 Unknown column 'old.delta' in 'field list'' in /var/aegir/drupal7b/includes/database/database.inc:1967 Stack trace: #0 /var/aegir/drupal7b/includes/database/database.inc(1967): PDOStatement->execute(Array) #1 /var/aegir/drupal7b/includes/database/database.inc(562): DatabaseStatementBase->execute(Array, Array) #2 /var/aegir/drupal7b/includes/database/mysql/query.inc(37): DatabaseConnection->query('INSERT INTO {fi...', Array, Array) #3 /var/aegir/drupal7b/sites/cope7f.kcud.net/modules/cck/modules/content_migrate/content_migrate.admin.inc(281): InsertQuery_mysql->execute() #4 [internal function]: _content_migrate_batch_process_migrate_data('field_invoice_p...', Array) #5 /var/aegir/drupal7b/includes/batch.inc(286): call_user_func_array('_content_migrat...', Array) #6 /var/aegir/drupal7b/includes/batch.inc(163): _batch_process() #7 /var/aegir/drupal7b/includes/batch.inc(81): _batch_do() #8 /var/aegir/drupal7b/modules/system/system.admin.inc(2303): _batch_page() #9 [internal function]: system_batch_page() #10 /var/aegir/drupal7b/includes/menu.inc(480): call_user_func_array('system_batch_pa...', Array) #11 /var/aegir/drupal7b/index.php(22): menu_execute_active_handler() #12 {main}
Data could not be migrated for field field_invoice_paid in bundle associatemember.

list_text

Data could not be migrated for field field_resolution_0 in bundle case.
exception 'PDOException' with message 'SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '1-798-0-0-und' for key 'PRIMARY'' in /var/aegir/drupal7b/includes/database/database.inc:1967 Stack trace: #0 /var/aegir/drupal7b/includes/database/database.inc(1967): PDOStatement->execute(Array) #1 /var/aegir/drupal7b/includes/database/database.inc(562): DatabaseStatementBase->execute(Array, Array) #2 /var/aegir/drupal7b/includes/database/mysql/query.inc(37): DatabaseConnection->query('INSERT INTO {fi...', Array, Array) #3 /var/aegir/drupal7b/sites/cope7f.kcud.net/modules/cck/modules/content_migrate/content_migrate.admin.inc(281): InsertQuery_mysql->execute() #4 [internal function]: _content_migrate_batch_process_migrate_data('field_resolutio...', Array) #5 /var/aegir/drupal7b/includes/batch.inc(286): call_user_func_array('_content_migrat...', Array) #6 /var/aegir/drupal7b/includes/batch.inc(163): _batch_process() #7 /var/aegir/drupal7b/includes/batch.inc(81): _batch_do() #8 /var/aegir/drupal7b/modules/system/system.admin.inc(2303): _batch_page() #9 [internal function]: system_batch_page() #10 /var/aegir/drupal7b/includes/menu.inc(480): call_user_func_array('system_batch_pa...', Array) #11 /var/aegir/drupal7b/index.php(22): menu_execute_active_handler() #12 {main}

@cnolle, please see #12. It's a bug that you have to do this, but visit and re-save the content type and see if your migrated data appears.

Hey,

I've have just tried the procedure described in #12:

1. Migrate field
2. Go to content type and re-save (field does not appear)
3. Edit field individually and re-save (field appears but without data)

Using D7 head and latest CCK.

Having same "Integrity constraint violation" error when migrating fields. Fields are created OK, but no data is moved over. Saving content type and field (as suggest in #12) does not correct this.

So if I'm looking at this correctly if you have a custom widget type you'll need to invoke hook_content_migrate_field_alter() to properly set the field type from say text to list_text if it needs changing? But you wouldn't need to invoke hook_content_migrate_instance_alter() because the widget module isn't changing?

Here's my first testing experience migrating data using Content Migrate:

I created a test site, imported all of my nodes (17k +) and cck data in to mimic my live site. Followed the instructions in Drupal 7's update.txt. Ran the update.php. The site upgraded without a problem.

Deleted my disabled Drupal 6 modules. Uploaded cck 7.x-2.x-dev. Enabled Content Migrate. Also uploaded and enabled contributed modules Link 7.x-1.0-alpha2 and Email 7.x-1.x-dev.

My cck field types from Drupal 6 included text, list_text, list_numeric, list_boolean, number_integer, email, link, content_taxonomy.

Content Migrate transferred all of the above except for link (throwing an error)* and content_taxonomy (no Drupal 7 version of the module exists yet).

As Cheek mentioned in #3, I did not have any data in my fields either. They were in the database but not in the fields when editing my nodes. As #12 mentioned, I then saved each data field again. Then I ran a cron. My data now shows up in the fields and in the edit screen.

Another issue is the 'Number of values' setting has changed from "1" to "Unlimited" in all of my fields. I think this was due to a core change of switching the numeric values of 1 and Unlimited but I can't seem to find that post now.

Thanks to all that are contributing to this.

*Update on Link throwing an error:
In the database: content_node_field and content_node_field_instance tables left over from Drupal 6 have the type and widget type called "link" respectively. In Drupal 7, this is now called "link_field". Once I changed this in the content_node_field and content_node_field_instance tables, I ran Content Migrate again. No more error and all the data made it over. So this is not a Content Migrate issue. Will post to the Link issues about this.

Hi Folks.

I'm the link field maintainer, looking in. As jcarlson34 noted in #28, there's a problem with link because the old field was 'link' and the new field is 'link_field'. This is a needed change, as Drupal core now has a widget named 'link' of it's own, and thus Link module must change. Is there a way to tell content_migrate that the data has moved?

And is there a project page for this? I'm not seeing one, but it seems like something people would need.

subscribe

Anybody having any luck with migrating data to the latest D7 RC3?

I think the cck upgrade path is 100% broken now. Probably some recent changes to the database structure.. :(

just fyi - upgrade docs for 6 > 7 #536854: Review and test the upgrade guide for Drupal 7 are needing to be finished asap, so if there is any info that might be relevant, would be awesome if someone can drop a note over there (especially if you can help write the udgrade docs for cck > field!) i'll keep an eye here as well.

@ comment 29 - the module will need a few such translations for modules that have gone into core. I created an issue for imagefield/filefield a few days ago: #1001410: How to handle fields with custom widgets. (but that is more complicated now that in drupal 7 the image style is a field of its own and not just a widget for filefield).

@ comment 31 - I previously could get a textfield to migrate after commenting out a line in cck Migrate. Now the migration there is also faulty, but that may be due to other tests I have been doing on my test site.

Okay, now that I'm able to update Drupal 7 I can look at this. I ran into several issues, and I was able to get around them to test out content_migrate.

Before I could get content migrate to work:

1. $data undefined in some contexts
2. undefined index: fieldname should be field_name
3. old keyword should be changed to old_table to improve db compatibility
4. etid: etid is now entity_type and the expression should be 'node' as entity_type
5. PDOStatement being serialized because of the try { } ???? I'm not sure if this is happening to everyone, but it's happening to me. I wasn't able to fix this, but it should have something to do with invoking __sleep()

After:

1. Field data & edit form won't show up until you specifically set "Number of Values" again and save. The fields I were testing only had 1 value, not multiple or unlimited.
2. Undefined indexes for 'required' and 'description' in function in field_multiple_value_form() field/field.form.inc on lines 156, 178, and 207.

The good news: data integrity for what I tested. text + textfield and list + custom widget. The comment I made in #27 was correct, and I was able to update my widget from D6 to D7 simply changing field_value['type'] and field_value['module']. ;-)

I've attached a patch that should be useful for developers to get things rolling and testing. It completely disables the try/catch statements, which is bad, but we can actually test things out.

Happy New Year!

I don't think throwing an exception is good in batch operations. It might be better to test each field's existence using the Field API, and just use the drupal_set_message() and/or watchdog() for "error" reporting (as it is now, but without a PHP exception).

I think I'm going to work on some automated tests for the entire process next. Maybe textfield, list, number individual migration test + multiple field migration test.

Here's the test case / framework I've been working on. I tried to make it as flexible as possible so others could extend the base test case with what they need for filefield, nodereference, etc... Many of the methods would need to be overridden and call parent for those cases.

The basic process as follows. Hopefully it makes sense and is helpful.

• add content_node_field, content_node_field_instance, and content_type_TYPENAME after creating TYPENAME. I have. The former are brought in by a schema method.
• The actual test case, in this example text, will handle actually inserting field, instance, and data into those respective tables (as well as creating a node). The base test case has public methods that should be used to do this.

Note: It's not complete or working fully at the moment, but I thought I'd upload it. createFieldData() isn't written at all, and createField() doesn't work. The drupal_write_record() statement to insert into content_node_field isn't working, and I'm having trouble finding out why. I'll probably figure it out or do it differently after I post this :P. The tearDown function I tried to write doesn't work so created tables aren't being cleaned up properly after a test.

Edit: I should also note I haven't written any tests for the actual migration of fields yet!

Oh, duh, I can't use drupal_write_record() because I don't have a schema. :)

This file has a working setUp() for text field with a textfield widget. I'll create a basic test to run through the upgrade on it next.

Here's one with a basic, "working" text field migration test. Also, fixed a bug I had where I forgot how cck values were stored in content_type_TYPE.

In this test we can see that the field was migrated, showing up in field ui admin page, but does not show up on the node. It also has some exceptions being raised in field_ui from the migration ;-)

Edit: don't forget to put a mention of this test in content_migrate.info when you're testing.

Yikes, that's quite a work around (happened to be back at this computer and checked the issue).

Yeah, it would probably be a lot of work to create every single edge case migration. However, I think some initial tests would help get content_migrate working again.

The big issue after what you fixed today is that the settings aren't being migrated quite right anymore - display settings and possibly the required and description are undefined in some cases.

7 fails down to 1 with your latest commit!

The last fail compares the "multiple" setting to what it should be. Somehow it's not getting set correctly.

This also tests multiple values and going through multiple nodes for one field.

Confirming the allowed values fix as well. Also, I got my widget to go through perfectly with the alter hook per comment #27.

Nice job, Karen!

Test file contains basic select list testing.

With the latest dev version I am now managing to migrate fields some text fields without any error messages.

When it doesn't work (on text fields) I get one of the following errors:

Duplicate error

An AJAX HTTP error occurred. HTTP Result Code: 500 Debugging information follows. Path: /batch?id=8287&op=do StatusText: Service unavailable (with message) ResponseText: PDOException: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'node-798-0-0-und' for key 'PRIMARY': INSERT INTO {field_data_field_advice} (entity_id, revision_id, field_advice_value, field_advice_format, bundle, entity_type, language, delta) SELECT old_table.nid AS entity_id, old_table.vid AS revision_id, old_table.field_advice_value AS field_advice_value, old_table.field_advice_format AS field_advice_format, 'case' AS bundle, 'node' AS entity_type, 'und' AS language, 0 AS delta FROM {content_type_case} old_table; Array ( ) in _content_migrate_batch_process_migrate_data() (line 282 of /Users/cnolle/Desktop/dev/cope7/sites/all/modules/cck/modules/content_migrate/content_migrate.admin.inc).

or

Column not found error

An AJAX HTTP error occurred. HTTP Result Code: 500 Debugging information follows. Path: /batch?id=8289&op=do StatusText: Service unavailable (with message) ResponseText: PDOException: SQLSTATE[42S22]: Column not found: 1054 Unknown column 'old_table.delta' in 'field list': INSERT INTO {field_data_field_post_code} (entity_id, revision_id, field_post_code_value, delta, bundle, entity_type, language) SELECT old_table.nid AS entity_id, old_table.vid AS revision_id, old_table.field_post_code_value AS field_post_code_value, old_table.delta AS delta, 'associatecorporate' AS bundle, 'node' AS entity_type, 'und' AS language FROM {content_field_post_code} old_table; Array ( ) in _content_migrate_batch_process_migrate_data() (line 282 of /Users/cnolle/Desktop/dev/cope7/sites/all/modules/cck/modules/content_migrate/content_migrate.admin.inc).

I am only migrating text fields. I hope this is useful.

These fields are in use across multiple content type. I haven't been able to migrate any fields across that appear in multiple.

Here's an export from the field_post_code which failed migrating (used on 5 content types).

Sorry, that's my fault. I thought I could change the function definition in a sub-class. I'll have to rework things.

Another good thing: I've been testing with both mysql and postgresql and there aren't any problems in the latter so far. ;-)

subscribe

It works soft of half way, data is migrated for 2 out of 5 content types.

An AJAX HTTP error occurred. HTTP Result Code: 500 Debugging information follows. Path: /batch?id=8293&op=do StatusText: Service unavailable (with message) ResponseText: PDOException: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'node-2383-0-0-und' for key 'PRIMARY': INSERT INTO {field_data_field_post_code} (entity_id, revision_id, field_post_code_value, bundle, entity_type, language, delta) SELECT old_table.nid AS entity_id, old_table.vid AS revision_id, old_table.field_post_code_value AS field_post_code_value, 'continuouspublication' AS bundle, 'node' AS entity_type, 'und' AS language, 0 AS delta FROM {content_field_post_code} old_table INNER JOIN {node} n ON old_table.nid=n.nid WHERE (type = :db_condition_placeholder_0) ; Array ( [:db_condition_placeholder_0] => continuouspublication ) in _content_migrate_batch_process_migrate_data() (line 290 of /Users/cnolle/Desktop/dev/cope7/sites/all/modules/cck/modules/content_migrate/content_migrate.admin.inc).

The above field (field_post_code) is shared among four content types. The field migrated fine into the first two content types but failed in the third one. It's using the Text Field widget in all four. The field is configured in the same across all content types.

---

With other fields (even non-shared ones I am still getting errors messages like this one. This field is also using text field widget (D6).

An AJAX HTTP error occurred. HTTP Result Code: 500 Debugging information follows. Path: /batch?id=8275&op=do StatusText: Service unavailable (with message) ResponseText: PDOException: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'node-2447-0-0-und' for key 'PRIMARY': INSERT INTO {field_data_field_invoice_name_first} (entity_id, revision_id, field_invoice_name_first_value, bundle, entity_type, language, delta) SELECT old_table.nid AS entity_id, old_table.vid AS revision_id, old_table.field_invoice_name_first_value AS field_invoice_name_first_value, 'journalinfo' AS bundle, 'node' AS entity_type, 'und' AS language, 0 AS delta FROM {content_type_journalinfo} old_table INNER JOIN {node} n ON old_table.nid=n.nid WHERE (type = :db_condition_placeholder_0) ; Array ( [:db_condition_placeholder_0] => journalinfo ) in _content_migrate_batch_process_migrate_data() (line 290 of /Users/cnolle/Desktop/dev/cope7/sites/all/modules/cck/modules/content_migrate/content_migrate.admin.inc).

I have had success with other field (non shared ones) - but I can't quite work why some work and other don't. These are all text fields.

I am attaching the content copy of a field where migration worked and one where it didn't. I hope this is helpful. Let me know if you want any further info.

subscribing

I don't think there's any code to update anything other than standard cck fields (minuse node & user reference).

Speaking of node/user reference. Is the upgrade path going to be dependent on 'references' module? I modified the test stuff to make it easier to write something for node reference, but I can't run through it because "nodereference" changed to "node_reference". I need to backtrack some changes out so I just have a patch to fix the bug above and a patch for all this new stuff. Hopefully, that's what I'll be doing at my Sunday Afternoon of Drupal meetup.

Should there be a content_migrate_update_N to make changes that these new contrib modules have done? Or is that up to the new contrib modules in their own hook_update_N?

I've now tried to migrate the same fields after removing all fieldsgroups (I removed these in D6, then ran the upgrade again) but I still get the same error with the same fields as above. Not sure what the issue is here - some of these fields are rather large (i.e. appear in a few thousands nodes across multiple content types). I've haven't got any contrib modules installed (except CCK migrate).

@KarenS: Thanx for your hard work with the migration issues. Yeah when the imagefield migration is working I'll guess that we allmost get the filefield migration for "free". I'm afraid that I'm out of my depth helping out with the migration code but let me know when you need some help testing it, I'll guess there a lot of people following this thread that are more then happy to do just that :)

@KarenS As @steinmb has pointed out above, thank you for your ongoing work with the migration issues. I imagine a lot people will be using it once its all up and working.

I was wondering if it could something to do with data integrity that some fields fail to migrate from 6 to 7? I am scratching my head trying to work out why this is happening. Is there something that should/could be done with the data in D6 before one attempts at migrating it it in 7?

The batch APi is sufficiently awful that I just use drush scripts which are not subject to timeout. If you have trouble with batch, I suggest you just go for a drush upgrade path. I know that some people can't use that on their current hosting, but we have zero upgrade path today so solving the problem for 80% is pretty valuable. Some folks on shitty web hosts without ssh could do the upgrade on own machines or elsewhere. There is also drush terminal project.

Some updates based on the latest code.

With this new code I am now able to import a few more fields. In this instance the field is created in all content types - but the data fails to migrate.
Before this latest update the field was only created in 2 out of 5 content types.

An AJAX HTTP error occurred. HTTP Result Code: 500 Debugging information follows. Path: /batch?id=8225&op=do StatusText: Service unavailable (with message) ResponseText: PDOException: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'node-2356-0-0-und' for key 'PRIMARY': INSERT INTO {field_data_field_city} (entity_type, bundle, entity_id, revision_id, language, delta, field_city_value) VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1, :db_insert_placeholder_2, :db_insert_placeholder_3, :db_insert_placeholder_4, :db_insert_placeholder_5, :db_insert_placeholder_6); Array ( [:db_insert_placeholder_0] => node [:db_insert_placeholder_1] => honorarymember [:db_insert_placeholder_2] => 2356 [:db_insert_placeholder_3] => 2691 [:db_insert_placeholder_4] => und [:db_insert_placeholder_5] => 0 [:db_insert_placeholder_6] => Sussex ) in drupal_write_record() (line 6776 of /Users/cnolle/Desktop/dev/cope7/includes/common.inc).

Here's another error message for a similar field, all of these are text fields.

An AJAX HTTP error occurred. HTTP Result Code: 500 Debugging information follows. Path: /batch?id=8226&op=do StatusText: Service unavailable (with message) ResponseText: PDOException: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'node-2356-0-0-und' for key 'PRIMARY': INSERT INTO {field_data_field_position} (entity_type, bundle, entity_id, revision_id, language, delta, field_position_value) VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1, :db_insert_placeholder_2, :db_insert_placeholder_3, :db_insert_placeholder_4, :db_insert_placeholder_5, :db_insert_placeholder_6); Array ( [:db_insert_placeholder_0] => node [:db_insert_placeholder_1] => honorarymember [:db_insert_placeholder_2] => 2356 [:db_insert_placeholder_3] => 2691 [:db_insert_placeholder_4] => und [:db_insert_placeholder_5] => 0 [:db_insert_placeholder_6] => ) in drupal_write_record() (line 6776 of /Users/cnolle/Desktop/dev/cope7/includes/common.inc).

I hope these are useful. Let me know if I can be of any other help.

Did some quick tests:
PHP Version 5.2.10
Drupal 7.0
CCK-7 dev

On loading "admin/structure/content_migrate" do I get:

    * Notice: Undefined index: inline in content_migrate_content_migrate_instance_alter() (line 577 of /Users/steinmb/htdocs/sites/elmcip.net/drupal-7.0/sites/all/modules/cck/modules/content_migrate/content_migrate.module).
    * Notice: Undefined index: inline in content_migrate_content_migrate_instance_alter() (line 578 of /Users/steinmb/htdocs/sites/elmcip.net/drupal-7.0/sites/all/modules/cck/modules/content_migrate/content_migrate.module).
    * Notice: Undefined index: inline in content_migrate_content_migrate_instance_alter() (line 577 of /Users/steinmb/htdocs/sites/elmcip.net/drupal-7.0/sites/all/modules/cck/modules/content_migrate/content_migrate.module).
    * Notice: Undefined index: inline in content_migrate_content_migrate_instance_alter() (line 578 of /Users/steinmb/htdocs/sites/elmcip.net/drupal-7.0/sites/all/modules/cck/modules/content_migrate/content_migrate.module).
    * Notice: Undefined index: inline in content_migrate_content_migrate_instance_alter() (line 577 of /Users/steinmb/htdocs/sites/elmcip.net/drupal-7.0/sites/all/modules/cck/modules/content_migrate/content_migrate.module).
    * Notice: Undefined index: inline in content_migrate_content_migrate_instance_alter() (line 578 of /Users/steinmb/htdocs/sites/elmcip.net/drupal-7.0/sites/all/modules/cck/modules/content_migrate/content_migrate.module).
    * Notice: Undefined index: inline in content_migrate_content_migrate_instance_alter() (line 577 of /Users/steinmb/htdocs/sites/elmcip.net/drupal-7.0/sites/all/modules/cck/modules/content_migrate/content_migrate.module).
    * Notice: Undefined index: inline in content_migrate_content_migrate_instance_alter() (line 578 of /Users/steinmb/htdocs/sites/elmcip.net/drupal-7.0/sites/all/modules/cck/modules/content_migrate/content_migrate.module).
    * Notice: Undefined index: inline in content_migrate_content_migrate_instance_alter() (line 577 of /Users/steinmb/htdocs/sites/elmcip.net/drupal-7.0/sites/all/modules/cck/modules/content_migrate/content_migrate.module).
    * Notice: Undefined index: inline in content_migrate_content_migrate_instance_alter() (line 578 of /Users/steinmb/htdocs/sites/elmcip.net/drupal-7.0/sites/all/modules/cck/modules/content_migrate/content_migrate.module).

If I try a field migration it self goes up in flame with the progress bar at 1%:

[12-Jan-2011 11:03:03] PHP Fatal error:  Unsupported operand types in /Users/steinmb/htdocs/sites/elmcip.net/drupal-7.0/modules/field/field.crud.inc on line 601
[12-Jan-2011 11:03:03] PHP Stack trace:
[12-Jan-2011 11:03:03] PHP   1. {main}() /Users/steinmb/htdocs/sites/elmcip.net/drupal-7.0/index.php:0
[12-Jan-2011 11:03:03] PHP   2. menu_execute_active_handler() /Users/steinmb/htdocs/sites/elmcip.net/drupal-7.0/index.php:22
[12-Jan-2011 11:03:03] PHP   3. call_user_func_array() /Users/steinmb/htdocs/sites/elmcip.net/drupal-7.0/includes/menu.inc:501
[12-Jan-2011 11:03:03] PHP   4. system_batch_page() /Users/steinmb/htdocs/sites/elmcip.net/drupal-7.0/includes/menu.inc:0
[12-Jan-2011 11:03:03] PHP   5. _batch_page() /Users/steinmb/htdocs/sites/elmcip.net/drupal-7.0/modules/system/system.admin.inc:2282
[12-Jan-2011 11:03:03] PHP   6. _batch_do() /Users/steinmb/htdocs/sites/elmcip.net/drupal-7.0/includes/batch.inc:81
[12-Jan-2011 11:03:03] PHP   7. _batch_process() /Users/steinmb/htdocs/sites/elmcip.net/drupal-7.0/includes/batch.inc:162
[12-Jan-2011 11:03:03] PHP   8. call_user_func_array() /Users/steinmb/htdocs/sites/elmcip.net/drupal-7.0/includes/batch.inc:285
[12-Jan-2011 11:03:03] PHP   9. _content_migrate_batch_process_create_fields() /Users/steinmb/htdocs/sites/elmcip.net/drupal-7.0/includes/batch.inc:0
[12-Jan-2011 11:03:03] PHP  10. field_create_instance() /Users/steinmb/htdocs/sites/elmcip.net/drupal-7.0/sites/all/modules/cck/modules/content_migrate/content_migrate.admin.inc:216
[12-Jan-2011 11:03:03] PHP  11. _field_write_instance() /Users/steinmb/htdocs/sites/elmcip.net/drupal-7.0/modules/field/field.crud.inc:491
 

Edit:
Had a seeking suspicion that going to manage fields and press save could fixe some of the obow problems and it did. Now it tried to convert the fields and failed with:

An AJAX HTTP error occurred. HTTP Result Code: 500 Debugging information follows. Path: /elmcip.net/drupal-7.0/batch?id=193&op=do StatusText: Service unavailable (with message) ResponseText: PDOException: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'node-47-0-0-und' for key 'PRIMARY': INSERT INTO {field_data_field_journalissue} (entity_type, bundle, entity_id, revision_id, language, delta, field_journalissue_value) VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1, :db_insert_placeholder_2, :db_insert_placeholder_3, :db_insert_placeholder_4, :db_insert_placeholder_5, :db_insert_placeholder_6); Array ( [:db_insert_placeholder_0] => node [:db_insert_placeholder_1] => critical_writing [:db_insert_placeholder_2] => 47 [:db_insert_placeholder_3] => 95 [:db_insert_placeholder_4] => und [:db_insert_placeholder_5] => 0 [:db_insert_placeholder_6] => 39/2009 ) in drupal_write_record() (line 6776 of /Users/steinmb/htdocs/sites/elmcip.net/drupal-7.0/includes/common.inc).

Edit II:
And I confirm that the field_* was created but non of the data was.

Upgraded D6.20 installation
PHP Version 5.2.10
Drupal 7.0
CCK-7 dev
Date & Calendar 7.x dev.

Migrated the date fields (field_date, datetime) and content with the cck migration tool with no errors, but node creation using the converted field give me WSOD though view and edit of existing node is ok. Created a separate bug on this in the Date issue q. #1024140: Migrated fields: WSOD "Cannot use string offset as an array"

After much suffering, I'm able to migrate my image fields from D6 to D7. Here's what I did:

On my D6 site:
- migrated my image nodes to filefield by following this page: http://drupal.org/node/432860
(the new content type I created was ImagePage, I named the filefield field_imagefield)
- disabled and uninstalled every module related to images
- renamed my database to "drupal7"

On my D7 site:
- run update.php
- enabled the Image module
- downloaded CCK and enabled the content_migrate module
- browsed to /admin/structure/content_migrate and migrated field_imagefield

This last step appears to migrate everything except for the file info in table "files". Here's a quick script I used to migrate the info from "files" to "file_managed" and "file_usage":

<?php

mysql_connect('localhost','drupal7','password');
mysql_select_db('drupal7');

$sql = 'SELECT * FROM field_data_field_imagefield';
$result = mysql_query($sql);
echo $result;
while ($row = mysql_fetch_object($result)) {
        $nid = $row->entity_id;
        $fid = $row->field_imagefield_fid;
        echo "Migrating $nid $fid<br />";

        $sql = "SELECT * FROM files WHERE fid=$fid";
        $result2 = mysql_query($sql);
        $row2 = mysql_fetch_object($result2);

        $sql = "INSERT INTO file_managed SET
                fid=$fid,
                uid=1,
                filename='{$row2->filename}',
                uri='public://images/{$row2->filename}',
                filemime='{$row2->filemime}',
                filesize={$row2->filesize},
                status=1,
                timestamp={$row2->timestamp}";
        mysql_query($sql);
        $sql = "INSERT INTO file_usage SET
                fid=$fid,
                module='file',
                type='node',
                id=$nid,
                count=1";
        mysql_query($sql);
}
?>

I put this in a file named convert.php and run it. Then, poof, images appear in D7 as they did in D6. Hopefully someone can incorporate the gist of this into content_migrate.

sub

Reading through the imagefield upgrade code and saw this comment.

// What is the difference between alt and custom_alt on the old field?

In D6.

'custom_alt' Is the flag that determines wether or not the user should be shown an alt text field when adding a new image to an imagefield.
'alt' is the optional default value to be used when there is no user entered alt text.

Looks promising. I am going to test upgrade a rather large site with thousands of imagefields. Will see how it goes.

I've tried it, but from what I see the filefield migration is not ready yet. So what happens now: imagefield migrated just fine, but there are no images (i.e. files which were handled by filefield).

#87 solution did't work for me unfortunately.

That script was a great start. I fixed a few issues that I had with it. I also converted for use after drupal bootstrap to make life easier for everyone.

Put this in a file called 'fix_images.php' in the drupal folder, change the $field_name variable to the right value, then run drush php-script fix_images.php

I'm going to see if I can come up with a patch for content_migration, but for now just wanted to post this.

$field_name = 'field_story_image';

$field_table = 'field_data_' . $field_name;
$field_fid_column = $field_name . '_fid';

$results = db_select($field_table, 't')->fields('t', array('entity_id', $field_fid_column))->isNotNull($field_fid_column)->execute();

while ($row = $results->fetchObject()) {
  $nid = $row->entity_id;
  $fid = $row->$field_fid_column;
  echo "Migrating $nid - $fid\n";

  $file = db_select('files', 'f')->fields('f', array('fid', 'uid', 'filename', 'filepath', 'filemime', 'filesize', 'timestamp'))->condition('fid', $fid)->execute()->fetchObject();

// Ignore this, just in case you need to rerun this script because something failed :P
//  db_delete('file_managed')->condition('fid', $fid)->execute();
//  db_delete('file_usage')->condition('fid', $fid)->execute();

  $public_path = str_replace('sites/default/files/', '', $file->filepath);

  db_insert('file_managed')->fields(array(
    'fid'       => $fid,
    'uid'       => $file->uid,
    'filename'  => $file->filename,
    'uri'       => "public://$public_path",
    'filemime'  => $file->filemime,
    'filesize'  => $file->filesize,
    'status'    => FILE_STATUS_PERMANENT,
    'timestamp' => $file->timestamp,
  ))->execute();

  db_insert('file_usage')->fields(array(
    'fid'   => $fid,
    'module'=> 'file',
    'type'  => 'node',
    'id'    => $nid,
    'count' => 1,
  ))->execute();
}

Great to see things moving here. Badly disguised subsribe post....

@loganfsmyth great! #92 is almost there in my case. The script worked fine except for (I'm not sure if it is just me or more general issue): in my D6 db file path is stored as files:filepath "files/blog/some.jpg".

So at the end in D7 we would have "public://files/blog/some.jpg" which is rendered img src="files/files/blog/some.jpg". And it's not right of course.

From what I noticed D7 upgrade itself would store it as "public://blog/some.jpg" i.e. without files/ prefix.

I'm still using the old school ./files instead of /sites/default/files due to many reasons.

Any ideas?

Here is a patch to cck that uses hook_content_migrate_data_record_alter.

This patch also makes it possible to set the $record to NULL, so that it will not be saved. Right now migration creates field_data rows even if the fid is NULL. Not the end of the world, but upon viewing a node with a NULL fid, I was getting some errors. Now I've made it just not save rows with NULL fids.

@betarobot, I think this should fix your issue too. This does a better job of checking previous directory settings.

The images get moved over correctly (hooray!) but the alt and title texts get messed up. I think this is due to a mismatch of field columns. In D6, imagefields have fid, list, and data, where data is a serialized array containing the alt and title for that image. D7 image fields have fid, alt, and title columns.

It looks like the columns are copied straight over, so now the Alt text is usually "1", and the Title text is a serialized array. I'll see if I can fix that in content_migrate_content_migrate_data_record_alter().

I'm not sure if the data column always has serialized content with different widget settings, so I tried to account for that. If that array has more in it than 'alt' and 'title', it'll be lost because there really isn't anywhere for it to go. Personally, I think that's a wild edge case and not worth worrying about, but it could be an issue.

Just a report on patch at #95: successfully updated a couple of very different sites (one d5 via d6 basic upgrade). All worked very well! Great job!

Though can't say anything for #97 as I din't have any alt tags submitted.

@betarobot:
Great to hear upgrades work. But as filefields dont yet get upgraded and imagefield depends on filefield for the data storage in later D6 CCK versions, isn't that missing to really get imagefields upgraded?

@eigentor not quite so: since some point in this thread you can perfectly upgrade D6 imagefield. The point at #87 was that imagefield is migrated just fine but filefield is not (kudos for @mattbatt for the way trough). So you'd have your CCK fields but no actual images(files) connected.

Current situations is: all CCK images are properly migrated with #92 (worked for me, or maybe #98) patch with imagefield and filefield properties. So Thomas, give it a try :)

The only thing I can't say/test yet is (custom?) alt attribute for imagefield as stated #97. Just don't have any.

@eigentor, content_migrate_content_migrate_field_alter() changes the field type to 'image' when the *widget* type's module is imagefield, so all of the subsequent alters look for 'image' in $field['type']. It shouldn't be too hard to do parallel alters to turn 'filefields' into 'file' fields since the storage is basically the same. I guess I'll have to mock up some filefields to test it out.

Here are imagefields and filefields getting migrated as properly as I can see.

Yeah, those tables are a bit annoying, but I usually delete and recreate the db to run.

I'm sorry I haven't looked at the tests recently. I've been meaning to for 2 weeks, but I haven't had a chance to look over things.

Great progress on this, I can now successfully migrate some imagefields. Unfortunately I am still getting the same error message on the same text fields that I've always had a problem with, see below. This is running the field migration with the latest dev version of CCK. What could cause this?

An AJAX HTTP error occurred. HTTP Result Code: 500 Debugging information follows. Path: /batch?id=8287&op=do StatusText: Service unavailable (with message) ResponseText: PDOException: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'node-2356-0-0-und' for key 'PRIMARY': INSERT INTO {field_data_field_post_code} (entity_type, bundle, entity_id, revision_id, language, delta, field_post_code_value) VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1, :db_insert_placeholder_2, :db_insert_placeholder_3, :db_insert_placeholder_4, :db_insert_placeholder_5, :db_insert_placeholder_6); Array ( [:db_insert_placeholder_0] => node [:db_insert_placeholder_1] => honorarymember [:db_insert_placeholder_2] => 2356 [:db_insert_placeholder_3] => 2691 [:db_insert_placeholder_4] => und [:db_insert_placeholder_5] => 0 [:db_insert_placeholder_6] => ) in drupal_write_record() (line 6776 of /Users/cnolle/Desktop/dev/cope7/includes/common.inc).

I should add that this particular CCK field is used across 5 content types.

I had some issues with the References implementation of the instance_alter. I posted an issue over there: #1032808: Undefined index: module in node_reference_content_migrate_instance_alter

~

In my case, textfields converted fine. During imagefield conversion the following error occured:

An AJAX HTTP error occurred. HTTP Result Code: 500 Debugging information follows. Path: /test7/batch?render=overlay&id=727&op=do StatusText: Service unavailable (with message) ResponseText: PDOException: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'public://1.jpg' for key 'uri': INSERT INTO {file_managed} (fid, uid, filename, uri, filemime, filesize, status, timestamp) VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1, :db_insert_placeholder_2, :db_insert_placeholder_3, :db_insert_placeholder_4, :db_insert_placeholder_5, :db_insert_placeholder_6, :db_insert_placeholder_7); Array ( [:db_insert_placeholder_0] => 1256 [:db_insert_placeholder_1] => 1 [:db_insert_placeholder_2] => 1.jpg [:db_insert_placeholder_3] => public://1.jpg [:db_insert_placeholder_4] => image/jpeg [:db_insert_placeholder_5] => 89864 [:db_insert_placeholder_6] => 1 [:db_insert_placeholder_7] => 1233939069 ) in content_migrate_filefield_data_record_alter() (line 163 of /public_html/test7/sites/all/modules/modules/content_migrate/modules/content_migrate.filefield.inc).  

@endrus For that to happen, I think you'd need to have multiple files in your D6 'files' table with the same 'filepath' field. The D7 files_managed table requires uniqueness in 'uri', but D6's equivalent 'filepath' does not. If that is the case, that would cause the issue you are seeing.

Could you confirm if this is the case by running this query and seeing if there are any rows that come up?
SELECT filepath, COUNT(filepath) as count FROM files GROUP BY filepath HAVING count > 1

@loganfsmyth. Ran the query. Here's what I am getting:

filepath 	count
files/001.JPG 	2
files/01.jpg 	2
files/011.JPG 	2
files/012.JPG 	2
files/013.JPG 	2
files/014.JPG 	2
files/03.jpg 	2
files/04.jpg 	2
files/06_0.jpg 	2
files/07_0.jpg 	2
files/1.JPG 	2
files/11.jpg 	2
files/1_1.jpg 	2
files/2.jpg 	2
files/3.JPG 	2
files/3_1.JPG 	2
files/4.jpg 	2
files/5.JPG 	2
files/karova.jpg 	2
files/SSA41746.jpg 	2
files/SSA41766.jpg 	2

Not to be a pest, but I am still getting the same error message. I've checked the version of CCK I was running is indeed the very latest and it is (Timestmap 2011/01/22 14:00:47)

An AJAX HTTP error occurred. HTTP Result Code: 500 Debugging information follows. Path: /batch?id=8284&op=do StatusText: Service unavailable (with message) ResponseText: PDOException: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'node-685-0-0-und' for key 'PRIMARY': INSERT INTO {field_data_field_publisher} (entity_type, bundle, entity_id, revision_id, language, delta, field_publisher_value) VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1, :db_insert_placeholder_2, :db_insert_placeholder_3, :db_insert_placeholder_4, :db_insert_placeholder_5, :db_insert_placeholder_6); Array ( [:db_insert_placeholder_0] => node [:db_insert_placeholder_1] => case [:db_insert_placeholder_2] => 685 [:db_insert_placeholder_3] => 2422 [:db_insert_placeholder_4] => und [:db_insert_placeholder_5] => 0 [:db_insert_placeholder_6] => ) in drupal_write_record() (line 6776 of /Users/cnolle/Desktop/dev/cope7/includes/common.inc).

NOW, this might have something to do with the fact that the CCK fields have travelled from D5 -> D6 and finally to D7. Everything works fine in D6 after the first upgrade.

Here's another one

An AJAX HTTP error occurred. HTTP Result Code: 500 Debugging information follows. Path: /batch?id=8285&op=do StatusText: Service unavailable (with message) ResponseText: PDOException: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'node-2381-0-0-und' for key 'PRIMARY': INSERT INTO {field_data_field_hon_id} (entity_type, bundle, entity_id, revision_id, language, delta, field_hon_id_value) VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1, :db_insert_placeholder_2, :db_insert_placeholder_3, :db_insert_placeholder_4, :db_insert_placeholder_5, :db_insert_placeholder_6); Array ( [:db_insert_placeholder_0] => node [:db_insert_placeholder_1] => continuouspublication [:db_insert_placeholder_2] => 2381 [:db_insert_placeholder_3] => 7723 [:db_insert_placeholder_4] => und [:db_insert_placeholder_5] => 0 [:db_insert_placeholder_6] => CP00002 ) in drupal_write_record() (line 6776 of /Users/cnolle/Desktop/dev/cope7/includes/common.inc).

@endrus That's definitely the source of the problem. I'm not sure what would have caused you to end up with duplicates in the files table. That's not an issue with the migration so much as a problem with your original D6 files.

@cnolle It looks like you have revisions, but if I understand correctly I don't think revisions are supported yet. I might get a chance to fix that, but it'll depend on if I have the time.

@loganfsmyth Maybe there was a problem when we ported from D5 to D6. Would appreciate if you could give me a hint how I could fix duplicates in the Files table.

@loganfsmyth I think that's it, revisions. I just deleted all the revisions for a content type in D6, ran the D7 upgrade on the database, tried migrating some fields that were failing before and voila, they now migrate beautifully. I am now going to go through and delete all revisions across the whole site.

@karens Thank you again for all your work on this.

@loganfsmyth I deleted all duplicates in Files table. However, when I rolled back imagefield and attempted to convert it again, the following error occurred:

An AJAX HTTP error occurred. HTTP Result Code: 500 Debugging information follows. Path: /test7/batch?id=736&op=do StatusText: Service unavailable (with message) ResponseText: PDOException: SQLSTATE[23000]: Integrity constraint violation: 1048 Column 'uid' cannot be null: INSERT INTO {file_managed} (fid, uid, filename, uri, filemime, filesize, status, timestamp) VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1, :db_insert_placeholder_2, :db_insert_placeholder_3, :db_insert_placeholder_4, :db_insert_placeholder_5, :db_insert_placeholder_6, :db_insert_placeholder_7); Array ( [:db_insert_placeholder_0] => [:db_insert_placeholder_1] => [:db_insert_placeholder_2] => [:db_insert_placeholder_3] => public:// [:db_insert_placeholder_4] => [:db_insert_placeholder_5] => [:db_insert_placeholder_6] => [:db_insert_placeholder_7] => ) in content_migrate_filefield_data_record_alter() (line 163 of /public_html/test7/sites/all/modules/modules/content_migrate/modules/content_migrate.filefield.inc).

@endrus Unfortunately just deleting the duplicates won't help because one of the duplicates is associated with one field, and the other one is associated with another.

Here is a patch that I think should fix your issue. It reassigns the field's associated file to use whichever file_managed entry already exists, instead of always trying to insert it's own based only on fid. I'm not sure if this is worth committing or not.

Give it a shot and let me know.

I tried the following: refreshed the conversion page until it was finished despite some mistakes along the way. Images converted fine with few exceptions. The conversion process stopped every time one of the following errors came up: image title is too long, uid is null, imagepath contains a duplicate (for example, 1.JPG and 1.jpg).

Glad to see all the progress here!

@KarenS I see in #35 you had "unset string offsets" problems a while back. I'm not sure if this is still an issue but if it is, a patch in this thread helped me solve that problem: #870444: PHP Fatal error: Cannot unset string offsets in content_migrate.module on line 163.

sub

I've upgraded (6.20 => 7.0) a decent size site, and then recently (within the last three days) installed the latest CCK(7.x-2.x-dev) and attempted to migrate my D6 custom content types and associated fields. It did error out on one shared field, but I was able to re-add that manually, and it appeared to be working fine. Then I noticed that my old content in the original table content_type_meeting did not make it into the new field-specific tables - for instance, one of the fields in content_type_meeting is field_theme_value - while the themes for the old meeting nodes are all still there, none of them made it into the new field table: field_data_field_theme. If I edit one of the old meetings, and then save it, it will populate the entries into the new field-specific tables, but the edit node screen won't populate from the old fields. I'm assuming that if this process had completed correctly, then it would have migrated the custom field content to the new field-specific tables.(?) Is there a way to manually kickstart a subprocess to do this step? Content Migrate says(smugly) it has imported all of these fields successfully and that there is nothing left to do...

@tcowin What type of field is field_theme? Each type of field potentially needs it's own custom handling added via hooks. The basic field types should be supported, but there are still a lot left unhandled.

@loganfsmyth - thanks for the response. This was just a text field. All of this was relatively basic CCK setup - mostly text fields with an integer here and there. It should have all been handled correctly, right? Is there a way to manually (re)run part of it?

Pardon my naivety, but perhaps something as simple as the attached patch is enough to properly handle revisions when updating cck fields? I think it worked for me.

My upgrade actually died on a field of type 'list_number', and I got the error:

# Error creating field field_weight
# exception 'FieldException' with message 'Attempt to create a field of unknown type list_number.' in /usr/local/setac/prod/www-drupal-7/modules/field/field.crud.inc:107 Stack trace: #0 /usr/local/setac/prod/www-drupal-7/sites/all/modules/cck/modules/content_migrate/includes/content_migrate.admin.inc(202): field_create_field(Array) #1 [internal function]: _content_migrate_batch_process_create_fields(Array, Array) #2 /usr/local/setac/prod/www-drupal-7/includes/batch.inc(285): call_user_func_array('_content_migrat...', Array) #3 /usr/local/setac/prod/www-drupal-7/includes/batch.inc(162): _batch_process() #4 /usr/local/setac/prod/www-drupal-7/includes/batch.inc(81): _batch_do() #5 /usr/local/setac/prod/www-drupal-7/modules/system/system.admin.inc(2282): _batch_page() #6 [internal function]: system_batch_page() #7 /usr/local/setac/prod/www-drupal-7/includes/menu.inc(501): call_user_func_array('system_batch_pa...', Array) #8 /usr/local/setac/prod/www-drupal-7/index.php(22): menu_execute_active_handler() #9 {main}

Going back and individually rolling back and then re-migrating the fields one by one seems to be effective, FWIW...

I split some of the errors reported here into their own issues:
#1044122: Cannot migrate 'list_number' field
#1043956: Content Migrate fails for nodes with revisions

Hey,

Now that the upgrade path is more or less working I've got a couple of question regarding left over database fields:
At the moment I got a ton of fields. Can I get rid of the following fields without any issues (once I've migrated successfully of course)

- content_field_
- content_type_

Then I also have quite a few fields like these:

field_deleted_data_
field_deleted_revision_

subscribing

Subscribe.

@KarenS - thanks for this module. I assume the plan is to include userreference at some stage (in reference to #77)? On a preliminary D6 to D7 conversion test I ran, user_reference fields show up under "Unavailable fields".

I also have a few more under "Unavailable fields" like content_taxonomy, formatted_decimal, etc which I imagine should be dealt with via drupal 7 versions of the modules, (and of course multigroup fields - discussed elsewhere).

Subscribing.

My five cents from the sandbox: you're doing a great job, keep up.
Custom text fields run OK.
Date field also remains intact. Unclear whether one still has to include the adodb library in the Date module.
Nodereference does not work, References installed.
Images are lost, because Imagecache presets are not recognized.

Subscribing.

Subscribe

Subscribe

Not able to update from D6 to D7 a simple text field used in one content type using latest dev.

An AJAX HTTP error occurred. HTTP Result Code: 200 Debugging information follows. Path: /batch?id=600&op=do StatusText: OK ResponseText: {"status":true,"percentage":"63","message":"Processed 1 out of 2.\u003cbr \/\u003eProcessing \u003cem class=\"placeholder\"\u003e97\u003c\/em\u003e : \u003cem class=\"placeholder\"\textsample\u003c\/em\u003e"}PDOException: SQLSTATE[22003]: Numeric value out of range: 1264 Out of range value for column 'changed' at row 1: INSERT INTO {tracker_node} (nid, published, changed) VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1, :db_insert_placeholder_2); Array ( [:db_insert_placeholder_0] => 158 [:db_insert_placeholder_1] => 1 [:db_insert_placeholder_2] => -2147483648 ) in tracker_cron() (line 103 of /public_html/modules/tracker/tracker.module).

Once I get it to D7 I really want to convert that text field to a media asset anyway since it references local mp3 files, so maybe someone knows a shortcut to that. I was planning to use VBO and maybe rules to convert, but have to have access to the old text field.
------
OK, I guess it was just a warning. The text field came over anyway.
Now to figure out how to work with the new media file reference.

like #20 I cannot update my fields because 'longtext' columns are converted to varchar(255)

@Stol in #148: I made you a new issue -- see #1078804: 'longtext' columns converted to varchar(255) on migration.

Trying to update a simple text field at shorter length. The fields are updated properly, but data is lost.

@OFF:

The basic code for upgrading field data is working. Lets close this issue and start opening separate issues for anything that still needs to be fixed. It's too confusing to try to figure anything out from this thread any more.

So file a new issue, okay?

Using the current dev. I've tried both with and without patch #125, this error is still being thrown on two separate occasions. Should I wait for this to be resolved or can it be ignored?

An AJAX HTTP error occurred. HTTP Result Code: 500 Debugging information follows. Path: /sandboxsite4/batch?id=1992&op=do StatusText: Service unavailable (with message) ResponseText: PDOException: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'public://IVAWChapterapplication1.doc' for key 'uri': UPDATE {file_managed} SET uid=:db_update_placeholder_0, filename=:db_update_placeholder_1, uri=:db_update_placeholder_2, filemime=:db_update_placeholder_3, filesize=:db_update_placeholder_4, status=:db_update_placeholder_5, timestamp=:db_update_placeholder_6 WHERE ( (fid = :db_condition_placeholder_0) ); Array ( [:db_update_placeholder_0] => 1 [:db_update_placeholder_1] => IVAWChapterapplication1.doc [:db_update_placeholder_2] => public://IVAWChapterapplication1.doc [:db_update_placeholder_3] => application/msword [:db_update_placeholder_4] => 206336 [:db_update_placeholder_5] => 1 [:db_update_placeholder_6] => 1269026651 [:db_condition_placeholder_0] => 93 ) in content_migrate_filefield_data_record_alter() (line 168 of C:\xampp\htdocs\sandboxsite4\sites\all\modules\cck\modules\content_migrate\modules\content_migrate.filefield.inc).

In another thread I was told by solotandem

I would suggest you:
-- search your database for duplicate uri's,
-- pick one row to keep,
-- change the references in other tables to the file table id you keep,
-- and delete the duplicate uri's from the file table,
-- rerun the content migration.

BTW, The file table issue is unrelated to this issue, and should be filed as a separate issue.

they suggested I use the following SQL to search for these duplicates

In the D6 database, try:

SELECT COUNT(*) AS Rows, filepath
FROM files
GROUP BY filepath
ORDER BY Rows DESC

Unfortunantly I've already migrated to D7, so I ran this against my D7 DB. I checked both the filepath and fid for duplicates. Their were none to be found. Thoughts?

UPDATE: This discussion is being continued here #1176186: D6 -> D7 upgrade: Duplicate files cause Integrity constraint violation: 1062 Duplicate entry 'xx' for key 'PRIMARY'

whats the fix? it'd be really nice if the fix or reason for closing was posted at the top of this page. searching through 150 comments of what may or may not fix the issues is not good.

@captaingeek, the answers and upgrade process are at http://drupal.org/node/1144136, which was extremely difficult for me to find, even though I already knew it existed.

Edit: I updated the issue summary (node body). @captaingeek, at the time this issue was closed, that option was not available to us.

@KarenS, or other maintainers, could you update the project page to point to http://drupal.org/node/1144136 instead of this issue?

I would like to provide a patch for content_migrate.filefiled.inc and it seems that the 7.x branch is not available on Git. How do I do ?

My patch is a single line in the switch...case: if ($file->uid == NULL) break; it really helped me to switch 2 D6 site with 1500+ images. It took hours, but it worked pretty good.

Thanks

Add a link to the documentation page.