[meta] Data migration from D6 contrib CCK fields to D7 Field API

It is no longer too early to be thinking about this, in fact it is probably one of the highest priority items at this point.

Some initial thoughts:

The CCK to Field API upgrade process will run in D7. We will therefore need D7 code that understands at least the D6 CCK content_* tables. This includes at least the D6 CCK CRUD functions to identify which fields and instances exist.

We will have a loop that iterates through the D6 CCK fields and instances and creates D7 fields and instances. We'll have to assume that all node types have been upgraded already.

We will have D7 field modules but not D6 field modules. Not all field types will map 1:1. For example, a D6 text field with allowed values will be converted to a D7 list field. This suggests that each D7 field module will need to register itself as being able to import certain kinds of D6 fields, and it will have to embed whatever info about the D6 CCK layout is necessary to do the upgrade. In fact this will probably look a lot like the "Field Conversion" logic for field_update_field(). So when we loop through all the D6 CCK fields, we'll call hook_field_upgrade_from_d6_to_d7($d6_field). In the allowed values case, perhaps list.module will implement that hook and handle any text or number fields with allowed values.

I guess hook_field_upgrade() can both create the D7 field and migrate the data into it. Or perhaps those should be separate passes/separate loops.

I do not think this can be a normal hook_upgrade_N() function because it is unlikely that all field modules a site is using will be available to upgrade all at once, so the upgrade process will require multiple passes. Thus, it should be a module with a UI, perhaps showing all the D6 fields & modules, the D7 fields and modules they map to, and the status of each (upgraded/not upgraded). As Karen points out, we have the luxury of being able to keep the D6 data tables forever, so the upgrade can be re-run multiple times until it is right. This also means that any errors during upgrade can be handled by just deleting the D7 field(s) and running the upgrade again.

subscribing

Karen had started a list of items for the upgrades, that she left in cck.install (HEAD).
I took them out while rolling #516138: Move CCK HEAD in core and posted them as a wiki page on g.d.o

subcribing.

This issue is about all upgrade path D6->D7 or only about the upgrade path of cck/field?

We need better title or a different component.

Better title.
Thing is, we now know this will need a process separate from update.php and possibly a UI (see #3), and thus will need to be provided in a contrib module.
As such, this does not fully belong to the core queue.

This seems like a duplicate of #394720: Migrate profile module data to field API

Please set as duplicated if that is so.

Nope, this issue is for D6 CCK fields, not user profile.

@Karen: Right, agreed.

It should also be a release blocker IMO.

If a CCK_to_D7 contrib module isn't ready the day D7 ships, it means people cannot upgrade D6 sites on day 1, but can start fresh D7 sites. I'm not sure this qualifies as 'release blocker' (or that drieschick will delay a release for this reason when the day comes)

My 2 cents:

1. I think this does definitely qualify as a release blocker, and is in fact the very definition of one.
2. I think anyone who thinks this upgrade path shouldn't be part of core itself is smoking a particularly string variation of crack. Can someone explain why this makes sense?

Tricky and difficult code is exactly where you want to leverage the advantages of the Drupal core queue with multiple eyeballs on it. I've no idea why it would be desirable to push this code to a dark corner in contrib.

"2. I think anyone who thinks this upgrade path shouldn't be part of core itself is smoking a particularly string variation of crack. Can someone explain why this makes sense?"

I don't see why we want to teach D7 core the intricacies upgrading the D6 Date contrib module to the D7 Date contrib module which isn't even written yet and may not be before D7 ships. Sure, core will provide some mechanisms for an upgrade path, and we need to validate that it is sufficient before release. But we can't expect a single magical CCK to Field API upgrade path to exist separate from all the contrib modules that provide CCK/Field API functionality.

The "more hands and eyeballs" argument is IMO the only one towards CCK_to_D7 in core. Other than that, I really don't see why core should burden with this.

- CCK D6 is contrib, core D6 has no fields to migrate. It's perfectly fine if you need a contrib module to get your contrib D6 fields after switching to D7
- as Barry outlined in #3 and #16, the upgrade cannot be a regular, one-off process running 'silently' along with other update.php updates (because core or even one single contrib module can't predict how to migrate date fields or wicked_contrib_field_type fields). The upgrade process needs to be iterative, provided by a cck_upgrade module that exposes a page with the list of D6 fields that it's able to migrate given the current state of enabled modules. Press 'submit', eligible fields are batch-migrated.
Until date.module D7 (or a submodule) is enabled and implements hook_cck_upgrade() or whatever, stating that it knows how to migrate date fields, your D6 date fields are not visible in D7. Come back to the 'migrate fields' page when you have the module.

tag

tagging :)

s.u.b.s.c.r.i.b.e. m.e.

tag

Adding the d7docs tag.

Is there a CCK issue for this somewhere? I couldn't find one searching the queue.

I still can't find that issue.

<catch> webchick: still agree with http://drupal.org/node/366364#comment-2169648 ?
<Druplicon> http://drupal.org/node/366364 => Upgrade path for D6 CCK fields => Drupal, field system, critical, active, 23 comments, 2 IRC mentions
<webchick> catch, Less and less every day. :P
<webchick> catch, My perfectionist side wants to still believe it, but my pragmatist side says the only way that's ever getting done is to release D7 without it.
<webchick> Else we wouldn't be sitting here 4 months later with you asking me that question. ;)
<catch> webchick: I couldn't find any issue anywhere for it apart from that one.
<catch> webchick: so I will downgrade it.
<catch> webchick: can I quote you?

<webchick> catch, LOL sure

While, technically this isn't a critical issue, a failure to address this issue, in neither Core nor contrib, could possibly negate all the good work of the D7CX movement.

A lack of contributed modules prevented a lot of site builders using D6 early, and hence D6 wasn't used much to begin with.
A consequence of that D6 is didn't get good early testing by site builders [as opposed to developers].
The D7CX movement could/should mean that site builders can give D7 a spin and see how it feels.

As webchick notes in the alpha 2 announcement.

You can help us reach the final release date sooner by testing this alpha and providing feedback.

An upgrade path for D6 CCK fields would broaden the available field of testers.
Some people are 'selfish' in the sense, that would only bother trying an alpha if they can test it for their own use case.
For many people who maintain one or two sites, they might not bother to test the Alpha unless they could run a test on a copy of their own site.

Similarly, while D7 could be released, it won't be 'ready' for many unless there is an upgrade path.

Alan

I'm sorry to reckon that I won't be able to lead the effort on the CCK upgrade path myself.
I couldn't find the time to move forward on this in the last 4 months, and things are not getting lighter in the forthcoming weeks :-/.

@Alan - there's two issues.

a. CCK field modules being upgraded to D7. Apart from the several that were added to core. user reference and node reference are all ported. This is light years ahead of D6 porting efforts (as is Views). There are sites being built (although not launching just yet) with Drupal 7 field API right now.

b. The second is the data migration between D6 CCK and Field API - so that people can port existing Drupal 6 sites with CCK fields to Drupal 7. This is unlikely to slow Drupal 7 adoption too much - people are generally much slower upgrading existing sites to newer releases than they are starting new sites with it - however it's absolutely critical that this in place fairly soon, because without it there'll be very few D6 sites able to port to D7 at all

This is pretty much what #yched said in #14.

I've re-titled this issue to make it more explicit that a. is the missing piece. Also Barry's thoughts in #3 sound eerily like table wizard and migrate..

From #3:

> We will have a loop that iterates through the D6 CCK fields and instances and creates D7 fields and instances. We'll have to assume that all node types have been upgraded already

I am (slowly) working on prototype code to do this. I'm planning a hook_schema sort of thing that defines what fields the system should create, before running a conversion process. I don't have that much free time to work on this, so if anyone wants to collaborate on this, I'll put code I have so far on github.

Please for the love of God not GitHub again on something this important! :(

Please just post it here as a patch, someone can take it and run with it if they have time. You can instruct them to check GitHub for the latest version.

Cross-referencing related: #727416: More complete CCK to Fields API update information needed

The whole point of github is that it's good for working collaboratively. I find it's much easier to see successive commits than muck about posting one patch after another. But I agree that you don't want to be having to learn something new at this point :)

So here's a zipfile :)

This is a module because I started out with the intention of making a helper module for contrib modules to use to migrate data to FieldAPI (eg user terms, term images, oldskool image).
It will need some work to go inside the update system in such a way that it can be used by core or contrib.

The idea is that you define migration plans which are then run as a batch.
Each plan tells you: what fields and field instances to create, how to get
the objects to manipulate, and then with a loaded object, how to transfer data
from one lot of object properties to another.
Eg your custom module might load stuff into $node->myprop, and this should go
into $node->field_new_field.
The plan tells you which nodes to do this to and how to get the data for the old
property if there are no longer hook_load() implementations to do this from old tables.

More explanations in code comments. There's a demo plan for dummy data and I made just now a very rough start on taxonomy, which as it's one of the more complex cases should help thrash out anything this system is missing.

I'm out rest of today but around tomorrow morning (0800-1200 UTC) if anyone wants to chat about this.

I would recommend to

1) agree on a proper name for this module (e.g. I'd favor upgrade_field [upgrade name-space, like upgrade_status module] or cck_upgrade [cck namespace, EOL])

2) create a corresponding contrib project

3) grant CVS access to all that want to help

My code is not aimed at just CCK, but any module that stores something that could migrate to FieldAPI.
Some good examples would be user terms, taxonomy image, image attach. All the code to help this should be in core.

A couple of notes about what I posted: I've not done the batch looping properly yet, nor implemented all the stuff that's in the info array such as the postprocessor. I think the whole plan needs pre- and post- callbacks too for cleanup, table deletion, etc.

I'm really glad to see that there has been some progress on this. I will try to pitch in. Like KarenS and yched, I worry about my ability to have the time to lead it. Unfortunately we're the three most likely candidates. :-/

I'm looking at taxonomy stuff right now, will be on IRC for the next hour or so.

This has some preliminary taxonomy work on it.

- creates a field for each vocab
- creates an instance for each node type the vocab applied to on D6
- TODO: the serialized data properties so the field know which vocab it is: the API needs to be able to handle the $field.data stuff.
- UNTESTED: bring over old data from {term_node} into the new fields.

What I could use a hand with:

- someone who knows more than me about update and batch API to rework this into something that can be used like this:

// any module can use this
function somemodule_update_7000() {
  // include the field convert API
  include_once('includes/field_convert.inc');
  $my_plan = // monster array describing the plan
  field_convert_plan($my_plan);
  // done!
}

// helper preprocess functions here

- someone who knows more about fieldAPI and CCK to make sure my code is sane
- and fill in the missing stuff like what to set in the data column values on {field_config}.

PS. I'll keep posting files here, but for those who do use github: http://github.com/joachim-n/field_convert You can also just download a zipfile from that page.

@joachim : Upgrading of taxonomy data is supposed to be already handled in core updates - except, well, it's currently broken : #706842: Improve comments for the taxonomy upgrade path :-/.
At any rate, this specific job is about migrating core D6 data (taxonomy), as opposed to contrib data (D6 CCK fields), so it has to be handled in core updates.

@Karen : the sample db you mention would be cool to ship in the CVS code, for testing purposes. Not sure if it could be used in a simpletest automated testing scenario, but having it at least available for everyone to try out and enrich sounds precious.

Agreed that 1) is a different animal. Data upgrades for 'X as field', where X was not a CCK field in D6, is a job for the module that implements X, be it core or contrib. From the examples we had in core so far (body, taxo, file uploads), those involve business logic specific to the task at hand, and I'm not sure there are lots to share with 2) and 3) (D6 CCK field -> D7 Field API field)

For 2) and 3), I don't think I see any reason why any of this should live in core ?

I think I was considering the following workflow - hook names are obviously negotiable :

• hook_cck_upgrade_upgradable($D6_field) lets modules say 'I'm able to convert that field'
• a page at admin/field/upgrade or whatever presents a table listing all the D6 CCK fields available, with an 'Upgrade' checkbox at the beginning of each row, and a 'Submit' button. Rows for which the hook above returned all FALSE are disabled.
• On submit, a batch API process is launched, migrating each one of the selected fields
  for each field :
  - One batch op creates the D7 $field and $instance arrays :
  The op calls hook_cck_upgrade_definitions($D6_field), returning array('field' => $D7_field, 'instances' => $array_of_D7_instances), and then performs the field_create_field() and field_create_instance() calls.
  - One multistep batch op migrates the field values :
  For each D6 value, the op calls hook_cck_upgrade_value($D6_field, $D7_field, $D7_instance, $value), returning the D7 value to insert, and then inserts the value.
  - Once all values are migrated, the D6 fields and data are moved out of the way "somehow" (simply removed from the db ? marked as 'upgraded' ?), for the next time you visit admin/field/upgrade.
• When crazy_contrib_field_type.module becomes available for D7, stating in its hook_cck_upgrade_upgradable() that it knows how to upgrade fields of type 'crazy_contrib_field_type', then all fields of this type become 'upgradable' the next time you visit admin/field/upgrade.

  This workflow, however, assumes that upgrading any given field is entirely hardcodable, without the need for any user-defined choice or adjustment (i.e, $field['foobar'] in D6 might translate to $instance['setting']['barbaz'] = 'a' or $instance['setting']['barbaz'] = 'b').
  If that's not the case, there might be a need for an additional 'configure migration' step before running the batch. I'm not sure we need to consider this case right now, though.

My code can already handle 1, 2, 3 and 3.

I think it's entirely possible to make this a generic API that any module, core or contrib, can make use of to upgrade itself, and I think that anything less is remiss.

PS. yched's proposed workflow is pretty much what my code does.

> Upgrading of taxonomy data is supposed to be already handled in core updates - except, well, it's currently broken : #706842: "taxo as field" update broken + wipes some node/term associations :-/.

Upgrading taxonomy is two things:
- upgrade term and vocabulary tables to become objects and bundles. That's currently outside this API.
- upgrade vocabulary node type associations and term_node association to become fields and field data. This API can handle that easily, and already does.

I am following the discussion to be ready if there are translatable fields issues to address: I hope to have time to give the code a look asap.

> Changing the core field updates to use this API would also involve re-writing core code that is already committed and in use,

I thought it was broken... :/

> That will be a lot more work for module developers

I disagree. Module developers have only to provide basic information about fields, rather than figure out how to create them and correctly migrate their data.

> I can't get this code working, it's making assumptions about what is in my database and throwing fatal errors

It's still pretty rough about the edges. So far it's working on test data that I've chucked into a D7 by grabbing it out of an entirely separate D6 install, which is quick and dirty but quicker than doing a whole D6-7 upgrade. I am still poking this whole concept with a stick to see if it works, but I think it can. It's prototype code, but it doesn't need *that* much cleaning up.

> And we don't have any way to see what is and is not converted, we only see fields that have provided information, we can't see anything about fields that are not converted.

CCK should get some special handling to list all existing D6 CCK fields as migration plans. Shouldn't be too hard to do.

> (Plus they apparently still show up as available for conversion after their data is converted).

Yup, not implemented that yet... Not too hard -- let each plan specify a table name: eg taxonomy says 'if term_node exists then I have not run yet'.

Ultimately, this is my opinion on the matter: I maintain or co-maintain several contrib modules that should really move to FieldAPI in D7. I am not going to write several upgrade paths which at their heart are identical. If every contrib module has to rebuild this one wheel, it's a colossal waste of Drupal developing time across the community. I am doing this once, and once only.

> Changing the core field updates to use this API would also involve re-writing core code that is already committed and in use,

I thought it was broken... :/

It happens to be broken currently, but this does not make it as if it was as good as not written and not existant ;-)

I am not going to write several upgrade paths which at their heart are identical. If every contrib module has to rebuild this one wheel, it's a colossal waste of Drupal developing time across the community. I am doing this once, and once only.

In practice, the bits that are identical are fairly minor IMO, merely a general structure (create fields and instances, then loop on all nodes). There are not much common parts to abstract out of the upgrade paths for 'node body as field', 'comment body as field', 'taxo as field' updates that currently exist in HEAD. Each time, the trickiest part is the business logic specific to the kind of data you're migrating to be a Field.
Abstracting out a generic framework able to handle all cases of 'data migration of X as field'
a) is a tough job (that's some quite 'meta' code you have in #45)
b) instead of writing a couple 'regular' (ok, multipass) update funcs, requires module authors to learn the framework
c) is not required for and won't directly serve the goal discussed in this thread (migrate D6 fields)

So as Karen said, I think this is a different topic. 'Migrate CCK D6 fields' is an issue with a fairly well defined scope, yet complex and challenging enough, let's not have it depend on first setting up a generic 'Migrate anything as field' solution.

I've simplified the plan array somewhat by having the API get information from hook_entity_info(), and I've added a way for plans to check if they've already run. Updated code on github.

I understand what you're both saying about the complexity here, but I really think it's worth it.
The $plan array is really just a few basic keys followed by chunks reproduced almost verbatim from what you need to pass to FieldAPI. I could maybe bring it in line to match completely, then module authors wanting to use this would just create a template field by hand on a demo site, export that from code, and copy-paste it into their plan definition.

The workflow I have is exactly what you describe in #50, and CCK can sit on top of it and provide further simplification: in other words, the CCK field migration is a special case which is easier because CCK already knows a lot about its own fields.

One question I'm not sure how to deal with is the matter of loaders that are gone by the time we are on D7. For example, in my taxonomy_migrate_pre_process() I'm having to do the work of the old D6 function that loads terms for a given node, because on a D7 installation that code has vanished. The same problem will happen to any module, CCK included: you no longer need this code on D7, so it's ripped out, but the migration system needs it. Any thoughts?

@joachim: you shouldn't use *any* API (read: hooks) in the upgrade path, for two reasons:

* The API can change, but the update functions still need to run,
* All updates functions will run (for enabled and disabled modules), but only the hooks for enabled modules will get fired.

> 1) I think you are still going down the road of creating something that will, or can, run in an update hook. All three of us, Yves, Barry, and I, think this has to be handled outside the update system in a way that it can be run on demand only when certain criteria are met.

What I've made so far is in fact an on-demand system because it seemed the quickest thing to build to prototype my idea. I had assumed anything of this sort should run in an update hook, but if the on-demand is generally deemed preferable then that's great! One less thing to figure out!

> 2) The three of have a visualization of a workflow that should work, but it will only work for fields, and it's not the same as the workflow you are creating here.

The workflow I have is really really just the same as yched outlined above. There are minor differences in things like "returning array('field' => $D7_field, 'instances' => $array_of_D7_instances)" but that's easily changed.

> 3) All your changes are on git and I don't have time to keep going over to see what has changed.

Go here: http://github.com/joachim-n/field_convert and click 'Download source'. You will get a zipfile identical to what I could upload here. I have not yet posted a patch because I don't yet know how to best work this into core. That's one of the things I was hoping for help with.

What fatal errors are you getting?

> The CCK field update module will create the old fields by reading the data in the content fields tables and constructing it into the right format and create the new fields by massaging that array into the new format and using the field api to create the tables.

Right - so CCK needs a generic preprocessor that all CCK field plans use to load up their data.

> Those techniques won't work for things that weren't fields to begin with.

They already do! I migrated test data from user_terms module to taxonomy fields created on users!

re Karen :
I've been thinking that maybe the script doesn't need to build the $D6_field array in the exact format CCK D6 builds and manipulates it (merges field + instance info).

Since the 'create fields and instances' step will need to iterate on instances of a given field, to build separate $D7_field and $D7_instance arrays, maybe it would be easier to have the script work on the raw data as loaded from the D6 tables content_node_field and content_node_field_instance (keeping $D6_field and $D6_instance separate), rather than having the records go through the D6 API that merges them into a series of $D6_field_plus_instance arrays that have to be sorted out (like, which ones of these are the 'same field').

Internally, the D6 raw db data is closer to the $D7_field / $D7_instance structure than the $D6_field_plus_instance API structure.

This issue is far too important to de-rail on discussions about optimal approaches. joachim, what you're working on sounds cool, but please open a separate issue for it (and please, for the love of God, post PATCHES instead of forcing people to go to some off-site thing and learn how to interact with it if you want anyone at all to review the code). Let's get the basic update path working here, in this issue, in the way that the three field API maintainers have already discussed and agreed upon.

Many thanks for attacking this, Karen. This looks great.

A couple remarks from a cursory read :

- It might be helpful to use var names like $D6_field, $D7_instance, instead of just $field and $instance, which get a little confusing ;-)

- I think we changed the semantics of values between D6 'multiple' and D7 'cardinality' (the value for 'unlimited' has changed, if I'm not mistaken), so we cannot just copy the value.
Also, cardinality is a property of the instance in D7, while it was a property of the field in D6, so it has to be set in field_migrate_get_instances(), and needs the $D6_field as a param, not just the instance.

This possibly is a sign that the creation of the D7 fields and instances from the D6 definitions is difficult to split into independant "a) do fields b) do instances" steps, but should be done globally (you need the D6 field and instances to create both the D7 field and the D7 instances)

- You can get D7 db API to get result rows directly as associative arrays, instead of objects that you then manually cast to arrays :

$query = db_select('content_node_field_instance', 'ni', array('fetch' => PDO::FETCH_ASSOC));

- The direct 'old table to new table' copy of field data is faster, but I'm not sure we can spare a value-by-value migration :
The values might need some custom massaging between D6 and D7 - that's the hook_cck_upgrade_value() step in #50 above.
Also, having the writing of values go through the regular field_attach_update() would let the script support migrating fields to other storage engines than the core default SQL one (there is already a mongodb storage engine out there : http://drupal.org/project/mongodb) or play nicely with PBS.

I've made a bunch more changes to my code:

- rearranged plan properties so chunks of it that are just FieldAPI settings look exactly like FieldAPI arrays. (Apart from a 'bundles' shortcut :)
- farmed out big chunks of code to helper functions so it's much easier to get an overview of what the batch process does. also it makes the debug code simpler.
- general cleaning up.

Code is now here: http://drupal.org/project/field_convert

subscribing

Subscribing..

Posit:

We should preserve fields, bundles, field instances, and field data from CCK to Field API. Everything else, including widget selection, widget settings, field settings, display settings, etc., we should not bother with, leaving them for a human to fix.

Rationale:

Upgrading from D6 to D7 will not be a simple completely automated process for any but the most simple sites. The most simple sites do not have many fields and custom settings, so re-creating them will not be that big a burden, whereas writing the code to completely automate it will be an enormous burden. Compare the Views 1 to Views 2 upgrade process, which never existed at all. People re-created their Views and survived.

Discuss. :-)

bjaspan's approach seems reasonable to me. In general, the Drupal philosophy is to preserve *data* from major release to major release, and that's what we're doing. If we preserve the data with clear expectations it's better than trying to preserve everything with huge expectations and lots of bugs.

I'd say this issue is now a collector for related contrib issues: changing status and title accordingly.

None updated the issue in 3 months, let's close it.

Yeah, #781088: Updating CCK Fields and Data from D6 to D7 has actually been pretty darn active and has some great progress.