Hi!
I am trying to write a field collection field handler for the migrate module. As starter I used the code from the simple value fields fields handler that comes with the migrate module as I thought the field from field collection is pretty the same than an integer field. Instead of storing an integer value from the user, it stores an integer value which is the entity id.
Am I right there?
My first question is, has my array down here the right format? As I understand Field API (I don't know much about Field API) Where the array key says value down here. This is the name of a column in the db that ends with value and where the entity id is saved? In my case "field_ca_competitor_value ".
$return[$language][$delta]['value'] = 5555; // Test with some fake Entity ID
If I run this in my migration I wolud assume that I get a new row in the field table that looks like this:
== Data for table field_data_field_ca_competitor |------ |entity_type|bundle|deleted|entity_id|revision_id|language|delta|field_ca_competitor_value |------ |node|competitor_analysis|0|6278|4818|de|0|5555 <-- This the fake ID from my migrate handler
But this is not happening. The node with the field inside is created, but the field is not populated.
So my second question is:
Can someone tell me if this is because there is no field colletion with the entity id (5555) at this moment? Does field collection check if there is data with this id and than avoiding that inconsistent state. I can't be a feature from the field API or? Because field API can not know that this field stores the entity id to a field collection, or can it? I thought migrate is just calling the field API for creating a row in the db with this field data. But maybe this works not like that, instead the data is written by the field collection module and at this place there is some consitency checking?!
Thanks for anything what helps me to understand this.
/*
Example from the Migrate Module how this is done with simple value fields
class MigrateValueFieldHandler extends MigrateFieldHandler {
public function __construct() {
$this->registerTypes(array('value', 'list', 'list_boolean', 'list_integer',
'list_float', 'list_text', 'number_integer', 'number_decimal', 'number_float'));
}
public function prepare($entity, array $field_info, array $instance, array $values) {
$migration = Migration::currentMigration();
$arguments = (isset($values['arguments']))? $values['arguments']: array();
$language = $this->getFieldLanguage($entity, $field_info, $arguments);
// Setup the standard Field API array for saving.
$delta = 0;
foreach ($values as $value) {
$return[$language][$delta]['value'] = $value;
$delta++;
}
if (!isset($return)) {
$return = NULL;
}
return $return;
}
}
*/
// Define a way to get data into the field_collectio field in a node
// This means not getting data in the fileds of the field collection
class MigrateFieldCollectionFieldHandler extends MigrateFieldHandler {
// Define the constructor of our Handler - it lets Migrate know what *types* of fields this handler is compatible against.
public function __construct() {
// Handle the field_collectio field type
$this->registerTypes(array('field_collection'));
}
// Based on the entity and return an array of field values to get processed.
public function prepare($entity, array $field_info, array $instance, array $values) {
$migration = Migration::currentMigration();
$arguments = (isset($values['arguments']))? $values['arguments']: array();
$language = $this->getFieldLanguage($entity, $field_info, $arguments);
// Setup the standard Field API array for saving.
$delta = 0;
$return[$language][$delta]['value'] = 5555; // Test with some fake Entity ID
if (!isset($return)) {
$return = NULL;
}
return $return;
}
}
Comment | File | Size | Author |
---|---|---|---|
#234 | extras.inc_.txt | 1.19 KB | albarin |
#234 | update.inc_.txt | 1.2 KB | albarin |
#222 | field_collection-migrate-1175082-222.patch | 6.04 KB | fuerst |
#220 | Screenshot_6_26_13_7_43_PM.png | 73.64 KB | bleen |
#209 | field_collection-migrate-1175082-209.patch | 6.5 KB | danylevskyi |
Comments
Comment #1
grendzy CreditAttribution: grendzy commentedI'm also a bit stuck on this; here's what I have so far.
Comment #2
grendzy CreditAttribution: grendzy commentedComment #3
mikeryanSubscribe
Comment #4
fagoProbably it would make sense to base this upon #1168196: Add generic-entity integration ?
Comment #5
abeger CreditAttribution: abeger commentedgrendzy: I'm currently attempting to do a field collection migration. Do you have any sample code for how to use this class?
Thanks!
Comment #6
grendzy CreditAttribution: grendzy commentedexample attached:
Comment #7
abeger CreditAttribution: abeger commentedThanks! I got it up & running yesterday and it's migrating my field collection like a champ.
Also, it's worth noting that this patch requires version Entity API 7.x-1.0-beta9 or later in order to work.
Comment #8
Bevan CreditAttribution: Bevan commentedThis patch has some whitespace fixes and registers the file with Drupal 7's class registry, so that Migrate can discover it.
Comment #9
fagoPlease do not RTBC your own patches.
I gave it a visual review:
Double spaces before "pre" and why is migration written in upper-case here? else it is't.
Missing point. Also in other comments.
Minor, but there should be an empty line before @return.
Missing space after (array).
Better use field_collection_item_load().
Comment #10
Bevan CreditAttribution: Bevan commentedAcknowledged. However, although I did upload that patch I don't consider it "mine". I only made one very minor change in addition to the previous patch.
Comment #11
pcambraHere's a patch applying fago comments and cleaning the file paths.
Comment #12
pcambraBy the way, I think this should be included in migrate_extras module (http://drupal.org/project/migrate_extras) more than in field collection, other projects are doing it that way (OG, Profile2...)
Comment #13
fagoMakes sense - so I'm moving this issue to the migrate_extras queue.
Comment #14
mikeryanJust copied over to the Migrate Extras project page from Migrate itself, don't know why I didn't have it in there before:
I would prefer to see this go into the File collection module itself rather than Migrate Extras (but that is ultimately up to fago).
Thanks.
Comment #15
fagoI'd be fine having it in field-collection either.
Comment #16
pcambraSorry about the confusion, I really thought that migrate extras was the place for this.
Anyways, I've tried this patch in a couple of migrations and seems fine to go.
Comment #17
pcambraHad some issues when doing rollback of non-imported items, attaching fixed version
Comment #18
roderikSome patch feedback:
#0
works great. No problems importing.
I was so happy that this is around. Like the whole module. As a non D7-experienced person, I jumped for joy when I discovered that field_collection would solve the big potential 'data problems' for me, in a nicer way then CCK3 could, let alone extra node types + nodereferences...
#1
It's not clear to me why 'hostEntityType' is defined as a destination field. This is for being able to create several different entity_types from items in the same import process?
Anyway, I didn't set it, after discovering that PHP gave a notice if I didn't pass it as an option in the constructor, like this:
And everything worked nicely, though the the "HostEntityType" field is still unmapped (and showing as such in the migration UI...)
It also wasn't clear to me what the HostEntityType should be set to. I understand by looking at the database after doing a test import, but... it would be nice if it was documented.
...actually, it probably needs a 'field_colllection_migrate_example' module with a documented import class?
(You can give me that task if you want... though I don't have time right now. More stuff to migrate urgently.)
#2
the last hunk in patch #18 (the only one for field_collection.module) fails. That's fine; it's in the -dev version already.
That's it.
So actually, I don't have any feedback except for 'documentation issues' (is that an RTBC blocker? I think yes?...) Great work.
Comment #19
roderikOh yeah,
#3
"Processed 1 (0 created, 0 updated, 0 failed, 0 ignored)"
...It seems an update counter is missing somewhere. The 'ignored' does work, but the 'created' doesn't.
Comment #20
pcambraThanks for the feedback and tests @roderik
As for #1 you need to provide a hostEntityType for loading the host entity in the import, I'd say that's required. You can use addUnmigratedDestinations in your _construct method for those mappings you prefer to ignore.
I've fixed #2 and #3 in the patch, I don't know what has happened with the code repeated, I carried that from the original patch, and I've added a call to increase imported elements.
As for the example, I don't know if fago would want to include that here, maybe it has more sense as a documentation page for the project.
Comment #21
roderikConcerning the example: I mentioned it because these example migration classes seem to be the best/standard thing for learning migrations. I picked more up from those than from the d.o. handbook pages.
I see the Date module also ships a date_migrate_example module -- though I'm not sure a complete module is necessary. A field_collection_migrate_example.inc file could be enough.
A documentation page would be good too, if linked from the project page.
---
As for the hostEntityType field: I guess I can see the point that you want to provide a mapping for every database field.
You don't need to provide a value/mapping for it, though. I did without, and most people can do without -- since you do need to provide the hostEntityType as an argument to the MigrateDestinationFieldCollection constructor.
That's the main thing I want to clarify for people using this code - be it in a documentation page or an example .inc file.
Comment #22
pcambraMaybe you could complete the patch with a field_collection_migrate_example :)
I have no special reason for keeping hostEntityType as a mapping itself if it can be done otherwise, feel free to change it in the patch and I'll give it a try for sure.
Comment #23
roderikComment #24
marcushenningsen CreditAttribution: marcushenningsen commentedThis works really well, however, I encountered a few problems. After migration the status show only 13 of 81 rows imported, however, all rows have been imported. Also, rollback has no effect.
Have you experienced these problems? Are you interested in more details?
Comment #25
mikeryanI'm now doing some field_collection migration myself, here's an updated patch with the following changes:
Comment #27
mikeryanRerolled - fixed a problem I had where the host node loaded by entity_load_single had an instance of my collection field with an empty value, and ultimately the real field value ended up at [1] with the empty [0] still present, so I ended up with nothing.
Comment #29
Ari Gold CreditAttribution: Ari Gold commented#27: collection_migrate-1175082-27.patch queued for re-testing.
Comment #31
jhedstromTesting the patch from #27, I'm getting these warnings (which trigger an exception, so the migration fails)
Comment #32
mikeryanHmm, looks like when MigrateSimpleFieldHandler was added to the Migrate module, it did not obey the change to the API from #1132034: Don't assume entity is an stdClass. I'll fix that in Migrate - in the meantime, guess I need to reroll this patch.
Comment #33
mikeryanRerolled patch.
Comment #34
jhedstromI may be using this incorrectly (I'm using a query that provides one row per collection item, rather than one row per host id), but the code from the patch above works in such a way as to only save the last collection to be imported for a given host id.
Comment #35
jhedstromDidn't mean to change status.
Comment #36
jhedstromSame patch as #33, but fixed the host entity structure (language is an array on the bundle rather than an object).
Comment #37
azinck CreditAttribution: azinck commented#36 is good except my host entity is a taxonomy term which doesn't have a language. This works for me:
Comment #38
alphageekboy CreditAttribution: alphageekboy commentedI am glad to see this patch for field_collection. Having I think it is currently missing one part that may be an edge case but probably should be accounted for. I can't seem to get this to work for a Field collection that has a file field with multiple files.
I currently have:
Field Collection Group
---- Meta Data About File
---- File Path
Field Collection Group
---- Meta Data About File
---- File Path
...
It seems that only the first file is associated with the collection and the additional files are ignored. I'm in the process of coming up with a solution for my migration, but thought I would throw this out there and see if anyone had a good idea on how to implement this.
Also, perhaps field_collection is working correctly and the issue is with Migrate updating existing nodes and has more to do with this issue: http://drupal.org/node/1348322.
Comment #39
bastnic CreditAttribution: bastnic commented@alphageekboy, i had a similar case but it was due to this test
$host_entity->language equals "en", but the field_collection was saved with "und", so the last saved field collection item was always unset.
Hope that's will help you.
Comment #40
alphageekboy CreditAttribution: alphageekboy commented@bastnic -
I noticed this too when tracing through the code. However, this did not fix my issue. It seems like the issue is deeper and related to the fact that my file is in a multi-value field within a collection. In tracing through the code, it doesn't look like the field_collection migrate takes into consideration there could be more than one value passed to it that it needs to account for. In my case, I'm passing a multi-value "description" and "file path". I'm still tracing through trying to figure out if this is a Migrate module issue or a field_collection migrate issue.
I am looking for alternative methods at this point, but if I find a solution for this I will post back here.
**UPDATE***
It's kind of a hack for right now, but the solution was to change where the for each loop begins and modify the code accordingly. Essentially, I need the entity_load and entity_create to be in the loop so that for additional values will have their entity created. To simplify troubleshooting I also modified my requirements so that I only need to add the file_path and not the file_descriptive_name. I will try to carve out some time to clean up the code and hopefully submit a patch back that works.
If you have any questions about this before I do that, please don't hesitate to contact me and I can explain in more detail.
Comment #41
marvil07 CreditAttribution: marvil07 commentedThis is the patch at comment 36, but it also includes an implementation of
hook_migrate_api()
.Comment #42
aendra CreditAttribution: aendra commentedAny update on an example file or some documentation? I need to import a tonne of content into field collections and I'd be super appreciative of any help you could give... Thanks!
Comment #43
aendra CreditAttribution: aendra commentedI've modified the example code at http://drupal.org/files/issues/people_org.inc_.txt, and when I try to use the UI to migrate my field collection, I get the following notice:
Notice: Undefined index: host_entity_type in MigrateDestinationFieldCollection->__construct() (line 50 of /home/username/sites/all/modules/field_collection/field_collection.migrate.inc).
Any clue what I'm doing wrong? Thanks!
Comment #44
azinck CreditAttribution: azinck commentedthis:
needs to become this:
You'll also find you need to change this:
to this:
Comment #45
aendra CreditAttribution: aendra commentedThank you so much! That helped a lot; at least it seems like I'm progressing.
Not to turn this into too much of a support ticket, but now I'm having issues with it detecting dependency migrations. Given the code in the top comments of field_collection.migrate.inc, I've tried adding the following:
$this->dependencies = array('Stories');
However, this gives me the following error:
If I remove the dependency line, I get the following lines:
I put dpm( debug_backtrace() ); at line 178 of entity.inc and it seems like it's being given a empty array or something.
Any idea what I'm doing wrong? If I can figure this out, I'd probably be willing to write an example module if that would help the cause...
Thanks!
Full pastebin of my code at http://pastebin.com/EpTmg0nV
Comment #46
azinck CreditAttribution: azinck commented@aendrew: You're running Stories before Bylines, right?
I'm not 100% sure as to the solution to your problem, but I could guess that maybe your DB records in Bylines aren't providing valid sid values. Here are a couple links that might help:
http://drupal.org/node/1323442
http://stackoverflow.com/questions/4798047/array-flipcan-only-flip-strin...
Comment #47
aendra CreditAttribution: aendra commentedThanks for the help. Apparently:
$this->softDependencies = array('Stories');
...Was what I was looking for. I found I was able to import *some* stories (some errored up) if I removed the dependency line, imported a bunch of stories first, then imported a bunch of bylines. Making it a soft-dependency ensures the correct order is used, without requiring my entire 15,000 entry table to be imported first.
Now I just need to figure out why my byline names aren't importing as taxonomy terms and I'll be set!
Comment #48
agentrickardIt doesn't appear to me that this current patch supports a single Field Collection that is attached to different node bundles.
For example:
* Field collection 'foo' is attached to both Page and Article nodes.
The fieldMapping for 'host_entity_id' expects a string, which would lock each migration in to a single target bundle.
Is that correct? If so, it needs to be fixed, since this is a very common use case.
Comment #49
aendra CreditAttribution: aendra commentedHi!
Apologies for once again hijacking this thread, but I'm running into the "flip STRING and INTEGERS" issue again -- though somewhat spastically, which is why I'm having problems troubleshooting it (Even though I've since used the patch at http://drupal.org/node/1003788#comment-5195682 to improve the level of error detail.).
To start: I'm still using the two migration classes listed above (stories and bylines), as well as a third for images (field collection with a term reference, plus two text fields). When I migrate the byline class, everything succeeds except the very first record, and I get the following messages via migrate_ui:
It doesn't matter if I migrate 5 or 500 (or 10 consecutive migrations of 50), each migration will always return one error.
When I migrate images, it happens anywhere from 12 to 24% of the time, with the same error.
What's up? Is it just unable to find a corresponding record? Any help you could give me would be great...Nevermind, I think it's because there are non-existent nids being pointed to by the authors table. Any way I can tell Migrate to skip non-existent entities? Thanks!
Comment #50
agentrickardCheck the value in prepareRow() and return FALSE to skip a bad record. Ran into the same problem.
Comment #51
coredumperror CreditAttribution: coredumperror commented@ agentrickard:
I believe that you could accomplish what you're trying to do by defining a different *Migration class for each pair of source/destination content types. As far as I know that's what you have to do for any other type of field migration, anyway. Just tweak your source SQL query to only grab the source data that's associated with a single source content type.
Comment #52
agentrickardI know you can do that, but I don't believe you should have to do that.
Being entities, Field Collections are a different beast than other fields, and should be treated that way.
Comment #53
coredumperror CreditAttribution: coredumperror commentedI've been using the code from #41 for a while now, and it works great. But there's a minor bug with how it treats errors during migration. When I'm developing my migration classes, my code often throws exception when I do test migrates, causing the individual field's migration to fail. However, those exception are apparently not counted as "failed" migrations by MigrateDestinationFieldCollection. Even if all 10 of my test migrations throw exceptions, I see:
When I do
drush mr SmallImageField
, though, it outputsSince no fields were actually migrated due to the exceptions.
I'd be happy to fix this myself, but I have no idea how.
Comment #54
coredumperror CreditAttribution: coredumperror commentedI'm curious: would it be possible to write an alternative version of MigrateDestinationFieldCollection to make it act as a field handler, instead of a migrate destination? It would be really handy to be able to migrate my field collections like fields, instead of like independent entities that happen to be parented by nodes. I realize that that's exactly what Field Collections *are*, but aren't Field instances essentially the same thing?
I'd be willing to spend the time to learn how to re-write MigrateDestinationFieldCollection as FieldCollectionHandler, but right now I'm essentially clueless. I can't afford to waste several days on a hopeless project if turns out that it's not possible to migrate Field Collections using MigrateFieldHandler.
Comment #55
agentrickardI suspect that it's not possible, having tried to write a custom field-based migration for a field collection before going with this patch instead.
Comment #56
bastnic CreditAttribution: bastnic commentedIt would also be very much more efficient, because actually each field_collection entity saved causes the parent's node update.
Comment #57
coredumperror CreditAttribution: coredumperror commentedYeah, that's the major reason I'd like this to work, since updating the parent node alters the 'changed' date. This forces me to store that date somewhere so that I can change the date back to the original value during the
complete()
proc for each field collection.Comment #58
mikeryanMy 2 cents: right now, the Migrate API handles fields that have internal structure - field collections with several sub-fields, file entities with alt/title/description/etc. values - very awkwardly. Doing the separate entity migration is much more convenient. On the other hand, as bastnic points out, it's inefficient because in the overall migration process you end up saving the parent node twice.
I certainly wouldn't want a field handler version to replace the current destination handler approach - if implemented, it would be an alternative which would make more sense in some applications than in others. I would suggest for now, though, that we go ahead and pursue getting the destination handler committed - the field handler alternative should be pursued as a separate issue in the queue.
Comment #59
zhensley CreditAttribution: zhensley commentedsubscribe
Comment #60
artis.bajars CreditAttribution: artis.bajars commentedHow would one use this to migrate data into a multi-valued field collection? I can't wrap my head around how to create more than one instance of the collection per host entity. My collection itself has 2 fields (single value) - a node reference and a textfield but I need to attach multiple instances of it to the target node. Am I missing something basic here or is this not possible with the current implementation?
Any help is appreciated, thanks!
Comment #61
coredumperror CreditAttribution: coredumperror commentedHere's a (slightly) stripped down version of my ImageMigration class, which does what you're asking about:
field_image_fc
is an unlimited-value Field Collection instance attached to the Press Release content type, whilefield_image_fc_image
,field_image_fc_caption
, andfield_image_fc_photo_credit
are the elements in the Field Collection. More than one source row can potentially have the samecategory_id
, which allows multiple copies of the Field Collection to be migrated into one Press Release node.Comment #62
artis.bajars CreditAttribution: artis.bajars commentedThanks @coredumperror, that worked. Also had a problem where only the last row per host node was being attached, but that turned out to be the same node/field language mismatch issue as already reported in #37. Shouldn't that line be included in the patch as well?
Comment #63
valderama CreditAttribution: valderama commentedI can confirm comment #62 - already imported field collection items get unset, when another field_collection item is attached to the same host entity, _if_ the language of the host enity does not match the language of the field collection.
This check needs to be more solid... the suggestion in comment #37 is a start...
Comment #64
drinkingcoffee CreditAttribution: drinkingcoffee commentedJust wanted to say thanks to everyone for these patches and example code -- it was extremely helpful.
Like #62 and #63, I had an issue with multiple field_collection items not being inserted. Trying the changes in #37 and other changes didn't work for me.
In the end (just when I was about to give up and write a script to insert directly into the db), I figured out a poor, hack-ish workaround, which I'll post here in case anyone finds themselves stuck in a similar situation:
- migrate in first set of field_collection items (so, all unique host entity ids)
- change the entity and revision id on the collection object in field_data_... and field_revision_... in some reversible way. I just added 100000 to all of them, and increment the delta value
- clear the migrate map table for this field_collection import (but leave the previous ones alone, so the host entity ids are still there)
- clear the cache
- migrate a second set of field_collection items
--repeat until all items are imported
- update the entity_id and revision_id back to the correct value (so, subtract 100000)
-??
-profit!
in my case, each of the items i was importing had between 1-4 attached field_collection items, so this whole process only looped 4 times, and while tedious (and easy to mess up at 2am), it only took 10-15 minutes, which is way faster than the alternatives.
obviously ymmv, and this would be a lot more complicated if you had to import objects that have a whole lot of field_collection_itmes in them.
Comment #65
mikeryanLast patch appears to need some work, based on comments since then. As it happens, I'm using this on another project now, one with multiple instances per host entities, so I'll be submitting a fresh patch soon.
Comment #66
cameron prince CreditAttribution: cameron prince commentedHow can a field_collection field be populated in conjunction with a node migration? I.E. using MigrateDestinationNode instead of MigrateDestinationFieldCollection. Is there a reason why there is no FieldHandler in the patch? I'm going to try and write one, but thought I'd ask first in case there's an architectural reason that prevents it.
I'm attempting to import some WordPress articles which have multiple images with captions. I have a field_collection field called field_article_images which is configured for unlimited values. The field_article_images collection contains a long text field called field_caption and a file field called field_image. This is my code from the migration's prepareRow function where I make a query back to the WordPress tables to gather all the attachment rows:
Can someone tell me if I'm on the right track here or if there's a better way?
Comment #67
aendra CreditAttribution: aendra commented@cameronbprince:
Re: no FieldHandler -- see the earlier posts, http://drupal.org/node/1175082#comment-5670794 and http://drupal.org/node/1175082#comment-5672314.
Having just done a migration similar to what you're wanting to do (except from a shoddily constructed custom DB instead of WP's), I get the feeling you should probably just use MigrateDestinationFieldCollection. I personally found it easier to troubleshoot migration issues when compartmentalizing the process between stories and field collections — that way, when the latter had problems, I was able to debug them fairly easily.
I'm hoping to write some more thorough documentation on using the MigrateDestinationFieldCollection class once I get a few moments to breathe this month; until then, there's a good example at http://drupal.org/files/issues/people_org.inc_.txt, though make sure to pay attention to the issue at http://drupal.org/node/1175082#comment-5529172 as some stuff has changed.
Comment #68
mikeryanFinally my latest version of the patch - only slightly tweaked from the previous one.
Comment #70
bastnic CreditAttribution: bastnic commented@mikeryan, I speed up this script a little bit with that trick. With that, while updating, it don't save the parent node.
Comment #71
mrfelton CreditAttribution: mrfelton commentedI don't quite understand the host_entity_id part of this, and can't get it work. In my source system (a drupal site) I have a content type with a title and body, standard. In the target system, the title maps to the title of a new content type, but the body needs to go into a field on a field collection attached to that content type.
So I have exampleChapterMigration which runs a MigrateDestinationNode Migration, creating a new node for each in the old system. And then I have exampleChapterBrandingMigration which I'm trying to use to add a field collection to each of the previously created nodes. In exampleChapterBrandingMigration I have
But when it runs, it gives me something like this:
followed by a buch of array_flip errors.
Any ideas how it is that $host_entity_id is not set, yet I have set it in my field mapping?
Comment #72
mrfelton CreditAttribution: mrfelton commentedTurns out I needed the nid included in the selected fields for the exampleChapterBrandingMigration class:
Now I'm just left with the array_flip errors :(
Comment #73
mrfelton CreditAttribution: mrfelton commentedReroll to apply cleanly against latest codebase.
Comment #74
mrfelton CreditAttribution: mrfelton commentedNot quit sure what happened with that last one... Got double inclusion of the new file. Try this one instead.
Comment #75
jpcurley25 CreditAttribution: jpcurley25 commentedIs there any idea as to an eta for this feature to be rolled out in a new version? (ie days, weeks or more?)
Comment #76
zorroposada CreditAttribution: zorroposada commentedThanks a lot for this patch #74.
I am using it as an include file in my custom migrate module.
It has so far worked perfect for migrating a field collection with a media field in it.
Comment #77
rocket777 CreditAttribution: rocket777 commentedsubscribe
Comment #78
agentrickard@rocket777 Please use the "Follow" button at the top of the page.
Comment #79
rocket777 CreditAttribution: rocket777 commentedThe patch #74 works just fine for field_collections migration.
Thanks for this.
I have attached a demo file of my script as an example.
Comment #80
attiks CreditAttribution: attiks commentedThis gives problems with multi value fields, in my case the $host_entity->language is 'nl', but the field language is 'und', so all previous values get removed :/
Removing those lines fixes it (for me)
Comment #81
jmenard CreditAttribution: jmenard commentedHow do I install the patch to get this working?
thx
Comment #82
dynamicdan CreditAttribution: dynamicdan commentedUm.. isn't this for the field collection and not file collection module?
Also, I don't believe that the field collection module currently supports multiple instances of the same field collection field. The closest solution is to create 1 field collection field and allow unlimited values. This however does not allow the fc field to be re-used with another label for example and is therefore not providing the ability to instance an existing fc field. Only the fc field entity can be instanced.
[EDIT] - ignore that last bit, this is drupal default functionality
Comment #83
alexrayu CreditAttribution: alexrayu commentedIs there a way to migrate textfields sharing the same person id table into multiple values of a node? I am specifying this:
In this code, person facts are to be migrated into a node Person, that had been migrated previously, creating multi-value fields inside a field collection. But it only applies 1 fact, skipping the rest.
Who can suggest what;s wrong? No errors, no "skipped" messaged on import.
Comment #84
brianV CreditAttribution: brianV commentedJust to echo Alexei's concern in #83 - has anyone managed to import multiple field collection entites against a multivalue field collection field on a content type?
In our situation, we are trying to migrate a Person content type (working flawlessly). Each person has a collection of facts associated with them.
When we try to use the code as indicated, it only migrates the first Fact for each person. The remainder silently fail with no messages. This appears to be tied to the lack of a highwater mark.
Comment #85
jpklein CreditAttribution: jpklein commentedHere's an addition to the patch in #74 to fail gracefully in the case that the host entity of the field collection can't be identified during the migration.
Comment #86
mducharme CreditAttribution: mducharme commentedI've managed to import multiple field collection items onto a single content type by commenting out the language check altogether. I tried the extended language handler patch to handle taxonomies as well which still only imported a single item per node:
This works for me but results in the field collection items being imported as language neutral on a node that hase a language set. I'm in a rush to get this done for a project and this is good enough for me right now. I'll hopefully have some time in the next two weeks to help out on this effort since I use field collections all the time.
Comment #87
mgiffordSetting the bot on #85 - would be great if we could get this patch into the next dev release.
Comment #89
Fidelix CreditAttribution: Fidelix commentedThis patch has wrong paths:
It tries to apply to
/docroot/sites/all/modules/contrib/field_collection/field_collection.migrate.inc
Comment #90
radiobuzzer CreditAttribution: radiobuzzer commentedSince it has already been 3 weeks from jpklein 's patch, I am re-submitting it with correct paths
Comment #92
radiobuzzer CreditAttribution: radiobuzzer commentedwoa, the patch should have been applied on the existing dev branch and not against the previous patch. Let's see now. I am submitting this patch without having reviewing it myself, it's just a repackaging of jpklein 's from 3 weeks ago.
Comment #93
radiobuzzer CreditAttribution: radiobuzzer commentedEDIT: Removed an issue to another thread
Comment #94
David_Rothstein CreditAttribution: David_Rothstein commentedHere is a reroll that removes the offending $entity->language code discussed in previous comments. This is the code which has been causing people trouble with multi-valued fields (it happened to me too), and I don't see the reason for the code being there in the first place. If the entity that is being loaded is broken and has incomplete field collections attached to it, it seems like that must be a bug somewhere else? Plus, I don't think that assuming "field language" = "entity language" is correct (as far as I understand the field API in Drupal 7, there is no such relationship).
This patch also makes a couple small improvements to the documentation.
Finally, I didn't change this:
But wondered: Why is this using 'cid' rather than 'item_id' (i.e., shouldn't the name match the primary key from the {field_collection_item} table)?
Comment #95
Pierre Paul Lefebvre CreditAttribution: Pierre Paul Lefebvre commentedThank you, this patch works great with multiple values.
Comment #96
Angry Dan CreditAttribution: Angry Dan commentedI've just been using this patch and so far so good, although it did take me ages to notice that second argument to the constructor '
host_entity_type
'.I would suggest (sorry, I don't have time to re-roll the patch now) that we throw an exception in the constructor. For example:
Edited: fix syntax error
Comment #97
jaydub CreditAttribution: jaydub commentedI have a question regarding the multiple value field collection issue for migration. The patches in the last dozen or so comments seem to be concerned with multiple values but I am wondering if this is in reference to multiple value fields _within_ a field collection item or multiple value field collection items?
An example to clarify:
Scenario 1
- field collection item consists of 1 select list field (text) and 1 textfield
- properties for the select list field and/or textfield are set to allow 'unlimited' values
- So a single field collection item could include 2 selections for the select list field and 3 values for the textfield
- field collection item in host entity is set to limit of 1 value
- array('field_collection_item' => array('field_1' => array(1, 2), 'field_2' => array('foo', 'bar', 'baz')));
Scenario 2
- field collection item consists of 1 select list field (text) and 1 textfield
- properties for the select list field and textfield are set to limit of 1 value
- field collection item in host entity is set to allow 'unlimited' values
- array('field_collection_item' => array('field_1' => array(1), 'field_2' => array('foo')), 'field_collection_item' => array('field_1' => array(2), 'field_2' => array('bar')));
We are needing to migrate field collection data as described in scenario 2 and the patches don't seem to work with that. If I manually add two field collection item rows (each with 1 value for select list field and textfield) I actually get 2 field collection entities created in Drupal. I am guessing that the reason the patch to add migrate support doesn't work in our case is that migrate is based on creating 1 single field collection entity per migration source row.
If I am right in my thoughts what would be the method for us to migrate source field collection data given scenario 2?
Comment #98
Pierre Paul Lefebvre CreditAttribution: Pierre Paul Lefebvre commented@jaydub In my case I am in scenario 2. It's right all patches wasn't addressing this issue, until the last one. Feel free to contact me via my contact form if you need help.
Comment #99
jaydub CreditAttribution: jaydub commentedThanks for the help @Pierre Paul Lefebvre
I believe that the patch as it currently stands needs to be rerolled as the changes made to field collection module and to entity module to support revisions means the $status of a migration via call to entity_save() will no longer return the expected value.
For background the changes to field collection to support revisions
(http://drupalcode.org/project/field_collection.git/commit/b21ecfc0b46a72...)
changed the save() method in the FieldCollectionItemEntity so that the method returns the result of an entity_save() of the host entity rather than the result of the entity_save() of the field collection entity as before.
before:
after:
For the migration patch the calling code in question is:
Before the changes for revision support the value of $status would be the result of the entity_save() of the field collection item, after the changes the value of $status would be the result of the entity_save() of the host entity. In my case the host entity is a node.
The code for entity_save() in entity.module is:
In the case of the entity_save() made in the new version of field_collection.module the resultant call in entity_save() is the 2nd if clause ($info['save callback']($entity);) which for a node is node_save().
Notice that the 2nd conditional in entity_save() does not return anything and node_save() which is what is called does not return anything (http://api.drupal.org/api/drupal/modules%21node%21node.module/function/n...).
So what this means is that the $status variable in the field_collection.migrate.inc patch will be NULL in this case which means the migration is seen as failing when in fact the field collection entity is successfully created and the host entity is successfully saved.
Now what I don't have here is a patch since w/o the call to entity_save() returning anything from the entity_save() call on the host entity there is no reliable way to say the migration succeeded or failed. I suppose we could check for the $entity->item_id since if the migration didn't successfully create the field collection entity then there wouldn't be an $entity->item_id but that test wouldn't work if we were in an update context.
Comment #100
marcoka CreditAttribution: marcoka commentedi am trying to get some names into a field collection that is attached to a node. somehow i dont get it. does the node already have to exist for importing the data into the field collection?
in my case its a node type called root_word, it has a field collection that consists of 3 fields containing two int fields and one field for a word.
Ignore the entityReference field at first i havent found out how it works yet
http://www.root.artwaves.de/screensnapr/1350233577-XpJwHU.jpg
is it needed to use the fields() function somehow?
Comment #101
alexrayu CreditAttribution: alexrayu commentedRight now, the code needs a host entity id. Which means, there should already be a node for that field collection.
So, your migration should go in 2 steps. In the first step you migrate the node, and then the field collection(s) as a dependent migration in the second step.
Comment #102
marcoka CreditAttribution: marcoka commentedok, that dependency stuff is standard migrate core i guess. i will give it a try. when i have some working code i will post it.
Comment #103
marcoka CreditAttribution: marcoka commentedhm ok i am having some problems here. i created the nodes where the field collection is attached to. the second step now would be importing the field collection itself.
but here is the problem. how do i know to what node a collection belongs? i would have to somehow load the node with id X and to that node attach the data from query "get data for field collection for node x".
can do that with some temporary field on the node and custom foreach code but i am not sure here. i want to use migrate the right way. any tips/samples?
Comment #104
mducharme CreditAttribution: mducharme commentedIf you're going to attach the field collection to a node that you manually created (not through migrate) then you will need to get the id of the node that you want to attach the field collection to. Assuming that you've already setup your "$this->destination", your field mapping will then need to provide the id. Otherwise, your source data needs to contain the id of the node that you want to attach the field collection to.
If that doesn't move you forward, post some code and I can try to help sort it out.
Comment #105
alexrayu CreditAttribution: alexrayu commented@e-anima Ideally, it should not work this way. Aren't there host entities in your migration source? Like, say, if you are migrating a page with maps, you will migrate the pages into a nodes in the first step, and then, as you are migrating maps locations, you specify the page-to-node migration as a source for your host entity it. And then, what it does, it automatically adds these field collections to the corresponding nodes.
And if you are making changes at import, and adding host entities where there were none in the migration source, then yes - you will need to write a script that creates nodes, and then feed those nodes to the FC import script. You will have to modify the FC script to do that.
You will also have to abstract the logic that will relate nodes with the database records for the Field Collections.
Comment #106
marcoka CreditAttribution: marcoka commentedI am migrating words (node type root_word) and the field collection holds words similar to the root word. Like root word is "cat" and then the field collection holds "kitty, pussy ...".
So the above code gets the root words from an external DB and creates the nodes (first step). The second step then should be migrating the Field Collections right? Question is is the second step inside this class or a new one?
Comment #107
mducharme CreditAttribution: mducharme commentedIt's good that your root word node is migrated as well, this will be the host_entity that your field collections will be attached to. Your "key" field is wort_link_id which you set it your MigrateSQLMap. Your similar words table contains a foreign key to the wort_link_id , right? This is how the field collection items will know what to map to (if not, see Alexei's comment above). Your field collection migration code should look similar to these snippets - and assumes that your query includes the wort_link_id field.
...and YES, do this in a new migration class with this root word class as a dependency!
Comment #108
marcoka CreditAttribution: marcoka commentedthe "wort" table
http://www.root.artwaves.de/screensnapr/1350732962-RuRLui.jpg
the table where all is linked together "wort_link"
http://www.root.artwaves.de/screensnapr/1350732991-Cu887m.jpg
wort_link_id is the PK and hauptgruppe_id is the field that groups the words with the root word (root_word = 1)
the tables are linked but not with wort_link_id
i used wort_link_id to import where root_node = 1 so i have the host entities. using wort_link_id in the migrate class for the FC doesnt seem to work. i think i am on the wrong path here.
"Could not find host entity of the field collection to import."
Comment #109
mducharme CreditAttribution: mducharme commentedSorry, but this is getting too case-specific for this issue. I recommend looking at your migration queries and try to get results that fit the suggestions that have been provided. Here are the generalized fields you'll want your migration queries to return:
If you can get results like this, you'll be able to move forward. Otherwise, you should open a new support request.
Comment #110
Pierre Paul Lefebvre CreditAttribution: Pierre Paul Lefebvre commentedI had a problem where the host_entity was loaded from cache which was not valid anymore. Changed entity_single_load for entity_load(*, *, TRUE) (reset = TRUE, so it loads the entity from the DB). It is obviously longer, but at least it stopping clashing with other modules on entity_presave.
Comment #111
brunodboThe patch in #110 doesn't work for me:
- It doesn't add
files[] = field_collection.migrate.inc
to the module's info file (so the class isn't found when running a migration).- After adding the .inc file to the module's info file and running a migration, I get the following error for every record that Migrate is attempting to import: "File /home/bruno/workspace/yale/yale_git/includes/entity.inc, line 290
Invalid argument supplied for foreach()", and nothing is imported in the fieldgroup.
The patch in #94 did work, and imported my data in the correct fieldgroup for the right nodes. However, with that patch, I got this error: "New object was not saved, no error provided". Migrate reported that all my records had failed to import (even though they got imported), and had no way to roll back the migration.
I also had to upgrade to the latest dev of Entity API, to avoid a fatal error ("Call to undefined function entity_revision_is_default()").
Comment #112
darrenmothersele CreditAttribution: darrenmothersele commentedI'm trying out the patch from #110. The "invalid argument supplied for foreach()" error is coming from lines 84 and 95 in field_collection_migrate.inc where entity_load is used incorrectly. It currently looks like this:
but it should look like this:
Fixing that now leaves me with another error, and no field collection items.
Comment #113
darrenmothersele CreditAttribution: darrenmothersele commentedUpdate: that error was caused by the fact the host entity was being passed in as an array (as returned by entity_load), rather than a single entity. Adding reset to the code on line 96 fixes it:
Although now I have the field collection items created, but migrate doesn't think they are and fails with this error:
Comment #114
darrenmothersele CreditAttribution: darrenmothersele commentedThis is because entity_save('field_collection_item', $entity); isn't returning a status.
That's because FieldCollectionItemEntity::save delegates saving the item to the host entity, i.e. in my case node_save(), and as we know node_save() doesn't return the nid.
Comment #115
darrenmothersele CreditAttribution: darrenmothersele commentedI've updated the patch from #110 with my changes to get this working. The only bit I don't like is where I have to force success if the host entity type == 'node' . If I don't do this then (as mentioned in comment #114) the field collection items are created, but Migrate doesn't know about them.
Patch attached...
Comment #116
Zahor CreditAttribution: Zahor commentedThis worked great for a DB to DB migration, but didn't work for a migration from an XML file as no other methods seem to be allowed once sourceMigration is used. Any ideas here? Did I miss something?
Comment #117
marcoka CreditAttribution: marcoka commentedwhat about when using a entityreference field inside a field collection?
Comment #118
busla CreditAttribution: busla commentedAmaaaazing! Worked great :-) thanks for contributing.
Comment #119
agupta CreditAttribution: agupta commentedTested patch for D6->D7 migration. Works great!
Comment #120
xcafebabe CreditAttribution: xcafebabe commentedWorks great!
But, entity_load returns an array and migration crash when you run a migration with option
--update
I just added an array_shift in entity_load call to avoid failures when you update a migration.
Comment #121
marcoka CreditAttribution: marcoka commentedanother problem needing to be documented is how to do that with multiple field collection values and what if an entity reference field is inside the FC. i did a manual hacky script for all that after trying it the clean way with migrate for too much hours.
Comment #122
alexweber CreditAttribution: alexweber commentedcan we get this committed please?
Comment #123
alexweber CreditAttribution: alexweber commentedUpon further testing with Migrate 2.5 this does not work. No time for a patch at the moment but we need to declare the destination handler in hook_migrate_api() and even after doing that the class is missing some required static methods that results in fatals in the Migrate UI.
Comment #124
danylevskyiThis is updated patch of #120. Now migration works if host_entity_type is field_collection_item.
Comment #125
Pls CreditAttribution: Pls commentedThanks for the effort guys, really appreciate patches. However, I've got stuck trying to migrate multivalue imagefield to multivalue field collection. As I understand ->sourceMigration('Type') is required for host_entity_id fieldMapping, but is there any way to bypass sourceMigration() method and still get multivalue field collection?
My problem is that I don't really need to migrate whole nodes, just migrate data from imagefield to multivalue field collection imagefield. Migration is happening in same drupal installation between different fields, so node migration is really not needed.
Maby I should use prepare() or prepareRow() functions for this particular situation? Thanks.
Comment #126
alexrayu CreditAttribution: alexrayu commentedI ended up writing a custom module subclassing the migration class to have multified fc's migrations. http://drupal.org/sandbox/alexrayu/1795572
Importing field collections in the complete() hook of the node migration is the choice I would choose now though.
Comment #127
alexweber CreditAttribution: alexweber commented@alexrayu that crossed my mind, it's probably much simpler because even with this patch in progress it requires 2 separate migrations 1 for the host entity and another for the field collection.
Comment #128
alexrayu CreditAttribution: alexrayu commented@alexweber I started with 'onion-skin' migrations - like, images - one migration. Images into fc's - another, and then fc's into nodes - yet another. If the site is simple, it goes well. But then if there are any problems in the intermediary level of migration, you will be crying bitterly trying to open that onion skin. So I used my module initially, but then, as changes, and fixes were demanded in my case, I ended up anstracting the field collections functionality, images functionality, etc, into functions in my migration class. I ended up importing everything with the node, and this made me happy and in control. Though of course if the site was less complex, all that would be unnecessary.
Comment #129
mikeryanWell, it's been a while, but I now have another project using collection fields, so I'll be giving the latest patch a fresh review.
Comment #130
mikeryanI can see right off the bat that the latest patch doesn't register the handler in hook_migrate_api(), which is required to for Migrate 2.5 and later - I'll add that.
Comment #131
mikeryanWell, that was silly - this isn't a handler, it's a destination class, so doesn't need the hook_migrate_api() registration.
Thus far, the last patch seems to be working fine for me.
Comment #132
alexweber CreditAttribution: alexweber commented@mikeryan, it still needs to be declared for Migrate to recognise it, doesn't it?
I have this on my local sandbox and it shows up under Migrate Registration ok (doesn't without it)
Comment #133
mikeryanNo, it doesn't - it's not a destination handler (derived from MigrateDestinationHandler), it's a destination plugin (derived from MigrateDestination). It's called directly in migration constructors, so there's no need for it to be registered.
Comment #134
alexweber CreditAttribution: alexweber commented@mikeryan, ah i see, awesome, thanks for clearing that up :)
Comment #135
danylevskyiGuys, it's time to commit it!
Comment #136
alexweber CreditAttribution: alexweber commented@mikeryan I have a question about the implementation but to keep this thread on topic created a new one: #1887092: Source ID for host_entity_id in migration
If you could please take a quick look I'd be really grateful, thanks! :)
Comment #137
alexweber CreditAttribution: alexweber commentedSo this patch works and the Migration runs ok but it returns a status message after complete: "New object was not saved, no error provided"
Despite what it says, the new object was in fact saved. However, since it doesn't think it saved it, it's not able to rollback the migration and doing that has no effect.
Any clues?
Thanks!
Edit: Figured it out, in line 112 there's the following if statement:
The problem with this is that even though the field collection item entity was saved properly in like 109, entity_save() didn't return anything. This is probably due to how the field_collection_item entity controller works.
Therefore, even though the field_collection_item was actually saved and mapped properly Migrate doesn't think so and doesn't fill in the "destid1" field in the migrate map table; which leads to errors when rolling back and also when migrating field collections that are fields of another field collection (inception ftw).
The fix is simple:
Since entity_save explicitly returns FALSE on an error, I'm thinking we can safely assume that anything other than false means its OK.
This solution now correctly stores "destid1" in the migratemap table and allows rolling back as usual.
Patch coming in a few minutes against dev.
Comment #138
alexweber CreditAttribution: alexweber commentedFor completeness.
Comment #139
alexweber CreditAttribution: alexweber commentedAttached patch does the following different from the previos almost working one (#124):
Comment #140
alexweber CreditAttribution: alexweber commentedOops, forgot to git add -N the file, correct patch now attached.
Comment #142
alexweber CreditAttribution: alexweber commentedGetting my butt whooped by git. Ok this one works, promise. :)
See #139 for more details.
Comment #143
mikeryan@alexweber - I don't get any "New object was not saved" messages, everything works cleanly, I think there's something else going on with your migration.
FWIW, here's my working collection migration. Note this is a Drupal 7 to Drupal 7 migration, based on the migrate_d2d module.
Comment #144
bdone CreditAttribution: bdone commentedalso confirming that #124 applies cleanly and works as expected: #1175082-124: Field collection field handler for migrate module.
Comment #145
alexweber CreditAttribution: alexweber commented@mikeryan,
My use case is a bit more complex: it's a migration from CSV of Profile2 profiles that have attached Field Collections.
The profiles and field collections are each in their own CSV files and after a couple hours of banging my head against the wall the above fix was the only thing that worked for me.
Have you tried running your own migration with my patch? I don't think it will have any negative impact on working migrations and might just help some edge cases.
Will be happy to provide any more info if needed!
Thanks,
Comment #146
setvik CreditAttribution: setvik commentedHi,
I'm using the patch in comment #120 (http://drupal.org/node/1175082#comment-6833952) and am trying to migrate data into a field collection comprised of a date field and two multi-value file fields. The field collection is attached to a node content type ('state').
The year field is migrating correctly but both file fields are not migrating and remain empty. Code is below and a screenshot of dpm($row) from the end of prepareRow() is attached. Any ideas on what's wrong with the code?
Comment #147
mikeryan@alexweber: Yes, I have no doubt your patch is harmless in the general case, and helpful if for some reason entity_save() returns nothing. I would just be concerned about the fact that entity_save() is returning nothing for you, that seems to suggest there's something going wrong there.
@setvik: In the dpm, what do you see if you expand profile and downloads? I would check to make sure that your Files migration has successfully migrated the fids you see there (i.e., migrate_map_files has rows with the profile and downloads fid as their sourceid1, with destid1 being the fid that should get filled in). Also try a dpm() in prepare() to make sure the file fields are properly constructed.
Comment #148
alexweber CreditAttribution: alexweber commented@mikeryan, just a quick follow-up: I've been rolling with my patch for a week now and have implemented 5+ migrations including field collections inside other field collections and have had zero issues thus far, they import and rollback beautifully :) Here are the migrations btw: http://drupalcode.org/project/drop_jobs.git/tree/refs/heads/7.x-1.x:/modules/custom/drop_jobs_demo
Comment #149
setvik CreditAttribution: setvik commented@mikeryan: Thanks for the pointers. The destination files (destid1 fid) weren't present. They're present when I do the files migration and then do the above field_collection migration, but what I'm finding is that when I roll-back the field_collection migration, not only are the field collections deleted, but the corresponding rows in the file_managed table are deleted as well. Is there something I'm doing in the code (or neglecting) that could be causing that?
Comment #150
mikeryanThat's a known Migrate issue, see #1676652: Find a better way to preserve files (there's a patch for it).
Comment #151
puddyglum#124 works for me as well, I think.
A confusing part was getting the field mapping of the host_entity_id and the entity_id working...
I tried using
but that was not working correctly... I ended up simply joining the migrate_map_mynode table and pulling the correct destid1 value.
Comment #152
setvik CreditAttribution: setvik commented@mikeryan: Thanks much!
Comment #153
clemens.tolboomShould I use #142 or #124
It's not clear enough to me :(
The interdiff shows one essential difference to me.
Comment #154
alexweber CreditAttribution: alexweber commented#124 is the vanilla solution that apparently works for all out of the box. I've had issues with field collections embedded in other field collections and also attached to non-core entities (profile2) and the patch in #142 is what has been working for me.
This has been going back and forth a bit and ultimately Mike will decide which one to commit :)
Comment #155
mikeryan@alexweber - your patch looks fine to me, was just cautioning you that it may obscure some other problem in your instance.
I do not have commit privileges here, and as one of several contributors to this patch it's not appropriate for me to RTBC it - some objective observer who's using it successfully should do that.
Comment #156
alexweber CreditAttribution: alexweber commented@mikeryan, thanks for clarifying! :)
I guess that excludes me to RTBC it as well, any takers?
Comment #157
clemens.tolboomTnx ... I'm testing #142 on XML containing Article nodes with embedded Authors (more then 1 per node) ... hope I dare to RTBC this later.
Comment #158
afox CreditAttribution: afox commentedWe'll start testing this in a scenario where we have the same field_collection field included in multiple content types, allowing multiple items, referencing field collections which have both single & multiple value fields. Hoping for the best...
Comment #159
danylevskyi#142 works great!
Comment #160
afox CreditAttribution: afox commentedLet's hold off the RTBC a little since we have at least two others testing.
Comment #161
bdone CreditAttribution: bdone commentedi've also tested #142, and it works well.
Comment #162
alexweber CreditAttribution: alexweber commented@afox can we RTBC this now?
Comment #163
afox CreditAttribution: afox commentedSorry I had to postpone my testing for tomorrow. I'm just concerned about testing with multi-value fields.
Comment #164
afox CreditAttribution: afox commentedAgh, my migration case a bit more complicated and needs some work from other perspective before I can properly test field_collections.
If somebody else has positive multi-value testing done, let's go with that.
Comment #165
alexweber CreditAttribution: alexweber commentedAccording to #159 and #162 we're good to go, feel free to change this if you find any errors.
Comment #166
mikeryanFWIW, my working example in #143 has a multiple value field (field_projects).
Comment #167
aendra CreditAttribution: aendra commentedYay, RTBC!
To everyone who contributed to this issue (Which I've been following seemingly forever) -- you're all awesome and totally deserve a cookie, a coffee, a beer -- or all three (Simultaneously! Cookie-flavoured Coffee Beer!).
Comment #168
alexrayu CreditAttribution: alexrayu commentedAnd you also deserve the patch to be included in migrate or migrate_extras module. Why is it still a standalone patch?
Comment #169
alexweber CreditAttribution: alexweber commented@alexrayu, Thanks! However Migrate only intends to handle core and work on Migrate extras has been put on an indefinite hiatus. The logic on their project page is that contrib modules should actually implement migrations themselves in order to keep things more organised and code contained within the respective projects. Come to think of it, it actually makes more sense than concentrating all migration-specific code in one module.
Therefore, this actually belongs in the Field Collection module :)
Comment #170
alexrayu CreditAttribution: alexrayu commented@alexweber I hope it gets accepted then. There has been a lot of work done, and it is also used often for migration as well. Should not fall in a hole in between the two modules.
Comment #171
mikeryanIt's not in a hole, it's right while it belongs - in the field_collection issue queue. It's taken a long time to get to rtbc, because field_collection is a complex module with a wide variety of use cases (it's not like some simple field that can be supported with a dozen lines of code).
Thanks not only to those who've contributed to it, but to those who have tested it out!
Comment #172
clemens.tolboomI've tested this on dev. It works as expected.
I had trouble with my XML file. I had to split it into a 'node' file and a 'collection' file. And I forgot to add the XPath to
$this->addFieldMapping('host_entity_id', 'url')->xpath('url')
RTBC to me :-)
Comment #173
alexweber CreditAttribution: alexweber commented@clemens.tolboom I'm not sure what you did there... you changed the status from RTBM to Needs Review, suggesting its not ok but then in your comment said it WAS ok.
Which is it? And if its fine, please change the status back!
Comment #174
star-szrSome documentation cleanup (http://drupal.org/node/1354), no code changes. Interdiff is from #142.
Comment #175
puddyglumFYI, I used patch #124 to migrate a multi-value field collection field:
Comment #176
danylevskyi#174 is ready for commit.
Comment #177
fagoI've not test this (yet), but I'm happy to commit it if it helps and works.
Also, I think we should have a short and fully working example somewhere? Maybe just describe it in the migrate-handbooks and put the link into the code-comments also?
This misses an @endcode.
I'd suggest using entity_get_controller($type)->resetCache() + the method or entity_load_single().
Comment #178
danylevskyiI've created basic documentation page describing field collection migrations.
Here is a patch with the fixes mentioned by fago.
Comment #179
clemens.tolboom@danylevskyi in #177 asked for a 'fully working example'. The doc ref in #178 is not. Thanks for the effort though :-)
Let's hope this is still RTBC :p
Comment #180
danylevskyi@clemens.tolboom, doc page now contains general info about field collection migrations. Later I'll prepare working example and add it.
Comment #181
alexweber CreditAttribution: alexweber commented@danylevskyi hows it going with the documentation? Need any help?
- Alex
Comment #182
danylevskyi@alexweber, I think that the best help is to test #178 patch to mark it as RTBC.
Comment #183
alexweber CreditAttribution: alexweber commentedSounds good, I'm going to run the migrations I've mentioned earlier in the queue and if things go by without a hitch it's good for me!
Comment #184
alexweber CreditAttribution: alexweber commentedWorks for me! I just ran about 10 different migrations, some of them with Field Collections embedded within others and with multiple values too and it all went smoothly!
Great work all!
Comment #185
radiobuzzer CreditAttribution: radiobuzzer commentedIt ran without a problem, for about 4 migrations. Thanks for this work.
Please commit before lines change again ;-)
Comment #186
evanbarter CreditAttribution: evanbarter commented+1.
Comment #187
danylevskyiWorking example of migration is ready, but page needs review.
http://drupal.org/node/1900640
Comment #188
darrenmothersele CreditAttribution: darrenmothersele commentedI have one minor issue, in that the host entity is saved for each field collection item that is added. Which, although most the time might not be an issue, in the case where you have lots of things firing in hooks based on updating the host entity, can cause issues. I got around this by using a custom FieldCollectionItem entity controller that I inserted using hook_entity_info.
Comment #189
alexweber CreditAttribution: alexweber commented@darrenmothersele, AFAIK you gotta save the host entity to attach the field collection item "item_id" to it. Curious as to how you worked around it but, either way, I'm happy with calling this an edge case.
@danylevskyi, thanks! I've added some minor fixes.
I think this is good to go... :) I can haz commit?
Comment #190
darrenmothersele CreditAttribution: darrenmothersele commented@alexweber I agree, this is edge case stuff. I was thinking that long term, a solution might be found that somehow collects up all the field collection items and only saves the host entity once, when they have all been done. I'm not sure there's any way of knowing when you're migrated the last item for a host entity?
Consequently, I didn't prevent the host entity saving, I just added a flag that stops all my custom entity hooks from firing during migration.
Comment #191
alexweber CreditAttribution: alexweber commented@darrenmothersele, maybe the correct way to do that would be to use stubs.
Either way, I think your case should be opened as a follow up. This issue and patches have been going back and forth for ages and we finally have something that everybody agrees upon with documentation, etc. I say commit it and then open a follow-up issue... no?
Comment #192
greenjuls CreditAttribution: greenjuls commentedThanx @danylevskyi for the last patch, it worked well for me, but I just had an issue when updating some field collection items with embedded field_collections.
In my case the responsible was in method MigrateDestinationFieldCollection::import() that use the whole original entity when updating. This cause all the original entity fields to be sent into functino prepare().
To fix my issue, I had to use the same behavior as the existing MigrateDestinationNode, copy non-existent fields after invoking migration prepare handlers.
I had to clone the $entity object at first and then use the following code:
Comment #194
greenjuls CreditAttribution: greenjuls commentedMy fault, here is the corrected patch.
Comment #195
greenjuls CreditAttribution: greenjuls commentedComment #196
Ronino CreditAttribution: Ronino commentedThe patch in #194 works great for me.
It took me some time to get my image data into a multi-value field collection field. I wasn't aware that the main source table with the node data and the image data table don't have the same primary key and so only the first image record per node was imported. After recognizing this, the migration worked great.
Comment #197
guile2912 CreditAttribution: guile2912 commented#194 worked fine for me too, I have updated the documentation page fixing the $this->dependencies and $this->sourceMigration parameter in the first part of the page.
Comment #198
asherry CreditAttribution: asherry commentedI'm not sure if it's just because I'm running a d6 - d7 migration, but #194 didn't work for me. It maps to an item_id, which is weird because field collections don't really have an item_id they have a host entity_id and a delta.
I changed the patch slightly and this works for me.
Let me know if I'm missing something though.
Comment #199
alexweber CreditAttribution: alexweber commented@asherry field collection items do have an item_id, it's the primary key
Comment #200
asherry CreditAttribution: asherry commentedRight, ok, sorry. There is an item_id in the field_collection_item table, that then gets referenced as [field_name]_value in the actual field table.
My issue was mapping a custom drupal 6 multiple value field to a new field collection item, it shouldn't have been a patch, more like a custom class proposed solution if ever anybody also encounters that scenario.
Thanks.
Comment #201
greenjuls CreditAttribution: greenjuls commentedJust a little fix from the #194 : I replaced isset() calls by property_exists(), otherwise field values can't get erased when updating a field collection item.
Comment #202
Jumoke CreditAttribution: Jumoke commentedCan anyone help me with this error I am getting here:
Post here: http://drupal.org/node/1944590#comment-7185072
Note: I am Using field_collection and the patch on #194. [Thank you all patching this].
And how do i sort migration list?
Comment #203
grendzy CreditAttribution: grendzy commentedPlease don't change the issue title for support requests.
Comment #204
Jumoke CreditAttribution: Jumoke commentedSorry, didnt realize i did that. Thot i was adding comment title :)
Comment #205
alanburke CreditAttribution: alanburke commentedSlight modification to patch.
This now allows the language to be set when attaching to the host entity.
Comment #207
puddyglum@alanburke, the patch seems formatted wrong, and additionally after comparing the code to #194 I couldn't spot any differences.
Comment #208
mccrodp CreditAttribution: mccrodp commentedWhat is the status of this, does field_collection now provide support for the migrate module? If not, I'll be working on this next week and willing to lend a hand on anything that needs doing.
Thanks,
Paul.
Comment #209
danylevskyiHi guys!
Here is the patch based on last working variants #194, #201. Some documentation and code formatting are fixed here.
We are all tired with this issue. We were working so much time on it. Let's finish it now!
Comment #210
Jumoke CreditAttribution: Jumoke commentedI am having trouble getting this logic to work - migrating into unlimited field_collections on a node.
unlimited fc has fields set as:
field_date (one value), field_document (multi-value)
field_date (one value), field_document (multi-value)
Note: Master node has "content_id", it is mapping to that content_id.
Table of data that should go into fc is below:
file_id, content_id, date, file
1, 1, 6/4/13,file
2, 1, 6/4/13,file
3, 1, 6/5/13,file
4, 1, 6/5/13,file
5, 2, 6/5/13,file
6, 2, 6/6/13,file
From data above, there should be 2 fc created on the content_id 1.
I am able to migrate into multiple field_collection for a node, but it looks like it's creating one for each document, so it must be that my logic is wrong (sql query handling).
Here is code: Any ideas, insights? Thanks in advance for your time.
Comment #211
alexrayu CreditAttribution: alexrayu commentedHello Jumoke,
Test your query with Devel and see if it's correct. Also, remember, that the files is a multi-value field (if I understood you correctly).
Comment #212
Jumoke CreditAttribution: Jumoke commentedThanks Alexei. My mistake was:
It should have been the unique id of the data, not the content_id that maps back to the node (and usually there are more than 1 of these).
Comment #213
Jumoke CreditAttribution: Jumoke commentedSo i have another question.
Has anyone successfully migrated into a field_collection with a node_reference? Shouldn't it be the same method? I'm spending too much time on this, i need ideas please :)
Comment #214
danylevskyiJumoke,
Comment #215
Jumoke CreditAttribution: Jumoke commentedThanks Dany,
Turns out there were some other problems with my migration.
Comment #216
puddyglumFor some reason when I import my field collection values attached to a node, it inadvertently deletes the Geofield value. It's simply doing a MigrateDestinationFieldCollection, so it shouldn't be altering any of the other fields, but it is. Anybody experiencing similar issues?
Comment #217
bleen CreditAttribution: bleen commentedWhen trying to migrate a field collection whose host is also a field collection item, then entity_load_single returns FALSE and everything craps out:
Comment #218
bleen CreditAttribution: bleen commentedignore my previous comment. I had a typo in my sourceMigration value.
That said, I've been doing a pretty complex migration with nested field_collections and (aside from my fat fingers) this has been working fabulously!! RTBC from me
Edit: I had a typo in the word "typo" above .... oh the irony
Comment #219
fuerst CreditAttribution: fuerst commented#209 works for me following the example at https://drupal.org/node/1900640. Thanks a lot!
Comment #220
bleen CreditAttribution: bleen commentedI dont think this is worth holding up committing this patch, but for what its worth whenever I see this exception thrown I see the message displayed twice
Comment #221
geerlingguy CreditAttribution: geerlingguy commentedMissing newline at end of file. Other than that, this patch looks good to me—and even this is just a whitespace problem. Looks like this issue's been open a while—can we get this accepted?
Comment #222
fuerst CreditAttribution: fuerst commentedAdded the missing newline using the patch from #209 as a basis.
Let's get this patch committed and no longer hold it back for minor reasons.
Comment #223
danylevskyiIt works great! RTBC!
Comment #224
weseze CreditAttribution: weseze commentedWe are using this patch in development on a several field_collection migrations. (migrating from noderefernces in D6 to field collections in D7)
It is working perfect, but very very slow...
The total number of field_collections migrated is around 25.000 and this takes over 3 hours to complete. I know creating that much entities will take time, but this is really a lot... We are also migrating about 100.000 nodes in the same project with lots of fields and fancy stuff and that migration is finished in 30-40mins.
What can be the cause of this and can we optimize somewhere?
Comment #225
jaydub CreditAttribution: jaydub commentedNot sure if it's something you're running into but you may want to check out #1839744: Add index on {$field_collection_field}_revision_id column.
Comment #226
busla CreditAttribution: busla commentedI´m creating about 100 field collection items in 65-85 seconds, with 6 fields on each item. This is a legacy MySQL db. Most of fields on the fc are taxonomy terms. I´m also running a rule that processes every item.
25.000 items would then take me 5.9 hours. I might be doing something wrong but everything works as it should be, and I´m not using the patch since I´m not migrating files.
I use the Drush --feedback and --limit parameters when I´m testing so the final migration doesn´t hurt as much :-)
Comment #227
rooby CreditAttribution: rooby commentedbusla:
I saw ridiculous times like that before once and the reason was that my databases were across the network, not on my local machine.
Moving the databases to my local machine, which removes the need for network traffic, made things way way faster.
That and making sure to use drush, not the migrate UI, for migrations saves a lot of time.
Comment #228
busla CreditAttribution: busla commentedrooby: I do all migrations locally but I´m creating several joins in some cases. That might be the cause. Also, the rules conditions might take their toll. I never use the MIgrate UI. I want all the feedback possible from the console... aswell as using --debug.
Comment #229
rooby CreditAttribution: rooby commentedI'm just doing a migration that has the entities (from a csv)
user
user > profile2_1
user > profile2_1 > field_collection_1
user > profile2_1 > field_collection_2
user > profile2_2
And the field collections don't really seem to be going any slower than the rest.
Maybe marginally but nothing really noticeable.
All the migrations have some custom code in prepareRow() but it is not doing any heavy lifting.
Seems it is possibly related to your customisations or maybe you have much larger and more complex field collections than I (mine are pretty small). Or maybe there is a difference because I am coming from CSV.
Comment #230
bleen CreditAttribution: bleen commentedMy fairly complex field collection migration does go noticeably slower but not to the degree that busla is describing. That said, this should not prevent the patch in #222 from being committed. Better to have working migrations that might be slower than no migrations. :)
Comment #231
rooby CreditAttribution: rooby commentedI agree.
When I ran my migration on the production server, which is much faster & better than my development one, I was easily getting well over 1000 items per minute (I can't remember the exact figure). My non-field_collection ones were still faster though.
I had a (very - & it's late, I could be talking complete rubbish :)) quick look at the code in the patch now and there is nothing outrageous in there.
It's possible some of it is on the field collection end of things.
Also, it has to deal with the host entity too, which other migrations don't, but there is likely nothing much that can be done about that.
It could end up just being one of those things that is a known issue.
Comment #232
nlisgo CreditAttribution: nlisgo commentedMigrating nicely with patch #222 :)
Thanks for the good work everyone. I used to avoid field collection because it just seemed like it was going to be painful to migrate content to it. I favoured the relation module. I will still use the relation module for other purposes but it's great to be using this and have migrate working!!
Comment #233
fuerst CreditAttribution: fuerst commentedJust a note: While working on a migration I got a PHP error in
MigrateDestinationFieldCollection::import()
: __clone method called on non-object in field_collection/field_collection.migrate.inc, line 96Problem here was: the Field Collection in question was imported earlier so there was a Destination ID (
$row->migrate_map_destid1
) available. Later the Field Collection was removed from the node. When migrating again the assumption inMigrateDestinationFieldCollection::import()
still was to update the existing because of$row->migrate_map_destid1
although the Field Collection did not exist anymore.I propose to just create a new Field Collection in this case. Any thoughts?
Comment #234
albarin CreditAttribution: albarin commentedHi, I'm not able to update existing field collection.
I have added this lines to the update migration class:
$this->systemOfRecord = Migration::DESTINATION;
$this->addFieldMapping('host_entity_id', 'id')->sourceMigration('Extras');
I always get this error message: Could not find host entity of the field collection to import.
I can't find any example.
I attach the initial migration and the update migration classes.
I would appreciate any help.
Thanks!!
Comment #235
rooby CreditAttribution: rooby commentedComment #236
pbuyle CreditAttribution: pbuyle commentedThe patch in #222 provides a working solution. The instructions are given in the comment of the the
field_collection.migrate.inc
file. The host entities have to be migrated first (in their own migration class), then the field collections are migrated in a second migration class.Comment #237
markdorisonI am attempting to use the patch in #222 to migrate into a multi-value field collection. I create an array of the objects in prepareRow() but when the migration completes and I review the nodes, only the first item in the array has populated the field collection and no subsequent items have been added. Am I missing something important?
Comment #238
markdorisonI have been unable to get this to work with multiple value fields in field collections. I am going to mark this as needs work unless someone can point out what I am missing. I am continuing to look at this.
Comment #239
aidanlis CreditAttribution: aidanlis commentedHere's an example migration from a D6 flexifield to a field_collection in the hope that it helps someone.
You can see I accessed the serialised field data rather than creating a query which used the content_type_flexifield table, this seemed easier.
Comment #240
fuerst CreditAttribution: fuerst commented@markdorison: The rule is: one field collection item per database row. Have a look at the documentation for
MigrateDestinationFieldCollection::import()
. That means you have to modify your source query to return one record for every ingredient. May be you have to normalize your data first. Alternatively you can create the additional Field Collections per recipe in thecomplete()
method.Comment #241
markdorison@fuerst: This did the trick. Thank you for pointing me in the right direction!
Comment #242
fagoThanks everyone for the hard work on this + documentation - this looks good now -> Committed.
Comment #243
aidanlis CreditAttribution: aidanlis commentedAmazing! Great to get this committed!
Comment #245
wonder95 CreditAttribution: wonder95 commentedHas anyone implemented this handler with migrate_d2d? I'm trying to do that with an image field, but all the examples I'm finding around the interwebs are for migrate, not migrate_d2d.
Comment #246
harpss CreditAttribution: harpss commented@markdorison: What was your solution? I'm at the point you were before you said this did the trick. What was your final code?
Comment #247
islalobo CreditAttribution: islalobo commented@markdorison: I'm also curious about how your code looked in the end.
Comment #248
daniroyo CreditAttribution: daniroyo commented¿Is it possible to migrate multilanguage fields with this handler?
I have tried to migrate some field_collections with this approach and I couldn´t.
I paste some of my code to view if I am doing something wrong
The migration is working almost fine, field collectios are created, but with "und" language.
I have to say that languages provided matched perfectly, so I am not sure what is wrong
Comment #249
daniroyo CreditAttribution: daniroyo commentedIs it known something about migrate languages with this destination?
Thank you!
Comment #250
gopinath1586 CreditAttribution: gopinath1586 commentedAny one tell me the final working migration code multi value field collection entity? @markdorison and @fuerst
Comment #251
osopolarIf someone experience the loss of field collection assignment on subsequent update migrations of the host entity, I added a comment with an example to the Migrate Field Collections handbook article.
Comment #252
herved CreditAttribution: herved at Randstad Digital commentedThank you all and thank you @mikeryan your example with migrate_d2d (#143) helped me a lot.
Comment #253
isalmanhaider CreditAttribution: isalmanhaider as a volunteer commentedWhich one is the final working patch that works? Anyone, please
Thanks
Comment #254
ccarnnia CreditAttribution: ccarnnia commentedthis is a old tread, i am posting because i see a lot heavy hitters here and am desperately trying to amplify my problem: https://www.drupal.org/project/field_collection/issues/3057027#comment-13120718
Comment #255
ccarnnia CreditAttribution: ccarnnia commentedUPDATE:
Thank you @osopolar @isalmanhaider and @Jumoke for desk checking this WITH me.
this worked: