I'm working on a project that will require a solution like field_collection aims to solve.
My project needs a node container that holds statistical data collections for several years.
node: title, array (year, text, number, number), taxonomy
After assessing the different options available to me, I've come to three possible solutions:
1. Custom module, make a cck sub module under d6 or d7 that gives me the fields for the array.
2. Use cck-3.x and the multigroup under d6.
3. field_collection on d7
Now, I think solution 3 is the best solution in the long term. However, it seems to me it is quite a while until this solution is ready to use. The greatest obstacle to me at the moment is the possibility to import large datasets. This leads to the following questions:
Is there planned a way to import datasets through for example feeds?
Can someone point me in the direction of how to manually import datasets into mysql?
Comment | File | Size | Author |
---|---|---|---|
#307 | 1063434-7.x-1.0-beta13-feeds-boolean-fix.patch | 9.93 KB | robdubparker |
#306 | 1063434-feeds-boolean-fix.patch | 1.07 KB | robdubparker |
#303 | add_feeds_integration-1063434-303-beta13.patch | 10.63 KB | philltran |
#261 | interdiff-1063434-239-261.txt | 2.19 KB | majorrobot |
#261 | add_feeds_integration-1063434-261.patch | 9.16 KB | majorrobot |
Comments
Comment #1
casey CreditAttribution: casey commentedFeeds integration seems a good idea.
Comment #2
casey CreditAttribution: casey commentedLet's make it a task.
Comment #3
bowersox CreditAttribution: bowersox commented+1. We use the Feeds module and it would be great if Feeds would import data into Field Collections.
Comment #4
inversed CreditAttribution: inversed commentedsubscribe
Comment #5
Gemma Morton CreditAttribution: Gemma Morton commentedWe are using Field Collections to have Multiple values on a node of:
Question & Rating.
We provide a form via custom code, and process the submit. You could do something similar with a custom module to import the data from a CSV.
This was found somewhere in the issue queue, but I will repost how we used the code to save field_collections:
Comment #6
fagoI think the right approach for feeds integration is #1033202: [Meta] Generic entity processor. That generic processor should basically work for field-collection too - if it needs some adaptions we can sub-class the generic solution to add them.
Comment #7
hellomobe CreditAttribution: hellomobe commentedChecking on the status. It wasn't clear to me if there was a solution in the generic entity thread. I'm using QueryPath as the parser, Node Processor. Fago, thank you for all your hard work and contributions to Drupal.
Comment #8
elly CreditAttribution: elly commented(subscribin' here and on the generic entity mapper thread too)
Comment #9
hellomobe CreditAttribution: hellomobe commentedhttp://drupal.org/node/1033202#comment-4721020 - Posting here too since it may have more to do with field-collection. The rules entity processor admin ui seems to work okay. I have two field-collections fields created.
If someone understands how the entity processor should or will work, can you give me a brief summary. Looking at it more closely (I'm not a programmer and I'm muddling through to grasp the concept as best I can), it seems that it would just create a field-collection entity, not a node that has a related field-collection. Yes, no? I think most people need it to create the node + field-collection (using this as the next phase to multi-groups). Just want to make sure I'm trying to hack the right approach.
Comment #10
Bevan CreditAttribution: Bevan commentedIt would be great to get some pointers on how to move this issue forward. I may need it soon and can probably work on this, though it would take me a while to understand all the moving parts here and I would probably do it the wrong way the first time.
Comment #11
hellomobe CreditAttribution: hellomobe commentedJust putting a thought out there since this is really over my head. Could we model the feed integration as a mapper like Location Mapper? Is location an "entity".
Would a sponsor pool help to get this going (some solution whatever that path takes)? I'm willing to chip in.
Comment #12
tim.plunkettPostponing on #1033202: [Meta] Generic entity processor.
Comment #13
franzI don't get why this is so attached to the issue on #12. Using this new generic processor, I'd have to create another importer for every field collection I have in my node type.
I have a complex node type with several field collections, and I want to import at once. Would be a lot easier to have a mapper "Field Collection 1: Field 1", etc...
Did I get wrong?
Comment #14
hellomobe CreditAttribution: hellomobe commented+1 for going this direction. I also noted this in #11 referencing the location mapper. It's kind of the same concept. I agree, as I understand it, that the generic processor just does one entity at a time, versus a field-collection + node at the same time, which is the most likely use case of a field-collection entity. With the generic entity, how would the field-collection know which nid to correspond to if it is imported separately (at a different feed process) than the node?
Should we change the status to active to get help with a mapper? I've tried creating one myself, but no success.
Comment #15
franz.glauber CreditAttribution: franz.glauber commentedI did create one, but it have some issues, it doesn't work with field collections inside field collections, because it is impossible to determine the target field collection.
Comment #16
franz.glauber CreditAttribution: franz.glauber commentedHere it goes, can be useful as a starting point.
Comment #17
franzdamn, wrong account =/
Comment #18
philipz CreditAttribution: philipz commentedPatch #16 shows the field-collection fields as targets for import. I'm looking through code but can't see anything about multiple value field-collection fields. Is this supported in your patch franz? I mean let's assume csv file import and then we're having columns for each field-collection fields with somehow (semicolon) separated values for each column multi value?
I'm going to try that and maybe modify it if it doesn't work.
Comment #19
franz#18 It is supposed to work, yes.
However, if you have a multiple-value field inside a multiple-value field-collection, it is impossible for feeds to determine which value belongs to which field-collection.
Comment #20
philipz CreditAttribution: philipz commentedOk so just to be sure let's say I have two fields in my field-collection ("fruit" and "color"). The field-collection field is unlimited.
Now to import two values into this field I have cvs file looking like this:
"fruit","color"
"apple;banana","red;yellow"
I map fruit source column to target fruit field and do the same for color.
This shuld give me two field-collection values:
1. [ apple ] [ red ]
2. [ banana ] [ yellow ]
The point is how to separate the values in single csv cell?
Comment #21
franzI see now, philipz. I'm not sure how is the best way of dealing with multiple values on CSVs (I'm working with XML). Maybe you can use http://drupal.org/project/feeds_tamper to explode the values so they are passed to targets as arrays. You map the fruit to field_collection:fruit and color to field_collection:color and they should work fine.
Comment #22
philipz CreditAttribution: philipz commentedThanks for this idea - I'm using Feeds Tamper already in other import so I'll give it a try. If all I have to do is explode this into array then this should be easy :)
I'm working with XML files as well so I understand that you're using XML with XPath XML parser for your imports and it's working?
Comment #23
franzYes, works like a charm. =)
Comment #24
zoraxworks like a charm with csv parser and taxonomy list. ;)
thanks
Comment #25
dgastudio CreditAttribution: dgastudio commentedso, the patch is functional? on import feeds, i only have to use feeds tamper to split array?
Comment #26
franzkervi, the explode is only needed if you have multiple values per field like #20
Comment #27
Jason Dean CreditAttribution: Jason Dean commented#16 working great for me :)
Comment #28
sir_squall CreditAttribution: sir_squall commented#19 I have tested the patch but he dident work for the "unlimited value",
A have done a new patch, for fixing that.
Comment #29
etiennechataignier CreditAttribution: etiennechataignier commentedTried the patch in #28.
I installed the feeds tamper module and used it to explode multiple values of a csv.
However, only the first element is inserted.
My CSV looks like :
On this example, the field collection {finition:2, ref:REF1B, price_euro:10, price_dollar:20, price_sterling:30} is inserted.
The field collection {finition:3, ref:REF1N, price_euro:11, price_dollar:21, price_sterling:31} was not.
[EDIT]
Tried with Feeds XPath Parser (without feeds tamper) and got the same result.
Here is the XML :
Context : //item
Retrieval of id_finition : finition/id_finition
Only the first finition is inserted.
I'm not an expert in Xpath, so maybe I missed something.
If anyone has a solution, it would be greatly appreciated.
Comment #30
franzHave you tried the patch on #16 too? AFAIR I used it for "unlimited" values without issues.
Comment #31
etiennechataignier CreditAttribution: etiennechataignier commentedThanks a lot !
Patch in #16 works perfectly, included with unlimited field collections.
Patch in #28 produces result provided in #29.
Comment #32
rsgracey CreditAttribution: rsgracey commentedWe applied #16, and nothing changed--still no field collection targets in the importer. When we applied the .dev version of field collection module, we got a white screen. FWIW.
Comment #33
franzrsgracey, can you post versions of drupal and feeds ? Maybe that could help.
Also, try clearing caches as this is a new hook implementation.
Comment #34
GaëlG#16 worked for me with Field collection 7.x-1.0-beta3 and Feeds 7.x-2.0-alpha4+40-dev.
Comment #35
arkjoseph CreditAttribution: arkjoseph commentedThis patch is a god send. Is there any chance #16 can be applied to 6.x ? Thanks for the awesome support.
For completeness, the attached is a sample of the csv I am attempting to import with field collections.
http://drupal.org/files/Screen%20Shot%202012-04-06%20at%2012.05.15%20PM.png
Many thanks.
Comment #36
Xearwe CreditAttribution: Xearwe commentedHi,
I was in front of the same problem as many of you lately: I wanted to import large amount of data, using feeds, into nodes having a "field_collection" field. This "field_collection" field is made of two fields: format, which is a unique "Term reference" field, and "Languages", which is an infinite "Term reference" field.
My goal was to set up a database of films, each film having several "physical support", and each support having a "format" and several "languages". (for instance the movie "Avatar" can be available on two DVDs: first one will have languages English and French, and second one Egyptian and Spanish)
My CSV file has the following format:
Here is how I solved this:
This plugin is really a "quick and dirty made" one, please review it if you plan to use it on a production environment.
==> Everything went well and my films are now created with correct "supports"
Thank you,
Alex
Comment #37
emptyvoid CreditAttribution: emptyvoid commentedI applied the patch from comment 16 and setup a multi-field entity with unlimited nodes on a parent content type.
It worked just fine right out of the box with no tweaks for fixes.
I'm not sure what the patch in 19 does to "fix" it but I found it the patch in 16 to work as advertised.
Comment #38
presleyd CreditAttribution: presleyd commentedI'm not finding the same success.I'm using a Field Collection made up of two fields: an entity reference and an integer. Both show up fine in the mapping UI but running the importer does not seem to bring either in. I wondered if the entity reference was causing the problem so I removed it from the import but to no avail. A infinite field collection made up of a single value integer didn't import either. I'm using Field Collection 7.x-1.0-beta4 and Feeds 7.x-2.0-alpha4+48-devEdit: My integer field had decimals in it and the entity reference doesn't work. Using a node reference from References and a text field it worked fine. In short I think this is working properly.
Comment #39
Shiraz Dindarsuccess with #16. thanks!
Comment #40
arvinsingla CreditAttribution: arvinsingla commentedThe patch in #16 has been working well for us save one issue. We are trying to fill in field collections where the data contains 0's. Currently the code does the following:
This is causing any 0's from being skipped. As a trade off I have updated the patch to include the following.
Comment #41
arvinsingla CreditAttribution: arvinsingla commentedUhg... Ignore the patch above I actually had to make it $v != 0 instead of $v !== 0
Comment #42
johnv@arvinsingla,
try this:
see also http://zainal.wordpress.com/2006/04/25/8/
Comment #43
franzThis is a major issue with Feeds -> #1107522: Framework for expected behavior when importing empty/blank values + text field fix
Comment #44
timfernihough CreditAttribution: timfernihough commentedHi franz.glauber,
I am working with arvin.singla on the same issue and our particular implementation requires the ability for us to map to a field collection nested inside a field collection. In your patch, for field_collection_feeds_processor_targets_alter, you immediately begin by ensuring that the nested entity is not of type field collection which of course is the reason why we cannot map to a nested field collection in feeds. Is there a reason behind why you did this? We have temporarily commented out this check on our local and we can then see the ability to map to these fields but we are concerned you did this for a reason.
Can you please share why you put in that check? Thanks so much for your time.
Comment #45
franzhi timfernihough,
AFAIR, this doesn't work well with multiple items (a multiple field_collection inside a multiple field_collection), beacuse Feeds cannot discern which piece of content belongs to which. If that's not your case, it should work ok too.
Comment #46
timfernihough CreditAttribution: timfernihough commentedThanks franz! After some debugging we were able to determine that is exactly what is happening.
For now, we've been able to settle by writing a hook to manipulate our incoming data feed on the fly and to avoid the need for feeds to be able to support nested data collections. If time permits, we may see if we can help by writing an update to the patch to help feeds be able to discern which content belongs to which field collection.
Comment #47
yurgon CreditAttribution: yurgon commented#16 Dont work with term link field ;(
any ideas?
Comment #48
franz@yurgon, what do you mean with don't work? Doesn't show up as a target or shows up and doesn't set a value?
Comment #49
GaëlG#42 is indeed needed when empty values make sense, for example with csv files like:
organization,contactsname,contactsmail
"Drupal","John|Jack|Joe","john@drupal.test||joe@drupal.test"
Comment #50
emilyf CreditAttribution: emilyf commented#16 is working for me, great work!
Comment #51
Norberto Ostallo CreditAttribution: Norberto Ostallo commented#16 is working fine for me too.
Why is this issue postponed? I don't think we need to wait for #1033202: [Meta] Generic entity processor to commit this as long as we simply add field collection fields as mapping targets. I will set this to "needs review", but feel free to prove me wrong on this. We could at least provide a way to import field collection fields with this patch until a generic entity feeds importer is created.
Comment #52
emilyf CreditAttribution: emilyf commentedseems like a good point -- once i added this patch from #16 and made the modifications i mention here http://drupal.org/node/1033202#comment-6218624 it allowed me to map my entity with field collection fields.
Comment #53
liquidcms CreditAttribution: liquidcms commented@franz and @tim, re: #46, yes, this is exactly what we are trying to do as well... import into a collection which in turn has a collection.
@tim - can you share what the hook code was you used for doing this?
@franz or @tim - i'll take a look at this as well tonight; but if either of you are interested in making this support nested collections; the project i am on could likely fund this work.
@franz... thanks a ton for patch in #16; seems to be working very well. with the feeds_tamper module and a little code in a _feeds_after_parse and a _feeds_presave we have been able to import into a bilingual node with a multi collection field
Comment #54
liquidcms CreditAttribution: liquidcms commentedmade a few assumptions and went for broke.. and surprised i got valid data imported (but not correct).
my structure is a node with a collection field (up to 3) and in that collection i have another collection (unlimited)
i removed the line in patch in #16 disallowing nested collections and sure enough i now get my nested collection showing up in the mapping options.
then i went out on a limb as to how to add data to the cell to represent this and assume feeds_tamper could do a nested explode.
so cell has this for one of the fields in the nested collection: yes,no%%yes,no,maybe%%yes,no,never
so basically in the 3 top level collections:
yes,no
yes,no,maybe
yes,no,never
and, oddly enough.. no errors and what i get imported is:
yes
no
yes
so only 1 item added to nested collection and seems random; but nested explode must have at least worked.. :)
Comment #55
liquidcms CreditAttribution: liquidcms commentedok, so yes, no, yes result is simply because feeds_tamper doesn't do what i hoped for explode.. it simply explodes into 1 long array as opposed to making nested arrays.. so i get yes, no, yes, no, maybe, yes, no, never..
Comment #56
liquidcms CreditAttribution: liquidcms commentedpretty late here so maybe not thinking all that straight.. but i think if we could get feeds_tamper to give a nested array then this should work but the field_collection_feeds_set_target() i think needs a recursive section which i don't see here.
Comment #57
liquidcms CreditAttribution: liquidcms commentedwow.. and this works.. :)
so the recursion is there in the fact that it calls the callback with an array; which in turn calls the same routine using the items of the array.
so the only issue here is getting the feeds_tamper module to do a nested explode.
Comment #58
liquidcms CreditAttribution: liquidcms commentedok, long night.. the patch i added here: #1688536: support nested explodes allows feeds_tamper to create a nested array.
that combined with the patch in #16, except removing the line that prevents it from doing nested, allows importing into nested collections.
Comment #59
charlie-s CreditAttribution: charlie-s commented@franz #16 working most well with an unlimited field_collection that contains many fields, using the JSONPath parser and Feeds dev. Hoping I don't have to stumble down the nested-unlimited-field-collections with this project, but who knows. You need to supply a link to paypal you beer (or coffee/tea).
Comment #60
yurgon CreditAttribution: yurgon commentedhi,
Dont update value of reference field in field_collection
Any ideas?
Thx
Comment #61
liquidcms CreditAttribution: liquidcms commentedhmm.. although my solution in #58 allows me to import into nested collections; it breaks re-importing (i.e. importing modified record to same node).
Comment #62
dotpex CreditAttribution: dotpex commented@mckeen.greg
@http://drupal.org/node/1063434#comment-4531630
Thanks!!!
Comment #63
gmclelland CreditAttribution: gmclelland commentedThe patch in #16 worked for me. Thanks everyone, huge timesaver!
Comment #64
seehive CreditAttribution: seehive commentedGreat!!! Patch in #16 works perfectly, included with unlimited field collections. just need to explode the "," for that particular field
Comment #65
argo27 CreditAttribution: argo27 commentedHi I have copied the patch in #16 renamed in fcollection.patch
Save the file to the root directory of feeds module sites/all/modules/feeds/
Then apply the patch by using git bash application
I typed
git apply -v fcollection.patch
and it returns and error
"checking patch field_collection.module...
error: field_collection.module: No such file directory"
I hve a field_collection module installed in my site.
So what I did to experiment is to copy the
field_collection.module from field_collection module folder and paste it
inside feeds module folder and run the patch again.
Then I got this error:
"error: patch_failed: field_collection.module:1142
error: field_collection.module: Patch does not apply."
How can I apply the patch successfuly? I badly needed this field-collection - feeds mapping integartion.
Im using drupal acquia in windows 7 OS
Comment #66
tim.plunkettThe patch no longer applies. Someone will need to reroll it.
Comment #67
argo27 CreditAttribution: argo27 commentedThanks for the reply.
So what can I do for now?
Comment #68
argo27 CreditAttribution: argo27 commented#16: field_collection-feeds_integration.patch queued for re-testing.
Comment #69
GaëlGIf you applied #42 but do not want empty field collection items to be saved, here's a patch.
Comment #70
btopro CreditAttribution: btopro commentedreroll of #28 against latest checkout of dev for testing
Comment #71
btopro CreditAttribution: btopro commentedOne issue noted with the targets in this patch: It's unable to do nested field collections. For example I have a type that goes node->field_collection->field_collection. it doesn't drill down into the deeper field_collection
Comment #72
btopro CreditAttribution: btopro commentedremoving
Enables at least 1 level of depth to sub-field_collections and works!
Comment #73
btopro CreditAttribution: btopro commentedHere's a helpful transformation method for getting your csv file to conform to the standard needed in order to import to feeds. This takes multiple rows and globs them together based on the unique key for import (in my case 'course')
Comment #74
argo27 CreditAttribution: argo27 commentedI always get this error whenever i apply any of the patch here for fieldcollection module:
error patch failed: field_collection.module:1859
error: field_collection.module:patch does not apply
I use drupal 7 field collection v 7.x-1.0-beta4
feeds v -alpha6
Is it not compatible?
Comment #75
argo27 CreditAttribution: argo27 commentedHow can I re quest for a re-roll of patch in #16 for field collection 7.x-1.0-beta4?
Comment #76
btopro CreditAttribution: btopro commentedneed to apply against dev. if you have issues applying then do it manually, it's a pretty basic patch (just remove + signs and add the two functions to the end of the .module)
Comment #77
argo27 CreditAttribution: argo27 commentedThank you so much btoro.
Your manual solution recommendation works.
I am no programmer and I have no idea I can just copy and paste the code of the patch to the module file.
Thanks so much. You saved me from several days of headache :)
Comment #78
argo27 CreditAttribution: argo27 commentedCan i also do the same thing to other patches and modules. What if I see a - sign than a + what does it mean? Does the patch always pasted on the .module file?
Comment #79
btopro CreditAttribution: btopro commented- means remove that line + means add that line. If you see a minus followed by a + that'll most likely mean replace this with that.
Comment #80
argo27 CreditAttribution: argo27 commentedThanks, it makes more sense,
sorry to ask you another question just to confirm
it this code:
diff --git a/entityreference.feeds.inc b/entityreference.feeds.inc
index 093fa5f..d90ecf9 100644
--- a/entityreference.feeds.inc
+++ b/entityreference.feeds.inc
@@ -16,14 +16,42 @@ function entityreference_feeds_processor_targets_alter
should I delete the "@@ -16,14 +16,42 @@" symbols before the function
and this means that I should paste it over the entityreference.feeds.inc file right?
Thanks
Comment #81
dgastudio CreditAttribution: dgastudio commentedi'm not sure, but after apply the patch, only the array[0] is imported.
i mean, from csv file like this.
"110";"city1,city2,city,3";"1,2,3";"a,b,c";"1,2,3";"i,b,e"
after import, field collection looks like
city, 1, a
nothing more.
field collection it's set to multivalue.
feeds is set to explode columns 2,3,4,5.
column 1 is node id. unique.
field_collection = latest dev
feeds = latest dev
any advice?
Comment #82
David_Rothstein CreditAttribution: David_Rothstein commentedThe issue raised in #40 (where a field collection with any empty fields inside of it is being treated as an entirely empty field collection and not being saved) is still a problem.
It's possible that #42 combined with #69 is the correct fix for that, but since #69 is a much more far-reaching change and the problem with empty field collections is being specifically introduced here during the Feeds import process, here is a reroll of #16 (which seems like it was the last working patch) with code to deal with that problem during the Feeds import process as well.
With this patch applied, a field collection with empty fields inside of it can still be imported, but an entirely empty field collection will be correctly skipped. (Whether or not the empty field inside of the not-entirely-empty field collection is itself imported or not depends on the properties of the field and is a separate issue, i.e. #1107522: Framework for expected behavior when importing empty/blank values + text field fix).
Comment #83
btopro CreditAttribution: btopro commentedSame patch with Sorcery added back in, you can do nested field_collections via this method.
Comment #84
Everett Zufelt CreditAttribution: Everett Zufelt commentedI haven't read through the entire thread. Does the proposed solution deal with removing (deleting) unnecessary field collection items on update?
If you have an entity with a field collection imported, and on update to that entity using fields find that a single item in that collection is no longer required, is it removed?
Comment #85
David_Rothstein CreditAttribution: David_Rothstein commentedJust a note: I had a situation where I needed to deal with a multi-valued field inside of a multi-valued field collection, which brings up the question of how do you distinguish between a single field collection item (that contains multiple values for the given field) and multiple field collection items (each of which contains only one value for the given field)? I posted a Feeds Tamper patch at #1819986: Add a plugin for reducing an array to a single element which was helpful in getting that to work.
(This comment is unrelated to Everett's question right above mine, by the way. I don't actually know the answer to that myself.)
Comment #86
g089h515r806 CreditAttribution: g089h515r806 commentedI have create a module http://drupal.org/project/field_collection_feeds which solve this issue in another way.
Comment #87
yurgon CreditAttribution: yurgon commentedthis module dont work 4 me
Comment #88
gboggs CreditAttribution: gboggs commentedWith the latest dev version of Feeds and Field Collection, and the Feeds Field Collection module, I can get field collections to import, and saving user profiles with embedded field collections no longer cause blank collections to be created, but I can no longer use feeds to import new users.
"SQLSTATE[23000]: Integrity constraint violation: 1048 Column 'field_listings_revision_id' cannot be null"
Ideas?
Comment #89
John Bickar CreditAttribution: John Bickar commentedThe patch in #83 would not apply for me ("fatal: corrupt patch at line 125") so I added the functions to the module manually, cleared caches twice, and it worked!
I re-rolled the patch against 7.x-1.x-dev.
Comment #90
ashedryden CreditAttribution: ashedryden commentedTested patch #89, applied cleanly and seems to have no issues.
Comment #91
xcf33 CreditAttribution: xcf33 commentedCreated a patch based on patch #89 (please apply patch on #89 before this).
This is purely for my use case but hopefully it could work for other folks. I wrote this very quickly and it is minimally tested.
In any case my issue was:
I have a field collection field on a node that has "Unlimited value", that field collection item contains 2 fields
1. Name
2. Email
When I create my mapping for feeds I mapped the [field_collection_item:name] and [field_collection_item:email] twice hoping that the field will create multiple instances of the field collection items i.e. $field[LANGUAGE_NONE][$delta];
However, looking at the code it looks like only if the mapping is passing in an array (so have to use feeds tamper or similar tool) in order to create multiple instances of the field collection item. Otherwise $delta will always be 0 and mapping the field collection item fields multiple times will always be overwritten by the last set of mapping.
Comments and feedbacks are welcomed!
Comment #92
arh1 CreditAttribution: arh1 commentedfranz/.glauber: thanks for saving my Friday :)
#16 successfully adds Field Collections as targets available for mapping. Cheers!
Comment #93
AaronBaumanhere's a new patch that includes both #89 and #91 (minus the extraneous whitespace changes)
i'm reverting back to "needs work" because after apply this patch,
i get the following Integrity constraint violation on import:
Integrity constraint violation: 1048 Column 'field_document_details_revision_id' cannot be null
This error may be related to #1810548: "Integrity constraint violation column [xyz]_revision_id cannot be null" when saving entity with attached field collection, in which case we'll have to postpone on that issue.
Regardless, though, the "multiple-values" behavior is a bit off.
For example, a multi-valued data source results in one field collection for each column of each row in the data source, instead of one field collection for each row in the data source.
Comment #94
AaronBaumanAnother issue I've run into with the approach in the current branch of patch development: updates to field collection items are more or less ignored.
If the parent entity is getting updated, and field collections are already assigned, they do not get updated with the new feed data.
I don't think this should necessarily prevent inclusion of this patch, but I thought it worth noting for dev and testing purposes.
Comment #95
ledom CreditAttribution: ledom commentedTested #93 patch on last dev version, works fine but any news about handling multi-values? One field collection for each row would be better.
Comment #96
praandy CreditAttribution: praandy commentedHi,
I have tried this approach with a complex field collection structure, but it does not work fine.
It couldn't match a nested field structure, for example a xml file:
The source
(text field)
(text area)
The imported content
(text field)
(text area)
I use json parser and xml parser (for this I have applied this patch http://drupal.org/node/1958660).
We need to transfer content "drupal-to-drupal" and we need to transform entity_reference, files and link of a field collection item in unique link field, so we create a custom "form_callback".
So I develop a patch (I have to thank Hernani for helping me).
Could you verify it, if it could be a good solution?
Comment #97
korn3lius CreditAttribution: korn3lius commentedHi,
First of all, thanks for the nice works for field_collection and feeds integration.
Patch #16 work fine for me, however, it only import first field and ignored the others.
Let me explain it clearly.
I had field collection with 2 fields => "visit" and "body weight"
and my csv files look like this:
visit | body weight
1 | 60
2 | 70
3 | 80
Patch #16 simply import the first field (visit 1 with body weight: 60) and ignored visit 2 and 3.
How can I import all the data properly. I think I must miss something here.
My environment:
Drupal: 7.21
Field collection: 7.x-1.0-beta5+0-dev
Feeds: 7.x-2.0-alpha7
Thanks you
Kornelius
Comment #98
vinmassaro CreditAttribution: vinmassaro commentedPatch in #93 working for me for basic mappings. For multi-value fields, I had to process myself with hook_feeds_presave().
Comment #99
Shiraz Dindarvinmassaro, can you post what you did in hook_feeds_presave()?
Comment #100
vinmassaro CreditAttribution: vinmassaro commented@Shiraz Dindar: Sure. Here is what the XML source looked like that needed to be mapped into the field collection. I am using Feeds XPath Parser.
I had a field collection containing four fields:
I couldn't get the multi-value country field to populate with all the countries. What I did was map the entire internationalActivity element to a temporary target (mapped as raw XML) instead of a field. I then converted it to a PHP array inside hook_feeds_presave() and took it apart to programmatically create the field collection items. If there are items, I remove and recreate them when the feed node updates since this was easier than trying to keep track of them. I used this page as a guide to programmatically create the field collection items. Hope this is helpful!
Comment #101
xcf33 CreditAttribution: xcf33 commented@aaronbauman re #93
I'm working on making the patch work from #93 for the following scenarios:
1. Multiple field collections, this means a node can have multiple field collection attached to it.
2. Multiple value fields in a field collection. This means create a field in the field collection that can have multiple values.
There are more complex cases such as field collection inside of field collection but I think this should work for that or at least maybe not supported at the moment.
It's a good note that field collection is not being updated when parent is updated by feeds, this is something we should build in as well.
Comment #102
xcf33 CreditAttribution: xcf33 commented@aaronbauman re #93
Patch based on #93 that should handle (Please patch against 7.x-1.0-beta5)
1. Multiple field collections, this means a node can have multiple field collection attached to it.
2. Multiple value fields in a field collection. This means create a field in the field collection that can have multiple values
This has been tested initially, need to also look at the update scenarios.
Did not test against field collection inside of field collections.
Comment #103
fehin CreditAttribution: fehin commentedJust applied patch in #102 and below is the message I got
Comment #104
codesmith#102 worked for me for a 2 field, field collection. I tried to add multiple values using a colon but that failed - not sure if it's supported or if I didn't set it up right. For example, using something similar to #20:
"fruit","color"
"apple:banana","red:yellow"
I got just one field collection entry:
fruit: apple:banana
color: red:yellow
instead of the expected:
(entry 1)
fruit: apple
color: red
(entry 2)
fruit: banana
color: yellow
Comment #105
Shiraz Dindar@vinmassaro -- a belated thanks for sharing your code. I didn't end up proceeding with this but I found your code and comments illustrative nonetheless.
Comment #106
franzThere are some ugly tabs and whitespace in this section
Comment #107
arh1 CreditAttribution: arh1 commented#102 (patched against 7.x-1.0-beta5) allows the field collections to be set as targets again, but doesn't save the values of text fields in field collections for me.
Digging a bit, I see that in text_feeds_set_target() (from feeds/mappers/text.inc) $field['und'] contains an element, whereas it's an empty array for the non-field collection fields I'm testing. This means that the foreach loop is always broken before setting the value in text_feeds_set_target() via this bit:
So I'm trying to understand enough of Feeds and the Field Collection integration discussed here to come up with a reasonable solution... I'm surprised no one else in this thread has hit the same problem.
Comment #108
arh1 CreditAttribution: arh1 commentedRe: my previous comment...
Oops, now I see the notes in #102 and #94 above.
Regarding updating existing field collection items, if I clear the value of the field's target in the entity object inside of field_collection_feeds_set_target before the callback with something like this:
..then the value of $delta inside of text_feeds_set_target in feeds/mappers/text.inc is set to 0 as desired, so that the field is iterated over once and the value saved correctly. This does not seem to break my initial import of the data, either.
This feels like a hack, though, and I'm sure it breaks some of the scenarios above, so I'm not providing a patch yet. Can anyone who's been working on this longer than I have comment on this approach?
Comment #109
arh1 CreditAttribution: arh1 commentedMeh, here's a patch after all, just to make it easier to discuss. Please don't apply to this to any site you care about.
I fixed the whitespace issues per #106, too.
Comment #110
gjones CreditAttribution: gjones commentedHi Everyone,
I was able to get Feeds 7.x-2.0-alpha8 to work with Field Collection 7.x-1.0-beta5 by utilizing the Explode feature from Feeds Tamper 7.x-1.0-beta4
Step 1: Apply the patch https://drupal.org/node/1063434#comment-7484110 onto Field collectoin 7.x-1.0-beta5
Step 2: This is my how spreadsheet is setup.
Name | Title | Phone | Email
-----------------------
Bob, Joe, Sally | Sir, Mr., Mrs. | 604-555-5555, 604-123-4567, 604-555-0011 | bob@test.com, joe@test.com, sally@test.com
John, Adam, Susan | Dr., Mr., Miss | 604-555-1111, 604-123-2222, 604-333-5511 | john@asdf.com, adam@asdf.com, susan@asdf.com
Note: Each row is an entire field collection and each column is a single field collection field. Also, I'm exporting as a TAB separated value, and having to convert it to UTF-8
Step 3: Setup your feeds mapping to the use the field collection fields.
Step 4: Setup Feeds Tampers
Using Feeds Tamper (must apply quick fix here: https://drupal.org/node/1929312#comment-7166832), I set a new Explode field tamper plugin for the each of my comma separated columns. Note: Your explode string must match your spreadsheet exactly. In other words, if your spreadsheet has a space after the comma, then your explode string needs to have this space for it to work.
Then run the feeds import.
Comment #111
arh1 CreditAttribution: arh1 commentedTo reiterate quickly, #109 is based on #102 but adds support for updating existing entities (via the line in #108) and cleans up some whitespace problems (I should have provided an interdiff).
I'm taking a top-down look here and providing a pragmatic fix for my use-case, but if #102 was working for people it's worth reviewing.
Comment #113
jeromec CreditAttribution: jeromec commentedHello,
i would like to be able to import a node with this structure :
-nid
-title
-fieldcollection :
--myfield1 (term reference with widget set to Autocomplete term widget (tagging), 1 value)
--myfield2 text
All is working perfectly except the insert of myfield1 who is not working. When i change the widget type of Select list it's works. But i want to be able to insert new term reference, the only way to to this is use the Autocomplete term widget (tagging) type.
i have configured my feed parser to CSV, delimiter (;) , processor node, in my target configuration for myfield1 i have checked Auto create and the search taxonomy term is by Term name.
Feeds : 7.x-2.0-alpha8
Field collection : 7.x-1.0-beta5
i also have applied the patch #16
I dont know what to do.
Can someone help me please ?
Comment #114
azin CreditAttribution: azin commented#109: field_collection-feeds-1063434-109.patch queued for re-testing.
Comment #116
azin CreditAttribution: azin commented#109: field_collection-feeds-1063434-109.patch queued for re-testing.
Comment #118
twistor CreditAttribution: twistor commentedJust a note, this should not be using feeds_alter(). A regular drupal_alter() will suffice.
Comment #119
humansky CreditAttribution: humansky commentedPatch #109 won't work for me because it is looking for the module in "sites/all/modules/field_collection/". That is probably also why it is failing testing. I've rerolled the patch with a more relative path to the modules' files
Comment #120
humansky CreditAttribution: humansky commentedComment #121
queenvictoria CreditAttribution: queenvictoria commentedI've updated #119 as work so far doesn't pass in the mapping to the callbacks on the sub_targets. Thus settings such as 'autocreate' on taxonomy fields don't passed in and therefore new terms don't get created.
See http://drupal.stackexchange.com/questions/83527/how-to-import-term-refer... for more information
Interdiff attached too.
Comment #122
jrao CreditAttribution: jrao commentedThis:
and this
doesn't seem to be correct, this makes it impossible to import multiple field collection items that has a subfield with multiple values (i.e. #36), in which case this code will just put everything in the first item's multi-valued field.
The code shouldn't care about cardinality of sub fields, or try to breakup the values to fit the sub fields, that's the job of tamper.
I think this should be changed to:
and
Comment #123
jrao CreditAttribution: jrao commentedA revised explode twice tamper plugin, fixed a small bug in the version from #36
Comment #124
ikeigenwijs CreditAttribution: ikeigenwijs commentedDo I understand correctly
That you need to do 2 imports with different processors
I have csv with
all node information including the collection field 1 level deep only
What i m trieing to do now, not succeeding yet is the following
Is 2 imports
1 for the node data with node processor
1 for the collection field date (https://drupal.org/project/field_collection_feeds)
both mapped to the unique field
csv file:
Title; uniq R number ; a lot of simple fields on the node ; collection field storage
"proc35";35; .a lot of fields; closet1.5.3.2,closet2.3.4.2
thx a lot for the help and directions
Comment #125
chhay CreditAttribution: chhay commented#16: field_collection-feeds_integration.patch queued for re-testing.
Comment #126
GaëlG#121 with #122 edits did not work well on our multilingual website. We had to change
to
in field_collection_feeds_set_target().
Comment #127
GaëlGFor anyone facing the same need, here is the code that worked for me to import an entity_translated field collection (with this patch: https://drupal.org/comment/8016755#comment-8016755) with parent field set untranslatable and some subfields set as translatable.
Comment #128
vladimir-m CreditAttribution: vladimir-m commentedHi GaëlG,
Thank you very much for this solution. Works fine for me.
Comment #129
colanPlease provide the above code in patch format.
Comment #130
lachezar.valchev CreditAttribution: lachezar.valchev commentedHi all,
I took the initiative to put GaëlG's code from #127 into a patch.
All credits go to GaëlG.
Regards,
Lachezar
Comment #131
pebosi CreditAttribution: pebosi commentedPatch and importing of field_collection fields works fine for me. Thanks
Comment #132
snowfly CreditAttribution: snowfly commentedHi which version of feeds does the patch in #130 works with
thanks
Comment #133
pebosi CreditAttribution: pebosi commentedI used latest dev.
Comment #134
ikeigenwijs CreditAttribution: ikeigenwijs commentedA short follow up,
I managed to do the import for field collections in field collections structure,
I added a custom uid (base on different fields) of the parent in all the children and used this field to map all the children to the correct parrent while the csv contains 1 row for each parent or child.
Comment #135
HarlsHicks CreditAttribution: HarlsHicks commentedI'm having issues with these patches. I've tried a few of the working ones, even #130. I'm coming up with the same issue, the callback $v is supposed to be an array, but for some reason it's not getting sent as one. I'm using the latest dev of both field collections and Feeds. Here's the error I get with patch #130:
ResponseText: Recoverable fatal error: Argument 4 passed to file_feeds_set_target() must be an array, string given, called in /modules/field_collection/field_collection.module on line 2018 and defined in file_feeds_set_target() (line 51 of /modules/feeds/mappers/file.inc).
Anybody have any ideas?
Comment #136
jason.blanda CreditAttribution: jason.blanda commented#135 All of the above patches will not work with Feeds 7.x-2.x-dev use Feeds 7.x-2.0-alpha8 #130 worked perfectly.
Comment #137
caw67 CreditAttribution: caw67 commentedptach #16 works great! can you implement it in the module?
Comment #138
colanNeeds a re-roll based on #136. All patches must apply cleanly to the dev branch.
Comment #139
AtlasX CreditAttribution: AtlasX commentedWorking without errors using patch #121 with a multi-value field collection containing a node reference field and a text field.
Tried patch #130 but gave me this error:
Notice: Undefined index: language in field_collection_feeds_set_target() (line 1976 of /home/ramjet/public_html/testsite/modules/field_collection/field_collection.module).
Still looking to update existing fields which I've been unable to do. It still won't remove fields that are no longer in an updated XML file or make changes to existing ones. It sometimes causes duplicates as well.
I've been using Feeds and importing XML for a while now without this module and it always works great. In that case, my new import seems to overwrite all existing fields and repopulates them from the updated XML, which, again, works great for what I need it for.
Comment #140
dwadson CreditAttribution: dwadson commented#135 - I added a line before 2018 to force $v to an array:
Comment #141
rcodina CreditAttribution: rcodina commentedPatch #16 work for me on Feeds 7.x-2.0-alpha8 However, with Feeds 7.x-2.0-alpha8 I'm not able to import images and files from their URI and I really need this feature. Could anyone do a patch which works with Feeds 7.x-2.x-dev?
Comment #142
johnvBetter title for my dashboard.
Comment #143
k_zoltan CreditAttribution: k_zoltan commentedRe-rolled patch from comment 130 #1063434-130: Add Feeds integration to FieldCollection so that it clearly applies and work with the most recent version of Feeds and Field Collection.
This patch also includes the fix suggested in comment 140 #1063434-140: Add Feeds integration to FieldCollection
Please note that the most recent dev vesion of Feeds includes the patch for the general Entity Processor #1033202: [Meta] Generic entity processor https://www.drupal.org/node/1033202
Comment #144
k_zoltan CreditAttribution: k_zoltan commentedLets see if this passes the tests ... if yes than I think this is ready to be reviewed.
Comment #145
JParkinson1991 CreditAttribution: JParkinson1991 commentedApplied patch #89 and it works perfectly so thanks alot for that.
I have a problem however, let me lay it out.
I have a field collection 'FAQ TAB'
Inside this i have the fields 'Tab Title', 'Tab Content'
'Tab Content' is a field collection containing 'Title', 'Body'
'Tab Content' can have unlimited values, so an unlimited number of title and body fields linked together.
How can i add the multiple title and body fields through the feeds importer?
I know i can use feed tamper and explode it etc, but this doesnt seem to work for unlimited value field collections.
Where do i start in trying to solve this?
Apologies for the lack of effort on my part, only been working within drupal for about a month and situations similar to mine seem not to exists on the internet.
Any help would be massively appreciated
Comment #146
hauruck CreditAttribution: hauruck commentedGreetings
I have just patched with the #143 and it works pretty well. My only issue remaining is how I use this feed integration with fields that have multiple values. Here is my example:
source CVS -> field collection field in drupal
Author 1: First name -> Author: First name
Author 1: Last name -> Author: Last name
then I map the next author to the same fields hoping with would create anothe entry:
Author 2: First name -> Author: First name
Author 2: Last name -> Author: Last name
unfortunatly only one value gets imported, the Author 1.
Any suggestion on how to import multiple value fields?
Comment #147
priyanka12chavan CreditAttribution: priyanka12chavan commentedRecently we faced similar issue where we need to add unlimited specifications for a each product. We created field collection with fields Category, Label, Value1, Value2 with unlimited fields. We came across Field collection feeds module. It's docs https://www.drupal.org/node/1831004 provides proper steps to setup importer.
Comment #148
imperator_99 CreditAttribution: imperator_99 commentedI've successfully applied the patch in #143 to no avail - my field collection fields don't show up as targets in feeds. I'm using field_collection 7.x-1.0-beta7 and feeds 7.x-2.0-alpha8. There's no errors in the logs, not really sure what to do from here...any advice appreciated!
Comment #149
hauruck CreditAttribution: hauruck commentedI found a solution to the multi value problems I was having. Use the tamper module and follow this guide:
http://thejibe.com/blog/12/09/feeds-importing-multiple-values-one-multi-...
Comment #150
bbdata CreditAttribution: bbdata commentedFor anyone using 7.x-1.0-beta7 and doesn't need the functionality from #127/#130/#143. Use the patch from #121.
Otherwise you will lose the ability to configure the target of taxonomy term references.
Comment #151
bbdata CreditAttribution: bbdata commentedSorry, changing version back.
Comment #152
bbdata CreditAttribution: bbdata commentedI would like to also add that there is an unexpected behavior found with a node reference field in a field collection item.
When the feeds importer is set to update a node with a field collection field that contains a node reference field, the node reference field is not updated. This is because the references_feeds_set_target() of the references module would append the source value to the existing array of values. This ends up getting ignored when the node reference field only allows one single value.
I am not 100% sure if this is a bug in the references module or in patch #121. A test was conducted by mapping directly to a node reference field that was part of the node, and it correctly updated with the new value. However, this will not happen when the reference field is inside a field collection item.
The following patch to #121 is rather specific and corrects this by emptying the reference field if it only supports one value before it is passed to references_feeds_set_target(). What happens now is that the source value gets correctly updated.
Perhaps someone with a better understanding of both modules can chime in where this fix will better fit.
Comment #153
GaëlGI could not have it work to delete field collections removed from the source. Besides, according to #1107522: Framework for expected behavior when importing empty/blank values + text field fix, the new default behavior is to erase field values for blank fields in source. So I made a new patch to handle this all.
It includes #127.
I removed the lines about $subfield because these variables are never used.
#130 is not included because it added
without explaining why and not even notifying it was added. It looks wrong to me because a field inside a field collection could be emptied in the source and this should empty the fied in Drupal on update. Avoiding empty collections creation is a task handled by
field_collection_feeds_presave()
.I did not keep #140 either because I don't see why this should be handled by the Field Collection implementation of feeds_set_target(). If there is a bug because $v is a string, the bug is either caused earlier (the function should be called with $value as an array of arrays) or more probably later in the subtarget mapping callback (the callback should be able to handle string values, it's something that
text_feeds_set_target()
does for example). I might be wrong on this but at least it is an argued point of view.Comment #154
jaysonjaynes CreditAttribution: jaysonjaynes commented@GaëlG, that one worked for me! I was having the issue of importing in a "0" for a decimal value on an exploded field, and it was ignoring it and shifting the next value into that position. Thank you!
Comment #157
Novitsh CreditAttribution: Novitsh commentedApplied patch from #153 and added the field collection field in my Feeds importer.
Getting this error back:
An AJAX HTTP error occurred. HTTP Result Code: 200 Debugging information follows. Path: /batch?id=167&op=do StatusText: OK ResponseText: Fatal error: Class name must be a valid object or a string in /drupal/sites/all/modules/contrib/entity_translation/includes/translation.handler_factory.inc on line 93
EDIT: disabling feeds_et did the trick there.
I've also figured out that for me it's working by only making the FC host translatable, not the child fields, and patching this line after #152:
Comment #158
SocialNicheGuru CreditAttribution: SocialNicheGuru commentedI could not get 153 to work.
Comment #159
mkhamash CreditAttribution: mkhamash commentedPatch #153 did work in my case but I think it needs more work to cover more use cases.
Comment #160
troybthompson CreditAttribution: troybthompson commentedI applied #153 to Field collection 7.x-1.0-beta8+11-dev (2015-Mar-26) and I'm using Feeds 7.x-2.0-beta1
Only some fields in the collections I have showed as available to be mapped. In my case, Entity Reference fields showed, but Text, Decimal, and Date did not. Thoughts?
Comment #163
firoz2456 CreditAttribution: firoz2456 commented@troybthompson Can you elaborate more, what type of fields you are trying to map?
Comment #164
troybthompson CreditAttribution: troybthompson commentedI don't know how much more details I have. Without the patch, none of the fields within my field collections show up for mapping.
I have multiple field collections in the content node type I'm trying to import to.
After the patch, only entity reference fields in my field collections show up to map. Within the same field collection that the entity reference field shows, there is also a text field that's not showing up to map. There are two of those field collections that are acting the same way. I also have a field collection that only contains a date field that is not showing up.
All fields were showing up with an older version of feeds.
Does that help? If there's any other details that would help, please let me know. I'm stuck with some feeds importers needing the new version of feeds and some needing the old method.
Comment #165
ScorpionGhost CreditAttribution: ScorpionGhost commentedtroybthompson, patch does not work with Field collection 7.x-1.0-beta8 and Feeds 7.x-2.0-beta1. Help!
Comment #166
troybthompson CreditAttribution: troybthompson commentedHas anyone been able to map to core fields inside field collections with the latest versions of things?
Comment #167
veronicaSeveryn CreditAttribution: veronicaSeveryn at Inclind Inc commentedI had a need to implement field_collection field through Feeds, and also discovered that the existing patch isn't longer working with the most recent code.. So I tried to re-work it a little. Attaching a patch (I am currently using it for my needs on one of the sites and it works for me). Not sure if it will need any more adjustments for more complex fields inside field_collection structure, but looks like it does the job at this moment.
The patch is based off of 7.x-1.0-beta8+11-dev field_collection code (March 26th, 2015 1:43:07 PM)
Comment #168
veronicaSeveryn CreditAttribution: veronicaSeveryn at Inclind Inc commentedComment #173
veronicaSeveryn CreditAttribution: veronicaSeveryn at Inclind Inc commentedUploading revised patch from comment #168 - I think I found a way to save complex fields within Field Collection structure. Tested with saving dates, text, numbers, lists.
Comment #174
Zatox CreditAttribution: Zatox commentedWorking for text fields with Feeds 7.x-2.0-beta1 and Field Collection 7.x-1.0-beta8
Thanks for updating the patch @veronicaSeveryn !
I think we'd need more testing to update the issue to RTBC. Especially on the complex fields... @troybthompson maybe?
Comment #175
peleo CreditAttribution: peleo commentedSorry but it did not work for me with Feeds 7.x-2.0-beta1. Doesn´t let me mapping field collections...
Comment #176
troybthompson CreditAttribution: troybthompson commentedThanks! This is working for me in all of my cases so far.
Feeds 7.x-2.0-beta1+6-dev (2015-Aug-05)
Field collection 7.x-1.0-beta8+11-dev (2015-Mar-26)
Now they just have to get Address Field feeds support working again and I'll be back in business. =^)
Comment #177
Novitsh CreditAttribution: Novitsh as a volunteer commentedMany thanks first to veronicaSeveryn. I've updated the patch from #173 a bit to comply with coding standards. Also removed some unused variables and declarations that are directly overriden.
More testing on this patch is still required.
Comment #178
SocialNicheGuru CreditAttribution: SocialNicheGuru commentedwill this let me do the following
my field_collection might have fields within it called field_my_name and field_my_birthday
the import data has a field name and birthday
will the patch allow me to map name to field_my_name within the field collection and birthday to field_my_birthday within the field collection?
Comment #179
veronicaSeveryn CreditAttribution: veronicaSeveryn at Inclind Inc commentedI had a Field Collection with the following field types inside: date, textfield, integer, decimal, addressfield, textarea, and I think URL. I was able to import them all successfully with the patch I had provided.
For the Feeds I am utilizing 7.x-2.0-dev version. And like I said, Field Collection patch was based off 7.x-1.x-dev branch but I was utilizing 7.x-1.0-beta8 in my site and it applied and worked with it without a problem for the fields I had to import.
UPDATING: I apologize, I typed the wrong version number previously: it might not work with FC 1.0-Beta1... So your best bet would be at least BETA 8, which worked for sure.
Comment #180
mark.bennett CreditAttribution: mark.bennett commentedI am currently using #177 successfully in order to recognize the first level of field collections in a content type. However, I have field collections inside of field collections and anything under that first level of field collections is not available to map.
For instance, I have a Profile content type. I have a variety of fields in Profiles (email, text, date, link, integer, etc) as well as numerous field collections; Address, Course, Presentation, etc. Under the Presentation field collection though, I have a few standard fields as well as its own "sub" field collection called Review. As a step farther, under the Review field collection, there is another field collection called Reviewer. So the hierarchy looks like this:
Profile
--> Presentation
------> Review
----------> Reviewer
With #177, I was finally able to recognize and map all field collections at the Presentation-level. However, I still cannot recognize any field collections or fields under that level (nothing in Review or below). Would it be possible to tweak the patch to recurse through this hierarchy?
Comment #181
pcambraI've found an issue when importing a file field inside a field collection, the mapper for file.inc does this:
Which checks the field collection settings and not the file inside the field collection, wondering if the best way around is to provide a custom mapper or to add a set targets alter.
Comment #182
pcambraIn case someone else finds #181, I've added a patch in feeds #2619788: Get instance info from the entity instead of from the source for file mappings, not sure if that's the way to go, but also this gist that handles file fields inside field collections https://gist.github.com/pcambra/9de9997ba8780e1f2a2e
Comment #183
pavlosdan CreditAttribution: pavlosdan commentedI had a similar problem as Mark in #180 and tried to play around with the patch a bit.
After commenting out the return statement at:
The items in the nested field collections started showing up and some test imports showed it worked fine with some simple textfields (against latest Dev versions of Feeds and Field Collection modules).
I am not sure though if this introduces more issues.
Was the taxonomy issue at #121 resolved? I can't get the feed to create taxonomy terms even when I check the auto-create checkbox. I am not sure if this is because the taxonomy term field is within one of the nested field collections. If that's the issue, is there any way to make this work? I've been trying to make sense of the code to see why it wouldn't work but so far no luck.
Any help is appreciated.
Thanks for all the work put into this already!
EDIT: the imported fields create one item each for the top level field collection, not multivalued fields of the nested collection items. More research and testing needed.
Comment #184
Novitsh CreditAttribution: Novitsh at Colruyt Group Services commentedChanged the patch based on the feedback of #183.
I'm wondering why that line was introduced in the first place?
With the patch taxonomy should be able to be created. Could it be a permission issue in your case @pavlosdan? Feeds can create items as anonymous or as a user. In that last case you need to make sure it has the correct permission to create taxonomy.
Comment #186
iajay CreditAttribution: iajay commentedI found one bug in the above patch(s)
When the feed is updated and the rows are deleted from the xml feed then the field collection values are not deleted and causes duplicates
I think I can explain this better with an example
I have a field collection - status_action - with fields action_date and action_text
This is the structure of the xml
When the xml is imported first it is working correctly and I get
2014-12-01 - Proposed Rules
2015-02-11 - Final Regulations
Now suppose the xml is updated to
So basically a value is deleted and the feed is again imported via cron
the new values that I get are
2015-02-11 - Final Regulations
2015-02-11 - Final Regulations
I hope I am able to explain the problem correctly
Is there a way around this?
Can we simply delete the existing values and then import the updated xml or can you guys think of some better option?
Awaiting reply
Regards
Ajjaykummar
Comment #187
nico.knaepen CreditAttribution: nico.knaepen as a volunteer and at Logic in Motion for Colruyt Group Services commentedComment #189
monsterplant CreditAttribution: monsterplant commentedI'm quite confused as to how this patch is meant to work. I have a csv with the following setup:
node title | ID (field collection field 1) | Quantity (field collection field 2) |
test | 1 | 1 |
test | 2 | 2 |
It imports fine for the first row, but for the second one it does not create a new field collection set in the same node. Can someone please enlighten me on how it is meant to be setup?
Comment #190
iajay CreditAttribution: iajay commentedI created a patch, now the code deletes the field_collection before saving.. This is my first patch.. so excuse me if I made some mistake
The patch is based on #184 above
regards
Comment #191
dpfitzsi CreditAttribution: dpfitzsi commentedSwitching to Needs Review so that Simpletest checks latest patch.
Comment #192
iajay CreditAttribution: iajay commentedOne more problem which I can explain better with an example
consider this xml
Considering the above the effective date value is assigned to the wrong position After important it looks like
Action Date:2015-03-19
Action Date: Action 1
Effective Date : 2015-04-23
Action Date:2015-03-19
Action Date: Action 2
Effective Date : 2015-05-23
Action Date:2015-03-19
Action Date: Action 3
Effective Date :
So is there a solution for this?
Regards
Comment #193
iajay CreditAttribution: iajay commentedI will try to simplify this further
In this case the action 2 gets assigned to first record and the action_text is not present in the first record
this is how it shows
Action text: action 1
effective date:2015-03-23
Action text: action 2
effective date:2015-03-23
Action text:
effective date:2015-05-23
Now even if a blank tag is added like then also it gives the same result
Comment #194
djcj CreditAttribution: djcj as a volunteer commentedIs there not any batch processing?
This does not work for large data sets (13,000 rows) (16MB xml file). It will error out with SQL timeout. Increasing/removing the timeout isn't a good approach, because a user will not wait that long to import.
Fatal error: Maximum execution time of 30 seconds exceeded in /home/djcjne5/public_html/music/drupal/includes/database/query.inc on line 1020
Comment #195
djcj CreditAttribution: djcj as a volunteer commentedWorking Alternative: Field Collection Feeds
http://drupal.org/project/field_collection_feeds
I ended up using this module instead, and it can handle BIG data (13,000+ collection items, 3 fields per collection item). Used XML as an input file and XPATH to grab the fields.
Versions I used:
Field collection feeds - 7.x-1.0-alpha3
Field collection - 7.x-1.0-beta11
EDIT:
Import is not actually working into field collections still. It takes too long and times out when running in front end and background job modes. The XML file is 400,000 lines. Will update this post if I make progress.
Comment #196
dpfitzsi CreditAttribution: dpfitzsi commentedTesting out the last patch against a field collection that has multivalue taxonomy checklists inside them appears to fail. Only the first term gets added. This also doesn't allow you explode terms using Feed Tamper (only one will get added).
It appears that in the previous few patches the sub targets carnality check was removed and this appears to be causing the problem. I have added a check back in and set it to use the single value version in the case that there is only one value (like a simple textfield) and to use a multivalue array when there are multiple terms (in the case of taxonomy).
Comment #197
hi.chawla CreditAttribution: hi.chawla commenteddpfitzsi patch is also not working for multiple entries, only first field is getting saved.
Our csv file contains following data
Node Id (Unique)|Field 1|Field 2
876|Ted;Mark|Red;Yellow
only Ted and Red and getting saved for node 876
Comment #198
NWOM CreditAttribution: NWOM commentedChanging back to dev version, since patches are always applied against the dev. Also, removing the tag according to Adding tags to issues and using tags.
Comment #199
arborems CreditAttribution: arborems as a volunteer commentedI also had problems with dpfitzsi patch #196, but was able to get it to work after changing the line "$sub_field_info = field_info_field($sub_target[0]);" to "$sub_field_info = field_info_field($args[0]);" I suspect this is what dpfitzsi originally intended. $sub_target is a string so $sub_target[0] just gives the first letter of the field name.
In my case $sub_target was either 'field_datetime:start' or 'field_datetime:end', so going back to the $args array was necessary to get only the first part, 'field_datetime'.
Comment #200
dpfitzsi CreditAttribution: dpfitzsi commentedThank you everyone for the feedback on my version of the patch. I will make an update this week and roll a new version.
Comment #201
djdevinUsing the patches above I can import to field collections however there is one case that is not working.
When importing to a boolean checkbox field (off/on checkbox) you cannot import to the opposite of the default. That is, if the checkbox is not checked, you cannot import it to checked, and vice versa.
I think what's happening is that the field collection is being created, and the default is being saved. Then when the new value is imported, instead of the value being overridden with what is being imported, it is being appended. The field ends up having a value of 0 and 1.
I made this change to fix the issue, it unsets the field before trying to set values on it. The callback is "list_feeds_set_boolean_target" and if you look at that function, it appends to whatever is there. All other fields seemed to still work.
Comment #202
edvanleeuwenI am not able to get the patch working using the latest dev (2016-Apr-29). I noticed that in hook_feeds_processor_targets_alter() the info type returned is entityreference, not field_collection. Am I missing something?
Comment #203
djcj CreditAttribution: djcj as a volunteer commented#201 solved my issue: check boxes were not updating on import
The import still throws errors, but at least it appears to work.
Comment #204
SocialNicheGuru CreditAttribution: SocialNicheGuru commentedpatch doesn't apply well for me. It stops at
so I get a WSOD:
PHP Parse error: syntax error, unexpected end of file in field_collection/field_collection.module on line 2004
Comment #205
NWOM CreditAttribution: NWOM commentedThat was most likely my fault. I changed the branch to dev, assuming that a patch would be created for it. Since usually only patches that are based on the dev version will get committed. This most likely only applies correctly on 7.x-1.0-beta11. I'll set it back to 7.x-1.0-beta11, just to be safe.
Could anyone else confirm whether this works on the dev release? I'm not currently in the office to double-check the version that I applied this on.
Comment #206
NWOM CreditAttribution: NWOM commentedPlease disregard the last post. It appears that the patch was indeed created for the dev release. Switching back to dev. It may need a re-roll.
Comment #207
sherakama CreditAttribution: sherakama as a volunteer commented[edit]
Aaahhh,
I totally started from the wrong patch. I will update accordingly.
sorry for the spam.
[edit]
Comment #208
sherakama CreditAttribution: sherakama as a volunteer commentedComment #209
sherakama CreditAttribution: sherakama as a volunteer commentedWith the patch in #196 I was only getting one field collection.
I have made a few more adjustments based on that work:
1. Use &drupal_static convention instead of raw static variables
2. Move the feed mappers loading out of a loop to ensure only loads once, even with static var. Minor performance improvement.
3. Remove the field_collection save in each field loop. Even though it was not saving the parent node it was still causing an enormous amount of database writes and had huge performance implications. Just save the field collections along with the parent entity for a write only once boost.
4. Only stop looping at the correct cardinality of a field. It is possible that a field could be set to 2, 3, 10, etc
5. Check for multiple values on callback save incase feeds_tamper or something explodes and provides an array.
Comment #210
NWOM CreditAttribution: NWOM commented#209 works perfectly for me. Thank you!
Comment #211
elpino CreditAttribution: elpino as a volunteer commentedPatch in #209 doesn't work for mapped fields on field collection if they are multivalue.
There's an error in delta handling and cardinality.
If source contains multiple values (and these are exploded into an array via Feeds Tamper), the code cycles through them but instead of assigning multiple values for a single field, the code creates new field collections for each value and stops at the field collection's cardinality instead of the individual field's cardinality. So on save, only the last value is saved.
Comment #212
sherakama CreditAttribution: sherakama as a volunteer commentedelpino was right. The patch in 209 did not account nor handle multiple value fields.
New patch attached.
Comment #213
sherakama CreditAttribution: sherakama as a volunteer commentedNope, that last patch did not work either.
Comment #214
sherakama CreditAttribution: sherakama as a volunteer commentedOk,
New patch with (hopefully) the correct handling of multiple value field collection items.
Comment #215
moonray CreditAttribution: moonray at Chapter Three commentedThe importer seems to work well for me, aside from a single notice:
Notice: Undefined offset: 1 in field_collection_feeds_set_target() (line 1952 of /sites/all/modules/contrib/field_collection/field_collection.module).
This was tested on a multi value field in a field collection.
Comment #216
lamp5In my case not work:
Fatal error: Call to undefined function stanford_feeds_helper_field_collection_file_feeds_set_target() in sites/all/modules/field_collection/field_collection.module on line 1987
I have got multi value field collection with textfield, textarea and imagefield
Comment #217
moonray CreditAttribution: moonray at Chapter Three commentedDuring a debug session for the following issue, I ran across this particular line, which looks very much out of place:
The problem I'm trying to debug is that when you have multiple sources mapped to the same target, it overwrites the field values rather than setting it as an additional value for the multi-value field.
Comment #218
sherakama CreditAttribution: sherakama as a volunteer commented@moonray,
You are 100% correct. There was an issue with file fields and I totally forgot to translate that function to the patch. I've wrapped all of the patch functionality in to a tidy little module until it gets added to the project here: https://github.com/SU-SWS/stanford_feeds_helper.
Updated patch attached to include the correct function and name.
Thanks for the testing!
Comment #219
moonray CreditAttribution: moonray at Chapter Three commentedWhy didn't you instead of this
Do this?
Also, I noticed a stray
dpm()
in your patch.Additionally, you might want to change
while ($value[$delta]) {
to
while (isset($value[$delta])) {
to prevent a notice mentioned in above comments.
Comment #220
guptahemant CreditAttribution: guptahemant commentedhi @sherakama
while using patch #218 i am getting the following notice while trying to import the image files and hence the image is not importing.
The file top_banner_2.jpg has an invalid extension.
I have already checked the field for allowed extension and it allows for the uploading of .jpg files
please help
Comment #221
hyside CreditAttribution: hyside as a volunteer commented@guptahemant
It may be unrelated to your problem, but is the extension of the image files you are importing in uppercase? If so, it might be worth checking out this issue I ran into some time ago...
https://www.drupal.org/node/2650600
Comment #222
guptahemant CreditAttribution: guptahemant commentedHi @hyside
Thanks for your reply but that wasn't the case.
Comment #223
dnmurray CreditAttribution: dnmurray commentedPatch 196 worked for me. I had the same problem as in #204 when using the current -dev; I fixed it by hand. (I'm not sure what I'm going to do to fix my drush make, though.) The 218 patch re-introduced the issue if trying to import lists using feeds_tamper. Only the first value is getting saved.
Comment #224
riddhi.addweb CreditAttribution: riddhi.addweb commentedComment #225
yfreeman CreditAttribution: yfreeman commentedCan the patch be updated as follows, as $value[$delta] will fail if == ""
Comment #226
moonray CreditAttribution: moonray at Chapter Three commentedThat should be
+ while (!empty($value[$delta])) {
if you want to keep the equation the same.Comment #227
elpino CreditAttribution: elpino as a volunteer commentedPatch #218 worked great for me, amended with moonray's comment on #266.
Comment #228
ScorpionGhost CreditAttribution: ScorpionGhost commentedIt does not update existing records.
My modules:
Feeds 7.x-2.0-beta2
Field collection 7.x-1.0-beta11
Commerce Feeds 7.x-1.4
Drupal 7.50
Patch field_collection_feeds-1063434-218.patch
It does not update the value of field collection. Any other well-field updates
Help me please. What should be changed?
Feed informs Updated Content 1. But the value of field Collection - old
Comment #229
ScorpionGhost CreditAttribution: ScorpionGhost commentedPlease include the version FEEDS, FIELD COLLECTION and the date of issue. On which the patch has been tested. I'm sure - it will be convenient for everyone to find bugs. I'm not running the update has added value fields field collection perhaps neither one version of the module. Although I tested on a beta 11 and DIV. Picture image added, too, only the first. Patch field_collection_feeds-1063434-218.patch
Comment #230
SocialNicheGuru CreditAttribution: SocialNicheGuru commentedthe patch includes a rogue dpm call https://www.drupal.org/node/1063434#comment-11476479
Comment #231
ScorpionGhost CreditAttribution: ScorpionGhost commentedNew update field_collection 7.x-1.0-beta12
Comment #232
junaidpvI came across this thread while trying to upgrade my field_collection to latest version.
Here is the re-rolled patch from comment#218 and updated as per comment#226.
Comment #233
majorrobot CreditAttribution: majorrobot at Major Robot Interactive commentedPatch in #232 worked like a charm for me. Using most recent Feeds module version (7.x-2.0-beta3) and Field Collection 7.x-1.x-dev.
Comment #234
alisonHi everybody, I'm extremely appreciative of your work on this issue.
Question: Does this patch intend to provide the same functionality as Field collection feeds? I feel like "yes," but I also feel like, I'm really not sure...
Thank you thank you!
Comment #235
NWOM CreditAttribution: NWOM commented@alisonjo2786: Nope both have different use-cases. The Field Collection Feeds module handles importing just Field Collections (Field Collection Processor).
Whereas this patch, allows importing nodes (as an example) that have field collections attached, so that both get imported in one go.
Comment #236
alisonOhhhhh okay, that makes total sense. Thank you -- and thanks so much for the super fast reply!
Comment #237
paulihuhtiniemi CreditAttribution: paulihuhtiniemi at Wunder commentedWhile testing this, I noticed two issues:
1. we can't use !empty() in while loop as then it will skip values like zero or boolean FALSE (as already mentioned in #225, I think isset() will do)
2. issue with update when new value has less collections than existing one. Let's say node initially had 5 field collections in a field. Now if we need to update this node and we have now have only 4 collections in incoming node, it won't remove that one extra value.
Here's a patch that tries to fix both of these issues.
Comment #238
paulihuhtiniemi CreditAttribution: paulihuhtiniemi at Wunder commentedHere's also the interdiff as I didn't see a way to upload multiple files??
Comment #239
paulihuhtiniemi CreditAttribution: paulihuhtiniemi at Wunder commentedHere's an updated version of my previous patch. Better way to unset extra collections. Also at least in my setup I was getting notices about undefined language config, so there's a check for that one, too.
Comment #240
NWOM CreditAttribution: NWOM commentedSetting to Needs Review, since it was set to Needs Work after establishing that the patch did not handle multi-value fields (this was implemented starting with #214).
Comment #247
alisonI applied #232 a couple months ago, but just recently discovered that when I have my importer set up to "delete" nodes when the item no longer exists in the source feed, orphan field collection items are left behind in the field_collection_item and field_collection_item_revision tables, and the field_data_field_*/field_revision_field_* tables for the field collection fields.
In other words, this query returns tons and tons of results -- all of which are field_collection items that do not have "host nodes" b/c the "host nodes" no longer exist b/c they were deleted by Feeds at some point:
mysql> SELECT entity_type,bundle,entity_id,field_my_field_on_a_fc_value FROM field_data_my_field_on_a_fc WHERE entity_id NOT IN (SELECT field_my_fc_value FROM field_data_my_fc);
(I hope that makes sense! If not, pls lmk, I'd be happy to try to restate!)
@NWOM @paulihuhtiniemi I think I know the answer -- and I realize the latest patch isn't fully functional yet anyway, but -- have y'all already handled this issue, or not yet? (If not yet, then for now I'm just chiming in, and I'll try to dig further into the code so I can actually pitch in rather than only chime in.) Thank you!
Comment #248
NWOM CreditAttribution: NWOM commented@alisonjo2786 I just ran into that issue as well the other day. I had ran the import multiple times, and come to find out every time I deleted all of the nodes (after test imports), the field collection entries were left behind. I had to use VBO to delete all entities without a host. I was also on #232 when running the import.
Can anyone confirm that this issue still exists for #239?
Comment #249
alison@NWOM phew I'm so glad it wasn't just me :) Yep I used VBO like that, too.
I'll give it a go, but I thought you were saying that the patch at #239 didn't work on multi-value fields? Or I may have misunderstood your comment #240.
Comment #250
NWOM CreditAttribution: NWOM commentedOh, it was because I had changed the status to "Needs Review", since it was originally set to "Needs Work", due to it supposedly not being able to handle multi-value fields. Looking at the comments it appears that it was fixed starting with #214, so I changed the status.
Am eager to hear your results. Deleting 400,000 host-less field collection entities was not fun :P.
Comment #251
alison[edit] OMG never mind, I can't read, bahhh never mind never mind...
[original comment...]
@NWOM I'm sorry but I have a stupid question!! Is the idea that I'd use #239 "on top of" my already-patched field_collection code? I thought I was supposed to reverse the patch I applied before, and apply this one instead, but the one at #239 doesn't actually have
hook_feeds_processor_targets_alter()
, obviously. If I'm supposed to essentially use #239 in addition to the older patch, then I'd want to reverse #232, apply #237, and then apply #239, correct? Or?Sorry, apparently somehow I've never run into this kind of situation before and therefore don't know what the intended action/procedure is.
Comment #252
alisonUPDATE: I deleted and re-imported all my feed items, and there are NO orphan field collection items, according to my VBO and according to
mysql> SELECT entity_type,bundle,entity_id,field_my_field_on_a_fc_value FROM field_data_my_field_on_a_fc WHERE entity_id NOT IN (SELECT field_my_fc_value FROM field_data_my_fc);
Wooooot!!
The *only* thing I didn't test, and I'm not sure that I can test it because I'm not permitted to mess with the data source (it's our employee HR system haha, and they don't grant access to the sandbox so I have to go thru their devs if I want test data changes) -- the only thing I didn't test is what happens if the data source of a FC item is removed but the host entity still exists (so in my case, the entity is a "person" and the field collection is a "job/position" -- if someone gets a promotion, they no longer have the old position, but they DO have a new position, so the person entity is still there, but a field collection item was removed). It just so happens that this scenario came up earlier today, before I was testing #239, and I found out about it because the rule I created to automatically delete orphaned FC items only works when the host node is deleted, which did not occur in this situation. But I can't force a position to change so I can test :( But it's a huge organization and there are person record changes every day, I might get lucky in the next 24 hours -- if not, I'll coordinate with the dev over in HR to do some testing via their sandbox.
Either way, I'll circle back here ASAP! (And lmk if you have occasion to test the scenario I described before I'm able to!)
THANK YOU bunches :)
Comment #253
alisonHmmmm I think maybe HR isn't super active on Good Friday... I've given my dev friend who'd be the person I'd work with for a sandbox test a heads-up that I might need a bit of his time on Monday -- meanwhile, gonna give it the weekend and see if anyone's records change by Monday late morning.
I hope you have a good weekend!
Comment #254
NWOM CreditAttribution: NWOM commented@alisonjo2786: Sorry wasn't in the office since Thursday. Thankfully we got a 4 day weekend here, so didn't check drupal.org daily like I generally do. Even though you figured it out, I just wanted to say that patches always consist of all changes thus far, so you would always revert a previous patch and then apply the new one, and if someone provides an "interdiff" then it's strictly to show the changes made between two patches to help understand what was changed between the two patches. Not a dumb question at all, since each ecosystem does things differently :)
Glad that the original issue is fixed with #239. That's one more issue down.
That other situation that you detailed would definitely be good to know. I personally only used the feeds field collection import once in order to import data to migrate from a closed source CRM system into drupal. If you run into a reproducible situation and need a 2nd confirmation, I can make a clone of the system and try to reproduce it next week.
I hope you have a great weekend as well!
Comment #255
alisonHappened again! I'm happy/sad lol. I'm being picked up for lunch, but I'll report back with full details after lunch, I just couldn't wait to say something.
Comment #256
alisonOkay! So, what happened is that someone's job title and job type changed, so the field collection item with their old job title/type was removed from the node, and a new one was added, with the latest job info for that person. BUT, the old FC item is hanging around in the background.
Attached are screenshots of the old ("unconnected") and new ("connected").
So, my VBO and my aforementioned mysql query confirm that there's one orphaned FC item.
There's one other weird thing, but it's different, but i'll mention it anyway -- it's weird to me that the existing FC item didn't just get updated. The "Position ID" is "used as unique" in my feed importer configuration. So maybe "used as unique" simply isn't possible for a field collection within a node importer -- and I can live with that, b/c as long as we can fix the orphan problem, it won't matter if the position data that gets updated is actually getting deleted and added in again, we don't have any fields on the field collection that get manually edited (so nothing gets lost).
So, I feel like I'm forgetting something, is there other info I should provide, are there more details that would be helpful for troubleshooting?
[EDIT: Just to clarify, the problem occurs regardless of whether the new FC has the same Position ID as the FC item that should have been but was not deleted from the system and now exists as a sad, sad orphan.]
Comment #257
NWOM CreditAttribution: NWOM commented@alisonjo2786: Yeah that definitely sounds like an issue still. Your information will definitely be useful for anyone able to contribute to the existing patch (my PHP skills are severely lacking :P).
If you're looking for a workaround for usage in a production environment, using Rules to delete the orphan item would be possible (Rules Event: After feeds item has been imported). Am on the #drupal-support IRC channel and the drupal slack group, if you need help setting something like this up temporarily.
Comment #258
mackpipe1 CreditAttribution: mackpipe1 commentedPatch # 239 works to map the fields in the creation of the import, but when importing the data, only saves the fields of the node, which are normal, does not save the information of the fields that are inside the collection. I have the following
Title
Descrcipicion
Region (Collecction)
----------------------------
In the collection Region I have
Text Mode
Objectives Text
Text Methodology
-------------------------------
When I import CVS, it only stores my data
Title
Descrcipicion
But the fields that are inside the collection are empty
I need to implement some hook to store ?, I have seen in several patches that have a hook_presave, this will be the hook that I need ?.
Thank you.
Comment #259
userspan CreditAttribution: userspan as a volunteer commentedPatch # 239 works to map the fields in the creation of the import, but when importing the data, only saves the fields of the node, which are normal, does not save the information of the fields that are inside the collection.
Reviewing information from tables.
Patch # 239 if you are saving the information but not performing the relationship with the node within the main field field_collection_name
Example
Field_collection_name (Does not save relation to fields)
-field_titulo_collec_producto (title field, the information is saved correctly)
-field_desc_collec_producto (field description, information is saved correctly)
Some form or patch to finish performing the relation with the parent of the field that is added in the node.
Comment #260
majorrobot CreditAttribution: majorrobot at Major Robot Interactive commentedI'm not sure if this is what's causing the issue mentioned in #258 and #259, but I have a found a bug in #239 that occurs when you are importing a node with multiple instances of a field collection and not all fields in your field collection are included in your feed source. The result is that the first instance of the field collection saves fine, but successive ones leave out fields.
Say I have a json feed like this:
Each item in "people" should map to a "person" field collection with fields for name, phone, and bio. This works beautifully if the data is complete, like the above. But, if your data is not as consistent, like the following, you run into issues:
The first person field collection saves fine, but the other 2 save without data in the "name" field. They do save with data in the last field ("bio," in this case), though.
I've spent several hours debugging the patch to figure out what's going haywire, and don't have anything yet. I suspect it has something to do with $delta in the while() statement. I also wonder if the $entity->{$target} is getting reset somehow instead of accumulating data as Feeds steps through the field collection's fields.
Comment #261
majorrobot CreditAttribution: majorrobot at Major Robot Interactive commentedI've found the cause of the bug I found above, and it looks like it's the code that removes extra field collections when you're importing:
This code does remove extra collections, but unfortunately also has the side effect of losing imported data if the feed source has a different number of values in one field than it has in another.
For example, in my second json example above, there should be 3 values for my person field collection (since there are 3 people records). However, since the second and third person do not have a phone number in the data, when iterating through the phone number field, the code finds count($value) not to be equal to count($field['und']). There is only one phone number value, but 3 $field['und'] values. Thus, the code slices out the second and third person field collections and leaves me w/ only $field['und'][0].
So, I'm attaching a patch where these lines have been removed. But, we still need a way to resolve the problem of removing extra field collections. The issue is finding a way to identify which collection should be removed when the number of collections don't match between the source and the current entity. It's easy enough to count them and compare the count, but without a unique field on each field collection, how do you know which to zap? I've thought about using something like the hash method that Feeds module uses for other entities, but I foresee that being a good bit of work that might include adding a database table. I've started some work in a hook_feeds_presave() function.
@paulihuhtiniemi, you kindly provided the code above ... any ideas for how to implement this another way?
Comment #262
majorrobot CreditAttribution: majorrobot at Major Robot Interactive commentedThere was a problem with the way I created my last patch. Oof, it's been a while. Replacement here.
Comment #263
alison@majorrobot are you in Baltimore? I'm hangin at the "Feeds" sprint table, even though I'm not acutally sprinting on feeds, b/c i'm thinking they might be able to help me if I need help :) :)
Re: this issue, I may be lost about what you're saying, but, when I used #239 I was able to import items that create nodes and have values map to field collection fields just fine -- and I could even have them update existing field items just fine. The *only* lingering problem for me after applying #239 was what I described in #256 re: "ghost" FC items hanging around in the background instead of being removed completely, when feed item nodes are deleted because they no longer exist in the source feed OR when the source data for the field collection items are no longer on the host record in the source feed (i.e. for me the feed items are person records, and the FC items are the jobs held by those persons -- so if someone's promoted, their old job is no longer on their record and the new job is -- but a "ghost" of the old job is still hanging around in the site database). (See also -- my explanations in #252 and #256.)
OH and also I guess, it's weird that the FC items aren't getting 'updated and instead they're being removed/replaced, even though the position ID is unique and stuff, and i have it all configured to use that. But I don't reeeeeally care about that, I only care that I don't continue to have ghost items :) :)
So like I said, I am going to be trying to work on this more til sprint time is over! If you're here and not busy, c'mon over to the Feeds table!
Comment #264
majorrobot CreditAttribution: majorrobot at Major Robot Interactive commented@alisonjo2786, I'm not in Balitmore (unless you count Detroit a remote suburb), but wish I were that close to the action!
So, my patch above (#262), actually eliminates the code that deals with the "old" field collections. I felt it important to do this since it was causing further breakages and, as you noticed, not really deleting the field collections anyway — just orphaning them.
There's gotta be a better way to do this. Maybe we can brainstorm something.
I see your note about using the unique id, and that's interesting. I've spent a lot of quality time in the code for this, and there's no application of the unique id there ... but maybe that's something we can do. I like that better than creating a hash of the whole FC and then comparing that to new data.
I can spend a little time working on the unique id idea today.
Any other ideas?
Comment #265
alisonAck! Bummer.
So actually, and I see now that I said it wrong in my last comment -- the thing I described was the case with #232, but since applying #239, FC items on **imported nodes that get deleted b/c they no longer exist in the source feed** ARE properly and cleanly removed along with their host nodes. FC items (that are on imported nodes that are not deleted) that SHOULD be deleted because the incoming feed item has a change that means the source data for that FC item is no longer there (like the "promotion" example I mentioned), THOSE field collection items become orphans/ghosts, haunting my DB and my dreams :) (In the admin UI, they get the label "Unconnected field collection item.")
Point being, #239 was helpful for me, just didn't go "all the way."
Could you maybe clarify/restate the problem you were having that you aimed to solve with the newer patch? Data gets mapped to FC fields just fine for me, but I'm not sure if I'm totally getting what you're describing.
Comment #266
majorrobot CreditAttribution: majorrobot at Major Robot Interactive commentedSo, the issue with the code in #239 is that if you have a feed where some field collections don't include every field, you end up with corrupted data. This has to do with the way that the code compares the number of field collections and the number of fields included in the feed, which, as it turns out, can be apples and oranges.
In my example above, some people (which are mapped to field collections) did not include phone numbers, but some did. The code counted the number of phone numbers in the feed and suddenly assumed that was how many field collections I had. So it dumped some of the field collections it was importing and generally mucked things up.
Does that help?
Comment #267
alisonOh! Okay I understand what you're saying now, yes that helps a lot, thanks. I haven't observed that in my usage, but I'll do some more testing to see if I can reproduce.
Y'know how if you want to be able to have feed items get updated when the feed re-imports, you have to designate one of the node fields to be "used as unique" (in your mapping settings) -- and in order to be able to do that, you usually have to use the field validation module so you can add a validation rule to that node field have to have unique values...? So, I have that set up for the ctype that my feed items become nodes of, so to speak (my "person" ctype has a field with a validation rule set up, and then I have it "used as unique" in my feed importer mapping settings -- it's a unique employee ID text field), AND I have a unique field set up for my "jobs/positions held by the person" field collection (it's a unique "position ID" text field). I'm saying all of this just to explain the unique ID thing I mentioned earlier that you sounded curious about -- hopefully my explanation makes some sense :) -- but also like, in case you think it might explain whey the data mucking isn't happening to me (like maybe b/c there's a unique ID field on my FC config, it's using that for knowing how many FC items there should be on each node...?). Totally speculating, but, there it is :)
Comment #268
alisonOkay and now I've re-read the interdiff for your patch on #262, and I get it a lot better now.
I had been focusing on what was added in #239 around line 2100 -- the lines you removed -- thinking that maybe I could add something that would actually delete those FC items. But I see what you're doing, so I'm going stare at that some more for the next 30ish minutes -- then they're kicking us out of the convention center, and like, I suppose I'll have to start driving back to Ithaca haha.
I'll let you know how it goes and/or if I think of anything else!
Comment #269
alison@majorrobot are you on Drupal slack?
- - - - - - - - - -
The first thing that's sticking out for me is that it _looks_ like $old_entities is actually the node, not the field collection field items array? But also, I'm feeling like I don't know what you're aiming at with the array_shift, even if $old_entities were the FC field items array. But I'm still staring/pondering.
Comment #270
majorrobot CreditAttribution: majorrobot at Major Robot Interactive commentedI see what you're saying about the unique field (I didn't realize you could do that with the Field Validation module. I've been writing my own hooks this whole time...). Just added back in the FC-removal lines of code with my same feed (some people missing phone numbers), and I get the same mucked up results, unfortunately. So, looks like the uniqueness doesn't apply at the FC level, only at the host entity level.
My current idea: Use hook_feeds_processor_targets() or hook_feeds_processor_targets_alter() and add a form_callback to field collection targets to generate a different checkbox in the mapping form that says "used as unique for field collection" or some such.
Then, later in the set_target callback or in the presave hook, this uniqueness is checked. If a field collection exists that does not have a unique value that exists in the feed, delete the field collection. If a field collections exists that does have the unique value, update that field collection instead of creating a new one, if possible.
Comment #271
alisonI think that what you're describing makes sense -- it feels like what I was trying to conceptualize before, with minimal success. It definitely sounds like the right track to me.
Have you looked at how Feeds uses the "use as unique" functionality when it looks for and then updates/deletes existing items? (I haven't looked into that yet.)
Aaaaaand, very sadly, I gotta get on the road :( :( :(
I'll check in again over the weekend or Monday, depending on how "post-DrupalCon-catchup" goes, in the office and at home haha.
Thanks so much for your work on this! I hope I can help more soon.
Comment #272
alisonOMG i totally just reproduced data mucking........ yikes!!!!!!!!!!!!!!! and now i'm getting on the road for real haha. bye for now.
Comment #273
majorrobot CreditAttribution: majorrobot at Major Robot Interactive commentedAll right, here is a new patch. It solves the issue of deleting field collection items that are no longer in the feed. I actually didn't rely on uniqueness at all, but just wiped out the old items and saved the new ones. The simplicity of this approach outweighs any downsides to me.
In the process, I discovered a few more issues that apparently already existed:
So, one step forward, and more steps discovered.
Comment #274
alisonVery cool, @majorrobot. The only thing I'm not clear on is #2 -- could you clarify/restate the situation you're talking about? Thank you!
Comment #275
majorrobot CreditAttribution: majorrobot at Major Robot Interactive commentedHi @alisonjo2786, #2 is just basically #1, but when a node is being updated rather than inserted. The code still (frustratingly) has a hard time when a feed item is missing a field.
I've wrestled with this for quite a while, and I've discovered that these issues (#1 and #2) are actually dependent on the parser. By the time the data is parsed and gets passed to our hooks, (at least in my case) it's in a structure that cannot possibly be saved into the correct field collection items, except by coincidence. In my case, I'm using Feeds JSONPath Parser, but other parsers may have the same issue (and my guess is that some do).
To explain: the JSONPath parser steps through the different subfields you've mapped and creates a simple array with the values that exist in the feed regardless of their position in the feed. The data does not maintain an association with *which* item in the feed that it comes from. So, take this example, where people will be mapped to field collection items:
The second people item is missing a phone number. JSONPath Parser creates an array of values for each field, and the array for phone number ends up like this:
Our field collection hooks read in that array and applies the first phone value to the first field collection item it saves, and the second phone value to the second item. Thus, you get a saved field collection where Bo from Bombay has a phone number of 333-3333. Corrupt data.
So, given all this, it's out of scope (and the realm of possibility) for the field collection feeds code to fix this. It has to be fixed at the parser level. Fortunately, JSONPath Parser has a hook that will let me fix the values specific to my project (I can share this code if it would be helpful).
If you're using the CSV parser, an XML parser, etc., you may or may not have this same problem. Not sure. But it has to be corrected at the parser level.
Comment #276
majorrobot CreditAttribution: majorrobot at Major Robot Interactive commentedI've continued working on this patch in an attempt to get nested Field Collections to import. I've made tiny, incremental progress, but the recursion of the code in field_collections_feeds_set_target() is driving me bonkers. Ideally the function would work for both parent field collection items and child field_collection_items.
Has anyone else attempted this? Or have any ideas?
Comment #277
atiras CreditAttribution: atiras commentedThe patch as #273 works for me -- sort of. I can confirm HTML XPATH Parser has the same issue as described in https://www.drupal.org/node/1063434#comment-12066933
I'm not a coder so can't help update the patch but will gladly test new iterations.
Comment #278
atiras CreditAttribution: atiras commentedUnfortunately I've had to back out the patch at #273 -- it breaks the importation of images with Fields (multi-value image field, image file name and image title parsed/tampered and exploded into the relevant arrays.) The file records are created but without titles and not linked to the right node.
Backing out the patch restores the correct functionality.
I've verified that the problem isn't caused by using the latest dev version of Fields Collection.
Comment #279
atiras CreditAttribution: atiras commentedTo elaborate on #278, I import a set of image files (URI plus title) into a node type; the URI and Title are parsed out of an HTML stream and then (after a few other Feeds Tampers) are exploded into a multivalue image field in each node.
Without patch from #273, this works fine. With the patch, it breaks -- the image records are created (without a title but with the correct image) but they aren't linked to the relevant node.
There are no field collections on the node type, so I'm not attempting to import any.
I'd wor backward through the patchers to try and find one that works for my use case, but it's impossible AFAIK to tell which patch was rolled against which verision of Feeds/Field Collection -- I'm running Feeds 7.x-2.0-beta3 and Field Collection 7.x-1.0-beta12 (but can update that one to dev if necessary -- could probably also update Feeds to dev.
Can anyone suggest which patch to try -- I need to import mutivalued field collections into a node, but each field collection doesn't contain multivalued items. #262 looked like a good possibility but it fails to apply.
Comment #280
atiras CreditAttribution: atiras commentedOn initial testing the patch in #262 (applied manually to 7.x-1.0-beta12) *doesn't break* the image importing -- I'll keep testing.
Comment #281
atiras CreditAttribution: atiras commented#262 is working pretty well as long as I ensure that all fields in the collection are populated.
Comment #282
m.stentaComment #175 accidentally cleared the original issue description (https://www.drupal.org/node/1063434/revisions/view/8887965/8894301), and then comment #200 set it to the same value as the comment (https://www.drupal.org/node/1063434/revisions/view/9504473/9535453). Restoring the description to what it was before #175...
Comment #283
m.stentaThanks @majorrobot for the patches - I tested #273 and it does work to create mapping targets for my field collection fields - and the import works.
Notably, I am using a CSV parser, and I do not have multi-value field collections, so I don't think I'm susceptible to the bug @majorrobot mentioned in #273.1.
However, the patch seems to have broken my entityreference mappings. I will do some debugging and see if I can figure out why... it may be related to what @atiras mentioned in #278/#279.
Comment #284
m.stentaOk, so the reason that entityreference mappings are broken with patch #273 (and probably image mappings as @atiras described) is because of a bad assumption in
field_collection_feeds_presave()
:That last
if
condition is making the assumption that any mapping target that contains a colon (:) is a field collection. Entity Reference fields also use colons, and so do image fields. A lot of others probably do as well.So we'll have to find a better way to check if the target is a field collection in that function...
Comment #285
m.stentaHmm also: this part is not working as intended...
On new entities,
$entity_id
is0
, which is not null in PHP, so the code in that block is always running - even on new entities.Should be this instead:
Comment #286
m.stentaAttached is a patch that fixes the two issues I described in my comments #284 and #285.
@atiras - Can you test this to see if it resolves your image import issue?
To check if the mapping is a field_collection field, I load the field info using
field_info_field()
and check the type.I'm leaving this as "Needs Work" because it sounds like there are other issues that need to be fixed.
Comment #287
atiras CreditAttribution: atiras commented@m.stenta -- sorry, been away for a while. Will test the new patch today.
Comment #288
atiras CreditAttribution: atiras commentedOK, patch in #286 works to link the imported images. Thanks.
Should it fix the import to field collections when not all fields are populated (I'm thinking not, but haven't tested it yet and won't if I'll be wasting my time).
Comment #289
m.stentaI'm not sure I know what you're referring to. Can you elaborate a bit?
Comment #290
giupenni CreditAttribution: giupenni commented#286 seems works for me
Comment #291
atiras CreditAttribution: atiras commentedSorry, I was thinking about another attempted solution to this issue, where you had to ensure that all fields in your field collection were populated to get the import to work correctly -- if you had an unpopulated field, the data got out of step during the import.
This seems to work fine.
Comment #292
m.stentaAnother issue that I'm experiencing is that field validation is not being run for fields within a field collection. Field validation was added to Feeds in the dev branch via #2379631: field_attach_validate() must be called before programmatic entity saves.
I have that patch applied to my Feeds module, along with the patch from #286 above for Field Collection. Field validation appears to be happening for fields that are attached directly to the entities I'm importing, but it does not work for fields attached to my field collection.
I have two "List (text)" fields (using the "Select list" widget), each with a set of available values to choose from. If I try to import an entity with an invalid value, I get a validation error for the field that is directly attached to the entity, but not for the field attached to the field collection.
Comment #293
SEVENTH7 CreditAttribution: SEVENTH7 as a volunteer commentedAdded patch but can't see new field in mapping? What is the extra functionality of this patch? We need to connect the field collections true the feed importer mapping system.
Comment #294
Siavash CreditAttribution: Siavash at OHO Interactive for Dartmouth commented#286 worked great in my case. Thanks!
Comment #295
rcodina CreditAttribution: rcodina commentedPatch on #286 works for me too. Many thanks! Maybe we can handle field collection field validation on a separate issue (see comment #292).
Comment #298
elpino CreditAttribution: elpino as a volunteer commented#286 worked for me too.
This should be ported.
Comment #299
SkyBuzz CreditAttribution: SkyBuzz commented#286 work for too with 7.x-1.0-beta12
Comment #300
sportch CreditAttribution: sportch commentedAny chance of an update for 7.x-1.0-beta13 ?
Comment #302
ram4nd CreditAttribution: ram4nd as a volunteer commentedComment #303
philltran CreditAttribution: philltran commentedHere is a patch for 7.x-1.0-beta13.
EDIT Looks like this has been committed to DEV while I was making the patch, so it probably will be in beta 14.
Comment #305
alison#303 works great for us on beta13!
@philltran are you saying that #261 was committed to dev, so we should use that?EDIT: never mind, now i see that #261 doesn't apply to beta 13 -- which I'm sure is why @philltran created the patch at #303 :)EDIT 2: Just, never mind...... (#303 does totally work on beta13, but it doesn't matter anymore :) sorry!)
Comment #306
robdubparker CreditAttribution: robdubparker commentedNoticed that boolean fields still don't work correctly with field collections and imports. I've attached a patch that seems to help fix the issue.
Comment #307
robdubparker CreditAttribution: robdubparker commentedrerolled patch for 7.x-1.0-beta13
Comment #308
alrueden CreditAttribution: alrueden commentedThis update still has a major problem with recursion. We don't even use Field Collection together with Feeds. Nevertheless, this caused a total failure for all the feed importers on our site.
After updating to the latest dev version (7.x-1.0-beta13+48-dev), our feeds stopped importing on cron. Attempting to edit the feed nodes resulted in a WSOD.
Server logs when attempting to edit a feed node or run the feeds importer on cron gave me this:
According to Acquia, this error is generally caused by an infinite recursion bug.
I removed the chunk of code relating to feeds integration (everything in commit ead602643387c4bca10d82fac1dfc17d425a241d) and now everything works again.