I'm working on a project that will require a solution like field_collection aims to solve.

My project needs a node container that holds statistical data collections for several years.
node: title, array (year, text, number, number), taxonomy

After assessing the different options available to me, I've come to three possible solutions:

1. Custom module, make a cck sub module under d6 or d7 that gives me the fields for the array.
2. Use cck-3.x and the multigroup under d6.
3. field_collection on d7

Now, I think solution 3 is the best solution in the long term. However, it seems to me it is quite a while until this solution is ready to use. The greatest obstacle to me at the moment is the possibility to import large datasets. This leads to the following questions:

Is there planned a way to import datasets through for example feeds?
Can someone point me in the direction of how to manually import datasets into mysql?

CommentFileSizeAuthor
#307 1063434-7.x-1.0-beta13-feeds-boolean-fix.patch9.93 KBrobdubparker
#306 1063434-feeds-boolean-fix.patch1.07 KBrobdubparker
#303 add_feeds_integration-1063434-303-beta13.patch10.63 KBphilltran
#286 add_feeds_integration-1063434-286.patch9.59 KBm.stenta
#273 add_feeds_integration-1063434-273.patch9.54 KBmajorrobot
#262 interdiff-1063434-239-262.txt2.05 KBmajorrobot
#262 add_feeds_integration-1063434-262.patch9.02 KBmajorrobot
#261 interdiff-1063434-239-261.txt2.19 KBmajorrobot
#261 add_feeds_integration-1063434-261.patch9.16 KBmajorrobot
#256 person-fc-connected.20170418.png8.7 KBalison
#256 person-fc-unconnected.20170418.png11.1 KBalison
#239 interdiff-1063434-232-239.txt1.41 KBpaulihuhtiniemi
#239 add_feeds_integration-1063434-239.patch7.9 KBpaulihuhtiniemi
#238 interdiff-1063434-232-237.txt986 bytespaulihuhtiniemi
#237 add_feeds_integration-1063434-237.patch7.92 KBpaulihuhtiniemi
#232 add_feeds_integration-1063434-232.patch7.62 KBjunaidpv
#218 field_collection_feeds-1063434-218.patch8.08 KBsherakama
#214 field_collection_feeds-1063434-214.patch5.89 KBsherakama
#212 field_collection_feeds-1063434-212.patch5.83 KBsherakama
#209 field_collection_feeds-1063434-209.patch5.32 KBsherakama
#207 field_collection_feeds-1063434-207.patch4.9 KBsherakama
#196 add_feeds_integration-1063434-196.patch5.23 KBdpfitzsi
#190 add_feeds_integration-1063434.patch4.88 KBiajay
#184 add_feeds_integration-1063434-184.patch4.49 KBNovitsh
#177 field_collection-feeds-1063434-177-7.patch4.59 KBNovitsh
#173 field_collection-feeds-1063434-173-7.patch4.78 KBveronicaSeveryn
#167 field_collection-feeds-1063434-167-7.patch4.32 KBveronicaSeveryn
#153 field_collection-feeds-1063434-153.patch9.57 KBGaëlG
#152 interdiff-121-152.txt1.69 KBbbdata
#152 field_collection-feeds-1063434-152.patch6.29 KBbbdata
#143 field_collection_1063434_143.patch8.8 KBk_zoltan
#130 field_collection-feeds-1063434-130.patch8.76 KBlachezar.valchev
#123 explode_twice.zip1.26 KBjrao
#121 field_collection-feeds-1063434-121.patch5.8 KBqueenvictoria
#121 interdiff-1063434-119-121.txt1.87 KBqueenvictoria
#119 field_collection-feeds-1063434-119.patch5.32 KBhumansky
#109 field_collection-feeds-1063434-109.patch5.45 KBarh1
#102 field_collection-feeds-1063434-102.patch5.13 KBxcf33
#96 field_collection-feeds_integration-new.patch11.96 KBpraandy
#93 field_collection-feeds-1063434-93.patch5.96 KBAaronBauman
#91 field_collection_feeds_multiple.patch4.51 KBxcf33
#89 field_collection-feeds_integration-1063434-89.patch4.51 KBJohn Bickar
#83 field_collection-feeds_integration-1063434-83.patch4.51 KBbtopro
#82 field_collection-feeds_integration-1063434-82.patch4.61 KBDavid_Rothstein
#82 interdiff-16-82.txt2.3 KBDavid_Rothstein
#70 field_collection-feeds_integration-1063434-70.patch3.19 KBbtopro
#69 0001-Do-not-save-empty-items-especially-useful-for-Feeds-.patch2.67 KBGaëlG
#41 field_collection-feeds_integration-1063434-41.patch3.14 KBarvinsingla
#40 field_collection-feeds_integration-1063434-40.patch3.14 KBarvinsingla
#36 explode_twice.zip1.23 KBXearwe
#28 field_collection-feeds_integration.patch3.19 KBsir_squall
#16 field_collection-feeds_integration.patch3.13 KBfranz.glauber
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

casey’s picture

Feeds integration seems a good idea.

casey’s picture

Component: Miscellaneous » Code
Category: support » task

Let's make it a task.

bowersox’s picture

+1. We use the Feeds module and it would be great if Feeds would import data into Field Collections.

inversed’s picture

subscribe

Gemma Morton’s picture

We are using Field Collections to have Multiple values on a node of:
Question & Rating.

We provide a form via custom code, and process the submit. You could do something similar with a custom module to import the data from a CSV.

This was found somewhere in the issue queue, but I will repost how we used the code to save field_collections:

// Set up your Node
 global $user;
  $new_node = new stdClass();
  $new_node->type = 'evaluation';
  $new_node->title = '';
  $new_node->nid = '';
  $new_node->is_new = TRUE;
  $new_node->uid = $user->uid;
  $new_node->created = strtotime("now");
  $new_node->changed = strtotime("now");
  $new_node->status = 1;
  $new_node->comment = 0;
  $new_node->promote = 0;
  $new_node->moderate = 0;
  $new_node->sticky = 0;
  $new_node->language = LANGUAGE_NONE;
  // Save your new node
  node_object_prepare($new_node);
  node_save($new_node);
  unset($new_node->is_new);
  
  // $node->nid must be valid at this point
  // Questions and ratings - field_question_ratings_col
  $delta = 0;
  foreach ($input['questions'] as $qid => $rating) {
    $field_collection_item = entity_create('field_collection_item', array('field_name' => 'field_question_ratings_col'));
    $field_collection_item->setHostEntity('node', $new_node);
    $field_collection_item->field_question_ref[LANGUAGE_NONE][$delta]['nid'] = $qid;
    $field_collection_item->field_question_rating[LANGUAGE_NONE][$delta]['rating'] = $rating['rating'][0];

    $field_collection_item->save();
    $new_node->field_question_ratings_col[LANGUAGE_NONE][$delta]['value'] = $field_collection_item->item_id;
    $delta ++;
  }
  // Save the New Node again, this time with Field Collections attached.
  node_object_prepare($new_node);
  node_save($new_node);
  return $new_node;
}
fago’s picture

Title: Import large datasets » add feeds integration

I think the right approach for feeds integration is #1033202: [Meta] Generic entity processor. That generic processor should basically work for field-collection too - if it needs some adaptions we can sub-class the generic solution to add them.

hellomobe’s picture

Checking on the status. It wasn't clear to me if there was a solution in the generic entity thread. I'm using QueryPath as the parser, Node Processor. Fago, thank you for all your hard work and contributions to Drupal.

elly’s picture

(subscribin' here and on the generic entity mapper thread too)

hellomobe’s picture

http://drupal.org/node/1033202#comment-4721020 - Posting here too since it may have more to do with field-collection. The rules entity processor admin ui seems to work okay. I have two field-collections fields created.

If someone understands how the entity processor should or will work, can you give me a brief summary. Looking at it more closely (I'm not a programmer and I'm muddling through to grasp the concept as best I can), it seems that it would just create a field-collection entity, not a node that has a related field-collection. Yes, no? I think most people need it to create the node + field-collection (using this as the next phase to multi-groups). Just want to make sure I'm trying to hack the right approach.

Bevan’s picture

It would be great to get some pointers on how to move this issue forward. I may need it soon and can probably work on this, though it would take me a while to understand all the moving parts here and I would probably do it the wrong way the first time.

hellomobe’s picture

Just putting a thought out there since this is really over my head. Could we model the feed integration as a mapper like Location Mapper? Is location an "entity".

Would a sponsor pool help to get this going (some solution whatever that path takes)? I'm willing to chip in.

tim.plunkett’s picture

Status: Active » Postponed
franz’s picture

I don't get why this is so attached to the issue on #12. Using this new generic processor, I'd have to create another importer for every field collection I have in my node type.

I have a complex node type with several field collections, and I want to import at once. Would be a lot easier to have a mapper "Field Collection 1: Field 1", etc...

Did I get wrong?

hellomobe’s picture

+1 for going this direction. I also noted this in #11 referencing the location mapper. It's kind of the same concept. I agree, as I understand it, that the generic processor just does one entity at a time, versus a field-collection + node at the same time, which is the most likely use case of a field-collection entity. With the generic entity, how would the field-collection know which nid to correspond to if it is imported separately (at a different feed process) than the node?

Should we change the status to active to get help with a mapper? I've tried creating one myself, but no success.

franz.glauber’s picture

I did create one, but it have some issues, it doesn't work with field collections inside field collections, because it is impossible to determine the target field collection.

franz.glauber’s picture

Here it goes, can be useful as a starting point.

franz’s picture

damn, wrong account =/

philipz’s picture

Patch #16 shows the field-collection fields as targets for import. I'm looking through code but can't see anything about multiple value field-collection fields. Is this supported in your patch franz? I mean let's assume csv file import and then we're having columns for each field-collection fields with somehow (semicolon) separated values for each column multi value?
I'm going to try that and maybe modify it if it doesn't work.

franz’s picture

#18 It is supposed to work, yes.

However, if you have a multiple-value field inside a multiple-value field-collection, it is impossible for feeds to determine which value belongs to which field-collection.

philipz’s picture

Ok so just to be sure let's say I have two fields in my field-collection ("fruit" and "color"). The field-collection field is unlimited.
Now to import two values into this field I have cvs file looking like this:

"fruit","color"
"apple;banana","red;yellow"

I map fruit source column to target fruit field and do the same for color.

This shuld give me two field-collection values:
1. [ apple ] [ red ]
2. [ banana ] [ yellow ]

The point is how to separate the values in single csv cell?

franz’s picture

I see now, philipz. I'm not sure how is the best way of dealing with multiple values on CSVs (I'm working with XML). Maybe you can use http://drupal.org/project/feeds_tamper to explode the values so they are passed to targets as arrays. You map the fruit to field_collection:fruit and color to field_collection:color and they should work fine.

philipz’s picture

Thanks for this idea - I'm using Feeds Tamper already in other import so I'll give it a try. If all I have to do is explode this into array then this should be easy :)
I'm working with XML files as well so I understand that you're using XML with XPath XML parser for your imports and it's working?

franz’s picture

Yes, works like a charm. =)

zorax’s picture

works like a charm with csv parser and taxonomy list. ;)
thanks

dgastudio’s picture

so, the patch is functional? on import feeds, i only have to use feeds tamper to split array?

franz’s picture

kervi, the explode is only needed if you have multiple values per field like #20

Jason Dean’s picture

#16 working great for me :)

sir_squall’s picture

#19 I have tested the patch but he dident work for the "unlimited value",

A have done a new patch, for fixing that.

etiennechataignier’s picture

Tried the patch in #28.

I installed the feeds tamper module and used it to explode multiple values of a csv.

However, only the first element is inserted.

My CSV looks like :

title;body;guid;category;language;finition;ref;price_euro;price_dollar;price_sterling
Product 1;Product 1;REF11;7;en;2,3;REF1B,REF1N;10,11;20,21;30,31

On this example, the field collection {finition:2, ref:REF1B, price_euro:10, price_dollar:20, price_sterling:30} is inserted.
The field collection {finition:3, ref:REF1N, price_euro:11, price_dollar:21, price_sterling:31} was not.

[EDIT]

Tried with Feeds XPath Parser (without feeds tamper) and got the same result.
Here is the XML :

<?xml version="1.0" encoding="UTF-8"?>
<products>
    <item>
        <title>Produit import XML</title>
        <body>Produit import XML</body>
        <guid>REF11</guid>
        <categorie>7</categorie>
        <langue>fr</langue>
        <finition>
            <id_finition>2</id_finition>
            <reference>REF1B</reference>
            <prix_euro>10</prix_euro>
            <prix_dollar>20</prix_dollar>
            <prix_sterling>30</prix_sterling>
        </finition>
        <finition>
            <id_finition>3</id_finition>
            <reference>REF1F</reference>
            <prix_euro>11</prix_euro>
            <prix_dollar>21</prix_dollar>
            <prix_sterling>31</prix_sterling>
        </finition>
    </item>
</products>

Context : //item
Retrieval of id_finition : finition/id_finition

Only the first finition is inserted.
I'm not an expert in Xpath, so maybe I missed something.

If anyone has a solution, it would be greatly appreciated.

franz’s picture

Have you tried the patch on #16 too? AFAIR I used it for "unlimited" values without issues.

etiennechataignier’s picture

Thanks a lot !
Patch in #16 works perfectly, included with unlimited field collections.

Patch in #28 produces result provided in #29.

rsgracey’s picture

We applied #16, and nothing changed--still no field collection targets in the importer. When we applied the .dev version of field collection module, we got a white screen. FWIW.

franz’s picture

rsgracey, can you post versions of drupal and feeds ? Maybe that could help.

Also, try clearing caches as this is a new hook implementation.

GaëlG’s picture

#16 worked for me with Field collection 7.x-1.0-beta3 and Feeds 7.x-2.0-alpha4+40-dev.

arkjoseph’s picture

This patch is a god send. Is there any chance #16 can be applied to 6.x ? Thanks for the awesome support.

For completeness, the attached is a sample of the csv I am attempting to import with field collections.

http://drupal.org/files/Screen%20Shot%202012-04-06%20at%2012.05.15%20PM.png

Many thanks.

Xearwe’s picture

FileSize
1.23 KB

Hi,

I was in front of the same problem as many of you lately: I wanted to import large amount of data, using feeds, into nodes having a "field_collection" field. This "field_collection" field is made of two fields: format, which is a unique "Term reference" field, and "Languages", which is an infinite "Term reference" field.

My goal was to set up a database of films, each film having several "physical support", and each support having a "format" and several "languages". (for instance the movie "Avatar" can be available on two DVDs: first one will have languages English and French, and second one Egyptian and Spanish)

My CSV file has the following format:

title;support_format;support_languages
AVATAR;DVD|DVD;English,French|Egyptian,Spanish

Here is how I solved this:

  • I used
    • Feeds
    • Feeds Tamper
    • Field_collection
  • I created a new plugin to Feeds Tamper by copying plugin "Explode": I named it "Explode Twice" since its purpose is to explode twice the value of the field "support_languages" (You'll find this plugin attached, it must be added to directory sites/all/modules/feeds_tamper/plugins)
    This plugin is really a "quick and dirty made" one, please review it if you plan to use it on a production environment.
  • I used patch provided in comment #16 to make fields of my "field_collection" field appear in the "targets" list
  • I configured mapping in my Feeds importer to use "Explode Twice" for "support_languages" field, first separator being "|" and second one ","

==> Everything went well and my films are now created with correct "supports"

Thank you,
Alex

emptyvoid’s picture

I applied the patch from comment 16 and setup a multi-field entity with unlimited nodes on a parent content type.

It worked just fine right out of the box with no tweaks for fixes.

I'm not sure what the patch in 19 does to "fix" it but I found it the patch in 16 to work as advertised.

presleyd’s picture

I'm not finding the same success. I'm using a Field Collection made up of two fields: an entity reference and an integer. Both show up fine in the mapping UI but running the importer does not seem to bring either in. I wondered if the entity reference was causing the problem so I removed it from the import but to no avail. A infinite field collection made up of a single value integer didn't import either. I'm using Field Collection 7.x-1.0-beta4 and Feeds 7.x-2.0-alpha4+48-dev

Edit: My integer field had decimals in it and the entity reference doesn't work. Using a node reference from References and a text field it worked fine. In short I think this is working properly.

Shiraz Dindar’s picture

success with #16. thanks!

arvinsingla’s picture

The patch in #16 has been working well for us save one issue. We are trying to fill in field collections where the data contains 0's. Currently the code does the following:

    if (empty($v)) {
      // Avoid creation of empty field collections
      continue;
    }

This is causing any 0's from being skipped. As a trade off I have updated the patch to include the following.

    if (empty($v) && $v !== 0) {
      // Avoid creation of empty field collections
      continue;
    }
arvinsingla’s picture

Uhg... Ignore the patch above I actually had to make it $v != 0 instead of $v !== 0

johnv’s picture

@arvinsingla,
try this:

-if (empty($v)) {
+if (!isset($v)) {

see also http://zainal.wordpress.com/2006/04/25/8/

franz’s picture

timfernihough’s picture

Hi franz.glauber,

I am working with arvin.singla on the same issue and our particular implementation requires the ability for us to map to a field collection nested inside a field collection. In your patch, for field_collection_feeds_processor_targets_alter, you immediately begin by ensuring that the nested entity is not of type field collection which of course is the reason why we cannot map to a nested field collection in feeds. Is there a reason behind why you did this? We have temporarily commented out this check on our local and we can then see the ability to map to these fields but we are concerned you did this for a reason.

Can you please share why you put in that check? Thanks so much for your time.

franz’s picture

hi timfernihough,
AFAIR, this doesn't work well with multiple items (a multiple field_collection inside a multiple field_collection), beacuse Feeds cannot discern which piece of content belongs to which. If that's not your case, it should work ok too.

timfernihough’s picture

Thanks franz! After some debugging we were able to determine that is exactly what is happening.

For now, we've been able to settle by writing a hook to manipulate our incoming data feed on the fly and to avoid the need for feeds to be able to support nested data collections. If time permits, we may see if we can help by writing an update to the patch to help feeds be able to discern which content belongs to which field collection.

yurgon’s picture

#16 Dont work with term link field ;(
any ideas?

franz’s picture

@yurgon, what do you mean with don't work? Doesn't show up as a target or shows up and doesn't set a value?

GaëlG’s picture

#42 is indeed needed when empty values make sense, for example with csv files like:
organization,contactsname,contactsmail
"Drupal","John|Jack|Joe","john@drupal.test||joe@drupal.test"

emilyf’s picture

#16 is working for me, great work!

Norberto Ostallo’s picture

Status: Postponed » Needs review

#16 is working fine for me too.
Why is this issue postponed? I don't think we need to wait for #1033202: [Meta] Generic entity processor to commit this as long as we simply add field collection fields as mapping targets. I will set this to "needs review", but feel free to prove me wrong on this. We could at least provide a way to import field collection fields with this patch until a generic entity feeds importer is created.

emilyf’s picture

seems like a good point -- once i added this patch from #16 and made the modifications i mention here http://drupal.org/node/1033202#comment-6218624 it allowed me to map my entity with field collection fields.

liquidcms’s picture

@franz and @tim, re: #46, yes, this is exactly what we are trying to do as well... import into a collection which in turn has a collection.

@tim - can you share what the hook code was you used for doing this?

@franz or @tim - i'll take a look at this as well tonight; but if either of you are interested in making this support nested collections; the project i am on could likely fund this work.

@franz... thanks a ton for patch in #16; seems to be working very well. with the feeds_tamper module and a little code in a _feeds_after_parse and a _feeds_presave we have been able to import into a bilingual node with a multi collection field

liquidcms’s picture

made a few assumptions and went for broke.. and surprised i got valid data imported (but not correct).

my structure is a node with a collection field (up to 3) and in that collection i have another collection (unlimited)

i removed the line in patch in #16 disallowing nested collections and sure enough i now get my nested collection showing up in the mapping options.

then i went out on a limb as to how to add data to the cell to represent this and assume feeds_tamper could do a nested explode.

so cell has this for one of the fields in the nested collection: yes,no%%yes,no,maybe%%yes,no,never

so basically in the 3 top level collections:
yes,no
yes,no,maybe
yes,no,never

and, oddly enough.. no errors and what i get imported is:
yes
no
yes

so only 1 item added to nested collection and seems random; but nested explode must have at least worked.. :)

liquidcms’s picture

ok, so yes, no, yes result is simply because feeds_tamper doesn't do what i hoped for explode.. it simply explodes into 1 long array as opposed to making nested arrays.. so i get yes, no, yes, no, maybe, yes, no, never..

liquidcms’s picture

pretty late here so maybe not thinking all that straight.. but i think if we could get feeds_tamper to give a nested array then this should work but the field_collection_feeds_set_target() i think needs a recursive section which i don't see here.

liquidcms’s picture

wow.. and this works.. :)

so the recursion is there in the fact that it calls the callback with an array; which in turn calls the same routine using the items of the array.

so the only issue here is getting the feeds_tamper module to do a nested explode.

liquidcms’s picture

ok, long night.. the patch i added here: #1688536: support nested explodes allows feeds_tamper to create a nested array.

that combined with the patch in #16, except removing the line that prevents it from doing nested, allows importing into nested collections.

charlie-s’s picture

@franz #16 working most well with an unlimited field_collection that contains many fields, using the JSONPath parser and Feeds dev. Hoping I don't have to stumble down the nested-unlimited-field-collections with this project, but who knows. You need to supply a link to paypal you beer (or coffee/tea).

yurgon’s picture

hi,
Dont update value of reference field in field_collection
Any ideas?

Thx

liquidcms’s picture

hmm.. although my solution in #58 allows me to import into nested collections; it breaks re-importing (i.e. importing modified record to same node).

dotpex’s picture

gmclelland’s picture

The patch in #16 worked for me. Thanks everyone, huge timesaver!

seehive’s picture

Great!!! Patch in #16 works perfectly, included with unlimited field collections. just need to explode the "," for that particular field

argo27’s picture

Title: add feeds integration » patch application problem

Hi I have copied the patch in #16 renamed in fcollection.patch
Save the file to the root directory of feeds module sites/all/modules/feeds/
Then apply the patch by using git bash application
I typed

git apply -v fcollection.patch

and it returns and error

"checking patch field_collection.module...
error: field_collection.module: No such file directory"

I hve a field_collection module installed in my site.
So what I did to experiment is to copy the

field_collection.module from field_collection module folder and paste it
inside feeds module folder and run the patch again.

Then I got this error:

"error: patch_failed: field_collection.module:1142
error: field_collection.module: Patch does not apply."

How can I apply the patch successfuly? I badly needed this field-collection - feeds mapping integartion.

Im using drupal acquia in windows 7 OS

tim.plunkett’s picture

Title: patch application problem » add feeds integration
Status: Needs review » Needs work

The patch no longer applies. Someone will need to reroll it.

argo27’s picture

Thanks for the reply.
So what can I do for now?

argo27’s picture

Status: Needs work » Needs review
GaëlG’s picture

If you applied #42 but do not want empty field collection items to be saved, here's a patch.

btopro’s picture

reroll of #28 against latest checkout of dev for testing

btopro’s picture

One issue noted with the targets in this patch: It's unable to do nested field collections. For example I have a type that goes node->field_collection->field_collection. it doesn't drill down into the deeper field_collection

btopro’s picture

removing

// Thou shalt not attempt Sorcery!
  if ($entity_type == 'field_collection_item') {
    return;
  }

Enables at least 1 level of depth to sub-field_collections and works!

btopro’s picture

Here's a helpful transformation method for getting your csv file to conform to the standard needed in order to import to feeds. This takes multiple rows and globs them together based on the unique key for import (in my case 'course')

// convert csv to an associative array
function csv_to_array($input, &$header, $delimiter=',') { 
  $data = array(); 
  $csvData = str_getcsv($input, "\n"); 
  foreach($csvData as $csvLine){ 
    if(is_null($header)) $header = explode($delimiter, $csvLine); 
    else{
      $items = explode($delimiter, $csvLine); 
      for($n = 0, $m = count($header); $n < $m; $n++){ 
        $prepareData[$header[$n]] = $items[$n]; 
      } 
      $data[] = $prepareData;
    }
  }
  return $data; 
}

$header = NULL;
$rows = csv_to_array(file_get_contents('import.csv'), $header);
$record = array();
// the key we want to group everything against
$primary_key = 'course';
// loop through and reassemble based on primary key
foreach ($rows as $row) {
  if (isset($record[$row[$primary_key]])) {
    foreach ($row as $item => $value) {
      // if this is not the primary key then append its value to its location
      if ($item != $primary_key) {
        $record[$row[$primary_key]][$item].=":$value";
      }
    }
  }
  else {
    // set values for the first time they are found
    $record[$row[$primary_key]] = $row;
    unset($record[$row[$primary_key]][$primary_key]);
  }
}
// open new file
$converted = "converted.csv";
$file = fopen($converted, 'w+') or die("can't open file");
// write headers
fwrite($file, implode(',',$header) ."\n");
// iterate and write rows back to file
foreach ($record as $name => $row) {
  $output = $name .','. implode(',',$row) ."\n";
  fwrite($file, $output);
}
fclose($file);
argo27’s picture

I always get this error whenever i apply any of the patch here for fieldcollection module:

error patch failed: field_collection.module:1859
error: field_collection.module:patch does not apply

I use drupal 7 field collection v 7.x-1.0-beta4
feeds v -alpha6

Is it not compatible?

argo27’s picture

How can I re quest for a re-roll of patch in #16 for field collection 7.x-1.0-beta4?

btopro’s picture

need to apply against dev. if you have issues applying then do it manually, it's a pretty basic patch (just remove + signs and add the two functions to the end of the .module)

argo27’s picture

Thank you so much btoro.
Your manual solution recommendation works.
I am no programmer and I have no idea I can just copy and paste the code of the patch to the module file.
Thanks so much. You saved me from several days of headache :)

argo27’s picture

Can i also do the same thing to other patches and modules. What if I see a - sign than a + what does it mean? Does the patch always pasted on the .module file?

btopro’s picture

- means remove that line + means add that line. If you see a minus followed by a + that'll most likely mean replace this with that.

argo27’s picture

Thanks, it makes more sense,

sorry to ask you another question just to confirm
it this code:

diff --git a/entityreference.feeds.inc b/entityreference.feeds.inc
index 093fa5f..d90ecf9 100644
--- a/entityreference.feeds.inc
+++ b/entityreference.feeds.inc
@@ -16,14 +16,42 @@ function entityreference_feeds_processor_targets_alter

should I delete the "@@ -16,14 +16,42 @@" symbols before the function
and this means that I should paste it over the entityreference.feeds.inc file right?

Thanks

dgastudio’s picture

i'm not sure, but after apply the patch, only the array[0] is imported.

i mean, from csv file like this.

"110";"city1,city2,city,3";"1,2,3";"a,b,c";"1,2,3";"i,b,e"

after import, field collection looks like

city, 1, a

nothing more.

field collection it's set to multivalue.

feeds is set to explode columns 2,3,4,5.

column 1 is node id. unique.

field_collection = latest dev
feeds = latest dev

any advice?

David_Rothstein’s picture

The issue raised in #40 (where a field collection with any empty fields inside of it is being treated as an entirely empty field collection and not being saved) is still a problem.

It's possible that #42 combined with #69 is the correct fix for that, but since #69 is a much more far-reaching change and the problem with empty field collections is being specifically introduced here during the Feeds import process, here is a reroll of #16 (which seems like it was the last working patch) with code to deal with that problem during the Feeds import process as well.

With this patch applied, a field collection with empty fields inside of it can still be imported, but an entirely empty field collection will be correctly skipped. (Whether or not the empty field inside of the not-entirely-empty field collection is itself imported or not depends on the properties of the field and is a separate issue, i.e. #1107522: Framework for expected behavior when importing empty/blank values + text field fix).

btopro’s picture

Same patch with Sorcery added back in, you can do nested field_collections via this method.

Everett Zufelt’s picture

I haven't read through the entire thread. Does the proposed solution deal with removing (deleting) unnecessary field collection items on update?

If you have an entity with a field collection imported, and on update to that entity using fields find that a single item in that collection is no longer required, is it removed?

David_Rothstein’s picture

Just a note: I had a situation where I needed to deal with a multi-valued field inside of a multi-valued field collection, which brings up the question of how do you distinguish between a single field collection item (that contains multiple values for the given field) and multiple field collection items (each of which contains only one value for the given field)? I posted a Feeds Tamper patch at #1819986: Add a plugin for reducing an array to a single element which was helpful in getting that to work.

(This comment is unrelated to Everett's question right above mine, by the way. I don't actually know the answer to that myself.)

g089h515r806’s picture

I have create a module http://drupal.org/project/field_collection_feeds which solve this issue in another way.

yurgon’s picture

this module dont work 4 me

gboggs’s picture

With the latest dev version of Feeds and Field Collection, and the Feeds Field Collection module, I can get field collections to import, and saving user profiles with embedded field collections no longer cause blank collections to be created, but I can no longer use feeds to import new users.

"SQLSTATE[23000]: Integrity constraint violation: 1048 Column 'field_listings_revision_id' cannot be null"

Ideas?

John Bickar’s picture

The patch in #83 would not apply for me ("fatal: corrupt patch at line 125") so I added the functions to the module manually, cleared caches twice, and it worked!

I re-rolled the patch against 7.x-1.x-dev.

ashedryden’s picture

Tested patch #89, applied cleanly and seems to have no issues.

xcf33’s picture

Created a patch based on patch #89 (please apply patch on #89 before this).

This is purely for my use case but hopefully it could work for other folks. I wrote this very quickly and it is minimally tested.

In any case my issue was:

I have a field collection field on a node that has "Unlimited value", that field collection item contains 2 fields

1. Name
2. Email

When I create my mapping for feeds I mapped the [field_collection_item:name] and [field_collection_item:email] twice hoping that the field will create multiple instances of the field collection items i.e. $field[LANGUAGE_NONE][$delta];

However, looking at the code it looks like only if the mapping is passing in an array (so have to use feeds tamper or similar tool) in order to create multiple instances of the field collection item. Otherwise $delta will always be 0 and mapping the field collection item fields multiple times will always be overwritten by the last set of mapping.

Comments and feedbacks are welcomed!

arh1’s picture

franz/.glauber: thanks for saving my Friday :)

#16 successfully adds Field Collections as targets available for mapping. Cheers!

AaronBauman’s picture

Status: Needs review » Needs work
FileSize
5.96 KB

here's a new patch that includes both #89 and #91 (minus the extraneous whitespace changes)

i'm reverting back to "needs work" because after apply this patch,
i get the following Integrity constraint violation on import:
Integrity constraint violation: 1048 Column 'field_document_details_revision_id' cannot be null
This error may be related to #1810548: "Integrity constraint violation column [xyz]_revision_id cannot be null" when saving entity with attached field collection, in which case we'll have to postpone on that issue.

Regardless, though, the "multiple-values" behavior is a bit off.
For example, a multi-valued data source results in one field collection for each column of each row in the data source, instead of one field collection for each row in the data source.

AaronBauman’s picture

Another issue I've run into with the approach in the current branch of patch development: updates to field collection items are more or less ignored.
If the parent entity is getting updated, and field collections are already assigned, they do not get updated with the new feed data.

I don't think this should necessarily prevent inclusion of this patch, but I thought it worth noting for dev and testing purposes.

ledom’s picture

Tested #93 patch on last dev version, works fine but any news about handling multi-values? One field collection for each row would be better.

praandy’s picture

Hi,
I have tried this approach with a complex field collection structure, but it does not work fine.
It couldn't match a nested field structure, for example a xml file:

The source

title field
(text field)
description
(text area)
links
1° item X (r) X X (a, b)
2° item X
3° item X (s) X X (c)





The imported content

title field
(text field)
description
(text area)
links
1° field collection X (r) X X (a)
2° field collection X (s) X X (b)
3° field collection X X (c)

I use json parser and xml parser (for this I have applied this patch http://drupal.org/node/1958660).

We need to transfer content "drupal-to-drupal" and we need to transform entity_reference, files and link of a field collection item in unique link field, so we create a custom "form_callback".

So I develop a patch (I have to thank Hernani for helping me).
Could you verify it, if it could be a good solution?

korn3lius’s picture

Hi,
First of all, thanks for the nice works for field_collection and feeds integration.
Patch #16 work fine for me, however, it only import first field and ignored the others.
Let me explain it clearly.
I had field collection with 2 fields => "visit" and "body weight"
and my csv files look like this:

visit | body weight
1 | 60
2 | 70
3 | 80

Patch #16 simply import the first field (visit 1 with body weight: 60) and ignored visit 2 and 3.
How can I import all the data properly. I think I must miss something here.

My environment:
Drupal: 7.21
Field collection: 7.x-1.0-beta5+0-dev
Feeds: 7.x-2.0-alpha7

Thanks you
Kornelius

vinmassaro’s picture

Status: Needs work » Needs review

Patch in #93 working for me for basic mappings. For multi-value fields, I had to process myself with hook_feeds_presave().

Shiraz Dindar’s picture

vinmassaro, can you post what you did in hook_feeds_presave()?

vinmassaro’s picture

@Shiraz Dindar: Sure. Here is what the XML source looked like that needed to be mapped into the field collection. I am using Feeds XPath Parser.

<internationalActivity>
  <title>UK Partner - South Essex Partnership NHS Foundation Trust</title>
  <location country="United Kingdom"/>
  <location country="Angola"/>
  <location country="Armenia"/>
  <location country="Azerbaijan"/>
  <description>Professional Development Program in International Healthcare Management</description>
  <startYear>2010</startYear>
</internationalActivity>

I had a field collection containing four fields:

  1. Activity title (text field)
  2. Countries (multivalue text field)
  3. Description (text field)
  4. Start year (date field)

I couldn't get the multi-value country field to populate with all the countries. What I did was map the entire internationalActivity element to a temporary target (mapped as raw XML) instead of a field. I then converted it to a PHP array inside hook_feeds_presave() and took it apart to programmatically create the field collection items. If there are items, I remove and recreate them when the feed node updates since this was easier than trying to keep track of them. I used this page as a guide to programmatically create the field collection items. Hope this is helpful!

function yale_profiles_feeds_presave(FeedsSource $source, $entity, $item) {
  if ($source->id == 'faculty_profile_importer') {
    
    // Load node object for modification.
    $node = node_load($entity->nid);
    
    // Remove field collection items to prepare for recreation during import.
    if (!empty($node->field_yp_international_activity[LANGUAGE_NONE])) {
      foreach ($node->field_yp_international_activity[LANGUAGE_NONE] as $key => $value) {
        
        // Take field collection item value and unset it so we aren't left with empty field collection entries.
        $field_collection_item_values[] = $value['value'];
        unset($node->field_yp_international_activity[LANGUAGE_NONE][$key]);
      }
      
      // Delete field collection items.
      entity_delete_multiple('field_collection_item', $field_collection_item_values);
    }
    
    // Process international activity and write it to the field collection.
    if (!empty($entity->{'Temporary target 1'})) {
      
      // If there is only one activity, it is set as a string. Cast it as an array so we can process it the same.
      if (is_string($entity->{'Temporary target 1'})) {
        $entity->{'Temporary target 1'} = (array) $entity->{'Temporary target 1'};
      }
      
      foreach ($entity->{'Temporary target 1'} as &$activity) {
        
        // Convert the raw xml to a PHP array (found from github.com/gaarf/XML-string-to-PHP-array).
        $activity = json_decode(json_encode((array) simplexml_load_string($activity)), 1);
        
        // Normalize the country array for all locations.
        if (!empty($activity['location']['@attributes'])) {
          $attributes = $activity['location']['@attributes'];
          unset($activity['location']['@attributes']);
          $activity['location'][]['@attributes'] = $attributes;
        }
        
        // Convert countries to ISO2 codes since they are saved in this format.
        foreach ($activity['location'] as &$location) {
          if (!empty($location['@attributes']['country'])) {
            $country = $location['@attributes']['country'];
            $location['@attributes']['country'] = countries_country_lookup($country)->iso2;
          }
        }
        
        // Convert year to Y-m-D H:i:s format so it can be created.
        if (!empty($activity['startYear'])) {
          $activity['startYear'] = date('Y-m-d H:i:s', strtotime($activity['startYear'] . '-01-01'));
        }
        
        // Prepare values to create a field collection item.
        $values = array(
          'field_name' => 'field_yp_international_activity',
          'field_yp_activity_description' => array(
            LANGUAGE_NONE => array(
              array('value' => (!empty($activity['title'])) ? $activity['title']: '')
            ),
          ),
          'field_yp_start_year' => array(
            LANGUAGE_NONE => array(
              array('value' => $activity['startYear'])
            ),
          ),
          'field_yp_country' => array(
            LANGUAGE_NONE => array(),
          ),
        );
        
        // Add countries to the $values array.
        foreach ($activity['location'] as &$location) {
          $values['field_yp_country'][LANGUAGE_NONE][] = array_fill_keys(array('iso2'), $location['@attributes']['country']);
        }
        
        // Begin by using the entity_create function to create a new entity of
        // type field_collection_item. The second parameter to the function
        // provides "an array of values to set, keyed by property name".
        $insert_entity = entity_create('field_collection_item', $values);
        
        // Attach the field_collection entity to the application node. This has to
        // happen separately so that it actually works -- you can't just specify
        // this via the $values array.
        $insert_entity->setHostEntity('node', $node);
        
        // Save the entity. Since it's attached to the right application node, this
        // will both create the field_collection entity and update the application
        // node to point to the new field_activity_log record.
        $insert_entity->save();
      }
    }
  }
}
xcf33’s picture

@aaronbauman re #93

I'm working on making the patch work from #93 for the following scenarios:

1. Multiple field collections, this means a node can have multiple field collection attached to it.
2. Multiple value fields in a field collection. This means create a field in the field collection that can have multiple values.

There are more complex cases such as field collection inside of field collection but I think this should work for that or at least maybe not supported at the moment.

It's a good note that field collection is not being updated when parent is updated by feeds, this is something we should build in as well.

xcf33’s picture

@aaronbauman re #93

Patch based on #93 that should handle (Please patch against 7.x-1.0-beta5)
1. Multiple field collections, this means a node can have multiple field collection attached to it.
2. Multiple value fields in a field collection. This means create a field in the field collection that can have multiple values

This has been tested initially, need to also look at the update scenarios.
Did not test against field collection inside of field collections.

fehin’s picture

Just applied patch in #102 and below is the message I got

field_collection-feeds-1063434-102.patch:51: trailing whitespace.

field_collection-feeds-1063434-102.patch:54: trailing whitespace.
  $field = isset($entity->$target) ? $entity->$target : array();
field_collection-feeds-1063434-102.patch:56: trailing whitespace.

field_collection-feeds-1063434-102.patch:70: trailing whitespace.
      $sub_field_info = field_info_field($sub_field[0]);
field_collection-feeds-1063434-102.patch:74: space before tab in indent.
              $callback($source, $field_collection_item, $sub_target, $v);
Checking patch field_collection.module...
Applied patch field_collection.module cleanly.
warning: squelched 5 whitespace errors
warning: 10 lines add whitespace errors.
codesmith’s picture

#102 worked for me for a 2 field, field collection. I tried to add multiple values using a colon but that failed - not sure if it's supported or if I didn't set it up right. For example, using something similar to #20:

"fruit","color"
"apple:banana","red:yellow"

I got just one field collection entry:

fruit: apple:banana
color: red:yellow

instead of the expected:

(entry 1)
fruit: apple
color: red

(entry 2)
fruit: banana
color: yellow

Shiraz Dindar’s picture

@vinmassaro -- a belated thanks for sharing your code. I didn't end up proceeding with this but I found your code and comments illustrative nonetheless.

franz’s picture

Status: Needs review » Needs work
+++ b/field_collection.module
@@ -1862,3 +1862,127 @@ function field_collection_devel_generate($object, $field, $instance, $bundle) {
+  ¶
+  // Iterate over all values.
+  $delta = 0;
+  $field = isset($entity->$target) ? $entity->$target : array();    ¶
+  $language = (isset($entity->language)) ? $entity->language : LANGUAGE_NONE;
+    ¶
+  foreach ($value as $v) {
+	    if (isset($field[$language][$delta]['entity'])) {
+	      $field_collection_item = $field[$language][$delta]['entity'];
+	    }
+	    elseif (isset($field[$language][$delta]['value'])) {
+	      $field_collection_item = field_collection_item_load($field[$language][$delta]['value']);
+	    }
+	    if (empty($field_collection_item)) {
+	      $field_collection_item = entity_create('field_collection_item', array('field_name' => $target));
+	    }
+      // let's see if the field in the field collection has multiple cardinality
+      // in case the field is complex type such as field_url:title, field_url:link
+      $sub_field = explode(':', $sub_target);
+      $sub_field_info = field_info_field($sub_field[0]);		
+	    if (isset($_sub_targets[$sub_target]['callback']) && function_exists($_sub_targets[$sub_target]['callback'])) {
+	      $callback = $_sub_targets[$sub_target]['callback'];
+	      if($sub_field_info['cardinality'] == 1) {
+  	      $callback($source, $field_collection_item, $sub_target, $v);
+	      } ¶
+	      else {
+  	      $callback($source, $field_collection_item, $sub_target, $value);
+	      }
+	    }
+	    $field[$language][$delta]['entity'] = $field_collection_item;
+	    $field[$language][$delta]['value'] = $field_collection_item->item_id;
+	    unset($field_collection_item);
+	    if ($info['cardinality'] == 1) {
+	      break;
+	    }
+      if ($sub_field_info['cardinality'] == 1) {
+        $delta++;
+      }	    ¶

There are some ugly tabs and whitespace in this section

arh1’s picture

#102 (patched against 7.x-1.0-beta5) allows the field collections to be set as targets again, but doesn't save the values of text fields in field collections for me.

Digging a bit, I see that in text_feeds_set_target() (from feeds/mappers/text.inc) $field['und'] contains an element, whereas it's an empty array for the non-field collection fields I'm testing. This means that the foreach loop is always broken before setting the value in text_feeds_set_target() via this bit:

    if ($info['cardinality'] == $delta) {
      break;
    }

So I'm trying to understand enough of Feeds and the Field Collection integration discussed here to come up with a reasonable solution... I'm surprised no one else in this thread has hit the same problem.

arh1’s picture

Re: my previous comment...

I'm surprised no one else in this thread has hit the same problem.

Oops, now I see the notes in #102 and #94 above.

Regarding updating existing field collection items, if I clear the value of the field's target in the entity object inside of field_collection_feeds_set_target before the callback with something like this:

unset($field_collection_item->$sub_target);

..then the value of $delta inside of text_feeds_set_target in feeds/mappers/text.inc is set to 0 as desired, so that the field is iterated over once and the value saved correctly. This does not seem to break my initial import of the data, either.

This feels like a hack, though, and I'm sure it breaks some of the scenarios above, so I'm not providing a patch yet. Can anyone who's been working on this longer than I have comment on this approach?

arh1’s picture

Meh, here's a patch after all, just to make it easier to discuss. Please don't apply to this to any site you care about.

I fixed the whitespace issues per #106, too.

gjones’s picture

Hi Everyone,
I was able to get Feeds 7.x-2.0-alpha8 to work with Field Collection 7.x-1.0-beta5 by utilizing the Explode feature from Feeds Tamper 7.x-1.0-beta4

Step 1: Apply the patch https://drupal.org/node/1063434#comment-7484110 onto Field collectoin 7.x-1.0-beta5
Step 2: This is my how spreadsheet is setup.

Name | Title | Phone | Email
-----------------------
Bob, Joe, Sally | Sir, Mr., Mrs. | 604-555-5555, 604-123-4567, 604-555-0011 | bob@test.com, joe@test.com, sally@test.com
John, Adam, Susan | Dr., Mr., Miss | 604-555-1111, 604-123-2222, 604-333-5511 | john@asdf.com, adam@asdf.com, susan@asdf.com

Note: Each row is an entire field collection and each column is a single field collection field. Also, I'm exporting as a TAB separated value, and having to convert it to UTF-8

Step 3: Setup your feeds mapping to the use the field collection fields.

Step 4: Setup Feeds Tampers
Using Feeds Tamper (must apply quick fix here: https://drupal.org/node/1929312#comment-7166832), I set a new Explode field tamper plugin for the each of my comma separated columns. Note: Your explode string must match your spreadsheet exactly. In other words, if your spreadsheet has a space after the comma, then your explode string needs to have this space for it to work.

Then run the feeds import.

arh1’s picture

Status: Needs work » Needs review

To reiterate quickly, #109 is based on #102 but adds support for updating existing entities (via the line in #108) and cleans up some whitespace problems (I should have provided an interdiff).

I'm taking a top-down look here and providing a pragmatic fix for my use-case, but if #102 was working for people it's worth reviewing.

Status: Needs review » Needs work

The last submitted patch, field_collection-feeds-1063434-109.patch, failed testing.

jeromec’s picture

Hello,

i would like to be able to import a node with this structure :
-nid
-title
-fieldcollection :
--myfield1 (term reference with widget set to Autocomplete term widget (tagging), 1 value)
--myfield2 text

All is working perfectly except the insert of myfield1 who is not working. When i change the widget type of Select list it's works. But i want to be able to insert new term reference, the only way to to this is use the Autocomplete term widget (tagging) type.

i have configured my feed parser to CSV, delimiter (;) , processor node, in my target configuration for myfield1 i have checked Auto create and the search taxonomy term is by Term name.

Feeds : 7.x-2.0-alpha8
Field collection : 7.x-1.0-beta5

i also have applied the patch #16

I dont know what to do.

Can someone help me please ?

azin’s picture

Status: Needs work » Needs review

Status: Needs review » Needs work

The last submitted patch, field_collection-feeds-1063434-109.patch, failed testing.

azin’s picture

Status: Needs work » Needs review

Status: Needs review » Needs work

The last submitted patch, field_collection-feeds-1063434-109.patch, failed testing.

twistor’s picture

Just a note, this should not be using feeds_alter(). A regular drupal_alter() will suffice.

humansky’s picture

Patch #109 won't work for me because it is looking for the module in "sites/all/modules/field_collection/". That is probably also why it is failing testing. I've rerolled the patch with a more relative path to the modules' files

humansky’s picture

Status: Needs work » Needs review
queenvictoria’s picture

I've updated #119 as work so far doesn't pass in the mapping to the callbacks on the sub_targets. Thus settings such as 'autocreate' on taxonomy fields don't passed in and therefore new terms don't get created.

See http://drupal.stackexchange.com/questions/83527/how-to-import-term-refer... for more information

Interdiff attached too.

jrao’s picture

This:

+      if($sub_field_info['cardinality'] == 1) {
+        $callback($source, $field_collection_item, $sub_target, $v, $sub_mapping);
+      }
+      else {
+        $callback($source, $field_collection_item, $sub_target, $value, $sub_mapping);
+      }

and this

+    if ($sub_field_info['cardinality'] == 1) {
+      $delta++;
+    }

doesn't seem to be correct, this makes it impossible to import multiple field collection items that has a subfield with multiple values (i.e. #36), in which case this code will just put everything in the first item's multi-valued field.

The code shouldn't care about cardinality of sub fields, or try to breakup the values to fit the sub fields, that's the job of tamper.

I think this should be changed to:

+      $callback($source, $field_collection_item, $sub_target, $v, $sub_mapping);

and

+    $delta++;
jrao’s picture

FileSize
1.26 KB

A revised explode twice tamper plugin, fixed a small bug in the version from #36

ikeigenwijs’s picture

Do I understand correctly
That you need to do 2 imports with different processors

I have csv with
all node information including the collection field 1 level deep only

What i m trieing to do now, not succeeding yet is the following
Is 2 imports
1 for the node data with node processor
1 for the collection field date (https://drupal.org/project/field_collection_feeds)
both mapped to the unique field

csv file:
Title; uniq R number ; a lot of simple fields on the node ; collection field storage
"proc35";35; .a lot of fields; closet1.5.3.2,closet2.3.4.2

thx a lot for the help and directions

chhay’s picture

GaëlG’s picture

#121 with #122 edits did not work well on our multilingual website. We had to change

$language = (isset($entity->language)) ? $entity->language : LANGUAGE_NONE;

to

$language = ($info['translatable'] && isset($entity->language)) ? $entity->language : LANGUAGE_NONE;

in field_collection_feeds_set_target().

GaëlG’s picture

For anyone facing the same need, here is the code that worked for me to import an entity_translated field collection (with this patch: https://drupal.org/comment/8016755#comment-8016755) with parent field set untranslatable and some subfields set as translatable.

/**
 * Implements hook_feeds_processor_targets_alter()
 */
function field_collection_feeds_processor_targets_alter(&$targets, $entity_type, $bundle_name) {
  foreach (field_info_instances($entity_type, $bundle_name) as $name => $instance) {
    $info = field_info_field($name);
    if ($info['type'] == 'field_collection') {
      $new_targets = array();
      feeds_alter('feeds_processor_targets', $new_targets, 'field_collection_item', $info['field_name']);
      foreach ($new_targets as $sub_name => $target) {
        $new_name = $info['field_name'] . ':' . $sub_name;
        $targets[$new_name] = $target;
        if (isset($target['name'])) {
          $targets[$new_name]['name'] = $instance['label'] . ': ' . $target['name'];
        }

        // We override callback for now and retrieve original later.
        $targets[$new_name]['callback'] = 'field_collection_feeds_set_target';
        $targets[$new_name]['form_callback'] = 'field_collection_feeds_mapper_form_language';
        $targets[$new_name]['summary_callback'] = 'field_collection_feeds_mapper_summary_language';
      }
    }
  }
}

/**
 * Process Field Collection items
 */
function field_collection_feeds_set_target($source, $entity, $target, $value, $main_mapping) {
  static $sub_targets = array();

  $args = explode(':', $target);
  $target = array_shift($args);
  $sub_target = implode(':', $args);

  // Retrieve the submapping so we can pass it to the callback
  $sub_mapping = array();
  $config = $source->importer()->getConfig();
  if (!empty($config['processor']['config']['mappings'])) {
    foreach ( $config['processor']['config']['mappings'] as $mapping ) {
      if ( $mapping['target'] == $target . ':' . $sub_target && $main_mapping['language'] == $mapping['language']) {
        $sub_mapping = $mapping;
        $sub_mapping['target'] = $sub_target;
        break;
      }
    }
  }
  // Now we retrieve old callbacks and keep then on a static cache
  if (!isset($sub_targets[$target])) {
    feeds_alter('feeds_processor_targets', $sub_targets[$target], 'field_collection_item', $target, $sub_mapping);
  }
  $_sub_targets = $sub_targets[$target];

  $value = is_array($value) ? $value : array($value);
  $info = field_info_field($target);

  // Iterate over all values.
  $delta = 0;
  $field = isset($entity->$target) ? $entity->$target : array();
  $language = ($info['translatable'] && isset($entity->language)) ? $entity->language : LANGUAGE_NONE;

  foreach ($value as $v) {
    if (isset($field[$language][$delta]['entity'])) {
      $field_collection_item = $field[$language][$delta]['entity'];
    }
    elseif (isset($field[$language][$delta]['value'])) {
      $field_collection_item = field_collection_item_load($field[$language][$delta]['value']);
    }
    if (empty($field_collection_item)) {
      $field_collection_item = entity_create('field_collection_item', array('field_name' => $target));
    }

    // Hack: feeds mapping callbacks (at least text_feeds_set_target), seem to
    // want an empty array for a Field Collection item target, to be sure the
    // item is iterated over at least once.
    //unset($field_collection_item->$sub_target);// Commented out for language handling. Might cause problems?

    // let's see if the field in the field collection has multiple cardinality
    // in case the field is complex type such as field_url:title, field_url:link
    $sub_field = explode(':', $sub_target);
    $sub_field_info = field_info_field($sub_field[0]);
    if (isset($_sub_targets[$sub_target]['callback']) && function_exists($_sub_targets[$sub_target]['callback'])) {
      $callback = $_sub_targets[$sub_target]['callback'];
        $callback($source, $field_collection_item, $sub_target, $v, $sub_mapping);
    }
    $field[$language][$delta]['entity'] = $field_collection_item;
    $field[$language][$delta]['value'] = $field_collection_item->item_id;
    unset($field_collection_item);
    if ($info['cardinality'] == 1) {
      break;
    }

      $delta++;

  }
  $entity->{$target} = $field;
}

/**
 * Implements hook_feeds_presave().
 */
function field_collection_feeds_presave($source, $entity) {
  // Do not save any empty field collection items that may have been created
  // during the mapping process. Since the mapping is done field by field in
  // field_collection_feeds_set_target(), we have to wait until this hook (when
  // the field collection item is fully built up) before we can check if it's
  // empty.
  $config = $source->importer()->getConfig();
  if (!empty($config['processor']['config']['mappings'])) {
    // Find all field collection mappings.
    foreach ($config['processor']['config']['mappings'] as $mapping) {
      if (isset($mapping['target'])) {
        $args = explode(':', $mapping['target']);
        $field_name = array_shift($args);
        if (($field = field_info_field($field_name)) && $field['type'] == 'field_collection') {
          // If the field collection item is empty, do not save it.
          if (!empty($entity->{$field_name})) {
            foreach ($entity->{$field_name} as $langcode => &$items) {
              foreach ($items as $delta => $item) {
                if (isset($item['entity']) && field_collection_item_is_empty($item['entity'])) {
                  unset($items[$delta]);
                }
              }
              // Clean up the final array.
              if (empty($items)) {
                unset($entity->{$field_name}[$langcode]);
              }
              else {
                $items = array_values($items);
              }
            }
          }
        }
      }
    }
  }
}


/**
 * Returns the language summary text for use in a mapper target summary_callback
 *
 * @param array $mapping
 *   Associative array of the mapping settings.
 * @param array $target
 *   Array of target settings, as defined by the processor or
 *   hook_feeds_processor_targets_alter().
 * @param array $form
 *   The whole mapping form.
 * @param array $form_state
 *   The form state of the mapping form.
 *
 * @return string
 *   Returns, as a string that may contain HTML, the summary to display while
 *   the full form isn't visible.
 *   If the return value is empty, no summary and no option to view the form
 *   will be displayed.
 */
function field_collection_feeds_mapper_summary_language($mapping, $target, $form, $form_state) {
  if (module_exists('locale')) {
    list($field_name, $subfield_name) = explode(':', $mapping['target']);
    $info = field_info_field($subfield_name);
    if ($info['translatable']) {
      $language_options = array(LANGUAGE_NONE => t('All languages')) + locale_language_list('name');
      if (empty($mapping['language'])) {
        return t('Language: <strong>@search</strong>', array('@search' => $language_options[LANGUAGE_NONE]));
      }
      return t('Language: <strong>@search</strong>', array('@search' => $language_options[$mapping['language']]));
    }
  }
  return FALSE;
}

/**
 * Returns tha language form for use in a mapper target form_callback
 *
 * @param array $mapping
 *   Associative array of the mapping settings.
 * @param array $target
 *   Array of target settings, as defined by the processor or
 *   hook_feeds_processor_targets_alter().
 * @param array $form
 *   The whole mapping form.
 * @param array $form_state
 *   The form state of the mapping form.
 *
 * @return string
 *   Returns, as a string that may contain HTML, the summary to display while
 *   the full form isn't visible.
 *   If the return value is empty, no summary and no option to view the form
 *   will be displayed.
 */
function field_collection_feeds_mapper_form_language($mapping, $target, $form, $form_state) {
  $form = array();
  if (module_exists('locale')) {
    list($field_name, $subfield_name) = explode(':', $mapping['target']);
    $info = field_info_field($subfield_name);
    if ($info['translatable']) {
      $language_options = array(LANGUAGE_NONE => t('All languages')) + locale_language_list('name');
      $form['language'] = array(
        '#type' => 'select',
        '#title' => t('Language'),
        '#options' => $language_options,
        '#default_value' => !empty($mapping['language']) ? $mapping['language'] : LANGUAGE_NONE,
      );
    }
  }
  return $form;
}

vladimir-m’s picture

Hi GaëlG,
Thank you very much for this solution. Works fine for me.

colan’s picture

Issue summary: View changes
Status: Needs review » Needs work

Please provide the above code in patch format.

lachezar.valchev’s picture

Status: Needs work » Needs review
FileSize
8.76 KB

Hi all,

I took the initiative to put GaëlG's code from #127 into a patch.

All credits go to GaëlG.

Regards,
Lachezar

pebosi’s picture

Patch and importing of field_collection fields works fine for me. Thanks

snowfly’s picture

Hi which version of feeds does the patch in #130 works with
thanks

pebosi’s picture

I used latest dev.

ikeigenwijs’s picture

A short follow up,

I managed to do the import for field collections in field collections structure,

I added a custom uid (base on different fields) of the parent in all the children and used this field to map all the children to the correct parrent while the csv contains 1 row for each parent or child.

HarlsHicks’s picture

I'm having issues with these patches. I've tried a few of the working ones, even #130. I'm coming up with the same issue, the callback $v is supposed to be an array, but for some reason it's not getting sent as one. I'm using the latest dev of both field collections and Feeds. Here's the error I get with patch #130:

ResponseText: Recoverable fatal error: Argument 4 passed to file_feeds_set_target() must be an array, string given, called in /modules/field_collection/field_collection.module on line 2018 and defined in file_feeds_set_target() (line 51 of /modules/feeds/mappers/file.inc).

Anybody have any ideas?

jason.blanda’s picture

#135 All of the above patches will not work with Feeds 7.x-2.x-dev use Feeds 7.x-2.0-alpha8 #130 worked perfectly.

caw67’s picture

ptach #16 works great! can you implement it in the module?

colan’s picture

Status: Needs review » Needs work

Needs a re-roll based on #136. All patches must apply cleanly to the dev branch.

AtlasX’s picture

Working without errors using patch #121 with a multi-value field collection containing a node reference field and a text field.

Tried patch #130 but gave me this error:

Notice: Undefined index: language in field_collection_feeds_set_target() (line 1976 of /home/ramjet/public_html/testsite/modules/field_collection/field_collection.module).

Still looking to update existing fields which I've been unable to do. It still won't remove fields that are no longer in an updated XML file or make changes to existing ones. It sometimes causes duplicates as well.

I've been using Feeds and importing XML for a while now without this module and it always works great. In that case, my new import seems to overwrite all existing fields and repopulates them from the updated XML, which, again, works great for what I need it for.

dwadson’s picture

#135 - I added a line before 2018 to force $v to an array:

$v = is_array($v) ? $v : array($v);
rcodina’s picture

Patch #16 work for me on Feeds 7.x-2.0-alpha8 However, with Feeds 7.x-2.0-alpha8 I'm not able to import images and files from their URI and I really need this feature. Could anyone do a patch which works with Feeds 7.x-2.x-dev?

johnv’s picture

Title: add feeds integration » Add Feeds integration to FieldCollection

Better title for my dashboard.

k_zoltan’s picture

Re-rolled patch from comment 130 #1063434-130: Add Feeds integration to FieldCollection so that it clearly applies and work with the most recent version of Feeds and Field Collection.
This patch also includes the fix suggested in comment 140 #1063434-140: Add Feeds integration to FieldCollection

Please note that the most recent dev vesion of Feeds includes the patch for the general Entity Processor #1033202: [Meta] Generic entity processor https://www.drupal.org/node/1033202

k_zoltan’s picture

Status: Needs work » Needs review

Lets see if this passes the tests ... if yes than I think this is ready to be reviewed.

JParkinson1991’s picture

Applied patch #89 and it works perfectly so thanks alot for that.

I have a problem however, let me lay it out.

I have a field collection 'FAQ TAB'
Inside this i have the fields 'Tab Title', 'Tab Content'
'Tab Content' is a field collection containing 'Title', 'Body'
'Tab Content' can have unlimited values, so an unlimited number of title and body fields linked together.

How can i add the multiple title and body fields through the feeds importer?

I know i can use feed tamper and explode it etc, but this doesnt seem to work for unlimited value field collections.

Where do i start in trying to solve this?
Apologies for the lack of effort on my part, only been working within drupal for about a month and situations similar to mine seem not to exists on the internet.

Any help would be massively appreciated

hauruck’s picture

Greetings

I have just patched with the #143 and it works pretty well. My only issue remaining is how I use this feed integration with fields that have multiple values. Here is my example:

source CVS -> field collection field in drupal

Author 1: First name -> Author: First name
Author 1: Last name -> Author: Last name

then I map the next author to the same fields hoping with would create anothe entry:

Author 2: First name -> Author: First name
Author 2: Last name -> Author: Last name

unfortunatly only one value gets imported, the Author 1.
Any suggestion on how to import multiple value fields?

priyanka12chavan’s picture

Recently we faced similar issue where we need to add unlimited specifications for a each product. We created field collection with fields Category, Label, Value1, Value2 with unlimited fields. We came across Field collection feeds module. It's docs https://www.drupal.org/node/1831004 provides proper steps to setup importer.

imperator_99’s picture

I've successfully applied the patch in #143 to no avail - my field collection fields don't show up as targets in feeds. I'm using field_collection 7.x-1.0-beta7 and feeds 7.x-2.0-alpha8. There's no errors in the logs, not really sure what to do from here...any advice appreciated!

hauruck’s picture

I found a solution to the multi value problems I was having. Use the tamper module and follow this guide:

http://thejibe.com/blog/12/09/feeds-importing-multiple-values-one-multi-...

bbdata’s picture

Version: 7.x-1.x-dev » 7.x-1.0-beta7

For anyone using 7.x-1.0-beta7 and doesn't need the functionality from #127/#130/#143. Use the patch from #121.

Otherwise you will lose the ability to configure the target of taxonomy term references.

bbdata’s picture

Version: 7.x-1.0-beta7 » 7.x-1.x-dev

Sorry, changing version back.

bbdata’s picture

I would like to also add that there is an unexpected behavior found with a node reference field in a field collection item.

When the feeds importer is set to update a node with a field collection field that contains a node reference field, the node reference field is not updated. This is because the references_feeds_set_target() of the references module would append the source value to the existing array of values. This ends up getting ignored when the node reference field only allows one single value.

I am not 100% sure if this is a bug in the references module or in patch #121. A test was conducted by mapping directly to a node reference field that was part of the node, and it correctly updated with the new value. However, this will not happen when the reference field is inside a field collection item.

The following patch to #121 is rather specific and corrects this by emptying the reference field if it only supports one value before it is passed to references_feeds_set_target(). What happens now is that the source value gets correctly updated.

Perhaps someone with a better understanding of both modules can chime in where this fix will better fit.

GaëlG’s picture

I could not have it work to delete field collections removed from the source. Besides, according to #1107522: Framework for expected behavior when importing empty/blank values + text field fix, the new default behavior is to erase field values for blank fields in source. So I made a new patch to handle this all.

It includes #127.

I removed the lines about $subfield because these variables are never used.

#130 is not included because it added

    if (empty($v)) {
      // Avoid creation of empty field collections
      continue;
    }

without explaining why and not even notifying it was added. It looks wrong to me because a field inside a field collection could be emptied in the source and this should empty the fied in Drupal on update. Avoiding empty collections creation is a task handled by field_collection_feeds_presave().

I did not keep #140 either because I don't see why this should be handled by the Field Collection implementation of feeds_set_target(). If there is a bug because $v is a string, the bug is either caused earlier (the function should be called with $value as an array of arrays) or more probably later in the subtarget mapping callback (the callback should be able to handle string values, it's something that text_feeds_set_target() does for example). I might be wrong on this but at least it is an argued point of view.

jaysonjaynes’s picture

@GaëlG, that one worked for me! I was having the issue of importing in a "0" for a decimal value on an exploded field, and it was ignoring it and shifting the next value into that position. Thank you!

Status: Needs review » Needs work

The last submitted patch, 153: field_collection-feeds-1063434-153.patch, failed testing.

Novitsh’s picture

Applied patch from #153 and added the field collection field in my Feeds importer.
Getting this error back:

An AJAX HTTP error occurred. HTTP Result Code: 200 Debugging information follows. Path: /batch?id=167&op=do StatusText: OK ResponseText: Fatal error: Class name must be a valid object or a string in /drupal/sites/all/modules/contrib/entity_translation/includes/translation.handler_factory.inc on line 93

EDIT: disabling feeds_et did the trick there.
I've also figured out that for me it's working by only making the FC host translatable, not the child fields, and patching this line after #152:

-  $language = ($info['translatable'] && isset($entity->language)) ? $entity->language : LANGUAGE_NONE;
+  $language = ($info['translatable'] && isset($mapping['language'])) ? $mapping['language'] : LANGUAGE_NONE;
SocialNicheGuru’s picture

I could not get 153 to work.

mkhamash’s picture

Patch #153 did work in my case but I think it needs more work to cover more use cases.

troybthompson’s picture

I applied #153 to Field collection 7.x-1.0-beta8+11-dev (2015-Mar-26) and I'm using Feeds 7.x-2.0-beta1

Only some fields in the collections I have showed as available to be mapped. In my case, Entity Reference fields showed, but Text, Decimal, and Date did not. Thoughts?

Status: Needs work » Needs review

Status: Needs review » Needs work

The last submitted patch, 153: field_collection-feeds-1063434-153.patch, failed testing.

firoz2456’s picture

@troybthompson Can you elaborate more, what type of fields you are trying to map?

troybthompson’s picture

I don't know how much more details I have. Without the patch, none of the fields within my field collections show up for mapping.

I have multiple field collections in the content node type I'm trying to import to.

After the patch, only entity reference fields in my field collections show up to map. Within the same field collection that the entity reference field shows, there is also a text field that's not showing up to map. There are two of those field collections that are acting the same way. I also have a field collection that only contains a date field that is not showing up.

All fields were showing up with an older version of feeds.

Does that help? If there's any other details that would help, please let me know. I'm stuck with some feeds importers needing the new version of feeds and some needing the old method.

ScorpionGhost’s picture

troybthompson, patch does not work with Field collection 7.x-1.0-beta8 and Feeds 7.x-2.0-beta1. Help!

troybthompson’s picture

Has anyone been able to map to core fields inside field collections with the latest versions of things?

veronicaSeveryn’s picture

I had a need to implement field_collection field through Feeds, and also discovered that the existing patch isn't longer working with the most recent code.. So I tried to re-work it a little. Attaching a patch (I am currently using it for my needs on one of the sites and it works for me). Not sure if it will need any more adjustments for more complex fields inside field_collection structure, but looks like it does the job at this moment.

The patch is based off of 7.x-1.0-beta8+11-dev field_collection code (March 26th, 2015 1:43:07 PM)

veronicaSeveryn’s picture

Status: Needs work » Needs review

The last submitted patch, 16: field_collection-feeds_integration.patch, failed testing.

The last submitted patch, 28: field_collection-feeds_integration.patch, failed testing.

veronicaSeveryn’s picture

Uploading revised patch from comment #168 - I think I found a way to save complex fields within Field Collection structure. Tested with saving dates, text, numbers, lists.

Zatox’s picture

Working for text fields with Feeds 7.x-2.0-beta1 and Field Collection 7.x-1.0-beta8
Thanks for updating the patch @veronicaSeveryn !
I think we'd need more testing to update the issue to RTBC. Especially on the complex fields... @troybthompson maybe?

peleo’s picture

Issue summary: View changes

Sorry but it did not work for me with Feeds 7.x-2.0-beta1. Doesn´t let me mapping field collections...

troybthompson’s picture

Thanks! This is working for me in all of my cases so far.

Feeds 7.x-2.0-beta1+6-dev (2015-Aug-05)
Field collection 7.x-1.0-beta8+11-dev (2015-Mar-26)

Now they just have to get Address Field feeds support working again and I'll be back in business. =^)

Novitsh’s picture

Many thanks first to veronicaSeveryn. I've updated the patch from #173 a bit to comply with coding standards. Also removed some unused variables and declarations that are directly overriden.
More testing on this patch is still required.

SocialNicheGuru’s picture

will this let me do the following

my field_collection might have fields within it called field_my_name and field_my_birthday

the import data has a field name and birthday

will the patch allow me to map name to field_my_name within the field collection and birthday to field_my_birthday within the field collection?

veronicaSeveryn’s picture

I had a Field Collection with the following field types inside: date, textfield, integer, decimal, addressfield, textarea, and I think URL. I was able to import them all successfully with the patch I had provided.

For the Feeds I am utilizing 7.x-2.0-dev version. And like I said, Field Collection patch was based off 7.x-1.x-dev branch but I was utilizing 7.x-1.0-beta8 in my site and it applied and worked with it without a problem for the fields I had to import.

UPDATING: I apologize, I typed the wrong version number previously: it might not work with FC 1.0-Beta1... So your best bet would be at least BETA 8, which worked for sure.

mark.bennett’s picture

I am currently using #177 successfully in order to recognize the first level of field collections in a content type. However, I have field collections inside of field collections and anything under that first level of field collections is not available to map.

For instance, I have a Profile content type. I have a variety of fields in Profiles (email, text, date, link, integer, etc) as well as numerous field collections; Address, Course, Presentation, etc. Under the Presentation field collection though, I have a few standard fields as well as its own "sub" field collection called Review. As a step farther, under the Review field collection, there is another field collection called Reviewer. So the hierarchy looks like this:

Profile
--> Presentation
------> Review
----------> Reviewer

With #177, I was finally able to recognize and map all field collections at the Presentation-level. However, I still cannot recognize any field collections or fields under that level (nothing in Review or below). Would it be possible to tweak the patch to recurse through this hierarchy?

pcambra’s picture

I've found an issue when importing a file field inside a field collection, the mapper for file.inc does this:

$v->setAllowedExtensions($instance_info['settings']['file_extensions']);

Which checks the field collection settings and not the file inside the field collection, wondering if the best way around is to provide a custom mapper or to add a set targets alter.

pcambra’s picture

In case someone else finds #181, I've added a patch in feeds #2619788: Get instance info from the entity instead of from the source for file mappings, not sure if that's the way to go, but also this gist that handles file fields inside field collections https://gist.github.com/pcambra/9de9997ba8780e1f2a2e

pavlosdan’s picture

I had a similar problem as Mark in #180 and tried to play around with the patch a bit.

After commenting out the return statement at:

  // Thou shalt not attempt Sorcery!
  if ($entity_type == 'field_collection_item') {
    //return;
  }

The items in the nested field collections started showing up and some test imports showed it worked fine with some simple textfields (against latest Dev versions of Feeds and Field Collection modules).

I am not sure though if this introduces more issues.

Was the taxonomy issue at #121 resolved? I can't get the feed to create taxonomy terms even when I check the auto-create checkbox. I am not sure if this is because the taxonomy term field is within one of the nested field collections. If that's the issue, is there any way to make this work? I've been trying to make sense of the code to see why it wouldn't work but so far no luck.

Any help is appreciated.

Thanks for all the work put into this already!

EDIT: the imported fields create one item each for the top level field collection, not multivalued fields of the nested collection items. More research and testing needed.

Novitsh’s picture

Changed the patch based on the feedback of #183.
I'm wondering why that line was introduced in the first place?

With the patch taxonomy should be able to be created. Could it be a permission issue in your case @pavlosdan? Feeds can create items as anonymous or as a user. In that last case you need to make sure it has the correct permission to create taxonomy.

iajay’s picture

I found one bug in the above patch(s)
When the feed is updated and the rows are deleted from the xml feed then the field collection values are not deleted and causes duplicates
I think I can explain this better with an example
I have a field collection - status_action - with fields action_date and action_text
This is the structure of the xml

<status_actions>
			<status_action>
				<action_date>2014-12-01</action_date>
				<action_text>Proposed Rules</action_text>
			</status_action>
			<status_action>
				<action_date>2015-02-11</action_date>
				<action_text>Final Regulations</action_text>
			</status_action>
		</status_actions>

When the xml is imported first it is working correctly and I get
2014-12-01 - Proposed Rules
2015-02-11 - Final Regulations

Now suppose the xml is updated to

<status_actions>
			<status_action>
				<action_date>2015-02-11</action_date>
				<action_text>Final Regulations</action_text>
			</status_action>
		</status_actions>

So basically a value is deleted and the feed is again imported via cron
the new values that I get are

2015-02-11 - Final Regulations
2015-02-11 - Final Regulations

I hope I am able to explain the problem correctly
Is there a way around this?
Can we simply delete the existing values and then import the updated xml or can you guys think of some better option?
Awaiting reply
Regards
Ajjaykummar

nico.knaepen’s picture

Status: Needs review » Needs work
monsterplant’s picture

I'm quite confused as to how this patch is meant to work. I have a csv with the following setup:

node title | ID (field collection field 1) | Quantity (field collection field 2) |
test | 1 | 1 |
test | 2 | 2 |

It imports fine for the first row, but for the second one it does not create a new field collection set in the same node. Can someone please enlighten me on how it is meant to be setup?

iajay’s picture

I created a patch, now the code deletes the field_collection before saving.. This is my first patch.. so excuse me if I made some mistake
The patch is based on #184 above
regards

dpfitzsi’s picture

Status: Needs work » Needs review

Switching to Needs Review so that Simpletest checks latest patch.

iajay’s picture

One more problem which I can explain better with an example
consider this xml

<status_action>
		<action_date>2015-01-30</action_date>
		<action_text>Action 1</action_text>
		</status_action>
	    <status_action>
		<action_date>2015-03-19</action_date>
		<action_text>Action 2</action_text>
		<effective_date>2015-04-23</effective_date>
	    </status_action>
	    <status_action>
		<action_date>2015-03-19</action_date>
		<action_text>Action 3</action_text>
		<effective_date>2015-05-23</effective_date>
	    </status_action>

Considering the above the effective date value is assigned to the wrong position After important it looks like
Action Date:2015-03-19
Action Date: Action 1
Effective Date : 2015-04-23

Action Date:2015-03-19
Action Date: Action 2
Effective Date : 2015-05-23

Action Date:2015-03-19
Action Date: Action 3
Effective Date :

So is there a solution for this?
Regards

iajay’s picture

I will try to simplify this further

<status_action>
  <effective_date>2015-03-23</effective_date>
</status_action>
<status_action>
  <action_text>action 2</action_text>
  <effective_date>2015-04-23</effective_date>
</status_action>
<status_action>
 <action_text>action 3</action_text>
 <effective_date>2015-05-23</effective_date>
</status_action>

In this case the action 2 gets assigned to first record and the action_text is not present in the first record
this is how it shows
Action text: action 1
effective date:2015-03-23

Action text: action 2
effective date:2015-03-23

Action text:
effective date:2015-05-23

Now even if a blank tag is added like then also it gives the same result

djcj’s picture

Is there not any batch processing?

This does not work for large data sets (13,000 rows) (16MB xml file). It will error out with SQL timeout. Increasing/removing the timeout isn't a good approach, because a user will not wait that long to import.

Fatal error: Maximum execution time of 30 seconds exceeded in /home/djcjne5/public_html/music/drupal/includes/database/query.inc on line 1020

djcj’s picture

Working Alternative: Field Collection Feeds
http://drupal.org/project/field_collection_feeds

I ended up using this module instead, and it can handle BIG data (13,000+ collection items, 3 fields per collection item). Used XML as an input file and XPATH to grab the fields.

Versions I used:
Field collection feeds - 7.x-1.0-alpha3
Field collection - 7.x-1.0-beta11

EDIT:
Import is not actually working into field collections still. It takes too long and times out when running in front end and background job modes. The XML file is 400,000 lines. Will update this post if I make progress.

dpfitzsi’s picture

Testing out the last patch against a field collection that has multivalue taxonomy checklists inside them appears to fail. Only the first term gets added. This also doesn't allow you explode terms using Feed Tamper (only one will get added).

It appears that in the previous few patches the sub targets carnality check was removed and this appears to be causing the problem. I have added a check back in and set it to use the single value version in the case that there is only one value (like a simple textfield) and to use a multivalue array when there are multiple terms (in the case of taxonomy).

hi.chawla’s picture

Version: 7.x-1.x-dev » 7.x-1.0-beta11
Issue tags: +Multiple Values are not getting saved

dpfitzsi patch is also not working for multiple entries, only first field is getting saved.
Our csv file contains following data

Node Id (Unique)|Field 1|Field 2
876|Ted;Mark|Red;Yellow

only Ted and Red and getting saved for node 876

NWOM’s picture

Version: 7.x-1.0-beta11 » 7.x-1.x-dev
Issue tags: -Multiple Values are not getting saved

Changing back to dev version, since patches are always applied against the dev. Also, removing the tag according to Adding tags to issues and using tags.

arborems’s picture

I also had problems with dpfitzsi patch #196, but was able to get it to work after changing the line "$sub_field_info = field_info_field($sub_target[0]);" to "$sub_field_info = field_info_field($args[0]);" I suspect this is what dpfitzsi originally intended. $sub_target is a string so $sub_target[0] just gives the first letter of the field name.

In my case $sub_target was either 'field_datetime:start' or 'field_datetime:end', so going back to the $args array was necessary to get only the first part, 'field_datetime'.

dpfitzsi’s picture

Issue summary: View changes

Thank you everyone for the feedback on my version of the patch. I will make an update this week and roll a new version.

djdevin’s picture

Using the patches above I can import to field collections however there is one case that is not working.

When importing to a boolean checkbox field (off/on checkbox) you cannot import to the opposite of the default. That is, if the checkbox is not checked, you cannot import it to checked, and vice versa.

I think what's happening is that the field collection is being created, and the default is being saved. Then when the new value is imported, instead of the value being overridden with what is being imported, it is being appended. The field ends up having a value of 0 and 1.

I made this change to fix the issue, it unsets the field before trying to set values on it. The callback is "list_feeds_set_boolean_target" and if you look at that function, it appends to whatever is there. All other fields seemed to still work.

@@ -100,6 +100,7 @@ function field_collection_feeds_set_target($source, $entity, $target, $value, $m
 
       if (isset($_sub_targets[$sub_target]['callback']) && function_exists($_sub_targets[$sub_target]['callback'])) {
         $callback = $_sub_targets[$sub_target]['callback'];
+        unset($field_collection_item->{$sub_target});
         $callback($source, $field_collection_item, $sub_target, array($v), $sub_mapping);
       }
edvanleeuwen’s picture

I am not able to get the patch working using the latest dev (2016-Apr-29). I noticed that in hook_feeds_processor_targets_alter() the info type returned is entityreference, not field_collection. Am I missing something?

djcj’s picture

#201 solved my issue: check boxes were not updating on import

The import still throws errors, but at least it appears to work.

Notice: Undefined index: value in field_collection_feeds_set_target() (line 1582 of /var/www/html/sites/all/modules/field_collection/field_collection.module).
Warning: array_flip(): Can only flip STRING and INTEGER values! in EntityAPIController->load() (line 219 of /var/www/html/sites/all/modules/entity/includes/entity.controller.inc).
SocialNicheGuru’s picture

patch doesn't apply well for me. It stops at

 } catch (Exception $e) {
    drupal_set_message($e->getMessage(), 'error');
$patch -p1 < add_feeds_integration-1063434-196.patch
(Stripping trailing CRs from patch; use --binary to disable.)
patching file field_collection.module
Hunk #1 succeeded at 1874 with fuzz 1 (offset 369 lines).
$ tail field_collection.module

      unset($field_collection_item);

      if ($info['cardinality'] == 1) {
        break;
      }
      $delta++;
    }
  } catch (Exception $e) {
    drupal_set_message($e->getMessage(), 'error');

so I get a WSOD:
PHP Parse error: syntax error, unexpected end of file in field_collection/field_collection.module on line 2004

NWOM’s picture

Version: 7.x-1.x-dev » 7.x-1.0-beta11

That was most likely my fault. I changed the branch to dev, assuming that a patch would be created for it. Since usually only patches that are based on the dev version will get committed. This most likely only applies correctly on 7.x-1.0-beta11. I'll set it back to 7.x-1.0-beta11, just to be safe.

Could anyone else confirm whether this works on the dev release? I'm not currently in the office to double-check the version that I applied this on.

NWOM’s picture

Version: 7.x-1.0-beta11 » 7.x-1.x-dev

Please disregard the last post. It appears that the patch was indeed created for the dev release. Switching back to dev. It may need a re-roll.

sherakama’s picture

[edit]
Aaahhh,
I totally started from the wrong patch. I will update accordingly.
sorry for the spam.
[edit]

sherakama’s picture

sherakama’s picture

With the patch in #196 I was only getting one field collection.

I have made a few more adjustments based on that work:

1. Use &drupal_static convention instead of raw static variables
2. Move the feed mappers loading out of a loop to ensure only loads once, even with static var. Minor performance improvement.
3. Remove the field_collection save in each field loop. Even though it was not saving the parent node it was still causing an enormous amount of database writes and had huge performance implications. Just save the field collections along with the parent entity for a write only once boost.
4. Only stop looping at the correct cardinality of a field. It is possible that a field could be set to 2, 3, 10, etc
5. Check for multiple values on callback save incase feeds_tamper or something explodes and provides an array.

NWOM’s picture

#209 works perfectly for me. Thank you!

elpino’s picture

Status: Needs review » Needs work

Patch in #209 doesn't work for mapped fields on field collection if they are multivalue.

There's an error in delta handling and cardinality.

If source contains multiple values (and these are exploded into an array via Feeds Tamper), the code cycles through them but instead of assigning multiple values for a single field, the code creates new field collections for each value and stops at the field collection's cardinality instead of the individual field's cardinality. So on save, only the last value is saved.

sherakama’s picture

FileSize
5.83 KB

elpino was right. The patch in 209 did not account nor handle multiple value fields.

New patch attached.

sherakama’s picture

Nope, that last patch did not work either.

sherakama’s picture

Ok,

New patch with (hopefully) the correct handling of multiple value field collection items.

moonray’s picture

The importer seems to work well for me, aside from a single notice:

Notice: Undefined offset: 1 in field_collection_feeds_set_target() (line 1952 of /sites/all/modules/contrib/field_collection/field_collection.module).

This was tested on a multi value field in a field collection.

lamp5’s picture

In my case not work:
Fatal error: Call to undefined function stanford_feeds_helper_field_collection_file_feeds_set_target() in sites/all/modules/field_collection/field_collection.module on line 1987
I have got multi value field collection with textfield, textarea and imagefield

moonray’s picture

During a debug session for the following issue, I ran across this particular line, which looks very much out of place:

<?php
        // HiJack the file callback so we can make it work.
        if ($callback == "file_feeds_set_target") {
          $callback = "stanford_feeds_helper_field_collection_file_feeds_set_target";
        }
?>

The problem I'm trying to debug is that when you have multiple sources mapped to the same target, it overwrites the field values rather than setting it as an additional value for the multi-value field.

sherakama’s picture

FileSize
8.08 KB

@moonray,

You are 100% correct. There was an issue with file fields and I totally forgot to translate that function to the patch. I've wrapped all of the patch functionality in to a tidy little module until it gets added to the project here: https://github.com/SU-SWS/stanford_feeds_helper.

Updated patch attached to include the correct function and name.

Thanks for the testing!

moonray’s picture

Why didn't you instead of this

+      $sub_mapping = array();
+      $config = $source->importer()->getConfig();
+      if (!empty($config['processor']['config']['mappings'])) {
+        foreach ($config['processor']['config']['mappings'] as $mapping) {
+          if ($mapping['target'] == $target . ':' . $sub_target) {
+            $sub_mapping = $mapping;
+            $sub_mapping['target'] = $sub_target;
+            // Needs language or feeds mappers shout php notices.
+            $sub_mapping['language'] = $config['processor']['config']['language'];
+            break;
+          }
+        }
+      }

Do this?

+      $sub_mapping = $main_mapping;
+      $sub_mapping['target'] = $sub_target;

Also, I noticed a stray dpm() in your patch.

Additionally, you might want to change
while ($value[$delta]) {
to
while (isset($value[$delta])) {
to prevent a notice mentioned in above comments.

guptahemant’s picture

hi @sherakama
while using patch #218 i am getting the following notice while trying to import the image files and hence the image is not importing.
The file top_banner_2.jpg has an invalid extension.
I have already checked the field for allowed extension and it allows for the uploading of .jpg files
please help

hyside’s picture

@guptahemant

It may be unrelated to your problem, but is the extension of the image files you are importing in uppercase? If so, it might be worth checking out this issue I ran into some time ago...

https://www.drupal.org/node/2650600

guptahemant’s picture

Hi @hyside
Thanks for your reply but that wasn't the case.

dnmurray’s picture

Patch 196 worked for me. I had the same problem as in #204 when using the current -dev; I fixed it by hand. (I'm not sure what I'm going to do to fix my drush make, though.) The 218 patch re-introduced the issue if trying to import lists using feeds_tamper. Only the first value is getting saved.

riddhi.addweb’s picture

Issue tags: +Field collection
yfreeman’s picture

Can the patch be updated as follows, as $value[$delta] will fail if == ""

diff --git a/field_collection.module b/field_collection.module
index 578704d..d7df3cd 100644
--- a/field_collection.module
+++ b/field_collection.module
@@ -1991,7 +1991,7 @@ function field_collection_feeds_set_target($source, $entity, $target, $value, $m
   $field = isset($entity->$target) ? $entity->$target : array();
   try {
 
-    while ($value[$delta]) {
+    while (isset($value[$delta])) {
 
       // Zero out.
       $field_collection_item = null;
moonray’s picture

That should be + while (!empty($value[$delta])) { if you want to keep the equation the same.

elpino’s picture

Patch #218 worked great for me, amended with moonray's comment on #266.

ScorpionGhost’s picture

It does not update existing records.

My modules:
Feeds 7.x-2.0-beta2
Field collection 7.x-1.0-beta11
Commerce Feeds 7.x-1.4

Drupal 7.50

Patch field_collection_feeds-1063434-218.patch

It does not update the value of field collection. Any other well-field updates

Help me please. What should be changed?

Feed informs Updated Content 1. But the value of field Collection - old

ScorpionGhost’s picture

Please include the version FEEDS, FIELD COLLECTION and the date of issue. On which the patch has been tested. I'm sure - it will be convenient for everyone to find bugs. I'm not running the update has added value fields field collection perhaps neither one version of the module. Although I tested on a beta 11 and DIV. Picture image added, too, only the first. Patch field_collection_feeds-1063434-218.patch

SocialNicheGuru’s picture

the patch includes a rogue dpm call https://www.drupal.org/node/1063434#comment-11476479

ScorpionGhost’s picture

New update field_collection 7.x-1.0-beta12

junaidpv’s picture

I came across this thread while trying to upgrade my field_collection to latest version.

Here is the re-rolled patch from comment#218 and updated as per comment#226.

majorrobot’s picture

Patch in #232 worked like a charm for me. Using most recent Feeds module version (7.x-2.0-beta3) and Field Collection 7.x-1.x-dev.

alison’s picture

Hi everybody, I'm extremely appreciative of your work on this issue.

Question: Does this patch intend to provide the same functionality as Field collection feeds? I feel like "yes," but I also feel like, I'm really not sure...

Thank you thank you!

NWOM’s picture

@alisonjo2786: Nope both have different use-cases. The Field Collection Feeds module handles importing just Field Collections (Field Collection Processor).

Whereas this patch, allows importing nodes (as an example) that have field collections attached, so that both get imported in one go.

alison’s picture

Ohhhhh okay, that makes total sense. Thank you -- and thanks so much for the super fast reply!

paulihuhtiniemi’s picture

While testing this, I noticed two issues:
1. we can't use !empty() in while loop as then it will skip values like zero or boolean FALSE (as already mentioned in #225, I think isset() will do)
2. issue with update when new value has less collections than existing one. Let's say node initially had 5 field collections in a field. Now if we need to update this node and we have now have only 4 collections in incoming node, it won't remove that one extra value.

Here's a patch that tries to fix both of these issues.

paulihuhtiniemi’s picture

FileSize
986 bytes

Here's also the interdiff as I didn't see a way to upload multiple files??

paulihuhtiniemi’s picture

Here's an updated version of my previous patch. Better way to unset extra collections. Also at least in my setup I was getting notices about undefined language config, so there's a check for that one, too.

NWOM’s picture

Status: Needs work » Needs review

Setting to Needs Review, since it was set to Needs Work after establishing that the patch did not handle multi-value fields (this was implemented starting with #214).

The last submitted patch, 184: add_feeds_integration-1063434-184.patch, failed testing.

The last submitted patch, 190: add_feeds_integration-1063434.patch, failed testing.

The last submitted patch, 196: add_feeds_integration-1063434-196.patch, failed testing.

The last submitted patch, 232: add_feeds_integration-1063434-232.patch, failed testing.

The last submitted patch, 237: add_feeds_integration-1063434-237.patch, failed testing.

Status: Needs review » Needs work

The last submitted patch, 239: add_feeds_integration-1063434-239.patch, failed testing.

alison’s picture

I applied #232 a couple months ago, but just recently discovered that when I have my importer set up to "delete" nodes when the item no longer exists in the source feed, orphan field collection items are left behind in the field_collection_item and field_collection_item_revision tables, and the field_data_field_*/field_revision_field_* tables for the field collection fields.

In other words, this query returns tons and tons of results -- all of which are field_collection items that do not have "host nodes" b/c the "host nodes" no longer exist b/c they were deleted by Feeds at some point:
mysql> SELECT entity_type,bundle,entity_id,field_my_field_on_a_fc_value FROM field_data_my_field_on_a_fc WHERE entity_id NOT IN (SELECT field_my_fc_value FROM field_data_my_fc);

(I hope that makes sense! If not, pls lmk, I'd be happy to try to restate!)

@NWOM @paulihuhtiniemi I think I know the answer -- and I realize the latest patch isn't fully functional yet anyway, but -- have y'all already handled this issue, or not yet? (If not yet, then for now I'm just chiming in, and I'll try to dig further into the code so I can actually pitch in rather than only chime in.) Thank you!

NWOM’s picture

@alisonjo2786 I just ran into that issue as well the other day. I had ran the import multiple times, and come to find out every time I deleted all of the nodes (after test imports), the field collection entries were left behind. I had to use VBO to delete all entities without a host. I was also on #232 when running the import.

Can anyone confirm that this issue still exists for #239?

alison’s picture

@NWOM phew I'm so glad it wasn't just me :) Yep I used VBO like that, too.

I'll give it a go, but I thought you were saying that the patch at #239 didn't work on multi-value fields? Or I may have misunderstood your comment #240.

NWOM’s picture

Oh, it was because I had changed the status to "Needs Review", since it was originally set to "Needs Work", due to it supposedly not being able to handle multi-value fields. Looking at the comments it appears that it was fixed starting with #214, so I changed the status.

Am eager to hear your results. Deleting 400,000 host-less field collection entities was not fun :P.

alison’s picture

[edit] OMG never mind, I can't read, bahhh never mind never mind...

[original comment...]
@NWOM I'm sorry but I have a stupid question!! Is the idea that I'd use #239 "on top of" my already-patched field_collection code? I thought I was supposed to reverse the patch I applied before, and apply this one instead, but the one at #239 doesn't actually have hook_feeds_processor_targets_alter(), obviously. If I'm supposed to essentially use #239 in addition to the older patch, then I'd want to reverse #232, apply #237, and then apply #239, correct? Or?

Sorry, apparently somehow I've never run into this kind of situation before and therefore don't know what the intended action/procedure is.

alison’s picture

UPDATE: I deleted and re-imported all my feed items, and there are NO orphan field collection items, according to my VBO and according to
mysql> SELECT entity_type,bundle,entity_id,field_my_field_on_a_fc_value FROM field_data_my_field_on_a_fc WHERE entity_id NOT IN (SELECT field_my_fc_value FROM field_data_my_fc);

Wooooot!!

The *only* thing I didn't test, and I'm not sure that I can test it because I'm not permitted to mess with the data source (it's our employee HR system haha, and they don't grant access to the sandbox so I have to go thru their devs if I want test data changes) -- the only thing I didn't test is what happens if the data source of a FC item is removed but the host entity still exists (so in my case, the entity is a "person" and the field collection is a "job/position" -- if someone gets a promotion, they no longer have the old position, but they DO have a new position, so the person entity is still there, but a field collection item was removed). It just so happens that this scenario came up earlier today, before I was testing #239, and I found out about it because the rule I created to automatically delete orphaned FC items only works when the host node is deleted, which did not occur in this situation. But I can't force a position to change so I can test :( But it's a huge organization and there are person record changes every day, I might get lucky in the next 24 hours -- if not, I'll coordinate with the dev over in HR to do some testing via their sandbox.

Either way, I'll circle back here ASAP! (And lmk if you have occasion to test the scenario I described before I'm able to!)

THANK YOU bunches :)

alison’s picture

Hmmmm I think maybe HR isn't super active on Good Friday... I've given my dev friend who'd be the person I'd work with for a sandbox test a heads-up that I might need a bit of his time on Monday -- meanwhile, gonna give it the weekend and see if anyone's records change by Monday late morning.

I hope you have a good weekend!

NWOM’s picture

@alisonjo2786: Sorry wasn't in the office since Thursday. Thankfully we got a 4 day weekend here, so didn't check drupal.org daily like I generally do. Even though you figured it out, I just wanted to say that patches always consist of all changes thus far, so you would always revert a previous patch and then apply the new one, and if someone provides an "interdiff" then it's strictly to show the changes made between two patches to help understand what was changed between the two patches. Not a dumb question at all, since each ecosystem does things differently :)

Glad that the original issue is fixed with #239. That's one more issue down.

That other situation that you detailed would definitely be good to know. I personally only used the feeds field collection import once in order to import data to migrate from a closed source CRM system into drupal. If you run into a reproducible situation and need a 2nd confirmation, I can make a clone of the system and try to reproduce it next week.

I hope you have a great weekend as well!

alison’s picture

Happened again! I'm happy/sad lol. I'm being picked up for lunch, but I'll report back with full details after lunch, I just couldn't wait to say something.

alison’s picture

Okay! So, what happened is that someone's job title and job type changed, so the field collection item with their old job title/type was removed from the node, and a new one was added, with the latest job info for that person. BUT, the old FC item is hanging around in the background.

Attached are screenshots of the old ("unconnected") and new ("connected").

So, my VBO and my aforementioned mysql query confirm that there's one orphaned FC item.

There's one other weird thing, but it's different, but i'll mention it anyway -- it's weird to me that the existing FC item didn't just get updated. The "Position ID" is "used as unique" in my feed importer configuration. So maybe "used as unique" simply isn't possible for a field collection within a node importer -- and I can live with that, b/c as long as we can fix the orphan problem, it won't matter if the position data that gets updated is actually getting deleted and added in again, we don't have any fields on the field collection that get manually edited (so nothing gets lost).

So, I feel like I'm forgetting something, is there other info I should provide, are there more details that would be helpful for troubleshooting?

[EDIT: Just to clarify, the problem occurs regardless of whether the new FC has the same Position ID as the FC item that should have been but was not deleted from the system and now exists as a sad, sad orphan.]

NWOM’s picture

@alisonjo2786: Yeah that definitely sounds like an issue still. Your information will definitely be useful for anyone able to contribute to the existing patch (my PHP skills are severely lacking :P).

If you're looking for a workaround for usage in a production environment, using Rules to delete the orphan item would be possible (Rules Event: After feeds item has been imported). Am on the #drupal-support IRC channel and the drupal slack group, if you need help setting something like this up temporarily.

mackpipe1’s picture

Patch # 239 works to map the fields in the creation of the import, but when importing the data, only saves the fields of the node, which are normal, does not save the information of the fields that are inside the collection. I have the following

Title
Descrcipicion
Region (Collecction)
----------------------------

In the collection Region I have

Text Mode
Objectives Text
Text Methodology
-------------------------------

When I import CVS, it only stores my data

Title
Descrcipicion

But the fields that are inside the collection are empty

I need to implement some hook to store ?, I have seen in several patches that have a hook_presave, this will be the hook that I need ?.

Thank you.

userspan’s picture

Patch # 239 works to map the fields in the creation of the import, but when importing the data, only saves the fields of the node, which are normal, does not save the information of the fields that are inside the collection.

Reviewing information from tables.

Patch # 239 if you are saving the information but not performing the relationship with the node within the main field field_collection_name

Example

Field_collection_name (Does not save relation to fields)
-field_titulo_collec_producto (title field, the information is saved correctly)
-field_desc_collec_producto (field description, information is saved correctly)

Some form or patch to finish performing the relation with the parent of the field that is added in the node.

majorrobot’s picture

I'm not sure if this is what's causing the issue mentioned in #258 and #259, but I have a found a bug in #239 that occurs when you are importing a node with multiple instances of a field collection and not all fields in your field collection are included in your feed source. The result is that the first instance of the field collection saves fine, but successive ones leave out fields.

Say I have a json feed like this:

{
	"departments": [
		{
			"nid": 1,
			"title": "Accounting",
			"people": [
				{
					"name": "Allie",
					"phone": "111-1111",
					"bio": "Born in Alabama"
				},
				{
					"name": "Bo",
					"phone": "222-222",
					"bio": "Born in Bombay"
				},
				{
					"name": "Callie",
					"phone": "333-333",
					"bio": "Born in California"
				}
			],
		}
	]
}

Each item in "people" should map to a "person" field collection with fields for name, phone, and bio. This works beautifully if the data is complete, like the above. But, if your data is not as consistent, like the following, you run into issues:

{
	"departments": [
		{
			"nid": 1,
			"title": "Accounting",
			"people": [
				{
					"name": "Allie",
					"phone": "111-1111",
					"bio": "Born in Alabama"
				},
				{
					"name": "Bo",
					"bio": "Born in Bombay"
				},
				{
					"name": "Callie",
					"bio": "Born in California"
				}
			],
		}
	]
}

The first person field collection saves fine, but the other 2 save without data in the "name" field. They do save with data in the last field ("bio," in this case), though.

I've spent several hours debugging the patch to figure out what's going haywire, and don't have anything yet. I suspect it has something to do with $delta in the while() statement. I also wonder if the $entity->{$target} is getting reset somehow instead of accumulating data as Feeds steps through the field collection's fields.

majorrobot’s picture

I've found the cause of the bug I found above, and it looks like it's the code that removes extra field collections when you're importing:

if (count($value) < count($field['und'])) {
  $field['und'] = array_slice($field['und'], 0, count($value));
}

This code does remove extra collections, but unfortunately also has the side effect of losing imported data if the feed source has a different number of values in one field than it has in another.

For example, in my second json example above, there should be 3 values for my person field collection (since there are 3 people records). However, since the second and third person do not have a phone number in the data, when iterating through the phone number field, the code finds count($value) not to be equal to count($field['und']). There is only one phone number value, but 3 $field['und'] values. Thus, the code slices out the second and third person field collections and leaves me w/ only $field['und'][0].

So, I'm attaching a patch where these lines have been removed. But, we still need a way to resolve the problem of removing extra field collections. The issue is finding a way to identify which collection should be removed when the number of collections don't match between the source and the current entity. It's easy enough to count them and compare the count, but without a unique field on each field collection, how do you know which to zap? I've thought about using something like the hash method that Feeds module uses for other entities, but I foresee that being a good bit of work that might include adding a database table. I've started some work in a hook_feeds_presave() function.

@paulihuhtiniemi, you kindly provided the code above ... any ideas for how to implement this another way?

majorrobot’s picture

There was a problem with the way I created my last patch. Oof, it's been a while. Replacement here.

alison’s picture

@majorrobot are you in Baltimore? I'm hangin at the "Feeds" sprint table, even though I'm not acutally sprinting on feeds, b/c i'm thinking they might be able to help me if I need help :) :)

Re: this issue, I may be lost about what you're saying, but, when I used #239 I was able to import items that create nodes and have values map to field collection fields just fine -- and I could even have them update existing field items just fine. The *only* lingering problem for me after applying #239 was what I described in #256 re: "ghost" FC items hanging around in the background instead of being removed completely, when feed item nodes are deleted because they no longer exist in the source feed OR when the source data for the field collection items are no longer on the host record in the source feed (i.e. for me the feed items are person records, and the FC items are the jobs held by those persons -- so if someone's promoted, their old job is no longer on their record and the new job is -- but a "ghost" of the old job is still hanging around in the site database). (See also -- my explanations in #252 and #256.)

OH and also I guess, it's weird that the FC items aren't getting 'updated and instead they're being removed/replaced, even though the position ID is unique and stuff, and i have it all configured to use that. But I don't reeeeeally care about that, I only care that I don't continue to have ghost items :) :)

So like I said, I am going to be trying to work on this more til sprint time is over! If you're here and not busy, c'mon over to the Feeds table!

majorrobot’s picture

@alisonjo2786, I'm not in Balitmore (unless you count Detroit a remote suburb), but wish I were that close to the action!

So, my patch above (#262), actually eliminates the code that deals with the "old" field collections. I felt it important to do this since it was causing further breakages and, as you noticed, not really deleting the field collections anyway — just orphaning them.

There's gotta be a better way to do this. Maybe we can brainstorm something.

I see your note about using the unique id, and that's interesting. I've spent a lot of quality time in the code for this, and there's no application of the unique id there ... but maybe that's something we can do. I like that better than creating a hash of the whole FC and then comparing that to new data.

I can spend a little time working on the unique id idea today.

Any other ideas?

alison’s picture

Ack! Bummer.

So actually, and I see now that I said it wrong in my last comment -- the thing I described was the case with #232, but since applying #239, FC items on **imported nodes that get deleted b/c they no longer exist in the source feed** ARE properly and cleanly removed along with their host nodes. FC items (that are on imported nodes that are not deleted) that SHOULD be deleted because the incoming feed item has a change that means the source data for that FC item is no longer there (like the "promotion" example I mentioned), THOSE field collection items become orphans/ghosts, haunting my DB and my dreams :) (In the admin UI, they get the label "Unconnected field collection item.")

Point being, #239 was helpful for me, just didn't go "all the way."

Could you maybe clarify/restate the problem you were having that you aimed to solve with the newer patch? Data gets mapped to FC fields just fine for me, but I'm not sure if I'm totally getting what you're describing.

majorrobot’s picture

So, the issue with the code in #239 is that if you have a feed where some field collections don't include every field, you end up with corrupted data. This has to do with the way that the code compares the number of field collections and the number of fields included in the feed, which, as it turns out, can be apples and oranges.

In my example above, some people (which are mapped to field collections) did not include phone numbers, but some did. The code counted the number of phone numbers in the feed and suddenly assumed that was how many field collections I had. So it dumped some of the field collections it was importing and generally mucked things up.

Does that help?

alison’s picture

Oh! Okay I understand what you're saying now, yes that helps a lot, thanks. I haven't observed that in my usage, but I'll do some more testing to see if I can reproduce.

Y'know how if you want to be able to have feed items get updated when the feed re-imports, you have to designate one of the node fields to be "used as unique" (in your mapping settings) -- and in order to be able to do that, you usually have to use the field validation module so you can add a validation rule to that node field have to have unique values...? So, I have that set up for the ctype that my feed items become nodes of, so to speak (my "person" ctype has a field with a validation rule set up, and then I have it "used as unique" in my feed importer mapping settings -- it's a unique employee ID text field), AND I have a unique field set up for my "jobs/positions held by the person" field collection (it's a unique "position ID" text field). I'm saying all of this just to explain the unique ID thing I mentioned earlier that you sounded curious about -- hopefully my explanation makes some sense :) -- but also like, in case you think it might explain whey the data mucking isn't happening to me (like maybe b/c there's a unique ID field on my FC config, it's using that for knowing how many FC items there should be on each node...?). Totally speculating, but, there it is :)

alison’s picture

Okay and now I've re-read the interdiff for your patch on #262, and I get it a lot better now.

I had been focusing on what was added in #239 around line 2100 -- the lines you removed -- thinking that maybe I could add something that would actually delete those FC items. But I see what you're doing, so I'm going stare at that some more for the next 30ish minutes -- then they're kicking us out of the convention center, and like, I suppose I'll have to start driving back to Ithaca haha.

I'll let you know how it goes and/or if I think of anything else!

alison’s picture

@majorrobot are you on Drupal slack?

- - - - - - - - - -
The first thing that's sticking out for me is that it _looks_ like $old_entities is actually the node, not the field collection field items array? But also, I'm feeling like I don't know what you're aiming at with the array_shift, even if $old_entities were the FC field items array. But I'm still staring/pondering.

majorrobot’s picture

I see what you're saying about the unique field (I didn't realize you could do that with the Field Validation module. I've been writing my own hooks this whole time...). Just added back in the FC-removal lines of code with my same feed (some people missing phone numbers), and I get the same mucked up results, unfortunately. So, looks like the uniqueness doesn't apply at the FC level, only at the host entity level.

My current idea: Use hook_feeds_processor_targets() or hook_feeds_processor_targets_alter() and add a form_callback to field collection targets to generate a different checkbox in the mapping form that says "used as unique for field collection" or some such.

Then, later in the set_target callback or in the presave hook, this uniqueness is checked. If a field collection exists that does not have a unique value that exists in the feed, delete the field collection. If a field collections exists that does have the unique value, update that field collection instead of creating a new one, if possible.

alison’s picture

I think that what you're describing makes sense -- it feels like what I was trying to conceptualize before, with minimal success. It definitely sounds like the right track to me.

Have you looked at how Feeds uses the "use as unique" functionality when it looks for and then updates/deletes existing items? (I haven't looked into that yet.)

Aaaaaand, very sadly, I gotta get on the road :( :( :(

I'll check in again over the weekend or Monday, depending on how "post-DrupalCon-catchup" goes, in the office and at home haha.

Thanks so much for your work on this! I hope I can help more soon.

alison’s picture

OMG i totally just reproduced data mucking........ yikes!!!!!!!!!!!!!!! and now i'm getting on the road for real haha. bye for now.

majorrobot’s picture

All right, here is a new patch. It solves the issue of deleting field collection items that are no longer in the feed. I actually didn't rely on uniqueness at all, but just wiped out the old items and saved the new ones. The simplicity of this approach outweighs any downsides to me.

In the process, I discovered a few more issues that apparently already existed:

  1. If fields in the feed are inconsistent (as discussed above), fields can still save on the wrong field collection item when inserting a node. This is because the field values are handled largely independent of the item they should be saved onto. The process relies on the $value[$delta] for each set of field values, which does not necessarily reference the correct FC item.
  2. The above gets even funkier when a node is being updated, and a field that existed in the feed no longer does.
  3. Nested FCs (field collections that have as a parent another field collection) are not being saved correctly.

So, one step forward, and more steps discovered.

alison’s picture

Very cool, @majorrobot. The only thing I'm not clear on is #2 -- could you clarify/restate the situation you're talking about? Thank you!

majorrobot’s picture

Hi @alisonjo2786, #2 is just basically #1, but when a node is being updated rather than inserted. The code still (frustratingly) has a hard time when a feed item is missing a field.

I've wrestled with this for quite a while, and I've discovered that these issues (#1 and #2) are actually dependent on the parser. By the time the data is parsed and gets passed to our hooks, (at least in my case) it's in a structure that cannot possibly be saved into the correct field collection items, except by coincidence. In my case, I'm using Feeds JSONPath Parser, but other parsers may have the same issue (and my guess is that some do).

To explain: the JSONPath parser steps through the different subfields you've mapped and creates a simple array with the values that exist in the feed regardless of their position in the feed. The data does not maintain an association with *which* item in the feed that it comes from. So, take this example, where people will be mapped to field collection items:

{
	"departments": [
		{
			"nid": 1,
			"title": "Accounting",
			"people": [
				{
					"name": "Allie",
					"phone": "111-1111",
					"bio": "Born in Alabama"
				},
				{
					"name": "Bo",
					"bio": "Born in Bombay"
				},
				{
					"name": "Callie",
					"phone": "333-3333",
					"bio": "Born in California"
				}
			],
		}
	]
}

The second people item is missing a phone number. JSONPath Parser creates an array of values for each field, and the array for phone number ends up like this:

array(
	0 => "111-1111",
	1 => "333-3333"
)

Our field collection hooks read in that array and applies the first phone value to the first field collection item it saves, and the second phone value to the second item. Thus, you get a saved field collection where Bo from Bombay has a phone number of 333-3333. Corrupt data.

So, given all this, it's out of scope (and the realm of possibility) for the field collection feeds code to fix this. It has to be fixed at the parser level. Fortunately, JSONPath Parser has a hook that will let me fix the values specific to my project (I can share this code if it would be helpful).

If you're using the CSV parser, an XML parser, etc., you may or may not have this same problem. Not sure. But it has to be corrected at the parser level.

majorrobot’s picture

I've continued working on this patch in an attempt to get nested Field Collections to import. I've made tiny, incremental progress, but the recursion of the code in field_collections_feeds_set_target() is driving me bonkers. Ideally the function would work for both parent field collection items and child field_collection_items.

Has anyone else attempted this? Or have any ideas?

atiras’s picture

The patch as #273 works for me -- sort of. I can confirm HTML XPATH Parser has the same issue as described in https://www.drupal.org/node/1063434#comment-12066933

I'm not a coder so can't help update the patch but will gladly test new iterations.

atiras’s picture

Unfortunately I've had to back out the patch at #273 -- it breaks the importation of images with Fields (multi-value image field, image file name and image title parsed/tampered and exploded into the relevant arrays.) The file records are created but without titles and not linked to the right node.

Backing out the patch restores the correct functionality.

I've verified that the problem isn't caused by using the latest dev version of Fields Collection.

atiras’s picture

To elaborate on #278, I import a set of image files (URI plus title) into a node type; the URI and Title are parsed out of an HTML stream and then (after a few other Feeds Tampers) are exploded into a multivalue image field in each node.

Without patch from #273, this works fine. With the patch, it breaks -- the image records are created (without a title but with the correct image) but they aren't linked to the relevant node.

There are no field collections on the node type, so I'm not attempting to import any.

I'd wor backward through the patchers to try and find one that works for my use case, but it's impossible AFAIK to tell which patch was rolled against which verision of Feeds/Field Collection -- I'm running Feeds 7.x-2.0-beta3 and Field Collection 7.x-1.0-beta12 (but can update that one to dev if necessary -- could probably also update Feeds to dev.

Can anyone suggest which patch to try -- I need to import mutivalued field collections into a node, but each field collection doesn't contain multivalued items. #262 looked like a good possibility but it fails to apply.

atiras’s picture

On initial testing the patch in #262 (applied manually to 7.x-1.0-beta12) *doesn't break* the image importing -- I'll keep testing.

atiras’s picture

#262 is working pretty well as long as I ensure that all fields in the collection are populated.

m.stenta’s picture

Issue summary: View changes

Comment #175 accidentally cleared the original issue description (https://www.drupal.org/node/1063434/revisions/view/8887965/8894301), and then comment #200 set it to the same value as the comment (https://www.drupal.org/node/1063434/revisions/view/9504473/9535453). Restoring the description to what it was before #175...

m.stenta’s picture

Thanks @majorrobot for the patches - I tested #273 and it does work to create mapping targets for my field collection fields - and the import works.

Notably, I am using a CSV parser, and I do not have multi-value field collections, so I don't think I'm susceptible to the bug @majorrobot mentioned in #273.1.

However, the patch seems to have broken my entityreference mappings. I will do some debugging and see if I can figure out why... it may be related to what @atiras mentioned in #278/#279.

m.stenta’s picture

Ok, so the reason that entityreference mappings are broken with patch #273 (and probably image mappings as @atiras described) is because of a bad assumption in field_collection_feeds_presave():

function field_collection_feeds_presave(FeedsSource $source, $entity, $item, $entity_id) {
  $mappings = $source->importer()->getConfig()['processor']['config']['mappings'];

  // Only hook in when updating the entity.
  if (!is_null($entity_id)) {

    // Go through $mappings and find the FC fields.
    foreach ($mappings as $mapping) {

      if (strpos($mapping['target'], ':') !== FALSE) {
        $parts = explode(':', $mapping['target']);
        $fc_name = array_shift($parts);

...

That last if condition is making the assumption that any mapping target that contains a colon (:) is a field collection. Entity Reference fields also use colons, and so do image fields. A lot of others probably do as well.

So we'll have to find a better way to check if the target is a field collection in that function...

m.stenta’s picture

Hmm also: this part is not working as intended...

  // Only hook in when updating the entity.
  if (!is_null($entity_id)) {

On new entities, $entity_id is 0, which is not null in PHP, so the code in that block is always running - even on new entities.

Should be this instead:

  // Only hook in when updating the entity.
  if (empty($entity_id)) {
m.stenta’s picture

Attached is a patch that fixes the two issues I described in my comments #284 and #285.

@atiras - Can you test this to see if it resolves your image import issue?

To check if the mapping is a field_collection field, I load the field info using field_info_field() and check the type.

I'm leaving this as "Needs Work" because it sounds like there are other issues that need to be fixed.

atiras’s picture

@m.stenta -- sorry, been away for a while. Will test the new patch today.

atiras’s picture

OK, patch in #286 works to link the imported images. Thanks.

Should it fix the import to field collections when not all fields are populated (I'm thinking not, but haven't tested it yet and won't if I'll be wasting my time).

m.stenta’s picture

Should it fix the import to field collections when not all fields are populated (I'm thinking not, but haven't tested it yet and won't if I'll be wasting my time).

I'm not sure I know what you're referring to. Can you elaborate a bit?

giupenni’s picture

#286 seems works for me

atiras’s picture

Should it fix the import to field collections when not all fields are populated (I'm thinking not, but haven't tested it yet and won't if I'll be wasting my time).
I'm not sure I know what you're referring to. Can you elaborate a bit?

Sorry, I was thinking about another attempted solution to this issue, where you had to ensure that all fields in your field collection were populated to get the import to work correctly -- if you had an unpopulated field, the data got out of step during the import.

This seems to work fine.

m.stenta’s picture

Another issue that I'm experiencing is that field validation is not being run for fields within a field collection. Field validation was added to Feeds in the dev branch via #2379631: field_attach_validate() must be called before programmatic entity saves.

I have that patch applied to my Feeds module, along with the patch from #286 above for Field Collection. Field validation appears to be happening for fields that are attached directly to the entities I'm importing, but it does not work for fields attached to my field collection.

I have two "List (text)" fields (using the "Select list" widget), each with a set of available values to choose from. If I try to import an entity with an invalid value, I get a validation error for the field that is directly attached to the entity, but not for the field attached to the field collection.

SEVENTH7’s picture

Added patch but can't see new field in mapping? What is the extra functionality of this patch? We need to connect the field collections true the feed importer mapping system.

Siavash’s picture

#286 worked great in my case. Thanks!

rcodina’s picture

Status: Needs work » Reviewed & tested by the community

Patch on #286 works for me too. Many thanks! Maybe we can handle field collection field validation on a separate issue (see comment #292).

The last submitted patch, 261: add_feeds_integration-1063434-261.patch, failed testing. View results

The last submitted patch, 273: add_feeds_integration-1063434-273.patch, failed testing. View results

elpino’s picture

#286 worked for me too.

This should be ported.

SkyBuzz’s picture

#286 work for too with 7.x-1.0-beta12

sportch’s picture

Any chance of an update for 7.x-1.0-beta13 ?

  • ram4nd committed ead6026 on 7.x-1.x authored by m.stenta
    Issue #1063434 by sherakama, majorrobot, paulihuhtiniemi, btopro, GaëlG...
ram4nd’s picture

Status: Reviewed & tested by the community » Fixed
Issue tags: -Field collection
philltran’s picture

Here is a patch for 7.x-1.0-beta13.

EDIT Looks like this has been committed to DEV while I was making the patch, so it probably will be in beta 14.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

alison’s picture

#303 works great for us on beta13!

@philltran are you saying that #261 was committed to dev, so we should use that?
EDIT: never mind, now i see that #261 doesn't apply to beta 13 -- which I'm sure is why @philltran created the patch at #303 :)

EDIT 2: Just, never mind...... (#303 does totally work on beta13, but it doesn't matter anymore :) sorry!)

robdubparker’s picture

Category: Task » Bug report
FileSize
1.07 KB

Noticed that boolean fields still don't work correctly with field collections and imports. I've attached a patch that seems to help fix the issue.

robdubparker’s picture

Version: 7.x-1.x-dev » 7.x-1.0-beta13
FileSize
9.93 KB

rerolled patch for 7.x-1.0-beta13

alrueden’s picture

This update still has a major problem with recursion. We don't even use Field Collection together with Feeds. Nevertheless, this caused a total failure for all the feed importers on our site.

After updating to the latest dev version (7.x-1.0-beta13+48-dev), our feeds stopped importing on cron. Attempting to edit the feed nodes resulted in a WSOD.

Server logs when attempting to edit a feed node or run the feeds importer on cron gave me this:

PHP Fatal error: Aborting! The New Relic imposed maximum PHP function nesting level of '5000' has been reached. This limit is to prevent the PHP execution from catastrophically running out of C-stack frames. If you think this limit is too small, adjust the value of the setting newrelic.special.max_nesting_level in the newrelic.ini file, and restart php. Please file a ticket at https://support.newrelic.com if you need further assistance. in /sites/all/modules/entity/includes/entity.wrapper.inc on line 319

According to Acquia, this error is generally caused by an infinite recursion bug.

I removed the chunk of code relating to feeds integration (everything in commit ead602643387c4bca10d82fac1dfc17d425a241d) and now everything works again.