When exporting a node, field_collection data is not exported.

This happens because the data for each field_collection is a separate entity. Node export references that entity using UUID, but there's no way to actually export those entities...

If node export could be made to work with entities instead of nodes... now that would be a major improvement. :) Failing that, we really could use support for field_collection export.

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

danielb’s picture

Priority: Major » Normal
Status: Active » Postponed

Node export dependency will maintain the relationship to the correct field collection entity (supposedly - I haven't actually tried it - but the code that plugs in to support field collections is there). The only thing is; you'd have to find your own way to copy the field collection entity to the new site.

See, this is node export, and that is the only entity this module actually exports. Except for files, which it supported before they became entities. It would be good if it was an all-round entity export though, but it would require a considerable redesign of the module. And once we cross into that territory it will become a nightmare to maintain.

The problem so far with Node export is that it's not simply a matter of coming up with a way to convert a node object into some code and then converting it back upon import. There are all kinds of hacks and shit to support all the little fiddly functionalities that are common in nodes. These are things like menus, publishing options, and dependencies. Then there are considerations of where this module provides it's functionality, like on the node page, content admin page, VBO, etc... How to identify a node or collection of nodes, e..g through features or drush. You have to understand everything about a node, specifically, to be able to handle it. There is no standard list of these functionalities or any automatic way to get this information. The only way you find out about this stuff is people making issues when they encounter a limitation. Because of this, handling just one entity type, the node, has become a monstrous task.

Preparing an entity for export (and handling the import) is really something that should be part of Drupal core so everyone that makes an entity type is aware that it may be exported/imported and that there are standard things to consider in that regard. I'm not saying Drupal core needs to provide the actual export mechanism or any interfaces - just some standard way of saying "get me the exportable version of this" and then to be able to plug that object back in and have it saved properly.

The entity API has got us part way there, but it seems when designing the entity system nobody really thought about the full scope of how entities would be used. This includes export/import of the entities, and even obvious things like a consistent way to search for, represent, reference, and link to entities - these are challenges I face in another of my modules.

If I support field collection, a contrib module, I may as well support exporting users and taxonomy - which are core entities and probably more relevant to a larger audience. Don't get me wrong, I'd love to do that - I just don't have a plan for it right now. I guess the obvious thing would be to take everything 'node' related out of the module, and add support for nodes as a plugin - to open the door for other entities to be handled. The problem is, a bulk of this module is in fact node specific.

Right now I'm not far off a full release of Node Export in Drupal 7, and I'd like to just fix up the few remaining issues related to that, and then consider what a future version of this module should look like. It would probably be a tighter relationship with modules that deal with entities in a broader sense.

danielb’s picture

Oh I could be overestimating the challenge multiple entities will cause, for all I know Nodes could be the only entity with such complexity. I suppose I could play around with it and see what happens.

danielb’s picture

I can't even get Field Collection to work with UUID. You mentioned you had it working - weird...

danielb’s picture

Status: Postponed » Closed (won't fix)

long story short: Field collection is not supported for now.

bcweaver’s picture

FWIW, the project page for Node Export contains this text which seems a bit confusing

Node export dependency (Drupal 7)
Supports: authors, translated content, books, organic groups, taxonomy term reference, file, image, node reference, user reference, entity reference, and field collection.

I couldn't find any modules called "node export dependency".

danielb’s picture

Ensure you have the latest version.

Anybody’s picture

It seems that there is still a problem with the order of uuid and field collection (views) installation. In one installation where I installed field collection first, everything is OK. In another where I first installed uuid for example the ID field inside views is missing for field collection views!

Any Ideas?

shenzhuxi’s picture

Status: Closed (won't fix) » Needs review
Issue tags: +UUID, +field_collection
FileSize
2.25 KB

The dev version of uuid module already supported Field collection.

I patch node_export_dependency to add field_collection export/import. File field, reference fields, field_collection fields and etc. inside field_collection_item are not supported yet.

I think File export mode "Inline Base64" should be separated (maybe put into node_export_dependency) so it can be re-used not only for field_collection_item but also other entities.

Ronino’s picture

shenzhuxi, I tried your patch, thank you. Export works fine, but when trying to import my nodes with field collections with file fields, I get the following errors:

Notice: Trying to get property of non-object in entity_revision_is_default() (line 323 of .../sites/all/modules/contrib/entity/entity.module).
Notice: Trying to get property of non-object in entity_revision_is_default() (line 322 of .../sites/all/modules/contrib/entity/entity.module).
Notice: Trying to get property of non-object in file_field_presave() (line 220 of .../modules/file/file.field.inc).
Strict warning: Creating default object from empty value in file_field_presave() (line 221 of .../modules/file/file.field.inc).
Notice: Undefined property: stdClass::$uri in file_save() (line 570 of .../includes/file.inc).
PDOException: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '' for key 'uri': INSERT INTO {file_managed} (filesize, status, timestamp, uuid) VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1, :db_insert_placeholder_2, :db_insert_placeholder_3); Array ( [:db_insert_placeholder_0] => 0 [:db_insert_placeholder_1] => 1 [:db_insert_placeholder_2] => 1361915972 [:db_insert_placeholder_3] => 0dcb9144-e3e2-15d4-c9ed-669156a3d690 ) in drupal_write_record() (line 7106 of .../includes/common.inc).

It obviously works for you with field collections without file fields?

shenzhuxi’s picture

Ronino, do you have file field in your field collection? It's not supported yet.

Ronino’s picture

do you have file field in your field collection? It's not supported yet

Yes, I do. But even if it's not supported, it should at least ignore them. Otherwise I can't use the export for any content type that has a file field in a field collection even if it's okay if the file data is left out.

shenzhuxi’s picture

@Ronino In this patch $field_collection->save() was used and field_collection module doesn't handle existing file entity during saving. I think we should patch field_collection module to fix that.

Ronino’s picture

shenzhuxi, I took your patch as a basis to create a new one which works for me.

Changes:

  • Clear-text export like other node_export data, no base64-encoded serialized objects.
  • The field collection items to import are added to $node and thus saved by node_save() (or some function called from it) automatically. This adds the data to the system as if saving it via the node edit form, correct import of new data and update of existing data works.
  • file/image fields are ignored during import for now (looks like node_export_file_field_export() and node_export_file_field_import() will be our friends when implementing it).

For me, it works for multi-value field collection items. It should work for multi-value fields inside the field collection itself, I doubt that it works with field collections containing field collections.

In this patch $field_collection->save() was used and field_collection module doesn't handle existing file entity during saving. I think we should patch field_collection module to fix that.

I first tried for quite a long time with FieldCollectionItemEntity::save() and EntityAPIController::save() which led to duplicated field collection item data. Then I found out that it's only necessary to add the data to the $node object. I don't think that patching field_collection is necessary.

brettbirschbach’s picture

The patch in #13 worked great for me, except for the file fields (which Ronino mentions as unsupported). In fact, there is also a bug in the workaround code that unsets the file field values in that it fails to unset file fields in data instances other than the first one when there are multiple data instances (i.e. multiple instances of the field collection with 1 file, not 1 instance of the field collection with multiple files).

At any rate, I wrote a quick patch that is meant to be IN ADDITION TO #13. #13 must be applied first.

This patch is meant to support the import of file fields. I basically stole the code from the node_export module and duplicated it here. Note that my patch also includes some lines of code from another file field patch @ http://drupal.org/node/1868450

jason.fisher’s picture

I am getting "corrupt patch" while using git apply on #14.

brettbirschbach’s picture

Sorry for that...might have had to do with the fact that I added some extra comments and such into my code after applying #13 and before creating #14.

I reverted my node_export module, and the cleanly applied patch #13 and my changes from #14, and re-rolled into a single patch. Please give this one a try.

Rhodungeon’s picture

Patch #16 works perfectly.

To be committed!

kytom’s picture

This might be a newbie question, but applying #16 gives me this:

$ patch -b -p0 < node_export-field_collection-1670740-16.patch
patching file modules/node_export_dependency/node_export_dependency.core.inc
patching file modules/node_export_dependency/node_export_dependency.module
patch unexpectedly ends in middle of line
Hunk #2 succeeded at 284 with fuzz 1.

Am I doing something wrong?

Other than that, importing a single node with field_collection threw an error:

PDOException: in field_sql_storage_field_storage_write() (line 451 of /.../modules/field/modules/field_sql_storage/field_sql_storage.module).

I should note that the target site has some nodes added, containing field_collections of different type.

ekes’s picture

Re-roll of #16 with git diff against head of 7.x-3.x Working nicely here, with one type of field collection. Anyone repeated #18?

Ronino’s picture

Hmm, I applied the patch from #19 and I don't get file data exported. node_export_dependency_handle_dependency() seems to import files via _node_export_file_field_import_file(), but I don't see any code in field_collection_node_export_dependency_field() to export file data.

Am I missing something?

Ronino’s picture

HitmanInWis wrote in #14:

This patch is meant to support the import of file fields.

Oookay. But how can we import files if they aren't exported?

I think we need to rework node_export_file_field_export() to handle field_collection items.

brettbirschbach’s picture

Maybe check if the patch applied correctly. I haven't played with this in awhile, but I know that field collection export/import was working for me as I used it for a production deploy.

Also, I havent really compared patch #19 to #16 - might be worth checking if something got missed.

Lastly, its possible that something changed in the node_export module that makes this patch no longer work. Perhaps something you can look into?

Ronino’s picture

HitmanInWis wrote:

Maybe check if the patch applied correctly. I haven't played with this in awhile, but I know that field collection export/import was working for me as I used it for a production deploy.

Did you really ever export files in field collections from one system and import in another one?

I mean, field_collection_node_export_dependency_field() in any of the patches does not export the real file data, only meta data like uri and filename, but not the file data itself. There is no e.g. base64 encoded binary data in the export files, so there is nothing to import.

brettbirschbach’s picture

Ahh...sorry...you mean the actual files are not being exported. I manually put the files into a folder, and imported them that way... Not ideal, but it was sufficient for my situation. So ya, I think there is probably more work to be done in that area.

Ronino’s picture

HitmanInWis wrote:

Ahh...sorry...you mean the actual files are not being exported.

Yes ;-).

I manually put the files into a folder, and imported them that way... Not ideal, but it was sufficient for my situation.

But sounds like a useful workaround. Thanks for the tip.

So ya, I think there is probably more work to be done in that area.

Definitely. Any volunteers?

vzblk’s picture

I'm trying to use patch#19 to export collection field with entity relation field (node) in this collection and failed. It is not enough information in exported data and i don't see code that can handle this information. So it is necessary to export UUID for referenced node and handle this during importing. I think the same is for other relation (user and taxonomy). I'm trying to make it work. And potentially we should look at field_collection_deploy module and try to use it.

ekes’s picture

Just for clarity. We're using #19 with references to entities that have been exported & are then imported additionally. The problem was that the field collection itself wasn't exported, the patch does that, with UUID in. It doesn't seem to matter if the referenced entity is imported after the field collection either, the UUID still match up once it's there.

mpv’s picture

#19 worked for me, thank you!

Honza Pobořil’s picture

It seems there is module Field Collection Deploy to export collections values.

loziju’s picture

#19 doesn't work for me when I have term reference field in field collection. UUID of the term is not showing anywhere (contrary to what ekes mentioned in #27).

caspervoogt’s picture

Tried #19 but no field collection data was included in the export..

lil123’s picture

Hi,

Image fields in field collections where not imported. With this it seems to work :

function field_collection_node_export_dependency_field($entity_type, $entity, $field, $instance, $langcode, $items) {
  $dependencies = array();
  node_export_dependency_add($dependencies, $items, 'field_collection_item', 'value');
  
  //PATCH (see https://drupal.org/comment/7647431#comment-7647431 )
  // Loop all items to add their data for export.
  foreach($items as $index => $item) {
		if ($field_collection_item = field_collection_item_load($item['value'])) {
			// PATCH TO ADD THE FILE URL
			$export_mode = variable_get('node_export_file_mode', 'inline');
			if ($export_mode == 'remote') {
				$field_collection_item_entity_array = (array)$field_collection_item;
				//print_r( $field_collection_item_entity_array );exit();
				foreach( $field_collection_item_entity_array as $field_name => $value ) {
					$field_info = field_info_field( $field_name );
					if( !$field_info || !($field_info['type'] == 'image') ) {
						continue;
					}
					// the field_collection's field is an image, add the full path
					$uri = $field_collection_item->{$field_name}[LANGUAGE_NONE][0]['uri'];
					$export_data = file_create_url( $uri );
					if( !(substr($export_data, 0, 4) === "http") ) {
						global $base_url;
						$export_data = $base_url . $export_data;
					}
					$field_collection_item->{$field_name}[LANGUAGE_NONE][0]['node_export_file_url'] = $export_data;
				}	      
	    }
			// END PATCH
			
			// Add the field collection item data to the exported data. Mimics
			// EntityAPIController::export() without the JSON conversion.
			$dependencies[$index]['node_export_field_collection_data']
			 = get_object_vars($field_collection_item);
		}
	}
	// END PATCH
  
  return $dependencies;
}
onigoetz’s picture

Hi @lil123,

Your code seems to work, but with a little modification.
the "fid" has to be reset to be able to be imported or Drupal will use the provided fid, even if it doesn't exist.

this would give us :

<?php
/**
 * Implements hook_node_export_dependency_field().
 */
function field_collection_node_export_dependency_field($entity_type, $entity, $field, $instance, $langcode, $items) {
  $dependencies = array();
  node_export_dependency_add($dependencies, $items, 'field_collection_item', 'value');
  
  //PATCH (see https://drupal.org/comment/7647431#comment-7647431 )
  // Loop all items to add their data for export.
  foreach($items as $index => $item) {
    if ($field_collection_item = field_collection_item_load($item['value'])) {
      // PATCH TO ADD THE FILE URL
      $export_mode = variable_get('node_export_file_mode', 'inline');
      if ($export_mode == 'remote') {
        $field_collection_item_entity_array = (array)$field_collection_item;

        foreach( $field_collection_item_entity_array as $field_name => $value ) {
          $field_info = field_info_field( $field_name );
          if( !$field_info || !($field_info['type'] == 'image') ) {
            continue;
          }
          // the field_collection's field is an image, add the full path
          $uri = $field_collection_item->{$field_name}[LANGUAGE_NONE][0]['uri'];
          $export_data = file_create_url( $uri );
          if( !(substr($export_data, 0, 4) === "http") ) {
            global $base_url;
            $export_data = $base_url . $export_data;
          }
		  
		  // Remove fid to be sure that imported node will rebuild the
          // file again, or keep an existing one with a different fid
    	  $field_collection_item->{$field_name}[LANGUAGE_NONE][0]['fid'] = NULL;
          $field_collection_item->{$field_name}[LANGUAGE_NONE][0]['node_export_file_url'] = $export_data;

        }
      }
      // END PATCH
      // Add the field collection item data to the exported data. Mimics
      // EntityAPIController::export() without the JSON conversion.
      $dependencies[$index]['node_export_field_collection_data'] = get_object_vars($field_collection_item);
    }
  }
  // END PATCH
  
  return $dependencies;
}
?>
jrb’s picture

After applying the patch in #19, it still wasn't correctly creating the field collections for me.

In my case, I had a field collection called "Location" (field_location) that included fields for an address (field_address) and a phone number (field_phone). After import, I was seeing new data in field_data_field_address, field_data_field_phone, and field_data_field_location; but the row in field_collection_item wasn't being created.

The problem was that the SQL in the patch was only checking if the collection existed in the field that was the field collection (field_location, in my case). SQL in patch:

  // Find the field collection item's current ID if it already exists.
  $item_id = db_query('
    SELECT ' . $dependency['field_name'] . '_value
    FROM {field_data_' . $dependency['field_name'] . '}
    WHERE entity_id = :nid AND delta = :delta',
   array(':nid' => $nid, ':delta' => $dependency['delta']))->fetchField();

Since it found data in field_location, it never created the full field_collection entity. I've attached an updated patch that just changes the SQL to add a join to the field_collection_item table as well. This makes sure that the field collection entity is fully created. New SQL:

  // Find the field collection item's current ID if it already exists.
  $item_id = db_query('
    SELECT d.' . $dependency['field_name'] . '_value
    FROM {field_data_' . $dependency['field_name'] . '} d
    INNER JOIN {field_collection_item} ci ON
      ci.item_id = d.' . $dependency['field_name'] . '_value
      AND ci.revision_id = d.' . $dependency['field_name'] . '_revision_id
    WHERE d.entity_id = :nid AND d.delta = :delta
    AND ci.field_name = :field_name',
    array(':nid' => $nid, ':delta' => $dependency['delta'], ':field_name' =>  $dependency['field_name']))->fetchField();

Now, I had also patched the Node Export module to make it work correctly with Organic Groups #2140075: Receiving EntityMalformedException: Missing bundle property on entity of type node when importing nodes with Organic Group dependencies, so it's possible that my problem may be related to how the patch in #19 works with that change. But, I think my new patch should work regardless of whether or not that patch is applied and shouldn't cause any additional issues.

jamsilver’s picture

That worked great except that it didn't support nested field collection fields.

I've just added a single extra line to add additional additional dependencies.

jrb’s picture

#35 made it export the nested fields, but the import didn't work correctly.

I have a field collection with an entity reference field. It wanted to add the entity reference field to the node rather than to the field collection.

Ronino’s picture

I took the patch from #35 and added the functionality to really export and import file data. That means in exports you have the file contents of field_collection fields hashed in the field 'node_export_file_data' like in non-field_collection fields.

I did this by adapting node_export_file_field_export() and node_export_file_field_import() to work with nodes and field_collection items. The changes suggested in #32 and #33 are not necessary anymore as that works automatically with this patch (apart from the fact that those changes only added the file URI, not the real data to the export).

This is the first time for me that exporting nodes with files in field_collection fields on one system and importing them on a different system fully works.

Ronino’s picture

Ronino’s picture

My patch from #37 contained the code of another patch ;-). Here is a working version...

zrhoffman’s picture

Are there any updates on this? I would find this bugfix extremely useful.

applegamuza’s picture

hopefully this module can be updated with this bug..

john.oltman’s picture

rcross’s picture

I see from the project page that this module supposedly supports both entity reference and field collection. Is this issue still relevant? Does it support including the referenced data in the export, or just the actual reference id?

dragon658’s picture

I have applyed #39 patch.
Field collection data was successfully exported to file.
But when I tried to import it to another site, images and body field of field collection were not exported.

Also I suspect that this solution does not support nested field collections.

I am going to write my own module for sync sites because of it. (field collection deploy module also does not work)
Did not find any good drupal module for syncing content.

rcross’s picture

@dragon658 - I think you misunderstand the point of this module and this issue.

You won't be able to import content across sites if the node types (and fields on that node) are the same. Look into the features module to managing changes of content types (i.e. node types) across sites.

This issue is about exporting of field collections. As the underlying structure of field collections are actually entities, and thus the node just holds a reference to that entity, then a basic node export would simply include the reference rather than the values. This issue is about how getting the values exported, as that is the expected/desired behaviour in most cases.

As for a more general solution to content synching, you would need to dig around more. Most cases, people are actually talking about content staging. (i.e. dev>stage>prod) rather than an arbitrary content synch. There are ways of achieving both, though from experience they all are a bit fragile and depend alot on the specific implementation. I would start looking here https://www.drupal.org/node/980186 but also check out https://groups.drupal.org/large-scale-drupal-lsd-projects-and-plans/cont... http://www.youtube.com/watch?v=akcK5DVaqA8 http://london2011.drupal.org/conference/sessions/content-staging-and-dep... (there is probably a video floating around online, but check out the slides) https://www.drupal.org/project/deploy https://www.drupal.org/project/content_staging

dragon658’s picture

Hello, rcross!
Thank you for your answer!

I have read articles you provide above. (long time ago)
I have tried Deploy module and lots of other drupal modules for content syncing.
Existing modules are very laggy and does not allow to properly sync difficult content types. (unfortunatelly!)

My two sites are syncing by remote script every day.
So uuid's of content on both sites are the same.
I tried to use node export to sinc single nodes between sites on demand.
It is working good.
But it does not work good with field collections.

It does not support nested field collections and does not import field collection content to new site.

smndey’s picture

https://www.drupal.org/node/1670740#comment-7061168 #8 Works fine for field_collection_item but doesn't provide support for files. #8 breaks the file export.

  • danielb committed a7ae133 on 7.x-3.x authored by Ronino
    Issue #1670740 by Ronino, HitmanInWis, shenzhuxi, ekes, jrb, jamsilver:...
danielb’s picture

Status: Needs review » Closed (fixed)
mibstar’s picture

Hi danielb,

I've reviewed your commit and while it does make reference to nested item collections I've not been able to import an example I've been working on either via features nor node-import.

The node gets created but none of the item are present. I've attached a screenshot of the source and destination. Does this take into account multiple levels of nesting?

mibstar’s picture

It seems like I can only attach one file at the time so here is the screenshot of the resulting import.

inversed’s picture

To any that may still be having trouble, make sure to enable the "Node export dependency (experimental)" module.