Field collections deriver and base migration

This is the last of the patches, and this one needs to be applied over https://www.drupal.org/project/paragraphs/issues/2911243 and https://www.drupal.org/project/paragraphs/issues/2911242. It still probably won't handle translations, and this patch won't pass tests without the other two, so I'm just leaving it here for review. The patch is included in #2897021: [META] Migrate support for importing field collections as paragraphs

I'm going to close this issue and roll the patch directly into the parent issue patch. I can't apply it on top of dev, and it's just getting impossible to manage as a separate issue.

Resetting this issue to prepare for more work. Currently blocked by #2911242 and #2911243

Adding in initial work. This patch is based on some previous work I did and still needs some love.

Hi @mikelutz,

I'm still stuck with this issue Input should be an array. lurking once I started migrating the node/content type, this is using the latest patch. Apologies, but this issue is really a PITA. I have tried different approaches even to the point of testing early versions of drupal 8.4 and I'm still stuck with the same issue. Already even tried using a different migrate-upgrade module an added the upgrade_ prefix manually still the same, that is how desperate I am in trying to resolve this issue. Even tried creating a new/fresh drupal 8 instance and following the steps and procedures in the following order.

- upgrade_d7_field_collection_type
- upgrade_d7_node_type
- upgrade_d7_comment_type
- upgrade_d7_taxonomy_vocabulary
- upgrade_d7_user_role
- upgrade_d7_user
- upgrade_d7_field
- upgrade_d7_field_instance
- upgrade_d7_view_modes
- upgrade_d7_field_instance_widget_settings
- upgrade_d7_field_formatter_settings
- upgrade_d7_field_collection_authors
- upgrade_d7_field_collection_revisions_authors
- upgrade_d7_node_press_release

Some migration requirements/dependencies that I need to migrate in order to resolve some of the migration errors

- upgrade_d7_node_type
- upgrade_d7_comment_type
- upgrade_d7_user_role
- upgrade_d7_user

Missing bundle entity, entity type comment_type, entity id comment_node_page.

YML Configuration

uuid: c44d4157-ce16-40a8-972c-2920ac098551
langcode: en
status: true
dependencies: {  }
id: upgrade_d7_node_press_release
class: Drupal\migrate\Plugin\Migration
field_plugin_method: null
cck_plugin_method: null
migration_tags:
  - 'Drupal 7'
migration_group: migrate_drupal_7
label: 'Nodes (Press Release)'
source:
  plugin: d7_node
  node_type: press_release
process:
  nid: tnid
  vid: vid
  langcode:
    plugin: default_value
    source: language
    default_value: und
  title: title
  uid: node_uid
  status: status
  created: created
  changed: changed
  promote: promote
  sticky: sticky
  revision_uid: revision_uid
  revision_log: log
  revision_timestamp: timestamp
  comment_node_press_release/0/status: comment
  body:
    plugin: get
    source: body
  field_authors:
    plugin: sub_process
    source: field_authors
    process:
      target_id:
        -
          plugin: migration_lookup
          migration: upgrade_d7_field_collection
          source: value
        -
          plugin: extract
          index:
            - 0
      target_revision_id:
        -
          plugin: migration_lookup
          migration:
            - upgrade_d7_field_collection_revisions
            - upgrade_d7_field_collection
          source_ids:
            upgrade_d7_field_collection_revisions:
              - revision_id
            upgrade_d7_field_collection:
              - value
          source: value
        -
          plugin: extract
          index:
            - 1
destination:
  plugin: 'entity:node'
  default_bundle: press_release
migration_dependencies:
  required:
    - upgrade_d7_user
    - upgrade_d7_node_type
  optional:
    - upgrade_d7_field_instance
    - upgrade_d7_comment_field_instance

I'm open to anything to try and resolve this issue.

@noel - I know you've been struggling with this, and I wish I had an idea on how to help you. I can tell you this much: "Input should be an array" is an error that occurs in the extract plugin, when the input is not an array (ok, duh..). The extract plugin takes in an array and returns the value at a specific index. If you are getting that error, then the previous migrate_lookup plugin is clearly not returning an array. This plugin returns the destination id(s) of a previous migration, and if your field collection migrations use the paragraphs destination (as in the latest patch), then that extends the entity_revision_reference destination, which has entity_id and revision_id as destination ids, so migrate_lookup should return an array, so extract should be happy.

Now if the migration lookup fails, the plugin attempts to create a stub paragraph, and thanks to a flaw in the system, rather than getting an array back, it just gets back the entity_id of the new paragraph. I'm aware of this, and working on a couple issues/workarounds to help resolve it, but at the moment it should only be a problem for nested field_collections/paragraphs. In your simple case, and with your field collection migrations running prior to the node migration, you should not be creating stub entities; your migration should succeed and you should get an array back for extract.

With the exception of nested field collections, I can't duplicate the error. If you could xdebug break at the line in the extract plugin that throws the error, and show the state of the migration, and other variables at that point it may help. If we could track down which of the two extract plugins is causing the problem, that might help. Maybe try getting rid of the target_id block and then switching out the target_revision_id block to see if the error message changes.

Finally, if you can catch me in slack (I know there's a time difference) I would be happy to try to talk through some debug stuff live. I hang out in the migration channel among others.

@mikelutz

Hello mike, thanks for sharing your inputs. Sent you message.

Just to add, here is the YML configuration for my field collection

uuid: 2969679c-b90a-4d99-9d48-6b27eb2c8836
langcode: en
status: true
dependencies: {  }
id: upgrade_d7_field_collection_authors
class: Drupal\migrate\Plugin\Migration
field_plugin_method: null
cck_plugin_method: null
migration_tags:
  - 'Drupal 7'
migration_group: migrate_drupal_7
label: 'Field Collections (Authors)'
source:
  plugin: d7_field_collection_item
  field_name: field_authors
process:
  type: bundle
  field_authors_name:
    plugin: get
    source: field_authors_name
  field_email:
    plugin: get
    source: field_email
destination:
  plugin: paragraphs
  default_bundle: authors
migration_dependencies:
  required:
    - upgrade_d7_field_collection_type
  optional:
    - upgrade_d7_field_instance

Totally understand that work still in progress. Just want to share my results with testing latest patch.

Still don't have any migrated node with at least one field_collection item attached. But I have exact number of paragaphs in database as I have FC items in D7 database. Tables with fields attached to paragraphs types contain no data though.

I'm using core Migrate UI module (8.5.*-dev) with dev version of Paragraphs and patch from #6.

Just a testing update from my end. I've been able to do a FC to Paragraph migration multiple times and with a code review from a coworker. We've pushed our code and run migrations on our dev server, just waiting for QA to verify the data is correct.

@mikelutz: It's now been 2 weeks so I'm just wondering if you've moved onto other things. I finally have some time to get a content migration done for one of my current projects, and ideally I'd have it working in the next half-week. The new site is already configured so I just need to migrate the data itself, nothing else. So I'm really wondering what's left for this.

If you have moved on, I was wondering about the following:

[#6] is based on some previous work I did and still needs some love.

As "love" isn't very specific, would you kindly let us know what you believe to be the outstanding bits?

#8:

Now if the migration lookup fails, the plugin attempts to create a stub paragraph, and thanks to a flaw in the system, rather than getting an array back, it just gets back the entity_id of the new paragraph. I'm aware of this, and working on a couple issues/workarounds to help resolve it, but at the moment it should only be a problem for nested field_collections/paragraphs.

Would you kindly point us to these issues/workarounds, or let us know what you're thinking so we can try to solve? I've got some nesting in my data, and I'm sure others do too.

If you haven't moved on, and are simply distracted, please let us know when you'd have a chance to get back to work on it. Any info you can add here would help the rest of us (possibly myself) finish it off (if you're too busy to do it yourself in the near future).

Thanks so much for getting it this far!

I've been trying to work with this stuff for the past week, trying to migrate data (only, everything else is already in place), but hitting roadblocks every step of the way, starting with #2956107: Database connection failures during migrations aren't logged anywhere. Some questions:

1. What difference does #2904765: Alter derived migration definition early in life make? Is the patch necessary for this this use case? For Drush only?
2. Do we still need src\Plugin\migrate\destination\Paragraphs.php if #2944836: Migrate destination plugin attempts to update new entities, preventing creation of stub entities is in? Does our code do anything different that's missing from ERR?

The @todo says:

Consider removing this and replacing with entity_reverence_revisions when #2944836: Migrate destination plugin attempts to update new entities, preventing creation of stub entities is resolved.

@noel.delacruz: I'm assuming #9 didn't work because this was missing? Please update us on your progress. Would be good to know where you're at, what worked, and what didn't so we can at least nail down and one working template for migrating a single field collection to a paragraph.

deriver: Drupal\paragraphs\Plugin\migrate\D7FieldCollectionItemDeriver

I'll keep hacking on this, and report back as I move forward. I'm now getting "No migrations found.", which is at least honest. Before fixing the DB connection issue, I was told that the migration was successful, and all I was getting was stub entities.

@colon It is a very long story, but I need #2904765: Alter derived migration definition early in life so that I can fix a bug with migrate_upgrade which causes migration_lookup to break. Basically the standard drupal migrate ui knows that when I tell it to do a migration_lookup from d7_field_collections I need it to look through ALL the derived D7 field migrations. When you convert these plugins to configs with migrate_upgrade and migrate_plus the derivative relationship breaks, and the plugin can't find all the field migrations to reference. It breaks taxonomy hierarchy as well, and a few other things. The workaround is to manually change the migration_lookup migration references in each content migration to directly reference the corresponding field migration.

#2890690: MigrateLookup plugin has inconsistent return values. breaks nested field collections, since I can't create and reference stubs.
#2944836: Migrate destination plugin attempts to update new entities, preventing creation of stub entities is in ERR now, so you could go back to using that destination plugin.

I haven't given up on this, or even been distracted really. All of those issues could potentially be worked around in this patch, btu it adds a lot of extra cruft, that I doubt would get committed, so I'm trying to fix the upstream issues before I do more work here.

Again, if you are working on a one-off migration, and don't mind hacking core a little or adjusting the migrate_upgrade configs, you can work around all of the content issues, I just don't have a clean committable way to code everything up yet.

@mikelutz: Thanks for your response.

I haven't given up on this, or even been distracted really. All of those issues could potentially be worked around in this patch, btu it adds a lot of extra cruft, that I doubt would get committed, so I'm trying to fix the upstream issues before I do more work here.

Totally understandable, and a good plan IMHO.

Again, if you are working on a one-off migration, and don't mind hacking core a little or adjusting the migrate_upgrade configs, you can work around all of the content issues, I just don't have a clean committable way to code everything up yet.

Understood. What would be good to have here though, until all of the upstream stuff is resolved, is a list of additional things we'd need to apply/do manually along with the patch here.

Here's what I think that list should be. Please let me know if I'm missing anything. Once we can agree on an initial list, we can add it to the issue summary (IS).

1. #2904765: Alter derived migration definition early in life
2. #2890690: MigrateLookup plugin has inconsistent return values.
3. #2954937: Object of class FieldCollectionItem could not be converted to int in FieldCollectionItem->fields()
4. The latest dev version of Entity Reference Revisions, the patch from #2944836: Migrate destination plugin attempts to update new entities, preventing creation of stub entities, or a release that contains it.

Anything else?

As far as #6 goes, let's say it definitely needs work (NW) based on the duplication of ERR code. We should also add this, and any other known NW items to the IS. So that's two lists really. Is there anything else that should be added to this one?

ERR duplication removed. The latest dev version of ERR is now required to get #2944836: Migrate destination plugin attempts to update new entities, preventing creation of stub entities.

The "entity_reference_revisions" plugin does not exist.

Not sure how that makes any sense. Even the stable version off ERR has the plugin, which is included in the test.

-  plugin: paragraphs
+  plugin: entity_reference_revisions

These would all need to specify the entity type derivitive

plugin: entity_reference_revisions:paragraph

Thanks guys. Hopefully this does the trick. Also fixed few coding standards violations the testbot was complaining about.

The ERR patch added a flag that was needed to save a new revision that wasn't necessary in the one-off plugin I wrote. Adding the flag should fix your test.

Okay, the test is passing locally, but failing on d.o because it's using the stable version of ERR, which does not have the commit needed to make this work. I'm adding a new patch here which adds a new lookup plugin and code to work around all the migrate_upgrade, migrate_plus, and migration_lookup issues and should support revisions, nested paragraphs/field_collections, and migrate-upgrade. The code is not cleaned up, this is meant more as a proof of concept and out for some manual testing. For a clean solution some of this code really needs to be refactored a bit and put upstream, but I wanted to get something working for people to use and test in the meantime.

I may put the destination plugin back in tomorrow, just to keep to passing tests and not being dependent on ERR-dev.

Anyway, I'm putting this up for review, not so much for a code review to be committed, because I don't think it should be, but for people to run some manual tests with migrate_upgrade and migrate_plus and let me know if it anything is seriously broken, and if not, I can start pushing the cleaned up versions upstream.

If you've already generated your migration configs with migrate upgrade, you will have to start again with this patch, delete them and re-generate.

This patch still does not handle translation, and I think that is fine for this issue queue. I'd like to get this code (once fixed up) committed first and then add translation support in another issue. This makes sense since we are still waiting on #2904705: Support asymmetric translation in experimental widget or something like it to manage translations and core migrate_drupal isn't stable for translations either.

I hope this helps move people forward on this issue. I know it seems like its been stagnant, but I'm actively working on two separate migration projects both heavily using field collections in D7 and needing paragraphs in D8. I'm pushing my code back as best I can, but for it to be right, I need things to happen in core, migrate_upgrade, migrate_plus and/or here, and I'm actively working to get the right code in the right places to do this right.

I pinged Miro on getting a new release on ERR, which should fix that test. I'd rather that than reintroduce the destination plugin, but we'll see what he says.

In the meantime, @colan, if you could try the latest patch with ERR-dev and let me know if it helps your nested field collections, I would like to hear how it goes.

Committed the Paragraphs issue. Is there anything that should go into ERR prior to a new release to unlock progress here?

I'm postponing an ERR release until THU to avoid potential challenges in the upcoming critical security issue.

About Asymmetric translation and related topics, see the priorities i outlined in the #2954487: New Roadmap
I think it's very important to care about Content Moderation and a clean storage first.

In the meantime, @colan, if you could try the latest patch with ERR-dev and let me know if it helps your nested field collections, I would like to hear how it goes.

So far, as has been happening earlier, the entities are getting created without field data. When I look at the entity before it's saved, I see that the entity's values array is populated, but SqlContentEntityStorage->saveToDedicatedTables() is skipping all of those because $this->entityManager->getFieldDefinitions() is only returning metadata field info. Maybe there's some field definition info we need to add to the entity? That's as far as I am with debugging at the moment.

I'm trying to do a single migration for the one nested field collection that I have. Once I can actually get field data, I'll add all of the other YAML files. I still have no idea how to grab the FC ID and use it for the Paragraph ID, which I'll need for other migrations, but that's my second problem. I don't even want to think about that until I can get field data to show up in a single migration. Here's my YAML.

# Migration for nested field_quality_issues field collections.
id: psia_ims_d7_nested_field_collection_field_quality_issues
label: PSIA IMS D7 Field Collection field_quality_issues Migration
migration_group: psia_ims_d7
deriver: Drupal\paragraphs\Plugin\migrate\D7FieldCollectionItemDeriver
source:
  plugin: d7_field_collection_item
  field_name: field_quality_issues
process:
  type: bundle
  field_qual_comments:
    plugin: get
    source: field_quality_iss_comments
  field_qual_completed_date:
    plugin: get
    source: field_quality_iss_completed_date
  field_qual_due_date:
    plugin: get
    source: field_quality_iss_due_date
  field_qual_start_date:
    plugin: get
    source: field_quality_iss_start_date
destination:
  plugin: entity_reference_revisions:paragraph
  default_bundle: quality
migration_dependencies:
  required:
    - psia_ims_d7_user
  optional: {  }
dependencies:
  enforced:
    module:
      - psia_ims
      - psia_ims_d7_migrations
      - migrate_drupal
      - migrate_tools
      - paragraphs

Any help to move this along would be greatly appreciated.

Maybe there's some field definition info we need to add to the entity?

That's not it. In the following code from EntityReferenceRevisions->getEntity():

    // Otherwise, create a new (possibly stub) entity.
    else {
      // Attempt to ensure we always have a bundle.
      if ($bundle = $this->getBundle($row)) {
        $row->setDestinationProperty($this->getKey('bundle'), $bundle);
      }

... $bundle gets set to "quality_issues". This should instead be "quality" as I have in my config.

destination:
  plugin: entity_reference_revisions:paragraph
  default_bundle: quality

"quality" is the target paragraph type, where I'd like the data to go, the one with the destination fields, so I have no idea where "quality_issues" is coming from (having trouble tracking this down; possibly somewhere in the source plugin strangely). So it can't find any fields on "quality_issues" because it doesn't exist.

When I hardcode $bundle to "quality", the field data actually comes through, and the migration works. Is this the correct way to specify the target bundle in the configuration? Am I doing it wrong? Or is there a bug in the code somewhere?

Actually, in that situation, you just need to remove the type: bundle line from your process array. The code is designed to work with the core migrate ui, and assumes you used the type and field migrations from the module. This maps the "field_quality_issues" field collection to a new 'quality_issues' paragraph bundle. your default_bundle in the destination isn't used because you specify the bundle from the source directly in your process array. For your custom migration, either remove it, or set it to a constant to override the bundle name derived from your field_collection.

Thank you! That worked. I just added the IDs like so, so I'm assuming I won't have to worry about look-ups later if I migrate dependent FCs first:

  id:
    plugin: get
    source: item_id

I saw this in both d7_field_collection_revisions and d7_paragraphs_revisions:

      index:
        -
          id

Should it instead be:

      index:
        - id

... as per the docs?

I updated the IS with everything we discussed above. I also updated the status as more review would be helpful before we start refactoring. For example, I haven't (and probably won't) be testing revisions.

Both are valid yml syntax, and there is no formatting standard for Drupal yml files beyond that they are valid yml syntax. Core configuration files format the arrays in both ways. We could change it for readability as a nit, but as everything will need to be refactored and moved around upstream projects, I'm more concerned with functionality as a proof of concept at the moment.

Not a problem then; I was just worried it would break something.

@miro_dietiker wrote:

I'm postponing an ERR release until THU to avoid potential challenges in the upcoming critical security issue.

Just wondering if you're still planning on cutting a release soon. It would be great to get this fix into stable.

Also, a Paragraphs release would be nice too.

@mikelutz: How can I get the paragraph ID from the parent ID? Can the paragraphs_lookup process plugin help with this? There's not much user/in-line documentation there, and it's rather long so it's tricky to figure out. (I'm sure this will be less of an issue once it gets refactored.)

I need to migrate some regular field data (not in a field collection) from a D7 node into a D8 paragraph that's a child of the node, where it doesn't contain the data in question. The destination paragraph would have already been migrated from the its corresponding FC, and the node would have been migrated as well. I've got the YAML file all written for this Node to Paragraph migration, except that I don't know how to get the ID of the previously-generated paragraph when adding additional data to it. In fact, the IDs are going to be the same on the source and destination nodes and FCs/paragraphs; not sure if that helps any.

I'm thinking of something like this:

process:
  id:
    plugin: paragraphs_lookup
    source: tnid
    type: parent

@colan It sounds like your node already contains a field collection, and your goal is to move some fields from the node into the new paragraph created from the field collection. The paragraphs_lookup is just the migration_lookup with a few workarounds for some migrate-upgrade and migration_lookup issues to help with nesting and derivative lookups. Assuming your original field collection field only allows a single delta, and you don't need to worry about multiples, you would be looking at running 3 migrations -

1:
Source: field_collection_item
field_name: your field
Destination: ERR:paragraphs

2:
Source: d7_node
node_type: your node_type
Process:
{{field maps}
id:
- plugin: migration_lookup
migration: {{migration #1 above}}
source: field_whatever/0/value
- plugin: extract
index:
- 0
revision_id:
- plugin: migration_lookup
migration: {{migration #1 above}}
source: field_whatever/0/value
- plugin: extract
index:
- 1
destination: ERR:paragraphs

3:
source d7_node
field_whatever: use the generated paragraphs_lookup plugins
destination: entity:node

You may need to play around with the source value for the id and revision id in the second migration, I'm not sure if you will need the delta off the top of my head. if your field only accepts a single value, field_whatever might be enough to get the lookup, I just don't know without checking the code.

You may need to tweak this if I'm misunderstanding your situation too, but the short of it is that you can't get the id from the parent id, but if you are using the parent as the source, then you should get the id from the field, or a migration lookup against the field.

If you catch me on slack, I can probably help more as well.

This patch no longer applies to the 8.x-1.3 release. Is it still needed or does it need a reroll?

This patch seems to still apply for me, but I've re-queued the test to check.

Hmm, odd, it works for me now too. Maybe it was a composer caching issue. Certainly had some other composer-related issues doing updates today. Thanks for getting back quickly.

A long Update method names to be more meaningful in MigrateFieldInterface, function processFieldValues() should be renamed to defineValueProcessPipeline().
Otherwise FieldCollection::processFieldValues() not calls for node migrations.

Update to patch to work on 8.6, still broken on 8.7.
Fix for that, and tests are coming.

One of the test failures for Drupal 8.7 is that in ParagraphsLookup the method skipOnEmpty() is called, which is replaced with skipInvalid(): https://www.drupal.org/node/3001283.

I see that ParagraphsLookup has some coding standard issues as well:

1. +++ b/src/Plugin/migrate/process/ParagraphsLookup.php
   @@ -0,0 +1,140 @@
   +
   +
   ...
   +
   +
   

   Double empty lines.

2. +++ b/src/Plugin/migrate/process/ParagraphsLookup.php
   @@ -0,0 +1,140 @@
   +
   +  public function transform($value, MigrateExecutableInterface $migrate_executable, Row $row, $destination_property) {
   ...
   +
   +  protected function check_migration(MigrationInterface $migration, $value, MigrateExecutableInterface $migrate_executable, Row $row, $destination_property) {
   

   Docblock missing.

3. +++ b/src/Plugin/migrate/process/ParagraphsLookup.php
   @@ -0,0 +1,140 @@
   +
   

   Extra empty line at the start of the method.

4. +++ b/src/Plugin/migrate/process/ParagraphsLookup.php
   @@ -0,0 +1,140 @@
   \ No newline at end of file
   

   Newline missing.

With Drupal 8.7 nearly arriving, would it be a good idea to drop support for Drupal 8.6 with new patches? It doesn't look like a final patch would make it before the next minor release of Drupal.

Worked fine for me. Patch created the migrations that create paragraphs out of field collections along with their fields, and then content migrations to migrate content and its revisions.

Here I am adding the respective Configuration and Content tags so the field collection migrations can run along others.

I realized that I was getting the wrong associations between paragraphs and their parent entities because the parent_id and parent_type properties were not being set. Here is a patch that fixes it.

I have install the patch #50 but i have an error during migration : Error : Call to a member function createInstance() on null dans Drupal\paragraphs\Plugin\migrate\process\ParagraphsLookup->transform() paragraphs/src/Plugin/migrate/process/ParagraphsLookup.php ligne 38
Regards,

@quatrys, I don't know what may be causing that error. Can you step into that line with a debugger to gather further insight?

@quatrys: I just experienced the same bug that you did. I don't know how I did not stumbled upon this before.

Here is a patch that fixes it.

A normal Paragraph Migration is ignoring the Paragraph Type, giving the field_name along with the bundle in the D7ParagraphsItemDeriver should fix this little issue.

Hi guys, thanks for all the work thus far on this. I'm trying to test this on a big and complex site (80 paragraph bundles and 4000+ paragraphs_item rows) and I'm running into a memory issue when trying to setup the migrations with drush migrate:upgrade.

PHP Fatal error: Allowed memory size of 12633243648 bytes exhausted in ...modules/contrib/paragraphs/src/Plugin/migrate/field/Paragraphs.php on line 73

As you can see I updated my allowed memory size to 12048M. Any ideas? I'm using php 7.1.29, Drush 9, and Drupal 8.6.14, and mysql 5.7.25.

Note, I'm only getting the error with the patch. Setting up drush migrate:upgrade worked fine prior to testing these patches.

Edit
I put a var_dump($field_name); in modules/contrib/paragraphs/src/Plugin/migrate/field/Paragraphs.php on line 73 and the output is continually one field name. In my case it's "field_accordion_items".

Edit 2
Oh I see on line 73 you have a call to the same method causing an endless loop. Did you mean to use parent::processFieldValues($migration, $field_name, $data);?

Edit 3
So based on my last edit I was able to setup the migrations. However, when I try to run one of them I get the following error:

[error]  Drupal\Core\Database\DatabaseExceptionWrapper: SQLSTATE[42S22]: Column not found: 1054 Unknown column 'parent_id' in 'field list': INSERT INTO {paragraphs_item_revision_field_data} (id, revision_id, langcode, status, created, parent_id, parent_type, parent_field_name, behavior_settings, default_langcode, revision_translation_affected) VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1, :db_insert_placeholder_2, :db_insert_placeholder_3, :db_insert_placeholder_4, :db_insert_placeholder_5, :db_insert_placeholder_6, :db_insert_placeholder_7, :db_insert_placeholder_8, :db_insert_placeholder_9, :db_insert_placeholder_10); Array
(
    [:db_insert_placeholder_0] => 3
    [:db_insert_placeholder_1] => 3
    [:db_insert_placeholder_2] => en
    [:db_insert_placeholder_3] => 1
    [:db_insert_placeholder_4] => 1560197887
    [:db_insert_placeholder_5] => 
    [:db_insert_placeholder_6] => 
    [:db_insert_placeholder_7] => 
    [:db_insert_placeholder_8] => a:0:{}
    [:db_insert_placeholder_9] => 1
    [:db_insert_placeholder_10] => 1
)
 in Drupal\Core\Entity\Sql\SqlContentEntityStorage->saveToSharedTables() (line 938 of /docroot/core/lib/Drupal/Core/Entity/Sql/SqlContentEntityStorage.php). 

So I'm thinking that the edit I made before is incorrect, or there is a new problem.

Tagging for DrupalCamp Asheville. Thanks!

This patch should fix tests.

Today I discovered that having parent::prepareRow() at the beginning of FieldCollectionItem::prepareRow() made it impossible for hooks_migrate_prepare_row() implementations to access the field data of each row. This patch fixes that.

I've gotten this to work pretty well for me now (80 paragraphs 4000+ paragraph_item rows), and I just wanted to share one bug I found.

In Drupal\paragraphs\Plugin\migrate\field\Paragraphs line 73, you need to change:

$this->processFieldValues($migration, $field_name, $data);
to
$this->defineValueProcessPipeline($migration, $field_name, $data);

Otherwise it causes an infinite loop. If you create a new patch that would be awesome.

Rerolled, and includes the feedback in #62.

I can confirm that patch #64 works for me as expected

I've made patch more universal.
Please look.

1. +++ b/paragraphs.module
   @@ -560,39 +566,42 @@ function _paragraphs_migration_bundle_adjust(array &$migration) {
   +  if (is_string($p)) {
   

   I think that we should use meaningful variable names.

2. +++ b/paragraphs.module
   @@ -560,39 +566,42 @@ function _paragraphs_migration_bundle_adjust(array &$migration) {
   +    throw new \Exception(sprintf('Unknown process element type: %s.', print_r($p, TRUE)));
   

   This would stop the migration process. Do we want that?

juampynr, thank you for review.

1. Let it name $process_element.
2. More precisely, we will stop generation of migration configurations. Ok. We can use drush_print().

Why do we remove these class checks?

We don't know, in what exact migrations "entity_type" will be used. E.g. it used in "user_picture_field" and "field_group" migrations...

I believe throwing \Exception will kill the migration from executing.

You are right. We should just drop a message. Maybe we can just do Drupal::logger("rg")->Error().

Source plugin name is not needed in my patch. And I just assume that 'entity_type' and 'targetEntityType' values all should be changed to 'paragraph'. Maybe it is just workaround...

Thank you!!!
I can confirm that patch #64 works for me as expected.

I had some problem with d7 field_colection revision table and the migration not working for me. (There were more rows than it should be. I think this one https://www.drupal.org/project/field_collection/issues/2000690 was the reason)
I did not need to migrate revision, so I just delete it.

DELETE FROM field_revision_field_NAME
WHERE NOT EXISTS (
  SELECT NULL
  FROM field_data_field_NAME
  WHERE field_revision_field_NAME.field_NAME_revision_id = field_data_field_NAME.field_NAME_revision_id
);

Maybe it will be helpful for someone.

Some fixes.

+++ b/src/Plugin/migrate/process/ParagraphsLookup.php
@@ -21,6 +21,13 @@ use Symfony\Component\DependencyInjection\ContainerInterface;
+  protected $migrationPluginManager;

Why adding this new dependency?

@super_romeo, can you post an intediff from #64 to #75? I am not clear on the set of changes that were introduced since then.

@juampynr, interdiff attached in #75.
We need $migrationPluginManager in transform() method.
I don't know how it works before :)
I will try to fix patch today.

Test fixed.

Another test should be fixed. I'll do it today.

My patch intendent for D8.8.x.
I understand:
1. Tests for D8.7.8 does not passed, because I use migrate.lookup service, that has been added in D8.8: New Migration Lookup and Stub services have been added.
2. Tests for D8.8.x does not passed, because tests intendent for D8.7.

Hi @super_romeo!

I am not sure how to help: tests were passing in #64. Then you made the patch, quoting your words in that comment, "more universal" at #65. Some of us gave you some feedback asking for the reasoning behind some of the changes that you made and then, in following patches, tests stopped passing.

Bases on the above, I suggest that we revert back to the patch at #64 and move forward from there.

Ok. I agree.
But what we should do then 8.8 will released? Will return to #80 patch?

@juampynr so #64 it is as RTBC? I'm really not deep into migrations. Please sign it off from your side so we can commit it then.

Good question.
Please read #82, 83. Patch #64 and 66 works. I think 66 better. But those patches for D8.7.8.
I use D8.8 and I couldn't develope patches for D8.7.8 so far. Last my patch works for D8.8 but tests should be fixed for D8.8.

Hi guys,
I have an old project (d8), which uses field collections. I would like to migrate data to paragraphs d8 (the same site). Is it possible to do that using above patches or are they only d7 field_collection to d8 paragraphs. Thank you in advance.

AFAIK only d7 field_collection to d8 paragraphs.

Indeed, the patch #64 was working fine for me and others and was passing the tests but D8.8.x introduced some modifications to migration lookup that should be addressed (as explained on comment #82).
Patch #80 works with Drupal 8.8-rc1 but it doesn't seem to make use of the new services (and I think it should). I believe we should focus on providing a patch that is compatible with core 8.8.x in order to be committed (and for D8.7.x one could use patch #64)

Thank you!!!
I can confirm that patch #64 works for me as expected with Drupal core 8.7 version.

Now patch #80 passes tests and needs review, I think.

Any thoughts on a 8.8 patch?

Nvrmnd, see it states that it is functional

Applying #80 to 8.8 I am seeing:

Fatal error: Cannot redeclare Drupal\paragraphs\Plugin\migrate\field\Paragraphs::defineValueProcessPipeline() in /app/web/modules/contrib/paragraphs/src/Plugin/migrate/field/Paragraphs.php on line 79

I see this error when visiting the migrations listing page for a migration group.

I have applied the patch at #80 on Drupal 8.8.

Can I check what the exact intentions of the patch are? I ask this because, with the patch I get migrations for both field collections AND paragraphs. Without it, these two different types are not present as migrations.

Field collection migration works well for me with said patch.

Paragraphs seems to get 3/4 of the way there. The paragraphs are migrated onto their nodes, but the fields on the paragraphs have no values.

Is it the intention of this ticket that it should provide migrations for both field collections and paragraphs?

We are getting same results as @iancawthorne, except we are getting successful paragraphs migration for the first paragraph type (alphabetically), but the other paragraph types are being ignored, which is resulting in empty values on the nodes. This is happening for both the single entity reference fields as well as the multiple entity reference fields. Any idea if this example is only for field collection only, and will not work if you have a mix of both field collections and paragraphs to migrate?

I'm experiencing the same as @iancawthorne and @fredonia_webteam. I can confirm that the paragraphs are migrated onto their nodes, but the fields on the paragraphs have no values when there's a mix of both field collections and paragraphs on the D7 side using the patch at #80.

I've been toiling on this for weeks. I've tried a simple migration within vanilla D7 and D8 environments, with paragraphs and no field collections, and it worked fine. With paragraphs AND filed collections, it failed.

EDIT: Here's a CSV export of the totals from migrate:status, which are all the same for both paragraphs and paragraphs revisions, respectively. I'm hoping that this may help diagnose the issue.

Group,"Migration ID",Status,Total,Imported,Unprocessed,"Last Imported"
"Import from Drupal 7 (migrate_drupal_7)",upgrade_d7_paragraphs_accordion_tabs,Idle,349,0,349,
"Import from Drupal 7 (migrate_drupal_7)",upgrade_d7_paragraphs_image,Idle,349,0,349,
"Import from Drupal 7 (migrate_drupal_7)",upgrade_d7_paragraphs_slideshow,Idle,349,0,349,
"Import from Drupal 7 (migrate_drupal_7)",upgrade_d7_paragraphs_tabs,Idle,349,0,349,
"Import from Drupal 7 (migrate_drupal_7)",upgrade_d7_paragraphs_text_block,Idle,349,0,349,
"Import from Drupal 7 (migrate_drupal_7)",upgrade_d7_paragraphs_revisions_accordion_tabs,Idle,9768,0,9768,
"Import from Drupal 7 (migrate_drupal_7)",upgrade_d7_paragraphs_revisions_image,Idle,9768,0,9768,
"Import from Drupal 7 (migrate_drupal_7)",upgrade_d7_paragraphs_revisions_slideshow,Idle,9768,0,9768,
"Import from Drupal 7 (migrate_drupal_7)",upgrade_d7_paragraphs_revisions_tabs,Idle,9768,0,9768,
"Import from Drupal 7 (migrate_drupal_7)",upgrade_d7_paragraphs_revisions_text_block,Idle,9768,0,9768,

After checking in the db for target_revision_id values, it is not getting the proper value to display the paragraph item on the node. This may be a larger issue with the paragraphs migration.

As for the patch in #80, the fields collection items are getting migrated properly into D8.

I have submitted a new issue here: https://www.drupal.org/project/paragraphs/issues/3122342

I found a problem while testing, the "field_name" property doesn't exist in the ParagraphsType row (D7ParagraphsItemDeriver::getDerivativeDefinitions line 120), it causes source plugin skipping the bundle condition and adding all paragraph items into each paragraph_item migration.

Here is the updated patch.

I migrated successfully from Field Collections (7.x-1.1) to Paragraphs with patch #98 without any problem. I'm using Drupal 8.8.5, Paragraphs 8.x-1.11, Entity Reference Revisions 8.x-1.8. Thanks for the patch!

@bubu, do you have any tutorials for migrating?

Thanks!

@Sseto, I didn't do anything special.

• I followed official Drupal 8 upgrade guide.
• I installed Paragraphs 8.x-1.11 and Entity Reference Revisions 8.x-1.8 and enabled them. After that I installed patch #98 from here with Composer.
• I performed migration with browser. I did nothing on command line.
• After migration I experienced that paragraph items display well but with views I cant get them. I resave all host content (admin/content, filter by content type, select all and node save action) and this problem gone.

@bubu, it worked! I migrated my Field Collections to Paragraphs.

The only thing that's missing is none of my content that's using Field Collections is migrated into paragraphs. Any idea what's missing?

Thanks!!

@Sseto: Please clarify:

• Did you just do a standard upgrade using core's migration UI or did you build a custom migration?
• Did all the field collection structures (types and fields) properly convert to Paragraph types?
• Did the source use multiple languages?
• Did any of the content migrate over?

Thank you.

@DamienMcKenna

• I used core's migration UI, no custom migration
• All field collection structures did properly convert to paragraph types
• The source does not use multiple languages
• All the content did migrate over, except content that's in field collection. It did turn into Paragraph field though.

Thanks Damien!

@Sseto: So, in summary, the data structures were converted correctly, but the content for them did not.

Putting this back to Needs work as it seems we might need some more test coverage.

@DamienMcKenna

I figured it out!

I followed @bubu by installing paragraphs 8.x-1.11 instead of paragraphs 8.x-1.x-dev. So I reinstalled paragraphs using the dev version, then applied the patch from #98, and finally used migrate UI. Everything got migrated. woohoo!

Thanks guys!

Ok, that's great to hear.

I still think it needs more test coverage.

Working on defining test coverage.

I'm done with migrating Field Collections to Paragraph on another site. Same setup and steps as in #99. Thanks.

The existing patch appears to round out the test coverage that was already there from previous issues, well done everyone on that work!

I think it would be worth filling out the test coverage a little more to make sure that all of the three nodes with field collections (nodes 8, 9, 10) have the expected data, I made a start on it.

Even though I need D7 paragraphs -> D9 paragraphs migration, this issue seems to be the best foundation for adding that as well.

But since the patch from #110 (that ends with -109) uses services that were removed from Drupal 9 (while the 8.x-1.x HEAD seems to be compatible with Drupal 9), I think that the next, reasonable step is adding a test for the state that #110 already provides on Drupal core 8.8.whatever.

The attached patch adds an extra functional test for testing Paragraphs' migrations via the Migrate UI.
* It adds the missing source fixture files (for the file migrations)
* It uses the Migrate UI for executing all of the migrations.
* Based on entity IDs and labels, it verifies that the expected entities exist after the migration.

@huzooka, I can't install #112 on drupal/paragraphs 1.12.0.
Output:

$ patch < paragraphs-field_collections_base_migration-2911244-112.patch
patching file d7_field_collection.yml
patching file d7_field_collection_revisions.yml
can't find file to patch at input line 64
Perhaps you should have used the -p or --strip option?
The text leading up to this was:
--------------------------
|diff --git a/migrations/d7_field_collection_type.yml b/migrations/d7_field_collection_type.yml
|index 79227c3..b0cdceb 100644
|--- a/migrations/d7_field_collection_type.yml
|+++ b/migrations/d7_field_collection_type.yml
--------------------------
File to patch: 

I'm before a migration. I can test your patch If you upload a new one.

@bubu: Are you sure that your local install of Paragraphs is 8.x-1.12? I just downloaded 8.x-1.12 and the patch applied ok with only this variance:

patching file paragraphs.module
Hunk #1 succeeded at 405 with fuzz 2.

@DamienMcKenna, thanks for the replay.

Yes, I'm sure. I'm working with composer. I'm patching with composer, and I got this issue:

$ composer update drupal/paragraphs
    1/1:	https://packages.drupal.org/8/drupal/provider-2020-2$1f50a3ee32f5f2cff0e1124fc90a4dabbf03929c3bc9245e1b58f0c2c8e3cd99.json
    Finished: success: 1, skipped: 0, failure: 0, total: 1
Gathering patches for root package.
Removing package drupal/paragraphs so that it can be re-installed and re-patched.
  - Removing drupal/paragraphs (1.12.0)
Deleting web/modules/contrib/paragraphs - deleted
Loading composer repositories with package information
Updating dependencies (including require-dev)
Package operations: 1 install, 0 updates, 0 removals
Gathering patches for root package.
Gathering patches for dependencies. This might take a minute.
  - Installing drupal/paragraphs (1.12.0): Loading from cache
  - Applying patches for drupal/paragraphs
    https://www.drupal.org/files/issues/2020-05-20/paragraphs-field_collections_base_migration-2911244-112.patch (Field collections deriver and base migration #112 (https://www.drupal.org/project/paragraphs/issues/2911244))
   Could not apply patch! Skipping. The error was: Cannot apply patch https://www.drupal.org/files/issues/2020-05-20/paragraphs-field_collections_base_migration-2911244-112.patch
                                                                                                                                                                                                           
  [Exception]                                                                                                                                                                                               
  Cannot apply patch Field collections deriver and base migration #112 (https://www.drupal.org/project/paragraphs/issues/2911244) (https://www.drupal.org/files/issues/2020-05-20/paragraphs-field_collect  
  ions_base_migration-2911244-112.patch)!   

I don't know why this happens. This is the first time this has happened to me... Here is the long output. But, yes patching by hand is working:

$ patch -p1 < paragraphs-field_collections_base_migration-2911244-112.patch 
patching file migrations/d7_field_collection.yml
patching file migrations/d7_field_collection_revisions.yml
patching file migrations/d7_field_collection_type.yml
patching file migrations/d7_paragraphs.yml
patching file migrations/d7_paragraphs_revisions.yml
patching file paragraphs.module
Hunk #1 succeeded at 405 with fuzz 2.
patching file src/Plugin/migrate/D7FieldCollectionItemDeriver.php
patching file src/Plugin/migrate/D7ParagraphsItemDeriver.php
patching file src/Plugin/migrate/field/FieldCollection.php
patching file src/Plugin/migrate/field/Paragraphs.php
patching file src/Plugin/migrate/process/ParagraphsLookup.php
patching file src/Plugin/migrate/source/d7/FieldCollectionItem.php
patching file tests/fixtures/drupal7.php
patching file tests/fixtures/sites/default/files/Babylon5.txt
patching file tests/fixtures/sites/default/files/ds9.txt
File tests/fixtures/sites/default/files/yellow.jpg: git binary diffs are not supported.
patching file tests/src/Functional/Migrate/MigrateUiParagraphsTest.php
patching file tests/src/Functional/Migrate/MigrateUiParagraphsTestBase.php
patching file tests/src/Kernel/migrate/ParagraphContentMigrationTest.php

I performed migration with manually patched (#112) Paragraphs 8.x-1.12 and Entity Reference Revisions 8.x-1.8. It seems me everything OK.

In one of my previous comment I mentioned a problem with views. I fount that is an another issue, not related to this.

Thanks for the patches!

Re-rolled #112.

It seems that being able to correctly handle multilingual situations for both paragraph and field_collection entities is too big task for me :(. I'm sorry.

I think we will be unable to solve the paragraphs AND field_collection migrations by relying only on the migration lookup and the migration order, especially for multilingual situations:

• If nodes are migrated first, we cannot know the destination id of the referenced field_collection|paragraph in the value process pipeline.
• If paragraphs|field_collections are migrated first, we only have the source ids and source ref ids of nodes - in d7, a translated node had different ids per language, and it is migrated into the same node, with different revisions. But these translations are migrated into the same node in D8 (the revisions will store the translations).
• If there is a nested paragraph|field_collection, its parent might be missing on migration (eg because it will be migrated in a next row).

For now, I'll create a new patch for making the existing migration plugins (except the paragraphs lookup) work with Drupal 9; but this one is only a re-roll.

ParagrapsProcessOnValue prevented the migration of comment field config (=field instance) config entities.

This patch went a bit forward, and adds the parent entity identifiers for the paragraphs entity migration.

There are still open questions and tasks, but I hope that this will pass on both Drupal 8.8 and Drupal 9.0. Let's see.

The failing tests on core 8.9.x of #121 are exactly the same ones that are failing with 8.x-1.x HEAD.

I have concerns about migrating the field collection IDs: I think that Paragraphs has to keep the paragraph entity IDs and revision IDs.

Anyway, I think that this is close to be 'finished' (issue title is about the deriver and the base migration), and only Remaining tasks.5 has to be addressed.

Please confirm this!

@heddn, thanks for the instant feedback!

I realised that we still have to finish `ParagraphContentMigrationTest`.

Hi all,
I try to migrate field_collection by drush.
paragraphs": "1.x-dev", entity_reference_revisions": 1.8
I have install the patch #121 but i have an error during migration of upgrade_d7_field_formatter_settings
[error] The "field_collection_fields" plugin does not exist. Valid plugin IDs
Do I need to change something in the upgrade_d7_field_formatter_settings ?

Issues #123.3, #123.4 and #123.6 still needs to be addressed.

I also finished the ParagraphContentMigrationTest kernel test I found following the coding standard errors raised in comment #124. (And also added ParagraphMultilingualContentMigrationTest.)

Re #126: @zorax, I think (I guess) that the upgrade_d7_field_formatter_settings migration you want to migrate misses the field type mapping what the patch(es) here trying to add. Could you please share its structure?

Re 130:

130.1: Yeah, I realized right after I hit the Save button :)

130.3:
I think that Drupal 8.8.x wont get the complete migration path. :(

For answering Huzoka, #128
Here is my class upgrade_d7_field_formatter_settings

status: true
dependencies: {  }
id: upgrade_d7_field_formatter_settings
class: Drupal\migrate_drupal\Plugin\migrate\FieldMigration
field_plugin_method: alterFieldFormatterMigration
cck_plugin_method: null
migration_tags:
  - 'Drupal 7'
  - Configuration
  - 'field'
migration_group: migration_bilan
label: 'Field formatter configuration'
source:
  plugin: d7_field_instance_per_view_mode
  constants:
    third_party_settings: {  }
process:
  field_type_exists:
    -
      plugin: migration_lookup
      migration: upgrade_d7_field
      source:
        - field_name
        - entity_type
    -
      plugin: extract
      index:
        - 0
    -
      plugin: skip_on_empty
      method: row
  entity_type:
    0:
      plugin: get
      source: entity_type
    paragraphs:
      plugin: static_map
      map:
        field_collection_item: paragraph
        paragraphs_item: paragraph
      bypass: true
  bundle:
    0:
      plugin: static_map
      source: bundle
      bypass: true
      map:
        comment_node_forum: comment_forum
    paragraphs:
      plugin: paragraphs_process_on_value
      source_value: entity_type
      expected_value: field_collection_item
      process:
        plugin: substr
        start: 6
  view_mode:
    -
      plugin: migration_lookup
      migration: upgrade_d7_view_modes
      source:
        - entity_type
        - view_mode
    -
      plugin: extract
      index:
        - 1
    -
      plugin: static_map
      bypass: true
      map:
        full: default
  field_name:
    -
      plugin: get
      source: field_name
  options/label:
    -
      plugin: get
      source: formatter/label
  options/weight:
    -
      plugin: get
      source: formatter/weight
  plugin_id:
    -
      plugin: process_field
      source: type
      method: getPluginId
  formatter_type:
    -
      plugin: process_field
      source: type
      method: getFieldFormatterType
  options/type:
    -
      plugin: static_map
      bypass: true
      source:
        - '@plugin_id'
        - '@formatter_type'
      map:
        file:
          default: file_default
          url_plain: file_url_plain
          path_plain: file_url_plain
          image_plain: image
          image_nodelink: image
          image_imagelink: image
        field_collection:
          field_collection_view: entity_reference_revisions_entity_view
        entityreference:
          entityreference_label: entity_reference_label
          entityreference_entity_id: entity_reference_entity_id
          entityreference_entity_view: entity_reference_entity_view
        datetime:
          date_default: datetime_default
        fivestar:
          fivestar_formatter_default: fivestar_stars
          fivestar_formatter_rating: fivestar_rating
          fivestar_formatter_percentage: fivestar_percentage
        link_field:
          link_default: link
    -
      plugin: d7_field_type_defaults
    -
      plugin: skip_on_empty
      method: row
  hidden:
    -
      plugin: static_map
      source: '@options/type'
      map:
        hidden: true
      default_value: false
  options/settings:
    -
      plugin: default_value
      source: formatter/settings
      default_value: {  }
  options/third_party_settings:
    -
      plugin: get
      source: constants/third_party_settings
  options/settings/view_mode:
    field_collection:
      plugin: paragraphs_process_on_value
      source_value: type
      expected_value: field_collection
      process:
        plugin: get
        source: formatter/settings/view_mode
destination:
  plugin: component_entity_display
migration_dependencies:
  required:
    - upgrade_d7_field_instance
    - upgrade_d7_view_modes
  optional:
    - upgrade_d7_field_collection_type
    - upgrade_d7_field
    - upgrade_d7_view_modes

I think that Drupal 8.8.x wont get the complete migration path. :(

I can confirm this — see @catch in #3076447-95: Migrate D7 entity translation revision translations:

I'm not opposed to backporting this to 8.8.x, except that we're gearing up to do the very last patch release there so there'll be no opportunity to release follow-up bugfixes should one surface. So it might be better to leave this in 8.9.x - doing that for now but re-open if you'd really like to see a backport.

Right after i fixed the migration for the Migrate Drupal UI (fixed means that I force the same migration execution order that we have in the paragraphs migration kernel tests for a while), I was unable to get the same results with the UI test :)

@heddn, I don't really want to touch ParagraphsLookup since I don't know how it should work. The way that may work is to create a 100% test coverage for its current functionality, and try to refactor after we have the test coverage.

Changes:

1. I've made a standalone MigrationPluginsAlterer service and moved every migrate-related, preexisting functionality into it.
2. Paragraph entity migration is now forced to happen before their parent entity migration, unless when the parent entity is a paragraph.
3. Since now the functional test that uses the Migrate Drupal UI is aIso able to repeat the same assertions that the paragraph content migration kernel test does, I think that this task is ready.

Further tasks discovered:

1. I think we should consider keeping the IDs of the source paragraph entities (and force paragraph migrations to be executed before field collection migrations). If we do so, and a migration needs to updated (to grab in updated or new nodes to the destination site) from source site that uses both field collection and paragraphs, then only the field collection migrations have to be rolled back. This might save some migration execution time.
2. The multilingualism still needs to be addressed. On a multilingual Drupal instance, the language of the paragraph revisions is grabbed from the parent entity. Right now, with this patch, the migrated paragraph entities get the default language of the Drupal instance.
3. Instead of entity_reference_revisions, I'd use the entity_complete migration destination plugin for migrating paragraphs and field collections, but unfortunately it isn't available on Drupal 8.8.x. Shouldn't we just copy it and it's tests into Paragraphs?.. That would simplify our content entity migrations...
4. The fields of the test node that has translations are referring to different paragraph entities. I don't know how we should handle these cases. (E.g. the English node has 3 paragraphs referenced in its paragraph field, while its Icelandic translation has only one.)
5. For a more robust migration path, we have to consider to derive paragraphs migrations based on their parent entity type (and parent entity bundle) and after that by their own bundle (e.g. d7_paragraphs:node:article:paragraph_bundle_one,) or based on their parent field (e.g. d7_paragraphs:node:article:field_paragraph_one_only). If we do continue derive them by their own bundles, it will easily end in orphaned paragraphs and partially migrated paragraph entities, since a paragraph can also have a paragraph reference field.
   
6. It would be nice to have other test nodes that are translated with Entity Translation module. Edit: Paragraphs 7.x does not support Entity Translation.

I'd like to ask for addressing all of these issues/ideas in separate issues, including also the ParagraphsLookup migration process plugin refactor.

Update for #135 Further tasks #5: Basically, this is how the current patch derives field collection migrations.

Last patch that somewhat worked for me was #98.

The #135's patch is producing requirements errors for all of the revisions, but the requirements have clearly been executed.

 [notice] Processed 13 items (13 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_paragraphs_2_columns'
 [notice] Processed 5 items (5 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_paragraphs_2_columns_aside'
 [notice] Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_paragraphs_aaa_viewer'
 [notice] Processed 21 items (21 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_paragraphs_accordion_tabs'
 [notice] Processed 2 items (2 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_paragraphs_image'
 [notice] Processed 35 items (35 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_paragraphs_node_reference'
 [notice] Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_paragraphs_slideshow'
 [notice] Processed 12 items (12 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_paragraphs_tabs'
 [notice] Processed 0 items (0 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_paragraphs_tag_content_with_terms'
 [notice] Processed 151 items (151 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_paragraphs_text_block'
 [notice] Processed 11 items (11 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_paragraphs_video'

 [error]  Migration upgrade_d7_paragraphs_revisions_2_columns did not meet the requirements. Missing migrations upgrade_d7_paragraphs_2_columns, upgrade_d7_paragraphs_2_columns_aside, upgrade_d7_paragraphs_aaa_viewer, upgrade_d7_paragraphs_accordion_tabs, upgrade_d7_paragraphs_image, upgrade_d7_paragraphs_node_reference, upgrade_d7_paragraphs_slideshow, upgrade_d7_paragraphs_tabs, upgrade_d7_paragraphs_text_block, upgrade_d7_paragraphs_video. requirements: upgrade_d7_paragraphs_2_columns. requirements: upgrade_d7_paragraphs_2_columns_aside. requirements: upgrade_d7_paragraphs_aaa_viewer. requirements: upgrade_d7_paragraphs_accordion_tabs. requirements: upgrade_d7_paragraphs_image. requirements: upgrade_d7_paragraphs_node_reference. requirements: upgrade_d7_paragraphs_slideshow. requirements: upgrade_d7_paragraphs_tabs. requirements: upgrade_d7_paragraphs_text_block. requirements: upgrade_d7_paragraphs_video.

 [error]  upgrade_d7_paragraphs_revisions_2_columns migration failed.

 [error]  Migration upgrade_d7_paragraphs_revisions_2_columns_aside did not meet the requirements. Missing migrations upgrade_d7_paragraphs_2_columns, upgrade_d7_paragraphs_2_columns_aside, upgrade_d7_paragraphs_aaa_viewer, upgrade_d7_paragraphs_accordion_tabs, upgrade_d7_paragraphs_image, upgrade_d7_paragraphs_node_reference, upgrade_d7_paragraphs_slideshow, upgrade_d7_paragraphs_tabs, upgrade_d7_paragraphs_text_block, upgrade_d7_paragraphs_video. requirements: upgrade_d7_paragraphs_2_columns. requirements: upgrade_d7_paragraphs_2_columns_aside. requirements: upgrade_d7_paragraphs_aaa_viewer. requirements: upgrade_d7_paragraphs_accordion_tabs. requirements: upgrade_d7_paragraphs_image. requirements: upgrade_d7_paragraphs_node_reference. requirements: upgrade_d7_paragraphs_slideshow. requirements: upgrade_d7_paragraphs_tabs. requirements: upgrade_d7_paragraphs_text_block. requirements: upgrade_d7_paragraphs_video.

 [error]  upgrade_d7_paragraphs_revisions_2_columns_aside migration failed.

---- SNIP ----

@ghalusa, what do you have in your paragraph entity migration yaml(s)?

I just filed an issue (with a patch) that addresses #135 Further tasks #5 at #3145755-2: Orphaned (nested) paragraphs entities after migration & (invalid) stub paragraph entity leftovers.

Addressed #144.

Also added a patch without the test improvements and new tests.

🥳🥳🥳

We are running into the same errors as @ghalusa has in #137. Patch #98 was working for field collections and paragraphs migration from D7. Updated to patch #146, now we are having errors with migrating D7 paragraphs_revisions. Not certain this patch is the problem as we did update to Drupal 8.9 and Paragraphs 1.12 also.

@fredonia_webteam,

It would help if you could give enough details so that others could reproduce your problem.

Do you guys know if it's safe to remove all my field_collection tables in my database after the migration?

I want to remove these tables:
migrate_map_d7_field_collection
migrate_map_d7_field_collection_revisions
migrate_map_d7_field_collection_type

Thank you!

Hey guys,

I recently migrated my D7 site to D8. In the upgrade, I migrated all my Field_Collection module content to use Paragraphs.

Everything worked perfectly until I decided to try out Layout Builder...

When I activate Layout Builder in a content type that uses paragraphs, I get:

The website encountered an unexpected error. Please try again later.

in dblog, it says this:

Drupal\Component\Plugin\Exception\PluginNotFoundException: The "field_collection_item" entity type does not exist. in Drupal\Core\Entity\EntityTypeManager->getDefinition() (line 150 of C:\wamp64\www\sseto\web\core\lib\Drupal\Core\Entity\EntityTypeManager.php).

Any idea how to fix this issue?

Thank you!

Lots of tests, RTBC from @heddn and most importantly, ASCII art from DS9. That's good enough for me to commit without checking too closely. Thanks everyone!

@Sseto: i suggest you create a separate issue for that, doesn't sound related to the migration.

This is amazing and perfect timing. Thanks to all involved.

Last time I tried to get both field collection and paragraphs migrations to work was with patch #80 on here. On that occasion field collections migrated fine in full, while paragraphs seemed to be a bit all over the place - wrong fields, wrong data etc.

I've just tried the latest dev release of paragraphs, which if I have understood correctly includes the committed patch here. Now I get paragraphs migrating 100% fine, but on the field collections, the fields of the collection are missing. No errors in the feedback during the migration using drush. Are there any follow up tickets for this one? Does anyone have any tricks to get it to work like running certain migration items before others?

There are at least a full example ymls that works?

At this time I can't get works the importer yml that set the paragraphs into node (d7_node)
I get the yml that uses the plugin d7_field_collection_item works well getting the field collection values into paragraphs structure.

A bug was reported and a fix was provided at #3192993: [migrate_upgrade compatibility] d7_field_instance upgrade fails due to redundant 'field_' prefix removal on bundle.

And … Drupal >=9.1.4 broke this: #3203739: Paragraphs migrations broken in Drupal >=9.1.4.