Improve source record count of translation migrate source plugins which use the "i18n_string" table

1. +++ b/core/modules/block_content/src/Plugin/migrate/source/d7/BlockCustomTranslation.php
   @@ -95,4 +95,41 @@ public function getIds() {
   +      $row_id = implode('.', array_intersect_key($result, $this->getIds()));
   +
   +      if (!empty($rows[$row_id])) {
   +        continue;
   +      }
   

   Surely this is not iterating over 100% identical rows? Why do different rows have the same value for ::getIds()?

2. +++ b/core/modules/block_content/src/Plugin/migrate/source/d7/BlockCustomTranslation.php
   @@ -95,4 +95,41 @@ public function getIds() {
   +  public function doCount($refresh = FALSE) {
   +    // Since the number of variables we fetch with query() does not match the
   +    // actual number of rows to migrate, we need to override count() to return
   +    // the correct count.
   +    $non_filtered_array = $this->prepareQuery()->execute()->fetchAll();
   +    $filtered_iterator = $this->removeDuplicates($non_filtered_array);
   +    return (int) $filtered_iterator->count();
   +  }
   

   Documenting the answer to the above here with a concrete example would make this far easier to understand.

   Furthermore, why is it safe to deduplicate at all? Is there something in the process pipeline that loads the data that lives in the rows we're removing here?

I get that you are trying to remove the duplicate rows. I suppose the reason is that when looks at the count of rows it will better match the real world number of translations. On the other hand it still feels odd because count() is supposed to return *all* rows discovered.

+++ b/core/modules/block_content/src/Plugin/migrate/source/d7/BlockCustomTranslation.php
@@ -95,4 +95,41 @@ public function getIds() {
+    // Since the number of variables we fetch with query() does not match the
+    // actual number of rows to migrate, we need to override count() to return
+    // the correct count.

Let's just explain here that the query gets duplicate rows because of the source data and the count is eliminating the duplicates. As it is it makes is sound like the query is broken.

This duplicate problem is true for any source plugin using i18nQueryTrait. Maybe we can do them all together and have a helper in the trait?

I suppose the reason is that when looks at the count of rows it will better match the real world number of translations.

I'm pretty sure this is indeed the intent. I believe @huzooka also pointed out that the migration definitions using this ignore the rows that he's omitting here; they don't do anything with them, and load the data in the missing rows directly, while ignoring these deduplicated rows.

@huzooka, can you please elaborate/clarify? 🙏

This duplicate problem is true for any source plugin using i18nQueryTrait. Maybe we can do them all together and have a helper in the trait?

Right! That is what I meant in #6's they don't do anything with them, and load the data in the missing rows directly!

Anything calling \Drupal\content_translation\Plugin\migrate\source\I18nQueryTrait::getPropertyNotInRowTranslation() suffers from this problem.

+++ b/core/modules/content_translation/src/Plugin/migrate/source/I18nQueryTrait.php
@@ -84,4 +84,41 @@ protected function getPropertyNotInRowTranslation(Row $row, $property_not_in_row
+    // Since the number of variables we fetch with query() does not match the
+    // actual number of rows to migrate, we need to override count() to return
+    // the correct count.

It is not true that a source plugin will only return the actual number of rows to migrate. The docs in SourcePluginBase::count say the count is 'the count of available source records'. The modification here is to make the count a more 'natural' one.

The Migrate Drupal Module was approved for removal in #3371229: [Policy] Migrate Drupal and Migrate Drupal UI after Drupal 7 EOL.

This is Postponed. The status is set according to two policies. The Remove a core extension and move it to a contributed project and the Extensions approved for removal policies.

The deprecation work is in #3522602: [meta] Tasks to remove Migrate Drupal module and the removal work in #3522602: [meta] Tasks to remove Migrate Drupal module.

Migrate Drupal will not be moved to a contributed project. It will be removed from core after the Drupal 12.x branch is open.

just updating tags

The Migrate Drupal and Migrate Drupal UI modules are deprecated and will be removed from Drupal 12. For these modules, effort is now focused on bug fixes and necessary tasks. Therefore, this feature request is closed as won't fix.

Thanks to all for working on this!