Hi there,

It would be nice to have multilingual support for the migrate module.
The attached patch adds a destination handler for migrate which takes care about translations.
It's not a really elegant nor a fast solution - but it works. (At least in my test cases ;) )

The patch adds support for node and content based translation.

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

plach’s picture

Hi Peter, thanks for working on this.

I'm afraid I'll have to won't fix this issue (and any other similar one): the reason is Content Translation aims to replace a core module, hence adding dependencies to contrib modules might render core inclusion more difficult. An exception is the Title project which will have to make its path into core too, somehow. Any other module with strong possibility to get into core might be taken in consideration for integration, though.

I'd like sun's feedback about this before actually closing this issue, but IMO the right way to go is either moving this feature request to the Migrate queue or creating a separate project.

das-peter’s picture

I fully agree with you. This is nothing that has to go into core, I'll ask mikeryan (migrate maintainer) how we can handle this - because an own project would be just overhead.
I'd like to see better multilingual support in migrate itself anyway :)

sun’s picture

+++ includes/translation.migrate.inc	1 Jan 1970 00:00:00 -0000
@@ -0,0 +1,106 @@
+        // Load old node to make sure it's the right id - mapping of migrate is not language sensitive
+        $current_node = node_load($entity->nid);

But why is this node-centric? (Sorry, if this is natural for Migrate module; didn't work with it yet)

Powered by Dreditor.

das-peter’s picture

Thanks sun, you're right!
Looks like I mixed some stuff - beside the fact that it's ugly like hell ;)
I hope all this can be cleaned by changing some stuff in migrate itself.

sun’s picture

Status: Needs review » Postponed

Alright, I hope you don't mind if we mark this issue postponed for now then. I think we have loads of more important, "basic" todos for this project currently, and contrary to those, this issue sounds fairly advanced to me. I'd be happy to revisit this at some point, because I've heard that Migrate becomes a very important solution, so we logically should make a best effort to support it. But yeah, "later". :)

das-peter’s picture

No problem - I need it anyway and it doesn't really matter where it resides. Thus I'll continue this in my project specific code. As soon as it's "later", I'll bring it back ;)
Btw: Let me know if you have some workpackages of the basic todos. I use the module already and I'm waiting for more feedback on #924968: Initial work - what's somehow related to this module. Means I'm interested in working on multilingual stuff ;)

das-peter’s picture

I'll try to keep my patch here up to date.

plach’s picture

Project: Content translation » Entity Translation
Version: 7.x-2.x-dev » 7.x-1.x-dev
bastnic’s picture

A quick update on this patch, seems to works with my use case.

bastnic’s picture

Status: Postponed » Needs review

Status: Needs review » Needs work

The last submitted patch, translation-migrate-support-929402-9.patch, failed testing.

bastnic’s picture

reroll without the path of my project

bastnic’s picture

Status: Needs work » Needs review

Status: Needs review » Needs work

The last submitted patch, translation-migrate-support-929402-9.patch, failed testing.

donquixote’s picture

Yeah, I want this!

@das-peter,
how is this designed to work? Is one row = one language, or is one row = all languages at once?
If one row = one language, what is the primary key? As far as I remember, primary keys in migrate destinations have to be single-value integers, or not?
I guess, as you did not add any destination class (just a destination handler), it is one row = all languages at once. So the field handler needs to deal with arrays as field values.
Do rollback and --update work?

@plach, sun,
what about node_save() with entity_translation enabled, does this save all languages or just one translation?

plach’s picture

what about node_save() with entity_translation enabled, does this save all languages or just one translation?

All field translations available in the $entity data structure are saved to the storage, It's the standard core behavior.

donquixote’s picture

@das-peter / bastnic:
One thing I still don't get...

<?php
    if (entity_translation_enabled($entity_type) && property_exists($entity, 'nid') && $entity->nid) {
?>

This will only work, if the $entity->nid is already set.
So, either during an --update run, or if the migration (class) that imports the translations is a different one than the migration (class) that created the node itself.

How do your migration classes look like, that use this handler?
How do you get the translations from the source db into $entity in the first place?

I suppose, at the time that MigrateTranslationEntityHandler::prepare is called, this has already happened, and now the translations sit in $entity->translations->data[$lang] ?

donquixote’s picture

I'm getting a little confused, what the EntityTranslationDefaultHandler::setTranslation() method actually does.
In fact, it is only writing on the $entity object, which it keeps as a protected variable.
Is there any documentation about this stuff?

KarenS’s picture

Just a note that this could be a really useful feature to create a system to bulk update translations. As noted in another issue, lots of translations are done offsite and the 'translation' work is mostly doing lots of copy/paste into the forms. How much nicer would it be to ask for translations to be provided in something Migrate can use (XML or a spreadsheet or a database), map the source to the site, then use Migrate to bulk update everything at once? So getting Migrate working properly could be very useful.

bastnic’s picture

@donquixote indeed, i began to save the node only in english in a first import, and then I do what @KarenS suggest to import all translations in another migrate import class.

das-peter’s picture

I'll need some time to get into this again.
As far as I remember I changed again some stuff to keep it working.

The approach I've used was to have a row per language. Where only the translatable content differs from row to row.

plach’s picture

I'm getting a little confused, what the EntityTranslationDefaultHandler::setTranslation() method actually does.
In fact, it is only writing on the $entity object, which it keeps as a protected variable.
Is there any documentation about this stuff?

That method just sets the updated translation records in the translation handler. These are subsequently stored through EntityTranslationHandlerInterface::saveTranslations().

donquixote’s picture

Ok.
But saveTranslations() is not called in the patch in #12. Is this triggered by something else?

plach’s picture

In entity_translation_field_attach_update() through node_save().

bastnic’s picture

I'm coming back on this ticket with a beautiful WTF effect. I really really did see my patch working but not anymore.

First :

      // Content based translation
      if (in_array($entity->type, variable_get('entity_translation_entity_types', array()))) {

can't work, we should have

      // Content based translation
      if (entity_translation_node_supported_type($entity->type)) {

Second,

// Preserve original language setting
$entity->language = $entity->translations->original;

... cause entity_translation not to work with (I think) migrate 2.3. So I did that:

--- a/entity_translation/includes/translation.migrate.inc
+++ b/entity_translation/includes/translation.migrate.inc
@@ -61,6 +61,7 @@ class MigrateTranslationEntityHandler extends MigrateDestinationHandler {
           );
         }
         // Preserve original language setting
+        $entity->field_language = $entity->language;
         $entity->language = $entity->translations->original;
       }
       // Node based translation

--- a/migrate/plugins/destinations/fields.inc
+++ b/migrate/plugins/destinations/fields.inc
@@ -91,6 +91,8 @@ abstract class MigrateFieldHandler extends MigrateHandler {
         return LANGUAGE_NONE;
       case isset($arguments['language']):
         return $arguments['language'];
+      case !empty($entity->field_language) && $entity->field_language != LANGUAGE_NONE:
+        return $entity->field_language;
       case !empty($entity->language) && $entity->language != LANGUAGE_NONE:
         return $entity->language;
         break;

I sincerely have no idea why my previous patch had ever worked!

sinasalek’s picture

Component: Code » Base system

I replaced name and description of a vocabulary and i can longer migrate to it.
This is the error i'm getting

SQLSTATE[21S01]: Insert value list does not match column list: 1136 Column count doesn't match value count at row 1: INSERT INTO {field_data_description_field} (entity_type, entity_id, revision_id, bundle, delta, language, description_field_value, description_field_summary, description_field_format) VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1, :db_insert_placeholder_2, :db_insert_placeholder_3, :db_insert_placeholder_4, :db_insert_placeholder_5, :db_insert_placeholder_6_0, :db_insert_placeholder_6_arguments, :db_insert_placeholder_7, :db_insert_placeholder_8); Array ( [:db_insert_placeholder_0] => taxonomy_term [:db_insert_placeholder_1] => 9783 [:db_insert_placeholder_2] => 9783 [:db_insert_placeholder_3] => food [:db_insert_placeholder_4] => 0 [:db_insert_placeholder_5] => en [:db_insert_placeholder_7] => [:db_insert_placeholder_8] => plain_text [:db_insert_placeholder_6_0] => [:db_insert_placeholder_6_arguments] => Array ( [format] => filtered_html ) ) (modules/field/modules/field_sql_storage/field_sql_storage.module:448)
radiobuzzer’s picture

Hi, until this is resolved, I have to mention how I did it, in case this helps anybody. I used the "prepare" method of the Migrate api, which is added in the end of any Migration class.

In this example I am migrating two fields, name_en and name_el into one translatable field. I also know that the default language is 'el' for all of the nodes, but this can be easily changed

class RegionMigration extends Migration{
  public function __construct() {
    ...
  }
  function prepare($entity, stdClass $row){
	$entity->language = 'el';
	$entity->title_field['el'][0]['value'] = $row->name;
	$entity->title_field['en'][0]['value'] = $row->name_en;
	$entity->translations = (object) array(
			'original' => 'el',
			'data' => array(
					'el' => array(
							'entity_type' => 'node',
							'entity_id' => $entity->nid,
							'language' => 'el',
							'source' => '',
							'uid' => '0',
							'status' => '1',
							'translate' => '0',

					),
					'en' => array(
							'entity_type' => 'node',
							'entity_id' => $entity->nid,
							'language' => 'en',
							'source' => 'el',
							'uid' => '1',
							'status' => '1',
							'translate' => '0',

					),
			)
	);
	return $entity;
  }
}

primozsusa’s picture

Hi, did anybody solved or has an example of using migrate with entity translation field. Maybe field handler example... #27 works kind of. if you have one record with multiple languages for import. but for taxonomy term with #27 the problem is that the data is not saved the same as if it would be done manually.
any suggestions welcome.
thx Primoz

mvdve’s picture

You probably want to look at Issue 1069774.
There is complete description of how to create the translation. This can be done in the complete function.

make77’s picture

Issue summary: View changes
Status: Needs work » Needs review
FileSize
5.67 KB

Hi,

I updated the patch for the latest version of the module (7.x-1.0-beta3).

Status: Needs review » Needs work

The last submitted patch, 30: translation-migrate-support-929402-30.patch, failed testing.

steinmb’s picture

Status: Needs work » Needs review
FileSize
5 KB

Broken patch. This patch apply cleanly on dev. Let's see if bot also is happy.

greenjuls’s picture

Hi, I just updated to version 7.x-1.0-beta3 of entity_translation module but it broke my migrations.
Problem is that the translations were not loaded in MigrateTranslationEntityHandler::prepare().

I replaced the following piece of code

// Load translations if necessary
if (!property_exists($entity, 'translations')) {
  $entity->translations = $translation_handler->getTranslations();
}

with:

// Load translations
$translation_handler->loadTranslations();
plopesc’s picture

Hello
I'm trying to use migrate and enityt_translation with no luck, and I think this patch can be helpful.
Could you include an example of how are you implementing it?

Thank you in advance

primozsusa’s picture

What finally worked for me:
- make sure your filed is set translatable in UI
- mapping

    $this->addFieldMapping('language')->defaultValue('en');
    $this->addFieldMapping('translate')->defaultValue(true);
    $this->addFieldMapping('field_tk_description', 'desc_en');
    $this->addFieldMapping('field_tk_description:format')->defaultValue('nc_filtered_html_editor');
    $this->addFieldMapping('field_tk_description:language')->defaultValue(array('en', 'sl'));

- prepareRow

  public function prepareRow($row) {
    if (parent::prepareRow($row) === FALSE) {
      return FALSE;
    }
    $row->language = array('en', 'sl');
    $row->opis_en = array($row->desc_en, $row->desc_sl);
    return TRUE;
    }

- prepare

  public function prepare($entity, $row) {
    //$entity->language = 'en';
    // this is not needed because $row->desc_en = array($row->desc_en, $row->desc_sl);
    // is already working from prepareRow
    //$entity->field_tk_description['sl'] = $entity->field_tk_description['en'];
    //$entity->field_tk_description['sl'][0]['value'] = $row->desc_sl;
  }

- complete: to update entity_translation table

  public function complete($entity, stdClass $row){
    $entity->language = 'en';

    $handler = entity_translation_get_handler('node', $entity);
    $translation = array(
      'translate' => 0,
      'status' => 1,
      'language' => 'sl',
      'source' => $entity->language,
    );
    $handler->setTranslation($translation);
    $handler->saveTranslations();

    return $entity;
  }
brockfanning’s picture

I may be missing something, but the latest patch in #33 didn't seem sufficient for my needs. It's limited to nodes, and also limited to existing nodes, which I don't think was intended. It also has a section about supporting "node-based translations" which I'm not clear on, as I thought that entity_translation was the alternative to node-based translations. Also I don't believe it is registering the destination handler correctly.

Given all that I wrote a new patch that is working for me.

The way I'm using it is to map an array of language codes to each translatable destination field's ":language" subfield, and a corresponding array of values to the field itself. For example, something like this in the migration constructor:

$this->addFieldMapping('field_subtitle', 'legacy_subtitle');
$this->addFieldMapping('field_subtitle:language', 'legacy_subtitle_languages');

Where $row->legacy_subtitle might be...

array(
  'My legacy subtitle',
  'Mon héritage de sous-titres',
);

And $row->legacy_subtitle_languages would be...

array(
  'en',
  'fr',
);

Also in combination with migrate_d2d and this patch: https://www.drupal.org/node/2389783 the mapping can be as simple as (assuming a D7 multilingual to D7 multilingual migration):

$this->addFieldMapping('field_subtitle', 'field_legacy_subtitle');
$this->addFieldMapping('field_subtitle:language', 'field_legacy_subtitle:language');
heldercor’s picture

@brockfanning, doesn't this break fields with multiple values?

brockfanning’s picture

@heldercor, not that I know of, though in my testing all translatable fields have been single-value. In theory it would be the same, but the $row value would be an array of arrays, like:

$row->legacy_numbers = array(
  array(
    'one',
    'four',
    'seven',
  ),
  array(
    'uno',
    'cuatro',
    'siete',
  ),
);

$row->legacy_numbers_language = array(
  'en',
  'es',
);

But as said, I haven't tested that.

nedjo’s picture

Thanks, this is working well on my testing.

For anyone who like me is wondering why it works to set the value and a language subfield as arrays, the answer is in migrate's field handling--see MigrateFieldsEntityHandler.

A couple of minor suggestions:

  1. +++ b/includes/translation.migrate.inc
    @@ -0,0 +1,96 @@
    +            'status' => 1,
    

    Maybe default this to $entity->status if set?

  2. +++ b/includes/translation.migrate.inc
    @@ -0,0 +1,96 @@
    +            'changed' => (empty($entity->changed)) ? time() : $entity->changed,
    +            'created' => (empty($entity->created)) ? time() : $entity->created,
    

    Could use REQUEST_TIME rather than calling time().

kristiaanvandeneynde’s picture

Working great on my tests!

@nedjo in #39:

  1. Published translations on unpublished nodes still show you an Access Denied, so it may be more user friendly to keep them published in which case they are automatically active when the node gets published (again).
  2. Agreed, see attached patch.

I also cleaned up some whitespaces and moved the hook to entity_translation.migrate.inc as suggested by the Migrate docs.

Seeing as everyone likes the patch and the change from #39 is small, I'm going to RTBC it immediately after testbot goes green.

kristiaanvandeneynde’s picture

Status: Needs review » Reviewed & tested by the community
.bert’s picture

@brockfanning & @kristiaanvandeneynde the patch in #40 (essentially the same as #36) doesn't seem to work for multiple values on a single field.

It could be my setup, but it works beautifully on single values. Any thoughts on where to look?

@heldercor, why did you think it might break on multiple values?

If anyone can point me in the right direction of where to start to troubleshoot this, that would be greatly appreciated.

Thanks

kristiaanvandeneynde’s picture

How did you provide the values? Did you map them as specified in #38?

Edit: To clarify, this patch only makes sure $entity->translations is properly set. The whole value populating mechanism is part of Migrate's "subfield" system.

.bert’s picture

Yes, the values are mapped out as described in #38.

Here's the basics of what we're doing.

<?php
// Mapping
$this->addFieldMapping('field_important', 'important_description');
$this->addFieldMapping('field_important:format')->defaultValue('full_html');
$this->addFieldMapping('field_important:language', 'important_description_language');
?>
<?php
// In prepareRow()
$row->language = array('en', 'fr');
$this->set_descriptions($row);
?>

The set_descriptions() method grabs the data and applies it to the translation fields

<?php
protected function set_descriptions($row) {
  // $result = connect to database and get results
  foreach ($result as $data) {
    // We will only see ENGLISH values in the HR_DESCRIPTIONS_IMPORTANT constant
    if (in_array($data->type, explode(',', HR_DESCRIPTIONS_IMPORTANT))) {
      $row->important_description['en'][] = '<h3>' . $data->title . '</h3><div>' . $data->description . '</div>';
      $row->important_description_language['en'] = 'en';
    }
    // If the data type is one of the language field values, add it the FRENCH
    // value of the important description field
    else if (in_array($data->type, explode(',', HR_DESCRIPTIONS_LANGUAGE_FIELDS))) {
      $row->important_description['fr'][] = '<h3>' . $data->title . '</h3><div>' . $data->description . '</div>';
      $row->important_description_language['fr'] = 'fr';
    }
    // Only add to supplemental description if the above failed
    else {
      $row->supplemental_description[] = '<h3>' . $data->title . '</h3><div>' . $data->description . '</div>';
    }
  }
}
?>

Adding multiple values in this fashion works fine on it's own without using entity translation, as seen in $row->supplemental_description. I also tried using an array to add the language data, like this:

<?php
$row->important_description_language['fr'][] = 'fr';
?>

So far, it will only add single values and will not add multiple values. I will attempt to troubleshoot, but if anyone has some guidance that would be greatly appreciated.

Thanks.

.bert’s picture

I can confirm that the method described in #38 does not work as described. Instead, the following works for that example when importing multiple values:

<?php
$row->legacy_numbers = array(
  'one',
  'four',
  'seven',
  'uno',
  'cuatro',
  'siete',
);

$row->legacy_numbers_language = array(
  'en',
  'en',
  'en',
  'es',
  'es',
  'es',
);
?>

Basically, you're setting the language on the subfield array using the same key as the value array. I'm not sure exactly "why", but appears to run without issue for our setup.

From my previous comment, the following works instead:

<?php
protected function set_descriptions($row) {
  // $result = connect to database and get results
  foreach ($result as $data) {
    // We will only see ENGLISH values in the HR_DESCRIPTIONS_IMPORTANT constant
    if (in_array($data->type, explode(',', HR_DESCRIPTIONS_IMPORTANT))) {
      $row->important_description[] = '<h3>' . $data->title . '</h3><div>' . $data->description . '</div>';
      $row->important_description_language[] = 'en';
    }
    // If the data type is one of the language field values, add it the FRENCH
    // value of the important description field
    else if (in_array($data->type, explode(',', HR_DESCRIPTIONS_LANGUAGE_FIELDS))) {
      $row->important_description[] = '<h3>' . $data->title . '</h3><div>' . $data->description . '</div>';
      $row->important_description_language[] = 'fr';
    }
    // Only add to supplemental description if the above failed
    else {
      $row->supplemental_description[] = '<h3>' . $data->title . '</h3><div>' . $data->description . '</div>';
    }
  }
}
?>

The other thing to note is that $row->language does not need to be set in prepareRow().

I hope this helps someone down the line.

kristiaanvandeneynde’s picture

Very nice write-up, I'll try to confirm this later on and see if this is indeed intended to work that way for subfields.

gambry’s picture

I implemented my own handler to deal with this, but I tested the patch anyway hoping to have pathauto/alias support. However even with the #40 patch, migrated translations still don't have aliases.

Is translations aliasing on migrating working for any of you?
I can see different issues about entity_translation and pathauto, but not of them refer to migrate.

I'll start having a look to find out why it doesn't create aliases, but it would be good if there is already a solution somewhere.

Thanks and great work!

gambry’s picture

So far the only solution I found is this, but to properly work with the #40 it needs to be inside the complete() handler.

Waiting for any better/official solution, I'll leave my comment here for those poor souls seeking help. :)
(If a better/official solution does NOT exist, I'm happy to update the patch).

kristiaanvandeneynde’s picture

That should probably go in some sort of PathautoMigrateDestinationHandler and be committed in that module.

gambry’s picture

@kristiaanvandeneynde I don't think that should be part of a Pathauto handler, but definitelly is NOT part of this Entity Translation handler.

My comment is a hint to whoever lands to this issue finding a solution on migrating/programmatically-creating an entity translation + URL aliases, and it can be ignored for the real purpose of the issue.

estoyausente’s picture

The patch in #40 work correctly.

I apply it and change my migration code and the entities (nodes in this case) are imported correctly.

In the construct:

		$this->addFieldMapping('title','TituloEspanol')
			->xpath('SpanishTitle');
		$this->addFieldMapping('translate', 'legacy_translated');
		$this->addFieldMapping('title_field','legacy_title');
		$this->addFieldMapping('title_field:language','legacy_language');
		$this->addFieldMapping('body','legacy_summary');
		$this->addFieldMapping('body:summary','legacy_summary');
		$this->addFieldMapping('body:language', 'legacy_language');

PrepareRow method:

	public function prepareRow($row){
		if (parent::prepareRow($row) === FALSE) {
			return FALSE;
		}

		// Multilanguage fields
		$row->legacy_title = array($row->xml->SpanishTitle, $row->xml->BrasilTitle);
		$row->legacy_summary = array($row->xml->SpanishSummray, $row->xml->BrasilSummary);
		$row->legacy_language = array('es', 'pt-br');

		//only mark as translated if have title in br
		$row->legacy_translated = !empty($row->xml->BrasilTitle);
		return TRUE;
	}

I hope that this example helps somebody :)

kristiaanvandeneynde’s picture

I can confirm the method .bert describes in #45 works for multi-value fields. I normally double-check this by analyzing the flow of the code, but time is short right now so you'll have to take .bert's and my word for it :)

mikeryan’s picture

Doing my first migration involving entity_translation, and I'm very happy to see this patch here. On a source code scan, all that jumped out at me is the one nit below (not worth moving out of RTBC for that). It's late in the day here, but tomorrow I'll try the patch for reals and report back on my experience.

Thanks!

+++ b/includes/translation.migrate.inc
@@ -0,0 +1,86 @@
+   * Registers all entites as handled by this class.

Nit: entities.

mikeryan’s picture

Well, I made the patch, reran my migration (which had previously properly migrated field translations, but not the entity_translation table stuff so the UI showed that it was translated), and it worked perfectly.

+1 RTBC from me!

rymo’s picture

another +1 RTBC here for #40

plach’s picture

Status: Reviewed & tested by the community » Fixed

Committed and pushed #40, awesome work, thank you all!

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.