Multilingual Feeds - Make field import language-aware

I get alot of:
"Notice: Undefined index: language i theme_feeds_ui_mapping_form() (rad 803 av /modules/feeds/feeds_ui/feeds_ui.admin.inc)."

...when I patch this one. Help?

Question: by default feeds import creates node with the language set to "language natural" which means it doesn't show the translate-tab. And also it won't show that the perticular language is translated. But if you click it, it shows the translated node.

Help?

Sub

I tested the patch in #5 (http://drupal.org/node/1183440#comment-4631354) and it works great.
However the patch doesn't apply cleanly on mappers/link.inc, so in attachment is an update patch that applies without problems on 7.x-2.x.

When using the entity_translation module I do indeed encounter some problems with the UI: the product entities are set to language neutral and thus no translate tab is shown to edit the translation of the fields. If I change the language manually on the entity, the field translations get lost...
But if I just don't touch the product entities at all, and just go about creating product displays referencing the product entity, the fields get translated perfectly...
This is good enough for me at this point. If I find some more time I'll investigate how to work nicer with the entity_translation UI.
But apart from that this patch is ready to be committed IMHO.

#8, thanks! Ill test this one.

----

Got the following errors:
can't find file to patch at input line 75
Perhaps you should have used the -p or --strip option?
The text leading up to this was:
--------------------------
|diff --git a/mappers/date.inc b/mappers/date.inc
|index 19aa65d..981ba0e 100644
|--- a/mappers/date.inc
|+++ b/mappers/date.inc

The git-way worked. But when I import, it still won't get a specific language, tho I set the GUID to a language.

#12, ok thanks. I have 4 different languages on every field.

1. Do I have to get all the translations in the same CSV-file or can I import from different files?

2. How do a node understand it is connected to another translation while importing? Do I need a connecting ID?

#pcambra: thanks! I'll try that and hope it works... Got a database which is going into about a 100 ckk-fields and 4 different languages, so it's important for me to get this right. Thanks, again.

1. Another problem I have noticed is that the languages doesn't say they have been translated. You have to hit "Translate" to see the actual translation. Is there any patch to correct this?

2. Also, I don't understand if or when I need to use default languages when I'm doing the mapping. If I have 2 languages I set "Title_en" to "title" as english and "Title_es" as spanish? Or do I set the "Title_en" as "default language" because it's the main language? Sorry if the questions are stupid, but I had some problems wrong languages showing up at wrong translations...

Updated the patch from #8 for mapping/taxonomy.inc, to include support for I18N_MODE_TRANSLATE, i.e. creating a separate term per language.
I had to change the lookup process, because a term / word could be the same in multiple languages, but this should still result in a separate term per language, otherwise that term won't be shown in all the languages it applies to.
Hope that makes sense. I added a check to make sure this functionality only kicks in when I18N_MODE_TRANSLATE is enabled, to avoid conflicts with other modes.

Thanks for reviewing this! I'd like to get this language awareness into Feeds badly.

applied patch from #17.

i have to importers
1. hotels_all
Do not update existing nodes
PropertyId - Guid (unique) (All languages)
Description - Description (All languages)

2. hotels_es
Update existing nodes (slower than replacing them)
PropertyId - Guid - Spanish
Description - Description - Spanish

i have activated content type translation.

now, after importing CSV with HOTELS_ES, instead of updating the existing nodes, all the nodes appear like standalone documents (with language setted to Spanish) instead of becoming traslations of already existing nodes imported earlier with HOTELS_ALL

i'm doing something wrong or this is bug?

pd. Sorry for my english

I've tried the patch from #17 and through the admin ui I can map the fields to the appropriate languages. But when I do the import it doesn't create any translations, it just overwrites the values of the title and body fields of the default language (english).

I'm importing all the languages in one csv file as recommended in #19.
(also tried as an xml file with same results)

Attached is an export of my importer.

Am I missing something?

subscribing..

I tried the patch from comment #8 and it mostly worked. However I can confirm #5, which says that the new nodes are language neutral, which seems to cause that the "Translate" tab is not displayed.

However, the fields themselves are translated correctly.

It helps a bit, if I use i18n module the set the current language as default for new nodes. Then the nodes created on import do have a language, and the "Translate" Tab is visible. When I click "Translate", there are the options to "Add Translation" for all languages. And, when I then chose to create a Translation, the fields which where translated on import are already pre-filled.

Soo, multi language field import works "a bit" at the moment, but needs a bit more work...

@pcambra, I am using the title and entity translation modules and have set the fields to allow translations.

after reading #22 I tried using the patch from #8 instead of the patch from #17 and I had a bit more success. As described in #22 the translate tab is available but no translations are listed as active, when I then click "Add translation" the fields are pre-populated with the imported data.

Also, the title of the base node (English) ended up with the French title.

Can you elaborate on what the problem is with the patch in #17?
It's the same as in #8, only handles taxonomy fields import better, so I don't know why it should work worse, except if you encounter taxonomy specific issues.

Ok I tried #17 again and you're right, it is behaving the same as #8 (I'm not dealing with taxonomy fields). I guess I previously just hadn't noticed that the fields were pre-populated as I was expecting the translations to actually be active rather than needing to go to "Add translation".

Subscribing, greetings, Martijn

subscribing

Patch #27 works great for me. Thanks guys for fixing this.
I've tested on node body & title (with title module).

Big thanks #pcambra for the this! #27 patch is working great. Can't wait this to be reviewed and committed!

I think this is very important to be included since I find Feeds very limited when working with multilingual content which comes from other sources... Almost lost my faith on this :S

One thing, I'm not able to get this work with taxonomy term processor o.O

The patch in #27 works great but it miss to add the traduction in {entity_translation}. The effect is that the translate tab in node edit doesn't show any translation.

subscribe! Me too. I'm not able to get this work with taxonomy term processor ...

It works very well except for the title it gets replaced by the other language imported. Also, I am using Search API to index a custom field with translation enabled, but it seems that the other language is not being indexed for the field. Is there a way to work around? Thanks!

Thanks pcambra, I have tried the title module but it doesn't work for me with the patch. Although i can manually create a node and make the title available in different languages, when import from feeds the original title would get replaced with an other language version.

I use Solr to make the search index, do I need to add related fields to the index? There is Translation source node, I don't know what it is different from the original node.

Also, do I need to specify a default language to use for the contents created by importing a file? I didn't specify one, so the contents created are all language neutral, and I cannot modify them in other languages unless I set the current language for them.

Thanks for your help!

Do I need to add translation for each node content after I do the import? It seems that the translated version would not be created automatically. Although the fields were imported in 2 different languages, only the default language ones were created. However, the other language version not created could be seen when going to the node view page. Is that correct?
Sorry for my English, Thanks!

Hi,
Love this patch #27!
greetings, Martijn

Can the translations get saved automatically instead of having to create them manually with pre-filled values after doing the import? Thanks!

Hi,
I reviewed the patch and it works well, thanks!
I see remarks of others that may be changed:
- #32 miss to add the traduction in {entity_translation}.
- #36 title issue.
When these things are handled, I think this patch is ready to commit!

Greetings, Martijn

The above patch gets us a good chunk of the way there. I'd like to thank everyone who worked on getting it this far.

I've been doing some work to get this initiative rolling again. Here are the other moving parts that need to work in conjunction with this issue, as alluded to by Summit:

• #1362790: Undefined property: stdClass::$title in title_entity_update()
• #1362378: Provide entity post-save hook
• #1364070: Record translations created by Feeds

In addition to those open issues, we need to explicitly state the prerequisites for using this new functionality effectively. I've added another paragraph of documentation to the Mapping page. This is included in the patch below, and is the only major change to the last patch (other than the previous patch not being properly Git-formatted, while this one is ;).

Prerequisites:

• Title (When mapping, select the second "Title" in the list. The first one is the core one not provided by the Title module.)
• Entity translation
• Multilingual content (Both options "Set current language as default for new content" and "Require language (Do not allow Language Neutral)" need to be checked in the Multilingual settings of each content type being imported.)

Tagging.

Locale and Content translation should be on

For Drupal 7:
I think 'Content translation' is not necessary for my site. Only 'Locale'.

I only want to import in one language (german), because my site only shows german pages to the visitors. If all imported nodes (and the views) know their language (and not 'und'), all is ok to me.

I would like recommend module Feeds: Entity Translation for importing data through Feeds into translatable fields handled by Entity Translation.

Please read issue guidelines, this is not at all critical.

@colan

i assume this is for nodes that user translatable fields; not translatable nodes. so, re:

(Both options "Set current language as default for new content" and "Require language (Do not allow Language Neutral)" need to be checked in the Multilingual settings of each content type being imported.)

I think, if the setting is set for translatable fields for a specific node type; then it is not possible to be set.

perhaps this is bugs in one of the other modules though??

@liquidcms: It's been a while, but IIRC, those are necessary to force an explicit language. It won't work otherwise. Yes, this could very well be a bug elsewhere, but I never looked into too deeply. I just made sure those options were enabled. ;)

too many issues still with translatable fields approach so we went back (yet again) to translatable nodes; in which case i use this new selector set as "all languages" for each field and have a column in import for language and assign this to language as a mapping field. then in one of the import hooks i set the tnid to link the translated nodes together.

I decided to do the same and was pretty confident that I could link the translations together with Rules since it supports Feeds. But I couldn't do it with Rules. Would you mind sharing your solution using import hooks?

for our particular case, the titles in FR and EN were the same so we had to after_parse to handle that.

/**
* hooks to importing questions
* 
*  - tweak the title by adding lang suffix to fool importer in to thinking they are unique records
* 
* @param FeedsSource $source
* @param FeedsParserResult $result
*/
function mymodule_feeds_after_parse(FeedsSource $source, FeedsParserResult $result) {
  foreach ($result->items as $key => $row) {
    if ($row['language'] == 'fr') {
      $result->items[$key]['title'] .= '_fr';
    }
  }
}

/**
* preocess the FR import record so we can go back and set TNID on the EN node (now that EN node has been created and we can also set TNID for impending FR node create
* 
* @param FeedsSource $source
* @param mixed $entity
*/
function mymodule_feeds_presave(FeedsSource $source, $entity) {
  if ($entity->language == 'fr') {      
    $realtitle =  str_replace('_fr', '', $entity->title);
    $query = new EntityFieldQuery;
    $result = $query
      ->entityCondition('entity_type', 'node')
      ->propertyCondition('type', 'question')
      ->propertyCondition('title', $realtitle)
      ->propertyCondition('language', 'en')
      ->execute();
    $nid = array_shift(array_values(array_keys($result['node'])));
    $entity->tnid = $nid;
    $entity->title = $realtitle;
    
    //  save tnid on english node too
    $ennode = node_load($nid);
    if ($ennode->tnid == '0') {
      $ennode->tnid = $nid;
      node_save($ennode);
    }
  }
}

Alright, reviewing this issue.

Is there a way that we could get this to work with the new mapping config options in dev?

Here's the original issue. #860748: Config for mappers?

There should be documentation in feeds.api.php, and you can find a simple implementation in path.inc

Not sure if this approach is feasible, or desired. I'm not terribly adept at this stuff. What are the options going forward? entity_translation vs. locale. I need to sit down and familiarize myself with this stuff.

For me this functionality is very important. When I'm using entity translations, there is only one node with translatable fields. feeds_et module does this, I'm not sure how is this patch different?

Currently I have all different languages in one XML file. I'm not sure is it possible to use a different importer to update the field translations from another XML file, at least the GUIDs have to match. It's better to have them in the same file I think.

Looking at the patch in #41, it looks like language support would have to be added to every target field, so the config mapping api is not a good fit. Is it really just as simple as adding $language to each set_target callback?

Added tnid to mapper and parser.
please see comment #66

Using the updated plugins in post #59, you now get tnid from the Feeds UI, can now map it like you can with nid and uid and other fields.

Again, I have tested this to work using the latest dev build of Feeds as of Friday sept 28 , 2012.

I successfully imported a dataset of 1400 nodes, (700 in english, 700 in french) including tnid.

Please review and schedule for include to the next dev build asap. We will need this functionality and from my tests, it is working.

Thanks

please see comment #66

Hi pcambra, please see comment #66

See comment #66

pcambra, here is the good patch, I promise!

Please re-test.

This one IS working for SURE.

this patch allows tnid mapping and parsing if the module "translation" is enabled (d7 core), I pulled the latest build and patched against it and named it following the naming convention of description-issuenumber-commentnumber.patch

I've successfully tested this code several times on my system.

I've tested a feed and dataset with a tnid mapping, and a feed and dataset without a tnid mapping.

Both work when tested against this code.

The title says "Make field import language aware", but the last patch is all about content translation (separate nodes without field translation), right? Should they be moved to another issue? I think they are different use cases. Both useful though, I need both types of translation in my project and multilingual importing as well.

mErilainen

I need both types of translation in my project and multilingual importing as well

Hi mErilainen, it seems like there already is another issue for node.

I've added a step by step HOWTO test the patch discussed in comment #66

Alternatively I could create a feature that would help the maintainers more easily test this patch.

I'd like to help the maintainers on this.

Thanks,

The last comments in this issue are moving this towards content translation, which is definitely not the same as field (entity) translation. Comment number 58 is the last one I'm hoping to see some progress on.

Do we need a separate setting per processor for the node language when there is the "Language" field where we can map to? Then content should have a source language so that it won't show up as "not translated".

I agree with you. The patch in #66 helped me a lot but is not the same issue.

I can't express enough how cool is to have configuration by mapping, I'll explore that further in commerce feeds, that's actually the intention of this patch at the beginning and I think that the language
should be just another setting for the mapping, here's a UI suggested:
https://skitch.com/pcambra/ei3hk/feeds-multilanguage-ui

Cool! That's exactly what this patch needs!

Hi I've implemented the patch 6 mentioned in #8, I only want to import description in English using Node ID to update existing nodes, I also using the 'Language' column as well to update only English version, I follow all the steps mentioned in #8 but it still store the description in 'und', hope you understand please help.

Thanks

Attached is a version of the code in #66 backported against Feeds-6.x-1.0-beta12

Need to make this work.

Please post patches regarding the tnid in the correct issue, thanks.

For now I disabled language, I feel this may come back to bite me :P
I need to find out more about how multiple languages should be imported using a .csv file very soon!

Hi Alar, to import bilingual content that do not already have tnid's using feeds you may be interested in this method:
(Credit to S.R. and J.B. for the work ).

See zip attached
Feeds Translation
=================

Allows Feeds to import nodes and apply Content Translation to them (i.e. tnid)

Installation
------------

1. Download Feeds 2.0-alpha8 or higher
2. Apply the following patch to Feeds: http://drupal.org/files/1994654-add-plugin-alter.patch
3. Enable feeds_translation

Usage
-----

1. Create a English Feeds Mapping (or language or your choice) -> Following field needs to exsist: ```GUID``` (your unique record selector for the Feeds Mapping).

2. Create a French Feeds Mapping (or language of your choice) -> Following fields needs to exsist: ```GUID``` (your unique record selector for the Feeds Mapping) and ```Translation node id```. ```Translation node id``` Feeds source field name can be any value as it will be overwirten in step 3. Example: Source: Blank Source 1, Target: Translation node id.

3. Go to the Feeds Tamper page for the French Feeds Mapping and for the ```Translation node id``` field add the "Get TNID from GUID" plugin

4. Run the English Feed

5. Run the French Feed

- Make sure to use the Language field so the mappings have a Language applied. See table example below to store the language directly in the data.

- When the French feed runs the Tamper plugin will take the GUID and look it up in the Feeds table to find the English/original node and apply the nid of the English node to the tnid of the French node using the ```Translation node id``` field.

- Make sure to use a GUID that is unique to the specific set of language imports. If you have a unreleated feed that shares a GUID value you will end up with conflicts that has the wrong node associated.

- The English and French GUID field should have the same values. Best way to do this is have your multilingual data in a single row data structure:

ID-GUID|Title (Eng)|Title (Fre)|Description (Eng)|Description (Fre)|Language(En)|Language(Fr)
-------|-----------|-----------|-----------------|-----------------|------------|------------
1|English Title|French Title|English Description|French Description|en|fr
2|sample2|sample3|sample4|sample5|en|fr

Thanks Joseph,
Feeds translation is just what I needed! Thanks for suggesting the bit about the Tamper plugin as well.
The french and english is maintained side by side on one line of excel and is loaded into Drupal as two separate nodes linked by a common GUID.
Merçi

I took a stab at adding field translation to feeds (in response to @pcambra's comment: https://drupal.org/node/1183440#comment-6530976). There's no dependency on the entity_translation module, since field translation is built into drupal core, but it does support it. That is, it will create an entity translation translation record (if necessary) if entity_translation is installed when adding fields that are in a different language than the default one. The code is partly adapted from some of the outdated patches earlier in this issue as well as Feeds: Entity Translation (which no longer works either).

A quick patch overview:

• Adds a mapping configuration form with a language option to text, number, file and link field mappings (provided that the target field is translatable).
• Adds a language option to taxonomy field mappings (provided that the target field is translatable)
• Uses the configured language code from the mapping configuration when importing instead of 'und'
• I've attached a little screenshot of the mapping page as an example.

I've only had a chance to test it with the 'text' field type, but so far it's been working. It would be nice to get some feedback though to see if I'm on the right track with the code.

Something that is currently not really supported is auto-creation of new taxonomy terms (for taxonomy term reference fields). This is a tricky one and I pondered over it for a while. Suppose you had a content type with a term reference field. The term reference field isn't translatable, but the terms themselves are. Now suppose you're importing into this content type and you have an english and french column in the source for the taxonomy field. When the importer gets to a source row where the term needs to be autocreated, it will create two different terms for the english and french source values. Depending on how we're translating our taxonomy terms, this either gets us halfway there (if we're using i18n_taxonomy), or gives us an extra term we don't need (if we're using entity_translate or simple string translation). Not to mention the fact that the 'name' property of a taxonomy term isn't "field" translatable anyway (you need entity_translate AND the title module to do that). Anyway, I just wasn't sure how much we wanted to support here so I've left it out for now. Ideally it would check what type of translation the taxonomy terms are using, then do the appropriate things when creating new terms... but I thought that might need more discussion...

The patch is against the current dev release. I don't know think it will apply cleanly against 7.x-2.0-alpha8... The file.inc file has changed quite a bit and it might choke on it.

Here's a patch against feeds-7.x-2.0-alpha8 in case anyone needs this right away and doesn't want to use the dev release. =)

Hello all.

I tested my changes from the patch in comment #78 today a little more rigorously and discovered a couple issues. Here's a re-rolled version that I've tested (I've uploaded one for the dev branch as well as the current alpha8 release). I can vouch for it working with text, number, file and taxonomy fields. I haven't tested the link field but from the looks of the code it should work...

See comment #78 (https://drupal.org/node/1183440#comment-7579839) for a better description of what the patch does.

drclaw

I updated the Issue Summary. Seemed like it needed one. I also think it might be worth discussion whether we split this issue off into two separate issues. One for Content Translation support (https://drupal.org/node/1183440#comment-7478058) and one for Field Translation Support (https://drupal.org/node/1183440#comment-7584721). Any thoughts?

drclaw, wow, great job!

Thanks for the update, it summarizes things well.

I would add one more point, that language should be mappable, per feed item, but that's a different issue, possibly a different module.

Overall very nice. It is a bit of a hack, but at this point it's the only way to proceed without a major overhaul. This definitely needs some tests.

+++ b/feeds.moduleundefined
@@ -875,6 +875,70 @@ function feeds_item_info_save($entity, $entity_id) {
 /**
+ * Adds an entity translation if the entity_translation module is enabled.
+ */

Parameters need docs.

+++ b/feeds.moduleundefined
@@ -875,6 +875,70 @@ function feeds_item_info_save($entity, $entity_id) {
+/**
+ * Returns the language summary text for use in a mapper target summary_callback
+ *
+ * @see mappers/text.inc

Param docs.

+++ b/feeds.moduleundefined
@@ -875,6 +875,70 @@ function feeds_item_info_save($entity, $entity_id) {
+function feeds_mapper_summary_language($mapping, $target, $form, $form_state) {
+  if (module_exists('locale')) {
+    $info = field_info_field($mapping['target']);
+    if ($info['translatable']) {
+      $language_options = array(LANGUAGE_NONE => t('All languages')) + locale_language_list('name');
+      if (empty($mapping['language'])) {
+        return t('Language: <strong>@search</strong>', array('@search' => $language_options[LANGUAGE_NONE]));
+      }
+      return t('Language: <strong>@search</strong>', array('@search' => $language_options[$mapping['language']]));
+    }
+  }
+  return FALSE;

This isn't going to work with more complex fields, since target isn't necessarily a field name.

+++ b/mappers/file.incundefined
+++ b/mappers/file.incundefined
@@ -21,6 +21,8 @@ function file_feeds_processor_targets_alter(&$targets, $entity_type, $bundle_nam

@@ -21,6 +21,8 @@ function file_feeds_processor_targets_alter(&$targets, $entity_type, $bundle_nam
         'callback' => 'file_feeds_set_target',
         'description' => t('The URI of the @label field.', array('@label' => $instance['label'])),
         'real_target' => $name,
+        'summary_callback' => 'file_feeds_summary_callback',
+        'form_callback' => 'file_feeds_form_callback',
       );
 
       if ($info['type'] == 'image') {
@@ -29,12 +31,16 @@ function file_feeds_processor_targets_alter(&$targets, $entity_type, $bundle_nam

@@ -29,12 +31,16 @@ function file_feeds_processor_targets_alter(&$targets, $entity_type, $bundle_nam
           'callback' => 'file_feeds_set_target',
           'description' => t('The alt tag of the @label field.', array('@label' => $instance['label'])),
           'real_target' => $name,
+          'summary_callback' => 'file_feeds_summary_callback',
+          'form_callback' => 'file_feeds_form_callback',
         );
         $targets[$name . ':title'] = array(
           'name' => t('@label: Title', array('@label' => $instance['label'])),
           'callback' => 'file_feeds_set_target',
           'description' => t('The title of the @label field.', array('@label' => $instance['label'])),
           'real_target' => $name,
+          'summary_callback' => 'file_feeds_summary_callback',
+          'form_callback' => 'file_feeds_form_callback',
         );
       }

Does it make sense to map each column to a language? I'm not sure we have a choice at this point.

+++ b/mappers/file.incundefined
@@ -129,5 +138,41 @@ function file_feeds_set_target($source, $entity, $target, $value) {
 
+  feeds_mapper_add_translation($entity, $langcode);
+

Is this safe to call so often? For every column for every field.

It seems like at the very least, it will be expensive.

+++ b/mappers/file.incundefined
@@ -129,5 +138,41 @@ function file_feeds_set_target($source, $entity, $target, $value) {
+/**
+ * Mapping configuration summary for text fields
+ *
+ * @param array $mapping
+ *   Associative array of the mapping settings.
+ * @param array $target
+ *   Array of target settings, as defined by the processor or
+ *   hook_feeds_processor_targets_alter().
+ * @param array $form
+ *   The whole mapping form.
+ * @param array $form_state
+ *   The form state of the mapping form.
+ *
+ * @return string
+ *   Returns, as a string that may contain HTML, the summary to display while
+ *   the full form isn't visible.
+ *   If the return value is empty, no summary and no option to view the form
+ *   will be displayed.
+ */
+function file_feeds_summary_callback($mapping, $target, $form, $form_state) {
+  return feeds_mapper_summary_language($mapping, $target, $form, $form_state);
+}
+
+/**
+ * Settings form callback.
+ *
+ * @return array
+ *   The per mapping configuration form. Once the form is saved, $mapping will
+ *   be populated with the form values.
+ */
+function file_feeds_form_callback($mapping, $target, $form, $form_state) {
+  return feeds_mapper_form_language($mapping);

All of these callbacks that wrap feeds_mapper_summary_language() and feeds_mapper_form_language() aren't necessary, just use the functions directly.

Hi twistor,

Thanks for the quick response and patch review! Here's some thoughts on your comments:

• RE: Param docs - I'll get working on these. I actually didn't include them because I wasn't sure if you would want to use those functions in the way that I was using them. I figured it made sense to make the language form it's own function to avoid code duplication, but wasn't 100% on the execution.
• RE: $target - In which case is $target not a field name? (so I can test it and account for it).
• RE: File field column mapping - I agree, I don't think we can avoid having to choose a language for each column.
• RE: Entity Translation support - I was a bit lazy on this one. It occurred to me a few days ago that I didn't need to set the translation for every field. Essentially we only need to do it once per entity per non-default language because all we need is the entity_translation record to be saved for the language. So, we can either keep it as it is and just check if there is already a translation for the language using $handler->getTranslations(), OR we could handle translations after the mappings are all done in a presave hook (or something). We would have to scan all the entity fields, however, for languages other than the default and build a list of langcodes, then run feeds_mapper_add_translation() on it. This would be on every row... seems a bit hacky...
• RE: Callback Wrappers - You know, it's funny. I originally had it without the wrappers, but then changed my mind. I liked the idea of having it wrapped to keep things open for adding more configuration options down the line (similar to the way taxonomy.inc defines additional options).

And yes... Tests are definitely needed. Should we extend the current mapper tests for field, file and taxonomy or create new ones specifically for multilingual testing?

Some things I've noticed in further testing of my patch:

The language of the entity isn't being set properly. For example, when using the Node Feeds Processor, the entity language is hardcoded to LANGUAGE_NONE ('und') in FeedsNodeProcessor::newEntity(). However, if you create a translated node normally using entity translation, the node language will always be saved as the default language (as far as I can tell). In my case this should have been 'en'.

One solution to this problem is to just add the language code to the source. However, sometimes people don't necessarily have the ability to alter their sources so I'd prefer to come up with a better solution. For now I'd suggest that we add a default language selector to the FeedsProcessor abstract class IF the site is multilingual. At least then people can set the language for the new entities. The only thing is we would then have to make sure the various processor sub classes support this new option. Additionally any contrib modules that provide custom processors (e.g. commerce_feeds) will need to add support, but I guess that was always going to be the case with multilingual.

Another bonus with the language selector is that we're probably going to need something like it for content translation support.

Anyway, if anyone has any thoughts or objections let me know. I'll be moving forward with the plan as it is, but am open to discussion.

drclaw

Related: #1311978: Can not edit nodes imported via Feeds/CSV

#80: feeds-field_translation_imports-1183440-80.patch queued for re-testing.

Hi
I'm currently trying to import nodes with Entity Translation enable. Two fields are enabled: Title_field and body. My default language (and you'll see reading this) : French.
I applied the patch #80: feeds-field_translation_imports-1183440-80.patchof feeds 7.x-2.x-dev (alpha8+7-dev). So I can import French and English Title and Description.
With Devel module, I can see that data is correctly inserted in title_field[en][0]['value'] and title_field[fr][0]['value']
So everything is good to render the fields. The language is correct with this.

The problem is with translation and edition. I made some test with the Target Language : try both FR and EN and array(EN;FR) (exploded with tamper) and I always have only one translation done. When I go to translate tab, I have strange cases

1. lang: English source: (original content) : no translation and when when I add a new one : translated data of fields is replaced with the "original content" (and are the wrong one actually...
2. lang : English source: French : but french translation does not exists.... and when I want to add a new translation the path points to [nid]/edit/Array

In all cases, I only have one translation existing for the node. No other are created even if the data exists in the DB and are correct.

I think there's a little more work on the mapper, so the values are correctely sent to the Entity.. But I don't know how to solve this.

Can someone either adapt the patch ?
or tell how Entity Translations are managed when creating the "Original Source" Content (either [Und] or for a specific language)?

So we can adapt the patch to handle this must have feature for Feeds! :-)

@nfavrod:

I believe the issue you're having is specifically with the title module's title field replacement. I had similar issues when I was setting up my multilingual feeds. After some inspection I discovered that you still need to set a value for the title property of the node as well as the title_fields. From what I can tell, the title module does some trickery with the node form to set the old title property (which is still technically required) in a submit handler (title_field_attach_submit). We need to replicate this in our feed mapper because we're not submitting a form. The solution is pretty simple actually, just add a mapping of your default language column to the core title property of the node. See my attached screenshot for a visual example.

As for amending the patch, I'm not sure if this needs to be accommodated for in feeds, or just documented. I think for now it would be best to just document it since feeds is already big enough to maintain without supporting yet another contrib module.

drclaw

Hello!

Thanks for your post. I actually retried many times and it works!!

What's strange is that I did a lot of tries without Deleting the node.

Finally, the correct mapping setup was :

• setup the hard node->title (not as unique)
• setup the node Title Field for every language
• AND setup a languageOtherwise the node were saved as und and it wasn't possible to edit translations, even if the data were correctly saved in fields

So the only thing that could be improved are some warnings or new logic, when you are about to import node's with translated fields. like:

1. Setup the language as requiered, otherwise the node is stored as UND and it's not possible to assign it to a language afterward. This value must be a two-char minuscule case (Feeds Tamper can help!)
2. Setup the Title Field in the language you're about to import is requiered.
3. Setup node->title could be automatically done if the conditions above are correct: it could be automatically copied in the background from the Title Field of the correct language.

Here we are! Thanks guys for your job and have a nice week-end!

Thanks a lot for the patch in #80, I've been waiting almost for a year for this feature. Hope someone will commit this for the next alpha version.

Don't know if someone mentioned it but for those you are not seeing the language button on some fields like the Body field : it has to be translatable. Go to your Content Type and Manage Fields, modify your field and at the bottom click the Enable Translation link.

Hi @drclaw,
Is there a new patch to review based on #80 and further?
Would love to be able to use language correctly with feeds.
Right now term-values will not show because somehow they need to be language based instead of "und".

thanks for the great work on this in advance!

Greetings,
Martijn

@Summit, any reason for removing the "Needs tests" tag?

Nope, sorry if I did this. This was not on purpose.

Hi, Whats the status on this patch related to the review: https://drupal.org/node/1183440#comment-7589933 ?
Would love to have language awareness within the feeds!
Greetings, Martijn

Hi,

I still need to write tests and make some adjustments based on twistor's comments. Time is tight these days, unfortunately! =(

drclaw

Okay, here's a patch re-roll with changes based on Twistor's input with tests. Here's some quick notes on the changes:

• Fixed the field_info_field($mapping['target']); issue. I didn't understand what twistor meant at first, but I figured it out when the first run of my test failed on a file field (since it uses "sub-field" syntax like field_name:uri). +1 for tests!
• Param docs should be all there now.
• I added a language option to the processor form. New entities were being created either with no language set, or 'und' hardcoded as the language (depending on which processor). It defaults to 'und' so it shouldn't affect existing feeds.
• I changed how translations are added. Instead of calling feeds_mapper_add_translation() on each mapping, I added a new method to the FeedsProcessor class (FeedsProcessor::processTranslations). It gets run before the entity is saved and figures out if translations need to be added. I think this is a far better way to do it because it a) only runs once per entity, and b) it takes the responsibility away from the mapper to manage translations
• I also added a little support for the title module (FeedsProcessor::processEntityTitleReplacements). Basically the title module replaces un-translatable field properties (e.g. $node->title) with regular fields, but it uses some form_alter trickery that we need to mimic. Without this little fix, people have to map their default language titles twice: once to the title property, and once to the replace title_field.
• The test is pretty simple. It just initiates a multilingual site with entity translation enabled and imports one item with multilingual field values (one for each supported field type). The test just checks that each value got correctly assigned to the correct language.

Also, just for reference, the language option and the title module support solves the issues discussed in comments #88, #89 and #90 above.

Thanks!
drclaw

Oops. Right, the automated test won't have entity translation... here's a re-roll that adds entity_translation as a test dependency. I also added the dependency to the test itself (in getInfo()). That should do it...

Issue summary update by drclaw

If you read this issue, you may also be interested in #2146719: Create aliases in all languages when importing with Feeds.

66: TNIDparserAndMapperFix-1183440-64.patch queued for re-testing.

I've tried the patch in #97, but I can't get the importer to create new translations of the nodes. If I manually create a translation of an imported node, and then run the import again, the field values of that language are being imported properly. So it's working to some extent.

I'm using:
Feeds 7.x-2.0-alpha8
Entity translation 7.x-1.0-beta3

Update:

I found the issue. The site had both english and swedish activated. The default language was set to english, but the nodes were set to be created in swedish (which was correct). However, I noticed that the function "processTranslations" from the patch looked at the default language. After changing default language to swedish and running the import again, everything worked. The imported nodes were created as swedish nodes, and english translations of them were added by the importer. The fields we're also filled properly.

In essence, the patch probably needs to be updated in order to handle this situation. Default language may very well be different from the language of the imported items.

Hm. That's a good point. Seems like we should use the language configured on the feed processor here. Here's an updated patch that should do it. Would you mind giving it a try?

Thanks!
drclaw

drclaw, are there any caveats to keep in mind while using this patch?

I don't think so, but it's sort of still pending more review and testing by the community so there might still be things I haven't considered. The only way to know for sure is to test it out! =)

The patch in #104 worked great for me, thanks!

Modified the patch at #104 for the latest dev version, and...

- Remove translations for languages without any field values. By calling removeTranslation() of the appropriate translation handler, we'll remove existing field values properly from the database.

- Check the language property of the entity when determining the default language instead of calling getLanguage(), since the entity of the language may have been specified during import.

Updated for the dev from february 11th, and..

- Use the entity_language() function when checking for the entity language rather than checking the entity property. This property doesn't exist for every entity type, such as taxonomy terms. This has been wrapped within the getEntityLanguage() function, which returns the language that has been configured for the processor if the entity object doesn't have any language support. This means that I renamed the getLanguage() function to getConfigLanguage().

- Don't add translations for undefined field languages.

Patch in #110 worked fine for me.

Fixing status (which will trigger the test bot).

Re-roll of #110

Nice work guys! I tested the most recent patch out and it works. The tests I added a while back got lost somewhere along the way though. I've added them back. =)

I tried the patch from 115 but it didn't create the translation.
I selected the Language: English and the Language: Greek from the mapping.
I'm not sure if I did something wrong. Could you please provide some instructions?

This patch dont work with custom entity's (like the ones created with ECK). Always get a undefined language index and processor settings form wont save.

@soraver: Gonna need a little more info to help

@PedroMiguel: That's interesting. I haven't tested it against ECK entities so I can't really help you there. I'll give it a try when I have some time though and see what I can do.

@drclaw: I can pm you the url and the logins of the site if this helps you

@drclaw, no problem, I end up using other "not conventional" solution for my needs, but as the last dev have support (experimental) to any entity the patch is not ready to apply because break that implementation.

I was using feeds + feed xpath parser, importing xml.

Ps: just to be documented to anyone interested how I did, I just split my feeds and make imports for each one with a default language assigned.

@soraver sure let's try that.

115: feeds-field_translations-1183440-115.patch queued for re-testing.

patch in #115 seems to work just fine.
Thanks for this, good job!

If I run into anything later on, I will definitely let you know.

If we can get one or two more positive reviews with basic use cases, I'd like to see this RTBCed soon. (I worked on this issue 2 years ago!)

For additional issues, such as getting this to work with ECK, let's open follow-up tickets. This thread is quite long, and we're already doing quite a bit in the latest patches.

About Patch #115, Tested on multilingual site (English, Welsh) using Drupal 7.24 and 7.28 Feeds 7.x-2.0- alpha8+33-dev (also Feeds XPath Parser 7.x-1.0-beta5)(also Entity Translation 7.x-1.0-beta3)

Case 1 (working):
Content type with text-field translatable (let’s say field_foo)
Feeds importer + patch, for single lang. I configure: Language:English OR Welsh
Import the xml -> perfect!

Case 2 (no working):
Content type with text-field translatable (let’s say field_foo)
Feeds importer + patch, but I create two mappings for field_foo, one for English and one for Welsh. At XPath XML parser Settings I map the first field_foo to field-foo-eng (the xml tag is < field-foo-eng >) and the second field_foo (it appears twice because we map it two times, one for each language) to field-foo-cym
Try to import the xml->
An AJAX HTTP error occurred. HTTP Result Code: 200 Debugging information follows. Path: /batch?id=23&op=do StatusText: OK ResponseText: Fatal error: Allowed memory size of 201326592 bytes exhausted (tried to allocate 65488 bytes) in \includes\utility.inc on line 29

Tested with File and http fetcher

Can anyone reproduce the issue in case 2? Perhaps @tvl would be willing to post a link to the feed, with any sensitive info blanked out?

I've tried patch #115 and it seems to work, except I still have an issue with the title-field. I'm using three languages, DE, EN and FR. The original language is DE. I'm using xpath-xml-parser. I'm mapping the original title to the node title, and the translated titles to the title-field (each language one).
But I only get the german an french title imported correct. The english title field is filled with the german title...
I changed the order of the fields in mapping, but no difference. I don't get the english title imported.
I'm doing the same with the body field and it works perfect.

I forgot the other two screenshots. (Still confused with the "new" style of posting things here)

Mmmh, don't know what I did, but now its working.
I don't know, if I have to activate internationalization. I first didn't, but after activating it (without any of its submodules) I get all three titles in their correct translation.

It seems $this->config['language'] is defined in FeedsProcessor but not in FeedsEntityProcessor. Adding this so that FeedsEntityProcessorTest passes (it appears the testbot is not running this test?)

--- a/plugins/FeedsEntityProcessor.inc
+++ b/plugins/FeedsEntityProcessor.inc
@@ -135,6 +135,7 @@ class FeedsEntityProcessor extends FeedsProcessor {
       'skip_hash_check' => FALSE,
       'bundle' => NULL,
       'values' => array(),
+      'language' => LANGUAGE_NONE,
     );
   }

@krabbe (#129), I don't know that Internationalization should be a dependency necessarily. If you weren't running the Title module previously, could you see if using that instead of i18n fixes your issue?

130: feeds-field_translations-1183440-130.patch queued for re-testing.

Thanks for that Stefan. I'm impressed with the work on this patch so far, although I now need to go and update my own Excel import module to ensure it works with this patch.

I believe this patch needs a reroll. It fails to apply for me against the latest git clone.

Hunk #2 FAILED at 44.
Hunk #3 FAILED at 63.
2 out of 3 hunks FAILED -- saving rejects to file mappers/text.inc.rej

Rerolled patch to latest git.

Patch works, but tests are failing because they expect LANGUAGE_NONE key in fields.

Suggestion to improve mappers config:
Currently it is possible to select any globally enabled language or LANGUAGE_NONE. It will be more foolproof if it checks field language configuration if field is translatable (require to select language) or not (use LANGUAGE_NONE and hide field).

1. I think tests are failing because it partly undoes the work in #2224643: Support for input format configuration on a per-field basis :

   +++ b/mappers/text.inc
   @@ -70,10 +80,10 @@ function text_feeds_set_target(FeedsSource $source, $entity, $target, array $val
   -        $field['und'][$delta]['format'] = $mapping['format'];
   +        $field[$langcode][$delta]['format'] = $format;
   

2. The patch in #137 also adds two files called text.inc.orig and text.inc.rej which should not be in there:
   

    mappers/text.inc.orig            | 139 +++++++++++++++++++++++++++++++++++++++
    mappers/text.inc.rej             |  62 +++++++++++++++++
   ...
   diff --git a/mappers/text.inc.orig b/mappers/text.inc.orig
   new file mode 100644
   index 0000000..235aea3
   --- /dev/null
   +++ b/mappers/text.inc.orig
   @@ -0,0 +1,139 @@
   ...
   diff --git a/mappers/text.inc.rej b/mappers/text.inc.rej
   new file mode 100644
   index 0000000..88e8d29
   --- /dev/null
   +++ b/mappers/text.inc.rej
   @@ -0,0 +1,62 @@
   

3. +++ b/mappers/text.inc
   @@ -25,6 +25,8 @@ function text_feeds_processor_targets_alter(&$targets, $entity_type, $bundle_nam
   +        'summary_callback' => 'feeds_mapper_summary_language',
   +        'form_callback' => 'feeds_mapper_form_language',
   

   This will not work when "Text processing" is set to "Filtered text (user selects text format)", because then these callbacks will be overwritten later on:

   if (!empty($instance['settings']['text_processing'])) {
     $targets[$name]['summary_callback'] = 'text_feeds_summary_callback';
     $targets[$name]['form_callback'] = 'text_feeds_form_callback';
   }

Currently it is possible to select any globally enabled language or LANGUAGE_NONE. It will be more foolproof if it checks field language configuration if field is translatable (require to select language) or not (use LANGUAGE_NONE and hide field).

Checking the field configuration for translatability sounds like a good idea, but on the other hand, wouldn't that make this feature more limited?

Hello All,

I am using the latest stable release of the module and i applied patch in #80 seems to work fine for me, except when creating new nodes on import. It does not set the node language to the default language so had to make some changes to the FeedsNodeProcessor.inc. I created a form field for language in the node settings form which defaults to the default language see the attached image. The language set in this form field is used for all the translatable fields, so we just need to set the language only once and not for every field.

Also adding a patch for the latest dev release, problem addressed in #136 has also been fixed in this patch.

Thanks
-Manish

Manish Jain: This is not good solution because we need to import multiple languages at once. For example my current configuration looks like this:

Btw, patch #137 does not support body field.

@Bobík: perhaps the body field works only when using Entity Translation?

Updated patch, let's see if this passes tests...

@stefan.r
I think you uploaded the wrong patch, there are only line deletions in there.

At least one of the reasons that the test failed is that the patch in #137 (and the patch in #141 too) partly undoes the work in #2224643: Support for input format configuration on a per-field basis , like I said in #140. Also in that issue, target configuration was added for text fields that have filtered text enabled (which the body field has by default). This conflicts with target configuration that is added for field language, as in the current design only one callback for target configuration per target can be defined.

I hope that this clears things up.

Yes that was the wrong file, here's another try.

@MegaChriz this patch passes tests but perhaps anyone could do a little code & functionality review to be sure this works well with #2224643: Support for input format configuration on a per-field basis ?

@stefan.r
I'll plan to do so, but it might take a while as I'm less experienced in the multilingual thing. With a first glance at the patch in #148, it looks like the language configuration would work in combination with the text format configuration for filtered text enabled text fields. Thanks for your work.

@stefan.r: Bad diagnosis, sorry. It did not worked with formatted long text fields. But with your latest patch it works :-)

I've added option to use $entity->language property as field language to support use case with import table like this:

lang	title
-----------------------
en		English content
cz		Český obsah

Then user can map "lang" column to Language mapper and select Use language property in Target configuration.

I did not wrote test for this feature, because I am quite lost in this tests.

Hi! You seem to be doing a great job. Thanks!!! In my opinion, this is a very necessary feature to feeds.

I'm new to this and I also need to import some bilingual feeds. Nothing too sophisticated: title and description in Spanish and English. I found some hope in this thread but after reading 155 comments I can't make myself clear on what patch I should apply to the feeds module that would make the thing. I'm trying Bobík's last one, but I don't know what's the file I should provide to the patch. Is there another module I should install as well?

Thank you very much for any clue on this.

#155 Seems to be doing the trick for me! Wonderful :) I can import nodes with default languages!

q./

@dpico you can apply the patch against the dev version of Feeds.

Here's an interdiff for Bobik's patch in #155.

I have not tried it but just so I understand his explanation correctly, one field in the feed will assign a language to a record and any other fields within that record that have the "Use language property" option selected will use that same language? And just to make sure this this is something we want, there is no other way to accomplish that, without changing the feed?

@stefan.r: Correct. It is useable for feeds where language of every record is in one of rows. But not only that, you can still set other fields by setting exact language in mapper.

If there is no other way to do that, it seems useful in order to import feeds with records in multiple languages.

In any way the current description is: "You can select particular field language (or language nautral) or use language code saved in entity's language property (e.g. $node->language) you set by some mapper above.", the select option is called "Use language property", and the "All languages" option is being renamed to "Language neutral". The wording of these is confusing, maybe someone can think of something clearer? This feature will need tests too.

Lastly I am wondering if this is really is a must-have in terms of this issue? If this is only a "nice to have", or there already is a possible workaround (Feeds Tamper?) I'd suggest o get something more basic basic RTBCed (such as the patch in #148), and put this feature in a separate issue as it is introducing quite some further complexity and still needs a bit of work (and testing). This issue is 3 years old already and I would like to see something committed at some point :)

Why not to commit current state? (my patch :-))

Yes it needs tests, but is it problem?

The current mapping API is a mess and we shouldn't make every mapper have to deal with this.
#2333029: Extend mapping API to allow for defaults and multiple callbacks.
This is "postponed" on the one above, which I should hopefully have done in the next couple days.

Just experimenting to see if I'm on the right track with #2333029: Extend mapping API to allow for defaults and multiple callbacks..

As I'm browsing through the code, I'm struck that there are too many things happening at once.
It would be nice if we could break out, title field, and entity_translation into separate issues. I'm not sure about the entity_translation, I have to dig into processTranslations() to see what it's trying to do.

Consider this a git stash.

Is there a reason we're using locale_language_list() in some places and language_list() in others. It seems to me, that if locale.module isn't enabled, there should be no need for translation at all. I'm probably wrong though.

Why were the hooks_feeds_processor_targets_alter() function changed to hook_feeds_processor_targets()?

See text.inc, taxonomy.inc etc. These aren't invoked anywhere and doing this breaks the import.

Edit: Disregard this comment, I see what's happening.

Can anyone help me out? I'm not so familiar with applying patches.

For some reason the patch won't apply to my Feeds.
I used feeds-field_translations-1183440-165-do-not-test.patch

I'm running Feeds 7.x-2.0-alpha8 (not the DEV version)

When I scan the patch and compare it to my files, the code is completely different.

Tried aswell with DEV but same results.

error: patch failed: feeds_ui/feeds_ui.admin.inc:635
error: feeds_ui/feeds_ui.admin.inc: patch does not apply
error: patch failed: mappers/taxonomy.inc:96
error: mappers/taxonomy.inc: patch does not apply
error: patch failed: plugins/FeedsProcessor.inc:669
error: plugins/FeedsProcessor.inc: patch does not apply

Thanks in advance

@onotic:

git clone --branch 7.x-2.x http://git.drupal.org/project/feeds.git
cd feeds
git checkout a6abe50
wget https://www.drupal.org/files/issues/feeds-field_translations-1183440-155.patch
patch -p1 < feeds-field_translations-1183440-155.patch 

This gets the git repo from back in august and applies the patch in #155 in it (I think the ones in #165 and #166 were just experiments)

@twistor: any pointers on further work you would like to be done on the patch in #166?

@stefan.r

Thanks a bunch! It works :) Also happy to see it works with commerce products.
You have any idea if and when there will be a new version of the feeds module implementing this patch?

I've tested the patch (#155) and tried to import taxonomy-terms with a name_field in 4 languages. Its works fine as long as the default language of the site is set to English. If I set German as the default language, then all English fields are filled with the German term-name.

Any ideas?

Thanks!

It seems the problem is caused by title module - title_entity_presave() sets current active language as the source language for the title field. Some really complicated crap, imho.

In practice one can avoid this problem, if you start the feeds import when your default language. E.g. If German is your default language, trigger the import from de/import/your_importer

I test the patch #166. It doesn't work for Inhereted Language.

In FeedsProcessor.inc, the patch add:

foreach ($mapping['preprocess_callback'] as $preprocess) {
          $preprocess($source, $target_item, $mapping['target'], $value, $mapping);
}

which must be

foreach ($targets[$this->id][$mapping['target']]['preprocess_callback'] as $preprocess) {
          $preprocess($source, $target_item, $mapping['target'], $value, $mapping);
}

The patch works correctly for me after this correction (including with targets of others modules).

But, apparently, this same code is add by the patch on issue 2333029 too. So, the patch of issue 2333029 must be apply before the patch #166 and the code between these issues shouldn't be repeat.

I add the patch #166 with my correction for now but isn't the best practice.

Patch without the modifications provided by issue 2333029. You must apply the patch number 18 of this issue before to apply this patch.

I've always 2 bugs: In administration page of importer, I can't configure some targets like textfield with html filter, I have an AJAX error. I suppose it's because I have 2 parameters on the target's configuration.

If I use Feature to configure this information, I have an other bug when I verify the content of $entity: The field have 2 languages. The inheret language and the undefined language. But for undefined language, value is empty.

Just a little correction in the name of preprocess.

For my understand i use the latest Feeds dev release and i need for first Patch #18 and and then your latest Patch?

I have an issue with updating Nodes/Entities to add translations with a second import.
https://www.drupal.org/node/2350443

I use at the moment the Feeds Entity Translation module

@neofelis1985 yes but that is patch #18 from this issue: #2333029: Extend mapping API to allow for defaults and multiple callbacks.

Alternatively you can use the patch in this comment which combines both patches (so that the testbot can pick it up).

i am confused - which patch can i use?

@ali_b, I would go with #148 as that is the last patch that passed tests.

re-roll of #148

thanks stefan, after apllying patch 148 I got error:
Parse error: syntax error, unexpected ''term_search'' (T_CONSTANT_ENCAPSED_STRING), expecting '(' in .../sites/all/modules/feeds/mappers/taxonomy.inc on line 300

Hi Ali_b , I also noticed that the recent commit 45152d374e6 (committed friday Jan 16th) breaks patch 191 from this issue

The workaround is to apply patch 191 on commit 006234fc3cc859e (Tue Dec 23 13:54) rather than the latest commit.

Here is what you have to do to test it:

git clone --branch 7.x-2.x http://git.drupal.org/project/feeds.git;
cd feeds;
wget https://www.drupal.org/files/issues/feeds-field_translations-1183440-191.patch;
git checkout 006234fc3cc859e4fec0cb261675577efcf389b8 --hard;
git apply < feeds-field_translations-1183440-191.patch

What has to happen now is for someone to refactor (reroll) patch 191 but using the latest commit from dev.
That doesn't mean we can't test patch 191 , if you want to test it, use the steps above with the previous commit from dec 23.

Ali_b, please test out the patch using the above steps, and let us know how it works.

Updating status based on #193.

I applied patch 191 on commit 006234fc3cc859e (Tue Dec 23 13:54) as was suggested and tested using taxonomy term processor. Having two name_fields (Title module) with different language assigned produces 'An AJAX HTTP error occurred. HTTP Result Code: 500 Debugging information follows. Path: /batch?id=.....' error when trying to import. Using only one name_field the import works including using a custom entity field wilth different translations mapped. Looked at the multilingual.csv from the patch (test) and see that the title should be translatable etc. Not sure multilingual taxonomy terms has actually been tested yet with the patch 191 or is a seperate issue.

When will this go into the dev version? I have no git and i need it urgent..

Does anyone have an update on nsurround's question ?

I have a taxonomy using entity translation.
Title and description of the terms ar translated using the Title module.

I would like to be able to use feeds to import terms for this taxonomy from a csv file with these fields for example:

• term_id
• term_name_en (this is the English term name)
• term_name_nl (this is the Dutch term name)
• term_description_en
• term_description_nl

When using content translation (not entity translation) I can comfirm that everything stated in #76 works.
I also found these instructions where I found the feeds translation module: https://github.com/CityofOttawa/feeds_translation

So following the instructions I was able to import and translate nodes from a csv file using Feeds and Feeds Tamper.

Next I will try to tackle my taxonomy with entity translation problem and see how that goes.

I'm happy to say that, after testing, using entity translation and Feeds also works for me :-). Using latest dev of Feeds, Feeds Tamper and also the Feeds Translation module.

For content translation you need to create 1 importer per language an follow instructions on #76. Each field must be included only once but need to have the correct language set.

For entity translation you only need 1 importer, and readd each translatable field for each language in that importer.

flyke, were you able to use entity translation with only 1 importer and be able to translate the taxonomy term name (title property). I was able to translate any custom field added to a term but not its term name title. For instance EN term name 'Red' could not translate to ES 'Rojas' etc using entity translation, feeds and taxonomy term processor. If you were successful with this could you explain a bit more the method or module versions used? Thanks for you input.

I'm hoping to also confirm the method in #1183440-199: Multilingual Feeds - Make field import language-aware with Entity Translation, of using the latest dev snapshot of Feeds and Feeds Tamper along with the Feeds Translation module attached in #1183440-76: Multilingual Feeds - Make field import language-aware. Fingers crossed.

I won't be looking at the Content Translation implementation as that's not the route being taken in this multilingual client project.

flyke: just to clarify, no actual patches were applied, correct?

I can confirm that core (Content Translation) translation method works with Feeds if you follow #76. Actually, you can even make it easier and skip Feeds Tamper part. In that case, here are the steps necessary:

Usage

1. Create a English Feeds Mapping (or language or your choice) and use the following fields in this example:
GUID (if you don't use Title as unique field), Title, TNID, language

2. Create a French Feeds Mapping (or language of your choice) -> and use the following fields in this example:
GUID (if you don't use Title as unique field), Title, TNID, language

3. Run the English Feed

4. Run the French Feed

- Make sure to use the Language field so the mappings have a Language applied.

Here is an example of the csv file without GUID being used:

Title (En)|Title (Fr)|Description (En)|Description (Fr)|Language(En)|Language(Fr)|TNID
English Title|French Title|English Description|French Description|en|fr|100

I have tried the #76 but the tnid of the French nodes is 0.
I have noticed that the Feed items' GUID is a URL (unique for each item).
The English feed item and its French translation have the same GUID though.
Does the GUID has to be numeric in order for the plugin to work?
If yes, how it can be modified to check for strings?

I've been reading up on this thread, and I'm missing what about when the feed, or feed element itself is specifying the language?

RSS allows specifying the language of the whole channel.
ATOM uses xml:lang which can be granular to the element.
iCal has a LANGUAGE property that can also be granular to the component.

Unlike CSV, or similar, where you have set up to have one column/field for one language, and another for another, and where you can then make multiple mappings; you have to map your DESCRIPTION to Body, or whatever, and allow the feed to specify the language - which can change per feed or field - all using the same configuration.

What seems obvious coming from this perspective is that the FeedsElement should have a property for language that could be set by the parser. Then the mapper can use this (as appropriate for the field).

How would this fit with the above configuration solutions?

@ekes that would work, but it assumes people have control over what their feeds look like. Multilingual elements don't always have a language property.

@ekes that would work, but it assumes people have control over what their feeds look like. Multilingual elements don't always have a language property.

At which point the FeedsElement::language remains LANGUAGE_NONE and the mapper continues as it is.
With the exception of the above discussion where the user is configuring this when configuring the Processor Mappings?

@ekes

I believe you can use Feeds tamper to set a default value on a language field. Feeds Tamper's plugin name is "Set value or default value".

I meant if the feed looks like this, maybe we don't want the language of all the elements to be LANGUAGE_NONE:

<feed>
  <item>
    <element_en>Hello</element_nl>
    <element_es>Hola</element_es>
  </item>
</feed>

We'd need to use something like Feeds Tamper to transform the format of the feed...

In any case this issue seems a bit stuck, maybe the Feeds maintainers can give some direction?

I think that #2333029: Extend mapping API to allow for defaults and multiple callbacks. first should get in, in order to fix this issue properly. I think we need to have the possibility to have multiple summary/form callbacks, for the reasons noted in #147.

Another reason this issue is hold back is because getting a new release out has more priority, see #2290465: [Meta] Create beta 1 release. To get this out sooner, a proposal is made to remove the generic entity processor, see #2469219: Remove the Generic Entity Processor.

With latest dev feeds and feeds_et I have managed to import a multilingual taxonomy vocabulary without applying any patch.
I just had to create a taxonomy with localize translation mode and replace the name field with an entity field.

The source for my taxonomy vocabulary has the following format:
term name | translation 1 | translation 2

The importer uses term name for both target Term name (name) used as unique and Name [English] (name_field:et:en)

I have two feeds, one in English and one in French where the GUID in both languages is the same. When I import these feeds I create nodes of a specific content type. I want somehow to link the English and French nodes and make them aware of the translation so I can use the language switcher when I read the article to switch to its translation.

None of the above patches or custom code worked for me so I created my own module and got my feeds translated.
I only used feeds tamper module so I can add the language to the target. I used blank source in the source and language field at the target and then in feed tamper I used a rewrite for that field with the value set to en for the English one and fr for the French one.

The fields that I used for both feeds (Source -> Target):
Item GUID -> GUID (Used as unique), Blank source -> Language, Title -> Title, Description -> Body, Published date -> Published date.

Here is my module's code. Sorry but I was too lazy to tidy it up :) It works very well though.

function my_module_feeds_after_save(FeedsSource $source, $entity, $item, $entity_id) {
  if ($entity->type == 'my_content_type' && $entity->language == 'en') {
    //When importing from the English feed
    $nodeid = db_query("SELECT entity_id FROM {feeds_item} WHERE guid =" . $entity->feeds_item->guid . "")->fetchField();

    if ($nodeid) {
      $node = node_load($nodeid);

      if ($node->language == 'fr' && $node->tnid == '0') {
        $ennode = node_load($entity->nid);
        $ennode->tnid = $ennode->nid;
        node_save($ennode);

        _add_link_item($ennode, 'your_parent_link_id', 'en');

        $node->tnid = $ennode->nid;
        
        /* Updates the French node's alias path with the English one. On my site we use only aliases in English.
        Skip this line if you want your alias in French. */
        $node->path = array('alias' => $ennode->path);
        
        node_save($node);
      } else {
        _add_link_item($node, 'your_parent_link_id', 'en');
      }
    }
  } else if ($entity->type == 'my_content_type' && $entity->language == 'fr') {
    //When importing from the French feed
    $nodeid = db_query("SELECT entity_id FROM {feeds_item} WHERE guid =" . $entity->feeds_item->guid . "")->fetchField();

    if ($nodeid) {
      $node = node_load($nodeid);

      if ($node->language == 'en' && $node->tnid == '0') {
        $node->tnid = $node->nid;
        node_save($node);

        $frnode = node_load($entity->nid);
        $frnode->tnid = $node->nid;
        node_save($frnode);

        _add_link_item($frnode, 'your_parent_link_id', 'fr');
      } else {
        _add_link_item($node, 'your_parent_link_id', 'fr');
      }
    }
  }
}


//This function adds a menu link to the feed item
function _add_link_item($node, $plid, $lang) {
  $linkitem = array(
    'menu_name' => 'main-menu',
    'link_title' => $node->title,
    'link_path' => 'node/'. $node->nid,
    'plid' => $plid,
    'language' => $lang,
  );

  menu_link_save($linkitem);
}

Let's see what happens.

This patch includes #2333029: Extend mapping API to allow for defaults and multiple callbacks.. If someone wants to review that, I could get to this fairly quickly.

I've applied successfully the patch at #214 to the dev version of the module.
I got this, though, in an installation with many other drupal plugins:
PHP Fatal error: Call to protected FeedsImporter::__construct() from context 'FeedsPlugin' in ...feeds/plugins/FeedsPlugin.inc on line 59
In a clean install i'm not able to reproduce the error and feeds seems stable, instead.
Thanks,
Da.

@planctus, are you sure that was from this patch at not updating the dev version of Feeds? It might be related to #2497507: Pass plugin definition to FeedsPlugin objects..

Can you post the full error message in that issue? It should be easy to track down.

I think you're perfectly right @twistor, i also had to upgrade feeds to apply the patch, indeed.
Da.

Alright! #2497507: Pass plugin definition to FeedsPlugin objects. was committed, that makes this patch much smaller.

Still needs tests.

Could really use some people manually testing as well.

The patch applies flawlessy, i've no clues about how to test a multilingual import with this new patch, though. Thanks.

@twistor There is a test called FeedsMapperMultilingualFieldsTestCase in the patch from #191.

@planctus
Things to test include:

• Behaviour with locale module turned on and off.
• Behaviour with Content Translation module turned on and off.
• Behaviour with Entity Translation module turned on and off.
• Behaviour when importing to content (nodes, terms, users) that are *not* multilingual.
• Behaviour with the various multilingual settings, for example:
  • Require language (do not allow language neutral) setting turned on and off.
  • Lock language (cannot be changed) setting turned on and off.
  • Limited amount of allowed languages vs all languages are allowed.

There are lots of things to test. Maybe it would be a good idea to come up with a testplan that describes a few use cases with steps to follow. Or, as a start, only describe the use cases and let the testers figure out the steps to take.

@MegaChriz, Good to know, I'll roll that in.

As for what to test:

• This patch doesn't deal with Content/node translation at all.
• It does support locale and entity translation.
  • Locale support allows you to assign a language to nodes and other entities will need minor patches to add support.
  • Entity translation support will allow you to assign a language to any translatable field.
• I haven't worked on the "Require language", "Lock language", or limited languages at all yet.

@Twistor and MegaChriz

Were there any changes in the previous releases that have made the patch in #76 redundant?

I'm using Feeds 7.x-2.0-alpha8+85-dev and I have managed to update TNID's just by installing Feeds translation module from #76. Patch wasn't necessary and translated nodes are connected properly to original nodes.

If this newest patch (#219) deals only with entity translation, please consider adding core translation support as well. #76 seems like an approach that's working well.

i d like to thank twistor for the patch

I applied the patch to the latest dev version (2015-Jun-04) and when i tried to use it it threw this error
An AJAX HTTP error occurred. HTTP Result Code: 500 Debugging information follows. Path: /batch?id=42&op=do StatusText: Service unavailable (with message) ResponseText: Recoverable fatal error: Argument 3 passed to locale_feeds_preprocess_callback() must be an array, string given in locale_feeds_preprocess_callback() (line 19 of /sites/all/modules/feeds/mappers/locale.inc).

i ve checked with watchdog debug prints and the arguement is string indeed.
I ve removed the array type in front of the $target argument (file feeds/mappers/locale.inc)
function locale_feeds_preprocess_callback(FeedsSource $source, $target_item, array $target, array &$mapping) {
to
function locale_feeds_preprocess_callback(FeedsSource $source, $target_item, $target, array &$mapping) {

and the module seems to work as expected
I m not quite sure what the implications are however

Thanks, that is a bug indeed.

Updated patch for the current dev.

So this just needs tests now? Any pointers on what would need to happen for this to support content translation as well?

Any pointers on what would need to happen for this to support content translation as well?

That should be a separate issue. I honestly haven't thought about it yet. Since content translation uses separate nodes, it sounds like it would be a very different solution.

Can anyone figure out why testMultilingualFieldMappings() needs to do $this->drupalGet('node/1/translate'); before the French translation will show up in the test? I'm stumped.