Associate feed with translation of a node in addition to language

Subscribe. I'm looking at a similar situation: an XML feed with bi-lingual records, so each imported record should result in two nodes.

subscribing

cross-referencing #840142: How can the created nodes be assigned their correct language? (and other "core" fields...), where work is being done on supporting content translation.

I don't know if there is a better way to solve this, but for now, I tackled my problem with a hook_feeds_after_import implementation.

As far as I can see, there is no way the parser can know the future ID of the node that is being created. As a result, the parser cannot set the tnid value for that node - at least in my case, because both the source language and its translation are imported simultaneously.

My importer created nodes of type 'object' and every object has an ID (stored in field_object_number) that is identical across its translations. The code below runs after finishing the import, because that's when the node id's are available.

function MYMODULE_feeds_after_import($importer, $source) {
  // Select dutch nodes and their object numbers from the database
  $base_nodes = db_query("SELECT cto.nid nid, cto.field_object_number_value fonv FROM {content_type_object} cto JOIN {node} n ON cto.nid = n.nid WHERE n.language = 'nl'");
  // For every result, set other nodes with the same object number as translations of the dutch object node.
  while ($row = db_fetch_array($base_nodes)) {
    db_query("UPDATE {node} n JOIN {content_type_object} cto ON n.nid = cto.nid SET n.tnid = %d WHERE cto.field_object_number_value = '%s'", $row['nid'], $row['fonv']);
  }
}

I hope that helps anyone.

My feed is huge, and I cannot quote from it publicly, but I'll give you a simplified example:

<exportXML>
  <recordList>
    <record>
      <object_number>a.001</object_number>
      <image>
        <value lang="">a001_1.jpg</value>
      </image>
      <image>
        <value lang="">a002_2.jpg</value>
      </image>
      <title>
        <value lang="en">This is an object</value>
      </title>
      <title>
        <value lang="nl">Dit is een object</value>
      </title>
    </record>
    <record>
      <object_number>a.002</object_number>
      <image>
        <value lang="">a002_1.jpg</value>
      </image>
      <image>
        <value lang="">a002_2.jpg</value>
      </image>
      <title>
        <value lang="en">This is another object</value>
      </title>
      <title>
        <value lang="nl">Dit is nog een object</value>
      </title>
    </record>
  </recordList>
</exportXML>

The example above results in 4 nodes: 2 English (en) and 2 Dutch (nl). The code from comment #5 makes sure that every English node is marked as translation of the Dutch node with the same object_number.

I think you're right about that. If you use hook_feeds_after_import to set the translations, it doesn't matter how and when the nodes got into the database. It would even be fine if one language is imported on one day and the translation arrives three days later. All you need is a way to identify which nodes are translations of each other (like the object_number in my case).

In fact, the code in #5 is not really dependent on Feeds module at all. It doesn't use any of the parameters passed to hook_feeds_after_import. It would run just as well in a hook_cron implementation, but in this case, hook_feeds_after_import just makes more sense.

That seems like the way to go, yes. Just like there is a 'unique target' flag right now when you configure the mapping for your feed, there could be a second checkbox called 'unique per language' or something like that. I assume that a hook_fields_after_import implementation can access that property and act accordingly.

As far as I know, there is no API function to tell Drupal "node x is a translation of node y". That's why my code updates the node table directly.

I would be interested to know if the Feeds maintainer will accept a patch that works like this...

I was just going to post a suggested solution like this. I realized I would have to use a different field for the unique ID if it has to be unique even for translations. But it would be perfect if you could just check any field to be used as translation mapping -- it could be the GUID or something else.

I'm more than happy to test/review/contribute to a patch as I need this ASAP... I'll start by trying to implement the code snippet and take it from there.

It would be great to see more multilingual support in Feeds!
m

I just came across this thread & patch for nodeferences with feeds - http://drupal.org/node/724536 - maybe that code could be adjusted to work for creating translations? It would just be a few tweaks (ie match on a given field instead of node title).

I'm also trying to adjust the code in #5 to only update the items imported (instead of everything) using $source->batch->items

Anyway, just some leads I thought I'd share as I'd love to see this move ahead quickly and it's a little beyond me...

I didn't know feeds_tamper yet, but it seems like a nice little framework to plug into (even though alpha 4 may not be too stable yet). However I'm not too sure about using Views to do the query; maybe I'm wrong but it sounds like a lot of overhead for a simple query.

It's hard for me to judge if plugging into feeds_tamper is in any way faster, easier, more maintainable or SimplyBetter^TM, but I would love hear the what the feeds_tamper maintainer thinks about this approach.

I have done something similar for feeds_tamper that handles node references when the feed items reference each other instead of existing nodes. I'll have to think on this a bit, just wanting to subscribe.

Thanks twistor, I'd be interested to hear your opinion on the best approach. I'm still not 100% sure about the best place to hook into the import process.

Well, here's how I ended up doing it. There are a couple of caveats : you need to import the base-language (in my case French) first, and you need to not mind if the GUIDs change (for me they are only used during import).

Issues:

*Even though I can set that the tnid is set in the node table, it isn't recognizing that there is a translation -- am I missing something?
*If I could use hook_feeds_after_import, both languages could be loaded at the same time, but it doesn't seem to have access to $source->batch, suggestions?

If feeds_tamper ends up being the way to go, I'm happy to jump on that bandwagon, I just need something that works for me ASAP and have not worked with that before...

>
function feeds_translations_feeds_after_parse($importer, $source) {
  $base = "fr";         //TODO make this config

  $items = $source->batch->items;
  foreach ($items as $key => $item) {
    if ($item['language'] != $base){
      //Get nid of base node
      $item['tid'] = db_result(db_query("SELECT n.nid FROM {feeds_node_item} f, {node} n WHERE f.guid = '%s' AND n.language = '%s' AND n.nid = f.nid", $item['guid'], $base));
      //make non-base language GUIDs unique by appending language code
      $item['guid'] .= $item['language'];
     }

   $items[$key] = $item;
  }

  //put everything back
  $source->batch->items = $items;
}

Even though I can set that the tnid is set in the node table, it isn't recognizing that there is a translation -- am I missing something?

Did you also set the tnid value for the base node? Your code looks as if you're only setting the tnid value on translation nodes.

Awesome! That's probably it.
I'll test a fix tomorrow and post new code in case it's of help to anyone else.

Thanks again for the quick reply. I'm sort of feeling my way around in the dark here... I couldn't find a good example how to do this.

That did the trick.

Just need to add this line:

db_query("UPDATE node SET tnid = '%d' WHERE nid = '%d'", $item['tid'], $item['tid']);

after setting the $item['tid']

Let me know if anyone thinks it's worth generalizing this...

thanks again.

A quick example to show you what a plugin might look like. Put this in set_tnid.inc in the plugins folder of feeds_tamper.

<?php

$plugin = array(
  'form' => 'set_tnid_form',
  'callback' => 'set_tnid_callback',
  'name' => 'Set translation node id',
  'multi' => 'skip',
  'category' => 'Other',
);

function set_tnid_form($importer, $element_key, $settings) {
  $mappings = $importer->processor->config['mappings'];
  foreach ($mappings as $mapping){
    if ($mapping['target'] == 'language') {
      $form['language_source'] = array('#value' => $mapping['source'], '#type' => 'hidden');
      break;
    }
  }
  $form = array();
  $options = language_list();
  foreach ($options as $key => &$value) {
    $value = $value->name;
  }
  $form['base_language'] = array(
    '#type' => 'select',
    '#title' => t('Base Language'),
    '#options' => $options,
    '#default_value' => isset($settings['base_language']) ? $settings['base_language'] : language_default('language');
  );
  return $form;
}

function set_tnid_callback($source, $item_key, $element_key, &$field, $settings) {
  $item = $source->batch->items[$item_key];
  if ($item[$settings['language_source']] != $settings['base_language']) {
    $query = "SELECT n.nid FROM {feeds_node_item} f, {node} n WHERE f.guid = '%s' AND n.language = '%s' AND n.nid = f.nid";
    $field = db_result(db_query($query, $item['guid'], $settings['base_language']));
    $item['guid'] .= $item['language'];
    db_query("UPDATE {node} SET tnid = %d WHERE nid = %d", $field, $field);
  }
}

This gives you configuration and exportability(Features) for relatively cheap.

I haven't tested this, I'm not set up here to. I'm also not very familiar with translations in general.
This could be generalized more. The GUID field could any unique field. This requires #840142: How can the created nodes be assigned their correct language? (and other "core" fields...) correct?

subscribe

any update on this? can the plugin from #24 be applied to d7 ?

i've spent some time thinking about this today for a client and have a short-term fix plus a suggestion for further work.

feeds_tamper is a fine module but it runs when a single field is being imported and doesn't have enough context to figure this out. we have to set node->tnid for both source and translation but that can't happen until later in the process after at least the first has been created.

i think solving this correctly would require adding a new mapper to feeds/mappers/, like the nodereference patch does in #724536: Mapper for nodereference field in Drupal 6. we would also need to store information about the source language node in the feeds_node_item table which we can reference later when importing the translation. i think this is reasonable since translation is a core module. but, the module maintainers should probably give their opinion on that, so i'm not going to start that work now since they might disagree with my approach.

so, in the mean time, i've solved this for my use case with the hook_feeds_after_import() approach. my code is below, and i've documented the assumptions it makes (ie, what you would need to do before you can use it).

@kervi: this thread is discussing the problem for D6; as i imagine the solution would be different for D7, perhaps you could search for or start a new thread for that conversation, and reference it here?

/*
 prerequisites:
 * the site is in drupal 6
 * the site only contains content in two languages
 * our feed importer is named resolutions
 * we are importing to the content type resolution in content_type_resolution
 * resolution has a single-value CCK integer field named
   content_type_resolution.field_translation_id
 * the data source contains a GUID field which is unique for each node and is
   set as unique target in the node processor mapping options
 * the data source contains a translation id field which is unique for each
   combination of language and node
 * two nodes with the same translation id are translations of each other
 * nodes in the default language will be imported first, translations second
 * if the node processor option "Replace existing nodes" is set, and the
   default language nodes are re-imported, then the translated nodes will also
   be reimported (not necessary when using the "update existing nodes" option)

 see also: drupal.org/node/997858
*/

function MY_MODULE_feeds_after_import($importer, $source) {
  $id = $importer->fetcher->id;
  $type = $importer->processor->config['content_type'];
  $language_default = variable_get("language_default", FALSE);
  if (!$language_default) {
    return;
  }
  if ($id != 'resolutions') {
    return;
  }
  $language_default = $language_default->language;
  // note: set the variable feeds_debug to TRUE to see these messages in /tmp
  feeds_dbg("in MY_MODULE_feeds_after_import() id=$id");
  // select default language nodes which were imported from this importer, plus
  // their associated translations, matching on field_translation_id
  $base_nodes = db_query("select n.nid as l1nid, tnid, n.language, title,
      ctr.field_translation_id_value as tid, ctr2.nid as l2nid from {node} n
      inner join {feeds_node_item} fni on n.nid=fni.nid left join
      {content_type_resolution} ctr on n.vid=ctr.vid inner join
      {content_type_resolution} ctr2 on ctr.field_translation_id_value =
      ctr2.field_translation_id_value where n.type = '%s' and fni.id = '%s' and
      n.language = '%s' and ctr2.nid > ctr.nid", $type, $id, $language_default);
  // set nodes as translations of the default language node
  while ($row = db_fetch_object($base_nodes)) {
    feeds_dbg("using $language_default node {$row->l1nid} as translation source for translation node {$row->l2nid}");
    db_query("update {node} n set n.tnid = %d where n.nid = %d or n.nid = %d", $row->l1nid, $row->l1nid, $row->l2nid);
  }
}

Closing this as a dupe of #1183440: Multilingual Feeds - Make field import language-aware. Any work done in that issue will have to be ported back to 6.x