FeedAPI Mapper 2 supports not only feedapi_node but any mapping target and it supports defining the uniqueness of arbitrary feed elements.

The paradigm changed between v1 and v2 from FeedAPI Mapper being completely transparent to FeedAPI processors (such as feedapi_node) - to shifting the responsibility of mapping to the processor itself. So while we did the mapping before in feedapi_mapper_nodeapi() when feedapi_node saved a feed item, now we need to do the mapping in feedapi_node itself.

The advantages of shifting mapping into processors are twofold:

1) we will be able to completely control how a processor maps feed elements to targets. For instance, with feedapi mapper 1 we can override what should be mapped to a node's body, but we can never completely remove the mapping from the feed item description to the node body - because it is hardcoded in feedapi_node. Related issue: #518180: Full overridability of node title, body, etc mapping

2) we will be able to define any element of a feed item as 'unique' - right now, feedapi_node just assumes that URL and GUID is unique, thus parsers have to always generate a GUID whether or not what they're parsing does actually merit a GUID. Further, many cases require GUID not to be unique.

TASKS:

A Remove feedapi_mapper_nodeapi() from feedapi_mapper()

B In _feedapi_node_unique() query feedapi_mapper_get_uniques() and determine the uniqueness of an item accordingly. This task will require a way to delegate the determination of whether something's unique or not to the specific implementation of a mapper: feedapi_node does not know anything about the storage of the mapping targets returned by _feedapi_node_unique(). I. e. we will have to come up with a $op = 'unique' for hook_feedapi_mapper() that lets the mapper implementation (e. g. feedapi_mapper/mappers/feedapi_mapper_content.inc

C In _feedapi_node_save() query feedapi_mapper_map($feed_node, $item) on an $item to build the node object for the feed item to save.

D (not necessarily part of this issue) As soon as feedapi_mapper 2 supports exportables, we should provide a default mapping by feedapi_node so that a user does not need to configure the mapper for getting sensible defaults. A stop gap solution here would be to fall back to the existing assembly of the feed item $node object if feedapi_mapper_map() does not return a useful result.

Notes:

We should aim for loose dependency = making feedapi_node not dependent on feedapi_mapper. If we do that by testing for function_exists() rather than module_exists() we should also take care of feedapi_node being compatible with both feedapi_mapper 1 and 2, because both functions feedapi_mapper_map and feedapi_mapper_get_uniques don't exist in feedapi_mapper 1.

Comments

aron novak’s picture

Status: Active » Needs work
StatusFileSize
new1.71 KB

I attached a patch that contains my first sight ideas.
A) Alright, but that function lives in the mapper. The comment in feedapi_mapper.module is a little bit misleading, this prepare op is not needed to move to feedapi_node's nodeapi implementation, but it should
B) Generally i like the idea of being able to override the logic of unique-ness, it's an old demand. I would like to propose a different behaviour, just as the _map() function. "delegate the determination of whether something's unique or not to the specific implementation of a mapper" - this should be the responsibility of the mapper, as in the case of mapping. See the patch, i think what i did here should be ok with the whole new concept.
C) Done.
D) Agree, task for the near future.

Loose dependency FTW, sure.

alex_b’s picture

A) - yes, only the functionality of feedapi_mapper_nodeapi() should be moved - not the hook implementation itself (A would remove hook_nodeapi(), C would add the functionality currently done by the hook_nodeapi() implementatin in feedapi_mapper into feedapi_node)
B) This is looking good. I would change the comment to "// Delegate to feedapi_mapper if available". We will need to think how we should implement the delegation of determining the uniqueness of an item in feedapi_mapper: I opened an issue on for that: #539752: Support 'unique' conditions.

alex_b’s picture

Created an issue for A on the FEMP issue tracker #539758: FeedAPI Node compatibility

aron novak’s picture

I created a new patch what really handles the return values from unique function from the mapper. See #539752: Support 'unique' conditions also.

aron novak’s picture

The tests will fail surely, because while the tests running, feedapi_mapper_map is an existing function, but no default mapping is defined.

alex_b’s picture

You can simplify the if/else statement in _feedapi_unique():

if (function_exists('feedapi_mapper_unique')) {
  $feed_node = node_load($feed_nid);
  $dup_items = feedapi_mapper_unique($feed_node, $feed_item);
  if (empty($dup_items)) {
    return TRUE;
  }
  elseif ($settings['x_dedupe']) {
    foreach ($dup_items as $id) {
      $feed_item->feedapi_node->duplicates[$id] = $id;
    }
    return array_pop($dup_items);
  }
  else {
    return array_pop($dup_items);
  }
}

Can be:


if (function_exists('feedapi_mapper_unique')) {
  $feed_node = node_load($feed_nid);
  $dup_items = feedapi_mapper_unique($feed_node, $feed_item);
  if (empty($dup_items)) {
    return TRUE;
  }
  elseif ($settings['x_dedupe']) {
    foreach ($dup_items as $id) {
      $feed_item->feedapi_node->duplicates[$id] = $id;
    }
  }
  return array_pop($dup_items);
}

For the tests: it should be easy to supply a mapping for feed content type by simply creating the proper variable. This can be a stop gap until #433388: Make mappings exportable with CTools/export lands.

aron novak’s picture

Status: Needs work » Needs review
StatusFileSize
new2 KB

#6: this control structure was simplified.

About the test: they fail because i mixed unit and functional tests, a direct module_invoke call is in the .test file. This is not so nice, but fixing this is not related to this patch at all. Basically at the test environment feedapi_mapper is not installed (of course), but at the direct function call, function_exists return true and it tries to use non-existing DB table.

alex_b’s picture

Status: Needs review » Needs work

This will need adjustments once #539752-8: Support 'unique' conditions is fixed.

aron novak’s picture

Status: Needs work » Needs review
StatusFileSize
new2.15 KB

Done.

aron novak’s picture

StatusFileSize
new2.94 KB

Updated as #539752: Support 'unique' conditions changed a bit conceptually.

alex_b’s picture

Status: Needs review » Needs work

2 minor nit picks:

1 This looks a little convoluted:

+  // Let the mapper hijack the decision
+  if (function_exists('feedapi_mapper_unique')) {
+    $feed_node = node_load($feed_nid);
+    $result = feedapi_mapper_unique($feed_node, 'feedapi_node', $feed_item);
+    if ($result === FALSE) {
+      // No user-defined mapping, use the built-in mapping logic (url with guid fallback)
+      unset($result);
+    }
+  }
+  if (isset($result)) {
+    if (empty($result)) {
+      return TRUE;
+    }

I would change it to:

+  // Call feedapi_mapper_unique() if available and use its results if unique elements are defined.
+  if (function_exists('feedapi_mapper_unique')) {
+    $feed_node = node_load($feed_nid);
+    $result = feedapi_mapper_unique($feed_node, 'feedapi_node', $feed_item);
+
+    if ($result !== FALSE) {
+      if (empty($result)) {
+        return TRUE;
+      }

2 Before the original code continues with "if (empty($feed_item->options->original_url)) {" let's add the comment "// If we reach this point feedapi_mapper_unique() is not available or no unique elements have been defined - fall back to URL/GUID for determining whether item is unique.

aron novak’s picture

Status: Needs work » Needs review
StatusFileSize
new10.97 KB

Fixed.

alex_b’s picture

Status: Needs review » Needs work

#12 is a wrong patch.

alex_b’s picture

After this patch feedapi_node should be compatible with feedapi_mapper 1.x and 2.x. We don't need to branch a FeedAPI 2.x therefore - correct?

After #11 is fixed and 1.7 is tagged, this is RTBC.

aron novak’s picture

Status: Needs work » Fixed

Whoo, this is not RTBC unfortunately.
When calling feedapi_mapper_map($feed_node, 'feedapi_node', $feed_item, $node);, this assumes 2.x API at the mapper. So i added more strict check there:

 // Only call feedapi_mapper_map() directly in the case of FeedAPI Mapper 2.x
301	 	   if (function_exists('feedapi_mapper_map') && function_exists('feedapi_mapper_unique')) {
302	 	     $node = feedapi_mapper_map($feed_node, 'feedapi_node', $feed_item, $node);
303	 	   }
304	 	   else {

After that I committed it.

alex_b’s picture

Could you quickly fix the comment to:

// Use feedapi_mapper_map() if FeedAPI Mapper is version 2.x (= feedapi_mapper_unique() is present)

aron novak’s picture

Status: Fixed » Closed (fixed)

I also updated the comment according to #16.