Last updated January 19, 2013. Created by alex_b on November 4, 2009.
Edited by balagan, 7wonders, that0n3guy, beanluc. Log in to edit this page.
Quick introduction to get started with improving Feeds or writing plugins or mappers for it.
For a detailed documentation of Feeds functions and a class hierarchy overview refer to Feeds Doxygen documentation (courtesy of indytechcook).
Architecture
Feeds consists of three modules:
- Feeds module (feeds.module) contains entry points to the Feeds API (feeds.api.php) and the production UI (feeds_ui/feeds_ui.module) (e. g. import forms).
- Feeds Admin UI (feeds_ui/feeds_ui.module) module contains the UI elements necessary for configuring and managing Feeds during a site build. It can be optionally shut off once a site is built out (to not much security or performance gain, however).
- Feeds Defaults (feeds_import/feeds_import.module via feeds_import/feeds_import.feeds_importer_default.inc) contains a couple of default importer configurations for getting started with feeds.
The feeds module itself provides the most important hook implementations, plugin functions and wrapper functions for retrieving objects of Feeds' API classes.
The most important API classes are:
class FeedsImporter: contains aFeedsFetcher, aFeedsParserand aFeedsProcessorplugin object and the configuration of the importer and its plugins.abstract class FeedsPlugin: base class for all pluginsabstract class FeedsFetcher,abstract class FeedsParser,abstract class FeedsProcessor: Base classes for the three possible types of plugins. Derived fromFeedsPlugin class.class FeedsSource: holds a source (i. e. a URL or a file path). AFeedsSourceobject is being passed intoFeedsImporterwhen importing from that source. AFeedsSourcecan be tied to a specific node or not.abstract class FeedsConfigurable: base class for configurable and persistent entities (FeedsImporter,FeedsPluginandFeedsSource)class FeedsScheduler: this class is responsible for periodic import (= aggregation).
See Feeds glossary for more information on Feeds terminology.
Plugins API
Feeds has a CTools based plugins API that works similar to Views. Examples can be found in Feed's very own feeds.plugins.inc or as an example stand alone module at Extractor module.
SUMMARY:
A module that wants to provide its own Fetcher, Parser or Processor plugin must declare a plugin and implement it.
Declaration of a Feeds plugin
Implement hook_feeds_plugins() and return an array of plugin definitions.
A plugin definition must contain a name, a description and a handler. The handler must contain a parent plugin name, the class name of the plugin, the file and path identifying the location of the definition of the plugin. Feeds distinguishes between the key of the plugin definition (the key of the plugin in the $info array) and the class name of the plugin handler. However, most of the times the class name and the key are the same.
Every plugin must be derived either directly or indirectly from either FeedsFetcher, FeedsParser or FeedsProcessor. The value of the parent property refers to the plugin key of another plugin, not its class name.
Example
<?php
/**
* Implementation of hook_feeds_plugins().
*/
function mymodule_feeds_plugins() {
$info = array();
$info['MyParser'] = array(
'name' => 'My parser',
'description' => 'Parses custom data.',
'handler' => array(
'parent' => 'FeedsParser', // A plugin needs to derive either directly or indirectly from FeedsFetcher, FeedsParser or FeedsProcessor.
'class' => 'MyParser',
'file' => 'MyParser.inc',
'path' => drupal_get_path('module', 'mymodule'),
),
);
return $info;
}
function mymodule_enable() {
//clear the cache to display in Feeds as available plugin.
cache_clear_all('plugins:feeds:plugins', 'cache');
}
?>Note: If you intend to extend the SimplePie parser, as described below, you will need to declare your plugin handler as follows: 'parent' => 'FeedsSimplePieParser' or else you will receive an error message about a missing class when you try to enable your parser.
Implementation of a feeds plugin
The implementation of the plugin needs to reside in the file indicated in hook_feeds_plugins(). When the plugin is requested (for instance when Feeds displays a list of available plugins to the user, or when a plugin is loaded to import a source), it is dynamically loaded from this file.
Depending on which plugin type (Fetcher, Parser or Processor) is implemented, a varying set of methods must or can be defined or overwritten. The classes used to implement Fetcher, Parser and Processor plugins do the main part of their work through methods named fetch(), parse() and process() respectively. For details, refer to FeedsFetcher, FeedsParser or FeedsProcessor class.
There are some differences between the Drupal 6 and Drupal 7 versions of the methods used to implement feeds plugins, as noted below.
Drupal 6 plugins
In Feeds 6.x-1.0, the Fetcher, Parser and Processor plugins rely on an object of class FeedsBatch and its subclass, FeedsImportBatch, to keep track of the state of a feed and its content while it is being fetched, parsed and processed. Documentation for the FeedsBatch and FeedsImportBatch classes can be found in the comments in file includes/FeedsBatch.inc.
The fetch() method for a Fetcher in Feeds 6.x-1.0 has the following function signature:
<?php
FeedsFetcher::fetch(FeedsSource $source);
?>The fetch() method returns an object of class FeedsImportBatch or a FeedsImportBatch subclass.
The parse() and process() methods for Parsers and Processors in Feeds 6.x-1.0 then receive and modify an object of class FeedsImportBatch. They have the following function signatures:
<?php
FeedsParser::parse(FeedsImportBatch $batch, FeedsSource $source);
FeedsProcessor::process(FeedsImportBatch $batch, FeedsSource $source);
?>The FeedsImportBatch object is therefore a single object which changes "states" as it is handed from Fetcher to Parser to Processor.
Drupal 7 plugins
In Feeds 7.x-2.0, the FeedsBatch/FeedsImportBatch class is gone and is replaced with different "result" classes that are returned by Fetchers and Parsers. Instead of returning an object of class FeedsImportBatch, therefore, the Fetcher returns an object of type FeedsFetcherResult.
The parse() and process() methods for Parsers and Processors in Feeds 7.x-2.0 therefore have the following function signatures:
<?php
FeedsParser::parse(FeedsSource $source, FeedsFetcherResult $fetcher_result);
FeedsProcessor::process(FeedsSource $source, FeedsParserResult $parser_result);
?>Documentation for the FeedsFetcherResult and FeedsParserResult classes can be found in the comments in files plugins/FeedsFetcher.inc and plugins/FeedsParser.inc.
In other words, Feeds handling still flows from Fetcher->Parser->Processor, but the objects used to pass information between each stage have changed. In Feeds 7.x-2.0, the process is as follows:
- The
fetch()method of the Fetcher returns an object of classFeedsFetcherResult. - The
parse()method of the Parser receives an object of classFeedsFetcherResultand returns an object of classFeedsParserResult. - The
process()method of the Processor receives an object of classFeedsParserResult.
Example of a Parser Plugin for Feeds 6.x-1.0
This example builds on the one in "Declaration of a Feeds plugin". It defines a MyParser that extends FeedsParser. NOTE: This example is based on Feeds version 6.x-1.0. The parse() method takes different parameters in Feeds version 7.x-2.0.
parse()parses the source document and populates aFeedsImportBatch.getMappingSources()declares the fields the parser returns. This information is being used for mapping (See Mapping API section).
<?php
/**
* Parses My Feed
*/
class MyParser extends FeedsParser {
/**
* Parses a raw string and populates FeedsImportBatch object from it.
*/
public function parse(FeedsImportBatch $batch, FeedsSource $source) {
// Get the file's content.
$string = $batch->getRaw();
// Parse it...
// The parsed result should be an array of arrays of field name => value.
// This is an example of such an array:
$items = array();
$items[] = array(
'guid' => 'MyGuid1',
'title' => 'My Title',
);
$items[] = array(
'guid' => 'MyGuid2',
'title' => 'My Other Title',
);
// Populate the FeedsImportBatch object with the parsed results.
$batch->title = 'Example title';
$batch->items = $items;
}
public function getMappingSources() {
return array(
'guid' => array(
'name' => t('GUID'),
'description' => t('Unique ID.'),
),
'title' => array(
'name' => t('Title'),
'description' => t('My Title.'),
),
);
}
}
?>Extending the SimplePie parser integration
To enable SimplePie:
- Download it from http://simplepie.org/
Download the most recent package to get the simplepie.inc file you need in the next step. (You don't need the source.) - Copy the simplepie.inc into feeds/libraries as mentioned in the readme
- Clear the cache (admin/settings/performance) to see the option
Alternatively, you can also use the Libraries API module, and put simplepie.inc inside sites/all/libraries/simplepie/ (so the end path would be sites/all/libraries/simplepie/simplepie.inc).
For customization purposes, Feeds' SimplePie parser integration can be extended by overriding parseExtensions() and getMappingSources().
Here is an example:
<?php
/**
* A simple parser that extends FeedsSimplePieParser by adding support for a
* couple of iTunes tags.
*/
class MyParser extends FeedsSimplePieParser {
/**
* Add the extra mapping sources provided by this parser.
*/
public function getMappingSources() {
return parent::getMappingSources() + array(
'itunes_keywords' => array(
'name' => t('iTunes:Keywords'),
'description' => t('iTunes Keywords.'),
),
'itunes_duration' => array(
'name' => t('iTunes:Duration'),
'description' => t('iTunes Duration.'),
),
);
}
/**
* Parse the extra mapping sources provided by this parser.
*/
protected function parseExtensions(&$item, $simplepie_item) {
$itunes_namespace = 'http://www.itunes.com/dtds/podcast-1.0.dtd';
if ($value = $simplepie_item->get_item_tags($itunes_namespace, 'keywords')) {
$item['itunes_keywords'] = $value[0]['data'];
}
if ($value = $simplepie_item->get_item_tags($itunes_namespace, 'duration')) {
$item['itunes_duration'] = $value[0]['data'];
}
}
}
?>Mapping API
Feeds allows other modules to define additional mapping targets by way of alter hooks. The API hooks are specific to the processor in use but they work very similar. An API implementer adds additional mapping targets via the alter hook. These targets contain a name, a description and a callback. The callback will be invoked by Feeds if a user has picked the mapping target for their importer.
See also list of implemented mappers.
Here is a step by step guide showing how to implement a new mapping target.
1) Implement target alter hook
First the presence of an additional mapping target needs to be declared to Feeds. This is done by implementing an alter hook that modifies the array of available mapping targets. Additional mapping targets must have a name, a description and a callback. Name and description are used on the mapping form of a processor, the callback is invoked if a site builder has added the mapping target to a processor configuration and a feed item is presently being imported.
There are different target alter hooks for node and term processors; they are documented in feeds.api.php. This guide focuses on extending the node processor.
<?php
/**
* Implementation of hook_feeds_node_processor_targets_alter().
*/
function mymodule_feeds_node_processor_targets_alter(&$targets, $content_type) {
if ($content_type == 'my_content_type') {
$targets['my_target_field'] = array(
'name' => t('My Target Field'),
'description' => t('Shows up in legend on mapping form.'),
'callback' => 'mymodule_set_target', // See 2)
);
}
}
?>Note: This function is for Drupal 6. The function signature for Drupal 7 is
<?php
mymodule_feeds_processor_targets_alter(&$target, $type, $bundle)
?>2) Implement callback
Now all that is left is to implement the callback. Pay attention to the parameters in a mapping callback: $node is the node object presently being assembled for storage. $target is the key of the target field. In this example it can only be 'my_target_field', because the hook only declared this one target key. $value can be a numeric, a string or a FeedsElement object. Alternatively it can be an array of values of these types. If FeedsElement is used like a string it converts automatically to a string.
This example assumes that any non-array value is acceptable as target value:
<?php
/**
* Mapping callback.
*/
function mymodule_set_target($node, $target, $value) {
if (!is_array($value)) {
$node->$target = $value;
}
}
?>Frequently asked questions on Mapping
Where can I learn more than what's documented here?
For further information, look at code. A good place to start reading is mappers/content.inc or for a more advanced example, mappers/filefield.inc. For learning more about the internal gears of mapping in Feeds, refer to the Doxygen documentation's module section.
Should I submit my mapper as a patch to Feeds?
Generally speaking, the most preferable location for a mapper integration is the module that is home to the mapping target.
If this is not possible (e. g. for core modules like taxonomy) a mapper can be implemented as a patch to Feeds. Another reason why a mapper would be included with Feeds is that the integrated module is very popular (e. g. CCK, Filefield, Date modules).
For easing maintenance, mappers that are included in Feeds must have tests verifying their functionality.
I can't find a mapper for targets like the node title
Simple mapping targets on the subject of the processor - for example the node title or the node body field in FeedsNodeProcessor - are implemented natively in the getMappingTargets() and setTargetElement() methods of the processors themselves.
Defaults hooks
Any module can define a Feeds Importer configuration by way of a hook_feeds_importer_default() implementation. See The site builder's guide to Feeds for more information.
Programmatic operations
Create a feed node programmatically
This example demonstrates how to create a Feed node programmatically. It assumes that there is an Importer configured and attached to the content type 'feed'. The Importer uses an HTTP fetcher.
<?php
$node = new stdClass();
$node->type = 'feed';
$node->title = 'My feed';
$node->feeds['FeedsHTTPFetcher']['source'] = 'http://example.com/rss.xml';
node_save($node);
?>If you want automatic node title from the feed URL, you can use this snippet below.
<?php
feeds_include_library('common_syndication_parser.inc', 'common_syndication_parser');
feeds_include_library('http_request.inc', 'http_request');
$node = new stdClass ();
$node->type = 'feed';
$node->title = '';
$feed = http_request_get($url);
if (!empty($feed->data)) {
$feed = common_syndication_parser_parse($feed->data);
if (!empty($feed['title'])) {
$node->title = $feed['title'];
}
$node->feeds['FeedsHTTPFetcher']['source'] = 'http://example.com/rss.xml';
node_save($node);
?>Trigger import programmatically
This example shows how to import a feed programmatically. It can be read in continuation of the previous example.
<?php
// Using Batch API (user will see a progress bar).
feeds_batch_set(t('Importing'), 'import', 'my_importer_id', $node->nid);
// Not using Batch API (complete import within current page load)
while (FEEDS_BATCH_COMPLETE != feeds_source('my_importer_id', $node->nid)->import());
?>* Note: To do this with a stand alone feeds importer (IE... if the importer form is not a node), you set the $node->nid to zero.
Create a feed node programmatically using services
Similar to the above, you can also create a feed node programmatically using Services 3. Here is an example in python to create a feed node that has a FeedsFileFetcher field to an already uploaded file (few dependencies in python such as requests and json python modules):
#Set the headers and cookie jar
headers = {'content-type': 'application/json'}
jar = cookielib.CookieJar()
#Set the endpoints and payloads
login = 'https://example.com/endpoint/user/login'
url = 'https://example.com/endpoint/node'
creds = {'username': 'USERNAME', 'password': 'ABRACADABRA'}
payload = {'title':'WHATEVER','type':'YOUR_FEED_NODES_TYPE','name':'Admin','language':'und', 'feeds':{'FeedsFileFetcher':{'source':'private://path_to_file'}}}
#First we login using our credentials
requests.post(login, data=json.dumps(creds), headers=headers, verify=False, cookies=jar)
#Then we create a new feed node
requests.post(url, data=json.dumps(payload), headers=headers, verify=False, cookies=jar)Testing
Use Feeds Test Site for testing. Follow instructions in its README.txt file.
Comments
Pass $node by reference in callback
Nicely documented, thanks for putting it together. One minor thing: shouldn't the $node object be passed by reference in the above 'mymodule_set_target' callback example?
I noticed the same thing. Any
Object are passed by reference.
See: http://php.net/manual/en/language.oop5.references.php
Thanks to @mongolito404 who pointed this out to me !
-Pol-
Find me on Google+ or Twitter
Extending the SimplePie parser integration
When using the code from the "Extending the SimplePie parser integration" above, make sure that when you implement hook_feeds_plugins you set the parent to "FeedsSimplePieParser" instead of "FeedsParser" or you will get an error about the Class FeedsSimplePieParser not being found.
.cw.
In D7 at least the hook is
In D7 at least the hook is hook_feeds_processor_targets_alter() (i.e. no 'node).
Mapping API Drupal 7
Hi , i added the functions:
1) 'Implement target alter hook'
2) 'Implement callback'
to my module at Drupal 7 and i don't see the target from function 1, at the Mapping for Node processor page(configuration) should i declare something for it?
Corrupt nodes.
I used:
<?php$node = new stdClass();
$node->type = 'feed';
$node->title = 'My feed';
$node->feeds['FeedsHTTPFetcher']['source'] = 'http://example.com/rss.xml';
node_save($node);
?>
... to create about 8 corrupt nodes in my db (node_delete, node_load, and content can't find.. although views can); and believe it might have something to do with the missing user field being passed, maybe. Please advise how to fix this code, and clean out the corrupt nodes. (Also posted at http://drupal.org/node/1122600#comment-4456702.)
You probably want
You probably want node_object_prepare:
http://api.drupal.org/api/drupal/modules%21node%21node.module/function/n...
<?php$node = new stdClass();
$node->type = 'feed';
node_object_prepare($node);
...
?>
Sean Robertson
seanr@webolutionary.com
Programmatically trigger import in Drupal 7
http://drupal.org/node/1151914
http://nerdery.com/workwithme/cw
Programmatically trigger import in Drupal 7
This works for me in D7:
<?phpfeeds_batch('import', 'my_importer_id', $node->nid, $context);
?>
feeds_batch_set() Becomes feeds_batch() in D7
Running feeds_batch_set() did not work for me on D7 so I did a quick search on this page for 'feeds_batch_set' and nothing came up besides the original article instructions. Now further searches for this should show up here (points up). Use feeds_batch() as described above.
Edit: Information on $context http://api.drupal.org/api/drupal/includes%21form.inc/group/batch/7
parse() and process() signatures are wrong
parse() and process() signatures have their parameters in the wrong order for Feeds 7.x-2.0 . It's $source, then $parser_result. Not $parser_result, then $source.
Implementing feeds Parser/Fetcher/Processor
Hi,
If you are implementing own feeds plugin (Praser, Fetcher or Processor) and after declaring hooks you get an error Missing plugin make sure you've added records about the your plugin's files in the module's .info file! Have a look at feeds.info file for an example.
Mapping custom fieldtypes.
If you like, me, have implemented custom field types in a module and need to use feeds with them, here's an example you may find helpful. This needs to go in the module that defines the fieldtypes. Well, it doesn't have to, but it's logical.
This is for Drupal 7.
<?php
function MYMODULE_feeds_processor_targets_alter(&$targets, $entity_type, $bundle_name)
{
foreach (field_info_instances($entity_type, $bundle_name) as $name => $instance)
{
$info = field_info_field($name);
unset($callback);
if ($info['type']=='MYFIELDTYPE1')
{
//The callback for this field type goes here.
$callback = 'TARGET_SETTER_FOR_MYFIELDTYPE1';
}
if ($info['type']=='MYFIELDTYPE2')
{
//The callback for this field type goes here.
$callback = 'TARGET_SETTER_FOR_MYFIELDTYPE2';
}
//Add cases for each field type here
//...
if (isset($callback))
{
$targets[$name] =array(
'name'=>check_plain($instance['label']),
'callback' => $callback,
'description' => t('The @label field of the node.', array('@label' => $instance['label'])),
);
}
}
}
?>
Then all you have to do is to define some target setters for your custom fields. Simple!
Create a feed node programmatically fails to import on cron
Can anyone give me advice on how to create a feed node programmatically that imports on cron?
When I create a feed node programmatically as described above, I can import import through the import button but no feed items import on cron. In contrast, feed nodes created through the gui do import on cron.
edit: It seems like cron did start working on programmatically created feed nodes so my problems were probably issues with the feed source.
For 2) Implement
For 2) Implement callback:
Does this need updating?
<?php/**
* Mapping callback.
*/
function mymodule_set_target($node, $target, $value) {
if (!is_array($value)) {
$node->$target = $value;
}
}
?>
feeds.api.php says:
<?phpfunction my_module_set_target($source, $entity, $target, $value, $mapping)
?>
Edit: Oops, reply fail. Meant to reply to main thread.