Fetch full contents from teaser-only RSS feeds

Yes, you're definitely on the right track. Getting the full contents of an RSS feed was one of the intended uses for XPath Parser. You'll see in a moment though; there's a small bit of functionality not included in Feeds or XPath Parser that can help accomplish this in a very clean manner.

First off, a basic principle when scraping remote content in Drupal is that the content will be presented as either a list or a record (todo: create a scraping glossary), and one feed will not work for both. The feed that imports the list of items (in this case an RSS feed, instead of a page scraper) will create a bunch of feed items but will not be useful for scraping the rest of the content. So, in most cases, the feed items will be incomplete.

The next feed needs to be configured specifically to import an individual record. It uses the imported source url or a user defined url to fetch the rest of the content. In a standard feeds configuration, a feed importer is separate from the item(s) that it creates. In this case however, one feed to import one other node would result in cumbersome duplication. Alternatively, with Feeds SelfNode Processor, individual records can imported with a feed importer that updates itself. This is sole purpose of the module.

I'll try to walkthrough the process using helpful names, but you can obviously change them to whatever it is you want:

1. Create the content types 'RISMedia Feed' for the attached list feed and 'RIS Media Article' for the articles.
2. Setup the feed that import lists of articles, 'RISMedia RSS'.

- Basic Settings: Attach to 'RISMedia Feed'
- HTTPFetcher
- RSS Parser
- Node Processor: 'RISMedia Article'
- Mappings: Title, Feed source (unique target) , Teaser (you don't really need it, since we will overwrite it with the full content)

3. Create another feed, 'RIS Media Article Unteaser'.

 - Basic settings: Attach to 'RISMedia Article'
 - HTTP Fetcher 
 - SelfNode Processor, XPath HTML: First add mappings, then define default xpath queries, eg. body field & article author (default queries are very important here so that every new record has the necessary configuration to import its content).  See http://drupal.org/node/1045878 for more information on what this might look like
 - Mappings: Setup target fields with XPath query holders, then go back up to define default XPath queries

General note: you can chain several list feeds and combine multiple single item processors. This would more so resemble a spidering scraper.

I have the same issue and error messages as dion6898 but content is also failing to update after the first import.

Any suggestions?

Thanks very much for the guidence on this open santa. For some reason, after installing feeds self node processor 6.x-1.x-dev I don't get the option for a 'SelfNode Processor, XPath HTML'. I set up the first importer and attach it to the feed ('RISMedia Feed') and when I set up the second feed importer I attach it to the content item ('RIS Media Article') but the only options for processors I have are the node processor and the feed node processor - both of which have xml parsers which is great - but the second importer requires me to select another content type which doesn't make sense as I want it to update itself. Is there something obvious that I'm missing? thanks!

My apologies. User too stupid. CTFC. I failed to clear the cache. Now I have the Self node processor.

I'm having the same problem - it creates nodes witht he title and body only, but no other xpath defined field values so it doesn't appear the the 'self node processor' is reaching into the full xml beyond the feed list. I know the xpaths are correct because I uploaded a sample of the full xml as a stand alone file and it populated the drupal fields correctly.

I'm sorry I haven't had time to contribute toward documenting this module. I'm working on some similar but very different data processing contributions.

To those commenting and working on this, thank you and godspeed. When the understanding emerges, please start fleshing out the handbook for this module.

I'll make at least a couple of further comments about my experiences with this module:
- I had similar issues when trying to set it up.
- There have been small quirks in each version. As kapayne pointed out, it was/is necessary to clear the cache after installation. Another one that hasn't been mentioned mentioned is to hit the submit button on a blank form in the processor settings page before adding the mappings (something to that effect, but I'm not sure where this was done or if twistor fixed this in a later release).
- Once I got it up, I released Example Web Scraper to try to help people around configuring these modules. That is only a workaround though; it would be helpful for many users to have the solution to this issue compiled into a handbook.

Thank you.

@mitchell the walkthrough you have mentioned doesn't work and it result in too many errors, is there any other method that pull the full content out of rss feeds ?, plus the way u mentioned need a new feed item for every site we need to grab, because every site has its own classes and id's and of course xpath identifiers.

I got exactly as far as #3, but I don't get any error messages.

The nodes are created, but as the update in #3 explains, the url of the article to scrape is not mapped, as when i edit a RISMedia Article node, the url to to article is blank!!

If I manually add the correct url into the feed field it gets the content correctly.

Can someone explain what is needed to be able to do this?

If someone is kind enough to publish an rss feed, shouldn't they get to decide what they want to share? Couldn't you for instance contact the source and ask that they provide full content, or explain why they don't? If they publish under a "free spirit" licence such as one of the Creative Commons, it shouldn't be too hard to convince to publish the content as a whole in their feeds, no? And if not, well... It's not because something is public that it is free to do whatever you like with it (come fair use provisions).

Just my 0.02$ on this topic.

I'm stuck with exactly the same issue. The self node items are not running. Here are my two exports, list and article:

List

$feeds_importer = new stdClass();
$feeds_importer->disabled = FALSE; /* Edit this to true to make a default feeds_importer disabled initially */
$feeds_importer->api_version = 1;
$feeds_importer->id = 'list';
$feeds_importer->config = array(
  'name' => 'list',
  'description' => 'List scraper',
  'fetcher' => array(
    'plugin_key' => 'FeedsHTTPFetcher',
    'config' => array(
      'auto_detect_feeds' => FALSE,
      'use_pubsubhubbub' => FALSE,
      'designated_hub' => '',
      'request_timeout' => NULL,
    ),
  ),
  'parser' => array(
    'plugin_key' => 'FeedsXPathParserXML',
    'config' => array(
      'sources' => array(
        'xpathparser:0' => 'title',
        'xpathparser:1' => 'link',
        'xpathparser:2' => 'link',
      ),
      'rawXML' => array(
        'xpathparser:0' => 0,
        'xpathparser:1' => 0,
        'xpathparser:2' => 0,
      ),
      'context' => '/rss/channel/item',
      'exp' => array(
        'errors' => 0,
        'debug' => array(
          'context' => 0,
          'xpathparser:0' => 0,
          'xpathparser:1' => 0,
          'xpathparser:2' => 0,
        ),
      ),
      'allow_override' => 0,
    ),
  ),
  'processor' => array(
    'plugin_key' => 'FeedsNodeProcessor',
    'config' => array(
      'expire' => '-1',
      'author' => '2',
      'authorize' => 0,
      'mappings' => array(
        0 => array(
          'source' => 'xpathparser:0',
          'target' => 'title',
          'unique' => FALSE,
        ),
        1 => array(
          'source' => 'xpathparser:1',
          'target' => 'feeds_source',
          'unique' => 1,
        ),
        2 => array(
          'source' => 'Blank source 2',
          'target' => 'field_source_website',
          'term_search' => '0',
          'autocreate' => 1,
        ),
        3 => array(
          'source' => 'Blank source 3',
          'target' => 'status',
          'unique' => FALSE,
        ),
        4 => array(
          'source' => 'xpathparser:2',
          'target' => 'field_read_more_link',
          'unique' => FALSE,
        ),
      ),
      'update_existing' => '0',
      'input_format' => 'plain_text',
      'skip_hash_check' => 0,
      'bundle' => 'imported_nieuws_item',
    ),
  ),
  'content_type' => '',
  'update' => 0,
  'import_period' => '0',
  'expire_period' => 3600,
  'import_on_create' => 1,
  'process_in_background' => 1,
);

Article

$feeds_importer = new stdClass();
$feeds_importer->disabled = FALSE; /* Edit this to true to make a default feeds_importer disabled initially */
$feeds_importer->api_version = 1;
$feeds_importer->id = 'article';
$feeds_importer->config = array(
  'name' => 'article',
  'description' => 'Article scraper',
  'fetcher' => array(
    'plugin_key' => 'FeedsHTTPFetcher',
    'config' => array(
      'auto_detect_feeds' => FALSE,
      'use_pubsubhubbub' => FALSE,
      'designated_hub' => '',
      'request_timeout' => NULL,
    ),
  ),
  'parser' => array(
    'plugin_key' => 'FeedsXPathParserHTML',
    'config' => array(
      'sources' => array(
        'xpathparser:0' => '//div[contains(@class, \'articleEntry\')][1]',
        'xpathparser:1' => '//div[@class=\'articleEntry\'][1]//img/@src',
      ),
      'rawXML' => array(
        'xpathparser:0' => 0,
        'xpathparser:1' => 0,
      ),
      'context' => '//html',
      'exp' => array(
        'errors' => 0,
        'debug' => array(
          'context' => 0,
          'xpathparser:0' => 0,
          'xpathparser:1' => 0,
        ),
      ),
      'allow_override' => 0,
    ),
  ),
  'processor' => array(
    'plugin_key' => 'FeedsSelfNodeProcessor',
    'config' => array(
      'expire' => '-1',
      'author' => '2',
      'authorize' => 0,
      'mappings' => array(
        0 => array(
          'source' => 'xpathparser:0',
          'target' => 'body',
          'unique' => FALSE,
        ),
        1 => array(
          'source' => 'xpathparser:1',
          'target' => 'field_news_image',
          'unique' => FALSE,
        ),
        2 => array(
          'source' => 'Blank source 1',
          'target' => 'language',
          'unique' => FALSE,
        ),
        3 => array(
          'source' => 'Blank source 2',
          'target' => 'tnid',
          'unique' => FALSE,
        ),
        4 => array(
          'source' => 'Blank source 3',
          'target' => 'status',
          'unique' => FALSE,
        ),
      ),
      'update_existing' => '2',
      'input_format' => 'plain_text',
      'skip_hash_check' => 0,
      'bundle' => 'imported_nieuws_item',
    ),
  ),
  'content_type' => 'imported_nieuws_item',
  'update' => 0,
  'import_period' => '0',
  'expire_period' => 3600,
  'import_on_create' => 1,
  'process_in_background' => 1,
);

My list is a standalone form, so not attached to a node.

Ok, everything seems to be working. Had some strange queue and cron issues but those seem resolved and now everything is working.

Lasac, can you elaborate on your queue and cron issues and how you fixed them? Looks like I have the same problem as you.

I dumped all the feeds node queues in the queue table, run cron couple of times and it started working as expected. Guess you could use the job queue module as well to run trough the queue.

Ok. Seems that the selfnode must contain data to import or else it's not working as expected. I have not digged deeper but for me, the body i tried to import seemd to be empty.

Same issue here, the body is empty then I click import from the node "Article" (#15) in the tab menu and click import button, so the body field being populated with xpath parser results.

Any idea how to queue per node "Article" import ?

hmm... all i need is just running cron for several time or wait until the cron job running to import selfnode (populated body field). now my setup working. But, there's another problem, every time cron job running all of the feed importer and article process to be update. So the cron queue increase significanly.

how can we stop article being updated for second and later times?

anyone has a clue?

The fact that selfnode_processor feeds are scheduled when update_existing = skip is silly and we should fix that.

#2349349: Unschedule feed when update_existing is skip.