Feeds Crawler Documentation

You are browsing documentation for an older version of Drupal, which is not supported any longer, and the information may not be correct. See current documentation for Contributed modules

The Feeds module is required. For documentation on installing the Feeds Crawler module, see the handbook entry "Installing contributed modules".

First a Feeds importer has to be configured. See The site builder's guide to Feeds.
Next, go to Administer > Site Building > Feeds Crawler.
Select the importer, or if the feed is attached to a node, the node.
If you'd like to start at a page other than the first, you can set the URL that you want to start with.
Optional: Try selecting auto detect. If the feed has a link with a rel="next" attribute the crawler will find it automatically. Any sites with the correct link that are not found should be posted as a bug.
Select whether the source is HTML or XML. If you're parsing an RSS or Atom feed select XML.
If auto detection did not work, put the XPath expression of the next link here. Make sure that the expression ends in "/@href". See below for documentation about XPath.
Finally, select the number of pages that you want to crawl. Putting 0 here will crawl until the there are not anymore link. Be careful with this.
Click Crawl!

todo

Documentation

Firefox XPath add-on modules

XPather "Feature rich XPath generator, editor, inspector and simple extraction tool"
XPath Checker "An interactive editor for XPath expressions"
FireXPath "a Firebug extension that adds a development tool to edit, inspect and generate XPath expressions."
Firefinder for Firebug "A Firebug add-on that finds HTML elements matching chosen CSS selector(s) or XPath expression"

Tutorials / Getting started

Here are some tutorials that should help you get started with Feeds Crawler. Feel free to add further tutorials.