Feeds Crawler Documentation

Installation

The Feeds module is required. For documentation on installing the Feeds Crawler module, see the handbook entry "Installing contributed modules".

Configuring

1. First a Feeds importer has to be configured. See The site builder's guide to Feeds.
2. Next, go to Administer > Site Building > Feeds Crawler.
3. Select the importer, or if the feed is attached to a node, the node.
4. If you'd like to start at a page other than the first, you can set the URL that you want to start with.
5. Optional: Try selecting auto detect. If the feed has a link with a rel="next" attribute the crawler will find it automatically. Any sites with the correct link that are not found should be posted as a bug.
6. Select whether the source is HTML or XML. If you're parsing an RSS or Atom feed select XML.
7. If auto detection did not work, put the XPath expression of the next link here. Make sure that the expression ends in "/@href". See below for documentation about XPath.
8. Finally, select the number of pages that you want to crawl. Putting 0 here will crawl until the there are not anymore link. Be careful with this.
9. Click Crawl!

Examples

todo

Links

Documentation

• XPath syntax reference
• Detailed XML Path Language (XPath) references

Firefox XPath add-on modules

• XPather "Feature rich XPath generator, editor, inspector and simple extraction tool"
• XPath Checker "An interactive editor for XPath expressions"
• FireXPath "a Firebug extension that adds a development tool to edit, inspect and generate XPath expressions."
• Firefinder for Firebug "A Firebug add-on that finds HTML elements matching chosen CSS selector(s) or XPath expression"