I'd like to see an HTML parser / scraper, (sometimes called a Web to RSS converter). Online services such as Feed43 to something similar, but it would be good to it without a third party service.

These services scan an HTML page for a start string, eg. <h1>News</h1>, then look for the a repeating pattern that represents a title, eg. <h3> * </h3>, then a pattern representing the content, and then a final end string pattern.

See also: Convert Web Pages Content Into RSS Feeds

This feature would allow the data mining of any web page. For reference, the following services look useful (and free):

Comments

iantresman’s picture

iantresman commented

The PHP Simple HTML DOM Parser looks like a step in the right direction, and includes example of scraping some web sites. Unfortunately I have no idea whether it could be converted to an HTML parser, or used as an HTML to XML converter.

darrenmothersele’s picture

darrenmothersele commented

subscribe

I'm already looking at using simplehtmldom to do something similar. It would be great to make this work with Feeds.

iantresman’s picture

iantresman commented

The simplehtmldom API looks interesting, I'll keep a keen eye on it.

alex_b’s picture

alex_b commented

Also take a look at querypath.

qandeel’s picture

qandeel commented

Hey,

I have already developed a module that works great with FeedAPI and it uses simplehtmldom API

Qandeel

iantresman’s picture

iantresman commented

Qandeel, what's your module called, and can you include a link to its home page.

qandeel’s picture

qandeel commented

I have not uploaded that on Drupal yet.. but i am planning to upload that as a separate project. Ill let you know once its done.. right now i am busy with my upcoming module "Advance Workflow Management"

twistor’s picture

twistor commented

I have a parser for Feeds that allows you to input XPath queries that map to fields. I'm actually trying to get CVS access at the moment so that I can post it. Here's a link to my request http://drupal.org/node/835276 you can download the current version. It's still pretty new, but I have been using it with much success.

twistor’s picture

twistor commented

Now at feeds_xpathparser.

hanno’s picture

hanno commented
Status: Active » Fixed

With the announcement of the feeds_xpathparser set this issue as fixed.

iantresman’s picture

iantresman commented

Sounds exciting, I shall look forward to test driving it in due course.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.