The following function adds two new fields to the HTTPFetcher. It's used to modifiy an URL with a prefix and a suffix.
The case scenario may be to use an URL entered by a user as an argument to build a request. An example might be to use the URL of a RSS feed
like "http://rss.cnn.com/rss/cnn_topstories.rss" to build a request to an API that will extract the full text of that feed.
By using a URL like this one as a prefix "http://fivefilters.org/content-only/makefulltextfeed.php?url=", all RSS feeds entered by the users would go to
"http://fivefilters.org/content-only/makefulltextfeed.php?url=http://url....".
Prefix and suffix can also be useful for sending requests to services such as SEOmoz, MajesticSEO or Google.
You'll notice that we've had a hard time fetching the array properly and that we have default to using a loop that counts each and every character.
This forces configuration names to be exactly 13 characters and descriptions must be empty. Anyway, it works well now as a prototype that serves our needs.

Anyone who wants to improve this code is welcomed.

CommentFileSizeAuthor
#3 FeedsHTTPFetcher.patch13.27 KBsylvanos
#3 Capture.png86.71 KBsylvanos
feed_importer.patch1.35 KBKarimh
HTTPFetcher.png56.47 KBKarimh
HTTPFetecher.patch13.09 KBKarimh

Comments

rickmanelius’s picture

rickmanelius commented

subscribe

patoshi’s picture

patoshi commented

patches arent working.. im also trying to integrate with fivefilters.

sylvanos’s picture

sylvanos commented
StatusFileSize
new Capture.png86.71 KB
new FeedsHTTPFetcher.patch13.27 KB

That's weird. Here is the patch again. Very important that your feed importer name has 13 characters.
Also included a screen capture to show an example of how we're using it.

patoshi’s picture

patoshi commented

i wonder if it would be feasible to integrate fivefilters with drupal or should it be seperate?

sylvanos’s picture

sylvanos commented

I like them separated. This way it is easier to balance the workload.

But I guess that if some folks wanted to include it directly within Feeds, then it coud become an option for the http fecthcer along with "Auto detect feeds" and "PubSubHubbub"

patoshi’s picture

patoshi commented

i still dont really understand how PubSubHubbub works and is that just another type of feed!? its for broadcasting your feed right? not receiving...

thanks!

sylvanos’s picture

sylvanos commented

It's both for broadcasting and receiving (http://en.wikipedia.org/wiki/PubSubHubbub), but I was just giving an example of where five-filters work could be integrated.

Robin Millette’s picture

Robin Millette commented

I needed something similar to this before checking the issue queue and came up with Feeds HTTP Arg Fetcher plugin.

BTW, I just love how easy it is to build on Feeds plugins the OOP way.

The HTTPArgFetcher plugin lets you configure a base, or template url with tokens. In my case, I was using the SelfNode processor with a "jamendo album" content type and importer.

I wanted to import information for the album http://www.jamendo.com/fr/album/64605 using their web API. I configured the "jamendo album" importer (attached to "jamendo album" content type AND ALSO outputting to the same content type - it's the only use case I tried with my HTTPArgFetcher plugin) and set a URL template (HTTPArgFetcher config) like
http://api.example.com/v1/albumid/[node:title] - notice the use of a token. You can basically use anything here to build the URL to be imported.

When creating a new "jamendo album" node, all you have to do is provide an albumid (ex: 64605) in the Title field. Submit, and
http://api.example.com/v1/albumid/64605 gets imported and according to my mapper and xpaths, values are set on the node. In my case, the albumid has its own field and 64605 is set there. Also, the title (set to "64605" by the user) is updated to reflect the actual title, according the regular mappers.

Give Feeds HTTP Arg Fetcher plugin a try and let me know if you think it helps... or (why) not :-)

To simplify data entry.

twistor’s picture

twistor commented
Assigned: Karimh » Unassigned
Status: Needs review » Closed (won't fix)

This is a pretty specialized use case that we are not going to support. The good thing is that Feeds is extensible. Feeds HTTP Arg Fetcher plugin also looks like a viable option.

Cleaning up old issues.