By breyten@www.drop.org on

There's a new version of the import module available for testing. In this implementation both feeds and items are nodes. It is able to
read RSS 0.9x, 1.0 and 2.0 and has support for some DC items, the SY module and the content:encoded stuff. In addition
to that it also makes use of RSS 2.0's guid and comments elemens (and some more). Adding a feed is as simple as
supplying the url and all the other options will be filled in automagically. There's also an option to promote news
items automatically .

It needs (lots of) testing and bugs fixed, while some features still need to be implemented. I need some help of taxonomy
specialists to find a way to set taxonomy terms on items automatically. When a feed is administered you should be able
to specify a set of taxonomy terms for the items.

If you want to test it out on your own installation read the INSTALL file carefully, as there are some
important changes when using the cron.php script.
You can see it in action on my test site.

Hope you like it. Please send feature suggestions, bug fixes, comments, questions etc. to me (breyten@dataloss.nl)

Comments

gotze’s picture

gotze commented

I am very interested in this new module. I am yet to install it, but now know what to do in the week-end ...
Does the node-implementation mean one can create new feeds? (e.g., composite feeds)
John

moshe weitzman’s picture

moshe weitzman commented

yes, this new module gives us composite feeds. i sent a patch to breyten a few days ago. onc he applies that, the taxonomy integration will be very strong in his new module(s)

dries’s picture

dries commented

Breyten,

  • Using this module one can quickly fill its database with thousands of nodes. I'm interested to know what this costs in terms of performance.
  • On a related note, does the aggregator offer a means to delete old news items, or items that haven't been commented on? Admittedly, this might be something to deal with at a higher level; like, it wouldn't hurt to remove forum topics that date back from 2001 and that still haven't been commented on. On the other hand, you don't want to remove blog items that date back from 2001 so it is rather tricky. They call is content management.
  • Why don't you use PHP's built-in parser? Nowadays, PHP's XML parser is enabled by default and is for example used by Drupal's XML-RPC library. Get rid of the xmlparser.inc!
  • Why are feeds/bundles nodes too? Personally, I'd prefer to have a single aggregator module only. After all, feeds/bundles are (or could be) created by means of taxonomy terms.
  • When developing the current aggregator module, I remember having problems with feeds that were being updated after they were imported (bloggers often massage existing blog items throughout the day). Make sure to test this.
  • I have applied one of Moshe's patches which should make it possible to set the author/site's name even though he is not a registered user.

Good stuff. Looking forward to an update.

moshe weitzman’s picture

moshe weitzman commented

I'm wondering if you are confusing feeds and bundles. Feeds cannot be mere taxonomy terms. We need to store the URL of the feed, the period of retrieval, the name of the feed, and so on. That can't be stored in the taxonomy tables of course. So Breyten has wisely made this a node type. One byproduct of this is that we can use taxonomy to catagorize feeds, which then automatically tag their respective items as items are imported (I sent breyten a patch for this). Thus, bundles are easily presented using taxonomy.

Your idea of optionally deleting old news items is a good one. That pretty much takes care of the node performance question for now.

I think that if a large node table becomes an issue, we need to deal with it specifically, and not write code to work around it. IMO, storing feed items and comments outside of the node table is writing code around a (possible) problem. A module author should be unafraid to jam thousands of nodes in the db if desired. It is the responsibility of the core engine to deal with this ... I wish we had more database expertise in this project (are you out there?).

breyten@www.drop.org’s picture

breyten@www.drop.org commented

Using this module one can quickly fill its database with thousands of nodes. I'm interested to know what this costs in terms of performance.

I don't know yet Only local images are allowed.. I can run a benchmark, just need to know how to do it.

On a related note, does the aggregator offer a means to delete old news items, or items that haven't been commented on? Admittedly, this might be something to deal with at a higher level; like, it wouldn't hurt to remove forum topics that date back from 2001 and that still haven't been commented on. On the other hand, you don't want to remove blog items that date back from 2001 so it is rather tricky. They call is content management.

It seems like a good idea to implement such an option. I'll make it a per-feed option, kinda like the watchdog module.

Why don't you use PHP's built-in parser? Nowadays, PHP's XML parser is enabled by default and is for example used by Drupal's XML-RPC library. Get rid of the xmlparser.inc!

Initially, I took the parser from revjim.net and extended that. Now I'm working on my own parser.

Why are feeds/bundles nodes too? Personally, I'd prefer to have a single aggregator module only. After all, feeds/bundles are (or could be) created by means of taxonomy terms.

See moshe's reply for this.

When developing the current aggregator module, I remember having problems with feeds that were being updated after they were imported (bloggers often massage existing blog items throughout the day). Make sure to test this.

Tested Only local images are allowed.. Since the introduction of the guids in RSS 2.0 this is a lot simpeler. But the problem still exists for 0.9x feeds. I mainly look at the link (least likely to change) and the title.

I have applied one of Moshe's patches which should make it possible to set the author/site's name even though he is not a registered user.

Great. I added that to the the module(s).

dries’s picture

dries commented

Initially, I took the parser from revjim.net and extended that. Now I'm working on my own parser.

But why? Nowadays, PHP comes with a built-in XML parser that is enabled by default ...

jsloan’s picture

jsloan commented

Is there any interest in adopting MagpieRSS as a "system module" (or perhaps more appropriately an "Include") and wrapping it with a Drupal API? The cooperation between two open source projects would benefit both and stabilize the development of RSS based modules within Drupal. I use RSS feeds from several systems within our Intranet to dynamically create menus within a node.
A while back this comment refered to using RSS weather feeds. My point is that RSS handling can be used throughout Drupal if it had a simple API.

MagpieRSS at Sourceforge

Steven’s picture

Steven commented

This was discussed extensively on the development mailinglist. Basically, MagpieRSS does more than RSS fetching, it also has an HTTP client and a cache for it. We don't need this in Drupal because we have our own framework for this.

If you take away everything except the parser, not much is left of MagpieRSS, and we would be much better off just adapting our own code so it incorporates the MagpieRSS's abilities that we currently don't have. MagpieRSS has problems of its own: for example it does not handle character encodings at all. This is something which Drupal already does.

http://lists.drupal.org/archives/drupal-devel/2004-07/msg00551.html
http://lists.drupal.org/archives/drupal-devel/2004-07/msg00447.html

nreese’s picture

nreese commented

I am new to Drupal, and have used it to set up the national newspaper of the Solomon Islands online. My problem is at the moment they are supplying a 35+ page document of all the stories to me with headings, sections, and body text, which is getting cut and pasted into the site.

Could someone please give me a process, or point me to some information, that will enable me to turn the document into an XML doc that I can then upload to drupal? (need the xml definition, and need the process for uploading)

Regards,

Nick Reese

See the national newspaper of the Solomon Islands online using Drupal:

www.solomonstarnews.com

Bèr Kessels’s picture

Bèr Kessels commented

the import system you are looking for can be found here: http://drupal.org/project/node_aggregator

[Moshe: fixed bad link. was pointing to node_import]

[Ber | Drupal Services webschuur.com]

ekadziela’s picture

ekadziela commented

I've been working with MagpieRSS and the flexinode module to feed random images from flickr. The php file that uses MagpieRSS works fine by itself. However, when i incorporate it into drupal, i get this warning message (along with my photos):

warning: mkdir() [function.mkdir]: Permission denied in /.../themes/bluemarine/magpierss/rss_cache.inc on line 34.
warning: fopen(magpie_cache/c25fb6c1d04909b44915bd950e3d5239) [function.fopen]: failed to open stream: No such file or directory in /.../themes/bluemarine/magpierss/rss_cache.inc on line 54.

I've screwed around with permissions and i've made sure that the current folder and parent folder and even the cache folder all have their permissions properly set and to no avail. Please help.