By fowlkes on

Friends of Drupal,

I'm wondering if I'm the only one that is concerned about this but...

We have just moved our main web site to Drupal 4.7. In our tests with 4.6, RSS feeds (as we wanted to see them) were so easy with Aggregator 2. Just create the feed, a list block was created in the blocks menu list, put it where you wanted, and click the RSS item titles to go to a newly created node with the feed detail on it.

Well, I'm desperate. I've tried Leech and it lacks documentation with all the wild things it can do. (I'm sure they are great but I just want basic functionality) I went back to Agg 2 and it is quirky in 4.7 and the basic aggregator doesn't create the nodes per RSS item that I need.

Anyone out there having the same problems and, if so, what are you doing about it? Is there someone out there with an alternative RSS agg 2 like tool? I'm so desperate, I'll just take some good docs on leech...

HELP!

Desperate in South Bend,

Keith :)

⋅ Categories: Drupal 4.7.x

Comments

newdru’s picture

newdru commented

why don't you just use aggregator (not agg 2)?

it seems to be working for me fine in 4.7. per the requirements you list above, it seems to do what you want it to (e.g. it automatically creates a block for each feed with a "more" link to the feed page).

boris mann’s picture

boris mann commented

I feel your pain. While Aggregator2 worked....that was actually deceptive. Leech module is an extension of the code and a bit crazy. The Development Seed folks are making improvements, but I fear that the base code is....scary.

Try the Feedparser project (http://drupal.org/project/feedparser) -- we've been very impressed by the clean-ness of the code.

fowlkes’s picture

fowlkes commented

I'll try it. I guess I'm just frustrated that it doesn't seem to be as user friendly as Agg2. I'm excited to check out feedparser and if anyone know where I can get good docs on Leech, please pass em along. Thanks again!!!

Keith

newdru’s picture

newdru commented

i took the time to try to post a reply to you.. can you be as courteous and reply to my question that i asked you above... mostly what feedparser and agg2 gives you that the standard aggregator doesn't?

we're all learning here together right?

thanks

funana’s picture

funana commented

Does Aggregator create nodes for the feed items? AFAIK not. That's one of the biggest lacks.
Does it handle not so wellformed RSS? AFAIK not.
Does it allow freetagging? AFAIK not.

newdru’s picture

newdru commented

i appreciate the reply..

if you have the time, can you tell me why i would want to create feed items as nodes?

i mean what benefit or extra feature can i provide on my site becasue the item is saved as a node? i know others want this feature but i'm not sure how i could use it to my advantage.

also, the normal aggreagator module has it's own categories (they are not taxonomy module based) thus they don't offer freetagging.

I guess feed items as nodes allows you to apply taxonomy freetagging to them. But what advantage does freetagging offer on feeds/feed items?

Finally, since you will be storing all feed items as nodes, and feeds are perpetual, won't this require huge amounts of db space? how do you manage old items (e.g. how and at what point do you delete them if ever)?

thanks much for any input you can provide
thanks

Finally

funana’s picture

funana commented

if you have the time, can you tell me why i would want to create feed items as nodes?

i mean what benefit or extra feature can i provide on my site becasue the item is saved as a node? i know others want this feature but i'm not sure how i could use it to my advantage.

If the items are published as nodes they become your own aggregated content, are taggable, rateble, you can comment them etc. It's better for search engines too.

I guess feed items as nodes allows you to apply taxonomy freetagging to them. But what advantage does freetagging offer on feeds/feed items?

In Aggregator you can only give feeds one category which I find a little bit restricted... If you freetag feeds in Leech the feeditems are tagged with the feed tags too and if you use the yahoo extension there is a function to autotag the items. If you combine this with community tagging module every member can tag items which is also a big advantage.

Finally, since you will be storing all feed items as nodes, and feeds are perpetual, won't this require huge amounts of db space? how do you manage old items (e.g. how and at what point do you delete them if ever)?

You can configure Leech to not leech feed items older than x days and I wouldn't delete old items though. There is no problem with DB space if you ask me. The items are just a little bit text and this is absolutely no problem for drupal and SQL. I run several sites with a lot of feeds and this has been no problem so far.

newdru’s picture

newdru commented

thanks... that helps a lot..

1) you mention the yahoo extension.. what does that exactly do?

2) could you point to some of your sites that use the leech features you sited in your previous two posts.. i'd like to see how others are using this functionality...?

thanks funana

alex_b’s picture

alex_b commented

leech_yahoo_terms.module tags content with the Yahoo Term Extraction Service: http://developer.yahoo.com/search/content/V1/termExtraction.html . The Term Extraction Service takes content and returns context-sensitive keywords. So an article that talks about "Paris" should get the keyword "France". Sounds fantastic, its accuracy could be better, though. It works very well for our purpose - we are tagging huge amounts of content to categorize it roughly.

If you want to see what leech and leech_yahoo_terms can do, please go to: http://leech.devseed.org/ . On that site you can also see a cool feature of leech's sister module url_profile: url_profile accumulates feed items from the same host. This is particularily interesting if you subscribe to keyword feeds like from news.google.com or blogsearch.google.com or other aggregators like Planet Drupal.

Alex

former username: lx_barth

funana’s picture

funana commented

1) Alex_b already answered this question

2) Yes, I set up one a few days ago: http://infopirate.org

fowlkes’s picture

fowlkes commented

Sorry, I got wrapped up in a bunch of development crap and didn't get back to this after that last post.

Standard agg doesn't create individual nodes per news item like agg2 does. I need this if I'm pulling feeds from somewhere else and I don't want to have to link back to another site.

Sorry again for the really late response. I'm usually on but I was off for a while.

K

khoogheem’s picture

khoogheem commented

For someone that has a couple of sites running leech I can say that the code is much cleaner and easy to edit vs. leech.

One of the big things I like with leech is the leech_yahoo_terms module, but found it was very easy to incorporate into feedparser by adding the node_api call right into function feedaggregator_node_insert($node) in the feedaggregator_node.module right before the db insert. (maybe not where it should go but it works ;) )

alex_b’s picture

alex_b commented

probably you want to do the small step and publish leech_yahoo terms as a standalone module: http://drupal.org/node/107814 ; )

- alex

former username: lx_barth

khoogheem’s picture

khoogheem commented

I actually was able to integrate leech_yahoo_terms into feedparser pretty easily.

khoogheem’s picture

khoogheem commented

I actually was able to integrate leech_yahoo_terms into feedparser pretty easily.

publishing’s picture

publishing commented

Hello K,

I saw your post re integratingLeech_yahoo_terms into feedparse. I am interested in utilizing leech_yahoo_terms, and was wondering if u have any information as to the "how to" of the integration? or if u have the patched files?

Thanking u in advance,

funana’s picture

funana commented

I understand well that you find Leech a little bit.... errr nerdy... But I'm really into aggregating content with Drupal and have collected a lot of experiences with it. I had the same problems like you and you can find comments of me here in the forums that are very desperate.

I didnt give Feedparser a try but I can say that Leech really rocks.

Once you have set it up it will fit all of your needs. I am very satisfied with it. And it is very simple to use.

If you need help with Leech, just contact me.

alex_b’s picture

alex_b commented

Leech is working fine. The code won't win any beauty contest, but we manage to improve it step by step. The documentation is rudimentary. I can only invite users of leech to contribute their experience and improve what's there!

fowlkes, if you have any particular questions about leech, don't hesitate to post them to leech's issue queue, I'd like to learn from them.

cheers, alex

former username: lx_barth

drriley’s picture

drriley commented

I'm going to make a suggestion regarding RSS -- and that is to not use modules. One of the reasons I decided against using modules for this task is the cost to my server resources, which are already taxed with a high volume traffic site. The option that I found that I liked was to put feeds in iframe tags -- and I can place them anywhere on my site that I want (in a block, in a node, etc.).

I am using this service to create the tags:
http://www.rss-info.com/en_rssinclude-simple.html

You can see an example of the feature in action via this link (warning: adult content/NSFW):

http://www.pornoworldonline.com/newest_dvd_releases_from_pink_visual