It would be nice if leech had the ability to limit the number of feed items created in a single run (either per feed, or globally). This would

a) stop a site getting too flooded with new content if a feed posted lots of items

b) allow other things that are triggered by node creation to happen

c) stop resource-heavy node creation causing a max-execution problem

Comments

alex_b’s picture

alex_b commented
Version: 5.x-1.8 » 5.x-1.x-dev

This is an interesting feature request, because I am rather leaning to the opposite :)

- remove "update frequency of feed" and "nr of feeds to visit per cron" settings
- let the module control how often to visit a feed (checking a feed for new content is relatively fast and can be throttled easily by observing the feeds usual update frequency).
- let it download for a given amount of time (UI would allow you to set how many seconds the module has time for downloading)
- tell user, when the given amount of cron time is not enough to download all items and which feeds didn't get updated.

In the end, when downloading is getting a serious performance critical issue, the update frequency and nr of feeds to visit per cron settings are kind of a joke - it's too hard to tweak 100 feeds so that they all come home in the available cron time. So, when we add a nr of nodes setting on top of that, I really don't think we are going to make things easier for the site maintenance.

So, addressing your questions from your post:

a) (content flooding) - i think this should be handled on a moderation level - limiting the nr of nodes coming in seems pretty random to me
b) (node creation actions) you could equally trigger actions when a node gets published (doesn't action module do that?)
c) that should be addressed by the set up I explained above.

alex_b’s picture

alex_b commented

- hit the submit button too early. So here's the question I actually wanted to get at: Would a configuration like the one explained above satisfy your requirements?

lyricnz’s picture

lyricnz commented

I'm not sure the approach you describe would satisfy my requirements, but agree that limiting the number of nodes created per run is a bit of a coarse approach. Perhaps if I quit being abstract, and describe my situation?

- I have a RSS feed, which refers to data which I want to download, and use to create a node. In my case, this is a podcast (ie: mp3 enclosure), but I can imagine other usages.

- Let's say I have my own parser (to save the extra information from the feed, like enclosure details), and my own item processor (that uses the extra information to curl-fetch the mp3, and create an audio node)

- Now obviously this fetch of the mp3 takes quite a few seconds, so I can't do all of the available items with the PHP timeout. So, I need a way to limit this (on the fly, or pre-configured, either way).

My local hack just limits the number of items created per run, but of course, I'd rather solve this in a general way.

So my primary drivers are :

- avoid php timeout

- avoid thrashing the RSS publisher, if I'm fetching stuff from her site

- (maybe) stop too many items arriving very quickly, giving people "50 new items". Not sure about this one.

lyricnz’s picture

lyricnz commented

Uh, forget that bit about my own parser etc. I've been using feedapi for a few weeks, and have it on the brain. My drivers remain the same, however.

alex_b’s picture

alex_b commented

Glad that you're using feedapi :)

I still think that limiting the download time is the way to go and also just posted a feature request to feedapi: http://drupal.org/node/170221

But I see your use case independent of that. You would need sth like a download queue that you can tell not just to "download everything that's there and delete items older than X days" (that's possible in leech and feedapi), but you want to configure sth in addition to that that says "only download new item if less than X items present".

In this case, I don't understand why you don't want to have the latest items... ?

Or are you thinking of a setting like "Only keep X latest items"?