Add Priority Setting for Feeds

Thank you.

FeedAPI's approach is to check feeds as often as possible, but not more often than once in 30 minutes (as of 1.x-dev which will be Feedapi 1.1 - which is in the works).

Defining the Feed's update rate is a very hairy business. Depending on the hardware you are running, other cron intensive modules there are activated (e. g. search) and on the number of feeds you have in the system, you get to a point very quickly, where it's not possible anymore to guarantee update rates - so you might have a setting to run updates every hour, but actually it takes you 12 hours to visit all feeds once.

So, instead of doing per feed update settings or per cron update settings which were very common in modules like leech, FeedAPI is using the approach of update as often as possible. This is complemented by a better reporting on how often a feed has been updated (management page, to be improved) and excellent caching by parser simplepie and a new patch for upcoming 1.1 which won't send feeds that haven't changed at all through the FeedAPI processing stack.

All that said, I would love to hear why you're asking for this feature - what's the user story here - or: what problem are you trying to solve by defining update rates on a per feed level?

Alex

I have the same need as MJH, and I have a user story. My site aggregates many, large feeds of different types:
a - Feeds that contain rapidly changing data (think financial quotes)
b - Feeds that contain slowly changing data (think scheduled maintenance events of a system)

As you stated, there is not enough time to process everything so that all feeds are up to date all the time, but I would like feeds of type a) to be privileged and statistically be updated more often than feeds of type b).

So my requirements are not as strict as those expressed by MJH, instead of scheduling I would like to be able to represent something like "dont update this feed more often that every x minutes".

I am using feedapi 1.0 and the common_syndication parser. Also, from what I see in the 1.1 dev branch, caching kicks-in only after parsing has occured (and just parsing can already take a lot of time).

I think I am going to take a stab at implementing this feature.

I can see in the code that there used to be a feed->refresh field in the database, and that the code still uses it to determine if the feed should be processed, but I don't know how to set this value on a feed after it gets loaded (and before it is parsed or processed).

Do you think it is a good idea to simply write an "update_throttle" processor that runs at the end and changes the feed->checked attribute to some time in the future (provided the feed was not half_done) ?

Nicholas

Subscribing, +1 for this feature! Greetings, Martijn

@nicholaswolff this is exactly what I was thinking. A way to give priority to important feeds with fast changing information.

I know what alex_b means though regarding unable to guarantee the feed will be parsed in any good time.

Did you get anywhere with your development nicholaswolff?

nicholaswolff made a good case. I think the event feed/financial data feed is a compelling example.

budda, I like your suggestion that the skip bit could have been implemented as time interval setting (http://drupal.org/node/232591#comment-809528 )

I'd like to get to a good approach for this. I am not happy with fine grained "once every hour" style settings because they create the impression that an aggregator could guarantee these frequencies.

What about a rough splitting? Think of a drop down on the feed node edit page and the content type edit page that looks like this:

Check feed
"as often as possible" (1) - default option
"twice a day" (43200)
"once a day" (86400)
"never" (0)

that could keep things simple while offering a way for people with specific needs to form alter this drop down and change it to their likings. A neat feature could be here to show the user in the help text of the drop down what's the current maximum update rate. We know that based on information in feedapi_stat table.

Another suggestion:

We should implement this not as a last_updated timestamp, but as a next_update timestamp. This could allow us to have add on modules do more fine grained control. Say: a smart scheduler that sets the next_update timestamp after it got a notification from a remote site.

Anybody up for a patch?

The next timestamp instead of last updated timestamp is a great point. Didn't think of it that way round.

If a patch was to be created for this functionality, could we just hijack the skip bit already put in to the database in the last version release?

Very much interested in this fuctionality. Anyone up for a patch please?
Thanks a lot in advance.
Greetings,
Martijn

Any luck or any requests for this feature (preferably for the D6 branch)?

Prioritization of feeds would be a nice way of spreading the stress on cron across time...

Changing subject to reflect the current status of the post.

all my feeds need to be refreshed at very frequent intervals.

alex_b...when you say it refreshes "as often as possible", how often is that? what is the criteria?

(and I'm using D6....but I didn't want to change the version on this request)

"what is the criteria?" :

// Minimum time that must elapse before a feed can be refreshed again on cron.
define('FEEDAPI_CRON_MIN_REFRESH_TIME', 1800);

in the begining of feedapi.module
If your special use-case requires really frequent updates, just alter this value.

fixed in 1.8, time to upgrade :)