We've discussed this already on the aggregation list, I post this as feature request now.

Currently, we throttle downloading by:

- adjusting how often cron is being called
- setting the PHP time out time
- setting an update frequency per feed ("Refresh interval")
- setting how many feeds are being visited per cron run ("Per-cron feeds refresh")
- setting how many feed items can be created/updated per cron run ("Maximal number of items updated/created in one refresh round")

This is simply not manageable.

I would like to suggest:

+ Remove "Refresh interval", "Per-cron feeds refresh" and "Maximal number of items updated/created in one refresh round"
+ Let feedapi download for a given amount of time (UI should allow to set how many seconds the module has time for downloading).
+ Let feedapi control how often to visit a feed (checking a feed for new content is relatively fast and can be throttled easily by observing the feeds usual update frequency).
+ Tell user, when the given amount of cron time is not enough to download all items and which feeds didn't get updated - see: http://drupal.org/node/164204

Thus, tuning downloading will merely be a matter of

- adjusting the available cron time for feedapi
- adjusting cron frequency
- adjusting PHP time out time

How does this sound?

Comments

budda’s picture

budda
Primary language English
Location Cheshire
commented

What about making use of the Curllib multi threading capabilities to read and process multiple feeds at once?

aron novak’s picture

aron novak
Primary language Hungarian
Location Hungary, Budapest
commented
Assigned: Unassigned » aron novak

Budda, I assume you talk about curl_multi_exec . It is a PHP5-only function. I think it's absolutely reasonable to use this as a possibility. (as the module uses simplexml php5 stuff as an option - one of the parsers)
Alex, yeah, i agreed this solution at the list, and now i also agree it. It 'll be implemented. (that's why i assign it to myself :) )

aron novak’s picture

aron novak
Primary language Hungarian
Location Hungary, Budapest
commented
Version: 5.x-0.x-dev » 5.x-1.x-dev

Just a few ideas on the exact implementation:

  • via ini_get we can determine the maximal execution time of a script. This is useful for "adjusting the available cron time for feedapi" setting. Maybe we'd like to enable the user to set the available cron time for feedapi only the half of max. execution time
  • compute md5 on raw feed data joined with serialize($feed_node). Store this value in feedapi table. Only refresh feed if the feed node or the feed content itself changed. It can be done before the parsing. It could boost things especially when multiple parsers are turned on
  • measure the time with drupal functions. evident
  • adjust feedapi_cron and feedapi_invoke_feedapi('refresh') functions to collect all the mentioned informations (see Alex's another ticket linked in the first post)
  • add the new columns to {feedapi} table. Delete: refresh. Add columns: download, update_rate, update_num, duplication, md5
  • Stop the processing when the timer shows bigger value than the limit. It's possible to stop between feed items, not only feeds.
aron novak’s picture

aron novak
Primary language Hungarian
Location Hungary, Budapest
commented

"compute md5 on raw feed data joined with serialize($feed_node). Store this value in feedapi table. Only refresh feed if the feed node or the feed content itself changed. It can be done before the parsing. It could boost things especially when multiple parsers are turned on"
- this was a bad idea definitely. Instead of this FeedAPI will implement If-Modified-Since HTTP header.

aron novak’s picture

aron novak
Primary language Hungarian
Location Hungary, Budapest
commented
Status: Active » Fixed

Despite the fact that this is fixed in the CVS, the code needs testing.
What is done:
parser_common_syndication: support of If-Modified-Since and Etag and Last-Modified http headers
parser_simplepie: simplepie's internal cache is enabled. It boosts the things extremely.
feedapi: the download and process related settings become a lot of easier.
hook_cron: it is process as many feeds and items as possible. It's depend on the elapsed time

Anonymous’s picture

(not verified) commented
Status: Fixed » Closed (fixed)