If a feed contains more items than the server is able to process, it times out and nothing is imported. It's probably wise to throttle not only the maximum number of feeds, but also the maximum number of feed items to import at once.

Comments

webchick’s picture

webchick
she/they
Primary language English
Location Vancouver 🇨🇦
commented

There's an option in the admin settings (admin/settings/simplefeed) to limit the # of pulls per cron. Or is that not what you mean?

Jeroen Coumans’s picture

Jeroen Coumans commented

As far as I can tell, that setting refers to the number of feeds which are processed at the same time, not the number of feed items. Using just 1 feed with 100 items is enough to max out my servers memory.

m3avrck’s picture

m3avrck commented

This might be a server issue that can't be fixed. SimplePie has to grab the full XML file the server is sending out so it can parse it. While it limits things internally, that initial grab could be it -- RSS feeds from sites usually show the latest 10 or so because of just that.

Additionally, we ahve put in some minor CPU/memory optimizations that might alleviate this but I'm afraid that is all we can really do here.

m3avrck’s picture

m3avrck commented
Status: Active » Closed (works as designed)

You can choose how many feeds to process per run now so this is the best that can be fixed. SimplePie has to grab the whole feed to parse regardless of whether half of it goes in or not. It's nearly impossible to choose certain # of items per feed.