when I set feedmanager to aggregate 100 feeds at each cron run it runs out of memory throwing an error like

Fatal error: Allowed memory size of 20971520 bytes exhausted (tried to allocate 179527 bytes) in /var/www/aggregator/modules/contributions/modules/feedparser/simplepie.inc on line 3993
Allowed memory size of 20971520 bytes exhausted (tried to allocate 1 bytes)

to make sure it's not a feed specific problem I changed feedmanager_cron() so that it will refresh a random set of feeds, it still gives the same error.

Comments

budda’s picture

budda
Primary language English
Location Cheshire
commented

It could be an issue with the 3rd party feed parser library SimplePie too.

As I don't have 100 feeds, or anything like that, available on my test setup its going to be hard to debug this easily. So any further details/research you can do on the issue would be appreciated.

alaa’s picture

alaa commented

I'll try to find out more but I'm not sure how to profile memory usage in php.

budda’s picture

budda
Primary language English
Location Cheshire
commented

Any further information?

How many feeds are you processing per cron run? (setting in feedmanager)
How many items do you have per feed on average?

alaa’s picture

alaa commented

messing around with different profilers in order to find out more, but since the pager $element bug I'm kinda worried it will be another weird and obscure thing.

I process 100 feeds per cron run, total number of feeds is around 1500, typically feeds either have one new item or none.

Chris Johnson’s picture

Chris Johnson commented

Are we sure this is a feedparser problem? cron runs all of the hook_cron() functions for all modules. Maybe the combination of all of the cron hooks together is resulting in the memory shortage, in which case it may or may not just be feedmanager.

alaa’s picture

alaa commented

no it is feedmanager cause it happens in the middle of feedmanager's hook_cron (that's the only thing I managed to confirm).

hickory’s picture

hickory commented

I tried to track this down a while ago, but never solved it. The memory usage was growing in a number of places, including during node_save, if I remember correctly.

budda’s picture

budda
Primary language English
Location Cheshire
commented

Could this be linked to the use of static variables in functions being used for temporary caching when creating new nodes?

hickory’s picture

hickory commented

Any idea how to go about profiling memory usage? My aggregator's (not Feedparser, but related) memory usage is going through the roof.

hickory’s picture

hickory commented

I believe this can be fixed by

a) unsetting the SimplePie feed object after it's been used in each loop

and

b) clearing the static cache in node_load once it gets above 100 (arbitrary number) items.