Just a warning for those still using php 4.3.9, a potential problem. This problem seems to be due to a regex memory leak in php version 4.3.9, i don't see this problem using 4.4.4 with the same feed data. I can't find any specific PHP bug which may be the cause, but there have been quite a few regex lib updates to PHP between 4.3.9 and 4.4.4, and a few regex memory leaks reported in between those versions. It is triggered by a call to:

db_query("INSERT INTO {simplefeed_feed_item_cache} (data, fid, url) VALUES (%b, %d, '%s')", serialize($item), $process_feed->nid, $item->get_permalink()));

PHP Fatal error:  Allowed memory size of 33554432 bytes exhausted (tried to allocate 5028886 bytes) in /var/www/vhosts/.../httpdocs/includes/bootstrap.inc on line 588

in the simplefeed_item_feed_parse() function. I performed memory usage tests after iteration of the feed items before the db call to insert them into the cache table. After one particular feed, on PHP 4.3.9 memory consumption increased +10MB after a call to db_query(), specifically after this:

 $query = preg_replace_callback(DB_QUERY_REGEXP, '_db_query_callback', $query);

On 4.4.4, with the same feed data, same code, i only ever see an increase of 1-2 MB at the most and everything works just fine.

CommentFileSizeAuthor
#15 simplefeed_item_0.patch3.74 KBJHeffner

Comments

dayre’s picture

dayre commented
Title: PHP 4.3.9 Fatal out of memory error when running simplefeed_item_feed_parse() » Fatal out of memory error when running simplefeed_item_feed_parse()
Status: Closed (won't fix) » Postponed (maintainer needs more info)

Ok... sorry folks... must always remember to view "closed" issues and always visit page 2 !

This looks to be a duplicate of http://drupal.org/node/120473 and is due to a mysql conf packet size issue. I'll try it and report back.

dayre’s picture

dayre commented

The suggestion in http://drupal.org/node/120473 which reported the same error was to increase the max_allowed_packet parameter in you my.cnf mysql conf file. It seams the default is 1M.

On the PHP 4.3.9 server in which i'm having this issue, i had a look at the current setting... and it's this:

max_allowed_packet=100M

On my PHP 4.4.4. server where i'm not seeing the problem, the max_allowed_packet is set to 1M and it's working just fine.

So if 16M should have fixed the problem as reported in the other issue... then 100M should really fix it... but it doesn't.... so... this still seems to be an issue... and i'm thinking perhaps with a php 4.3.9 regex library.... but i don't have time to pull out my debugger skillz and really nail down the problem.

dayre’s picture

dayre commented

Just to clarify the issue some more:

Test feed:
http://www.gtrlabs.org/temp/nfl_news

php 4.4.4 
memory_limit = 8M
mysql 5.0.27
max_packet_size = 1M

Works fine with default low memory and low packet size setting.

php 4.3.9
memory_limit = 32M
mysql 4.1.12
max_packet_size = 100M

Doesn't work. PHP Fatal out of memory error even though much higher memory and packet settings. Should work ! Upping the memory limit will bypass the out of memory settings (in most cases), but then triggers a mysql packet size error.

Likely nothing to do with php or mysql config settings but a php bug.

JHeffner’s picture

JHeffner commented

I'm having the same Fatal Errors as everyone else, and I was able to find out more information. I found two feeds from movabletype blogs that one of our staff members created that breaks feed parsing with the "Fatal Error: Allowed memory size" errors.

Here are two examples:

1. http://www.personal.psu.edu/ejp10/blogs/gotunicode/

This breaks in /includes/database.mysql.inc on line 380 and imports up to the 12th feed. The 13th post ( http://www.personal.psu.edu/ejp10/blogs/gotunicode/2007/02/the_schwa_ups... ) which presumably contains unicode characters never gets imported. The feed parsed successfully through the SimplePie test.

2. http://www.personal.psu.edu/ejp10/blogs/tlt/

This breaks in /modules/simplefeed/simplefeed_item.module on line 393 but successfully imported all entries. This feed did not parse successfully through the SimplePie test.

The unfortunate downside is that once the parser breaks during a cron, the cron job stalls and has to be released manually. I'm thinking the problem is illegal characters in the feeds but I haven't had a chance to look further into it yet. I wanted to post what I have found out so it might help diagnose the issue.

JHeffner’s picture

JHeffner commented

I'm now thinking it is cache handling on feeds with enclosures. The same thing happens under PHP4 and PHP5. I also tried to increase my memory settings for MySQL,

JHeffner’s picture

JHeffner commented

Wow, did I ever have to increase the memory limit on MySQL. It is fixed now for me, but I also had to clear out simplefeed_feed_item_cache before I refreshed the feed items.

JHeffner’s picture

JHeffner commented

There is definitely a memory leak somewhere in the code.. there is no way simplefeed_feed_item_cache should get upwards of 10+MB entries just for simple feeds. This is where I have been experiencing the memory issues which still happen on certain feeds independent of memory limits.

dayre’s picture

dayre commented
Status: Postponed (maintainer needs more info) » Active

Well, i had an opportunity to upgrade a server to php 5.1.4. My original thinking was that this was a php version issue with php's regex library, but i still get this same fatal error.

I changed the status of this to active from active (needs more info)... since we do have more info from JHefner.

JHeffner’s picture

JHeffner commented

I bumped up the settings in /etc/my.cnf to 100M for max_allowed_packet size and still have issues. I found PHP5 did handle the problem better but not perfectly either. I'm trying to trace down the problem since this module seems to be the best one out there for feed parsing so far that I can tell.

The following feed breaks it every time http://www.personal.psu.edu/ejp10/blogs/tlt/ . It breaks when loading the feed into simplefeed_feed_item_cache. It appears as if the first feed data stored starts out at 55KiB increasing gradually up to the 12th feed item at 8.8MiB. I'm assuming it breaks on the 13th feed item.

I've been trying different configurations and looking at the code, without rewriting the way the caching works.

m3avrck’s picture

m3avrck commented

Well we can figure out if it SimplePie or the Drupal module testing it here...

http://simplepie.org/demo/?feed=http%3A%2F%2Fwww.personal.psu.edu%2Fejp1...

Seems to be the Drupal module, hmm...

JHeffner’s picture

JHeffner commented

You will want to test it off of the trunk version of SimplePie. The tried updating to the latest trunk as well.

http://php4.simplepie.org/trunk/demo/?feed=http%3A%2F%2Fwww.personal.psu...

m3avrck’s picture

m3avrck commented

Best bet is to hop in #simplepie on IRC and we can work with the SimplePie devs to figure this out...

alaa’s picture

alaa commented

I caught two memory bugs, one was actually caused by http://drupal.org/node/147314 where feeds with large number of new items and large number of categories would consume all memory due to the accumulative category bug.

the other seems to be PHP version specific cause it only happens with PHP 5.2.1 on my Mandriva test box

I'd like to investigate more but have no idea how yo profile mysql memory usage, how do you guys do that?

JHeffner’s picture

JHeffner commented

The problem occurs when using serialize on the simplepie item; possibly it has references to itself. Maybe creating a new array to hold this data temporarily and storing that in the cache might work.

simplefeed_item.module, ine 395
db_query("INSERT INTO {simplefeed_feed_item_cache} (data, fid, url) VALUES (%b, %d, '%s')", serialize($item), $process_feed->nid, $item->get_permalink());
      }

I'm going to code up a workaround for myself so I can get my site back up, but it will be far from perfect. The developers may want to code up some workaround of their own.

JHeffner’s picture

JHeffner commented
StatusFileSize
new simplefeed_item_0.patch3.74 KB

Here is the temporary patch I put in place. It's ugly but works, and I also noticed a significant speed increase; where I can set the throttle much higher now. It's been running for about 12 hours without any problems.

JHeffner’s picture

JHeffner commented

One week and I haven't had any problem with memory errors. I even set MySQL back to sane memory limits.

ezra-g’s picture

ezra-g commented
Status: Active » Needs review

I was having the same issue, even with a PHP memory limit of 256MB. The patch in issue #15 solved this issue for me! Updating to CNR.

m3avrck’s picture

m3avrck commented

Very interesting -- I'll be exploring this in detail next week but JHeffner I think you might be onto something with the multiple references issue...

JHeffner’s picture

JHeffner commented

m3avrck,

I've been running it for about a month now on 30 different feed sources and haven't had any issues with memory or the cron job dying on me. It also runs much faster than the original. I set mysql memory limits back to what they were before I started troubleshooting. I did change the patch to reflect the tags issue I posted about over at http://drupal.org/node/147314. I probably shouldn't have included it above.

m3avrck’s picture

m3avrck commented
Status: Needs review » Fixed

This is fixed, w00t! Thanks JHeffner -- I improved your patch a bit, fixed a few more reference issues, and improved it a bit.

Please confirm :-) Thanks!

JHeffner’s picture

JHeffner commented

Thanks m3avrck. However I won't be able to test until I have a chance to create a migration path for the new database schema. I can't lose the current data and just start over. I see you made the updates needed by the new simplepie. Nice Job!

JHeffner’s picture

JHeffner commented

you have a type mismatch somewhere. I'm looking for it now. All tags imported from feed items get imported as "Object".

JHeffner’s picture

JHeffner commented

It appears how categories are handled was changed in the new version of simplepie and should follow the example at http://simplepie.org/wiki/reference/simplepie_item/get_categories since it is returned as an SimplePie_Item. I tried to do this and can only get the first category from the feed however by using this code 

          if (variable_get('simplefeed_categories', 0)) {
            $tags = $process_feed->tags;
               foreach ($item->get_categories() as $category) {
                  $tags.= ', ' . $category->get_label();
               }
          }
m3avrck’s picture

m3avrck commented

Thanks JHeffner! I committed this change, not sure why only one tag would appear, but if it's still borked, pleae create another issue, let's keep this one closed since it's fixed now (or so we think ;-))

Anonymous’s picture

(not verified) commented
Status: Fixed » Closed (fixed)