I have been using feeds for a site these last few months, I have setup 33 incoming feeds, both CSV and XML.

When I run these through the UI they run perfectly and complete as expected - creating/updating as required.

I have them scheduled to run automatically once a day through cron. Cron is called twice an hour.

When everything is reset, some of the feeds complete but ~9 hang and never complete - stopping any future runs of feeds in the queue.

The feeds source table contains values for state and fetcher result similar to :-

state
a:4:{s:10:"start_time";i:1353892518;s:5:"parse";O:10:"FeedsState":8:{s:8:"progress";d:0.316131983410587957816773041486158035695552825927734375;s:7:"pointer";i:51833;s:5:"total";i:163960;s:7:"created";i:0;s:7:"updated";i:0;s:7:"deleted";i:0;s:7:"skipped";i:0;s:6:"failed";i:0;}s:7:"process";O:10:"FeedsState":8:{s:8:"progress";d:1;s:7:"pointer";N;s:5:"total";i:0;s:7:"created";i:0;s:7:"updated";i:50;s:7:"deleted";i:0;s:7:"skipped";i:0;s:6:"failed";i:0;}s:5:"fetch";O:10:"FeedsState":8:{s:8:"progress";d:1;s:7:"pointer";N;s:5:"total";i:0;s:7:"created";i:0;s:7:"updated";i:0;s:7:"deleted";i:0;s:7:"skipped";i:0;s:6:"failed";i:0;}}

fetcher result:
O:22:"FeedsHTT
(although the fetcher result field says it's a (Binary/Image).

There is nothing in the error log, in the feeds log or in watchdog to indicate why they failed.

When I reset the fields by running :-

UPDATE feeds_source SET state = 'b:0;';
UPDATE feeds_source SET fetcher_result = 'b:0;';

The feeds begin again and then eventually stall again.

Can anyone help shed light onto what the problem might be? Are there batch settings I could play with?

Comments

Similar issue. Subscribe

This issues might be related to #1553190: data not import during cron run, which has a patch (#7). You should probably try it.

Hi,

Thanks, I already had that patch in place after searching (I think!) exhaustively. I am still having the issue, I get no error messages in the php error log but running cron manually and checking watchdog I receive these logs (in order) :-

Warning: fgets(): supplied argument is not a valid stream resource in FeedsFetcherResult->sanitizeFile() (line 88 of /var/www/vhosts/mydomain.com/httpdocs/modules/feeds/plugins/FeedsFetcher.inc).

Warning: fclose(): supplied argument is not a valid stream resource in FeedsFetcherResult->sanitizeFile() (line 89 of /var/www/vhosts/mydomain.com/httpdocs/modules/feeds/plugins/FeedsFetcher.inc).

Warning: fopen(public://feeds/FeedsHTTPFetcherResult1354905901) [function.fopen]: failed to open stream: "DrupalPublicStreamWrapper::stream_open" call failed in ParserCSVIterator->__construct() (line 19 of /var/www/vhosts/mydomain.com/httpdocs/modules/feeds/libraries/ParserCSV.inc).

Warning: fopen(public://feeds/FeedsHTTPFetcherResult1354905901) [function.fopen]: failed to open stream: "DrupalPublicStreamWrapper::stream_open" call failed in FeedsFetcherResult->sanitizeFile() (line 87 of /var/www/vhosts/mydomain.com/httpdocs/modules/feeds/plugins/FeedsFetcher.inc).

Recoverable fatal error: Argument 2 passed to feeds_tamper_feeds_after_parse() must be an instance of FeedsParserResult, null given in feeds_tamper_feeds_after_parse() (line 19 of /var/www/vhosts/mydomain.com/httpdocs/modules/feeds_tamper/feeds_tamper.module).

Cron errors out and the feeds are hung until I clear the feeds_source fields mentioned earlier.

Running the feeds manually through the /import UI every single one completes as expected without error, running them scheduled with cron a few get through then they hang :( Really would like to get to the bottom of this one :(

Cheers!

I found this issue :-
http://drupal.org/node/1029102

It mentions a 'life' for temporary feed files, they have a lifespan of 6 hours (Drupal core). So if your feed is batched, is very large and takes more than six hours to run (say for example you're running cron once a day, or once an hour) then the file may not exist after a certain amount of time when the process comes back to run it again.

There is a simple patch here :- http://drupal.org/node/1029102#comment-6565716

Implementing this, at first glance, it appears to have freed up the scheduling again. Will see how it goes over the next 48hours and report back...

Status:Active» Closed (fixed)

Feeds haven't been hanging since that patch was implemented.

So you can either up the frequency of cron, or, implement the patch mentioned above.