Feeds Not Being Updated

Try the DRUPAL-5 branch from CVS? There are some bug fixes in there that may solve your problem.

Does this only happen on some feeds while others update without a problem?

Having the same issue here. Cron is running just fine, feeds update manually - but not automatically. I have it set to update feeds every 15 minutes, but it's a no go.
Using the latest 6xdev.

From what I understand, this is happening in the Drupal 5 branch as well. If you have any details please provide them. I'm going to try and write up some code to track what's going on at some good test points and launch it on a site to model what's happening.

When refreshing a feed manually, it does not matter how long ago the feed was refreshed, it will always pull new results if it finds them. When refreshing a feed via cron, a feed will only refresh every so often based on your settings for that particular feed.

If you check the feed edit page you will see an area called "Feed Settings" and, under that, a field called "Check feed every". By default this period is an hour, so your feed would only be refreshed every 4th cron run (if set to run every 15 mins, every 60th cron run if set to run every min, etc).

Simulating the conditions (using Drupal 6), feeds refresh correctly on cron once an hour has elapsed since the last cron run, but will not refresh (nor show status messages) until that period has elapsed.

Could this be the issue that you are encountering?

A relatively simple check for this (if you can update mysql directly) is to revert the checked date back to 0 before running cron. If you do this, and your feed does _not_ refresh correctly on the next cron run, please post any further details you might have that could expedite fixing this issue. Thanks!

@csevb10 - There has been a problem with feeds not updating when they are supposed to. That was the reason for http://drupal.org/cvs?commit=125012. It looks like that may not have corrected all the problems.

Ok. I've replicated the issue at this point, and I'm researching the solution.

@mfer: Sorry, got pulled into a 3 hr meeting, but I did a little more research, and the problem seems to be unrelated to manual or automatic refreshing of the feeds in my tests. It appears that the actual issue is with feed expiry. The cache is set for an hour term by default from simplepie (since we're now using their caching methods). As a result, regardless of the period that you set as a user, your feed will be cached for an hour. This might cause minor race conditions if the values are set to the same value (15 min interval, say, for both the feed and the simplefeed cache), but even then, the cache timeout would at least coincide with the refresh period. The simplest solution is probably to wipe the hash value whenever you re-save the feed node, and set the timeout to coincide with the selected period. I'll do a little more work on this and let you know.

@mfer: Ok. I'll put together a patch and ping Ted to review it tomorrow. Clearing the hash is the easy solution to forcing a feed refresh, but it orphans cache files, so it's obviously not the preferred solution. Right now we're running 2 timing methods side by side: ours (refresh + checked) + simplefeed's (mtime). I'll check if there's a good way to do a cache flush tomorrow so that things sync well and put the patch together. Thnx.

Actually, maybe we can remove the local SimplePie file cache support stuff now.

Way back in 1.x, we used SimplePie cache locally for the feed. SimplePie checked that cache locally and requested the feed to see if change. Cost = file lookup + HTTP request

Now, in 2.x, we store a hash of the feed. If we remove SimplePie cache, we still have our HTTP request, and just compare that to the stored hash.

In theory should run a bit faster, and be simpler setup.

What have you Bill and Matt? :)

Scott seems to be in favor of removing SimplePie cache too... he wrote this to me:

When I looked at this briefly I decided the issue probably was in the
simplefeed_feed_refresh() on the limit query, cause that appeared to be
the only difference between refreshing a feed by hand and refreshing it
via cron. Be nice if we had a someone test to make sure that unlimited
fails similarly.

>From what I gather the issue is the following:
If SimplePie's cache is still valid, it grabs that cache stuff in init()
(and returns true). Therefore, that passes the if($success && data) but
the $feed_node->hash == $hash, thereby skipping processing the nodes.
But evently SimplePie's cache will get old and then $feed_node->hash !=
$hash, thus it will parse the new stuff. SOOO not sure if its the
problem.

Getting rid of it would make it faster though thats for sure and FAR
easier to follow what is going on(my head hurts....). It IS getting in
the way, though not convinced it is the source of the problem. I think
it is the way it is updating the table, EVEN if the $feed_node->hash ==
$hash.

Furthermore, might want to add another else{} right here:
if ($success && $feed->data) {
...
}
else if ($feed->error) {
...
}
else {
watchdog("UNKNOWN ERROR!!!")
}

That would hopefully, produce the error spot as well by producing more
output

I'm in line with what Scott and Ted wrote. It would be simpler and faster to remove the cache and it would be one less place our problem of not updating could be happening.

I've also used something like

if ($success && $feed->data) {
...
}
else if ($feed->error) {
...
}
else {
watchdog("UNKNOWN ERROR!!!")
}

when I've been testing. This might be a good idea to put right into the module. Thoughts?

@csevb10 - are you still up for rolling a patch? If not, I'll volunteer.

Sorry. I totally dropped the ball on this one. Got busy and forgot to follow up.
When I traced it back the issue was that, if a cached file existed, then the hash always came out equal to the previous data ($process_feed->hash). $feed->data hash is always equivalent to our old hash (since $feed->data is set to the cached version of the file which we created our old hash out of).

I'm down with removing the file caching (I definitely think it simplifies things when viewing the code and our hash validation will 1. work correctly and 2. should be adequate). I've attached the patch for Drupal 6, and all the changes are very straightforward. Just checking to make sure we're all on the same page and that I'm not missing something. If you want me to roll the D-5 patch, let me know. It's pretty quick and easy.

I didn't include the change of the if-else statement, so if we want to include that, we'll just need to roll a slightly larger patch.

Looks good to me. Just need to update the comments for that function to remove $cache_path tho, otherwise commit away!

Be sure to update README.txt with fix.

And definitely backport this to Drupal 5 (should be same fix) since that is broke too, thanks!

Then we'll test it a bit and put out a new release.

Seems like a straight forward change. Be great to add the extra else though to the if($success) block. Not totally convinced this is whats causing the error (though I think its the most likely suspect). Extra else gives more output on failure which is (Y)++ in my book

Bill if you don't have time to do that this week let me know and I'll roll and commit patches. We need to get this live on a few our of sites.

I should be able to do it tonight, probably later today. :-)

w00t!

Ok, this has been updated along with the readme for D5 & D6 branches. It has *not* been updated for 5--2-2 or any of the tagged revisions (which I believe is as expected). Perhaps we can now test, and then log issues back in here as necessary.
--Bill

Ya there is only the DRUPAL-5 and DRUPAL-6 dev branches.

We can test those and if not issues make an official DRUPAL-5--2-3 tag release and for 6 too.

Commits look good Bill!

One issue, in D6 you added

 else {
435	 	     watchdog('simplefeed', 'You shouldn\'t be here. Something has gone terribly wrong.');
436	 	   }

We should prob add to Drupal 5 too. After that feel free to close ticket! We'll reopen if these issue comes back up, I'm testing it on a live site right now.

Done & Done.

We're still seeing this issue on our sites...

Seems to be related to Feedburner: http://drupal.org/node/307686