No new items on cron (issues with job scheduler configuration)

I have the same problem. For some feeds (not all) i have to manually import new items. Cron does not import them. Any ideas?

This is a major bug and has been a couple of versions of betas.

You find it in: http://drupal.org/node/705444
(issue closed for some reason)

Have tried to fix it myself but im new to this module and not a master on cron.

Can you link to the feed you are importing and explain your importer configuration?

That other issue was marked a duplicate of #916962: Fix undefined feeds_reschedule() when upgrading (feeds not updating on cron) but that's evidentially not the whole problem because I just checked out the latest from CVS and it's still not importing on cron.

This was working fine for me in beta 5. I did a jump up to beta 9 shortly after it came out and it's been broken ever since. I always delete the old directory before untarring (or checking out) the new one so that's not it for me.

re #3: I'm importing about 20 different feeds from different places. It's not the feeds with the problem because they import manually just fine. It's only on cron that it's not working. They're all set up so there's a feed node. HTTP Fetcher / Common syndication parser / Node processor.

I'm trying to poke around in the code but not finding anything so far. Any tips on where, exactly, to look would be great. :)

Thanks,

Michelle

Did you adjust the "hidden" job scheduler settings in the the settings.php? ( look in the readme of both modules ) They changed when feeds outsourced the job_scheduler.

No, but I don't see anything in either readme that says I have to set something to make cron work. Could you be more specific?

Thanks,

Michelle

#6/#7 No hidden settings need to be adjusted for having cron aggregating feeds. If you have changed a hidden setting to 'fix' something, you're at best working around another issue.

I am really interested in fixing you guys' issues with Feeds on cron. My simple problem is that I can't reproduce these reported issues. I run literally dozens of instances of feeds and there is no issue with any feeds not being updated.

Grab me on IRC.

Michelle, try setting this to an very high number ( like 5000 ) and see if it works then

Name: job_schedule_num
66 Default: 5
67 Description: The number of jobs to execute on cron time.
68 Only has an effect if Drupal Queue is *not* enabled.
69 http://drupal.org/project/drupal_queue

from job_scheduler readme.

No hidden settings need to be adjusted for having cron aggregating feeds.

But they need to be adjusted to make feeds work as expected. And i'm pretty sure that most of those errors are simply "not working as intended" errors. E.G. feeds is working fine, but only updating 5 feeds with no new content, which results in the user thinking something's wrong.

Look into the old "feeds not updating" thread. There where like 10 people saying its not working ( including me ) and after someone posted that fix ( to simply higher the schedule_num ) none has every posted there again :)

*XiaN also pokes on #914542: The hidden variables should be editable via an admin setting page again :P*

try setting this to an very high number ( like 5000 ) ... But they need to be adjusted to make feeds work as expected.

Please. We have discussed this before. It should be clear that this is not going to peform and hence this is not a solution. If you do that and you have a very high number of feed subscriptions, you also need to adjust your max execution time and your max memory size accordingly. You'll see that you quickly get to numbers that aren't practical. High execution times will expose you to DoS attacks, the max memory size will be at ranges that you simply don't want to allocate for your apache processes (> 512 MB).

XiaN Vizjereij: could you please stop handing out this advice? I repeat, setting job_schedule_num is *not a solution*. It's not helpful to keep harping on it.

That said, Michelle: could you set it to 5000 and see whether this fixes your problem? If yes, we have another indicator to go on in order to figure out what is going on.

If you do that and you have a very high number of feed subscriptions, you also need to adjust your max execution time and your max memory size accordingly.

Of cause you need to do that and there is nothing wrong with it.

You only need to

* Mention it somewhere in the readme. So that people know what they are up to, when they want to aggregate a large number of feeds
* Provide a way to limit these values somewhere in the drupal UI (!). Thinking behind it is : "Ok, the unlimited values to not work for me any more, i now need to override them with capped numbers"

But i can only say it again : The user expects that EVERY feeds updates on cron ( unless he explicitly says not to do so ) and that every item gets updated/created ( unless he explicitly says not to do so ). The user does NOT expect that only a few number of feeds get updated, that only a limited number of nodes get created/updated and that he has to edit those values somewhere deep in the settings.php. Btw. no other module i used so far required a manual setting in the settings.php

XiaN Vizjereij: could you please stop handing out this advice? I repeat, setting job_schedule_num is *not a solution*. It's not helpful to keep harping on it.

I sure can. But from 5 people on IRC and 3 friend of mines asking me why feeds is not working, 7 reported that my advice helped them ( fixing the problem ). They all had small amounts of feeds and unfortunately only the onces without new content got updated on cron. Which resulted in them thinking its not working ( and also why it worked on manual import ). The other one had his MySQL server dying from #625196: Fatal errors (Unsupported operand types) and warnings (Argument is not an array) in FeedsConfigurable spam in the watchdog :D

Thats why i keep poking that issue. Not to be rude or give wrong advice.

I think i would also help identifying the rare real problem with feeds on cron, when we could sort out all the "false not-working-as-intended" once.

With help from Alex on IRC, I got my live site working. The trick was to change the feed refresh rate to something else (you can change it back again; just save in between) and then run cron 2, maybe 3 times. Running it just once isn't enough, which confused the heck out of me. LOL!

He wanted me to note my upgrade path since it seems to be upgrade related. Unfortunately, my upgrade path is messy. I started using it in December of 2009 which would have been either alpha 8 or the dev right before alpha 9. I was on a patched version for a very long time waiting on #632920: Inherit properties from parent feed node (taxonomy, author, OG, language) and I only upgraded feeds sporadically when it started getting horribly out of date because every upgrade required me to repatch it. I know I was on beta 5 and it was working fine there as it had been all along. I then jumped right to beta 9 and it broke. So my guess is it was in that jump that the problem happened.

Michelle

I am seeing this issue on a clean install of Feeds (and the fast feed module), having had no prior versions of Feeds installed. Items got imported initially but then nothing new gets pulled in on cron.

geodaniel: grab me on irc, let's work through this issue.

Alex, I'll try and catch you on IRC today. I tried out the tip you gave to Michelle, and updated the refresh rate for the feed. This morning I came in to find there are new feed items that have been pulled in, so it must have done the trick.

Too bad - I am having the same problem.

No new content by running cron. Only manual.

I am also using 6.BETA-9 ( http://drupal.org/node/957654#comment-3651812 ) and I tried Michelles Trick under http://drupal.org/node/927782#comment-3623886 but for me this didn´t work.

subscrine

Same problem, with:

Feeds 6.x-1.0-beta10
Chaos Tools 6.x-1.8
Job Scheduler 6.x-1.0-beta3

Running cron manually works.

I am also having the same issue. I have all of the latest modules installed. Manual updates of the feeds work, but I can't seem to get them to update on cron, I've set up a cron job using cpanel with the following command:

cd 'my_file_structure_path' ; php -q 'cron.php' > /dev/null ;

I'm in the Drupal Support IRC channel as "Poet" right now.

I'm having the same issue... brand new install. I create a feed and it will only pull in items if I do a manual import. I'd be happy to go on IRC to try and get this resolved. I have an hourly cron job that runs through crontab, and I've also tried drush cron, and cron from the admin_menu. No new items, unless I do a manual import.

Sorry, now seems to be working for me. I basically uninstalled job scheduler and feeds and reinstalled them, and I also set the minimum time to refresh to "as often as possible" Resetting it to 30 mins still seems to work as well.

Same with me. I run drupal 6.19, feeds in beta10. I built my website over alpha5 or near that locally, then upgraded to beta9 and moved the whole thing online. Then I sarted getting duplicate items on every cron run. After I uninstalled beta9 and installed fresh beta10, and assigned unique elements, both URL and GUID, I never got dupe again, but my feed stopped getting updated. The site runs one single feed now, it's a yahoo pipe. At first, I thought all troubles derrive from quirky URl format or perhaps standards that Yahoo uses. Then I burned the feed with Feedburner and nothing changed. I tried to run cron manually (click "run cron manually" on Status reports page?) – no new items.

So. I'm not getting any updates to my feed. What am I doing wrong?

I am having the same problem on Drupal 6.20. Automatic updates used to work but now I can only get new nodes through manual import

Subscribing...

Using beta10 since the get the get-go, feed updates (set to minimum 1 hour in Feed Importer settings) worked for 2-3 days, then I noticed the site is now about 24-48 hours behind in updates. Cron is set to run every 15 minutes. We have about 70 slowly updating feeds.

Did use Drupal Queue for a bit, but that+Job Scheduler is plain broke for D6 pending this patch being backported, so I uninstalled it.

Gonna delve into the code/deep settings.

Actually, I'm pretty certain this is a duplicate of #705444: Feed Nodes don't import on Cron with Feeds Beta7 + Job Scheduler.

What's strange is that I just copied the database from our live environment - which exhibits this bug - to my dev box and, despite the code and databases being the same the feed updates started working again. Immediately.

The only two differences then, since DB and code is identical, is the OS and PHP version and the use of SSL: my development environment on my Ubuntu box which runs PHP 5.3.3 without SSL, whereas our production & test environments run Debian Lenny, which has version 5.2.6 with SSL for admin & user pages...

Could it be that the various issues along these lines (this, #705444: Feed Nodes don't import on Cron with Feeds Beta7 + Job Scheduler, #959406: beta3 + Feeds = no automated job. etc) that Alex_b has no issue with, but other do, are simply operating environment differences? Are there any places in the Job Scheduler or Feeds code that could introduce this bug due to a difference PHP a version difference? Or does SSL louse things somehow?

UPDATE
Feeds on the test environment also started working after importing the database in backup/migrate. So the PHP verison/os is not relevant, but something that happens after restoring a database. Something happens after a DB import...

Gonna keep digging... (might be barking up entirely the wrong tree, but something odd this is going on...)

OK, after importing the live environment's database back into the live environment, feeds started coming through. Hmm...

So I looked at the job_schedule table and saw ~480 rows. We have only 76 feeds, so including the housekeeping and a couple of other 'special' feeds, that's 79. So I truncated that table and manually ran feeds_reschedule(TRUE) and 79 rows appeared in the table.

This appears to be a bug somewhere between Feeds and Job Scheduler because it seems deleting a feed node does not remove references to it from the job_schedule or feed_source tables. (Update: a manifestation of #797228: Deleted Feed records still in database tables)

Anyway a quick calc: 480 (rows) / 5 (feeds per cron) * 15 (minutes between crons) = 1440 minutes = 1 day. Which is about the time I'd not seen any updates for...

Could this be an issue for other people? I mean, if the job_scheduler table fills up with dead entries, could the cron scheduled tasks could spin blindly, either indefinitely or for long enough to give the impression of not updating? Could re-importing the db have cause the feeds/scheduler to start from the beginning of the table again, giving the impression of being fixed?

Not saying this is the only issue, but I'm fixed for now. Will keep an eye on it an report back...

Hi

I am trying to import a csv file and when this csv file is updated for the changes to be picked up and changed in the appropriate nodes.
I have set feeds to refresh to every 15mins but its not doing anything?
Also how do i manually run the cron job for feeds refresh

Pete

Please do not change active issues to another topic. If you need help, create another issue or search the issues list and documentation. And don't assign an issue to yourself unless you are personally going to manage and fix it! Good luck.

Subscribing.

I'm having a similar issue. One of my importers works fine. The other is configured essentially the same way, but will not import anything on cron.

Hi bmgrimes, can you try the thing that fixed it for me? See #25 but essentially:

1. Truncate (empty) the job_schedule table in the database (e.g. truncate job_schedule; at MySQL command line or PHPMyAdmin)
2. Run feeds_reschedule(TRUE); (I just added that line to a page with PHP input format and viewed it)
3. Run cron a few times

And let us know how that goes please!

Thanks a lot Jim, that worked perfectly!

It seems that the feeds I added to existing nodes weren't getting added to the job_schedule table. I think the feeds that were created with new nodes were updating properly. Running feeds_reschedule(TRUE) after truncating the table corrected the issue.

My job scheduler table looks normal but the posts are being imported. Have no idea why

Job Scheduler issue: #959406: beta3 + Feeds = no automated job.

Let's make some FAQ to track this bug.
First of all:
1. feeds_cron() doesn't do importing, job_scheduler_cron does.
2. For devs: If you removed the record manually from job_schedule, saving Feeds Importer form won't help, because you've to change 'Minimum refresh period' to different one (at: admin/build/feeds/edit/%/settings)
See:

// File: FeedsImporter.inc
  public function configFormSubmit(&$values) {
    if ($this->config['import_period'] != $values['import_period']) {
      feeds_reschedule($this->id);
    }

3. If you use Devel, disable it.
4. For devs: Make sure that your cron is not locked out (especially when you did some die()'s in the code, it doesn't release the semaphore).
drush vdel cron_semaphore || variable_del('cron_semaphore');
5. To be able to import anything on cron, you've to have at least one record in job_schedule table.

Bug 1:
Even if you did the change of refresh period and feeds_reschedule() is run, it still doesn't create job_schedule row.
Bug 2:
No matter what refresh period you change, it's always set as 1h (job_schedule table).
By default job_scheduler setting the new job at the next period time. So if you set up every 15min, this job will be executed after 1h anyway (doesn't matter that you executed the cron manually).
Theoretically you can shorter that, by executing following query:

UPDATE job_schedule SET next = UNIX_TIMESTAMP();

This bug fixing Bug 1, described in #33

Patch fixed Bug 1 & Bug 2, described in #33

@kenorb: thanks for the updates... I can't test the patch for a while but will do as soon as I can.

Don't forget bug 3 also affecting people here as I alluded to in #25: #797228: Deleted Feed records still in database tables - It's quite possible to have hundreds or thousands of orphaned job_scheduler entries...

I think the configuration is very confusing.
I typed URL in the Description by mistake. Then I was looking half an hour where I can set up the URL in Feeds Importer.
Finally following the wiki page: http://drupal.org/node/638804
I release that there is some separate page /import
Where you have to import it manually.
I thought it will be automatic.
Then even when you import it and run cron again, still nothing happen.
So crazy.

To those who didn't import:
http://drupal.org/node/638804
Try: /import
Don't forget to choose Parser, XML wasn't working for me, I chosen CSV.
Make sure that you have write permission to files.
In my case looks like something being imported.
I'll test as well if it does work on cron.

I'm still trying to understand what should happen when cron runs a batch from an import.

If my feed has 2000 items, and my batch is 150, and my feed update period is "often as possible", and cron is set to every 5 minutes should it

a) iterate through all 2000 in batches of 150, with each cron running a batch, or

b) each cron processes 150, but because I have a period of "often as possible", every time cron has feeds look at the feed it thinks everything needs to be processed again, meaning all the things that were batched in the previous cron get batched again, leading to an ever growing queue?

I'm hoping for a obviously, but I'm not confident.

And of course the feeds import form manages to run the entire import in one long batched process. Does anyone have instructions or an idea as to how to achieve this on a single cron? That would be much more to my liking than these 5 minute interval small batches, with a feed that I know will be refreshed once per day at a known time.

I achieved cron job by testing CSV, but XML parser looks like it doesn't work at all.
I've downloaded Simplepie as well, no difference.

I've performed a full reinstall of job_scheduler & feeds, and everything seems to be working ok.

Just:
Uninstalled (both feeds and job_scheduler)
Deleted everything (folders)
Enabled job_scheduler + feeds + feeds admin ui (in this specific order, and step by step - i've read in another thread that problems can arise when a dependent module is not enabled previously). I don't recall if i reinstalled Chaos Tools, but it's possible as i was having issues with Context also, so if you want to be sure, include that as first on the list, as listed below:
- Chaos Tools
- Job Scheduler
- Feeds
- Feeds Admin UI

Avoid enabling all at once, enable one at a time.

Working with
- Chaos tools 6.x-1.8
- Feeds 6.x-1.0-beta10
- Job Scheduler 6.x-1.0-beta3

Ok, in my case my XML was not well-formated as Atom feed.
I've installed: http://drupal.org/project/feeds_xpathparser

If you have similar issue, check using some XML validator:
http://validator.w3.org/
http://www.validome.org/xml/
http://www.w3schools.com/xml/xml_validator.asp
etc.

Make sure that you XML file starts with proper xml tags (<?xml ...?>)

I think, there should be at least some errors informing that you XML is not right.
In my opinion following issue is too generic, so if you have similar problem, please raise a new ticket with more details.
Created new tickets:
#1116804: Provide some messages and logs if feed file is not well-formed

Its still one of the stalking problems on feeds. Lets keep this open so other people can find help here.

Wrong status >.<#

Actually, this should still be at "needs review" because the patch at #35 still hasn't been committed to the module. I don't think it should be "needs work" since the patch is not proven to have issues yet.

Confirmed that the patch at #35 gets feeds items importing on cron run.

Also confirming that #25 fixes the issues with cron.

#35 alone did not fix this for me, but I did get around this

The reason was that drupal_queue was not installed, which makes a difference in the job_scheduler:

$num = module_exists('drupal_queue') ? variable_get('job_schedule_queue_num', 200) : variable_get('job_schedule_num', 5);

So if you have more than 5 cron jobs scheduled, only the first 5 would run.

Either increase the default in the code or install drupal_queue

@snez, where did you put this code?

Could #35 be commited? It fixes the problem.

Permissions

I'm pretty sure #35 fixed the problem for me as well - I say 'pretty sure' because I applied the patch, re-saved my importer, ran cron manually, and nothing happened. But the next time cron ran after that, it did work. Weird.

Confirming that patch #35 worked like a charm. Whether I manually run cron or schedule cron, the feeds now get imported.

Please commit. Many, many thanks!

A few things:

• Patch should apply from root of the feeds directory.
• I don't see anything in this issue mentioning problems with expiration.
• + $this->config['import_period'] = $values['import_period']; is not needed at all.
• $this->schedule(); duplicates what happens in feeds_reschedule().

That said, I agree that we should move scheduling of importers to submit and remove it from feeds_reschedule().

See #1446516: Feeds begins automatically deleting nodes for some ideas.

This is just a repost of #35 in Git format.

HI there is the above patch part of 7.2 dev. Or do i have to patch it? . I am having the same isssue with the feeds not imported on cron. Iam using feeds 7.2 apha8
Many thanks

open editing