The message that Cron run failed should, IMHO, be at least a warning level message not just a notice level.

The real problem is Attempting to re-run cron while it is already running. But it is not running, and has not run in over 3 days.

If it matters: Drupal 5.1 running on Windows XP (SP2), Apache/2.2.3 (Win32) mod_ssl/2.2.3 OpenSSL/0.9.8d PHP/5.2.0 mod_perl/2.0.3-dev Perl/v5.8.8, MySQL database/5.0.27

CommentFileSizeAuthor
#31 146551-cron-semaphore-reset-D7.patch698 bytesdave reid

Comments

catch’s picture

There's currently no way to fix this without doing:

DELETE FROM `variable` WHERE name = 'cron_semaphore';
DELETE FROM `variable` WHERE name = 'cron_last';

It'd be good to have a "restart cron" button which does this somewhere - maybe in /admin/logs/status next to "run cron manually".

nancydru’s picture

Yes, I did this. The "problem" is Update_Status and the length of time it takes it get data from DO.

BTW, I also added a check for this condition in my Site Documentation module.

catch’s picture

/wonders if drupal.org slowness has anything to do with update status. Haven't checked out your site documentation module yet, but it looks interesting.

nancydru’s picture

This feature has not been committed yet.

nancydru’s picture

Status: Active » Fixed

The problem has not recurred since the big update and table moves. And I know how to fix it easily, so I'm closing this.

Anonymous’s picture

Status: Fixed » Closed (fixed)
gdtechindia’s picture

even after deleting the values from variables table, Cron failed to run for my site.

DELETE FROM `variable` WHERE name = 'cron_semaphore';
DELETE FROM `variable` WHERE name = 'cron_last';

nancydru’s picture

try clearing the cache

gdtechindia’s picture

it worked for me after trying for many times.

running the command.

restarting apache

and clearing cache

OpenChimp’s picture

I was getting this error for over an hour (thought waiting for the process to finish might help, but didn't seem to). So, I searched and found this issue and this article: http://www.mojahmedia.net/drupal-cron-run-failed-cron-busy-probably-stuc..., which has more details, that I haven't tried yet.

The first thing that is mentioned here and there:

DELETE FROM `variable` WHERE name = 'cron_semaphore';
DELETE FROM `variable` WHERE name = 'cron_last';

seemed to fix everything.

Thanks

filiptc’s picture

Version: 5.1 » 7.x-dev
Status: Closed (fixed) » Active

Bumping this thread because I ran over the same problem with D6. This needs something like in #1 (or a *real* fix). This is especially confusing for novice users.

Setting version status to 7.x-dev for discussing about this functionality to getting included, and for drawing attention to the subject which is not so trivial as one may think (broken cron is serious). Cheers.

nmashruwala’s picture

Same symptoms as everyone else.

Fixed by disabling the Search module.

As a suggestion, instead of printing only "Cron run failed", some sort of error information would be helpful as it seems cron can break in a few different ways.

lias’s picture

Disabling search also worked for me on drupal 6.5

jaxond’s picture

Version: 7.x-dev » 6.8

Just upgraded from 6.6 to 6.8 and updated all modules. Tried the http_host patch, deleting the 2 variable records and disabling search module. Still get "Cron run failed" and "Attempting to re-run cron while it is already running" in the log. Found nothing more helpful in the server logs. (FYI, cron was failing prior to the 6.6 to 6.8 upgrade.) Is there any way to expose cron's progress, or lack thereof, while it is running as a way of troubleshooting? Maybe an optional verbose runtime mode would be useful.

nancydru’s picture

Poormanscron can log progress. It might also be a better debugging tool for this.

jaxond’s picture

Thanks for the tip. I turned on Poormanscron progress logging. It logs that node_cron, dblog_cron and filter_cron have executed, but then reports no problems or errors. The site still says cron has never run, so it must be running partially.

ClearXS’s picture

Version: 6.8 » 6.9

Cron has been running for more than an hour and is most likely stuck.

Engine failure, or something..?

Was on upgrade 6.8 to 6.9 (coinciding or due to other cause).

Well; let's just download the drupal6.9 another time, upload/extract (or extract/upload) and only replace cron.php..? But the cron.php almost contains nothing & seems intact:

include_once './includes/bootstrap.inc';
drupal_bootstrap(DRUPAL_BOOTSTRAP_FULL);
drupal_cron_run();

So what core-file should I replace? Apparently "bootstrap.inc". So renamed and copied another time and newly down&uploaded bootstrap.inc to that place.

Status report
Cron ran successfully.

Just accidental, or this was it..?!

Then I see I have to clear all caches on update (sometimes even related to cron), but can't find how I should do that..? Well, luckily this question is not that important anymore! But?

nancydru’s picture

Clear cache button at Administer>>site configuration>>performance or you can go to the modules page and ask update.php to run.

pxlar8’s picture

Tip from above to disable Search module worked for me on 6.9.

1. Disable Search in modules admin
2. Run cron
3. Re-enable Search in modules admin

marqpdx’s picture

Tried all these good steps.
Cron finally worked again after search module was disabled.
Re-enabling the search module, however, made cron inoperable again.
Something not right in the search module, it seems.
thanks!
m

Anonymous’s picture

After trying all the steps mentioned, I had the same experience as described in #20. Cron functioned with search disabled (not uninstalled) but my experience was different after reenabling search. Both sites: D6.8

For one site - 9.000 pages cron functioned well after reenabling search with max index per cron run set to 100 because a higher number caused a time out for cron.

In another site of 50 pages, cron stopped again after a few seconds. The messagelog mentioned a timed out for cron.
These two sites use different modules, so maybe there is a conflict between modules in the small site or an useful index missing in the database.
I'll try investigate this further but my knowledge and experience with these kind of things is small.

marqpdx’s picture

Turns out there was bad content which was killing the cron when it got to the search module.

After all the above steps, i'm now good to go again!

thx all,
m

mainebob’s picture

Title: "Cron run failed" » "Cron run failed" due to core Search

Wow, This cron fail has been bugging me for a month....
After upgrading to D6.9 I did the following:

- turned off core search /admin/build/modules
- ran /update.php (nothing needed updating)
- cleared cache with button on /admin/settings/performance page
- run cron manually /admin/reports/status/run-cron

"Cron ran successfully." Woo Who! YES!

Glad to find this thread... I had looked at many other cron fail threads back when this was
a problem with D5....

and then, turn on core search
"Cron run failed." Booo Hisssss! Seems like this is a bug in search.

So, Is this a bug to be fixed in 6.x ? AND I agree there needs to be better
error messages when cron fails... I was misdirected alot... there were tons of error messages in the log
similar to those that #17 reported.

I updated the subject line...

-Bob O

baim78’s picture

Thanks all,

to fix cron failed, I do the steps:

1. do sql query command

DELETE FROM `variable` WHERE name = 'cron_semaphore';
DELETE FROM `variable` WHERE name = 'cron_last';

2. disable search module
3. restart apache
4. clear cache in administer > site configuration > performance
5. do cron.php or from administer > report > status, wait until successfully
6. re-enable search module
7. do cron.php again, wait until successfully
8. remember dont do cron until your last cron ran successfully

Thanks all

catch’s picture

Version: 6.9 » 7.x-dev

Bumping this to Drupal 7 - we could at least use a 'reset cron when it's broken' button to update the cron semaphore, but better feedback would be ideal.

vacilando’s picture

In D6.9 (after upgrade from D5.15) - same situation as #20 - things worked after deleting the two variable records from the database and disabling search, but after enabling search the problem returned :-(

I appreciate marqpdx talking of "bad content" in #22 but I have 171k nodes... where to look and for what problems?

The problem is I don't see any useful error message - cron.php finishes silently and /admin/reports/status/run-cron ends on an empty page with "PHP Fatal error: require_once() [function.require]: Failed opening required './sites/all/modules/cck/theme/theme.inc' (include_path='.:.:') in (SNIP)/httpdocs/sites/all/modules/cck/content.module on line 180" ... any idea can that mean?

I'll appreciate any suggestions as I've run out of ideas after many days of research and trial&error!

nancydru’s picture

Reduce your search depth and re-delete those variables.

alexanderpas’s picture

we can't simply reset apache on shared hosting...

justin.hopkins’s picture

I'm having the same problem, and suspect it's due to "bad content" as marqpdx found. I'm not sure how to find this content though.

coupet’s picture

this might help in identifying bad content.

search indexing gets stuck: node_index()
http://drupal.org/node/139537#comment-993610

dave reid’s picture

Status: Active » Needs review
StatusFileSize
new698 bytes

Patch to reset the semaphore variable when running manual cron from admin/reports/status

dave reid’s picture

Title: "Cron run failed" due to core Search » Reset cron_semaphore when cron is run manually in admin/reports/status

Changing title to fit the current direction.

j.somers’s picture

Status: Needs review » Reviewed & tested by the community

Small fix, applied cleanly and works.

boombatower’s picture

+1 !

dries’s picture

I'm OK with this code, but it does introduce a race condition. A manual cron run can now interfer with an automatic cron run. Resetting the cron semaphore might not always be a smart thing to do.

I wonder if this problem will automatically go away once cron is using the task/job queue ...

Status: Reviewed & tested by the community » Needs work

The last submitted patch failed testing.

dave reid’s picture

Status: Needs work » Needs review

Hah! I don't believe you testing bot. 23 fails, 62 exceptions from one little variable_del()?

EDIT: Yep, it was a testbot failure.

boombatower’s picture

Actually this appears to be related to the partial commit made the other day that hosed the bot for a short period of time. Thus bot reported what it saw. :)

@see http://testing.drupal.org/node/25

mikejoconnor’s picture

Status: Needs review » Reviewed & tested by the community

Since this was already marked RTBC, and kicked back due to a testbot issue, I'm marking it back to RTBC.

Aside from that I do think this would be a good usability improvement. I agree with Dries that this could introduce some issues if an automatic cron run is in process, however the likelihood of that is small, and the usability benefit of this patch is well worth the risk

dries’s picture

Status: Reviewed & tested by the community » Needs work

I'm OK to proceed with this patch, but let's update the code comment so (1) it describes _why_ we do this and (2) to acknowledge that this might introduce a race condition. It's good to document oddities so we don't forget about them two years down the road. You can set this back to RTBC once the documentation was extended. Thanks!

alexanderpas’s picture

how about preventing the most obvious race conditions by disallowing another cron run within the php max_execution_time from the start of the cron run.

stephencarr’s picture

Bumping this because it plagues my content heavy aggregation sites and I am tired of manually re-setting cron.

damien tournoud’s picture

Erm. No! The cron could legitimately be running when you click on the "run cron manually" link. There is really no need to introduce yet another race condition here. I suggest adding a "reset cron semaphore" link to the "Fail to run cron" error message.

boombatower’s picture

#43: seems like the best option.

wmfinck’s picture

Thank you! Clearing the cache solved this same problem for me too!

dave reid’s picture

Component: base system » cron system

Moving to new cron system component.

brionace’s picture

phew!!
i too was having this problem and i followed the steps suggested by mainebob, and it worked.
thanks

Junro’s picture

Hello, I can't run cron manually when search core module is enable.

I done this:

1. do sql query command
DELETE FROM `variable` WHERE name = 'cron_semaphore';
DELETE FROM `variable` WHERE name = 'cron_last';
2. disable search module
3. restart apache
4. clear cache in administer > site configuration > performance
5. do cron.php or from administer > report > status, wait until successfully
6. re-enable search module
7. do cron.php again

But it doesn't work for me.

Any ideas?

Thanks

sebasje83’s picture

I followed the same 7 step procedure several times, but still getting the http 500 internal error and in the log it says: Cron run exceeded the time limit and was aborted.
Can someone help me locate the problem, it seems that some content, module or block is killing the run cron I guess.. Hopefully you can help me out.

Thanks in advance

sebasje83’s picture

I just solved the problem for my database. It seems that the Twitter module I had installed was causing the run cron to fail.. I tried all version for Drupal 6 but all of them are making the run cron fail.. So for me unfortunately no Twitter module.....

catch’s picture

Status: Needs work » Closed (duplicate)

#805702: Cron should use locking framework solves the same problem, but by removing the annoying cron semaphore altogether, marking this as duplicate since that issue has a more up-to-date patch.

brightbold’s picture

The Backup & Migrate module also seems to cause this under some circumstances. So if the steps in #24 are not working for you, try disabling Backup & Migrate. That worked in my case, although obviously not a good long-term solution since I do need to back up!

Dret’s picture

Version: 7.x-dev » 6.19
Issue tags: +stop indexing particular node, +cron blocked

My problem is still on 6.19.

My cron is blocked, I followed the instruction on #49 and this my results:

1) If Search modules is disabled cron works fine.
2) Search index is blocked on a particular node (2194, in my case)

So I made this operations:

1) I followed all procedure + Uninstalling Search module + rebuilding the Search index

Results:

All worked good until the new index procedure arrived to node 2194: here the Cron get stopped again and Search index too.

So I DELETED the node 2194... and made all once again!

But now the index stops at node 2193...!! :O

Some ideas about?
Thanks!
Bye.

jonathan1055’s picture

@Dret in #54
Did you see #361171: How to debug cron stoppage problems with bad PHP content comment #11 and #33 for hints on how to identify the problem pages?

Jonathan

ikeigenwijs’s picture

also having cron problems on 6.20 started 1 week ago , out of the blue.

tedstein’s picture

I just made a small module to help get cron unstuck.

http://drupal.org/project/unstick

Hopefully this will help.

el_reverend’s picture

I am having the same problem, with different modules. By turning off all modules and turning them on one by one and running cron after each one it did result in a successful cron completion. I have stumbled across a module, path_auto. It is there that the variable cron_semaphore is called again (or only then does it appear in my db again) and entered into the variable table. When cron is then called and the scripts are being executed it seems to execute, but I have the suspicion that the cron tasks executed are taking longer than the module will wait for an answer. I do not see a message when loading is completed. When it is re-run manually again it will result in failure since cron is already running but has not yet terminated?

How can I trace the failure to an exact process or file failure? I cannot figure out why this would run on not only a local environment but two remote environments, but not the newest one! I believe that there may be a problem with the transfer process of the site. I have tried the cron_semaphore workaround, but unless there is a SERVER specific setting that the scripts in cron don't like, it should work.

I feel that the problem of unsticking a process could be too taxing on the server, if I have to do it too often.

kenorb’s picture

+1
unstick looks ok