Different cron processes randomly lock up and require a manual unlock. Before Monday it had not happened for a month or longer. It happened twice Monday, once Tuesday, twice yesterday, and several times today. I need help figuring out where to look for some information that can tell me why this is happening.

Drupal 6.22 on Ubuntu 11.10
Production site, but relatively low traffic.
Running under SSL.
We have about 40 modules installed, several of which have cron hooks:
Captcha
Chaos Tools
Date Timezone
Progress
Ubercart (Credit Card, Cart, Order, Recurring Payments API)
plus 1 custom fax module (receives faxes and sends them as emails)

The custom module is basically the point of the site, and all heck breaks loose when that stops delivering. The custom module is set to run every 3 minutes, and usually runs for 5-15 seconds if there is a fax to process (or less than 1 second if there is none.)

When I say "locks up", all I can really tell you is that the Start Time is 10 minutes to several hours ago, where the locked process normally completes in a few seconds.

I think I've seen every process lock up. As I am typing this, "ultimate_cron:background_process_cron" has been running for over 20 minutes, and "ultimate_cron:search_cron" has been running for 21 minutes. In the log file for each, their normal duration is 00:00:00. No other processes are locked up. The fax process has run several times while these two were locked up.

I've unlocked 3 dozen processes today. I can't find any pattern to how often or when any given process will lock up. I'm now just refreshing the /admin/settings/background-process/overview page every few minutes and unlocking when I see a process get to 25 minutes (except for the fax process, then I unlock it at about 10 minutes.)

I did notice something odd with unlocking the fax process. Sometimes it immediately sends emails--like it was queued up to send. At first I thought it was sending on the next execution, which makes more sense. But it is definitely sometimes sending immediately after I hit 'unlock'.

Ubuntu's syslog is silent on anything related to cron.
Apache's error log has no mention of any of these pages.
There are no errors in Mysql's error.log.
There are no errors in the watchdog table.

When I unlock a process then it gets written to the ultimate_cron_log table, the status is '0', and the msg is "Manually Unlocked".

Where else can I look?
What else can I do?

Thanks in advance.
-Randy

CommentFileSizeAuthor
#39 server_status9.txt65 bytesdhcadmin
#39 bg_process_and_watchdog9.txt738.38 KBdhcadmin
#38 bg_process.debug_.access.patch485 bytesgielfeldt
#37 server-status_8.txt65 bytesdhcadmin
#37 bg_process_and_watchdog.txt394.38 KBdhcadmin
#33 cron.txt1.47 KBdhcadmin
#29 background_process.module.rej_.txt1.23 KBdhcadmin
#29 BackgroundProcess.class_.php_.rej_.txt1.32 KBdhcadmin
#28 bg_process.debug_.patch2.94 KBgielfeldt
#25 server-status7.txt65 bytesdhcadmin
#22 server_status6.txt65 bytesdhcadmin
#16 server_status5.txt65 bytesdhcadmin
#10 server_status_4.txt65 bytesdhcadmin
#9 Apachess.zip98.02 KBdhcadmin
#6 server_status_3.txt65 bytesdhcadmin
#4 Server-Status2.txt65 bytesdhcadmin
#2 server-status1.txt65 bytesdhcadmin

Comments

gielfeldt’s picture

gielfeldt commented

Hi

Determining why processes die can be a pain. Ultimate Cron should detect and capture fatal errors, timeouts, etc., and most of the time memory exhaust. What it can't detect, is if the apache process running the job dies, is killed or restarted in such a way that php's shutdown handlers won't execute.

Do you have server-status enabled on apache? If so, also enable ExtendedStatus and go to the server-status page, to see if the URL for the stalled job is there, and if it is, then if it's running.

URL example: POST /bgp%3Astart/ultimate_cron%253Abackground_process_cron

If, according to server-status, a job is still running after 20 minutes even though it should, it is most likely caught in some sort of infinite loop. If it happens to all jobs, it's most likely a module that hooks into something like hook_init() or hook_exit() or another hook that's very general.

If the job is not running according to server-status, then the job has somehow died without Ultimate Cron's knowledge. This will happen if e.g. apache is restarted while jobs are running.

A theory to some of these unexplained deaths is memory exhaust. A memory exhaust so hard, that the shutdown handlers won't even run. To this there may be a remedy. You can try this in your settings.php:

// OUT OF MEMORY PROTECTION
// To reduce the risk of shutdown handlers failing, we reserve 4mb of memory
$GLOBALS['__RESERVED_MEMORY'] = str_repeat('0', 1024 * 1024 * 4);
register_shutdown_function('_out_of_memory_protection');
function _out_of_memory_protection() {
  unset($GLOBALS['__RESERVED_MEMORY']);
}

For production-grade operations, I've created the module Background Process Apache Server Status (bundled with Background Process) which uses Apache's server status to determine if a job is actually running. If it's not running it unlocks it, so it can be run again. You can try this module out, but you may very well need to upgrade Ultimate Cron to the dev-version, due to a limitation of 64 chars in URLs in Apache's extended server status.

If you're not familiar with the output of Apache Server Status, just paste/attach the output along with a list of the jobs that has stalled, then I'll help you out deciphering it.

dhcadmin’s picture

dhcadmin commented
StatusFileSize
new server-status1.txt65 bytes

Hi gielfeldt,

Thanks for the help.

I did not have server status enabled. I had to restart Apache when I enabled it. Nothing has locked up since I restarted about 5-6 hours ago.

I've been watching closely for a lock-up. In fact, when I sit and hit refresh on the /admin/settings/cron/overview page none of the cron processes ever show that they are running. I sat there and hit refresh every second for several minutes, and I never got the running icon. I'm afraid that enabling the server-status and extended status has changed something.

I've attached a text file with the output from the /server-status page. It sure looks like several cron processes are hung, but you can see that some of them have duplicates running. Does this make any sense to you?

I've added some notes at the bottom of the file to explain some of the IP addresses and hostnames.

Thank you again! Have a great day!
-Randy

gielfeldt’s picture

gielfeldt commented

There doesn't seem to be any jobs "currently" running. When a job is running it usually has a "W" in the CPU column. When a job has ended, it is still present in the server-status, but the CPU changes to ".", so that slot is ready for a new connection.

If/when a job hangs again, take a new dump from the server-status page.

dhcadmin’s picture

dhcadmin commented
StatusFileSize
new Server-Status2.txt65 bytes

gielfeldt,

I had a handful of locks over the weekend, but I couldn't get to the Apache server status from home. So I just unlocked and went on. Now I have two processes locked this morning. The Apache server status is attached.

There are two processes locked up, according to the
/admin/settings/background-process/overview page:
* ultimate_cron:background_process_ass_cron 2012-02-20 07:39:07
* ultimate_cron:background_process_cron 2012-02-20 07:36:02

I see 3 background_process_cron listed and only 1 background_process_ass_cron. I don't see a discernible pattern as to why these two processes are still "running", but rules_cron and progress_cron are not also still "running".

Thank you again for your help with this.

-Randy

gielfeldt’s picture

gielfeldt commented

The processes don't seem to be "really" running according to the server status. One of two things could have happened:

1.) The process never started properly, but Background Process thinks it did.
2.) The process died in a way that the shutdown handlers did not execute.

Like discussed before, #2 is particularly painful to debug. However we might be able to confirm/reject #1. Would it be possible for you to install 6.x-1.x-dev? If so, be aware that no cron jobs must be running during the update.

Another thing just struck me. The problem might be very simple .... How many servers are you running on, and what is you MaxClient setting in apache? There's a setting in Ultimate Cron called "Simultaneous connections" which defaults to 40. This setting should be equal to or less than Apache's MaxClient setting, otherwise connections might not make it through, and Ultimate Cron won't know about this.

I'm puzzled that especially the two Background Process cron jobs were stalled, as these should be self-healing. Which version of Background Process are you running?

dhcadmin’s picture

dhcadmin commented
StatusFileSize
new server_status_3.txt65 bytes

gielfeldt,

I had left these two processes locked, and waited for others to lock up as well. Once my fax process locked I had to unlock everything. Attached are the apache server status just before and just after I unlocked everything. These are the 6 processes that showed to be locked:
ultimate_cron_cleanup_log 2012-02-20 08:38:02
background_process_ass_cron 2012-02-20 07:39:07
background_process_cron 2012-02-20 07:36:02
fax_cron 2012-02-20 10:36:07
date_timezone_cron 2012-02-20 10:40:07
rules_cron 2012-02-20 10:40:13

To your questions:
1. I will disable cron, update ultimate_cron to that dev release, and then restart cron. I cannot do this until my next maintenance window late one evening, because I cannot take the production site offline to run update.php during the day. I have a maintenance window tentatively scheduled for Friday night, and I might not be able to get one before then.

2. I am running on a single server. I have MaxClients set to 150 (I don't remember changing that.) I have 325MB of physical RAM available and 4GB of Swap available.

The only server-level limitation I remember changing from its default is for PHP. I set the memory_limit up to 128MB.

3. My Simultaneous Connections is set to 40.

4. Background Process is version 6.x-1.10. Progress is version 6.x-1.4.

Again, thank you for your help.

dhcadmin’s picture

dhcadmin commented

gielfeldt,

I just got an error out of my custom fax_cron process. It looked like it was hung up. I let it sit for a few minutes, to see if it would unlock, and it did. Then the cron log for that process showed that it ran for 3 minutes and 5 seconds, and gave this error:

"
htmlspecialchars() expects parameter 1 to be string, array given in /home/port/www/includes/bootstrap.inc on line 856. Never started (auto unlock due to timeout)
"

That's the first I've seen of this error. The fax cron kept on churning and generated the same error on it's next run (2 minutes and 51 seconds). But the run after that took 1 second and gave no error.

Not sure if this is cause, effect, or coincidental to the general locking problem; but I'm going to dig in to it.

A quick search indicates that PHP 5.2.5+ (I'm on 5.3.6) has a history of this error in several modules. Yuck. See http://drupal.org/node/829250#comment-3444480

Thanks,
Randy

gielfeldt’s picture

gielfeldt commented

Hmmm. It seems check_plain() is only used in translations containing arguments with the '@' prefix, so I cannot understand why "Never started (auto unlock due to timeout)" would fail?

However, if the socket connection fails from background process it uses the '@' prefix for the error message. I will dig into this to see if this is an issue.

For trial'n'error purpopses, you can try commenting this line in background_process.module:

$result->error = trim($errstr) ? trim($errstr) : t('Error opening socket @socket', array('@socket' => $socket))

You might also want to update background process to 6.x-1.x-dev, as there's a fix there that solves some strange issues with cwd and apache in shutdown handlers (which could be very relevant to your problem). The upgrade should be non-invasive, you don't have to stop all cron jobs or anything.

dhcadmin’s picture

dhcadmin commented
StatusFileSize
new Apachess.zip98.02 KB

gielfeldt,

I've got your change to the background_process.module loaded in my test environment. I'm going to move it to production shortly.

I'm also using devel in my test environment and getting some interesting backtraces on the various ultimate_cron related pages. I'm not sure that I'm reading this right, so I'm attaching it for you. This is the backtrace on the Apache Server Status Settings page (/admin/settings/background-process/ass). Two things I'm seeing here:

1. Array 0's args are in the form of an array. The function is check_plain, which doesn't accept arrays.

Maybe this is a rabbit trail, but check_plain being called with an array is exactly the same problem that my hung cron job reported. So maybe the rabbit trail will tell us something useful.

2. Array 5: args: element 0 is 'background_process_ass_max_age', and it is an array of 24 elements. When you expand to see the 24 elements, it says 'recursion'. Ditto for Array 6: args: element 1, which is an array with 32 elements. Maybe this is just a limit in devel's backtrace. Those are the largest arrays on the page. Or maybe there is some recursion happening, and check_plain getting called with an array is a downstream failure.

What do you think?

Thanks,
Randy

dhcadmin’s picture

dhcadmin commented
StatusFileSize
new server_status_4.txt65 bytes

gielfeldt,

No joy on the change to background_process.module. Modified version of background_process_module with that line commented out was put into production about an hour ago. Everything worked fine for a while.
Now three processes are locked up:

fax_cron 2012-02-21 11:33:02
progress_cron 2012-02-21 11:30:03
background_process_cron 2012-02-21 11:25:02

Apache server-status is attached.

Thanks again,
Randy

gielfeldt’s picture

gielfeldt commented

In your dump, I think 0's args as an array is correct, as this is backtrace's way to represent multiple parameters in a uniform way.
I cannot see why the Background Process Apache Server Status would cause this "recursion" inside the form object though. I must add, that I don't get this backtrace on my system.

The Background Process Apache Server Status settings form is as simple as it gets, I'm almost 100% certain that it's not doing anything wrong. Perhaps another module with a form_alter messes something up?

Nevertheless, none this "should" cause locks not to be released. Can you dump the relevant rows from the background_process table, next time you have a stalled job?

dhcadmin’s picture

dhcadmin commented

gielfeldt,

I do not know how to identify the relevant rows from the background_process table.

That makes sense that backtrace could be putting the values in an array. I'll try to dig into that.

Could a form_alter problem be located in a theme, or only in a module?

Also, we had an unrelated problem yesterday afternoon. I had to restart the server. There has not been a single locked process since the restart. Similarly, when I restarted Apache to enable server-status the other day, there were no lock-ups for a day or so, then very few for a day or so. It was only after a few days that the lock ups really began in earnest. If that pattern holds then it makes me think I am looking for a memory leak, or something else that has to build up over time.

If that pattern holds then I won't have another locked process until this time tomorrow.

It's making me consider setting up a cron job to restart Apache every night. Or many to just reload Apache, and see if that is enough. It's just a band-aid, but maybe that is the right answer.

Thanks again for all of your help.
Randy

gielfeldt’s picture

gielfeldt commented

If you have database access, you can run the query:
SELECT * FROM background_process;

If not, it's still possible to find what I'm looking for, by installing the dev version of Ultimate Cron. What I'm looking for, is whether the jobs die or never start at all.

I'm not sure, by I believe it's possible to have a hook_form_alter() in a theme.

gielfeldt’s picture

gielfeldt commented

BTW: The dev version of Ultimate Cron has better drush support, so if Apache is the problem, this might solve a great deal. Unfortunately the automatic unlocking feature from Background Process Apache Server Status, doesn't work with this (for obvious reasons), but if that feature didn't seem to work anyway, Drush may be a good option.

Hopefully I will make a new release of Ultimate Cron soon containing this feature, and try to document it properly on the project page.

gielfeldt’s picture

gielfeldt commented

Also, fyi, I just double checked the @socket line you tried removing. It cannot be this causing problems either, as $socket is always a string. You can safely re-add the line.

dhcadmin’s picture

dhcadmin commented
StatusFileSize
new server_status5.txt65 bytes

gielfeldt,

46 hours after the server restart I finally had my first lock-up. I think this has to be connected to some accumulating process, like a memory leak.

I had put the @socket line back into production before the restart.

I dumped the ultimate_cron, ultimate_cron_log, and watchdog tables before I unlocked the processes this time. The uc_log table had almost 44,000 records. So I just included the tail in the attached file.

I also noticed that there were more of the "htmlspecialchars() expects parameter 1 to be string," errors than I had expected. Of the ~44,000 records, there were 77 instances of that error. I included all of those as well, just to see which services were generating the error. You will see that there were several different processes guilty of this, but the large majority of the instances are background_process_cron.

After looking at this I think that the htmlspecialchars() problem is not related to the lockups. The timing is all wrong. The processes don't lock up when those errors are presented; they lock up at other times. But I could be wrong, so I'm including the data for you.

Thank you again. I would be really lost in this without you.

-Randy

gielfeldt’s picture

gielfeldt commented

Ah, shame ... the table I needed dumped was "background_process".

BUT ... There's is a minor bug in background_process_cron that causes these warnings! Thanks, I'll fix that right away. But like you, I cannot see how this can cause lockups.

To explain the processes that "magically completes", it is probably when the background_process_cron runs and cleans up stale locks that have timed out.

To confirm/deny this, I really need to see a dump of the table "background_process" next time you get lockups.

What is the "Cleanup age" set to on /admin/settings/background-process ?

gielfeldt’s picture

gielfeldt commented

omg, the background_process_cron is faulty as hell .... thanks for bringing this to my attention.

gielfeldt’s picture

gielfeldt commented

Hi Randy

Background Process updated, committed to 6.x-1.x, and tarball for 6.x-1.x-dev is ready for download.

This fix should actually mitigate your locking problems. Though It doesn't solve the root cause of the jobs being locked, now at least the automatic unlock of stalled jobs should work for background processes that never started.

dhcadmin’s picture

dhcadmin commented

gielfeldt,

I'm sorry that I sent you the wrong table dump. I really need to know the root cause of the locking for my custom fax module, so I'm going to stay on the current version of background process through one more lock-up. And then I'll send you the right table. I would really appreciate your help getting to the bottom of that.

After that I'll update background process and move on.

I think all of my background process settings are still at default:
Service timeout: 0
Connection timeout: 2
Stream timeout: 2
Cleanup age: 120

Should I change any of those?

Thanks again!
-Randy

gielfeldt’s picture

gielfeldt commented

Hi Randy

Settings look good, no need to change them this time.

I'll standby for the table dump.

dhcadmin’s picture

dhcadmin commented
StatusFileSize
new server_status6.txt65 bytes

gielfeldt,

Yeah, it locked up again.

The list of what was locked up, dumps from background_process and ultimate_cron, andApache server status are attached. I also have the ultimate_cron_log and watchdog, if you need them.

Thanks,
Randy

dhcadmin’s picture

dhcadmin commented

gielfeldt,

Do I need to run update.php when I install the new dev version of background_process?

Thanks,
Randy

gielfeldt’s picture

gielfeldt commented

Hi Randy

Running update.php is not necessary for this update.

Having looked at the dump, it is as I suspected: Background Process seems to think the job is launched, but it just never starts. I've experienced this, when Apache is trying to handle too many connections. There might also be a bug in background_process_http_request(), but it will be very hard for me to debug, when I don't have a system where it fails :-(

The new version of Background Process should be able to clean these locks up after the bug-fix on background_process_cron. However the question is if it will matter much, if it's problem that accumulates over time?

BTW: Are running Apache in threaded or prefork mode? I believe of heard of the problem with Apache accepting requests, but not actually running those requests in threaded mode.

EDIT: insertED missing words in above sentence...

dhcadmin’s picture

dhcadmin commented
StatusFileSize
new server-status7.txt65 bytes

Hello gielfeldt,

I am running Apache in whatever the default mode is for Ubuntu 11.10. I think it is prefork, but I'm not sure.

I installed the background_process dev version into production about 3-4 hours ago. I was having many lock-ups this morning before installing this. The lock-ups have been less, but they usually are after an Apache restart.

By the way, I tested Apache restart (and reload and graceful), and that does not always cause the site to behave for hours afterwards. Usually the site behaves better afterwards, but not always and not always for very long. So I have not implemented a nightly restart of Apache.

Anyway, the attached file has the normal stuff: the list of process that show to be locked up, the Apache server-status, and the dump of the background_process table.

What can I do next to troubleshoot this?

Thank you again!
Randy

gielfeldt’s picture

gielfeldt commented

Hi Randy

Hmm, have to admit, I'm running out of ideas. Next step for me would be to through in some watchdog messages in the code to try to determine exactly where things go wrong. I can supply a patch if you're up for it?

dhcadmin’s picture

dhcadmin commented

Hi gielfeldt,

Yes, I'm up to it. There may be a lag before I can install it, though, especially if I have to run update.php.

Thanks,
Randy

gielfeldt’s picture

gielfeldt commented
StatusFileSize
new bg_process.debug_.patch2.94 KB

patch with watchdog stuff...

When it locks up next time, please send table dump of background_process, watchdog and a dump of the server status page.

dhcadmin’s picture

dhcadmin commented
StatusFileSize
new BackgroundProcess.class_.php_.rej_.txt1.32 KB
new background_process.module.rej_.txt1.23 KB

gielfeldt,

I've never installed a .patch file before. I thought I found good instructions, and it seems to have tried to work. But it did not fully work. This site has background_process 6.x-1.10+4-dev installed, just like my production site. What am I doing wrong here?

My session:
# cp background_process.module background_process.module.backup
# patch background_process.module < bg_process.debug_.patch
patching file background_process.module
Hunk #1 FAILED at 169.
1 out of 1 hunk FAILED -- saving rejects to file background_process.module.rej
patching file background_process.module
patch unexpectedly ends in middle of line
Hunk #2 succeeded at 429 with fuzz 1.
# cp BackgroundProcess.class.php BackgroundProcess.class.php.backup
# patch BackgroundProcess.class.php < bg_process.debug_.patch
patching file BackgroundProcess.class.php
patching file BackgroundProcess.class.php
Hunk #1 FAILED at 416.
patch unexpectedly ends in middle of line
Hunk #2 FAILED at 428.
2 out of 2 hunks FAILED -- saving rejects to file BackgroundProcess.class.php.rej

The two .rej files are attached, renamed to .txt so I could upload them.

Sorry for the trouble. Thank you again for your help.
Randy

gielfeldt’s picture

gielfeldt commented

Hi Randy

No worries, I should have posted how to apply:

While standing in the background_process directory, do:

patch -p1 < bg_process.debug.patch
vimaljoseph’s picture

vimaljoseph commented

#7 @dhcadmin , I also got the same error (htmlspecialchar). Please see the updates on http://drupal.org/node/1470002

gielfeldt’s picture

gielfeldt commented

Hi Randy

You mentioned that you were running the site on SSL. Is the website accessible through no-SSL from localhost? If so, we could try setting up Background Process not to use SSL to see if this is the culprit. Let me know, I'll post how to set it up.

dhcadmin’s picture

dhcadmin commented
StatusFileSize
new cron.txt1.47 KB

@vimaljoseph: Thank you for the pointer. I will update my background_process to pick up that fix.

gielfeldt,

My site is not available through non-ssl.

With those install instructions, the patch installed just fine on my test server. The output seems to be getting written to the watchdog log. I've got a really busy day today. I don't want to install this onto production unless I can sit and watch it for a few hours. So it will probably be tomorrow before I can install it there. I'll keep you posted.

I did something else yesterday that has reduced the number of lock-ups. I did a complete reschedule of all of my enabled cron processes within Ultimate Cron. I have my custom fax module (that is probably the primary cause of the lock-ups) running every 3rd, 6th, and 9th minute, and nothing else except background_process and background_process_apache_server_status (which both run every minute) running at those times. Everything else is scheduled to run on some iteration of the 0th, 1st, 2nd, 4th, 5th, 7th, or 8th minute of each 10-minute cycle.

It made no difference for the first hour or two, and then suddenly everything quieted down. I have not had a single locked process since about 2 hours after I made this change. I'm not sure this is a permanent fix, but it helps reduce my stress. When I have your logging patch installed I'll just edit a few cron schedules to get more things happening at the same time.

I've attached my new cron schedule, in case you are interested.

Thanks,
Randy

gielfeldt’s picture

gielfeldt commented

Hi Randy

Ok. From the sound of your "solution", I'm back to thinking of some sort of congestion, where the webserver drops connections. I've been planning to implement some sort of "skew" feature for the rules, especially upon install of Ultimate Cron. I've had reports of sites crashing when Ultimate Cron was installed, because too many cron jobs began running in parallel at the same time, as they all have the same default rule. But with only 16 cron jobs it must be when users are hitting the webserver that it begins to congest. How many webservers are you using?

Have you experienced lockups since you changed the rules? If yes, you could also try to reduce the "Simultaneous connections" from 40 to e.g. 10.

Also, you might still be interested in trying Drush out.

dhcadmin’s picture

dhcadmin commented

gielfeldt,

Sorry for the delay. Something else came up and I've been away from this project for several days.

The production server returned to locking up a day or so after I made the last post. It is locking up regularly again. I'm about to install your patch on production, because my test site where it is installed seems to be stable and behaving.

I'd rather not change the simultaneous connections and apply the patch at the same time. But I'll do that in a few days.

I have only one production server. It is a relatively robust linux box. I've been watching the memory and processor utilization, and I have plenty of available resources.

Is there a simple process for unintalling this patch, when the time comes? Or will I just reinstall the base version?

Thanks,
Randy

gielfeldt’s picture

gielfeldt commented

Hi Randy

Yes, just install the base version, it's safe.

dhcadmin’s picture

dhcadmin commented
StatusFileSize
new bg_process_and_watchdog.txt394.38 KB
new server-status_8.txt65 bytes

gielfeldt,

And here it is.... The two files have the requested information. Anything interesting in here?

Thanks,
Randy

gielfeldt’s picture

gielfeldt commented
StatusFileSize
new bg_process.debug_.access.patch485 bytes

Hi

Yes, It seems as that the webserver accepts the HTTP request, but it never actually runs it. Either that, or the http request part in background process, is broken in a way i cannot see.

I've attached one last patch, to see if the access handler is called at all. Could you try it on? Don't revert the other patch, just apply this one.

And send the background_process, watchdog and server status dump again.

dhcadmin’s picture

dhcadmin commented
StatusFileSize
new bg_process_and_watchdog9.txt738.38 KB
new server_status9.txt65 bytes

gielfeldt,

New files attached.

gielfeldt’s picture

gielfeldt commented

Hi again

From the looks of it Background Process issues an HTTP requests, the server accepts, but nothing happens. Either there's a bug in the way Background Process issues an HTTP request, or there's something wrong with your webserver somehow. I won't rule out the first part, but I'm not able to replicate it, so I think I might have to give up at this point. I could try to make a version that uses CURL instead to see if that works, but I don't think I have the time for that right now.

My only suggestion at this point to get Ultimate Cron working for you would be via Drush. If you are interested in this, let me know, I can explain here how it's done.

Sorry to leave you hanging like this :-(

dhcadmin’s picture

dhcadmin commented

gielfeldt,

I understand. Thank you for your help. I would like to try the Drush route. I was planning on installing it on my test server today for another task, anyway.

Thanks again,
Randy

dhcadmin’s picture

dhcadmin commented

gielfeldt,

I have drush running on my test server. I see the ultimate_cron drush commands. I don't understand how enabling a cron job via drush will make any difference. Is it more complicated than just drush cron-enable'ing each job?

Thanks,
Randy

gielfeldt’s picture

gielfeldt commented

Using drush can make a difference because it doesn't use the webserver to run the jobs.

To use drush.

1.) Disable "Poormans cron" (if enabled) at /admin/config/system/cron/settings
2.) Wait until all cronjobs have finished (or restart webserver and unlock all jobs manually afterwards).
3.) In your servers crontab instead of the usual "wget", add "* * * * * drush cron-run all --cli --check-rule". Your syntax might vary. Here is my snippet from /etc/cron.d/drupal (change paths appropriately)

* * * * * root  (cd /home/thomas/develop/drupal7 ; /home/thomas/drush/drush cron-run all --cli --check-rule --logfile=/var/log/drupal-cron >> /var/log/drupal-cron 2>&1)

Please remember that for this to work you must be using the dev version of Ultimate Cron.

Rules and other cronjob specific settings are still controlled through the Drupal interface (/admin/config/system/cron)

Advantages of using Drush for cron is:
1.) Doesn't use the webserver (less memory/network/cpu footprint), so there's more room for your clients.
2.) Possible to use a different php.ini for e.g. more memory consumption for cron.

Disadvantages:
1.) It's not possible to use the Background Process pool management and load balancing features
2.) There's no cleanup module for dead jobs like "Background Process Apache Server Status" (to my knowledge).

dhcadmin’s picture

dhcadmin commented

gielfeldt,

Thank you for those instructions. I think I can set that up and test it out.

Two minor clarifications:
Will I still be able to see and unlock locked processes from the web interface, like I do now?
I could disable the background process apache server status, right?

gielfeldt’s picture

gielfeldt commented

Yes, you will still be able to unlock, and you can disable the background process apache server status module.

dhcadmin’s picture

dhcadmin commented

gielfeldt,

Thanks for that. I've got it drush ultimate cron running on my test server. I'm planning to put it into production on Friday night.

For anyone who is following along, I had trouble getting the cron drush command to work. I had to chmod the /user/share/php/drush/drush.php file and add execute permission for the owner. Maybe there is some other way to get that working, but cron gave me permission errors running the command until I did that.

Thanks,
Randy

gielfeldt’s picture

gielfeldt commented

Did things work out?

dhcadmin’s picture

dhcadmin commented

gielfeldt,

Drush cron went into production on Friday night. So it has been almost a week. There have been no lock-ups since then. Maybe it is premature, but I think I'll declare victory.

Thank you for all of your help!

Randy

gielfeldt’s picture

gielfeldt commented

You're welcome. If you don't mind, I would like to keep this ticket open if I should stumble upon a reason for the original problem.

Btw, lock ups can of course still occur in event of server reboots/crashes etc.

gielfeldt’s picture

gielfeldt commented

Just out of curiosity: Is it still working?

gielfeldt’s picture

gielfeldt commented
Status: Active » Fixed

I'll revisit this after releasing Background Process 2.x I think

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

b_man’s picture

b_man
he/him
Location California, USA
commented