Multi-Threaded Cron Jobs [#87528]

So, I have started an ambitous patch. For those who do not want to read the explination, and just gain knowledge by reading the patch:

This patch allows a user to run each modules cron job as it's own thread.

Right now, there is not a lot of information about cron runs other than "they run". I, as an admin, do lik elike this "black box" feature of something running every so often, but not telling me much about what it's doing, errors etc. What this patch does is allow a user to give drupal the command it uses to run cron.php, and it then fires of each modules cron hook in it's own thread. This has big benefits:

Modules that my be "broken" do now interfear with other module runs
Modules that take up a lot of time (e.g. search.module indexing a large site), will not "hog" all the run time. Each cron job has it's own "sandbox" to run in.
Indivual tracking of modules

This patch adds the ability to see the time that each module took to run.
This reporting of each modules run can help a site admin better turn their site.

Note: This also has an option to "truncate" and keep only so many crons in history (simliar to watchdog

Note:It still allows for a "basic" cron.

I think this patch could lead to some interesting things, such as:

Setting individual time outs for each module
Capturing error messages during a cron run on a per module basis
Emailing an admin upon a "cron error" escalation, and information about which module it is.
Among other things.....

The basic program flow is as follows:
Cron Explincatoin

Here are some screen shots:

Cron Confiuration
Cron Configuration

Main Run Screen with 2 runs (1 basic, 1 advanced)
Cron Runs

Advanced Run

Thread Run

Basic Run

Comment	File	Size	Author
#24	issue-87528-24.patch	22.84 KB	lilou
#24	cron-admin.png	47.92 KB	lilou
#15	threaded_cron_51.patch	19.62 KB	Souvent22
	threaded_cron.patch	14.57 KB	Souvent22

Comments

Comment #1

Souvent22 commented 5 October 2006 at 03:08

Status:

Active

» Needs review

Up for review now.

Comment #2

boris mann commented 5 October 2006 at 07:09

This is a very important patch....we need this to do more complicated and larger data sets.

Comment #3

moshe weitzman commented 5 October 2006 at 21:45

Considering how ambitious the change is, I am surpirsed and pleased thhat the patch is not too large. Will review soon.

Comment #4

dries commented 9 October 2006 at 07:13

This won't make it for 5.0 but might be useful for 6.0. My initial reaction is that we should avoid adding configuration settings. The UI needs some work too. That said, more advanced cron functionality is a useful feature.

Comment #5

coreb commented 30 November 2006 at 02:18

Version:

x.y.z

» 6.x-dev

Moving out of the "x.y.z" queue to a real queue.

An important patch like this needs to be in the right place.

Comment #6

coreb commented 30 November 2006 at 02:19

Status:

Needs review

» Patch (to be ported)

By the time someone gets to this, it will need to be ported.

Comment #7

RobRoy commented 30 November 2006 at 06:25

Status:

Patch (to be ported)

» Needs review

The "to be ported" state is for patches that have been contributed to HEAD and need to be backported to 4.7 or 4.6 for example. So this still needs review before that would happen.

@coreb - Not sure if there's any other issues that were incorrectly marked "to be ported". Let me know if you find any and have questions on the correct status. Thanks!

Comment #8

m3avrck commented 17 December 2006 at 03:31

Subscribing...

Comment #9

FiReaNGeL commented 5 January 2007 at 18:04

Subscribing... and at the same time, could we implement some sort of cron ordering? Let's say I want to make sure that aggregator cron (or some other module) runs before one of my custom module (I don't think it's possible right now).

Comment #10

forngren commented 5 January 2007 at 18:30

Subscribing and adding to my 6.x todo-list.

Comment #11

moshe weitzman commented 21 January 2007 at 04:02

HEAD is again open. lets get this one in front of the committers.

Comment #12

chx commented 29 January 2007 at 12:50

Status:

Needs review

» Needs work

I advise using proc_open not popen.

Comment #13

killes@www.drop.org commented 29 January 2007 at 13:35

The patch is somewhat dangerous, cron_run_command should be a bit sanitized and even then, I am not sure I like it.

Comment #14

Souvent22 commented 29 January 2007 at 14:43

Yes. Agreed about the dangerousness of the current patch. I have cleaned it up a lot. I shall try and post the updated patch this week.

Comment #15

Souvent22 commented 12 February 2007 at 20:26

Status	File	Size
new	threaded_cron_51.patch	19.62 KB

This is just a "re-sync" patch to get back in line with 5.1. It has been tested though and works.

The way I currently tested it is I have a few custom modules for clients paired with large datasets that have very long run-times.
Another easier way to test this is to modify existing cron runs. E.g.:

Try putting a sleep(15); line in the watchdog cron function, and run your cron manually. You will see that the page returns saying cron run-secucessful. If you go over to admin/logs/cron and select the newest cron run, you'll see that watchdog will wtill still be running and the amount of time it has been running.

The basics are there, just needs some heavy tweaking, cleaning and polishing.

Comment #16

Souvent22 commented 12 February 2007 at 20:41

Killes,

I'm trying to come up with a scenario where the cron_run_command could be compromised as a "primary" opening. Meaning, if someone gains access to your MySQL database, and changes the variable in the database...there are larger security issues at hand.
However, my current scenario that I can think of is someone gains a drupal installations administrator's username/password, and changes the executable to "rm -r " or something malicious like that.

However, if you're running an important drupal site a secure sever, then Drupal (PHP) should not be running as a privileged user anyways.

Thoughts?

Comment #17

Souvent22 commented 7 March 2007 at 06:35

I'll update this module if/after SQL Server goes into head, thus I don't have to keep 2-3 sets of db creation patches up to date after each head change. blah. :)

Comment #18

m3avrck commented 16 May 2007 at 02:10

Reroll?

Comment #19

owen barton commented 18 May 2007 at 17:55

Subscribing

Comment #20

wim leers

Ghent 🇧🇪🇪🇺

commented 20 August 2007 at 22:52

Version:

6.x-dev

» 7.x-dev

Subscribing.

Comment #21

liquidcms commented 13 February 2008 at 11:20

subscribing

Comment #22

boris mann commented 19 February 2008 at 17:02

Maybe work on this for HEAD and backport. It's needed.

Comment #23

moshe weitzman commented 12 April 2008 at 02:16

Anyone up for reviving this. HEAD is open and accepting patches :)

Comment #24

lilou commented 8 November 2008 at 20:53

Status	File	Size
new	cron-admin.png	47.92 KB
new	issue-87528-24.patch	22.84 KB

Mark #246380: Module wise cron scheduling interface. as duplicated.

Update patch #15 for D7, but fail on install :

Notice: Undefined index: rid in drupal_cron_run() (line 2800 of includes\common.inc).

Comment #25

kehan commented 26 January 2009 at 13:43

subscribing

Comment #26

Antinoo commented 8 February 2009 at 04:50

subscribing

Comment #27

frankcarey commented 13 May 2009 at 20:22

bump

Comment #28

lilou commented 13 May 2009 at 23:35

Issue tags:

+cron

Add tag.

Comment #29

Souvent22 commented 19 May 2009 at 18:33

Ok. I have a need and some time :). I'm going to start to port this to a contrib-module instead of a patch against core.
After it gets vetted there, we can look at perhaps bringing into core.
I'll update as time permits. I'm going to work against D6 for the first implementation.

Comment #30

owen barton commented 19 May 2009 at 22:00

@Souvent22 - did you see http://drupal.org/project/cron_mt? It uses pcntl for managing threads, and it looks like the basic framework is there, and a lot of the issues solved (or at least identified). It would be great to work on that module (and perhaps see if a pclose(popen()) type approach is indeed sufficient) rather than duplicating work here.

Comment #31

cpliakas commented 22 May 2009 at 13:39

I would be very willing to work with, or at least have a discussion with, anyone who wants to add support for non-unix-like systems to Cron MT. Right now it is a fairly simple solution that works very well on Linux, which is the platform on which I exclusively develop. To goal of the module it to be as light weight and transparent to the end user as possible, and unfortunately I have not been able to come up with a similar solution on Windows probably because of my lack of low-level knowledge of the OS.