Multi-Threaded Cron Jobs
| Project: | Drupal |
| Version: | 7.x-dev |
| Component: | base system |
| Category: | feature request |
| Priority: | normal |
| Assigned: | Souvent22 |
| Status: | duplicate |
| Issue tags: | cron |
So, I have started an ambitous patch. For those who do not want to read the explination, and just gain knowledge by reading the patch:
This patch allows a user to run each modules cron job as it's own thread.
Right now, there is not a lot of information about cron runs other than "they run". I, as an admin, do lik elike this "black box" feature of something running every so often, but not telling me much about what it's doing, errors etc. What this patch does is allow a user to give drupal the command it uses to run cron.php, and it then fires of each modules cron hook in it's own thread. This has big benefits:
- Modules that my be "broken" do now interfear with other module runs
- Modules that take up a lot of time (e.g. search.module indexing a large site), will not "hog" all the run time. Each cron job has it's own "sandbox" to run in.
- Indivual tracking of modules
- This patch adds the ability to see the time that each module took to run.
- This reporting of each modules run can help a site admin better turn their site.
Note: This also has an option to "truncate" and keep only so many crons in history (simliar to watchdog
Note:It still allows for a "basic" cron.
I think this patch could lead to some interesting things, such as:
- Setting individual time outs for each module
- Capturing error messages during a cron run on a per module basis
- Emailing an admin upon a "cron error" escalation, and information about which module it is.
- Among other things.....
The basic program flow is as follows:
Here are some screen shots:
Cron Confiuration
Main Run Screen with 2 runs (1 basic, 1 advanced)
Advanced Run
Basic Run
| Attachment | Size | Status | Test result | Operations |
|---|---|---|---|---|
| threaded_cron.patch | 14.57 KB | Ignored | None | None |
#1
Up for review now.
#2
This is a very important patch....we need this to do more complicated and larger data sets.
#3
Considering how ambitious the change is, I am surpirsed and pleased thhat the patch is not too large. Will review soon.
#4
This won't make it for 5.0 but might be useful for 6.0. My initial reaction is that we should avoid adding configuration settings. The UI needs some work too. That said, more advanced cron functionality is a useful feature.
#5
Moving out of the "x.y.z" queue to a real queue.
An important patch like this needs to be in the right place.
#6
By the time someone gets to this, it will need to be ported.
#7
The "to be ported" state is for patches that have been contributed to HEAD and need to be backported to 4.7 or 4.6 for example. So this still needs review before that would happen.
@coreb - Not sure if there's any other issues that were incorrectly marked "to be ported". Let me know if you find any and have questions on the correct status. Thanks!
#8
Subscribing...
#9
Subscribing... and at the same time, could we implement some sort of cron ordering? Let's say I want to make sure that aggregator cron (or some other module) runs before one of my custom module (I don't think it's possible right now).
#10
Subscribing and adding to my 6.x todo-list.
#11
HEAD is again open. lets get this one in front of the committers.
#12
I advise using proc_open not popen.
#13
The patch is somewhat dangerous, cron_run_command should be a bit sanitized and even then, I am not sure I like it.
#14
Yes. Agreed about the dangerousness of the current patch. I have cleaned it up a lot. I shall try and post the updated patch this week.
#15
This is just a "re-sync" patch to get back in line with 5.1. It has been tested though and works.
The way I currently tested it is I have a few custom modules for clients paired with large datasets that have very long run-times.
Another easier way to test this is to modify existing cron runs. E.g.:
Try putting a sleep(15); line in the watchdog cron function, and run your cron manually. You will see that the page returns saying cron run-secucessful. If you go over to admin/logs/cron and select the newest cron run, you'll see that watchdog will wtill still be running and the amount of time it has been running.
The basics are there, just needs some heavy tweaking, cleaning and polishing.
#16
Killes,
I'm trying to come up with a scenario where the cron_run_command could be compromised as a "primary" opening. Meaning, if someone gains access to your MySQL database, and changes the variable in the database...there are larger security issues at hand.
However, my current scenario that I can think of is someone gains a drupal installations administrator's username/password, and changes the executable to "rm -r " or something malicious like that.
However, if you're running an important drupal site a secure sever, then Drupal (PHP) should not be running as a privileged user anyways.
Thoughts?
#17
I'll update this module if/after SQL Server goes into head, thus I don't have to keep 2-3 sets of db creation patches up to date after each head change. blah. :)
#18
Reroll?
#19
Subscribing
#20
Subscribing.
#21
subscribing
#22
Maybe work on this for HEAD and backport. It's needed.
#23
Anyone up for reviving this. HEAD is open and accepting patches :)
#24
Mark #246380: Module wise cron scheduling interface. as duplicated.
Update patch #15 for D7, but fail on install :
Notice: Undefined index: rid in drupal_cron_run() (line 2800 of includes\common.inc).#25
subscribing
#26
subscribing
#27
bump
#28
Add tag.
#29
Ok. I have a need and some time :). I'm going to start to port this to a contrib-module instead of a patch against core.
After it gets vetted there, we can look at perhaps bringing into core.
I'll update as time permits. I'm going to work against D6 for the first implementation.
#30
@Souvent22 - did you see http://drupal.org/project/cron_mt? It uses pcntl for managing threads, and it looks like the basic framework is there, and a lot of the issues solved (or at least identified). It would be great to work on that module (and perhaps see if a pclose(popen()) type approach is indeed sufficient) rather than duplicating work here.
#31
I would be very willing to work with, or at least have a discussion with, anyone who wants to add support for non-unix-like systems to Cron MT. Right now it is a fairly simple solution that works very well on Linux, which is the platform on which I exclusively develop. To goal of the module it to be as light weight and transparent to the end user as possible, and unfortunately I have not been able to come up with a similar solution on Windows probably because of my lack of low-level knowledge of the OS.
#32
Does #578676: Use queue for cron render this patch obsolete?
#33
It does.