Multi-Threaded Cron Jobs
| Project: | Drupal |
| Version: | 7.x-dev |
| Component: | base system |
| Category: | feature request |
| Priority: | normal |
| Assigned: | Souvent22 |
| Status: | patch (code needs work) |
So, I have started an ambitous patch. For those who do not want to read the explination, and just gain knowledge by reading the patch:
This patch allows a user to run each modules cron job as it's own thread.
Right now, there is not a lot of information about cron runs other than "they run". I, as an admin, do lik elike this "black box" feature of something running every so often, but not telling me much about what it's doing, errors etc. What this patch does is allow a user to give drupal the command it uses to run cron.php, and it then fires of each modules cron hook in it's own thread. This has big benefits:
- Modules that my be "broken" do now interfear with other module runs
- Modules that take up a lot of time (e.g. search.module indexing a large site), will not "hog" all the run time. Each cron job has it's own "sandbox" to run in.
- Indivual tracking of modules
- This patch adds the ability to see the time that each module took to run.
- This reporting of each modules run can help a site admin better turn their site.
Note: This also has an option to "truncate" and keep only so many crons in history (simliar to watchdog
Note:It still allows for a "basic" cron.
I think this patch could lead to some interesting things, such as:
- Setting individual time outs for each module
- Capturing error messages during a cron run on a per module basis
- Emailing an admin upon a "cron error" escalation, and information about which module it is.
- Among other things.....
The basic program flow is as follows:
Here are some screen shots:
Cron Confiuration
Main Run Screen with 2 runs (1 basic, 1 advanced)
Advanced Run
Basic Run
| Attachment | Size |
|---|---|
| threaded_cron.patch | 14.57 KB |
#1
Up for review now.
#2
This is a very important patch....we need this to do more complicated and larger data sets.
#3
Considering how ambitious the change is, I am surpirsed and pleased thhat the patch is not too large. Will review soon.
#4
This won't make it for 5.0 but might be useful for 6.0. My initial reaction is that we should avoid adding configuration settings. The UI needs some work too. That said, more advanced cron functionality is a useful feature.
#5
Moving out of the "x.y.z" queue to a real queue.
An important patch like this needs to be in the right place.
#6
By the time someone gets to this, it will need to be ported.
#7
The "to be ported" state is for patches that have been contributed to HEAD and need to be backported to 4.7 or 4.6 for example. So this still needs review before that would happen.
@coreb - Not sure if there's any other issues that were incorrectly marked "to be ported". Let me know if you find any and have questions on the correct status. Thanks!
#8
Subscribing...
#9
Subscribing... and at the same time, could we implement some sort of cron ordering? Let's say I want to make sure that aggregator cron (or some other module) runs before one of my custom module (I don't think it's possible right now).
#10
Subscribing and adding to my 6.x todo-list.
#11
HEAD is again open. lets get this one in front of the committers.
#12
I advise using proc_open not popen.
#13
The patch is somewhat dangerous, cron_run_command should be a bit sanitized and even then, I am not sure I like it.
#14
Yes. Agreed about the dangerousness of the current patch. I have cleaned it up a lot. I shall try and post the updated patch this week.
#15
This is just a "re-sync" patch to get back in line with 5.1. It has been tested though and works.
The way I currently tested it is I have a few custom modules for clients paired with large datasets that have very long run-times.
Another easier way to test this is to modify existing cron runs. E.g.:
Try putting a sleep(15); line in the watchdog cron function, and run your cron manually. You will see that the page returns saying cron run-secucessful. If you go over to admin/logs/cron and select the newest cron run, you'll see that watchdog will wtill still be running and the amount of time it has been running.
The basics are there, just needs some heavy tweaking, cleaning and polishing.
#16
Killes,
I'm trying to come up with a scenario where the cron_run_command could be compromised as a "primary" opening. Meaning, if someone gains access to your MySQL database, and changes the variable in the database...there are larger security issues at hand.
However, my current scenario that I can think of is someone gains a drupal installations administrator's username/password, and changes the executable to "rm -r " or something malicious like that.
However, if you're running an important drupal site a secure sever, then Drupal (PHP) should not be running as a privileged user anyways.
Thoughts?
#17
I'll update this module if/after SQL Server goes into head, thus I don't have to keep 2-3 sets of db creation patches up to date after each head change. blah. :)
#18
Reroll?
#19
Subscribing
#20
Subscribing.
#21
subscribing
#22
Maybe work on this for HEAD and backport. It's needed.
#23
Anyone up for reviving this. HEAD is open and accepting patches :)