Imagine there are N servers with the following line in their crontab:
* * * * * /usr/bin/wget -O - -q http://example.com/cron.php
This means example.com/cron.php will be executed N times each minute, right?
Is this something that should be avoided?
Will it cause jobs to run more times than if cron.php was run only once per minute?
How much overhead does this result in?
Comments
Comment #1
gotheric commentedYes, that way you execute cron N times each minute.
If you use standard "cron.php" there is a little overhead in each call (for cache handling, that causes all variable cache to be refreshed in memory and db), and calling it a lot of times means a lot of overhead!
To avoid it use elysia cron's own "cron.php".
* * * * * /usr/bin/wget -O - -q http://example.com/sites/all/modules/elysia_cron/cron.php
Read "ADVANCED INSTALL" section of elysia cron README.TXT for more info.
Comment #2
gotheric commentedComment #3
mrP commentedresurrecting an old post regarding new drush integration in 2.x
i've got the same configuration with all sites using crontab:
* * * * * cd ~/public_html; drush @sites elysia-cron --yes
Every minute, there are N requests on this particular server to cron.
I'm definitely experiencing a lot of overhead in terms of server load and memory utilization. A few questions regarding this.
1. Is elysia_cron intended to handle this type of configuration efficiently?
2. Would it be better to script drush aliases for all N sites and have a single crontab line at root level crontab?
3. Staggered (ie, not all * * * * *) crontab entries?
4. Any insight into wget vs drush performance?
5. When calling elysia-cron via drush, is this equivalent to triggering http://example.com/sites/all/modules/elysia_cron/cron.php with wget?
Many thanks.
Comment #4
gotheric commented@pricejn:
- calling elysia_cron via drush is nearly the same than running elysia_cron/cron.php. The only difference are in security checks (drush run skip check_key and IP checks) and in running users setup... nothing that involves less (or better) performance.
- so, performance optimizations are active on the elysia-cron side. But i don't know the "drush" side: if drush, during is normal task execution, does some time-consuming operation, running it very often could be a bottleneck. For example: if it sets some drupal variable it invalidates the variable cache that should be rebuilt each time. This is the same that standard cron do, and that elysia cron avoid (when running it directly with elysia_cron/cron.php).
- generally the best thing if you want performance is to run few things... so it's better to run ONLY elysia_cron than drush+elysia_cron... however you can try to compare the 2 configurations. Try running only elysia_cron and see if performance are better. Maybe yes, maybe not...
- however you must consider that even with the max performance tricks, running a cron chain is NOT a time-free operation! Running it every minute should be fine, but running it hundreds of time every minute will definitely has a lot of load!
I don't understand you (multi-site?) configuration, but don't run cron (or everything else) too much if it's not needed...
Comment #5
mrP commentedThanks for your comments and feedback @gotheric. Its really helpful.
For clarification, you mean to remove the crontab (drush) entries and let the elysia_cron module handle cron tasks?
The site configuration is fairly standard/simplistic in many ways. It is not multi-site but rather multiple (N) single sites. So the effect is that N php processes are spawned every minute. I understand that a lot of this is probably specific to my server configuration and I certainly don't want to waste your time in that regard.
But in general with this site config (which I think is still fairly typical among smaller drupal shops) where each user has a single site:
1. /home/site1/public_html/sites/all/modules/elysia_cron.php
2. /home/site2/public_html/sites/all/modules/elysia_cron.php
.....
N. /home/siteN/public_html/sites/all/modules/elysia_cron.php
And each site has a single cron job entry:
1. * * * * * wget -O - -q -t 1 http://www.site1.com/sites/all/modules/elysia_cron/cron.php
2. * * * * * wget -O - -q -t 1 http://www.site2.com/sites/all/modules/elysia_cron/cron.php
...
N. * * * * * wget -O - -q -t 1 http://www.siteN.com/sites/all/modules/elysia_cron/cron.php
Obviously, this turns into a lot of server overhead occurring every minute, especially as N gets larger. From your previous comment, it sounds like the cron job entries should only be created when necessary rather than by default to avoid excessive php requests.
Comment #6
gotheric commentedI meant to use elysia_cron without drush, calling it directly in your system crontab (like in your examples).
Consider that each time system cron invokes "http://www.site1.com/sites/all/modules/elysia_cron/cron.php" there is some work that the site is doing (and without elysia_cron this is even more):
1. a full drupal bootstrap (this is needed for drupal to work): this involves also a variable init that usually reads all variables from cache table
2. a query in elysia_cron table to see if there are channel/tasks to execute
3. the work of each task executed (obviously it may vary)
#3 in the majority of executions should do nothing (if you have few jobs configured in drupal sites... but if you have a job that is configured to run every minute "* * * * *" this is not true and you should check about it!)
If you have a lot of sites, and these sites are used, elysia_cron should not be a real problem: running elysia_cron usually does less work than a person that is doing a standard http request in your site. So if it's a problem running elysia_cron every minute in every site, it means that if your sites are visited by only a person with 1 request/minute this will be a problem too! And this is a very little load for a site (a single spider does a lot more requests when indexing your sites).
So this usually means that your machine is too small to have all that sites...
Maybe a problem should about the PEAK load that occours by executing all cron simultaneously (all site's cron are executing, teorically, at "0" second of each minute).
You can prevent this by executing a bash script that runs every cron sequentially.
For example, in your system crontab, something link this:
[code]
* * * * * /path_to_script/run-cron.sh
[/code]
in /path_to_script/run-cron.sh:
[code]
wget -O - -q -t 1 http://www.site1.com/sites/all/modules/elysia_cron/cron.php
wget -O - -q -t 1 http://www.site2.com/sites/all/modules/elysia_cron/cron.php
...
wget -O - -q -t 1 http://www.siteN.com/sites/all/modules/elysia_cron/cron.php
[/code]
This way each cron is executed sequentially after the previous cron is terminated.
(If you have a lot of sites executions run-cron.sh will overlap, you can do some shell-script tricks to avoid this - having a lock file or similar).
Comment #7
mrP commentedThis is definitely the key observation here:
Many thanks.