boost_cron function failing, due to fsockopen() not respecting user given timeout value [#577578]

Comment	File	Size	Author
#14	boost-577578.patch	2.48 KB	mikeytown2

#5	cron.patch	860 bytes	mikeytown2

#1	boost-577578.patch	1.35 KB	mikeytown2

File	Size
boost-577578.patch	1.35 KB

Comment #2

mikeytown2 CreditAttribution: mikeytown2 commented 15 September 2009 at 01:43

Status:

Active

» Needs review

Log in or register to post comments

Comment #3

dbeall CreditAttribution: dbeall commented 15 September 2009 at 04:24

just added patch to 4 shared host sites and 1 vps site. Don't know what effect it will have, but will report anything I can find. Have cleared watchdog, clean slate.

Off Topic: will be updating handbook soon to reflect any changes...

Log in or register to post comments

Comment #4

dbeall CreditAttribution: dbeall commented 15 September 2009 at 18:02

update: Don't know if this applies here or to this thread. Sometimes just not sure which issue to use.
getting some erratic cron behavior, sometimes DbLog says cron has been running for over an hour and is most likely stuck. Other times it's normal.. hmmm
I am seeing a lot of sleeping cron notices.
It tries to crawl user, user/*/track and user/*/contact and gets access denied. Have added those pages to not-crawl text box.
It's difficult to tell whats going on from the DbLog sometimes.
Have cleared everything and cleared DbLog, will watch to see what happens..

Log in or register to post comments

Comment #5

mikeytown2 CreditAttribution: mikeytown2 commented 15 September 2009 at 19:31

File	Size
cron.patch	860 bytes

The crawler is doing something that php was never designed to do (multi-threaded, long running, batch operations), so there will be bumps in the road. This patch took care of cron timing-out for me, but it could be another module as well; there is no good way of figuring out what cron operation failed (unless you want to hack core).

http://drupal.org/project/cron_mt is an alt, but requires a core patch and PCNTL to be compiled with php (most don't do this).

Below is a patch; should allow you to ID what cron process is failing (yes your hacking core here).
http://api.drupal.org/api/function/drupal_cron_run/6
http://api.drupal.org/api/function/module_invoke_all/6

Log in or register to post comments

Comment #6

dbeall CreditAttribution: dbeall commented 15 September 2009 at 20:02

not a problem, i know i shouldn't, but have hacked core many times.(don't tell anyone).
Will investigate after this boat is fixed.

Log in or register to post comments

Comment #7

mikeytown2 CreditAttribution: mikeytown2 commented 15 September 2009 at 20:03

btw, I know about boats; I have learned to hate the OMC 800 outdrive, if you know what that is.

Log in or register to post comments

Comment #8

dbeall CreditAttribution: dbeall commented 15 September 2009 at 20:36

not exactly sure, but think i know the one. I am a boat nut, live on buckeye lake, make boat covers for a living because i don't really want to fix em for everyone, have 6 boats -slow to fast- 14' row boat, 24' pontoon, 14' 90hp checkmate, 21' 250hp lazer vee, 15' 250hp tunnel, 19' 320hp mod racing tunnel that runs 120mph. i'ts fun, quick and fast.lol (hacking core is painless and costs $0, lol).

Log in or register to post comments

Comment #9

mikeytown2 CreditAttribution: mikeytown2 commented 15 September 2009 at 23:55

Just noticed on my host that it's not respecting set_time_limit(0);. I should set the default batch size to a variable based off of the average page generation time & the servers set max_execution_time variable. Let end user up that if they wish.

Also noticed that if I call the crawler directly, it doesn't assign a thread to that process. I should do that.

Log in or register to post comments

Comment #10

dbeall CreditAttribution: dbeall commented 16 September 2009 at 00:26

Since running Drupal and having php.ini access, i have just blindly upped a few things:

max_execution_time 600 600
max_input_nesting_level 64 64
max_input_time 120 120
memory_limit 96M 96M
open_basedir no value no value
output_buffering no value no value
output_handler no value no value
post_max_size 96M 96M

Off Topic, i remember now about allow_url_fopen on my server. It was because of PHPgedview or B2evo. I have gallery2 running, but that wasn't the one that wanted it. Both PHPgedview and B2evo have since updated(mine are old), but I'm a Drupaler now.. B2evo is on the way out, but PHPgedview is a keeper.

Log in or register to post comments

Comment #11

dbeall CreditAttribution: dbeall commented 16 September 2009 at 02:50

it seems the root installed sites are doing ok, call cron from browser while logged out, flushes and rebuilds nicely, watching it regenerate by refreshing FTP. Not really any issues there. watching it.

it's the sub directory one that is having crawler sleeping repeating. Redid the timeout setting to 0, clear all, *Found a bot of some kind may have been causing a problem, -blocked the IP, watching it.

Log in or register to post comments

Comment #12

dbeall CreditAttribution: dbeall commented 16 September 2009 at 07:28

I love that core hack! Thank you, keeping that handy I have no idea why it is not enabled by default with a verbose setting or a setting in each module to turn it on or off. No wonder nobody can figure out what is happening. Maybe it should be called DbDumbedDownLog without it.
the root installed sites are doing ok..
issue is only on the shared host site in sub folder, debrafetty.us
The cron was stuck for 3 days. Had to pull boost, delete cron entries from database to make it run.
Back tracking, start from scratch.

Log in or register to post comments

Comment #13

mikeytown2 CreditAttribution: mikeytown2 commented 16 September 2009 at 07:30

#578546: Reset crawler

Log in or register to post comments

Comment #14

mikeytown2 CreditAttribution: mikeytown2 commented 16 September 2009 at 08:36

File	Size
boost-577578.patch	2.48 KB

Still need to assign a thread to a rouge processes, or kill it.

Log in or register to post comments

Comment #15

dbeall CreditAttribution: dbeall commented 16 September 2009 at 09:28

applying patch now..

ON Topic: I saw your chipin on the new release page, I have a policy of contributing to module projects(see my user page). This project is on my list after my storage boats come in.
Start Ranting...
Some people may take development work for granted, but I am not one of them. Drupal and Modules may be 'free' for download and use, but it all comes at a cost to someone.
Hopefully others will contribute too, the power of it is when many people help.
I contributed to xmlsitemap a few months ago and I was upset to see only 20 some people helped.(sucks)
I wish I was a rich guy or wish the boat business was worth a dam(this year).. I would be passing out lots of bucks to lots of great projects.
Quit Ranting....

Log in or register to post comments

Comment #16

dbeall CreditAttribution: dbeall commented 16 September 2009 at 22:28

Reporting with update:
Shared host single domain, Full Drupal install in Sub folder.
Boost 6.x-1.0 with both (2) patches from this thread, #577578(same number).
Scheduled Cron runs as normal, flushes, rebuilds, No errors.
Cron on command while logged in; cron runs as normal, flushes, rebuilds, No errors.
Looks good, no issues to report.

EDIT:
Will install Boost with patches in more sites tonight while upgrading core.

EDIT#2: The timeout or throttle on the test site was set to 0.
Changed php.ini on my accounts. max_execution_time (600) to (150)
Honestly, i have no idea why i set that at 600, all the others were at 120/200

Log in or register to post comments

Comment #17

mikeytown2 CreditAttribution: mikeytown2 commented 16 September 2009 at 23:22

600 is 10 min. For the crawler I set that to some ridiculously high number so php won't timeout. It's only under certain circumstances that a timeout this high is needed. In general it's there to prevent infinite loops and things of that nature. My recommendation to you is to keep the default, somewhere around 1-3 min usually.