i have been noticing that our cron hasn't been completing on a semi-regular basis and i've narrowed it down to the fact that the linkchecker cron hook isn't finishing.

to figure that out i installed supercron, and ran each one separately, moved the linkchecker to the last hook called, etc, till i could rule out any of the other cron hooks hanging.

then my next task was to figure out where in the linkchecker cron hook it was hanging up. i added quite a few watchdog calls all over the cron hook and kept an eye on my watchdog log to see what was going on. i surrounded line 144 with watchdog calls so that it would log every link that was checked when it started and then after it received a response like so:

watchdog('linkchecker', 'checking: '.$link->url, array(), WATCHDOG_NOTICE);
$response = drupal_http_request($link->url, array('User-Agent' => 'User-Agent: ' . $useragent), $link->method, NULL, 1);
watchdog('linkchecker', 'done checking: '.$link->url, array(), WATCHDOG_NOTICE);
_linkchecker_status_handling($link, $response);

what i noticed was that sometimes, what seems like random links, that seem to work when i go to them manually, never get to that second watchdog call.
that causes the linkchecker module to hang, and therefore cron never completes.

when i find that link in the linkchecker_links table, and set it to status 0, and then rerun cron, the linkchecker module runs just fine and cruises through all the links it should and cron finishes as it should.

what i'm wondering is:
is there a way to automatically set the status of a link, or maybe a "no response" flag, and then skip it and move on to the next one?
should these be getting set to a "timeout" broken link status but for some reason aren't?
is this something that i might have messed up in my config?

the way i have to do it now is to monitor that watchdog table and if i see a linkchecker link that starts checking but never finishes, i manually set that link to status = 0

i'm not sure what is causing it to get no response from these random links since when i go to them manually in a browser, they seem to load just fine...

thanks!

Comments

hass’s picture

hass commented
Status: Active » Postponed (maintainer needs more info)

That's strange... is your PHP time-out 240 seconds? Make sure 240s can get set by drupal cron! drupal_http_request() have a time-out of 15s/20s (http/https) and the module stops checking links after the half cron time is completed. So if the last check is executed at 119s and the server does 3 redirects you get a sum of max 3 x 20s = 60s + 119s = 179s. This is ~61 seconds before PHP times out... add some processing time and other stuff on top... we should always be save.

I know about a core bug that the time-out is not respected correctly... #156582: drupal_http_request() should support timeout setting.

So over all you need to identify the URLs that are broken... please post them... I will give them a try to. In general you can test yourself with "wget --spider http://example.com". spider does a HEAD request - that is different from what a browser does. Browsers normally do a GET. So if the server admin have blocked HEAD requests you may run into issues. You can try to change them to GET - but *never* do this if you check files like downloads or drupal_http_request() will try to download the file - what will for sure time out if it's a big file. So more and more issues...

To make you hidden for the server admin, try a different browser identifier... maybe they blocked the Drupal user agent... but first of all, please post the URLs so I can take a look.

hass’s picture

hass commented

Do you have this cron failures with the original code or with your customizations posted in #739970: customizable peak time max link checker add-in thing?

chadd’s picture

chadd commented

it's always the first url of the cron hook that seems to hang, never the last.
[edit: well, it could be the last, but then the next cron it's the first and only. it just seems to hang when it gets to these particular url's]
whether it is max execution time of 30 or 300, it is always the only url attempted.

here's a few that have failed:
http://www.vetmedcenter.com/vmc/vmc_index.jsp
http://www.bpacelearning.com
https://webmail.matcmadison.edu/owa

my default max execution time is 30.

i'm not quite sure how to do that wget, or what i should be getting for a result. when i run:
<? exec('wget --spider https://webmail.matcmadison.edu/owa'); ?>
it just returns a blank page.

and cron failures happened with the original code AND my new customizations.

hass’s picture

hass commented

http://www.vetmedcenter.com/vmc/vmc_index.jsp
-> 200, OK, no delays.

http://www.bpacelearning.com
-> 403 Forbidden, no delays.

https://webmail.matcmadison.edu/owa
-> no delays, SSL cert is not correctly installed - have you enabled SSL ìn your PHP install? If not, do it!

No idea why it should hang... wget is only to debug if the server or request works well. Don't use it in execs. Higher your max execution time to 240s. This could be the main reason for your issues. 30s /2 = 15seconds and this is *less* then 20 seconds time-out you have for SSL. Many bad sites do not answer sooo quick...

I also had cron failures on my xampp box, but this was caused by a disabled openssl module...

chadd’s picture

chadd commented

when i look at my php info page, it looks like my max execution time is 30, but in my watchdog, when linkchecker runs, it is telling me that the max execution time is 240.
not sure what's going on there...

[edit: looks like when i run cron manually it sets max execution time to 240, and when i invoke linkchecker via supercron, it sets max to 30]

(i'll tell those people in charge of that server to install that SSL correctly. that's a different server run by a different bunch o' folks.)

hass’s picture

hass commented

Core's hook cron tries to sets 240s - in some limited environments this may fail. I don't know what supercron is...

chadd’s picture

chadd commented

http://drupal.org/project/supercron

chadd’s picture

chadd commented

when using lynx to check these links they hang indefinitely:

# lynx https://webmail.matcmadison.edu/owa
>  Making HTTPS connection to webmail.matcmadison.edu:
>  Alert!: Access without authorization denied -- retrying
>  Username for 'webmail.matcmadison.edu' at server 'webmail.matcmadison.edu:443':

[..and it sits there until you enter something... hit enter and then yes...]

>  Proceed without a username and password? (y/n)
>  HTTP request sent; waiting for response.

[and then it just sits there... let it go for like 5 minutes last time.. nothing.]

the same steps for http://www.bpacelearning.com return an error 403/Forbidden, then it asks about cookies, and then it just sits there forever.
It doesn't matter if you accept or deny the cookies, it still hangs.

could that be something to look into?
from what i can tell, linkchecker is using the drupal_http_request() function http://api.drupal.org/api/function/drupal_http_request/6
is the drupal_http_request() running into the same issue that these lynx examples are hitting? maybe it's a cookie or validation thing?

and it looks to me that if the drupal_http_request function doesn't work for whatever reason (maybe just like these lynx examples) it just sits there forever and never errors or times out.
it would seem like that drupal_http_request should be able to time out or something.

[edit: looks like people are aware of the issue: http://drupal.org/node/156582 ]

hass’s picture

hass commented

Jepp... but normally it should time-out after 20s with HTTPS as said above. With HTTP after 15seconds, but for sure if drupal_http_request() have issues - linkchecker have a problem, too. :-(. You need to monitor the other case #156582: drupal_http_request() should support timeout setting or work on the patch... :-)

chadd’s picture

chadd commented

the problem, as i see it, is that the linkchecker code, is dependent upon the drupal_http_request returning a response.
but, from what i can tell, no response is returned for these specific examples, so it just sits and waits for one.
and it never gets to the line in the linkchecker.module that should time it out.
the linkchecker.module timeout seems to work fine IF it gets a response from the drupal_http_request.

hass’s picture

hass commented
Status: Postponed (maintainer needs more info) » Fixed

Ok, than this is the issue linked above. :-(

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.