Add support with non-blocking parallel link checking

When I opened this case I hoped to have this implemented within a week with the approach from http://www.onlineaspect.com/2009/01/26/how-to-use-curl_multi-without-blo..., but the status code and error handling and auto-updating of 301 links seems to be a quite complex task and made me working on other tasks.

If someone likes to jump in here - go on... we definitively need this feature.

http://www.jaisenmathai.com/blog/2008/05/29/asynchronous-parallel-http-r...

If the tests pass without this patch and with not - there may be bugs :-)))

+++ linkchecker.module Locally Modified (Based On 1.7.2.138)
@@ -137,20 +139,187 @@
+    // GET or HEAD?
+    if ($request->method == 'HEAD') {
+      curl_setopt($curl_handles[$index], CURLOPT_NOBODY, TRUE);
+    }
+    else {
+      curl_setopt($curl_handles[$index], CURLOPT_HTTPGET, TRUE);
+    }

By default we use HEAD, not GET. POST is not yet possible, we need to do some more stuff to get this working.

+++ linkchecker.module Locally Modified (Based On 1.7.2.138)
@@ -137,20 +139,187 @@
+      $responses = array(
+        100 => 'Continue', 101 => 'Switching Protocols',
+        200 => 'OK', 201 => 'Created', 202 => 'Accepted', 203 => 'Non-Authoritative Information', 204 => 'No Content', 205 => 'Reset Content', 206 => 'Partial Content',
+        300 => 'Multiple Choices', 301 => 'Moved Permanently', 302 => 'Found', 303 => 'See Other', 304 => 'Not Modified', 305 => 'Use Proxy', 307 => 'Temporary Redirect',
+        400 => 'Bad Request', 401 => 'Unauthorized', 402 => 'Payment Required', 403 => 'Forbidden', 404 => 'Not Found', 405 => 'Method Not Allowed', 406 => 'Not Acceptable', 407 => 'Proxy Authentication Required', 408 => 'Request Time-out', 409 => 'Conflict', 410 => 'Gone', 411 => 'Length Required', 412 => 'Precondition Failed', 413 => 'Request Entity Too Large', 414 => 'Request-URI Too Large', 415 => 'Unsupported Media Type', 416 => 'Requested range not satisfiable', 417 => 'Expectation Failed',
+        500 => 'Internal Server Error', 501 => 'Not Implemented', 502 => 'Bad Gateway', 503 => 'Service Unavailable', 504 => 'Gateway Time-out', 505 => 'HTTP Version not supported'
+      );

I'd like to move this into an extra function if not already available to reduce lines of code.

+++ linkchecker.module Locally Modified (Based On 1.7.2.138)
@@ -137,20 +139,187 @@
+          // If retry is set, fall back to drupal_http_request()
+          if ($retry) {
+            $result = drupal_http_request($result->headers['Location'], $headers, $method, $data, --$retry);
+            $result->redirect_code = $result->code;
+          }
+          $result->redirect_url = $location;

By this drupal_http_request() we have a blocking request as I know... :-(

Powered by Dreditor.

This curl modul from #12 should get a review if it provides us the same API/interface for installations with and without cURL.

#1. Your else statement defaults to GET. This is logic wise not correct and need to default to HEAD.

#3. I'm not sure... but if it blocks we don't have "non blocking requests". We could address this later if there is no way to solve...

Aside, should we try to depend on http://drupal.org/project/curl? I don't know the code, but it may easify some things if people don't have curl!?

D7 first

#8: linkchecker_CURL-380052-4.patch queued for re-testing.

http://drupal.org/project/httprl may also an alternative to curl. It's written as API compatible with core drupal_http_request(). In http://drupal.org/node/1272542 they plan to fallback to curl if stream wrappers are not available.

The good thing is we get all possibilities in one module and have a consistent API and we don't need to care about anything linkchecker internally. I love the ideas, but have no experience with httprl yet.

@janusman: I guess you spend a lot of time on the curl patch... I'm sorry for this, as the HTTPRL module was a chance find for me, too. But it looks really promising!

Just made a test with 1000 URLs (~3 minutes) and it's really cool and API conform to core.

Per #732626: Supporting checking link via Ajax in node view we should first move the checking logic out of hook_cron() and than we can make the HTTPRL module plug-able. hopefully #1268096: Implement a rate limiter get's solved as I do not like to commit a patch before. We can really overload any server with this feature.

Patch attached is a prove of concept. We need an UI so users can decide to stay with Core or use the HTTPRL module (Experimental). Changing topic to be more general.

Patch requires HTTPRL latest DEV or future v1.5. This is waiting for #1427958: Run callback after stream has been fetched as current logic is not that fast as expected because of some request timeouts that are blocking speed.

New patch

Missed to remove one workaround leftover from previous broken HTTPRL versions.

Fixed the link to httprl module.

Thanks a lot for the -2 hint. I missed this change. I'm currently testing the callback logic we discussed in #1589122: background_callback array with two parameters?. New patch will follow soon.

Latest patch attached with callback logic. This is a memory killer. No idea why.

I pushed only 560 links in the queue and when cron exits after some time only 60 links have been checked with a thread limit of 4. CPU is 100% and 1.5GB of memory has been used by Apache (httpd) process what is not acceptable.

@task: add global httprl timeout. This need to be calculated.

I can give "callback" a try tomorrow. I thought the background one is the one with less memory usage as it free's up after it's completed. How exacly can I make a decission for one or the other callback/background?

It's very simple... Fill up the site with many links... I have 1500 test urls. Than run cron with web url from status page. That's really all.

I plan to create a final linkchecker release very soon. Would be great to get this done before.

+++ b/linkchecker.module
@@ -385,7 +385,25 @@ function _linkchecker_link_block_ids($link) {
+  if ($has_httprl && !$background) {
+    // Setup callback options array; call linkchecker_cron(TRUE) in the background.
+    $callback_options = array(
+      array(
+        'function' => 'linkchecker_cron',
+      ),
+      TRUE
+    );
+    // Queue up the request.
+    httprl_queue_background_callback($callback_options);
+
+    // Execute request.
+    httprl_send_request();
+    return;
+  }
+

I'm sorry, but I do not really understand what you are doing here or how this works. I thought we are already running the link checks with httprl_send_request() after we filled the queue. Can you explain this a bit, please?

+++ b/linkchecker.module
@@ -395,40 +413,70 @@ function linkchecker_cron() {
+      if ($links_remaining == 0) {
+        // Make sure we have enough time to validate all of the links.
+        drupal_set_time_limit(240);
+
+        httprl_send_request();
+      }
...
+      _linkchecker_status_handling($response, $link);
...
+      if ((timer_read('page') / 1000) > ($max_execution_time / 2)) {

I'm fine with this drupal_set_time_limit() change. But as I know core already set's 240 seconds and it's not set again to 240 seconds as I know. This is only there for the case that any other module may not set it. e.g. core set 240s and any other module require 60 seconds before linkchecker comes in the sequence. Than we only have 180s left up to 240s. With timer_read('page') we can calculate the remaining seconds and make sure that httprl is able to execute the _linkchecker_status_handling() before php times out. To be a bit safe, we should not use more than 220s seconds over all or only 180s max. We need to add the global timeout for httprl to e.g. "220 - (timer_read('page') / 1000)" in $options array to make httprl exit before php kills the process the hard way.

Aside, do you know why the CPU load is so heavy (not tested latest two patches)? Normally core drupal_http_request() is low on CPU with 0% load. I also think httprl need to be low on CPU, too. Most of the time we are waiting for the remote servers to answer... My best guess is that there are some very aggressive loops without a sleep that will fire the CPU to 100%. That's all guessing, but what httprl is doing while it runs - shouldn't be that CPU intensive and the size of the url array should also not eat sooo much memory. 32 socket shouldn't be a problem as a server should theoretically be able to open up to ~65536 ports, too. Ok, there are timeouts blocking and we'd still like to be able to be accessible from outside, but all is relative here... 32*240s = 7680 checks (I do not expect these many) and this is not that much :-)

Thanks for all these details. I tried both of your patches. This is really heavy with 8 threads (<100% CPU, ~400MB httpd). Can we reduce CPU load?

I see httprl has a default global_timeout of 120s. Is this intentional? I would set it to 180s or so.

Moving the complete code outside of hook_cron() should be done for sure. There is some need to reuse the logic in other ways like checking one link with an ajax request. So we can do this for sure.

#39: ~1223 links with 8 treads in only one cron run... Dreams come true :-)))

This patch adds missing global_connections and global_timeout. You may remember I asked already to allow setting these global values with httprl_send_request(). Here we see how this could help. Now we add 8000 links to the queue and also 8000 times the global settings. This requires a lot more memory than needed and I'm also not able to set the global_timeout correctly. Is there a chance that this may change soon?

I've committed a cleanup patch to reduce the size of these patch and the stuff that is not really part of this new feature here. New patch attached that also moves the code outside of hook_cron().

Missed to update the hook_cron() call in the drush file.

Suxxx windows line feeds

#48: linkchecker_380052+Add+support+with+non-blocking+parallel+link+checking_callback5.patch queued for re-testing.

+++ b/linkchecker.moduleundefined
@@ -418,12 +448,48 @@
+        'global_connections' => $linkchecker_check_connections_max,
+        'global_timeout' => 180 - (timer_read('page') / 1000),
...
+        httprl_send_request();

Looking for feedback on how we can move this into httprl_send_request($global_options);. Maybe HTTPRL 1.8 can support this?

Would like to move link checking logic logic out of the cron hook so the new thread/process will call something other than a linkchecker_cron (having linkchecker_cron call linkchecker_cron is a quick hack). Make this new function have as little side effects as possible (don't call drupal_set_time_limit) so it can be used elsewhere without any time limits enforced (drush).

It's already changed in the last patch or have I misunderstood this? The function is named _linkchecker_check_links().

Looks like we do have a new function so I'll see what I can do in order to reduce the side effects

What side effects? The high CPU load? :-)

I fear to commit this patch :-(. The latest one brings httpd up to 1.2GB of memory, 100% CPU with permanent load (machine is non-responsive) and runs *only* with 8 concurrent HEAD connections. Without the reference it looked smarter on the memory and CPU side.

HTTPRL need to be made more performant and a lot less memory and CPU intensive first.

Aside, shouldn't we have a lock on the function call? Otherwise cron may be executed more than once and this will start up N background processes what will kill nearly every machine and will check the same urls N times. Is this something that HTTPRL need to handle or linkchecker? Any idea how we can add a lock?

We can use below:

http://api.drupal.org/api/drupal/includes%21lock.inc/function/lock_acqui...
http://api.drupal.org/api/drupal/includes%21lock.inc/function/lock_relea...

I'm open minded to optimizations that work reliable... :-)

What is head_only? The option for HEAD or GET is "method" only.

+++ b/linkchecker.moduleundefined
@@ -386,31 +386,65 @@ function _linkchecker_link_block_ids($link) {
+  $max_execution_time = ini_get('max_execution_time') == 0 ? 240 : ini_get('max_execution_time');
+  if ($time_limit > 0) {
+    // Make sure we have enough time to validate all of the links.
+    drupal_set_time_limit($time_limit);
+  }
+  // Make sure this is the only process trying to run this function.
+  if (!lock_acquire(__FUNCTION__, max($max_execution_time, $time_limit))) {
+    return FALSE;
+  }

Why is there a need for two time limits? We know from $max_execution_time that this is the maximum time we can run a script, isn't it?

+++ b/linkchecker.moduleundefined
@@ -386,31 +386,65 @@ function _linkchecker_link_block_ids($link) {
+  if (!lock_acquire(__FUNCTION__, max($max_execution_time, $time_limit))) {
+    return FALSE;
+  }

I'm going to add a watchdog in here later to warn about the locked process. Maybe useful information to know. :-)

This should be enough, isn't it?

/**
 * Run link checks.
 */
function _linkchecker_check_links() {
  // Get max_execution_time from configuration, override 0 with 240 seconds.
  $max_execution_time = ini_get('max_execution_time') == 0 ? 240 : ini_get('max_execution_time');
  // Make sure we have enough time to validate all of the links.
  drupal_set_time_limit($max_execution_time);

  // Make sure this is the only process trying to run this function.
  if (!lock_acquire(__FUNCTION__, $max_execution_time)) {
    return FALSE;
  }

New patch

Based on #61.

• Removed $time_limit
• Added some watchdogs

I do not know why, but the memory usage is not always logged on my machine.

Ok, we found my local problem. :-( I have disabled all links to http://localhost and now all run smooth. Memory usage is good (~80MB), CPU mostly low, short spikes with max 25%. Sorry for stressing on this side. But it clearly shows me that we can kill a remote server with linkchecker very quickly. I guess I will set this domain limit down to 2 threads how the RFC suggests originally and may allow overriding this via settings.php.

+++ b/linkchecker.moduleundefined
@@ -386,31 +386,66 @@ function _linkchecker_link_block_ids($link) {
+  $max_execution_time = ini_get('max_execution_time') == 0 ? 240 : ini_get('max_execution_time');

@@ -418,26 +453,69 @@ function linkchecker_cron() {
+        'global_timeout' => 180 - (timer_read('page') / 1000),

Need to move this outside the foreach and set all to the same time. $max_execution_time - 30s or 60s if we know we are running in an extra process we have 100% of the time and there is no need to read the page timer.

+++ b/linkchecker.moduleundefined
@@ -418,26 +453,69 @@ function linkchecker_cron() {
+        'head_only' => TRUE,

Removal, we are using the Range header.

+++ b/linkchecker.moduleundefined
@@ -418,26 +453,69 @@ function linkchecker_cron() {
+      // If connecting to this server, limit connections to 1.
+      if (strpos($link->url, $base_http) === 0 || strpos($link->url, $base_https) === 0) {
+        $options += array(
+          'domain_connections' => 1,
+        );
+      }

I'm not a fan of committing this... this case just shows us a box may be quickly overwhelmed by 8 threads at the same time and on the end of the day the "per domain" limit may be too high with a thread limit of 8. :-)

+++ b/linkchecker.moduleundefined
@@ -418,26 +453,69 @@ function linkchecker_cron() {
+  watchdog('linkchecker', 'Link check run completed.', array(), WATCHDOG_NOTICE);
+  watchdog('linkchecker', 'Memory usage: @memory_get_usage byte, Peak memory usage: @memory_get_peak_usage byte.', array('@memory_get_peak_usage' => memory_get_peak_usage(), '@memory_get_usage' => memory_get_usage()), WATCHDOG_NOTICE);

I do not see both entries in the watchdog logs. Maybe httprl_send_request() does not come back and is also not able to remove the named lock. This needs further investigation.

+++ b/linkchecker.moduleundefined
@@ -456,6 +534,11 @@ function _linkchecker_status_handling($link, $response) {
+    case -4: // HTTPRL: httprl_send_request timed out.
+      // Skip these and try them again next cron run.
+      break;

Good catch. I planed to look into httprl to see if there may be more I have missed in past moths...

Last patch, this one will be committed if all lights are green.

Removed 1 from Number of simultaneous connections.

Better formatting of memory usage

HTTP Parallel Request Library is now named HTTP Parallel Request & Threading Library

@mikeytown2: It was a pleasure to work with You on this feature. Thanks a lot for all your help!!!

D7: http://drupalcode.org/project/linkchecker.git/commit/b56dda1

http://drupalcode.org/project/linkchecker.git/commit/52d44fe