Just a heads up I was able to do sudo multi threading on vanilla php. it works on windows, ect...


    // Return html so connection closes
    boost_async_opp('async');

        //Spin Up Threads
        $thread_primed = variable_get('boost_crawler_threads_primed', FALSE);
        $threads = 2;
        if (!$thread_primed) {
          variable_set('boost_crawler_threads_primed', TRUE);
          while ($threads > 0) {
            $threads--;
            boost_async_call_crawler($self);
          }
          watchdog('boost', 'All threads started');
          exit;
        }

/**
 * Output text & set php in async mode.
 */
function boost_async_opp($output) {
  // Prime php for background operations
  ob_end_clean();
  header("Connection: close");
  ignore_user_abort();

  // Output text
  ob_start();
  header("Content-type: text/html");
  header("Expires: Wed, 11 Nov 1998 11:11:11 GMT");
  header("Cache-Control: no-cache");
  header("Cache-Control: must-revalidate");
  header("Content-Length: " . (strlen($output)-1));
  header("Connection: close");
  print($output);
  ob_end_flush();
  flush();
  // wait for 1 second
  sleep(1);

  // text returned and connection closed.
  // Do background processing. Time taken after should not effect page load times.
}

/**
 * Call a URL with a timeout of 3 seconds.
 *
 * @param $self
 *  URL to restart the loop.
 */
function boost_async_call_crawler($self) {
  $timeout = ini_get('default_socket_timeout');
  ini_set('default_socket_timeout', 3);
  @file_get_contents($self);
  ini_set('default_socket_timeout', $timeout);
}

Code is in the latest version of boost
http://drupal.org/project/boost

Comments

irakli’s picture

Something not right.

I tested the code you posted with 20 seconds wait, to see the picture more clearly:

// wait for 1 second
sleep(20);

and it took: 6 seconds. Commented out the "sleep" and it took 0 seconds. Quite strange. While it's not necessarily waiting, it's nor working as expected either.

mikeytown2’s picture

@irakli
The crawler code is still under a lot of development, so some of what you see above has changed in the boost codebase; means new code should work better.

cpliakas’s picture

Status: Active » Closed (won't fix)

Due to end of life announcement.