HTTP Parallel Request & Threading Library

This is a library module. It provides no out of the box functionality other then providing an API that other modules/code can use. Other projects might require/recommend this module. Install HTTPRL only if other modules recommend or require it.

What does httprl do?

Using stream_select() it will send http requests out in parallel. These requests can be made in a blocking or non-blocking way. Blocking will wait for the http response; Non-Blocking will close the connection not waiting for the response back. The API for httprl is similar to the Drupal 7 version of drupal_http_request().

As a bonus, a simple threading library is built on top of the parallel http requests functionality. This allows you to split a job and have multiple http "workers" running this split job in parallel. Anything that takes a long time to do can greatly benefit from using threads.

Benchmarks

Single httprl request are slightly faster then drupal_http_request() because the headers are parsed mid stream; thus making preg_split() parse less text resulting in faster execution time. Multiple requests using httprl can be over 5x faster then core's drupal_http_request() because multiple connections are open and the streams are downloaded in parallel. If doing a non-blocking request, the URL is requested but the connection is closed not returning the result. Non-Blocking requests are good for doing background processing as it takes about 1ms to ping 1 url. When doing a normal request you can expect httprl to take on average 5ms per connection in order to process it (excluding network time).

Requirements

Requires PHP 5 and stream_socket_client(), stream_select(), stream_set_blocking(), stream_socket_get_name(), & stream_get_meta_data() must be available on the server. Some hosting providers disable these functions; but they do come standard with PHP 5.

Example Code/Usage

See the readme for examples on how to use httprl.

Projects using it

SMTP User Authentication
Imageinfo Cache
7.x Boost Crawler
Link Checker
Apache Solr Parallel Indexing
Advanced CSS/JS Aggregation

Downloads