drupal_http_request() does not handle 'chunked' responses

The drupal_http_request() behavior hasn't changed since Drupal 4.6, or maybe longer.

If I understand the protocol correctly, we can look for the Transfer-Encoding header and then strip away the extra lines. The problem is, sometimes the content-length encodings appear in the middle of the message body, not just the beginning and end.

Possible (partial) solution:

http://us3.php.net/manual/en/function.fsockopen.php#71849

Many thanks for the bug report. Drupal actually just switched from HTTP/1.0 to HTTP/1.1 for drupal_http_request() (see http://drupal.org/node/104693), although I think this change was after rc1. This might be helpful in figuring out why gmap times out.

Do you still have this problem with using the latest CVS version of Drupal?

Oops, I was wrong about drupal_http_request() not changing. I didn't spot the HTTP/1.0 to HTTP/1.1 difference.

The problem disappears when I revert to HTTP/1.0. Thanks for the tip.

This patch attempts to handle "chunked" transfer encodings. My tests indicate it works.

Hrm. This patch uses standard filenames and paths. Sorry 'bout that.

It needs minor work to conform to drupal coding standards (see http://drupal.org/node/318) - operator spacing and comment styles both need to be tweaked.

It might be best to hold off until 6.x to add handling for chunked responses.

I had ported some code from php.net, hence the coding style. This patch cleans it up.

Chris: any chance you can test this with your gmap setup?

I think we need to postpone this to Drupal 6.

What's the advantage of the chunked protocol? I'd document that in the patch.

One advantage of the chunked transfer encoding is a theoretical ability to handle long responses efficiently, by syncing the HTTP level blocks with TCP-level reads, effectively adding minor synchronization points to TCP, which theoretically can make it more efficient: reads can be made to the announced chunk size instead of being to the default block size (see the "fetch response" part in drupal_http_header) and, if the underlying network supports it, will therefore be processed at the optimal speed. In addition, since the content length is known in advance from the chunk length, the end of socket read needn't be done using feof($fp), which relies on the protocol closing sequence and therefore introduces at least one latency iteration, but can close directly when the exact length of data has been read, since it is known in advance. See http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.6.1 for details.

The patch does not seem to take into account the optional final headers which may appear after the data when a Transfer-Encoding is defined as under the trailer BNF production for chunked encoding.

See http://www.w3.org/Protocols/rfc2616/rfc2616-sec19.html#sec19.4.6 for a recommended way of processing such encodings.

FWIW, I just encountered the problem today, where a Wordpress blog would return Transfer-Encoding: chunked to drupal_http_request in D5 and D4.7, although the request is send under HTTP/1.0 and such encodings are not allowed for HTTP/1.0 agents (as per section 3.6: A server MUST NOT send transfer-codings to an HTTP/1.0 client). Maybe someone familiar with WP should create an issue on their site too.

Any progress here?