Support for Drupal 7 is ending on 5 January 2025—it’s time to migrate to Drupal 10! Learn about the many benefits of Drupal 10 and find migration tools in our resource center.
I encountered this problem when using aggregator.module to load web feeds. Sometimes the process would return with an message about a "syntax error on line 1." As it turns out, chunked HTTP/1.1 responses have extra content-length encoding in the message body.
drupal_http_request() passes the extra encodings along as part of the message body. This confuses the XML parser.
Comment | File | Size | Author |
---|---|---|---|
#8 | chunked_1.patch | 1.06 KB | njivy |
#6 | chunked_0.patch | 1.26 KB | njivy |
#5 | chunked.patch | 1.25 KB | njivy |
Comments
Comment #1
njivy CreditAttribution: njivy commentedThe drupal_http_request() behavior hasn't changed since Drupal 4.6, or maybe longer.
If I understand the protocol correctly, we can look for the Transfer-Encoding header and then strip away the extra lines. The problem is, sometimes the content-length encodings appear in the middle of the message body, not just the beginning and end.
Comment #2
njivy CreditAttribution: njivy commentedPossible (partial) solution:
http://us3.php.net/manual/en/function.fsockopen.php#71849
Comment #3
ChrisKennedy CreditAttribution: ChrisKennedy commentedMany thanks for the bug report. Drupal actually just switched from HTTP/1.0 to HTTP/1.1 for drupal_http_request() (see http://drupal.org/node/104693), although I think this change was after rc1. This might be helpful in figuring out why gmap times out.
Do you still have this problem with using the latest CVS version of Drupal?
Comment #4
njivy CreditAttribution: njivy commentedOops, I was wrong about drupal_http_request() not changing. I didn't spot the HTTP/1.0 to HTTP/1.1 difference.
The problem disappears when I revert to HTTP/1.0. Thanks for the tip.
Comment #5
njivy CreditAttribution: njivy commentedThis patch attempts to handle "chunked" transfer encodings. My tests indicate it works.
Comment #6
njivy CreditAttribution: njivy commentedHrm. This patch uses standard filenames and paths. Sorry 'bout that.
Comment #7
ChrisKennedy CreditAttribution: ChrisKennedy commentedIt needs minor work to conform to drupal coding standards (see http://drupal.org/node/318) - operator spacing and comment styles both need to be tweaked.
It might be best to hold off until 6.x to add handling for chunked responses.
Comment #8
njivy CreditAttribution: njivy commentedI had ported some code from php.net, hence the coding style. This patch cleans it up.
Comment #9
Dries CreditAttribution: Dries commentedChris: any chance you can test this with your gmap setup?
I think we need to postpone this to Drupal 6.
What's the advantage of the chunked protocol? I'd document that in the patch.
Comment #10
fgmOne advantage of the chunked transfer encoding is a theoretical ability to handle long responses efficiently, by syncing the HTTP level blocks with TCP-level reads, effectively adding minor synchronization points to TCP, which theoretically can make it more efficient: reads can be made to the announced chunk size instead of being to the default block size (see the "fetch response" part in drupal_http_header) and, if the underlying network supports it, will therefore be processed at the optimal speed. In addition, since the content length is known in advance from the chunk length, the end of socket read needn't be done using
feof($fp)
, which relies on the protocol closing sequence and therefore introduces at least one latency iteration, but can close directly when the exact length of data has been read, since it is known in advance. See http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.6.1 for details.The patch does not seem to take into account the optional final headers which may appear after the data when a Transfer-Encoding is defined as under the
trailer
BNF production for chunked encoding.See http://www.w3.org/Protocols/rfc2616/rfc2616-sec19.html#sec19.4.6 for a recommended way of processing such encodings.
FWIW, I just encountered the problem today, where a Wordpress blog would return Transfer-Encoding: chunked to
drupal_http_request
in D5 and D4.7, although the request is send under HTTP/1.0 and such encodings are not allowed for HTTP/1.0 agents (as per section 3.6: ). Maybe someone familiar with WP should create an issue on their site too.Comment #11
PanchoAny progress here?
Comment #12
Steven Jones CreditAttribution: Steven Jones commentedThis is still an issue in Drupal.
Comment #13
mikeytown2 CreditAttribution: mikeytown2 commentedHelps to know what this looks like: http://en.wikipedia.org/wiki/Chunked_transfer_encoding#Example coming from #1320222: Bring in other drupal patches
Various functions on php.net. IMHO the regex ones seem like a bad idea.
http://php.net/fsockopen#96146
http://php.net/fsockopen#85572
http://php.net/fsockopen#73581 (this one handles compression)
Comment #14
Damien Tournoud CreditAttribution: Damien Tournoud commenteddrupal_http_request()
is a HTTP/1.0 client.If a server replied with chunked encoding to a HTTP/1.0 request, fix the server.
Comment #15
andypostSuppose we should upgrade drupal_
http_request()
to be a HTTP/1.1 client.There's a lot of code written so make it migrate easier!
HTTP Kernel is enough to make drupal_http_request() work as wrapper
Comment #16
mikeytown2 CreditAttribution: mikeytown2 commentedSee issue: #1447736: Adopt Guzzle library to replace drupal_http_request()
And code to implement this if that issue get's blocked: http://drupalcode.org/project/httprl.git/blob/d4710a3cb78dadaa4da0064326...
Comment #17
mikeytown2 CreditAttribution: mikeytown2 commentedGuzzle is in. Moving this to D7