Proxy https support for drupal_http_request()

RFC 2187 is titled "Upgrading to TLS Within HTTP/1.1"

You patch has 'protocol' => 'HTTP/1.0' so I don't understand why that's not fundamentally wrong.

@jpsoto - there are a lot of incorrect code-style changes here that make it hard to see the relevant (meaningful) changes, plus a lot of the new code doesn't conform to coding standards either.

http://drupal.org/coding-standards

Also, this patch seems to mix the definitions of HTTP 1.0 and 1.1, e.g. from:

http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html

Since HTTP/1.0 did not define any 1xx status codes, servers MUST NOT send a 1xx response to an HTTP/1.0 client except under experimental conditions. 

But this patch has the client handling 1xx codes when properly the client should indicate a failure. However, I realize that some HTTP servers like Varnish improperly serve 1.1. responses to 1.0 clients due to the pigheadedness of their developers.

@jpsoto - please fix the code style - current patch is way outside the expected.

#14: 924498_drupal_http_request_101004.patch queued for re-testing.

The patch in #20 seems to work on a Drupal 7.0 common.inc - it applies with a bunch of fuzziness, but so far it appears to be working properly. I don't feel confident enough to change the status but wanted to include my $.02

[mike@host web]$ patch -p0 < proxy.patch
patching file includes/common.inc
Hunk #1 succeeded at 722 (offset 4 lines).
Hunk #2 succeeded at 757 with fuzz 2 (offset -11 lines).
Hunk #3 succeeded at 781 (offset 4 lines).
Hunk #4 succeeded at 856 (offset -11 lines).
Hunk #5 succeeded at 893 (offset 4 lines).
Hunk #6 succeeded at 932 (offset -11 lines).
Hunk #7 succeeded at 1098 (offset 4 lines).
Hunk #8 succeeded at 1095 (offset -11 lines).
patching file sites/default/default.settings.php
Hunk #1 succeeded at 444 (offset 15 lines).
[mike@host web]$

Ok,

Sorry I'm fairly new to this all, I'm coming from mostly Microsoft environment.

I'm trying to setup a Drupal 7 instance on our Windows2008 (64bit) server:
- Installed WAMP
- Installed Drupal 7 successfully (standard install)

the server is behind a firewall and uses a proxy to reach the internet. I was able to apply the patch in #20 successfully. Uh... now what? How do I set the proxy machine/port? I've enabled the OpenID module on Drupal and would like to test it out.

Thanks

Thanks so much jpsoto! Your solution/patch is EXACTLY what I needed.

Our servers need to reach a proxy server to make outgoing HTTP/HTTPS requests. This was done on the IIS world by command line proxycfg.exe for our .NET applications. I was trying to look for an Apache equivalent.

I did some more digging and ended up hard-coding the values into the common.inc file before I saw your post.

I updated my settings.php with the correct values and VOILA! I am able to login via OpenID! thank you so much. I'm surprised this isn't more of an issue with other Drupal environments.

The patch gave me an error when I did not set a username or set it to '' in settings.php

$conf['proxy_username'] = '';
or commenting out that line completely results in an error

Notice: Undefined index: Proxy-Authorization in drupal_http_request() (line 910 of /var/www/drupal-7.0/includes/common.inc).

I can see it is caused by the code on line 829 of common.inc that isn't executed without a non-empty username. Setting the username to 'foo' allowed OpenID login to work because my proxy server ignores username/password being passed in if it has no username configured. I don't know what the correct value for $options['headers']['Proxy-Authorization'] should be in the case of no value for username.

    if ($proxy_username = variable_get('proxy_username', '')) {
      $proxy_password = variable_get('proxy_password', '');
      // As described by RFC-2617 for Basic Authentication Scheme.
      // Digest Access Authentication Scheme is not supported (by now)
      $options['headers']['Proxy-Authorization'] = 'Basic ' . base64_encode($proxy_username . (!empty($proxy_password) ? ":" . $proxy_password : ''));
    }

The above patch worked OK for me and solved the OpenID authentication issue I was having by allowing HTTPS connectivity via the proxy.

However, it needs default proxy settings configured in settings.php or else drupal_http_request will generally fail.

Perhaps it can be adjusted to not require the default settings.php proxy settings, but rather to set the defaults itself, if not set.

Otherwise fine, thanks!

DT

Bumping to 8.x

Hmm I think is too late to solving it only for D7. I won't play version ping-pong, but this patch needs to be fixed first in D8 and backported to D7 IMO since it is new feature and since D8 is out for couple of weeks.

@jpsoto yes, but it is impossible to put this to D7 directly without putting this to D8 first because if so, then this feature will be missing in D8, so it will be a regression. That's why all new features or bugs have to go to the latest dev first. You can read more about this here: #1050616: Figure out backport workflow from Drupal 8 to Drupal 7. There are also some thoughts about how to make this process better and less painful for developers, so if you are angry about how this process is designed and if you have some better idea how to push and manage patches without introducing regressions, you can comment there.

BTW: If you want to increase the chances to get this into core, you also need to provide tests for your feature.

#29: 924498_drupal_http_request_110313.patch queued for re-testing.
Patch fails on 7.2

The patch for 8.0 in http://drupal.org/node/7881#comment-4406026 works fine for 7.2.

Thanks to jpsoto for his work on the 7.0 patch.

subscribing

#29 does not fulfill Drupal core code style rules. It breaks with many.

After applying the patch for Drupal 7.14 on Ubuntu 12.04 LTS I've got this message:

patch -p0 < 924498_drupal_http_request_101004_0.patch
patching file includes/common.inc
Hunk #1 succeeded at 714 with fuzz 2 (offset -4 lines).
Hunk #2 succeeded at 767 with fuzz 2 (offset -1 lines).
Hunk #3 succeeded at 776 (offset -1 lines).
Hunk #4 succeeded at 866 (offset -1 lines).
Hunk #5 succeeded at 888 (offset -1 lines).
Hunk #6 FAILED at 943.
Hunk #7 succeeded at 1053 (offset 1 line).
Hunk #8 FAILED at 1064.
2 out of 8 hunks FAILED -- saving rejects to file includes/common.inc.rej
patching file sites/default/default.settings.php
Hunk #1 succeeded at 510 (offset 81 lines).

For another patch:

patch -p0 < 924498_drupal_http_request_140111.patch
patching file includes/common.inc
Hunk #1 FAILED at 730.
Hunk #2 succeeded at 774 (offset 6 lines).
Hunk #3 succeeded at 864 (offset 6 lines).
Hunk #4 succeeded at 886 (offset 6 lines).
Hunk #5 FAILED at 934.
Hunk #6 succeeded at 1051 (offset 8 lines).
Hunk #7 FAILED at 1055.
3 out of 7 hunks FAILED -- saving rejects to file includes/common.inc.rej
patching file sites/default/default.settings.php
Hunk #1 succeeded at 510 (offset 66 lines).

I can figure out what went wrong.

@jpsoto can you reroll this patch since #7881: Add support to drupal_http_request() for proxy servers (http not https) is commited? i guess the patch will be less complex and easier to review

I went through the patch proposed in #38 based on D8 & what was committed in Node 7881 (wow that's old!) and the only thing I could find was the position of:

timer_start(__FUNCTION__);

Now considering that the thread went on a lot further than issue #341 in the other queue I suspect it has more to do with that.

So I assume that some things were fixed in the 7.17 release to address the errors popping up in:

http://api.drupal.org/api/drupal/includes!common.inc/function/drupal_htt...

I'm just trying to move this issue along...

#29: 924498_drupal_http_request_110313.patch queued for re-testing.

Well... it wasn't easy to grasp whats going on in the patch....tried to transfer what ii could find for https proxy and CONNECT method..i am pretty sure i forgot somethin

i think this will pass tests

Hello,

I tested the patch in #48 with a basic setup:

drush dl
# applied the patch from #48
cp sites/default/default.settings.php sites/default/settings.php
# Added adequate $conf statements within sites/default/settings.php to declare my outgoing HTTP proxy

and a basic test query:

# simple test
drush php-eval 'print_r(drupal_http_request("https://www.google.fr/"));'
# debug
strace -t -f -e write=3,4,5 -e read=3,4,5 drush php-eval 'print_r(drupal_http_request("https://www.google.fr/"));' 2>&1 | vim -

Unfortunately, I encountered a few issues...
First, a minor one: 'protocol' => 'HTTP/1.0'; I am surprised by the choice of HTTP/1.0 as default protocol -- I tend to consider the only remaining HTTP/1.0 requests nowadays are CONNECT requests while some HTTP/1.1 features such as the Host field are more and more considered as a must-have.

Next, an easily fixed bug when generating the CONNECT request:

$request = $options['method'] . ' ' . $uri['host'] . ':' . $uri['port'];

There is no guarantee that $uri['port'] is filled, so a fallback value (443 sounds right given the goals of this issue) is required:

$request = $options['method'] . ' ' . $uri['host'] . ':' . (strlen(trim($uri['port'])) ? $uri['port'] : 443);

Otherwise, we end up with something like CONNECT hostname: HTTP/1.0, which should lead to a 400 response from most proxies.

Also, the value returned by stream_socket_enable_crypto should be tested, e.g.:

    $crypto_enabled = stream_socket_enable_crypto($fp, TRUE, STREAM_CRYPTO_METHOD_SSLv23_CLIENT);
    if ($crypto_enabled !== TRUE) {
      $result->code = HTTP_REQUEST_TLS_FAILED; // constant to be defined
      $result->error = 'TLS negotiation failed';
      return $result;
    }

Finally, a blocking issue: since the stream is initially opened as pure TCP, stream_socket_enable_crypto() uses the hostname (or IP address) of the outgoing proxy during the TLS negotation with the target HTTPS server and failure ensues -- this is probably done to check whether the common name issued by the server matches the expected one. I tried to play with the SSL context options this way:

    $options_connect = array(
      'headers' => array(),
      'method' => 'CONNECT',
      'protocol' => 'HTTP/1.0',
      'context' => stream_context_create(
        array(
          'ssl' => array(
            'CN_match' => $uri['host'], // attempt to fix the Common name expected during the TLS negotiation
            'verify_peer' => TRUE, // by the way, there is no Curl-like "verify_host" option
            'verify_depth' => 5, 
            'cafile' => '/etc/ssl/certs/cacert.org.pem', // don't blame me, it's a test
            'allow_self_signed' => TRUE, // tolerate self-signed certificates
          )
        )
      ),
    );

... but no luck. The following PHP bug report from 2009 summarizes the problem (see its last comment):
https://bugs.php.net/bug.php?id=47030
... and I am afraid it is blocking the current CONNECT+TLS approach...

I will continue to look for a workaround on my side...

yeap, unfortunately the server i was needing this for does not have stream_socket_enable_crypto
so i ended up using curl for https requests
not beautiful but it does its job.
Changing status per #49

I also tend to think the Curl extension is the way to go. Attached is my first draft (for Drupal 7.17, not tested with real life requests yet) to provide a drupal_curl_request() function that does the same job as drupal_http_request() but does so by relying on curl_* functions. This function features several ways to modify the curl handler before curl_exec() is called; this should enable anyone to apply custom policies (especially regarding TLS negotiation, proxy, authentication, etc.) while being limited only by the Curl API itself (which lacks a curl_getopt() function btw...).
The drupal_http_request() is also modified to delegate HTTPS requests that need to go through the outgoing proxy (and only those ones) to the drupal_curl_request() function. This should avoid breaking requests that already work (i.e. direct HTTP, direct HTTPS and HTTP through proxy) during the test phase.

I intend to provide more feedback tomorrow...

Note: cURL support has been added to D8 via Guzzle #1447736: Adopt Guzzle library to replace drupal_http_request()

yeah, at least we wont have the same problem in d8..
but for d7 i guess drupal_http_request will remain as is..if you need https and you are behind proxy, just use curl and be done with it..

If we are gonna add drupal_curl_request i propose using this every time if curl is available..but i doubt this will happen..

so maybe the solution for d7 would be adding a hook before drupal_http_request starts doing its stuff, to let other module override the method to get external data. and if this alter hook returns data then just return those data..else let drupal_http_request do its stuff

I would re-open this issue as D7 if you wish to go that route #64866: Pluggable architecture for drupal_http_request()

Side note is I've opened up an issue for HTTPRL to support https proxies as best as I can (no verify peer support) #1867088: Add in support for https proxy

Thanks for redirecting me on this issue:)
I am not going to work on this, unless i get to work once more for a server behind a proxy.
But imo, we definitely need a drupal_http_request() hook as a D7 only solution.
So closing this issue and reopening #64866: Pluggable architecture for drupal_http_request() for 7.x-dev seems to me the best way to act

Something else of interest if you wish to go this route
http://drupal.org/project/chr
http://drupal.org/sandbox/daeron/1490922
from #1447736-11: Adopt Guzzle library to replace drupal_http_request()

Hello,

First, thank you for your feedback.

Indeed, the best way to address this issue would be to make drupal_http_request pluggable for Drupal 7.x.

I had found some time to continue working on my draft to make its return value closer to drupal_http_request()'s, but some tests are still failing due to the delegation of the "follow location" feature to curl.

However, I will not work anymore on that patch since the chr module (which I discovered thanks to #56) already does the job; I attached drupal_curl_request.seconddraft.diff for information and completeness.
I now intend to solve the issue on my Drupal instances by relying on the chr module + a trivial patch to drupal_http_request:

--- drupal-7.17.orig/includes/common.inc	2012-12-10 17:52:21.000000000 +0100
+++ drupal-7.17/includes/common.inc	2012-12-20 10:23:37.000000000 +0100
@@ -785,6 +785,9 @@
  *   - data: A string containing the response body that was received.
  */
 function drupal_http_request($url, array $options = array()) {
+  if (function_exists('curl_http_request')) {
+    return curl_http_request($url, $options);
+  }
   $result = new stdClass();
 
   // Parse the URL and make sure we can handle the schema.

See also #1664784: drupal_http_request() has problems, so allow it to be overridden.

Hi, all,
I came across this as I was trying to find solutions to the error messages I get as I try to download modules via admin/modules/update. Ultimately this boils down to how drupal_http_request in common.inc works. I am behind a proxy and obviously the download is from the drupal site in under SSL.

HTTP error 501 occurred when trying to fetch https://ftp.drupal.org/files/projects/abc_module.tar.gz.
Downloading updates failed:
Failed to download abc_module from https://ftp.drupal.org/files/projects/abc_module.tar.gz

In order to concentrate on the bits where it opens a socket using proxy and send a request, I have created a stand alone PHP file which will access a non-SSL site (http) using proxy to test if my site location actually talks to out side via proxy:

$sock = stream_socket_client('tcp://proxy.mycomp.com:80', $errno, $errstr, 30, STREAM_CLIENT_CONNECT);
if(!$sock) exit();

$request = array();
$request[] = "GET http://php.net/manual/en/function.fgets.php HTTP/1.1';  
$request[] = 'Host: php.net';  
$request[] = 'User-Agent: TestClient';
if(!fwrite($sock, implode("\r\n", $request)."\r\n\r\n")){
  exit();
}
$response = '';
while(!feof($sock)){
  $response .= fgets($sock, 4096);
}
echo $response;
fclose($sock);

Which echos the page content fine.

Then I have tried to access SSL site (https://ftp.drupal.org/files/pictures/Drupalcon%20SF%202010%20Group%20Ph...) behind proxy. I chose the image rather than a tar file as I thought it is easier to display on a browser:

$sock = stream_socket_client('tcp://proxy.mycomp.com:80', $errno, $errstr, 30, STREAM_CLIENT_CONNECT);  
if(!$sock) exit();

$request = array();
$request[] = 'CONNECT ftp.drupal.org:443 HTTP/1.1';
$request[] = 'Host: ftp.drupal.org';
$request[] = 'User-Agent: TestClient';
$request[] = "Proxy-Connection: keep-alive\r\n";
if(!fwrite($sock, implode("\r\n", $request)."\r\n\r\n")){
  exit();
}
$response01 = '';
while(!feof($sock)){
  $response01 .= fgets($sock, 4096);
  if(preg_match('/^http/i', $response01)) break;
}
# Some of the crypto methods:
# STREAM_CRYPTO_METHOD_TLS_CLIENT, STREAM_CRYPTO_METHOD_SSLv3_CLIENT, 
# STREAM_CRYPTO_METHOD_SSLv23_CLIENT, STREAM_CRYPTO_METHOD_SSLv2_CLIENT 
stream_socket_enable_crypto($sock, true, STREAM_CRYPTO_METHOD_SSLv2_CLIENT);

$request = array();
$request[] = 'GET https://ftp.drupal.org/files/pictures/Drupalcon%20SF%202010%20Group%20Photo%20-%20Seated.jpg HTTP/1.1';
$request[] = 'Host: ftp.drupal.org';
$request[] = "Connection: keep-alive";
$request[] = 'User-Agent: TestClient';

if(!fwrite($sock, implode("\r\n", $request)."\r\n\r\n")){
  exit();
}
$response02 = '';
while(!feof($sock)){
  $response02 .= fgets($sock, 4096);
}

echo $response02;
fclose($sock);

In above it completes the routine and returns me something like "X-Bst-Request-Id: F74l2h:9pqc:24831" on echo. I do not know well enough why it returns X-Bst-Request-Id rather than the actual content, which is what I want. I just assumed this at least did the handshake ok.

From this excercise, I can say this much that the current drupal_http_request in common.inc does not deal with access to https sites behind proxy in the way the second set of the code deal with. Seeing the thread here, I can see that you are tackling this using CURL and last updated was two years ago. Has this seen the conclusion else where or has it been suspended? And most importantly, is there any modules or solutions which resolve the problem comprehensively? Or #43 is as good as it gets for the moment? I am trying to decide the direction as to where I go from here and the conclusion here if any is a concern to me. Your comments here to my question will be much appreciated.

My environment is Drupal 7.50 and on OpenSuse with Postgres and PHP 7. Proxy server runs on squid. I am making an intranet site.

Hi, I'm actually working on a Drupal 7.x environment with a proxy, and several requests in Drupal are made through SSL, so updates checking or some web services are not working. I am always getting Bad Gateway (502) errors, although the request is properly done in a non-proxied environment.

Please, consider patching this method because this is extremely important in site building.

Thanks.