Add support to drupal_http_request() for proxy servers

Did this ever get taken care of in 4.5.2. I can't seem to get any feed to work from behind a firewall. Getting error....

Failed to parse RSS feed Microsoft Security Info: 10060 A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond..

At home this same feed works fine, so I know it's not the feed.

Thanks,

SlackNet

+1 on having a facility to modify proxy settings...

here's a patch of /includes/common.inc for drupal-4.4.2 version

+1 on having a facility to modify proxy settings...

here's a patch of /modules/system.module for drupal-4.4.2 version

patch for /includes/common.inc 4.5.2 version to facilitate rss feeds behind firewall

patch of /modules/system.module 4.5.2 version to set proxy settings

hi, tso small issues. one is described at: http://drupal.org/node/9706
The other is that the code is not "drupal compliant". Drupal developrs are (luckily) very picky about spaces, indentation, etc.

i forgot to remove my hardcoded proxy name.. sorry

Ber, which is not drupal compliant?

1) New features only get added to CVS/HEAD, not to the 4.5.x branch.

2) 'Blind' variables that cannot be changed through an admin interface are not a good idea.

3) Your proxy codepath ignores the $defaults array completely. At this point in the code, this doesn't just contain default headers but also any extra headers passed to drupal_http_request(). This is not good as they are simply not used anymore.

4) Code style:
//use proxy settings
There should be a space between '//' and text. MInd capitalization and punctuation.

$request = "$method ".$uri['scheme']."://".$uri['host'].$path." HTTP/1.1\r\n";
Rules for string concatenation: never a space between . and a quote, always a space otherwise.

Tab usage: we never use tabs due to editor differences. Your patch contains several.

String quote usage: we prefer single quotes in Drupal to double quotes, except in the case where single quotes would be too hard to read (e.g. backslash escapes should be avoided as well).

In a very tight setup, users must authenticate to access the proxy, to prevent unauthorised usage, or to track authorised usage.

Can the proxy support also be modified to allow username/password to be supplied to the proxy?

Thanks Steven.

I hope this patch works and be accepted... This will allow proxy variables to be modified using the admin interface.

compliant patch for proxy suport, thanks Steven.

-1 on this. Please do not introduce more config options. IT makes no sense, since you only set that option once.

Also: you solve a single case of fopen, please look for an overal solution that will solve all drupa fopens, rather than hack into the feed code to fix that specific issue.

* either make the code smarter so it can detect proxy errors (http://drupal.org/node/9706) OR
* add this to conf.php

I'd also like to avoid YAS (yet another setting). Can't we include proxies settings.php?

I tried to apply the patch and from the fsockopen(): unable to connect to error i get the Warning: Unknown: A session is active. You cannot change the session module's ini settings at this time. in Unknown on line 0 error.
I am on the version v4.6.3 on windows with
php
PHP 5.0.5 (cli) (built: Sep 22 2005 10:49:43)
Copyright (c) 1997-2004 The PHP Group
Zend Engine v2.0.5, Copyright (c) 1998-2004 Zend Technologies
with Zend Extension Manager v1.0.8, Copyright (c) 2003-2005, by Zend Technol
ogies
with Zend Optimizer v2.5.10, Copyright (c) 1998-2005, by Zend Technologies
apache 2.0.55 for windows and mysql as backend.
Any help or pointers in the right direction much appreciated.
my commons.inc file version is 1.443.3.10
Regards,
Rajesh

As dlr noted before (http://drupal.org/node/34340), wouldn't it be cleaner to add the proxy support in the core system? Have I missunderstood, but don't I have to patch (http://drupal.org/node/7881) common.inc every time after an update? If it were possible to define the proxy-configuration as an option, it wouldn't be neccesery to do the patch all over again after each update.

I bet there are lots of other corporate users (in addition to dlr and me) behind firewall that aren't able to get aggregated content without proxy support.

I would also love to see the possibility to configure a proxy server as part of the drupal distribution.

Until that happens I have created a patch with the "won't make it into a proper release" hacks from above that applies cleanly to 4.7.2.

1. Features only go to HEAD
2. We must avoid the cluttering of the "admin" settings page for "system" variables like this one, that are defined only once. (Perhaps we should create a special page with all variables in the table, if someone does not like to play with settings.php)

Proxy server should be a core based setting. For instance, here at Cingular Wireless all the servers are sitting behind a proxy server as well as the users. So anything that is hosted on this network is forced to access external links through a proxy server.

I need this functionality too. I have just finished dealing with support at my hosting company that have implemented a proxy server for security. This is only going to become a bigger issue as more hosting companies implement similar techniques to increase security of thier servers. Please, can we have a generalised solution - and may it not take 2 years to implement? Unfortunately, I'm not a coder so can't do it myself. I agree that it needs to be generalised not just for RSS feeds, but for any external communication by Drupal.

Made the changes described to a fresh install of Drupal 5.1 and it worked like a charm. Thanks ;-)

Would be better if it was just a config settings tho.

@#24:

What did you do to get it working with 5.1? I've tried it with my fresh install here and it doesn't work, it fails with the following error:
The feed from BBC News: Education seems to be broken, due to " ".

I've attached the changes I made to my common.inc file.

To get drupal_http_request() working with a proxy server which requires authentication, you need to add the username and password to the request, like this:

+1 to this feature.

One of the greatest pushes for Drupal that I've seen has been behind proxies, using Drupal as a knowledge base or as an intranet. Some form of proxy awareness would be a godsend. In my instance, I am currently behind a NTLM proxy (which makes life real interesting on Linux) although I've managed to get NTLMaps set up so apt-get (etc, etc) can talk to the outside world. Some method to allow proxy details to be passed to Drupal (whether it's a pointer to the NTLMaps application, or to the main proxy itself) would allow us to restore a serious slab of functionally back to the installation.

I might also point out, that with Drupal6, I now have a permanent alert on my admin page, saying "One or more problems were detected with your Drupal installation. Check the status report for more information." ... why?

"Update notifications are not enabled. It is highly recommended that you enable the update status module from the module administration page in order to stay up-to-date on new releases. For more information please read the Update status handbook page."

So now, it's a security issue.

I feel that this issue is starting to grow in significance.

All issues are now fixed in 7.x, then backported if applicable.

I've attached a patch to common.inc to make drupal_http_request work through a proxy. I've allows the proxy host, port, username and password to be specified.

I don't have a proxy that required authentication so authentication has not been fully tested.

This is against Drupal CVS HEAD.

Perhaps, instead of amending drupal_http_request(), we should make it plugable/replaceable?

We already do that with drupal_mail().

Every once in a while somebody discovers another deficiency in drupal_http_request() and suggests to improve it. I myself had to forgo drupal_http_request in a certain project, in favor of CURL, because I discovered it doesn't behave well in strained environments.

+1 on proxy settings.

I'd really like a place to edit in common.inc or through the Drupal interface.

I've been trying to demonstrate Drupal 6.0 inside the corporate network. We have a proxy firewall, so I'm unable to complete the setup with OpenID, Update Status, and RSS. I need to specify (unauthenticated) proxy access to [ip address]:8080.

Check out the post from agentrickard. He provided a link to an fsockopen example

Since the Update Status module is now in the core, and Drupal nags if it isn't on, I do not consider this a feature request. This is a bug. OpenID is now in the core and cannot work for many Intranet sites without an area to specify a proxy. RSS does work under the same environment as well. These individual module developers should not have to create their own proxy module, either.

All changes get made to 7.x first.

Added configuration options to the UI under Site Information. Probably needs to be tweaked some for wording and such.

Probably be changing this tomorrow so that it adds a new link called Proxy Settings rather than throwing under Site Information, but it's a start.

okay, i know nada about php, but i found this commentary in a discussion of this specific issue from a web host blocking http requests for security reasons and they offer these ideas:

1) Using the PHP environment variable $_SERVER['DOCUMENT_ROOT'], which returns the absolute path to the web root directory.
2) cURL is another method that could be used.

would option 1 be possible in the current common.inc file as a manual edit? and if so, how?

I've added changed the patch in 39 to include:
* Validation of the proxy port to be a number and in the range of valid tcp ports
* Use a valid port by default in the settings page
* Fix a bug in my original patch with the setting of $auth_string.

@zxombie good catch.

Opinions on creating a new menu item under Site Configuration called "Proxy Settings"? Also, I'm thinking there should be an on/off option.

@zilla the fsockopen method used in this patch is much better than either of those two alternatives.

I'm thinking we should also allow the option of a persistent connection (via a checkbox), and use pfsockopen rather than fsockopen. On a dedicated server, it would make sense to use a persistent connection, which is probably the majority of sites that are stuck inside of an intranet.

@boydjd, I've updated the patch to create a "Proxy Server" page that allows setting the 4 proxy settings. I've added to the description a note that when the proxy host name is empty it will use a direct connection to the internet.

I've applied this patch in a 6.1 test site on my school webserver which is behind a firewall requiring proxy authentication. There were no errors reported.

The Proxy Settings form appears in Administer-->Site Configuration-->Proxy Server and accepts input.

But I'm still getting "HTTP request status Fails" from Status Report and the Aggregator module reports "" The server can't issue HTTP requests".

I appreciate the work that has gone into this, but I am also amazed that this is even still an issue. There must be countless educational/corporate intranet/webservers that must authenticate through an upstream proxy service.

As someone observed, this is not a feature request, it is a design flaw.

I hope this can be resolved sometime soon and built into core. 6.2 perhaps?

Cheers

@Slim Pickens, what method of authentication does the proxy server expect? The patch only implements basic authentication.

If you need this environment to support NTLM (which continues to be the bane of many of our existence), you will need to potentially review a solution that uses http://ntlmaps.sourceforge.net/ ... I use this on my Ubuntu machine at work to handle the NTLM Proxy so apt-get (etc) works.

#53 zxombie - the proxy server expects a username and password. I thought the common.inc patch was creating that capability?

Hey guys -- I haven't tested this yet, but it looks reasonable to me. A couple of suggestions:

1. We don't capitalize each word in a sentence or title. Only the first word gets a capital letter. So things like 'Forward Proxy Settings' should become 'Forward proxy settings'.

2. Proxy settings are quite technical. It would be good if there was a little bit of help on the settings page. If my mom reads it, she should probably be able to determine that this is not a page she should make changes to.

3. What is an 'empty Drupal site'?

4. Add an entry to CHANGELOG.txt please.

Given some additional polish, it should be good to go.

I've re-rolled the patch so that it applies cleanly against head. Working on implementing Dries' changes.

Thanks, it works!

'Forward Proxy Settings' should be 'Forward proxy settings'.

'the Internet' should be 'the internet'.

'Proxy Server' should be 'Proxy server'.

Etc.

Actually "Internet" is a proper noun, and should therefore be capitalized.

I'm attaching a tarball of a module + core patch which enables the use of a http proxy in drupal 5.7. It's based almost entirely on proxy_11.patch at #57 above.

One exception though, TCP port 0 is reserved so I don't allow it.

Thank you for the module/core patch.... does this allow for the option to have only secure https requests to go through a proxy server? That feature would be awesome as it would allow Drupal to run on shared hosting plans (such as GoDaddy). GoDaddy requires https calls to run through s proxy server but not http.

Edit: While my initial post referenced the Drupal 5 patch, I think it would be of great value to separate the http and https calls through proxy options in Drupal 7 for the same reason. Some hosts require only secure https calls to be done through a proxy server.

I edited my files with the proxy_11.patch above for version 6.2. It is currently working and solved my problems completely...THANK YOU to everyone who worked on this and provided this solution.

I have attached my common.inc, system.admin.inc, and system.module patched files in a zip for those who are unable to patch their files. Hopefully, this will work for you as well.

Thanks again.

Any system that does not support NTLM/Digest authentication is just not ready for the Enterprise! Proxies have been around for along time. How come Mozilla Firefox gets it right?

Suggestion for Proxies that require Digest/NTLM authentication.
If the proxy you are authenticating against requires than try NTLMAPS from sourceforge.
NTLMAPS works in Linux as well as Windows or any OS for that matter as it require Python.
I have used it in Ubuntu and Windows XP and it works well.

This is not a support forum. This needs to go into D7 first.

Subscribing....

Hi all,

I'm new here so sorry if I tread on toes - just wanted to note that the patch in #57 *almost* works behind our basic-authentication proxy at the University of Newcastle for Drupal 6.2. Common.inc is not quite right though and needs to be modified slightly as given in the attached patch. I now have proper update and RSS functionality through the proxy as far as I can tell, though more testing is obviously required.

I also can't confirm that this works for 7.x-dev as I'm not actively involved with HEAD.

Very cool, I'd love to test it, but I haven't encountered the need of a proxy server. New features have to go into Drupal 7, and then back ported to Drupal 6. It's wonderful to get as many patches and tests out there as possible though, so thanks! If I get the chance, I'll try porting this to HEAD.

@boydjd:

I think I found following problem with the patch from your comment#57: your modification of drupal_http_request would by itself work fine in my site (which is behing an ISA proxy with basic authentification) if there wasn't the call for system_check_http_request at the beginning. Here Drupal checks if it is able to load a local page per HTTP request. The relative url for this check is 'admin/reports/request-test' and gets completed to an absolute url (like 'http://www.example.com/admin/reports/request-test' if 'http://www.example.com' is the url of the Drupal Site). This check is now done over the modified drupal_http_request function and it fails in my site because it tries to make the request over the proxy I configured before in the new proxy settings. And because this check fails, drupal_http_request returns with an error and does not try further requests.

So the self_test (system_check_http_request) has to be a direct request on my site and I think on every site behind a proxy to be successful.

To achieve this behaviour I added the parameter No proxy for to the proxy settings. It works like the same parameter in Firefox. If you define something like 'example.com' as "No proxy for", then every request to a site which has the string 'example.com' in it's url is requestet directly, not over the proxy.

With the additions in the patch (against Drupal 7 head) I provide here I could use update status and the feed aggregator behind the proxy.

Hi asilva,

here is the patch from comment#83 above for D6.3 and D6.4. It works for my site behind an ISA proxy server with authentification. On another drupal site I had still troubles with this patch. There I added some code to switch off Drupal's HTTP self test to make the patch work. Tell me if the patch here works for you. If it doesn't I will provide the other patch with the disabled HTTP self test.

To apply the patch for D6.3 please copy the file drupal-6.3-proxy.patch into your Drupal base directory and apply it with the command: patch -p0 < drupal-6.3-proxy.patch.

The 2nd file drupal-6.4-proxy.patch is the same patch as patch file for D6.4

Four things in response to the patch at #83:

1. I'm not quite sure how I feel about a is_in_no_proxy_list() function as it isn't really that descriptive. drupal_in_proxy_list() maybe?
2. I'm also not sure how I feel about foreaching through the list whenever the drupal_http_request function is called, but can't really think of a cleaner alternative.
3. $output .= '<li>'. t('basic configuration options for .... The space before and after the period is part of Drupal 7 coding standards ;-) .
4. Is there anywhere else that the proxy settings could live? They seem pretty small for just one page.

Hi Rob and thanks for your response.

I'm not quite sure how I feel about a is_in_no_proxy_list() function as it isn't really that descriptive. drupal_in_proxy_list() maybe?

I suggest to rename the function is_in_no_proxy_list() to drupal_is_proxy_exception(). But I'm not sure if it's good to use the suffix drupal_ for this internal helper function. Wouldn't it be better to name it _is_proxy_exception() instead?

2. I'm also not sure how I feel about foreaching through the list whenever the drupal_http_request function is called, but can't really think of a cleaner alternative.

I agree with your concern. Maybe it would be a better alternative to use an additional variable proxy_enabled which would be represented by a new checkbox labeled "Use proxy server" in the proxy settings page? Then the foreaching through the list would only be necessary when "Use proxy server" is checked.

3. $output .= '<li>'. t('basic configuration options for .... The space before and after the period is part of Drupal 7 coding standards ;-) .

Shame on me, I used ancient coding convention for string concatenations (Drupal 4 I think...). I'll adjust the syntax in my next patch file.

4. Is there anywhere else that the proxy settings could live? They seem pretty small for just one page.

You are right, but looking at D7's site configuration menu I find some other small settings pages like "Clean URLs", "File system", "Image toolkit" or "Error reporting". On the other hand I don't feel that the proxy settings would match with any of them from a topical point of view. So I suggest to keep the proxy settings on their own page for now. Maybe in the future someone will collect all those small pages into one or more bigger pages?

Please tell me what you think of my suggestions, so I will be able to modify my patch.

Hi rizamp,

here is a patch for D6.4 with the disabled self test. But in the meantime I found a better replacement for the problem I had on one of my sites behind a proxy server. I had to enable the reverse_proxy setting in settings.php:

With this setting the http self test workes fine for me. please try to enable the reverse_proxy setting, before you apply the new patch and disable the http self test (by checking the checkbox "Skip HTTP self test" at the end of the proxy settings page).

If the proxy patch still does not work on your site and your proxy server regards authentification: be sure that your proxy server accepts "basic authentification method". If he does not allow basic authentification but only allows other methods like Digest, Radius or NTLM the proxy patch won't work. It works only for basic authentification (or if no proxy authentification is required at all).

Another thing we have to think about is testing. Is there anyway to create an effective SimpleTest for this for self-testing?

• Removed the is_in_no_proxy_list entirely for a stripos call
• Fixed some strings that didn't seem that helpful
• Added hook_help text
• Fixed some of the minor logic which looked kind of weird (variable scope issues)
• Replaced != '' with calls to empty()
• Removed the fieldset in the settings, following what they have for Clean URLs, and other settings with just straight textfields
• Logic in the validation wasn't optimal

To test:

1. Apply patch, clear cache
2. Visit admin/settings/proxy and enter your proxy settings
3. Make a PHP call using drupal_http_request

... And I'd consider this a feature.

Hi,

This is interesting feature, however I believe the word "proxy" used here might be confusing with different kinds of proxies. For instance, default/settings.php use "reverse_proxy" variables for something different.

I would suggest to be a bit more explicit and use "forward_proxy" or something similar as prefix for variables, etc.

Hi Rob,

your fixes look good and make the code cleaner. But after applying your patch from comment#91 I get following error when I access my site:

The reason for this is that empty() only checks variables as anything else will result in a parse error. In other words, the following will not work: empty(trim($name)) or - like it is done in common.inc - !empty(variable_get('proxy_server', '')) and !empty(variable_get('proxy_username', '')).

Hi subru77, unfortunately this can't be implemented through a module because it requires modifying the drupal_http_request function. We could implement a hook in drupal_http_request to expose how HTTP requests are made. Then we could create add on module that could make different kinds of requests, like HTTP requests, HTTPS requestions, FTP requests, etc. But that would have to be thought about, and still wouldn't work if we required a proxy in from of the request..... Thoughts?

Thanks for taking a look at it, brahms. Is that the only issue you saw with it?

Hi Rob,

I'm not shure if the two places in code with empty(variable_get()) are the only issues. I just tested the code (with fixes for the empty issue) on my site behind an ISA proxy. First it didn't work because the variable drupal_http_request_fails had been set to 1 before. This caused a call to the self test (function system_check_http_request) and the self test always seems to fail on my site. The reason for this may be the special network configuration I have to use to reach my customer's site. I am connecting to the web server over an SSH tunnel and there is a network address translation configured between the client I use and the web server.

If I set the variable system_check_http_request) to 0 or delete the variable everything works fine. Starting the update status check I am able to connect from my site to the drupal.org update server.

So I think the patch itself works fine. But I am not sure if Drupal's http self test is working from behind a proxy server. (This is why I tried a piece of code which turns of the self test as I provided in my comment#89) In the next days I will visit my customer with the affected Drupal site. There I'll try to check the self test in a network environment without the SSH tunnel and without the NAT.

EDIT:
I just returned from a visit at my customer. I had enough time there to test the code and everything worked fine including the http self test. For me - and much more: for Drupal - it would make sense if your code from comment#91 will make it's way into core after the empty issue has been fixed.

Yep, the !empty() is broken. This patch looks close though. :)

A workaround for those not comfortable patching Drupal is to use cntlm (http://cntlm.sourceforge.net/) and then using apache to rewrite the URL to point to the cntlm proxy. You then need to alter the URL of what you are looking for to point to your local box. I have posted an example of configuring update module to authenticate with an IIS proxy at http://drupal.org/node/172708#comment-1003511.

subscribing

same as #91 backported to Drupal 6.4

• corrects the diff context respect Drupal 6.4
• corrects the empty calls to variables
• adds one line to make a callback be valid

I applied the patch provided in post #102 to a fresh drupal-6.4 installation without issue (thank you for that). After configuring my proxy server I still can't check for updates. I am now getting the HTTP Request Self-Test error. I followed the instructions in post #89 to enable reverse_proxy, that didn't work for me either. I'm going to attempted to patch drupal with the patch from post #89 (drupal-6.4-proxy-skip_selftest.patch) but I have to mess with it first. In the mean time can someone answer these questions:

- Since I'm getting the Self-Test error should my update checks still be failing?

- Does someone have another solution to the Self-Test problem.

- It seems like most people applied the patch (#91 or #102) like I did and were able to get updates. Is this true or did the rest of you have to do something more to get it to work? The reason I ask is I have been hitting my head on this for 2 weeks and can't get it to work.

Thanks

I'm having second thoughts on this. Instead of adding this straight into core, would we be able to create some kind of hook that would manage altering the request so that if we wanted a proxy contrib module to handle this, it could?

Anyway, here's a patch with the !empty() fix along with an added user permission (administer proxy configuration) because you don't want to give your administrator user team permission to see your proxy server password......

is this for D5/6/7?

I'm having second thoughts on this. Instead of adding this straight into core, would we be able to create some kind of hook that would manage altering the request so that if we wanted a proxy contrib module to handle this, it could?

+1 of course!!!

I'm having second thoughts on this. Instead of adding this straight into core, would we be able to create some kind of hook that would manage altering the request so that if we wanted a proxy contrib module to handle this, it could?

+1

I suggested this in comment #35. On the other hand, I see people are feeling very strongly that this proxy thing should be in core.

====

It will be interesting to see how this plug/hook mechanism will be implemented. It will require only few lines of code, but nevertheless we should "standardize" the way we "plug" custom code into Drupal. I'll have to do the same for format_date() (to enhance i18n support). drupal_mail() too has a "plug" mechanism and it could be replaced with the one we finally settle on.

about the mail there is some work already done http://drupal.org/project/smtp, but it doesn't seems to be proxy aware

I'd prefer to have this in core, but I can be convinced (as always).

If someone can confirm that this solution works, I'm ready to commit it to core.

We should probably add a CHANGELOG.txt entry though.

I can confirm that the most recent patch from Rob in comment#104 works in following environments:

• Drupal 6.4 Site behind a proxy server without authentification
• Drupal 6.4 Site behind a proxy server which allows basic authentification

(I tested those Drupal 6.4 Sites with a back port of the patch, because I didn't have the chance to install Drupal 7 in those environments. I include this backported patch here as g-zipped file for Drupal 6.4).

Though there are proxy configurations where the solution will not work (if there is another authentification method than basic authentification required) I think having the given solution here in core is an important step for bringing Drupal to enterprise environments. Nevertheless we should continue to work on solutions for the other authentification methods, but this should go into a new issue.

I tested #91 backported to D6 in #102 on Drupal 6.4 behind a proxy server without authentification

I vote for inclusion in core. I work in public education. Our State department has more than 1600 schools - all of them are obliged to use a proxy server for internet access. The other 5 Australian States education departments would operate similarly.

Surely a proxy server is the norm for corporate environments? Proxy setup should be a part of the site configuration through core. If you don't need it, don't configure it. We're talking a few lines of code here. No biggie.

Cheers

Include in core.

A major wish and focus of mine is to get Drupal better recognised by large corporations and government. This is not going to happen without "thinking corporate" right out of the box.

The patch at #104 still applies.

Did someone test #104?

On a quick code style and string review, there are several concatenation operators that aren't set off with spaces, plus several code comments not ending in a period. Also, one of the things we need to clean up (in another patch) is the variety of ways we show examples. This patch uses "eg.". Elsewhere in code we have "e.g.," and "Example: xxxx" and "Example: xxx". As long as this patch is internally consistent and hits one of the existing styles it should be fine, especially since there are other issues in the queue to conform all these to a single standard.

What about:

The host name of the proxy server. Leave empty if your site has an existing connection to the Internet or does not use a proxy server. Example: localhost.

The password used to connect to the proxy server, if necessary. Warning: This password is stored as plain text.

"comma seperated" should be "comma-separated".

These could just be "Configure proxy server settings."

I can roll some of these changes into a new patch tomorrow if nobody beats me to it.

Patch #104 modified according comment #117.

Doesn't seem to apply anymore for me ...

The attached patch works on my installation of Drupal 6.5. Please review. Looks like 2 hrs, 33 minutes. A new record! Hurraaahhh!

patch #123 patch without errors for my 6.5, but in status report HTTP request status fails and check manually Drupal core update status also fails. Proxy settings are correct without auth.

patch #123 removes slowing down from update-status module behind a proxy.

Has someone other infos?

@Gos77:

Just some things you can try to make the proxy settings work on your site:

reverse proxy settings

I had to enable the reverse_proxy setting in settings.php to make HTTP connections work on my site.

Drupals HTTP Self Test and the variable drupal_http_request_fails

When the HTTP self test in Drupal fails, Drupal sets the variable drupal_http_request_fails to 1. This is the intended Drupal behaviour and it was the case on my site before I applied the proxy patch. Then I applied the patch but though the proxy patch itself would work, update status still failed because the self test did not reset the variable to 0. I had to reset the variable by SQL (setting the column value to b:0; in the variable table) to make the proxy connection work. An alternative way is to delete the variable drupal_http_request_fails from the table. (This is no bad hack because Drupal itself writes this variable into the table once the self test fails. If Drupal's self test on your site never had a problem, there is also no row for the variable drupal_http_request_fails in the table.)

As in #125 me tends to add the remark to think about stuff like this: if the self-test request fails, drupals internal logic will ZAP any further http request. Adding a proxy in this whole drupal-http logic is not only imlementing it the technical way. The whole dataflow needs to be thought of. I would love to see proxies out of drupal therefore, it makes configuration too complex. A reverse proxy transparently configured somewhere else can do the job as well.

Instead of this, an own connection and transport class for php5 based druapl version(s) might lead to a more and propper way to solve such things. in this case, you could switch to a simple http proxy transport class instead of the default http one.

works for my blank 6.5 with patch from #123 and hint #125 .. thx

Just reviewd the patch. Looks OK for me so far (6.5 patch). Right now I still have no Idea to solve the problem with a sleaner integration if the http self test fails. I had self test fails with DNS problems but they recovered automatically. Why doesn't this happen here. Any Idea brahms, Gos77?

subscribe

This is a feature request, and as such, goes to HEAD, not 6.x.

Looks like the patch doesn't apply to HEAD cleanly anymore, I'll re-roll when I find a spare moment. :)

Changing to CNW, since it doesn't apply anymore.

No, new features go only against the dev version, regardless of how long they've been requested. Sometimes people maintain backport patches of them, but they do not go into core. Of course, this has to get committed to Drupal 7 first.

Why not use cURL instead of implementing it ourselves? As a bonus, we'll get its 'timeout' feature (it's good for Aggregator).

For Drupal 7, we could define a Drupal::HTTP interface. The default implementation will be the current one. A 'cURL' module, when enabled, will plug in its own implementation.

without getting into the politics of whether or not to backport here is my patch for 6.6 - it works for me, others may find it useful as i found there patches extremely helpful.

Now I am back at work I have attached a patch file for Drupal 6.8 - I hope this helps anyone who needs it.

Nitin - have you tried anything else that requires proxy such as checking for module updates - does that work?

There have been some changes in 6.9 that affect the patch. Try this version, it works for me.

I can confirm the 6.9 proxy patch to work fine with D6.9. Thank you silid!

If we got this into Drupal 7, a patch for Drupal 6 would be very easy. As a side note, #64866: Pluggable architecture for drupal_http_request() would be helpful to stick proxy support in contrib.

Hi everyone!

The patch outlined in the original post didn't work on our proxy. We needed to leave this bit that the author advised to comment out:
$request .= implode("\r\n", $defaults);
With this line commente dproxy complained about missing content-length header. With the line in it works.
Cheers,
Michal

Subscribing

Here is a patch for 6.10

I haven't tested this so I don't know that it works. Please give feedback

patch from #158 working fine for me for D6.10

only getting (Stripping trailing CRs from patch.) for all three patched files while patching.

Feature additions do not get committed to stable branches, unless it is necessary to close a security hole. Getting this into Drupal 7, however, is something that really does need to happen. Can we focus on that? (Even if it were to go into D6, it won't until it's landed in D7. Focus your efforts there.)

Crell, I know that this is the rule - but there is no rule without exceptions. For example, see http://drupal.org/node/243524 and http://drupal.org/node/276174 that now went into 6.10 without being a security fix.

This issue is in fact a security fix because the lack of proxy support breaks the Update status module on servers behind a proxy, with the consequence that it no longer warns you about essential security updates. It's not an option that some may want to enable because it's a fancy feature - it's essential to work behind a proxy. Without proxy support, all those servers are locked away from using Drupal 6 at all. So I don't see this patch as a new feature, but rather as a critical bug fix for the Update status, Aggregator and other modules to work.

The D6 patch is working excellent - what's so hard in porting it to D7?

This patch has been made

The last submitted patch failed testing.

Missing the is_in_no_proxy_list() function?

Patch for 6.10 works great for me for http requests. But it appears that https requests are not proxified so any module that uses https, such as OpenID, does not work - request times out.

Hmm. That appears to be true. I must say that I have never needed to access https by proxy and so didn't realise, but looking at the patch it doesn't do anything to https requests at all.

So in other words, it isn't broken it just hasn't ever done it. I will have a look at the patch when I get time but feel free to submit your suggestion.

Si

I applied the 6.11 patch in #176 and I have the 'available updates" report working for the very first time on a PC and Drupal servers in our company. Thanks for all the great work.

However our team is debating whether we should start the precedence of patching the core. If we do, when we upgrade to future 6.x releases, we will have to keep patching (either relying on someone releasing a patch or we create the patch ourselves).

Proxy support is essential in a corporate environment. D7 supports proxy but it is still a long way away.

Not being able to check "available updates" is a security risk. Including this patch into the next 6.12 release as a security fix will make the life of many corporate Drupal users easier and is another step towards getting Drupal wider acceptance as an enterprise CMS.

Thanks.

This is 6.11 patch updated for 6.12.

Attached file contains the three files to add proxy support to D 6.12

I am closing this issue and deleting every "I tried to patch" comment from it and then reopen.

Please focus on getting a working patch for HEAD (http://drupal.org/node/320). Also, please refrain from posting questions regarding 'how to apply patches' and 'why this is not in Drupal N". Those who do will be rewarded with a two week ban.

I've done my best to ignore the horrible // PROXY-HACK comments, so here is a review of the patch in #181.

Drupal doesn't use value-as-first-component comparisons. You can simply rewrite that as $proxy_not_required && ($proxy_server = variable_get('proxy_server', ''))

No support for HTTPS (via the CONNECT proxy command)? It seems like this could be useful in several use cases I can think about.

This indentation is wrong, and same remark as above for the condition.

Same remark.

The function is poorly named and it is not documented.

The condition (that uses type-agnostic comparison on the result of strstr()) is severely broken: it would allow only partial matches that do not start at the first characters (ocalhost will match localhost, but localhost will not match localhost).

We probably want this match to be either on the hostname or on the IP address. This is not possible right now.

This form should use the new autodefault feature of system_settings_form() in D7.

$form_state['values']['proxy_server'] != '' should be strlen($form_state['values']['proxy_server']) > 0, or "0" will not match. The two error messages should be collapsed in one.

Could anyone give me a hint where I have to enter the settings of the proxy server (IP, Port, maybe authentication) with proxy6.12.patch? I searched the issue with no idea on that...

Administer -> Site configuration -> Proxy Server (admin/settings/proxy)

Sorry to ask here, but does the 6.x patch also work with Drupal 6.13?

Hi roball .. patch from #181 works fine for my 6.13 installation.

As requested here is a patch for 6.13 - I haven't reviewed the suggestions or made any amendments to the 6.11 patch I last uploaded except to apply it to 6.13.

As mentioned previously I haven't written the code included here but have produced the patch from other code submitted in the thread to help other users.

The 'horrible // PROXY-HACK comments' are to make it easier for me to roll the patch from one version to the next.
It doesn't use any D7 feature because it is a patch for D6.

I haven't yet upgraded my site to 6.13 so this is untested but I see no reason why it shouldn't work. When I get some time I may try to take all feedback into consideration but that won't be soon.

Thanks a lot silid, your patch works fine with 6.13 :-)

As far as I understand, nobody wants this feature to go into D7. Marking as won't fix accordingly.

If you do want this to go into Drupal 7, please reroll a patch taking into account my comments in #187 and previous remarks by others.

It will get the work when there is time. This thread is full of people that want this feature.

Until we have a proper patch for this, please let the status at "won't fix".

Still waiting for a proper patch.

The handbook says that 'needs work' is defined as:

The patch needs additional work before it should be reviewed. The author of the patch may indicate that it is incomplete, or the patch has gone through the reviewing process and has received feedback about areas where it needs improvement.

where as 'won't fix' is defined as:

It has been decided that this issue will not be fixed. This includes feature requests that are deemed to be outside the scope of the project, and bug reports that cannot be reproduced or are unsupported.

By your own comments you have described it as a 'needs work', but for some reason you don't want it to say 'needs work' until the work is done. See http://drupal.org/node/156119 for details.

I believe the point is that since no one is trying to make a Drupal 7 version of this patch, that must mean that "it has been decided that this issue will not be fixed". If it was going to be fixed, someone would have taken then 20 min to make a Drupal 7 version of it rather than rerolling a Drupal 6 version forever, which is a dead end.

Take-away: Work on getting better proxy support into Drupal 7 and stop wasting time on anything else. This thread is too long already. Those who keep rolling Drupal 6 versions in this issue, I'm looking at you...

subscribe

Who actually needs this feature?

Also, re-titling to reflect that this is for upstream proxies, not downstream (like Varnish in front of Drupal).

Are you kidding?

Thousands of people who want to/must run Drupal on a server behind a proxy *needs* it.

subscribe

I'm confused. I run drupal behind a firewall and need to use a proxy. Is this hard to understand?

Many many thanks to those who provide this very essential patch.

I *hate* patching my site; it's a maintenance nightmare and my management isn't happy.

And no relief planned for Drupal 7?

What's wrong with this picture?

Anyway, thanks again to those who provide this critical patch!

Another note: it's in no way a waste of time to make my site work as required by my management. Have we been taken over by "kids" w/no real responsibilities and no real life experience?

What's a waste of time is me and many others having to find and apply this patch every time drupal is updated; the right thing to do is *fix the damn problem*, not hide behind "policies."

The right thing to do is for *someone* to roll a patch for *Drupal 7*, so that it can be committed there. Apparently nobody is willing here to do that simple thing... how can I not conclude that nobody wants this feature?

Come on, people.

I'm probably not the right person to roll that patch, but will give it some consideration.

Seriously, the problem may be that the folks who need the proxy capability may be like myself and not really in a position to contribute code. Hence, we're reduced to writing grumpy posts :-)

glowrocks: If your management hates how much time you spend repatching Drupal, tell them that you can instead spend a couple of hours getting it patched in Drupal 7 so that after you upgrade to D7 you'll never have to run a forked version of Drupal again (at least not for this reason). That's how open source works. :-) What's wrong with this picture is that no one who needs this feature is willing to take any time at all to carry it home so that it can be solved. Help yourself by helping Drupal. It won't happen unless someone steps up to work on it.

"Apparently nobody is willing here to do that simple thing". Not true - see #166 - silid has already posted a patch for D7, but the validation failed for some reason. If you say that is so simple, or other "Drupal core people" mention it only takes 20 minutes, so why do you waste your time in making this thread more end more endless instead of helping out with that "easy 20 minutes work"? Not everybody here has these skills, but a lot simply need this patch for D6 now!

Thanks Crell. I probably read this thread too quickly; I came away with the impression that patches were being rejected as no one saw this as a problem.

I wanted to make the case that this is a problem.

If the meta problem is that no one is working on a patch, then that's what needs to be addressed (as you so clearly stated).

Here is my attempt to provide a patch for Drupal 7. I took the last D7 patch from comment 166 respecting most of the recommendations from comment 187.

One thing I kept is the strstr comparison between the host part of the request and the proxy exception patterns. I am not able to see why this should be broken. strstr('localhost', 'localhost') returns 'localhost' and so localhost does match localhost. Maybe there has been a mix-up of strpos with strstr in the review?

@brahms: sorry to tell you this but that patch is far from ready. You have included code that is commented and haven't resolved several of the issues noted in #187. Also, the accepted patch should provide unit tests so that it can be tested properly using the ATS. In fact, unit tests should be written first.

@Damien Tournaud:
@Crell:
@chx:
You don't seem to realise that most of the people in this thread are from enterprises running a Drupal RELEASE, not CVS. Setting up and maintaining a CVS site behind a proxy is not a trivial task, especially when the expected result is that some things won't work and other things work differently to the version you are actually trying to run (D5/6). Many enterprises are not prepared to pay someone to test their less than perfect code against the production proxy server and most workers leave their proxy server behind when they go home at night. This is a significant issue with significant work required, not a 20 minute re-roll.

@everyone: if you have nothing to say that will progress this to resolution, please don't say anything. Posting a 'subscribe' comment or correcting someone's grammar is pointless. D.o. really needs #34496: Add Flag integration - a way for people to subscribe without cluttering the issue with 'subscribe' comments.

What this issue needs is an expert who will see it through to the end. None of us seem to be experts but I am prepared to do what is necessary to get it into D7, with unit tests, all passing the ATS with full support of HTTP and HTTPS (using CONNECT). My only problem is time - I won't be able to start until mid August. If somebody wants it sooner, maybe they should start a Chip-In bounty for it that might attract a D7 expert.

I've probably said before that this should be part of a standard D7 setup. Unfortunately I do not have the skills to create an approved standard patch for consideration.

I work for a school system with at least 1600 schools behind a firewall/authenticated proxy server - and this is just in one state of Australia.

For promoting the use of Drupal around the world, this feature is a no-brainer - it must be done. How hard can it be? Seems like the work has been done - and the argument is over dotting 'i's and crossing 't's.

Come on Dries - throw some resources at it to get it into D7, please!

New patch for D7 based on the latest patch by brahms.

The patch adds proxy support to the new browser in includes/browser.inc instead (this is supposed to replace the implementation in drupal_http_request(), see #553280: Integrate browser.inc and drupal_http_request()).

I haven't tested the proxy username/password stuff (because my local proxy does not require this).

@c960657: thanks for working on this.

The first part is just $exception[0] == '.', and I find the substr_compare() here a little bit cryptic. substr($hostname, -strlen($exception)) == $exception seems easier to understand.

This should probably be FILTER_FLAG_IPV4, because gethostbyname() doesn't support IPV6 yet.

cURL options are persistent. Don't we need a else clause here?

Please, try to avoid network lookups during the tests (the secret plan is to allow tests to run in full network isolation one day). You can probably safely assume that localhost resolves to 127.0.0.1.

Those should be #element_validate functions of their own element.

There is a small indentation issue here.

This review is powered by Dreditor.

Great comments. All points are addressed in this patch.

This patch is getting even more awesome by the day ;)

The second parameter should probably be FILTER_FLAG_IPV4.

cURL can be a strange beast sometimes. Have you verified that this works?

form_set_error() should be form_error($element) here.

Why not abstracting those element validation a little bit? This is simply a system_port_validate() ;)

I'm on crack. Are you, too?

cURL can be a strange beast sometimes. Have you verified that this works?

Yes, setting it to FALSE seems sufficient (and not resetting it like in proxy-1.patch does indeed cause trouble).

The other points are fixed.

(BTW I look forward to the day when FAPI will support something like this: '#type' => 'number', '#min' => 0, '#max' => 65535 - and transform it into a nice HTML5 input field <input type="number" min="0" max="65535" /> and of course also do server-side validation :-)

Good, let's get this in ;)

Next stop: make drupal_http_request() use the browser ;)

This will need a sign-off from Dries, IMO, since this patch has changed significantly since the incarnation where he was on the cusp of committing it. Too bad there was so much jiggering with the version selector, or this likely would've been resolved (and possibly backported) a year ago. Sigh...

I'm not sure of this new approach. It doesn't look like the latest patch actually fixes the problem; just fixes it in the brower.inc which hinges on drupal_http_request() moving over to it. There is quite literally nothing at #553280: Integrate browser.inc and drupal_http_request(), which doesn't give me confidence that we'll end up doing this in time for the end of code slush. And even if we do get a patch over there, we're going to need to talk over the implications of basically forcing core to require cURL (unless I am mistaken about the state of the final browser.inc that made it in).

Couple of minor things:

From a usability perspective, it'd be nice to have a line or two description in hook_help() for this form that would indicate to me as an end-user whether I need to worry about this or whether I can ignore it. It seems very odd to me to include such a 'scary' settings form in core when only 1/10000000 sites will use it, but this passed Dries's sniff test, so that's fine.

Why display it as plain-text as well? That's odd. Shouldn't this be a password field?

I'm pretty sure 0 is not a valid proxy port. So probably make this > 1. Also, there should be a period at the end of this comment.

Other than those small things, this looks fine from my perspective.

Why display it as plain-text as well? That's odd. Shouldn't this be a password field?

Changed to '#type' => 'password'. I'm not sure if this is the best way to populate the field: '#attributes' => array('value' => variable_get('proxy_password')) The point is that we want to indicate to the user whether a password is specified or not.

I'm pretty sure 0 is not a valid proxy port.

According to http://en.wikipedia.org/wiki/TCP_and_UDP_port#Technical_details, 0 is a valid port (but probably not much used in practice).

Added a line to hook_help().

There is quite literally nothing at #553280: Integrate browser.inc and drupal_http_request(), which doesn't give me confidence that we'll end up doing this in time for the end of code slush.

I share your concern, though it was initially the goal to get that fixed in time for D7 (see #340283: Abstract SimpleTest browser in to its own object comment #99). I suggest we postpone this patch until #553280: Integrate browser.inc and drupal_http_request() has landed.

Changing the priority to critical, since the code freeze for D7 already began.

The last submitted patch failed testing.

subscribe.

Waiting for a 6.14 patch.

ganglion, "proxy6.13.patch" from #192 works also with D6.14.

I have tried, and I get this:

Am I doing something wrong? The patch looks OK when I compare it visually to the code.

I just tried the 6.13 patch again, against a freshly downloaded drupal-6.14, and it fails with the same errors.

I haven't tried it against 6.14 yet. Have you tried increasing your fuzz factor?

Si.

I can't do this - the version of patch on my server doesn't have a -F option.

Are you likely to have a 6.14 patch ready in the next few days, or should I try to make my own from the 6.13 patch?

It *did* work for me:

I haven't tested this. I applied the patch for 6.13 to 6.14 and rolled a new patch.

This also fails on the live server (a solaris machine):

Patch #223 was with DOS carriage return settings...

Same patch after dos2unix attached below.