HTTP request status fails (web in maintenance)

My humblest apologies to everyone.

I have discovered that my problem with the HTTP request status fails was a result of the .htaccess redirect to a secure site. When I commented out the .htaccess redirect lines the problem was resolved. I then removed the comments and was still able to update the feeds. I will keep an eye on this over the next few days and see if there is anything else I can learn.

Sorry for raising unwarranted concerns. I hope others did not spend too much time pondering my problem.

Thanks,

Izzy

Well I think there is still a problem here. And it has to do with the error recovery in drupal_http_request in common.inc for version 6.

The code in question is:
if (!$self_test && variable_get('drupal_http_request_fails', FALSE)) {
$self_test = TRUE;
$works = module_invoke('system', 'check_http_request');
$self_test = FALSE;
if (!$works) {
// Do not bother with further operations if we already know that we
// have no chance.
$result->error = t("The server can't issue HTTP requests");
return $result;
}
}

which tries to recover in the event that a request has failed previously. This snippet then calls check_http_request:

function system_check_http_request() {
// Check whether we can do any request at all. First get the results for
// a very simple page which has access TRUE set via the menu system. Then,
// try to drupal_http_request() the same page and compare.
ob_start();
$path = 'admin/reports/request-test';
menu_execute_active_handler($path);
$nothing = ob_get_contents();
ob_end_clean();
$result = drupal_http_request(url($path, array('absolute' => TRUE)));
$works = isset($result->data) && $result->data == $nothing;
variable_set('drupal_http_request_fails', !$works);
return $works;
}

which tries to do a http request on admin/reports/request-test and there-in the real problem lies, at least for me on my hosting service. You see, I'm using godaddy free hosting that places some js on each page. So when this recovery routine tries to kick in because of some past failure, it never recovers because of the following clause: $result->data == $nothing.

I was able to correct my problem by removing the failing check ($result->data == $nothing). But this brings up whether or not checking admin/reports/request-test is a reliable solution to the problem being solved. I propose that instead a file hosted on drupal.org should be used for the recovery check.

I had the same problem, the reason was that I had a password-lock on the website (Basic-Auth via .htaccess-file) while I was working on the website.

When I swithed Basic-Auth off, everything worked fine. So your explanation with system_check_http_request is exactly what happened.

It seems, Drupal first checks whether it can connect to itself. In case it can't, it stops trying to connect to remote servers. That's something I'd say is a bit weird, because in most cases, the website should be blocked via basic-auth anyway while in development mode.

Well, I appreciate that you two are looking at this closely but you are above my head.

I am still looking for a way to keep the feeds working when I have .htaccess redirecting to SSL.

The lines I have in .htaccess are:

SSLOptions +StrictRequire
SSLRequireSSL
SSLRequire %{HTTP_HOST} eq "mysite.com"
ErrorDocument 403 https://mysite.com

When these are commented and the redirect inactive the feeds work. When I remove the comments and activate the redirect the feeds fail.

I thought it might be improper for me to have this showing as a bug report so I changed the status to "closed" and posted this question as a new topic.

As someone new to the Drupal community I would appreciate any help and guidance. Should the status of this thread be kept open?

Thanks,

Izzy

Izzy,

If you feel like hacking the code a bit, do this:

Edit common.inc and in function drupal_http_request() comment out the recovery code. That is:
// Try to clear the drupal_http_request_fails variable if it's set. We
// can't tie this call to any error because there is no surefire way to
// tell whether a request has failed, so we add the check to places where
// some parsing has failed.
/*
if (!$self_test && variable_get('drupal_http_request_fails', FALSE)) {
$self_test = TRUE;
$works = module_invoke('system', 'check_http_request');
$self_test = FALSE;
if (!$works) {
// Do not bother with further operations if we already know that we
// have no chance.
$result->error = t("The server can't issue HTTP requests");
return $result;
}
}
*/

This will prevent the recovery code from running, but will also allow all http requests to be attempted. In my opinion there is no harm in doing so.

Then add the following line so that the flag gets cleared:
variable_set('drupal_http_request_fails',FALSE);

You *could* add it right after the code that was commented out, but that would defeat the purpose of the flag. Probably a better place for it would be in the success path of the code in this function. Try putting it before these lines at the end of the function:

$result->code = $code;
return $result;
}

So you would get at the end of the function something looking like:

variable_set('drupal_http_request_fails',FALSE);
$result->code = $code;
return $result;
}

Let me know if you try it out and if it works for you.

My HTTP requests were working fine, until one day they stopped.

This had been driving me crazy, because I thought it was a problem in the module I was developing, until I noticed that the update status wasn't working either.

I set drupal_http_request_fails to 0, and now it all works again!

I did a little poking around, and it turns out the requests fail because the site has HTTP authentication enabled. You have to login before you can view anything. So when system_check_http_request looks at $result->data, it finds an error message (Authorization Required), and the check fails.

So, for anyone using HTTP authentication, system_check_http_request breaks drupal_http_request. I don't see why that part of the check is necessary in the first place, but even if it does something useful, there has to be a solution that doesn't kill people's otherwise healthy http requests.

I cannot comment on your findings. But I have found that the problem I was encountering was indeed related to redirecting to an SSL site.

I am using .htaccess and when I commented out the redirect and entered the site with http: the feeds worked.

I also found when I then went back and removed the comments so the redirect was active to the https: site the feeds would work but stop doing so after several hours.

I repeated the same steps and found the same results.

Shortly after that I added Google analytics and the problem went away.

I am not sure if the problem would have gone away anyway after the short span of a day or two if I hadn't added Google analytics but I haven't bothered to remove that and test again.

Izzy

I think the problem might be related to sqlite install not working.

I noticed on my server the HTTP request status fails and yum not working due to sqlite issue at the same time.
This suggest the http request are not working due to the sqlite issue.

here is the error on yum command
Warning, could not load sqlite, falling back to pickle

here is what I have on my Red Hat EL4
> rpm --query yum python python-sqlite sqlite
yum-2.4.2-0.4.el4.rf
python-2.3.4-14.4.el4_6.1
python-sqlite-2.3.3-1.el4.rf
sqlite-2.8.17-1.el4.rf

working on finding a solution and will let you know if that fixes this http error.
If anyone else has any ideas on how to update this sqlite prob please do post.

ok, I got the HTTP request status fails issue fixed.

It had to do with my BIND going out of whack

if you run a nslookup command like so, and you get the following ..then Bind is working fine.
> nslookup google.com
Server: 127.0.0.1
Address: 127.0.0.1#53

Non-authoritative answer:
Name: google.com
Address: 64.233.187.99
Name: google.com
Address: 72.14.207.99
Name: google.com
Address: 64.233.167.99

If the nslookup gives error than you need to fix bind - start by checking these 2 files
1) /etc/sysconfig/network

mine looks like this.
NETWORKING=yes
GATEWAY="168.X.X.X"
HOSTNAME=yourservername

and file
2) /etc/resolv.conf

which looks like this
nameserver 127.0.0.1
nameserver 165.x.x.20
nameserver 165.x.x.21
nameserver 168.x.x.x

I just added the 'hosting' provider ips for nameserver as backup (165.x.x.20/21)

I'm getting this message in the STATUS REPORT...
HTTP request status Fails
Your system or network configuration does not allow Drupal to access web pages, resulting in reduced functionality. This could be due to your webserver configuration or PHP settings, and should be resolved in order to download information about available updates, fetch aggregator feeds, sign in via OpenID, or use other network-dependent services.

I looks like there are different views in this post. I was wondering if someone would post their ideas to this proplem in very plain speak. I'm a newbie!!!
Thank you,
Keith

I had the same issue...until I realized I had my webpage in "maintenance": > settings> site-maintenance
As soon as I put the web online again I got rid of the problem.

Hopefully this is Keith's case.

:)

That fixed it for me to. As a suggestion/bug fix - could the status report be set to say "It looks like you're running your site in offline mode. Try switching it to online and back again, which might fix the problem"?

(Putting my site back into offline mode does not re-trigger the problem)

You read my mind. Even thought it is clear once you know, you tend not to associate one issue with the other since time has passed. It will be nice to add a comment similar to the one you propose.
The current message only tells you there is a problem but not a single suggestion is given. I was almost ready to call my hosting provider thinking that it was "their" problem.

It seems it takes a little bit of time to recreate the issue if you get into maintenance mode.

Perhaps there are other issues involved with this matter but I do agree with ged3000 that a short suggestion in the status report won't hurt and can avoid some headaches.

Thanks everyone for saving me the headaches. I only spent an hour researching this and none of my hair has been pulled out. I turned off my HTTP authentication as you suggested and it works now! I agree that this probably is due to some unnecessary steps being done in the HTTP check, but I haven't decided if I want to hack anything yet...

age3141592, thanks for your guidance.

I experienced this failure in D6.2 like others described. Only I don't even have agregator enabled but I guess it started after installing openidurl and/or facebook_auth. But even after uninstalling _and_ removing the directories this persisted.

I also played around with permissions for guests/anonymous and maintenance permutations but to no avail.

Only after I did what you suggested (variable_set... put to end of that function) and running cron manually all was well. I even could put common.inc back to what it used to be and running cron seams to work, still.

To me it seams as if drupal "locks itself out" once this situation has occured. So, though I'm new to drupal, would consider this a bug (even critical?). I'll keep an eye on it and come back here.

Thanks again!

You're a hero! That fixed it for me! I was doubly puzzled (nay, tripply puzzled) because I'm running several copies of Drupal (various rev levels) on the same ISP and this was only affecting one of them. Weird, or what?

Thanks again.

edit:

This was supposed to be in reply to:

age3141592 - April 19, 2008 - 12:43

But it simply appears at the bottom of the page out of sequence...

edit 2:

Well, what can I tell you? It worked, my site checked Drupal for updates, there were none, everything looked tickety-boo, and then a few minutes later I've got the same problem back again, big pink window saying HTTP request status fails.

Great.

hi all,
I tried all those but they didn't work...

I later figured out my problem. My site couldn't connect to itself, the link in the cron was broken as i had moved my site to a different directory.

Updating crontab, everything is fine...

Regards,
shreyank.

OK, I could spend ages trying this lot out to make my site check for updates.

What I need is a 'dummies guide' to fixing this problem.

Any volunteers?

Well, karlcw, it's really not that hard. I'll try my best:

If I recall correctly the only thing I did was:

1. Get a copy (or edit it online) from the file common.inc in your drupal directory and open it with your text editor of choise (not word or wordpad, texteditor!)
2. Find the line where it says: function drupal_http_request( (for D6.2 it should be line 416)
3. Now find the end of that function which in D6.2 case should be line 563. Above that line it says:
   
   
     $result->code = $code;
  return $result;
   
   Now you need to paste in the following line just above $result->code
   
     variable_set('drupal_http_request_fails',FALSE);
   
   so now the last three lines read:
   
     variable_set('drupal_http_request_fails',FALSE);
  $result->code = $code;
  return $result;
4. After you saved this change on the server (via ftp or whichever way you use) run your update function from drupal's admin area once (www.yourdomain.tdl/admin/reports/updates/check)which now shouldn't produce any errors.
5. Now comment out the line you just added which means to prefix that very line with two slashes (//) and save it to your server again.

On my site drupal somehow only needed this flag cleared (i.e. set variable drupal_http_request_fails to FALSE) so it runs without complains ever since. If the update script does produce errors after you did the last step, i.e. comment out, maybe you might need to leave that line in place until 6.3 fixes this (or which ever version it will be), i.e. don't do the last step.

Hope that helped,
cheers.

The fix to add -- variable_set('drupal_http_request_fails',FALSE); -- to common.inc doens't work for me.

The problem with HTTP request status fails on my site occours out of the blue. Site works great for a few weeks.
This weekend it stops working, nothing changed.

What went wrong.

For those that don't understand it yet, here is an explanation of this bug:

- Drupal wants to make an HTTP request; it calls system_check_http_request().
- That function does three things:
* Makes an internal call to get admin/reports/request-test
* Makes an external call (via HTTP to itself) to get admin/reports/request-test
* Compares the two; if they match, it returns true.

The problem is that the external call might not be same result. If you put admin behind authentcation, or insist on it being SSL or your server can't resolve to itself, or any number of other things that can happen very, very easily without HTTP requests actually failing, Drupal will insist that you fail.

I also don't understand why all that mess in drupal_http_request() is necessary when the cURL module is so widely deployed. Additionally, why is it hitting a page in the admin section when it could easily just hit a /ping and check for an HTTP 200/301/302 response?

I just upgraded to drupal 6.3. This issue was happening for me before the upgrade, and it's still happening now. I had hoped it would resolve it, but it doesn't.

I just verified that the issue as marcelesser describes it is exactly what is happening on my system. Of course I want the admin section behind authentication. Duh! Who doesn't. And since drupal isn't going to submit your admin user name and password to itself when making the request, there is about zero likelihood that this will work on any real production drupal installation.

For me, even the solution to use variable_set('drupal_http_request_fails',FALSE) doesn't work. And I don't know why. Very frustrating.

I'm in same boat as JCwatson EXCEPT, I never had issue in 6.1/6.2...Only after upgrade to 6.3. Also tried marcelesser solution with no impact...??

SORRY! IT's a non-issue when you get out of maintenance mode as mentioned above....

Hi, the same problem here upgrading from 6.2 to 6.3. As other user says on this comment http://drupal.org/node/227791#comment-860911 "update check manually" fixed the problem and after that the status report was clean.

The bug is also in Drupal 7, so bumping release.

Here is a patch for D6 and D7 that implements a bypass of the check. Because system_check_http_request() is used in several places in core (in OpenID, in Update, in common.inc, etc.), the bypass must be right inside that function.

To test:

• Patch your Drupal installation
• Add $conf['assume_http_request_works'] = TRUE at the bottom of your settings.php

I have restored functionality using the patch, however the status reports that "http request status fails." As I said I am able to check updates so I think all is working, just would look better if I could reset the error message.

Patch from #25 works but it need work because loop in system_check_http_request - drupal_http_request
Suppose problem caused by aggregator or system_check_http_request

debug: (if drupal_http_request_fails set to TRUE)

drupal_http_request(http://drupal.org/security/rss.xml) $self_test: false
system_check_http_request()
drupal_http_request(http://www.drupal.ru/admin/reports/request-test) $self_test: true
system_check_http_request() $works=false (but
Aggregator: error.. The server can't issue HTTP requests
system_check_http_request()
drupal_http_request(http://www.drupal.ru/admin/reports/request-test) $self_test: false
system_check_http_request()
drupal_http_request(http://www.drupal.ru/admin/reports/request-test) $self_test: true
system_check_http_request() $works=false
system_check_http_request() $works=false

Hello,
Well, my hosting service doesn't allow loopback connection, thus, the HTTP Request fails (in the same way, I can't use normal cron). Is there any mean to get this working again?

Thanks.

David

This fixed a problem I have now in my hotel room. Thx.

Hi I am a newbie to these forums.
I have received this error in my status report. I have read this discussion and others related to this error. Also, nearly everything that has been suggested has been tried to no avail.

I've:
- Taken my site "offline" and put back into "online" mode and ran cron in the status report --- Error still
- Applied the patch as discussed in #19 -- Error still
- Taken out the modules which I thought were affecting it (OpenID and OpenX Manager) -- Error still

I'm not sure what else to do, other than uninstall OpenAds, OpenX. My OpenX database and drupal database are different names, and different from my site names. I haven't tried the BIND thing as suggested in #9, but I'm not sure what that does or how it would help.

I also don't have aggregator. I got this status report error right after I installed and configured OpenX (which runs on a different directory--its not in the same one as my Drupal installation) so maybe that might be the error?

I've also added my other databases to the settings.php file, but I haven't called them, nor do I know if thats whats affecting the HTTPRequest.

I would appreciate any feedback!! Thanks in advance.

**EDIT** So magically, I re-activated my OpenX manager (after having disabled it before above), and put in the new criteria to link to my OpenX server, and BAM NO ERRORS!! I am so confused, but whatever, I'm happy :)

Rerolled #25 with fixed documentation and an additional clearing of drupal_http_request_fails do deal with the edge case described in #27. No real functional change, so back to RTBC as per #29.

The by-pass is cute but does not fix the problem for 99% of our install base. Less than 1% will know/understand that they can flip a magical variable. I'd like to see us make the code more robust, rather than adding a variable. I don't think this is an adequate/complete solution.

If the code itself is being revisited, might I suggest that the admin/reports/request-test page returns JSON (using drupal_json();) instead of HTML? This should help prevent some hosts from injecting HTML into the page since it will no longer be text/html. This assumes that the code revisit doesn't do away with this check.

It's also worth mentioning that in researching this error I found that I hadn't enabled OpenSSL on my PHP installation (openssl.so or openssl.dll). Once I enabled this setting, I was able to get this check to work properly (I have anything under /admin forced to SSL).

To sum up the other errors I've read in this issue, the check will always fail when:

• The server/target page is protected by server-level authentication
• The site is offline (anonymous users will all get the site maintenance page)

Whitelisting the URL during maintenance mode would clear up that last problem (I have no clue how easy that would be), but feels like a hack. The server-level authentication is a tougher nut to crack.

I'm using Drupal 6.4 and even with the above #31 patch applied I'm still getting the error. Not keen on hacking core modules but it's an important time for the site I'm working on. Seems strange that's it's been going on for that past 3 releases of Drupal too :(

subscribing

I've tried the above patch and I'm still getting the error.

I'm very much a Drupal novice and I'm very unsure about patching core files etc.

Please help!!!

Tossing an idea out there. What if we changed the url that drupal_http_request uses to test itself to the url http://example.com/ that's provided via RFC 2606. That way this wouldn't fail if a site is in maintenance mode?

This fixed the problem for me :)

http://collectiveidentity.net/node/13

Thx physcab,

I was running through all the different possible causes and solutions. Thankfully I tried yours too. It worked. I think my HTTPRequest issue was initiated when I enabled parts of the Devel Module. I was pulling my hair out, un-enabling those parts of the Devel Module, and attempting the various fixes ... then I read your post. I immediately re-enabled the portions of the Devel Module and the HTTPRequest Error was no longer reported in my Status Report. THX for posting your process. I might not otherwise have made the connection!!!

I need help please..I am really at the end of my rope on this.

I am getting the "Server can't administer HTTP requests" on feeds. I am on a clean install of 6.4 with Aggregator 6.4
I have as suggested (don't freaK) hacked common.inc with dreamweaver and changed it so it reads:

variable_set('drupal_http_request_fails',FALSE);
$result->code = $code;
return $result;
}
put the site offline then put it online, logged on and off and manual ran of Cron. Still no feeds.Nothing works..It has been at least six hours of troubleshooting..please help-anybody

The fix at http://collectiveidentity.net/node/13 solved the problem for me as well.

Thanks for the reply JuliaKM but thats the fix I tried and no luck

Subscribing

This note is for anybody on Godaddy under the freehosting plan who experienced a loss in feeds once they upgraded from 6.2 upward.
I have spent WEEKS on this issue and can say that NOTHING works. Hacking core won't work. the countless suggestions won't work in these nodes won't work

All related posts on what clearly is an existing problem
http://drupal.org/node/227791

core hack suggestion & patch (which users report may not work):

http://drupal.org/node/245990

Collective Identity in depth discussion and walk thru of core hack
http://collectiveidentity.net/node/13

Moving over to Acquia won't work..None nada none works. This is an issue that for whatever reason is just being ignored as a Service Provider problem and not a Drupal problem.