RSS feeds bringing Drupal.org down

I'm seeing this message now: Our servers are currently a bit busy, please try refreshing the page after a few seconds.
Thanks!

I'll see what I can find for a parent issue.

Infrastructure team might have blocked release feeds because of performance issues. I am checking with them now.
In the mean time feel free to choose any other issues with Drupal.org 7.1 tag to work on :) https://drupal.org/project/issues/search?text=&projects=&assigned=&submi...

Correct. I currently have release feeds blocked globally as they were taking down the site. This need to be addressed.

-N

Any idea why they were taking down the site?

nnewton, can you give more information? Why they were taking the site down? How can we fix it?

I saw that most of the servers are blocking it still but for some reason www5 is not:

ricardo@ricardo-VPC:~$ curl -I https://drupal.org/node/3060/release/feed
HTTP/1.1 503 Service Unavailable
Server: nginx/1.4.7
Date: Fri, 28 Mar 2014 13:50:29 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 211
Connection: keep-alive
Retry-After: 0
X-Varnish: 771684814
Age: 0
Via: 1.1 varnish
X-Cache-Svr: www1.drupal.org
X-Cache: MISS

ricardo@ricardo-VPC:~$ curl -I https://drupal.org/node/3060/release/feed
HTTP/1.1 503 Service Unavailable
Server: nginx/1.4.7
Date: Fri, 28 Mar 2014 13:50:33 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 211
Connection: keep-alive
Retry-After: 0
X-Varnish: 1248611135
Age: 0
Via: 1.1 varnish
X-Cache-Svr: www7.drupal.org
X-Cache: MISS

ricardo@ricardo-VPC:~$ curl -I https://drupal.org/node/3060/release/feed
HTTP/1.1 200 OK
Server: nginx/1.4.7
Date: Fri, 28 Mar 2014 13:50:52 GMT
Content-Type: application/rss+xml; charset=utf-8
Content-Length: 338801
Connection: keep-alive
X-Powered-By: PHP/5.3.28
X-Drupal-Cache: MISS
Etag: "1396012027-0"
Content-Language: en
Cache-Control: public, max-age=0
Last-Modified: Fri, 28 Mar 2014 13:07:07 +0000
Expires: Sun, 19 Nov 1978 05:00:00 GMT
Vary: Cookie,Accept-Encoding
X-Varnish: 1217607366 1217469374
Age: 2624
Via: 1.1 varnish
X-Cache-Svr: www5.drupal.org
X-Cache: HIT
X-Cache-Hits: 12
Front-End-Https: on

so I would advise for us to move the configurations from CFEngine to puppet, so every Infra Sysadmin can change them and update them across servers

This is the actual code that's blocking the feeds:

default.vcl:

     if (req.url ~ ".*/node/[0-9]+/release/feed") {                                                                                                            
       error 503;                                                                                                                                             
    }     

Updating issue summary.

While we're figuring out how to fix this issue - is there any possibility to provide a better message on this blocked feeds? Currently it says:

Our servers are currently a bit busy, please try refreshing the page after a few seconds.
Thanks!

Which is not really a good idea, considering refreshing won't help.

I'm not sure what the root cause is here. These pages are another Display of https://drupal.org/node/3060/release, which is doing okay, as far as I know. We may be able to turn these back on, any pay attention to the DB and logs.

@drumm you mean someone can remove this code from varnish?

   if (req.url ~ ".*/node/[0-9]+/release/feed") {
       error 503;
    }

or is that still a problem?

Yes, I think that should be removed. (Or we need to know what is problematic about those pages.)

@drumm where can i change those vlcs?

$ git commit -am"drupal.org/node/2129427 - RSS feeds"
[master 5c3f7f2] drupal.org/node/2129427 - RSS feeds
 1 file changed, 4 deletions(-)
$ git push
Counting objects: 15, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (7/7), done.
Writing objects: 100% (8/8), 656 bytes | 0 bytes/s, done.
Total 8 (delta 5), reused 0 (delta 0)
To git@bitbucket.org:drupalorg-infrastructure/puppet.git
   aa15496..5c3f7f2  master -> master

this is back to normal now

while true; do curl -sI https://drupal.org/node/3060/release/feed  | egrep "HTTP|Cache-Svr"; sleep 1; done
HTTP/1.1 200 OK
X-Cache-Svr: www5.drupal.org
HTTP/1.1 200 OK
X-Cache-Svr: www6.drupal.org
HTTP/1.1 200 OK
X-Cache-Svr: www6.drupal.org
HTTP/1.1 200 OK
X-Cache-Svr: www5.drupal.org
HTTP/1.1 200 OK
X-Cache-Svr: www7.drupal.org
HTTP/1.1 200 OK
X-Cache-Svr: www2.drupal.org
HTTP/1.1 200 OK
X-Cache-Svr: www7.drupal.org
HTTP/1.1 200 OK
X-Cache-Svr: www2.drupal.org
HTTP/1.1 200 OK
X-Cache-Svr: www7.drupal.org

Thanks Ricardo