This is a spin off from this infra issue http://drupal.org/node/255204

When a site is put offline, the responses have an HTTP code of 200, and hence search engines will index the offline page. This is not desirable for obvious reasons.

What we need to do is send a 503 and a Retry-After message when a site is offline. This way, (well behaved) search engines would not index the offline page.

If no one else gets to this I will roll a patch for it.

Comments

greggles’s picture

greggles
he/him
Primary language English
Location Denver, Colorado, USA
commented
Title: Need to send correct headers so site offline would not get indexed in search engines » Send "Retry-After" headers so robots know when to crawl again
Status: Active » Closed (fixed)

Headers from a test site in "offline" mode.

We're sending the correct HTTP 503 code, it's just the (lower priority) "Retry-After" that needs to be added.

From my Ubuntu7.10 localhost using Firefox LiveHTTPHeaders extension:

http://gvs1/m/tester/



GET /m/tester/ HTTP/1.1

Host: gvs1

User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.14) Gecko/20080418 Ubuntu/7.10 (gutsy) Firefox/2.0.0.14

Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5

Accept-Language: en-us,en;q=0.5

Accept-Encoding: gzip,deflate

Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7

Keep-Alive: 300

Connection: keep-alive

Referer: http://gvs1/m/tester/admin/settings/site-maintenance

Cookie: SESS5d2d03bf42b711b3079654709448e81a=18f086c317c1cfc8007846332e0b583f

If-Modified-Since: Tue, 06 May 2008 16:17:39 GMT

Cache-Control: max-age=0



HTTP/1.x 503 Service unavailable

Date: Tue, 06 May 2008 16:17:46 GMT

Server: Apache/2.2.4 (Ubuntu) PHP/5.2.3-1ubuntu6.3

X-Powered-By: PHP/5.2.3-1ubuntu6.3

Expires: Sun, 19 Nov 1978 05:00:00 GMT

Last-Modified: Tue, 06 May 2008 16:17:46 GMT

Cache-Control: store, no-cache, must-revalidate, post-check=0, pre-check=0

Content-Length: 3567

Connection: close

Content-Type: text/html; charset=utf-8

IMO, we actually should not add a Retry-After. Some sites will stay in maintenance mode for a long time. The way to get the Retry-After "right" would be to ask users to specify when they expect to be back online but that overly clutters the UI.

If we set the Retry-After to "1 hour after entering maintenance mode" or some other made up value, then that might be wrong as well...better to just not send it than to send the wrong value.

So, I set this to closed as both "fixed already" for part of the idea and "won't fix" for the other part.