As per my understanding the drupal core itself and the CustomError moduledoes not send the HTTP/1.1 404 Not Found in header.

Would like to know that why it is designed in such a way? wasn't that will keep search engine spider coming back to the "not found page".

Comments

nevets’s picture

Drupal core provides the function drupal_not_found() which does set the HTTP header and as far as I know it is how core reports page not found errors.

Foodster’s picture

Thanks nevets,

I think i know what is causing

When running Drupal on a server with PHP as CGI you have to change line 288(?) in /includes/common.inc from

drupal_set_header('HTTP/1.0 404 Not Found');

to

drupal_set_header('Status: 404 Not Found');

toomanypets’s picture

Thank you Foodster. Changing the header sent by drupal_not_found() in /includes/common.inc solved a problem that I had been struggling with for hours -- 500's instead of 404's. Instead of looking at the error log, I wasted many hours assuming that this problem was related to RewriteRule's in my root folder's .htaccess file (Drupal is installed in a subdirectory). When I finally reached the conclusion that I was approaching the problem incorrectly, I examined the error log and found:

/home/httpd/vhosts/default/fcgi-bin/php5fcgi" aborted: error parsing headers: malformed header 'HTTP/1.1 404 Not Found'

So, based on your post I changed:

function drupal_not_found() {
- drupal_set_header('HTTP/1.1 404 Not Found');
+ drupal_set_header('Status: 404 Not Found');

Everything is working fine now. In the future, on new installs, the first thing I will look at is the "Server API" section of the page returned by phpinfo(). If it "CGI/FastCGI" instead of "Apache 2.0 Handler" or similar I will make this change to common.inc.

See also http://drupal.org/node/175855 (#24).

kenorb’s picture

Similar problem.