I am helping a new client with some updates to their site and one of the biggest issues they are having is that they aren't indexed in Google. In looking at Google's Webmaster Tools, it appears that since they moved to drupal and a new server last December, they haven't been indexed.

I looked at the robots and htaccess and those match what I run on my drupal site which is indexed daily.

The previous developer said he thinks it has something to do with the views module.

They are running Drupal 5.3. Any ideas would be greatly appreciated.

Thanks!

Comments

allen074’s picture

ok i ran a header check and it came back with
500 INTERNAL SERVER ERROR

hrm - any ideas where to look at for this since i dont see it when i view the page?

thanks!

snufkin’s picture

yeah apache error log

dman’s picture

Can't help debug much if you don't want to post an URL.
It's odd that you are able to see the site yourself but the robots are not. We need real header investigation. Is the domain redirected in some way? It's possible (I guess) for that to be malformed in some way that robots don't like but the browser allows.

Are you sure there is no deliberate access control happening? In Drupal or even earlier. It could be an anti-leech script at the Apache level or .. well, we need an URL to probe to really see more.

.dan.
How to troubleshoot Drupal | http://www.coders.co.nz/

allen074’s picture

url is:
http://www.jcreport.com

thanks for the help!

dman’s picture

Yeah, that's wack.

dman:~$  wget  -d --save-headers http://www.jcreport.com/
Setting --save-headers (saveheaders) to 1
DEBUG output created by Wget 1.10.2 on linux-gnu.

--14:42:31--  http://www.jcreport.com/
           => `index.html'
Resolving www.jcreport.com... 67.192.120.241
Caching www.jcreport.com => 67.192.120.241
Connecting to www.jcreport.com|67.192.120.241|:80... connected.
Created socket 3.
Releasing 0x08089b20 (new refcount 1).
Date: Thu, 17 Jul 2008 02:41:58 GMT
Server: Apache/2.0.52 (Red Hat)
X-Powered-By: PHP/5.2.6
Set-Cookie: SESSbe7fc6e2f96e9592116e6c095d6adcd9=5tg3tjlr2klns1ob694lcv23c4; expires=Sat, 09 Aug 2008 06:15:18 GMT; path=/; domain=.jcreport.com
Expires: Sun, 19 Nov 1978 05:00:00 GMT
Last-Modified: Thu, 17 Jul 2008 02:41:58 GMT
Cache-Control: store, no-cache, must-revalidate
Cache-Control: post-check=0, pre-check=0
Connection: close
Content-Type: text/html; charset=utf-8

---response end---
500 Internal Server Error
cdm: 1 2 3 4 5 6 7 8
Stored cookie jcreport.com -1 (ANY) / <permanent> <insecure> [expiry 2008-08-09 18:15:18] SESSbe7fc6e2f96e9592116e6c095d6adcd9 5tg3tjlr2klns1ob694lcv23c4
Closed fd 3
14:42:33 ERROR 500: Internal Server Error.

---request begin---
GET / HTTP/1.0
User-Agent: Wget/1.10.2
Accept: */*
Host: www.jcreport.com
Connection: Keep-Alive

---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.0 500 Internal Server Error

But the page comes up in FF fine.
Something is screwy in the server. Could be it's bouncing requests that don't have the right headers (user-agent?) .. not really.

dman:~$ telnet www.jcreport.com 80
GET /

responds as expected however.
Something about the HTTP protocol?
... Hm, same even if I enter the identical headers from the commandline as wget sent.

GET / HTTP/1.0
User-Agent: Wget/1.10.2
Accept: */*
Host: www.jcreport.com
Connection: Keep-Alive

works OK as expected! Same for both www and not (no redirects that I can see)

That's messed up.

Even worse, requesting files directly
wget -d http://jcreport.com/files/imagecache/small_w150px_sq/files/article_thumbs/liberty1.jpg
DOES work. So yeah, it's looking like not Apache, but Drupal-specific!

Is there anything custom in your .htaccess?
You must have some module in there that is expecting something that's not available during non-interactive sessions. Any third-party user-tracking? logintoboggin?
Creepy.
I guess the apache logs for the error are the only clue. You may try bumping up the error-reporting level in PHP to return to the screen... but it's looking like the death is fairly early on in the request, possibly before drupal bootstrap really kicks in.
Anything custom in the settings.php?

This is also meaning that thinks like the validator can't see anything either. Pretty serious malfunction.
The strangest thing is that real browser requests DO work!
:-(

.dan.
How to troubleshoot Drupal | http://www.coders.co.nz/

dman’s picture

extra - I get a sort-of correct 404 to 'not found' pages, not a 500. Not sure what that tells us.
It's a Header 404 with no content though, not the full (nice) 404 page the browser gets.

allen074’s picture

wow - what a reply - thank you so much! here's the htaccess - pretty standard i think:

#
# Apache/PHP/Drupal settings:
#

# Protect files and directories from prying eyes.
<FilesMatch "\.(engine|inc|info|install|module|profile|po|sh|.*sql|theme|tpl(\.php)?|xtmpl)$|^(code-style\.pl|Entries.*|Repository|Root|Tag|Template)$">
  Order allow,deny
</FilesMatch>

# Don't show directory listings for URLs which map to a directory.
Options -Indexes

# Follow symbolic links in this directory.
Options +FollowSymLinks

# Customized error messages.
ErrorDocument 404 /index.php

# Set the default handler.
DirectoryIndex index.php

# Override PHP settings. More in sites/default/settings.php
# but the following cannot be changed at runtime.

# PHP 4, Apache 1.
<IfModule mod_php4.c>
  php_value magic_quotes_gpc                0
  php_value register_globals                0
  php_value session.auto_start              0
  php_value mbstring.http_input             pass
  php_value mbstring.http_output            pass
  php_value mbstring.encoding_translation   0
</IfModule>

# PHP 4, Apache 2.
<IfModule sapi_apache2.c>
  php_value magic_quotes_gpc                0
  php_value register_globals                0
  php_value session.auto_start              0
  php_value mbstring.http_input             pass
  php_value mbstring.http_output            pass
  php_value mbstring.encoding_translation   0
</IfModule>

# PHP 5, Apache 1 and 2.
<IfModule mod_php5.c>
  php_value magic_quotes_gpc                0
  php_value register_globals                0
  php_value session.auto_start              0
  php_value mbstring.http_input             pass
  php_value mbstring.http_output            pass
  php_value mbstring.encoding_translation   0
</IfModule>

# Requires mod_expires to be enabled.
<IfModule mod_expires.c>
  # Enable expirations.
  ExpiresActive On
  # Cache all files for 2 weeks after access (A).
  ExpiresDefault A1209600
  # Do not cache dynamically generated pages.
  ExpiresByType text/html A1
</IfModule>

# Various rewrite rules.
<IfModule mod_rewrite.c>
  RewriteEngine on

  # If your site can be accessed both with and without the 'www.' prefix, you
  # can use one of the following settings to redirect users to your preferred
  # URL, either WITH or WITHOUT the 'www.' prefix. Choose ONLY one option:
  #
  # To redirect all users to access the site WITH the 'www.' prefix,
  # (http://example.com/... will be redirected to http://www.example.com/...)
  # adapt and uncomment the following:
  # RewriteCond %{HTTP_HOST} ^example\.com$ [NC]
  # RewriteRule ^(.*)$ http://www.example.com/$1 [L,R=301]
  #
  # To redirect all users to access the site WITHOUT the 'www.' prefix,
  # (http://www.example.com/... will be redirected to http://example.com/...)
  # adapt and uncomment the following:
  # RewriteCond %{HTTP_HOST} ^www\.example\.com$ [NC]
  # RewriteRule ^(.*)$ http://example.com/$1 [L,R=301]

  # Modify the RewriteBase if you are using Drupal in a subdirectory and
  # the rewrite rules are not working properly.
  #RewriteBase /drupal

  # Rewrite old-style URLs of the form 'node.php?id=x'.
  #RewriteCond %{REQUEST_FILENAME} !-f
  #RewriteCond %{REQUEST_FILENAME} !-d
  #RewriteCond %{QUERY_STRING} ^id=([^&]+)$
  #RewriteRule node.php index.php?q=node/view/%1 [L]

  # Rewrite old-style URLs of the form 'module.php?mod=x'.
  #RewriteCond %{REQUEST_FILENAME} !-f
  #RewriteCond %{REQUEST_FILENAME} !-d
  #RewriteCond %{QUERY_STRING} ^mod=([^&]+)$
  #RewriteRule module.php index.php?q=%1 [L]

  # Rewrite current-style URLs of the form 'index.php?q=x'.
  RewriteCond %{REQUEST_FILENAME} !-f
  RewriteCond %{REQUEST_FILENAME} !-d
  RewriteRule ^(.*)$ index.php?q=$1 [L,QSA]
</IfModule>

# $Id: .htaccess,v 1.81.2.3 2007/09/21 12:24:22 drumm Exp $

There are a variety of extra modules I've never seen before including the one you mention:
LoginToboggan

I get a warning on some admin pages about jquery not being installed correctly.

Thanks for your help!

dman’s picture

Yeah, nothing odd in the .htaccess.
No rewrites that I can see. Not related to clean URLs (same results either way)
Some death within some module running at init time is my current guess.

Sorry to say on a live site, but I'd have to start turning things off to narrow it down (without logs).
Or duplicate a full test site on the same server to experiment.
It's thorny.

.dan.
How to troubleshoot Drupal | http://www.coders.co.nz/

allen074’s picture

im sure you are right that it's a module issue - i will plug away at it more - thanks dman!

allen074’s picture

any other suggestions or thoughts before i start turning off stuff and who knows what else will break :)

allen074’s picture

ok so i turned off logintoboggan and it still spits out 500 error - i tried changing the theme - still 500 - any other ideas? im still shocked that the site is browsable - i've never seen this before

thanks

mmmtiger’s picture

Any more updates on this issue?
I'm having what seems to be the exact same problem. Thanks! Thanks!

egbertb’s picture

Did you ever find a solution to this problem. I am having the same problem on a 5.15 site. Upgraded tot 5.19 but problem persists.

Egbert - webdevelopment and implementation - www.overboord.nl

egbertb’s picture

Ok, I figured it out

Problem:
Your site disappears from the google index

-Site runs normal in a browser
-When you check your headers they throw a 500 error
-Googlebot throws a 500 error in your php error log
-You have a persistent error in your logs, something like:
PHP Parse error: syntax error, unexpected ')' in /var/www/vhosts/ggg,hdjdh,ext/httpdocs/includes/common.inc(1473) : eval()'d code on line 5, referer: http://www.noubreektmijnklomp.nl/krak
-Validating the site with a w3c validator gives you something like
I got the following unexpected response when trying to retrieve < http://www.noubreektmijnklomp.nl/krak >:
500 Internal Server Error
If you made recent changes to your domain name (DNS) configuration, you may also want to check that your domain records are correct, or ask your hosting company to do so.
-Your dns records are ok
- when you copy the html of a page and place it on your site (as html or as php) it validates correctly

Googlebot and the w3c validator simply do not get past the 500 header. So you have to find that error and correct it, or be banned from the googleindex for ever.
The error is probably somewhere in the php content on your site, f.i. the visibilty settings of a block (where you use php code), or php code for generating the content of a block. The log error message just states that code being evaluated in common.inc throws an error, not what code.
It can also be somewhere in a module (you can check that by simply disabling all contributed modules and see if the problem then persists.)
Or it can be somewhere in the templates (Change theme and see if the problem then persists)

Egbert - webdevelopment and implementation - www.overboord.nl

briinums’s picture

Has anyone tried changing the imagecache module to save files in different folder? (eliminating "imagecache" which could be detected by google :/ )

also keep in mind that your default robots.txt contains lines that block google from indexing images, hence lines "/files/" and "/sistes/".. even after removing these lines, my images still dont get indexed, while text content is indexed frequently..

rajarju’s picture

Thnx man..
I found the issue in one of my clients site
The issue was in the theme ..!!