By allen074 on
I am helping a new client with some updates to their site and one of the biggest issues they are having is that they aren't indexed in Google. In looking at Google's Webmaster Tools, it appears that since they moved to drupal and a new server last December, they haven't been indexed.
I looked at the robots and htaccess and those match what I run on my drupal site which is indexed daily.
The previous developer said he thinks it has something to do with the views module.
They are running Drupal 5.3. Any ideas would be greatly appreciated.
Thanks!
Comments
ok i ran a header check and
ok i ran a header check and it came back with
500 INTERNAL SERVER ERROR
hrm - any ideas where to look at for this since i dont see it when i view the page?
thanks!
yeah apache error log
yeah apache error log
Can't help debug much if you
Can't help debug much if you don't want to post an URL.
It's odd that you are able to see the site yourself but the robots are not. We need real header investigation. Is the domain redirected in some way? It's possible (I guess) for that to be malformed in some way that robots don't like but the browser allows.
Are you sure there is no deliberate access control happening? In Drupal or even earlier. It could be an anti-leech script at the Apache level or .. well, we need an URL to probe to really see more.
.dan.
How to troubleshoot Drupal | http://www.coders.co.nz/
.dan. is the New Zealand Drupal Developer working on Government Web Standards
url
url is:
http://www.jcreport.com
thanks for the help!
Yeah, that's wack. dman:~$
Yeah, that's wack.
But the page comes up in FF fine.
Something is screwy in the server. Could be it's bouncing requests that don't have the right headers (user-agent?) .. not really.
responds as expected however.
Something about the HTTP protocol?
... Hm, same even if I enter the identical headers from the commandline as wget sent.
works OK as expected! Same for both www and not (no redirects that I can see)
That's messed up.
Even worse, requesting files directly
wget -d http://jcreport.com/files/imagecache/small_w150px_sq/files/article_thumbs/liberty1.jpgDOES work. So yeah, it's looking like not Apache, but Drupal-specific!
Is there anything custom in your .htaccess?
You must have some module in there that is expecting something that's not available during non-interactive sessions. Any third-party user-tracking? logintoboggin?
Creepy.
I guess the apache logs for the error are the only clue. You may try bumping up the error-reporting level in PHP to return to the screen... but it's looking like the death is fairly early on in the request, possibly before drupal bootstrap really kicks in.
Anything custom in the settings.php?
This is also meaning that thinks like the validator can't see anything either. Pretty serious malfunction.
The strangest thing is that real browser requests DO work!
:-(
.dan.
How to troubleshoot Drupal | http://www.coders.co.nz/
.dan. is the New Zealand Drupal Developer working on Government Web Standards
extra - I get a sort-of
extra - I get a sort-of correct 404 to 'not found' pages, not a 500. Not sure what that tells us.
It's a Header 404 with no content though, not the full (nice) 404 page the browser gets.
.dan. is the New Zealand Drupal Developer working on Government Web Standards
wow - what a reply - thank
wow - what a reply - thank you so much! here's the htaccess - pretty standard i think:
There are a variety of extra modules I've never seen before including the one you mention:
LoginToboggan
I get a warning on some admin pages about jquery not being installed correctly.
Thanks for your help!
Yeah, nothing odd in the
Yeah, nothing odd in the .htaccess.
No rewrites that I can see. Not related to clean URLs (same results either way)
Some death within some module running at init time is my current guess.
Sorry to say on a live site, but I'd have to start turning things off to narrow it down (without logs).
Or duplicate a full test site on the same server to experiment.
It's thorny.
.dan.
How to troubleshoot Drupal | http://www.coders.co.nz/
.dan. is the New Zealand Drupal Developer working on Government Web Standards
im sure you are right that
im sure you are right that it's a module issue - i will plug away at it more - thanks dman!
any other suggestions or
any other suggestions or thoughts before i start turning off stuff and who knows what else will break :)
ok so i turned off
ok so i turned off logintoboggan and it still spits out 500 error - i tried changing the theme - still 500 - any other ideas? im still shocked that the site is browsable - i've never seen this before
thanks
Having same Problem
Any more updates on this issue?
I'm having what seems to be the exact same problem. Thanks! Thanks!
Me too
Did you ever find a solution to this problem. I am having the same problem on a 5.15 site. Upgraded tot 5.19 but problem persists.
Egbert - webdevelopment and implementation - www.overboord.nl
Solution and root of all evil
Ok, I figured it out
Problem:
Your site disappears from the google index
-Site runs normal in a browser
-When you check your headers they throw a 500 error
-Googlebot throws a 500 error in your php error log
-You have a persistent error in your logs, something like:
PHP Parse error: syntax error, unexpected ')' in /var/www/vhosts/ggg,hdjdh,ext/httpdocs/includes/common.inc(1473) : eval()'d code on line 5, referer: http://www.noubreektmijnklomp.nl/krak
-Validating the site with a w3c validator gives you something like
I got the following unexpected response when trying to retrieve < http://www.noubreektmijnklomp.nl/krak >:
500 Internal Server Error
If you made recent changes to your domain name (DNS) configuration, you may also want to check that your domain records are correct, or ask your hosting company to do so.
-Your dns records are ok
- when you copy the html of a page and place it on your site (as html or as php) it validates correctly
Googlebot and the w3c validator simply do not get past the 500 header. So you have to find that error and correct it, or be banned from the googleindex for ever.
The error is probably somewhere in the php content on your site, f.i. the visibilty settings of a block (where you use php code), or php code for generating the content of a block. The log error message just states that code being evaluated in common.inc throws an error, not what code.
It can also be somewhere in a module (you can check that by simply disabling all contributed modules and see if the problem then persists.)
Or it can be somewhere in the templates (Change theme and see if the problem then persists)
Egbert - webdevelopment and implementation - www.overboord.nl
Same issue here
Has anyone tried changing the imagecache module to save files in different folder? (eliminating "imagecache" which could be detected by google :/ )
also keep in mind that your default robots.txt contains lines that block google from indexing images, hence lines "/files/" and "/sistes/".. even after removing these lines, my images still dont get indexed, while text content is indexed frequently..
Thnx man.. I found the issue
Thnx man..
I found the issue in one of my clients site
The issue was in the theme ..!!