By ken512-1 on

I have a new site at http://www.kensgaming.com that I setup and google will not index it. I have traced the problem down and it appears for some reason my site is serving different data when the user agent is set to googlebot. I used the tool on this website http://web-sniffer.net/ to check my server headers and try different user agents. If I set the user agent to Internet Explorer, Mozilla, Safari, or none then status code 200 is returned along with my sites content. You can see that at this link. http://web-sniffer.net/?url=http%3A%2F%2Fwww.kensgaming.com%2Fmain%2Find...

That all works as expected. The problem is if you set the user agent to googlebot then suddenly my site returns 302 redirect error and displays no content. Because no content is displayed googlebot obviously has nothing to index. You can see that at this link. http://web-sniffer.net/?url=http%3A%2F%2Fwww.kensgaming.com%2Fmain%2Find...

I have Drupal 6.1 along with several modules installed. I can't figure out if this is a problem with Drupal or something caused by one of the modules. I have searched all over the internet for the last several days and I do find links referring to several modules such as the xml sitemap or pathauto, but I have neither of those installed. Here is my complete list of modules.

Flag content
Frequently Asked Questions
Google Analytics
Persistent Login
phpBBforum
Service links
Video Filter
CAPTCHA
Atom

Where inside Drupal would it display different content based on the user agent field? I have tried to search and can't find what might be causing this problem. Any help would be greatly appreciated.

⋅ Categories: Drupal 6.x

Comments

ken512-1’s picture

ken512-1 commented

Alright I think I found the problem. It had to do with the phpbbforum integration mod, because when I turn that off everything starts working. Specifically the problem is because the forum integration mod tries to sync the drupal users up with the phpbb database. The problem is phpbb3 identifies search bots so it can show when they are online. phpBB sees googlebot and assigns it as a registered user under the name google [bot], Drupal however sees googlebot as an anonymous user and assigns it as anonymous. So the forum integration mod gets confused and just keeps redirecting to infinity. As soon as I went into phpBB ACP under the Spiders/Robots section and disabled all the bots then suddenly everything works. I guess this is really a bug in the phpbbforum mod. I won't know for sure if the problem is fixed for a couple days when google tries to index me again, but I feel pretty good that I fixed it.