Google DoS: .htaccess RewriteRule, globalredirect or GoogleBot?

By hass on 3 Mar 2007 at 23:58 UTC

I have extracted this small part from my weblogs, and it looks like GoogleBot does something like a DoS, i don't know!? It looks like this is related to the .htaccess setting or globalredirect.module. I have no clue what the real source of this problem is. Someone experienced this problems, too? GoogleBot requests ~330.000 times the "/" path in one night... and the site only have ~50 webpages at all... every second ~5 requests to the same URL :-(((.

if i'm manually check the URL "http://www.yaml-fuer-drupal.de" with LiveHTTPHeaders i cannot see this 301 redirect. therefor i don't understand what's going on there and where is my possible fault... i have configured this rewrite rule in .htaccess:

RewriteCond %{HTTP_HOST} !^www\.yaml-fuer-drupal\.de$ [NC]
RewriteRule .* http://www.yaml-fuer-drupal.de/ [L,R=301]

207.210.234.135 - - [28/Feb/2007:16:10:40 +0100] "GET / HTTP/1.0" 301 332 "http://www.yaml-fuer-drupal.de" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
207.210.234.135 - - [28/Feb/2007:16:10:43 +0100] "GET / HTTP/1.0" 301 332 "http://www.yaml-fuer-drupal.de" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
207.210.234.135 - - [28/Feb/2007:16:10:43 +0100] "GET / HTTP/1.0" 301 332 "http://www.yaml-fuer-drupal.de" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
207.210.234.135 - - [28/Feb/2007:16:10:44 +0100] "GET / HTTP/1.0" 301 332 "http://www.yaml-fuer-drupal.de" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
207.210.234.135 - - [28/Feb/2007:16:10:44 +0100] "GET / HTTP/1.0" 301 332 "http://www.yaml-fuer-drupal.de" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
207.210.234.135 - - [28/Feb/2007:16:10:44 +0100] "GET / HTTP/1.0" 301 332 "http://www.yaml-fuer-drupal.de" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
207.210.234.135 - - [28/Feb/2007:16:10:44 +0100] "GET / HTTP/1.0" 301 332 "http://www.yaml-fuer-drupal.de" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
207.210.234.135 - - [28/Feb/2007:16:10:45 +0100] "GET / HTTP/1.0" 301 332 "http://www.yaml-fuer-drupal.de" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
207.210.234.135 - - [28/Feb/2007:16:10:45 +0100] "GET / HTTP/1.0" 301 332 "http://www.yaml-fuer-drupal.de" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
207.210.234.135 - - [28/Feb/2007:16:10:45 +0100] "GET / HTTP/1.0" 301 332 "http://www.yaml-fuer-drupal.de" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
207.210.234.135 - - [28/Feb/2007:16:10:46 +0100] "GET / HTTP/1.0" 301 332 "http://www.yaml-fuer-drupal.de" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
207.210.234.135 - - [28/Feb/2007:16:10:46 +0100] "GET / HTTP/1.0" 301 332 "http://www.yaml-fuer-drupal.de" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
207.210.234.135 - - [28/Feb/2007:16:10:46 +0100] "GET / HTTP/1.0" 301 332 "http://www.yaml-fuer-drupal.de" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
207.210.234.135 - - [28/Feb/2007:16:10:46 +0100] "GET / HTTP/1.0" 301 332 "http://www.yaml-fuer-drupal.de" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
207.210.234.135 - - [28/Feb/2007:16:10:47 +0100] "GET / HTTP/1.0" 301 332 "http://www.yaml-fuer-drupal.de" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
207.210.234.135 - - [28/Feb/2007:16:10:47 +0100] "GET / HTTP/1.0" 301 332 "http://www.yaml-fuer-drupal.de" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
207.210.234.135 - - [28/Feb/2007:16:10:47 +0100] "GET / HTTP/1.0" 301 332 "http://www.yaml-fuer-drupal.de" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
207.210.234.135 - - [28/Feb/2007:16:10:51 +0100] "GET / HTTP/1.0" 301 332 "http://www.yaml-fuer-drupal.de" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
207.210.234.135 - - [28/Feb/2007:16:10:54 +0100] "GET / HTTP/1.0" 301 332 "http://www.yaml-fuer-drupal.de" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
207.210.234.135 - - [28/Feb/2007:16:10:54 +0100] "GET / HTTP/1.0" 301 332 "http://www.yaml-fuer-drupal.de" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
207.210.234.135 - - [28/Feb/2007:16:10:54 +0100] "GET / HTTP/1.0" 301 332 "http://www.yaml-fuer-drupal.de" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
207.210.234.135 - - [28/Feb/2007:16:10:55 +0100] "GET / HTTP/1.0" 301 332 "http://www.yaml-fuer-drupal.de" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
207.210.234.135 - - [28/Feb/2007:16:10:55 +0100] "GET / HTTP/1.0" 301 332 "http://www.yaml-fuer-drupal.de" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
207.210.234.135 - - [28/Feb/2007:16:10:55 +0100] "GET / HTTP/1.0" 301 332 "http://www.yaml-fuer-drupal.de" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
207.210.234.135 - - [28/Feb/2007:16:10:56 +0100] "GET / HTTP/1.0" 301 332 "http://www.yaml-fuer-drupal.de" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
207.210.234.135 - - [28/Feb/2007:16:10:56 +0100] "GET / HTTP/1.0" 301 332 "http://www.yaml-fuer-drupal.de" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
207.210.234.135 - - [28/Feb/2007:16:10:57 +0100] "GET / HTTP/1.0" 301 332 "http://www.yaml-fuer-drupal.de" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
207.210.234.135 - - [28/Feb/2007:16:10:57 +0100] "GET / HTTP/1.0" 301 332 "http://www.yaml-fuer-drupal.de" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
207.210.234.135 - - [28/Feb/2007:16:10:57 +0100] "GET / HTTP/1.0" 301 332 "http://www.yaml-fuer-drupal.de" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
207.210.234.135 - - [28/Feb/2007:16:10:57 +0100] "GET / HTTP/1.0" 301 332 "http://www.yaml-fuer-drupal.de" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
207.210.234.135 - - [28/Feb/2007:16:11:01 +0100] "GET / HTTP/1.0" 301 332 "http://www.yaml-fuer-drupal.de" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
207.210.234.135 - - [28/Feb/2007:16:11:01 +0100] "GET / HTTP/1.0" 301 332 "http://www.yaml-fuer-drupal.de" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
207.210.234.135 - - [28/Feb/2007:16:11:01 +0100] "GET / HTTP/1.0" 301 332 "http://www.yaml-fuer-drupal.de" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
207.210.234.135 - - [28/Feb/2007:16:11:02 +0100] "GET / HTTP/1.0" 301 332 "http://www.yaml-fuer-drupal.de" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
207.210.234.135 - - [28/Feb/2007:16:11:02 +0100] "GET / HTTP/1.0" 301 332 "http://www.yaml-fuer-drupal.de" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
207.210.234.135 - - [28/Feb/2007:16:11:02 +0100] "GET / HTTP/1.0" 301 332 "http://www.yaml-fuer-drupal.de" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
207.210.234.135 - - [28/Feb/2007:16:11:03 +0100] "GET / HTTP/1.0" 301 332 "http://www.yaml-fuer-drupal.de" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
207.210.234.135 - - [28/Feb/2007:16:11:03 +0100] "GET / HTTP/1.0" 301 332 "http://www.yaml-fuer-drupal.de" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
207.210.234.135 - - [28/Feb/2007:16:11:03 +0100] "GET / HTTP/1.0" 301 332 "http://www.yaml-fuer-drupal.de" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
207.210.234.135 - - [28/Feb/2007:16:11:03 +0100] "GET / HTTP/1.0" 301 332 "http://www.yaml-fuer-drupal.de" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
207.210.234.135 - - [28/Feb/2007:16:11:04 +0100] "GET / HTTP/1.0" 301 332 "http://www.yaml-fuer-drupal.de" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

Comments

I have noticed this too

CharlieHipHop commented 4 March 2007 at 01:25

Not quite as severe as yours, but yes.

Wish I could tell you what to do about it. Short of banning the googlebot, I don't really see a solution.

Google DoS: .htaccess RewriteRule, globalredirect or GoogleBot?

Comments

I have noticed this too

New forum topics

News items

Our community

Documentation

Drupal code base

Governance of community