A standard robots.txt example file to be added to the tarball and deployed in the docroot as requested in the mailinglist (http://lists.drupal.org/archives/drupal-support/2004-03/msg00654.html)

CommentFileSizeAuthor
robots.txt915 bytesbertboerland

Comments

gerardryan’s picture

gerardryan commented

Why not handle this in the database?
I have a working, form editable model -- that could make a great module -- that I'm using in my current project which includes a robot trap. I use a simple .htaccess mod_rewrite redirect to grab any requests for the file.

I can give you some example code.

robert castelo’s picture

robert castelo commented

"that could make a great module"

Yes please, I was thinking of doing something like this. At the moment I've set this up manually, but I think it would make a great Drupal feature.

I notice that drupal.org is being increasingly hit by aggressive spiders, so might be something the maintainers would be interested in.

matt westgate’s picture

matt westgate commented
Project: Path » Drupal core
Component: Code » other
bertboerland’s picture

bertboerland commented

some discussion on this has been going on on the develop / infra list. most likley to be picked up, will update when relevant.

bertboerland’s picture

bertboerland commented
Tobias Maier’s picture

Tobias Maier commented

i think it would be great to have a basic robots.txt included in drupal which could be replaced by modules...

bertboerland’s picture

bertboerland commented
Status: Postponed » Fixed

is in 5.0

Anonymous’s picture

(not verified) commented
Status: Fixed » Closed (fixed)