By Lcribb on

Hi all

In my admin stats I can see that the file robots.txt is being reached for constantly by spiders.

My questions are:
What is robots.txt?
What is it for?
Should I have it?
What use is it?
What should be in it?

Your help is much appreciated,

cheers
Lev
Http://www.GlobalDetail.com

Comments

pizdindim’s picture

pizdindim commented

Search engines look for "robots.txt" to tell them which files and directories they should index, and just as importantly, which they shouldn't. See this tutorial for more:

http://www.searchengineworld.com/robots/robots_tutorial.htm

As an exercise, try accessing the robots.txt file from some of the existing websitites you now visit.

Lcribb’s picture

Lcribb commented

Cool Thanks!

Lev

Http://www.GlobalDetail.com
Travel Articles and Blogs - For you and by you

JymBrittain’s picture

JymBrittain commented

This sounds like a response from an educator. Are you, by chance?
Jym
technology4teachers.com

boris mann’s picture

boris mann commented

See http://drupal.org/node/22265