By kc on

I keep getting this in my error logs. Is any of the code looking for that. I know that it is for search engines, but I cannot see a robots.txt in the distribution. Or was there one and I deleted it or something?

Thanks

KC

Comments

Morbus Iff’s picture

Morbus Iff
they/them
commented

As you mentioned, robots.txt are strictly for search engines. They don't exist unless you make them exist (they're not default files in the webserver, or in Drupal). If you're sick of the 404's in your logs (I was), you can just create a zero-byte robots.txt file, and you'll be fine (nothing adverse will come of it).

http://disobey.com/
http://gamegrene.com/

http://www.disobey.com/

Nick Wilson’s picture

Nick Wilson commented

about not having a robots.txt file:

  • You may have sections of your site indexed that you'd rather not. like /node/add/ or /legal etc etc
  • Some paranoid folks think it harms rankings (i think this is rubbish but i cant prove it conclusively)
  • Robots.txt can be used to filter out the riff-raff, ban all those skanky lil bots that screw with your bandwidth but send you no traffic

Although saying that, morbus is of course right, it does no real harm at all.

Nick

kc’s picture

kc commented

Thanks guys, i figured so as well but wanted to double check not to miss anything important.

KC

Nick Wilson’s picture

Nick Wilson commented

Here's mine:

User-agent: *
Disallow: /user/
Disallow: /search/
Disallow: /spellcheck/
Disallow: /comment/reply/
Disallow: /node/add/
Disallow: /emailpage
Disallow: /taxonomy/
Disallow: /filter/

That keeps the bots out of pages I consider useless in terms of Search. If you copy this, you might have to remove the taxonomy line unless like me, ALL of your vocabs and terms are aliased...

Nick

Woodlums’s picture

Woodlums commented

so this is just some file that the "spiders" are looking for? It's nothing you need to have and no error that will keep you from being ranked or anything?

Where would you put the robots.txt file? What would that line look like?

Thank you!

Woodlums

m_freeman2004’s picture

m_freeman2004 commented

Hi,

I had the same problem, but just created an empty robots.txt file and uploaded it to the root directory.

If you google robots.txt you get this page, which was useful:

http://www.searchengineworld.com/robots/robots_tutorial.htm

Regards,

Marcus.

scriptnews’s picture

scriptnews commented

Hi

where would I put the robots.txt for multisite installations.

If I put it to the root, then it would be the same for all sites.

But if I want to have different instructions for every site within the multi-site installation, then I would need a different approach.

Please be so kind to share suggestions or experiences ...

Would symlinks help, and how would this work in this case

Idea:

make robots.txt in root a symlink to a php script,
which in turn would produce the desired output
for each site from the respective

/sites/www.example.com/robots.txt

For the normal Index/Noindex stuff etc, we could use meta-tags either, but what about instructions for Google Media-Partner.

Thanks in advance for ideas or solutions

Roland

freyquency’s picture

freyquency commented

I am sick of the error messages in my multisite installation as well.