By mwu on

If I tell robots.txt to disallow all when building the site, and then later want site indexed, is there a problem?

I'm thinking of telling robots.txt not to index anything when I'm building the site. I don't want to just take the site off line because a large part of what I'm testing is what anonymous sees, blah blah.

If I disallow all, and then later want the site indexed by Google (and others), will there be a problem?

Comments

erdemkose’s picture

erdemkose commented

No, there won't be any problem. Because bots like Google, Yahoo! request robots.txt for each indexing. When you switched to normal robots.txt they will start indexing your site.

--------------------------------------------------------------
http://erdemkose.com/
Drupal Türkiye - Drupal Turkey

vizechef’s picture

vizechef commented

Depends on what you call a "problem" !
Keep in mind that spiders DO NOT HAVE to care about your robots-file.
Usually they do but they don't have to.
They can index any content they can access !