Google hasn't crawled my site since 12/12/06. I had Disallow / for the whole site.

2 days ago, I changed my robots.txt file to this and google still hasn't crawled since 12/12/06. Shouldn't the crawl happen daily?

[code]# small robots.txt
# more information about this file can be found at
# http://www.robotstxt.org/wc/robots.html
# lines beginning with the pund ("#") sign are comments and can be deleted.

# if case your drupal site is in a directory
# lower than your docroot (e.g. /drupal)
# please add this before the /-es below

# to stop a polite robot indexing an exampledir
# add a line like (delte the #'s)
# user-agent: polite-bot
# Disallow: /exampledir/

# a list of know bots can be found at
# http://www.robotstxt.org/wc/active/html/index.html
# see http://www.sxw.org.uk/computing/robots/check.html
# for syntax checking

User-agent: *
Crawl-Delay: 10
Disallow: /aggregator
Disallow: /tracker
Disallow: /comment/reply
Disallow: /node/add
Disallow: /user
Disallow: /files
Disallow: /search
Disallow: /book/print
Disallow: /node/
Disallow: /comment/
Disallow: /archive/
Disallow: /taxonomy/
Disallow: /files/
Disallow: /cgi-bin[/code]

Comments

uzbekjon’s picture

Hi,

Google does not crawle your site daily. Usually every week, sometimes longer...

Use google sitemaps to see statistics and crawling rates...

---------------------------------
Uzbek Lyrics Database - http://lyrics.boo.uz
Uzbekistan Forum - http://www.uzforum.com

JamesGardner’s picture

1. Submit the site to Google directly so it goes into their crawling queue (instructions @ Google.com)
2. Get inbound links pointing to your site from sites already known to Google
3. Be patient ... it's a big WWW that Google is mapping and organizing ;-)
------------------
adverlicio.us | online advertising archive
Powered by Drupal