i dont want my site indexed until i am finished with it. there seems to be a large number of guests on my site. i added a robots.txt file and the number of guests immediately dropped way down. in the past week the number of guests has been getting higher than i think is realistic and i think it might be some new or other bots. i then 'beefed up' my robots.txt file and added:
Options +FollowSymlinks
RewriteEngine On
RewriteBase /
to my .htaccess
i still i have unrealistically high numbers of guests. so:
how does drupal 'sense' a guest and which files process that information and present it to the web page? in other words how can i 'see' the identity of the current guests whether they be bots or just anonamous viewers?
AND
is there anything else i can do to make robots go away for the time being?
here's my current robots.txt file
thanks.
=-=-=
User-agent: *
Crawl-delay: 5
Disallow: /
Disallow: /aggregator
Disallow: /tracker
Disallow: /comment/reply
Disallow: /node/add
Disallow: /user
Disallow: /files
Disallow: /search
Disallow: /book/print
User-Agent: Googlebot
Disallow: /
User-Agent: Googlebot-Image
Disallow: /
User-Agent: Googlebot-Image
Disallow: /images/
User-Agent: Scooter
Disallow: /
User-Agent: T-Rex
Disallow: /content1.html
Comments
there are hundreds (if not
there are hundreds (if not thousands) of bots that wont pay attention to the robots.txt file.
google bad bots and block their IP addresses. If you did not want any bots/guests at all, it would have benefitted you to develop on your local machine where it is untouched by internet activity.
Sessions
Any interaction with a Drupal site creates a PHP session. The number of guests reflects the number of currently active PHP sessions mapped to user 0 (anonymous). The "Who's online" block displays this number -- and the corresponding number for users whose uid is not 0, and lists up to N most recently active users. Both the value of N and the time window defining "active" are configurable in the block settings.
A well written spider should store and use cookies. Google's and Yahoo's spiders will do so, and they will show up as only one anonymous user at any given time.
The net is overrun by badly behaved spiders that actually are trying to inject blog spam or referer spam. If your anonymous user number suddenly spikes, that is probably the cause. Check your server logs, especially referer logs. Wikipedia has some suggested solutions.
perhaps securesite is what you need
I use the securesite module to block search engines (and other unwanted guests) before I'm ready. Only users with an account can see the site.