How to exclude drupal site from search engine

In case you literary mean: not use the robots.txt file, but are ok with using the same technology, you could use the Robotstxt contributed module (http://drupal.org/project/robotstxt). By using this module, it is not the robots.txt file that is served to the spider of the search engine, but Drupal itself is generating the file. The purpose of the module is to serve different robots.txt files for different sites in case you run Drupal in multisite config.

Mark

_

Posted by tryitonce on September 6, 2010 at 8:42am

... just a thought. I use the following method to keep websites that are under construction out of search engines:

I password protect the root (public / www) directory on the server. That blocks the search engines - so far - and adds a second security layer giving access to those working on the project / site only.

-----------
_{Good luck .....

the results of trying Drupal just once are
www.mallsandmore.com
www.sds-i.com
www.proRotaTherm.com}

Robots.txt / SEO

Posted by ben.acheson on September 6, 2010 at 2:30pm

Create a text file called robots.txt - paste the text below, change 'privatefolder1' and 'privatefile2' to all the files/folders you wish to block, and drop the robots.txt into the root directory of your Drupal site:

User-agent: *
Disallow: /privatefolder1/
Disallow: /privatefolder1/privatefile1.htm

(If you want to disallow everything use this - use with caution):

User-agent: *
Disallow: /

http://benacheson.com

_

Posted by tryitonce on September 9, 2010 at 4:47pm

ben,

just a quick Q - what is the story behind your highlighted warning?? "(If you want to disallow everything use this - use with caution):"

Thanks ....

-----------
_{Good luck .....

the results of trying Drupal just once are
www.mallsandmore.com
www.sds-i.com
www.proRotaTherm.com}

You are simply telling all

Posted by groenm on September 10, 2010 at 6:56am

You are simply telling all search engine spiders: "Go away!"

That might not be what you want. The effect will be that your site will not show up any more in any of the better known search engines.

Mark

_

Posted by tryitonce on September 10, 2010 at 7:13am

Thanks Mark,

it's what I expected and what was part of the discussion. Since there was no explanation on ben's note I was wondering if there is another issue hidding somewhere.

The point was discussed that for virtual office websites for logged-in members SEO and crawling spiders are usually not wanted.

And to prove the point I recently changed the robots.txt file on an existing website that never really went live:

User-agent: *
Disallow: /

But Google duly ignores the changes and keeps crawling the site. So - Troll - http://drupal.org/project/troll and banning - http://www.yoursite.org/admin/reports/visitors these offenders seems to be the only way right now.

-----------
_{Good luck .....

the results of trying Drupal just once are
www.mallsandmore.com
www.sds-i.com
www.proRotaTherm.com}

Thanks for all the

Posted by raym0nd on September 7, 2010 at 3:55am

Thanks for all the replies.

As I thought there is a way to disable in the drupal settings itself.

_

Posted by WorldFallz on September 7, 2010 at 2:11pm

webcrawlers browse the site as anonymous-- uncheck the 'access content' permission for the anonymous role and webcrawlers can't see anything.

__{Don't be a Help Vampire - read and abide the forum guidelines.

If you find my assistance useful, please pay it forward to your fellow drupalers.}

_

Posted by tryitonce on September 7, 2010 at 3:07pm

.... but then user that are not logged in cannot see anything either.

I guess it depends on what you want to do.

We are creating some websites for (smallish) companies at present that need to have a "virtual office website".

These sites are used to exchange information between teams and members of teams. Thus the sites are private and the content should not be crawled = SEOed.

Apart form not doing any SEO work on these sites, we also use the double login described earlier, the robotxt exclusion and are always happy to learn some new tricks ....

-----------
_{Good luck .....

the results of trying Drupal just once are
www.mallsandmore.com
www.sds-i.com
www.proRotaTherm.com}

_

Posted by WorldFallz on September 7, 2010 at 3:18pm

but then user that are not logged in cannot see anything either.

huh? Just grant 'authenticated users' the 'access content' permission.

__{Don't be a Help Vampire - read and abide the forum guidelines.

If you find my assistance useful, please pay it forward to your fellow drupalers.}

_

Posted by tryitonce on September 7, 2010 at 3:32pm

.... as I said - it depends what you want to do.

So far we do not know what raym0nd wants to do.

May be he wants a site that is not searched for some proper or not reason and let those who know where to find it see at least the front page.

As far as I understand it without 'access content' permission you would not be able to see anything but the login page.

I would assume that is exactly what one would want - and what we do one some projects mentioned.

Let's see may be raym0nd will enlighten us a bit more and tell us what he is upto and how he will have succeeded in the quest.
.....

-----------
_{Good luck .....

the results of trying Drupal just once are
www.mallsandmore.com
www.sds-i.com
www.proRotaTherm.com}

_

Posted by WorldFallz on September 7, 2010 at 4:02pm

I totally read your previous content wrong, lol. I thought you were saying that logged in users couldn't see anything, duh.

In any case-- you can also use the http://drupal.org/project/front module to allow anonymous users see just the front page so you can add some text there about the site and how to 'register for more info' and such.

But no, there's really no 100% reliable way to stop search engine indexing while allowing anonymous 'user' browsing afaik.

__{Don't be a Help Vampire - read and abide the forum guidelines.

If you find my assistance useful, please pay it forward to your fellow drupalers.}

Actually one of the drupal

Posted by raym0nd on September 8, 2010 at 1:23am

Actually one of the drupal site I am working on is for internal company usage which I do not want search engine to to locate.

I know that robot.txt can do the job but I heard that drupal setting itself can set which I am curious to find out.

Can this method uncheck the 'access content' permission for the anonymous role ensure that search engine won't see anything? As my site require authentication.

_

Posted by tryitonce on September 8, 2010 at 7:30am

... I would recommend not to use anonymous role for an internal website. Having only logged in = authorized roles would give you better recording of what people are doing (have done).

In my mind anonymous role = is for public websites and on the forefront of SEO, promotion, marketing ....

In other words - if you want to keep a website private - than there should be no space for anonymous role.

-----------
_{Good luck .....

the results of trying Drupal just once are
www.mallsandmore.com
www.sds-i.com
www.proRotaTherm.com}

Thanks for the explanation :)

Posted by raym0nd on September 8, 2010 at 12:17pm

Thanks for the explanation :)

_

Posted by WorldFallz on September 8, 2010 at 1:39pm

robot.txt is not 100% -- search engines are not required obey it. The only 100% guaranteed way to not allow your content to be indexed is to remove access for the 'anonymous' user role.

__{Don't be a Help Vampire - read and abide the forum guidelines.

If you find my assistance useful, please pay it forward to your fellow drupalers.}