Hiding from search engines - URGENT - please help
janp13 - August 25, 2009 - 20:33
The site I run is a private site for families of students at a school. In the past it has been successfully hidden from random Net users but since switching to Drupal we have a problem.
The front page of the site still seems to be hidden, but the User Account page shows up in Google (the one with the options to: Create new account, Log in, and Request new password).
I'm hoping this will be simply a matter of putting the robots code in the appropriate place - but where do I put it?
I guess this is unusual in that most people want their websites found by search engines. :) This is done to keep our students safe from Internet preditors.
I would appreciate any help . . .
Cheers
Jan
... anyone?
... anyone?
Set up your web server to
Set up your web server to require http basic authentication; if you do this, users will have to enter a username and password into a pop-up dialog before any information from any page is displayed.
On the Drupal end, you can then use the Secure Site module to log into Drupal using the credentials entered in the http auth dialog. This will make your entire site inaccessible to search engines.
- Greg
Thank you for replying.
Thank you for replying.
I do still need users to be able to create their own accounts. I authenticate them and make them active.
In our old system users applied for an account and I had to go through a long process to set them up. With Drupal it's one click and save - so I don't want to go back to the old way of handing out usernames and passwords.
I just don't want Google (or any other) picking up the page. We give the web address to parents so they can create their own account.
I think with this method they wouldn't be able to?
But thanks anyway - I appreciate the answer.
Jan
robots.txt
robots.txt should already exist in the root htdocs directory of your web host. Edit it.
Disallow: /
See http://www.robotstxt.org/faq/prevent.html
Thank you
I will try this and let you know :)
Jan
Thanks for the link too
Interesting - I've always used the html meta tags in the way they describe. Not sure if they could be used with Drupal though :0)
I need to pass this on to my hosting services as I can't get into the htdocs any more. But I really appreciate your help - thanks again.
Jan
Umm, disable create new account?
Janp,
I am not sure exactly what the core problem is - my guess is you dont want especially the "create account" funtionality available to users ?
If that is the case, go to admin/user/settings and update the User registration settings to "Only site administrators can create new user accounts. "
If the problem is that the entire login block should not be available, check out http://drupal.org/node/315082#comment-1035933 and http://drupal.org/node/180962 <-- this is a helpful post.
Also, the previous poster's reply about secure site will be a great addition.
Hope this gets you some pointers.
Thanks for replying
I think this is similar to the answer above ...
I need users to be able to create new accounts - saves me so much time :)
But I don't want the search engines to index any part of the site.
Thanks again - I've definately got some pointers :)
Jan
You can use the Export Users
You can use the Export Users to DBM module to manage your Apache .htaccess file in Drupal; however, this only gets you part way to your desired solution, since as you noticed, the http basic authentication setup is going to block new users from creating accounts.
There are a couple of ways around this. Perhaps the easiest is to configure your http authentication such that it does not protect the home page or the pages that allow users to create new accounts. This is not too difficult, but does require familiarity with Apache authentication directives (see my original link above). If this is beyond your level of experience, perhaps you could hire a paid consultant to set it up for you. Probably wouldn't require too many hours of work. You'll need to make sure that your hosting provider gives you access to your httpd.conf or .htaccess file; then you should be good to go. (NOTE: Search engines would still see your sign-up pages in this scenario, so you'd need some way to identify real students from unauthorized users. You can 'solve' this with robots.txt as mentioned above, but note also that this file is advisory only, and some search engines / malware crawlers might not respect it.)
A slightly more "hacky" solution would be to create a well-known username and password that you hand out to your students. Configure this account so that it does not have any rights in Drupal except to go to the register new users page. I haven't tried this technique; you'd have to make sure that Drupal allows you to register a new user while you're logged in as an existing user. Some fiddling and experimenting would likely get you something that worked well enough for your purposes. This solution would have the added advantage that your registration pages do not show up in search engines.
Good luck,
- Greg
Will talk to my host
Thank you for this info. I'll send it to my web host and see what we can come up with.
I'm really surprised that this is happening.
I was alerted today when I received a bogus request for an account. Looked like it had been generated rather than human input.
Thanks again
Jan