The 2026 Drupal Business Survey is open. Your response will help shape the future of the Drupal business ecosystem.
By MountainX on
How do I allow Google and other search engines to index (but not cache) my content when I want to require registration (and admin approval) to view all content?
Comments
_
http://lmgtfy.com/?q=google+index+password+protected+content
Drupal specific help needed
Thanks for the appropriate search terms (and the cool site lmgtfy -- although isn't it rude to assume everyone who asks a question is too lazy to search?).
Adding "Drupal" to that query leaves no helpful results... so I need some Drupal specific help. E.g., where would I put some php code like what is shown here in Drupal? Will this even work in Drupal.
_
Not sure if it's rude, but with a large part of the hundreds upon hundreds of posts I read and respond to being exactly that (from people who couldn't be bothered searching first), it provides a bit of comedy in addition to a gentle nudge.
This isn't a drupal issue which is why there's not much drupal specific info. It's a google / se issue. Google spiders a site as an anonymous user-- if there's a way around that, it would be google info applicable to all sites, not just drupal.
There are ways around it
There are ways around it, but they tend to be application specific. See this link: http://davidwalsh.name/google-password-protected-areas
The suggestion is to do this:
$allow_inside = ($is_logged_in) || substr_count($_SERVER['HTTP_USER_AGENT'],'Googlebot');I understand the idea, but I do not know enough about Drupal to be able to implement that. That's why I need some advice specific to Drupal. Where would I put this code? Which file? What are the Drupal variables? Etc.
_
Besides the enormous security hole that enables, afaik, drupal's access control cannot be bypassed by setting a variable. You're better off doing something like the http://drupal.org/project/premium module does.
Thank you - premium module
Thank you. the http://drupal.org/project/premium module looks interesting.
re-think
A bit rude, maybe. But seldom incorrect when faced with open questions that don't show evidence of doing their own homework. Kudos on an accurate post title however!
Anyway...
There is a problem in what you are asking - in that it's deliberately cheating search engines. Giving the search engine one result and humans another is pure lying, and is actively punished by Google ... when it can detect it.
ExpertSexchange.com got away with this for ages and skewed search results with a bait and switch. They lasted so long by setting a cookie and allowing a "first one's free" sorta preview. But even they couldn't prevent Google-caching, and it seems that these days they have been penalized and ranked down a lot from where they were last year. If you want Google to index it, then Google can cache what it indexed. If you don't want that, then don't get it indexed.
Trying to cheat that system (although temporary exploits do exist) can only be regarded as 'black-hat' SEO, and you might not find many legitimate web professionals to help you do that. It is extremely likely to get you blacklisted.
Just saying, you may be on your own there. If you don't know enough code to implement this plan yourself, you probably don't understand why it's a technically bad idea.
I understand that you think you are just asking for technical answers, not a moral judgment ... but this advice is honestly relevant.
.dan. is the New Zealand Drupal Developer working on Government Web Standards
_
yeppers-- oddly enough I was purposely staying away from my abhorrence to the old bait and switch but dman is quite right-- it's pretty shady. Trying to have your cake and eat it too usually is.
you know what they say about assuming...
OK, I can understand that someone would not know I searched for more than an hour before posting my question, so I appreciate the recommendation to describe what I have done so far.
In regard to the moral judgment, you are way off base. You assumed something that doesn't apply. You could have simply asked why I wanted to do this and saved yourself a lot of wasted ranting.
(You also made another assumption that someone who doesn't know Drupal-specific coding is almost surely an idiot. Maybe you should rethink that.)
I took my site offline a couple days ago and I did not intend to put it back up. But there is some valuable content there (thousands of articles) and I got a lot of requests to put it back up. For some very specific reasons, the only way it makes sense to put it up is to have people agree to a disclaimer (and that means they need to register, which is free). But if the site is not indexed by Google, then I see no value in putting it up and paying for hosting and putting all that time into it. So the only solution I thought of is the one I asked about. As far as I am concerned, I'm doing a service (even spending my own time and money to give people free content) and all I require is that people agree not to violate certain regulations in order to see the content they want to see.
(My site has no ads and I make no money. I am offering a free service. Furthermore I have invested a large amount of money and 4 years of full time work to create the free content.)
move forward
And this demonstrates why it's more constructive to describe the goal, not the step.
There are other solutions to requiring a disclaimer! Many of which are more SEO-friendly. If what you really want is an acceptance of terms click-through, it may be that you don't need passworded signups, and you can give Google a proper view of the content.
My points were mainly technical, making the point that you still don't acknowledge (or understand?) that deliberately providing false information to a search engine is prohibited in their terms of use, harmful to the web, and more than just not working, may in fact penalize you. That's free advice.
However altruistic your stated motives, what you describe is "bad behaviour", just as someone sending out a million emails of information they genuinely think is important would rate as spam. It's not about motives, content, demand or commercialism - it's about method. Although you may think you are doing the world a favour, this behaviour is the same as that of amateur SEOs.
I am stating this again, because I don't need to call you an idiot - I need to make a point that you didn't seem to be aware of:
Such arbitrary (and unfortunately condescending) statements can be made in many professions, not just web coding. I've gone out of my way to explain the why of this. Exceptions can exist, but only after you know the rules in the first place. Work forwards and there may be better solutions to your actual challenge than trying to deceive Google.
.dan. is the New Zealand Drupal Developer working on Government Web Standards
If you want to do this project
If you want to do this project, see paid jobs forum here:
http://drupal.org/node/560650
You can add but if the pages
You can add
but if the pages are indexed (which you want) then anyone searching for the term can enter the site and view your content that way.
You could also use the above while also permitting full access by specific search engine IPs while all others require login. This is not an ethical dilemma either-it's not cloaking. Your visitors may view the exact same content that the SEs do and it's done by a number of highly ranked subscription sites.
Take a look at First Click for Free here: http://googlewebmastercentral.blogspot.com/2008/10/first-click-free-for-...
That's good advice
Yeah, look into that!
.dan. is the New Zealand Drupal Developer working on Government Web Standards