By benone on

Have any of you tried it before ?
I have a content on my drupal site visible only after registering.
I would like to have this content indexed by search engines.
Is there a module or other tricky method to let bots crawl my full site ?

Comments

carnevaledesign’s picture

carnevaledesign commented

The search engine bots would have to log in which they aren't capable of doing.

cestmoi’s picture

cestmoi commented

I have been researching same feature and best solution that complies with Google guidelines. I hope this post of mine, with my research/experience so far in this area, would help anyone in need for the same functionality.

(please click through the links I posted in this post to see the issues I refer to re the different modules)

Googlebot access should be achievable using ip_login but this would need to set up ip's ranges while Google advises allowing access by referrer (best after checking using Reverse DNS) and not by ip's since there is no fixed ip range for googlebot. I haven't tested this module yet, however. If anyone did, please share.

Another module that is supposed to do the job too, I guess, is Role Referrer but after testing it on referring sites (not Googlebot), I couldn't make it work and there sadly does not seem to be any support for the module :(

A third module that should be helpful, and possibly best of all, would've been Sesame as it allows for access that fits best with Google requirements

This module allows you to set up access to the site on base of request referrer, remote IP address or User Agent with reverse-lookup check of domain name (for searchbots)

Unfortunately, the Sesame (sesame_access) module seems to have no support and a promised Drupal 6 port for it was promised to be released more than 1.5 year ago but never done !

The "First Click Free" ( FCF ) module - while it allows access of already crawled pages- does not seem to allow googlebot crawl the protected content. This or similar module will be needed though with any of the above mods to make sure complying with Google terms. The FCF module does work to allow searchers access members-only pages when users click on an already-crawled page on Google.

EDIT: I also like to add a further issue that needs consideration. If using xmlsitemap, the sitemap (at this time) does not support protected nodes either and so Google would not see and crawl those nodes through xmlsitemap. Unfortunately, I have yet to get any answer re fixing this issue !

It would be great if anyone has any further insights on this post.

Cheers