I am try to get our Google Mini to index our Drupal site, including pages that require authentication. I was hoping this module would make that possible, but after installing the module, the Google Mini reports a 4xx error on those pages.
Is this a feature available in this module?

Comments

dugnap’s picture

dugnap commented
Title: searching pages require authentication » searching pages that require authentication
larskleiner’s picture

larskleiner commented

You can configure user names and passwords to be passed to sites that get crawled within the Google Mini admin console. In my version it's under Crawl and Index > Crawler Access.

dugnap’s picture

dugnap commented

Thanks for responding. I did create a user for the search engine and gave it permission to access the members-only area, and configured the Google Mini to match the user name and password. Unfortunately, it doesn't work for me. I have read that Google Mini is not compatible with Drupal's user authentication (but that the full GSA is compatible). Is this something you have done yourself with the Mini?
I was able to get the Mini to crawl password protected pages using IPAuthenticate and putting the session cookies in the http header of the Mini.
Thank you again for taking the time to respond.

larskleiner’s picture

larskleiner commented

The Google Mini we use crawls a Samba share via NTLM HTTP. We configured the Samba user credentials in Crawler Access and it works fine.

I don't think there is an easy way to control the Google Mini crawler behaviour through Drupal. The Mini receives authorisation to crawl content either from a web server or Samba, without Drupal ever taking part in this process.

This link might be helpful:
http://code.google.com/apis/searchappliance/documentation/62/QuickStart/...