Multi-site search functionality [#867526]

For multi-site search, here is the plan that has been discussed:
1. Content from all relevant sites will filter into a single solr index.
2. Search boxes on _all_ sites utilizing that solr index will send search to drupal.org (i.e. association.drupal.org's search box will send users to drupal.org instead of searching directly on association.drupal.org)
3. No special effort will be made to explicitly hide the search pages on other sites, however, other sites will not have the project/module faceting available to them.
4. Results will link back to the relevant site.
5. Any potential faceting based on originating site will be discussed and mocks created.

Work that will be needed:
1. All sites on the same version of apachesolr
2. All sites directed to the same index
3. Force search box to post to drupal.org explicitly
4. Test apachesolr_multisite and determine if we should use and why
5. Determine if/how we'd facet for any off-site content
6. Create mocks for any faceting/site-specific elements
7. Determine if any special theming is required, particularly with regards to off-site content.

Comments

Comment #1

drumm

he/him

NY, US

commented 29 July 2010 at 04:19

3. Other sites should have the same facets near the search box, since they do all go to drupal.org. The refine your search menu is working on redesign.drupal.org/...

Otherwise, looks good.

Comment #2

pwolanin commented 29 July 2010 at 18:48

some comments on this plan:

I would be very frustrated if I want to find something on the association site and I cannot without doing a search and then filtering down by finding the 'site' facet in the sidebar after getting sent over to drupal.org.

The multisite module adds a second tab - so one option is to make that tab the default, but let uses switch tabs to search only the current site.

of course - the relative pain of varied implementations may choose the path for you too

Also on g.d.o currently there is a "search within this group" box - per-site searches like this are likely going to continue to be in demand and mean that the per-site search pages shouldn't be neglected unless you really want to send everybody over to drupal.or for every search.

The requirements seemed very implicit around multi-site.

One important note - you might make your life easier up front if you define the site hash values as something human readable (might as well do this today, so it slowly filters into the search index). This will make coding easier since e.g. when you are hacking in something to limit to d.o when starting with a multi-site search you can just add it like $query->add_filter('hash', 'd.o'), rather than having to put some random and hard-to-remember hash value into your code.

Comment #3

csevb10 commented 15 August 2010 at 00:19

After discussions with various people, the following implementation has been suggested:
1. Single index (of course)
2. Each site searches ALL content in the index
3. All initial results are displayed within the context of the originating site (i.e. if I search on association.drupal.org, I'll stay on association.drupal.org initially)
4. "Meta" facets (currently themes, modules, forums/issues, etc) will push to a particular site. At present, that means d.o, but in the future, a "groups" facet might push all users to g.d.o.
5. Users can filter the displayed contents based on the site value of any sites within the result set.

This should allow us the flexibility to do certain things such as:
1. Have robust theme filtering, module filtering, group filtering, etc as desired on a site-by-site basis.
2. Only implement that robust filtering on the site with the base information. (i.e. only implement theme filtering on d.o)

This will require us to implement at minimum - the following on each site utilizing the shared index:
1. apachesolr
2. apachesolr_multisitesearch
3. a helper module for meta-type faceting block

Comment #4

drumm

he/him

NY, US

commented 15 August 2010 at 02:34

For maintenance, we ideally want only one site to do search results. We do have some decent tools to keep all the sites in sync, but I don't want to depend on code going out to all at the same time. One site to do results, means only one to deal with when working on them.

If we do have some big advantages to doing groups facets on groups.drupal.org, then all facets being strongly linked to a site is acceptable.

5 - What would this look like UI-wise?

Comment #5

pwolanin commented 16 August 2010 at 16:29

Doing all the search on one site is certainly possible - as above, though, you lose some of the robustness and richness that a single site search can deliver.

For example, the per-group search could send you over to do where filters for the g.d.o site hash and the groupID could be applied. But you are now out of the context of the group and on a different site. note also - this is another good reason to start now to set human-readable site hashes for each site.

A middle ground solution would be to have (for example) only d.o serve multi site searches. So you limit the complexity of managing that full set of code to one site. For all other sites you just have the normal apachesolr module, plus break out a little piece of the multisite module that indexes meta data and adds the current site hash filter to all searches.

Comment #6

drumm

he/him

NY, US

commented 16 August 2010 at 17:50

I think the main requirement is that each page only comes from one site, so we aren't trying to keep things synced. So each meta-facet result page should be rendered by one and only one site.

(If someone manually switches around the URL, it's okay to render an unusual page, but we won't put resources into making sure it is correct.)

Comment #7

pwolanin commented 16 August 2010 at 18:04

I'm not sure what "3. a helper module for meta-type faceting block" means.

As above - I would suggest we either hack out a chunk of the multisite module, or re-architect it a little so that is can provide certain services (meta data indexing, etc) on all sites without actually providing a search tab, etc.

Comment #8

csevb10 commented 17 August 2010 at 01:00

I was suggesting that we carve out some of what is in drupalorg_search and move it into a separate module (if drupalorg_search cannot be leveraged in its entirety) so that every search would be presented with relevant meta-type faceting (modules, themes, forums/issues, etc) and not just when the search is on drupal.org.

In general, it's determining how much we need to do to each site in the index to get it to work conform with the multisitesearch.
--Bill

Comment #9

csevb10 commented 20 August 2010 at 22:23

Plan:
We're going to follow the original plan to push all sites sharing the index to d.o. for the time being. Sites with special requirements in terms of faceting will follow special rules and be handled once we have an applicable use case.

Work:
1. #889320: Move search box customization and meta-type handling into drupalorg_crosssite
2. #889314: Add apachesolr_multisitesearch to redesign
3. #889324: Re-work block placements to display on multisitesearch pages
4. #889328: Determine how site filtering will work/look (Issue)
5. #889336: Determine new options - if any - for meta-type faceting (Issue)
6. Bring in the first site and test in dev environments.
7. Document process for bringing new sites into the shared index.
8. Document expected functionality for bringing a new site with special faceting (similar to theme/module faceting on d.o.) into the shared index.