Support collation for Did You Mean spelling suggestions, and support multiple suggestions

Code didnt come through in original post. The XML in SOLR config to test this would be:

<str name="spellcheck">true</str>
<str name="spellcheck.onlyMorePopular">true</str>
<str name="spellcheck.extendedResults">true</str>
<!--  The number of suggestions to return -->
<str name="spellcheck.count">5</str>

Unified diff attached.

Bug identified in the previous attachment. Slight ammendment included in this version to cater for the suggestion object changing name.

This goes deeper than I initially thought once you start looking at the configuration options of the spelling suggestions component.

If <str name="spellcheck.extendedResults">true</str> then SOLR returns an object of the suggestions including a count of the number of results each suggestion will return. This final patch checks if the suggestions are an object or not. If so, it reads the object and displays the count of results as well as the phrases, if not it just returns the phrases from the array.

Sorry for multiple submissions so quickly!

Sorry, I've read up on the standards a bit more and attached a revised patch. Please let me know if there are any problems with this one.

I've been doing some more investigation into SOLR spelling suggestions, and have written more code to handle

<str name="spellcheck.extendedResults">true</str>
<str name="spellcheck.collate">true</str>

within the spelling suggestions to handle more possibilities in the SOLR configuration files.

Would you like that provided as an update to this patch (i.e. against the original), or as a diff against the patched version? Should I create a new issue for this?

First time patcher so this is all a little new to me!

Ok, I've attached the updated version of my patch which handles the different configuration options for SOLR spelling suggestions that a user can make. The reason for the patch is that if a user changes the SOLR configuration files without this patch then errors were being displayed when using the unpatched module.

The main change in this version of the patch is that when:

<str name="spellcheck.extendedResults">true</str>

then SOLR passes back a more complex object than the array that was being handled. Extended results means that the frequency of the spelling suggestions occurring in the index can be displayed to the end user.

Other changes are to handle:

<str name="spellcheck.collate">true</str>

Collation looks at the phrases for which spelling suggestions are being provided, and puts together the most likely group of phrases based rather than a single phrase. For example, if I were to search for "Druapl slro", with collate = false I would be given suggestions of "Drupal" and "solr" as individual phrases. With collate = true, solr will return "Drupal solr" as a collated term. In this patch, I have put the collated term first in the list of suggestions, as this is likely to be most relevant.

There is a patch for SOLR in the works which will enhance the collation functionality (https://issues.apache.org/jira/browse/SOLR-2010) which if it makes it into the full SOLR release may mean some refactoring of this patch will be required. The SOLR patch will allow multiple collations to be returned, and ensure that only collations which will return search results are returned.

I've also attached my solrconfig.xml with these parameters in, for reference.

Let me know if any further changes or information are needed.

Yes, there's an interesting issue with multiple search phrases. Essentially, for each word in the query SOLR will do a spell check. For each word which it thinks is spelt incorrectly it will return an item in the search response object containing the number of suggestions specified in <str name="spellcheck.count">5</str> in solrconfig.xml.

With collation enabled, another aspect this patch addresses (<str name="spellcheck.collate">true</str>) it will return the combined suggestions for the whole phrase using the most likely suggestion for each word in the query.

What I'm not sure about is how best this should be displayed to the user... if a user searches for a query with 5 words in it, all spelt incorrectly, should we return 5 x 5 = 25 spelling suggestions for every word? The approach i've taken with the patch is to return spelling suggestions for the first incorrectly spelt word in the phrase, and the collated spelling suggestion, so as not to overload the user. I'd be grateful for any suggestions.

As an aside, I'm using this module in conjunction with the Apache Solr Autocomplete module (which I am writing a similar patch for to enhance the spelling suggestions of the autocomplete) so the user gets instant feedback on their spelling mistakes which would mitigate the problem of only suggesting the first word... Any thoughts?

Are there any suggestions on how to proceed on this, or is the approach in the patch ok?

Having had a think, and experiment with a few different searches, one spelling suggestion seems to be the sensible way to approach this. If used in conjunction with the apache solr auto-complete module and the spelling suggestion collation then that should give the best chance of returning something useful to the user.

The flip side is that since SOLR offer the configuration option to return multiple results, maybe the module needs to detect whether collation is enabled and if so just display the collated correction and the original phrase. If collation is not enabled then it should return however many corrections the SOLR configuration specifies.

Yes, in SOLR config, there is an entry:

<str name="spellcheck.collate">true</str>

When set to true, in the SOLR query response there is a collation element in the response object (see the patch in #15 - it does a check for whether collated result was returned).