Implementation of solr spellchecker

Sounds great! But I noticed you are only using the title field for building the dictionary. Is there a reason not to use the "text" field (and/or others)?

BTW am I right in that PHP5.2 required for the new SolrPhpClient?

Well, a simple regex would be
/\w+:(\([^)]+\)|\w+)/i
But since colons inside quotes should remain (and parentheses can be nested, and...), a completely correct regex isn't possible, this would require a grammar and therefore a complete funciton dedicated to parsing the query for this syntax. But if the Solr_Base_Query::parse_query() functions correctly, this could easily be reduced to just checking each field for this case, which might even be possible with just a regex.

Although this seems very nice, I *feel* that hacking away at the SolrPhpClient code apart from using Solr1.3 (which is not really out yet?) adds a touch too much of complexity; I would move for us to work on the simpler solution on #230380: spelling suggestions instead, which relies on Solr1.2 and only slight modification; perhaps that other code can be worked on to provide some of the neat stuff this patch proposes?

What do you think? For now I am marking as duplicate while we come to an agreement (we can revert this).

Thank you for your comments.

BTW this is not my *decision* but rather my *opinion*. My suggestion is to continue this thread over in the (older) issue I mentioned to focus the discussion, and not at all to stop this (on the contrary!) =) I am just tyding up =)

As for the code, I realize these are exciting benefits, however code should be modularized (I know, slowing things a bit) to keep it sustainable; for example, when a new SolrPhpClient comes out, how would we manage this great code you put in? Right now the mantainers of this module have contacted the SolrPhpClient developer(s?) and in the future are going to try to influence that other project to put stuff like this in there.

For now, ?is it possible to extract the code you put into your modified SolrPhpClient to another place? To one of the existing modules, or a new module inside the contrib/ directory.

@brainski: Solr1.3.0 support is now go. Since you proposed this for 1.3 and I was working on the same thing for Solr 1.2, do you think you want to still collaborate on bringing this to the current dev version?

If not, and if it's ok, I will continue the great work you started... let us know, ok? Thanks.

Please feel free to propose changes to the core query (for example, a parameter that sets the handler) if it makes it easier to elegantly implement these features.

This looks like a very nice feature, but will need to be reworked a bit because I'm about to put a patch in to use the dismax handler which changes a lot of code.

I also see some style/architecture nits - such as using a global var to pass around the suggestions.

I don't think you need a new request handler, you just need to add spell checking capabilities to the standard handler.

We should not alter Service.php.

@Jacob- the standard solrconfig.xml already defines the spellcheck handler:

  <!-- a request handler utilizing the spellcheck component -->
  <requestHandler name="/spellCheckCompRH" class="solr.SearchHandler">

Yes, but you're not meant use it.

it is an example. You can copy the properties inside of it, and add it to your standard search handler if you want spell checking.

From http://svn.apache.org/repos/asf/lucene/solr/trunk/example/solr/conf/solr...

<!-- A request handler utilizing the spellcheck component.  
  ################################################################################################
  NOTE: This is purely as an example.  The whole purpose of the SpellCheckComponent is to hook it into
  the request handler that handles (i.e. the standard or dismax SearchHandler)
  queries such that a separate request is not needed to get suggestions.

  IN OTHER WORDS, THERE IS REALLY GOOD CHANCE THE SETUP BELOW IS NOT WHAT YOU WANT FOR YOUR PRODUCTION SYSTEM!
  ################################################################################################
  -->

Also, there are whitespace issues (tabs vs. spaces, I think) in the patch.

Looking here: http://svn.apache.org/repos/asf/lucene/solr/trunk/example/solr/conf/solr...

the HEAD version of this has more useful comments in solrconfig.xml:

  <!-- A request handler utilizing the spellcheck component.  
  ################################################################################################
  NOTE: This is purely as an example.  The whole purpose of the SpellCheckComponent is to hook it into
  the request handler that handles (i.e. the standard or dismax SearchHandler)
  queries such that a separate request is not needed to get suggestions.

  IN OTHER WORDS, THERE IS REALLY GOOD CHANCE THE SETUP BELOW IS NOT WHAT YOU WANT FOR YOUR PRODUCTION SYSTEM!
  ################################################################################################
  -->
  <requestHandler name="/spellCheckCompRH" class="solr.SearchHandler">
    <lst name="defaults">
      <!-- omp = Only More Popular -->
      <str name="spellcheck.onlyMorePopular">false</str>
      <!-- exr = Extended Results -->
      <str name="spellcheck.extendedResults">false</str>
      <!--  The number of suggestions to return -->
      <str name="spellcheck.count">1</str>
    </lst>
    <arr name="last-components">
      <str>spellcheck</str>
    </arr>
  </requestHandler>

I think this is what Jacob is referring to - we should add this component to our usual request handler, not query it separately.

i.e. like this.

whoops - cross-posted above.

how about something like this for the changes to the config files?

Useful reference pages:

http://wiki.apache.org/solr/SpellCheckComponent
http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters
http://wiki.apache.org/solr/SpellCheckerRequestHandler

here's a patch w/ no UI - enable spelling via $conf['apachesolr_search_spellcheck'] = TRUE; in settings.php. You may also have to issue a query directly to solr to build the spelling index.

a little cleaner code and comments.

now w/ a little UI to enable/disable spellcheck.

actually - I was trying to be too clever here. The substring replacement will fail (wrong offset) if there is more than 1 replacement and any but the last have a length different from the misspelled word

don't need 'all' params in the response except for debugging.

need to move the build directive to make it work.

committed #31 to 6.x

Nice. Two things.

1: Can we change

Did you mean:  food

to

Did you mean <em>food</em>?

2: Can we put the spelling suggestion in the current search block as well?

Indexing documents from an empty index is now über-slow.

The solr log shows that the spell index is being built continuously (for every document, it seems!)

I would sugest for spell index rebuilding be called with cron; also perhaps let admins rebuild the spell index manually (and advise them to do so in the Drupal status report?)

Will work on a patch for this.

@Janusman - yeah, it's set to rebuild on each commit now. Unfortunately, the option to build on optimize is not going to be available until Solr 1.4

An earlier patch above had a rebuild-on-cron option. Alternatively, you can omit the rebuild directive from your solrconfig.xml for now.

Included is a patch against a fresh checkout of DRUPAL-6--1 to address some of the previous concerns:

• reverts rebuilding the spellchecker index to cron runs (once daily) only, and also adds a local menu task (tab) to manually rebuild by the admin.
• adds the "Did you mean XXXX?" line to the Current Search block, like @robertDouglass requested. However I did not actually choose to do hard markup () to what @pwolanin already added, because it can be styled via CSS.

Marking as Code needs review.

It would be better to not have the rebuild be a GET request - it should happen on form (or confirm form) submission.

I would still include a question mark and get rid of the : in the did you mean sentence. You should also put the spelling suggestion inside of the t() function. Here's what that line should look like, in my opinion:

$output .= '<div class="spelling-suggestions">' . t('Did you mean %suggestion_link?', array('%suggestion_link' => $suggestions_search_link)) . '</div>';

Also note that I've generally adopted the D7 coding convention of surrounding the . operator with one space on each side.

@robertDouglass: Re: #40; totally right, will fix.

I've been thinking a bit abot the UX side of things...

I think that the correct thing is not put this in the "Your Current Search" block; my reasoning is because a suggestion is not "Your Current Search" but actually a "future" search that (I think) actually belong with the facets (they also represent "future" searches, stemming from the current one).

I propose this:

• When a search returns 0 results, the suggestion should be prominent, along/instead of the "Your search returned no results" message returned by search.module. Similar to this: http://biblioteca.mty.itesm.mx/pasteur/en/search/apachesolr_search/ererwe (an earlier version of this patch for D5).
• When a search returns >0 results, also provide a block with "just" the suggestion, so that site admins can choose to place it wherever (with the facets, for example?). Perhaps this could then (later on) mutate into not just spellchecking but also recommending additional (popular? complementary?) searches.

Any thoughts? Will wait a few and if no comments then I'll roll a patch and see if y'all like it =)

@janusman: I still like the idea of putting the spelling suggestion in current search for zero results. I dislike the idea of another block. We're drowning in blocks. It makes one more step that you have to administer and configure, whereas putting it in current search "just works".

Subscribing.

Since we are moving to 1.4, we should take probably advantage of the on-Optimize building of the search index instead of doing it on cron, since I added a cron hook to optimize once per day.

subscribing

opened a separate issue for building on optimize: #375991: Use 1.4 feature - generate spelling index on optimize

New patch rolled for current 6.x-dev, following up on comment #42.

"in current search for zero results"

I don't see this check.

Could you elaborate? @robertDouglass 's comment was "I still like the idea of putting the spelling suggestion in current search for zero results", and that's the way it is.

See attached screenshot, perhaps we can discuss over it.

@Janusman - it looks in your patch like the suggestions will always be in the current search block. My understanding of Robert's suggestion was to only display them in the block if there are zero results.

IT seems to me we want to use spellcheck.onlyMorePopular

And I think we should use a threshold for showing. Only showing with zero results doesn't always work great. For instance, search for druppal on this site :)

spellcheck.extendedResults will also give us a frequency and the origFreq so

search for druppal

origFreq = 20 (if 20 nodes use the word druppal)
and
freq (for Drupal) = 100k or somehting

which would give us a good indicator of an error. However, this may incur a performance hit (I dunno), and I'm not certain how you know that 100k is a big # and 20 is a small one. What if the index is 50,000,000. In other words, if I have a lot of documents, then the difference is not as great.

and I'm not certain how you know that 100k is a big # and 20 is a small one. What if the index is 50,000,000. In other words, if I have a lot of documents, then the difference is not as great.

I think the size of the index hasn't got much to do with it - a search with 100k results is in any case much more popular / likely to be correct than one with 20. I think just using the origFreq/freq ratio and maybe a threshold (e.g., no suggestions for origFreq > 1k) should work.

But the performance penalty sure is something to keep in mind and test before implementing.

@Jacob - I thin the idea was to show it in the current search result when there are zeros hits - it.e. as an extra hint that there might be an error.

Probably onlyMorePopular is a good idea too - otherwise we are likely to be returning as many spelling errors as corrections.

This feature is already implemented. Closing out.