Ability to control the 'Did you mean?' by asking Solr for spellcheck.extendedResults

Very interesting of course.

+++ b/apachesolr_search.moduleundefined
@@ -852,15 +856,27 @@ function apachesolr_search_get_search_suggestions() {
+        unset($suggestions['correctlySpelled']);

Can this be commented? Can we prevent this?

+++ b/apachesolr_search.moduleundefined
@@ -841,6 +843,8 @@ function apachesolr_search_process_response($response, DrupalSolrQueryInterface
+  $search_page = apachesolr_search_page_load('core_search');

This is also not wanted, since we do allow searches from more then 1 environment

I'll try to test it out now :-)

Rerolled the patch so it applies on the latest dev version.

There is however still one problem it seems. The following code did not validate on my system.

if ($min_freq >= $value->suggestion[0]->origFreq) {
  $replacements[$word] = $value->suggestion[0]->word;
}

I only got

0 (Object) stdClass
word (String, 5 characters ) testt
freq (Integer) 15

Maybe there is a difference between Solr 1.4 and 3.x?

The line if ($min_freq >= $value->suggestion[0]->origFreq) had the incorrect logic. Altered the patch in #2 to use correct logic.

Ok, I gave it another try with my standard test and it was not giving me the same result. We should have a way to enable or disable the advanced spelchecker so the original behavior does not get lost.

I'm a bit doubtful for the final implementation for this. It seems that the purpose is very specific and the UI to configure it will not be very clear for most of the people?

Are you able to do this without hacking the module? In other words, by using the hooks provided? You could easily make a new contrib module that extends this module and if I was you I would go for that solution?.

Would this be sufficient?

And now with a newline

@Nick_vh this hook doesn't allow us to add $params['spellcheck.extendedResults'] = 'true'; which is needed to get those extra parameters.

hook_query_alter? That should be sufficient to add that parameter?

/**
 * Alter the query after it's prepared and cached.
 *
 * Any module performing a search should call
 * drupal_alter('apachesolr_query', $query). That function then invokes this
 * hook. It allows modules to modify the query object and its parameters.
 *
 * A module implementing HOOK_apachesolr_query_alter() may set
 * $query->abort_search to TRUE to flag the query to be aborted.
 *
 * @param object $query
 *   An object implementing DrupalSolrQueryInterface. No need for &.
 */
function hook_apachesolr_query_alter($query) {
  // I only want to see articles by the admin!
  $query->addFilter("is_uid", 1);

  // Only search titles.
  $query->replaceParam('qf', 'label');
}

Ah yes, that does work :)

However, adding this parameter changes the structure of the suggestions array, making (apahcesolr_search.module +856):

foreach ($suggestions as $word => $value) {
  $replacements[$word] = $value->suggestion[0];
}

incorrect as the word is now under $value->suggestion[0]->word. Any ideas of how to get around this?

This patch fixes the above problem for both cases.

Without documentation this patch is as fuzzy as a patch could be? Would there really not be any other way?

Nope, because when you add the query parameter to return extended results it changes the suggestions from an array of strings to an array of objects.

Looking at the drupal_alter call on the suggestions as well, this doesn't provide what we need as that will only pass in the string that it will suggest. This means that none of the properties (origFreq etc) are available in the alter hook. It would be much better to do it like this:

function apachesolr_search_get_search_suggestions() {
  $suggestions_output = array();
  if (apachesolr_has_searched()) {
    $query = apachesolr_current_query();
    $keyword = $query->getParam('q');
    $searcher = $query->getSearcher();
    $response = apachesolr_static_response_cache($searcher);
    // Get spellchecker suggestions into an array.
    if (!empty($response->spellcheck->suggestions)) {
      $suggestions = get_object_vars($response->spellcheck->suggestions);
      if ($suggestions) {
        drupal_alter('apachesolr_suggestions', $suggestions, $env_id);
        $replacements = array();
        // Get the original query and retrieve all words with suggestions.
        foreach ($suggestions as $word => $value) {
          $replacements[$word] = $value->suggestion[0];
        }
        // Replace the keyword with the suggested keyword.
        $suggested_keyword = strtr($keyword, $replacements);
        // Show only if suggestion is different than current query.
        if ($keyword != $suggested_keyword) {
         $suggestions_output[] = $suggested_keyword;
        }
      }
    }
  }

  return $suggestions_output;
}

Sorry, wrong again. This patch includes the correct placing and variables to use in the alter hook so we have something useful, and the change in the foreach loop with documentation as to what it's for :)

I don't like this approach. It would be better if you actually remake the array so it fits in to the function properly.

@Nick_vh:
Could you please exand on that comment? We're flying in the dark here about what you want, and why you're rejecting our suggestions.

sorry if I sounded rude :-)

I would prefer that the logic of this $value->suggestion[0]->word happens in the alter.
We are adding query specific syntaxes here and that is not necessary.

I'd prefer if we reworked it to something similar like :

 // Get spellchecker suggestions into an array.
     if (!empty($response->spellcheck->suggestions)) {
       $suggestions = get_object_vars($response->spellcheck->suggestions);
       // pre processing could happen here so we have a flat suggestion array?
       drupal_alter('apachesolr_suggestions', $suggestions, $env_id);
       if ($suggestions) {
         $replacements = array();
         // Get the original query and retrieve all words with suggestions.
         foreach ($suggestions as $word => $suggestion) {
           $replacements[$word] = $suggestion;
         }
         // Replace the keyword with the suggested keyword.
         $suggested_keyword = strtr($keyword, $replacements);

This way we do everything in the alter and we can add more suggestion query weirdness if one would want?

I think you can address some of these problems through changes to Solr config.

Regarding the original issue:

E.g. a search for 'woods' (in 11 documents) would return 'did you mean? foods' (in 69 documents)

Try settings spellcheck.onlyMorePopular to "false". Solr keeps it off by default, but the config for apachesolr.module turns it on. It's a misleading option, which sounds helpful, but I find it leads to worse suggestions overall.

And regarding the other part:

This way an admin can easily suppress the 'did you mean?' to words that appear in fewer than 5 documents for instance, or alternatively only show for misspelled words (appear in 0 documents in the index).

Another possibility is to use a spellchecker config option called "thresholdTokenFrequency". Seems to be undocumented, but here is a stack overflow example). This tells the spellcheck component to ignore terms found in less than a certain percentage of documents.

This thread looks like it ran out of steam - did it get picked up somewhere else. In the meantime - can someone clarify if the suggestion to "Try settings spellcheck.onlyMorePopular to "false"" is a UI setting or actually in the module? I am assuming the latter but happy to get a nice surprise (though I can't find it)

@petednz: It's a Solr parameter that can be set in solrconfig.xml or overridden with URL parameters. The solrconfig.xml bundled with apachesolr.module sets it to true. You can use hook_apachesolr_query_alter() or hook_apachesolr_query_prepare() to override that setting.

sweet. really appreciate your prompt and useful response.

@milesw, I'm assuming you're running Solr 1.4? That feature doesn't seem to be available in Solr 3, or at least it's never worked for me.

@daniel.nitsche: I was using 3.x. Are you referring to the onlyMorePopular feature? If you changed it in solrconfig.xml don't forget you have to reload the config. :)

The only change I'm aware of is in Solr 4.x where you can optionally use the DirectSolrSpellChecker instead of the default IndexBasedSpellChecker. As I understand it, the onlyMorePopular param irrelevant with that new spell checker.

I completely lost track of this issue as I moved off the project that was using the patch. While there's valid suggestions as to what other things you can do with solrconfig and parameters to achieve similar functionality, an alter hook gives you far more power and flexibility. We've been using this extensively since the patch #22 was posted. This is the example of how we use it:

/**
 * Implements hook_apachesolr_suggestions_alter().
 */ 
function mymodule_apachesolr_suggestions_alter(&$suggestions, $env_id) {
  $incorrect_spelling_only = variable_get('mymodule_spellcheck_correctly_spelled', 0);
  $min_freq = variable_get('mymodule_spellcheck_min_orig_freq', '0');

  if ($suggestions['correctlySpelled'] == TRUE && $incorrect_spelling_only) {
    // If we've set it so we only want incorrectly spelt words to be suggested and the suggestion
    // is spelt correctly then we don't want to do anything.
    return;
  }
  unset($suggestions['correctlySpelled']);
  
  // If min_freq is 0, retain normal functionality
  if ($min_freq != 0) {
    foreach ($suggestions as $word => $value) {
      // If the word occurs more (or the same as) the minimum frequency, don't use it.
      if ($value->origFreq >= $min_freq) {
        unset($suggestions[$word]);
      }
    }
  } 
}

Setting to needs review to try kick this off again

Woops, needed a reroll.

I'll give in and commit this. But we should actually get a cleaner solution but let's take this as part of the solution. Committed to 7.x-1.x and needs backport now

Thanks @Nick_vh, feel free to add a comment above the line you are not happy with as a TODO to fix :)