The current implementation of DrupalApacheSolrService uses solr's JSON Response Writer to receive the response as json.

Unfortunately the json response does not contain all information compared to the other response writers available. That's a documented behavior (see section "JSON specific parameters"):

Using a JSON object (essentially a map or hash) for a NamedList results in the loss of some information.

In Apache Solr Multilingual we support multiple spell checkers in different languages. If you use the standard response writer the corresponding part in the XML looks like this:

 <lst name="spellcheck">
  <lst name="suggestions">
    <lst name="intolerans">
      <int name="numFound">1</int>
      <int name="startOffset">0</int>
      <int name="endOffset">10</int>
      <arr name="suggestion">
        <str>Intoleranz</str>
      </arr>
    </lst>
  </lst>
</lst>
<lst name="spellcheck">
  <lst name="suggestions">
    <lst name="intolerans">
      <int name="numFound">1</int>
      <int name="startOffset">0</int>
      <int name="endOffset">10</int>
      <arr name="suggestion">
        <str>intolerance</str>
      </arr>
    </lst>
  </lst>
</lst>

The example shows a German and an English correction of "intolrans". Using XML it's no problem to access both suggestions.

If you use JSON, all suggestions are present in the result, but json_decode() collapses them and the latest element wins:

"spellcheck":{
    "suggestions":{
      "intolerans":{
        "numFound":1,
        "startOffset":0,
        "endOffset":10,
        "suggestion":["Intoleranz"]}}},
"spellcheck":{
    "suggestions":{
      "intolerans":{
        "numFound":1,
        "startOffset":0,
        "endOffset":10,
        "suggestion":["intolerance"]}}},

An interesting point is, how the available PHP Response Writers deal with that. Both, the php and phps responses automatically add counters to identically named elements:

'spellcheck'=>array(
    'suggestions'=>array(
      'intolerans'=>array(
        'numFound'=>1,
        'startOffset'=>0,
        'endOffset'=>10,
        'suggestion'=>array('Intoleranz')))),
'spellcheck 1'=>array(
    'suggestions'=>array(
      'intolerans'=>array(
        'numFound'=>1,
        'startOffset'=>0,
        'endOffset'=>10,
        'suggestion'=>array('intolerance')))),
a:1:{s:11:"suggestions";a:1:{s:10:"intolerans";a:4:{s:8:"numFound";i:1;s:11:"startOffset";i:0;s:9:"endOffset";i:10;s:10:"suggestion";a:1:{i:0;s:10:"Intoleranz";}}}}s:12:"spellcheck 1";a:1:{s:11:"suggestions";a:1:{s:10:"intolerans";a:4:{s:8:"numFound";i:1;s:11:"startOffset";i:0;s:9:"endOffset";i:10;s:10:"suggestion";a:1:{i:0;s:11:"intolerance";}}}}}

So I see different ways to solve this issue:
1. Use a better response writer than json in general. (Heavy rewrite and API changes)
2. Create patches for solr itself to let the json response writer behave like the php response writers. (Might be the best solution in the future but might not solve the issue for older solr 1.x and 3.x versions)
3. Fake a valid json result on our side.

I see two ways to implement solution 3 until solution 2 will be available. I will post a patch soon.

Comments

mkalkbrenner’s picture

Status: Active » Needs work
StatusFileSize
new3.1 KB

Here's my quick workaround to make just search requests "save".

mkalkbrenner’s picture

Status: Needs work » Needs review
StatusFileSize
new3.1 KB

enable test bot

Status: Needs review » Needs work

The last submitted patch, 1924448_save_json_response.patch, failed testing.

mkalkbrenner’s picture

Status: Needs work » Needs review
StatusFileSize
new3.1 KB
mkalkbrenner’s picture

Issue summary: View changes

fixed indention

mkalkbrenner’s picture

StatusFileSize
new3.1 KB

fixed typo: safe instead save

nick_vh’s picture

So, looking at this I don't really advocate the phps.

Other issues about this :
http://drupal.org/node/1338342
http://drupal.org/node/396548

Quote :

https://issues.apache.org/jira/browse/SOLR-1967
Using php or phps can expose a number of unexpected security issues as mentioned by the thread above. Security is more important than performance and therefor it is wise to stick with json as it is more tested and widespread.

Solution?
Here's some random code I'm making to overcome this issue. Since json is plain text and solr seems to not respect the json standards, we can fix this issue ourselves before parsing the json.

function give_unique_id($value) {
    return $value . "-" . uniqid();
}

$json_ouput = spellchecker_foo_get();
$json = preg_replace_callback('/"spellcheck/":','give_unique_id' , $json_output);
$output = json_decode($json);
nick_vh’s picture

Status: Needs review » Needs work
mkalkbrenner’s picture

Using xml will solve the issue as well, but my edge case is not worth that effort and we will lose performance.

If we only deal with the specific spell checking problem, we should move and adjust the code that is currently used in Apache Solr Multilingual to this module. That will increase performance a little because we avoid an additional json_decode:

/**
 * Retrieve all of the suggestions that were given after a certain search
 * Mostly copied from @see apachesolrsearch_get_search_suggestions();
 * @return array()
 */
function apachesolr_multilingual_get_search_suggestions($env_id, $filter_languages = array()) {
  $suggestions_output = array();
  if (apachesolr_has_searched($env_id)) {
    $query = apachesolr_current_query($env_id);
    $keyword = $query->getParam('q');
    $searcher = $query->getSearcher();
    $response = apachesolr_static_response_cache($searcher);
    // @see http://wiki.apache.org/solr/SolJSON
    // "Using a JSON object (essentially a map or hash) for a NamedList results
    // in the loss of some information."
    // That's the reason why the multiple language specific spell check results
    // get lost during json_decode(), because they are all named "spellcheck".
    // Therefor we rename the the language specific spell checks and
    // json_decode() twice.
    $language_ids = array_keys(apachesolr_multilingual_language_list());
    foreach ($language_ids as $language_id) {
      $response->data = preg_replace('@"spellcheck"@', '"spellcheck_' . $language_id . '"', $response->data, 1);
    }
    $result = json_decode($response->data);
    if (is_object($result)) {
      foreach ($result as $key => $value) {
        $response->$key = $value;
      }
    }

    ...
}

nick_vh’s picture

Priority: Critical » Major
pwolanin’s picture

I'd say this is really an issue with Solr, but the problem is documented in terms of the format.

Why not try a change to json.nl=arrarr ? The steps to reproduce this are also unclear - can you create a simple test case?

I'm rather opposed to taking in serialized PHP.

mkalkbrenner’s picture

Status: Needs work » Closed (won't fix)

I think we should stay with json as it is. The spell checker is a real edge case. I introduced a new class DrupalApacheSolrMultilingualService that solves that issue for Apache Solr Multilingual individually:
http://drupalcode.org/project/apachesolr_multilingual.git/commit/1b98526

mkalkbrenner’s picture

Issue summary: View changes

typo