Implemented highlighting feature using the SolrPHPClient

Hi janusman

It seems that we are developing on the same issues. Maybe you have see that I have developed a spellchecker to (also supporting phrases)
http://drupal.org/node/303937

I have implemented highlighting text snippets using solr.

Step 1: Replace Solr PHP Client with attached one
The author of SolrPHPClient has released a new SolrPHPClient using json_decode instead of xml results. This is a performance gain.
https://issues.apache.org/jira/browse/SOLR-341

I have attached a modified version of the SolrPHPClient.

Step 2: Edit apachesolr_search.module

      try {
        $params = apachesolr_search_get_params();

        //Enable highlighting
        if(variable_get('apachesolr_solrhighlighting', false)) {
        
        	//Add new parameter to the search request
        	$params['hl'] = 'true';
        	$params['hl.fragsize']= variable_get('apachesolr_textsnippetlength', 100);
        	$params['hl.simple.pre'] = '<strong>';
        	$params['hl.simple.post'] = '</strong>';
        	
        }
        
        $response = $solr->search($query->get_query(), $params['start'], $params['rows'], $params, $search_servlet);

        apachesolr_has_searched(TRUE);
        
        //New since json_decode
        $highlight = $response->highlighting;
        $response = $response->response;

            $extra['score'] = $doc->score;
            
            if(variable_get('apachesolr_solrhighlighting', false)){
				 $snippet = each($highlight);
				 $snippet = $snippet['value']->text[0];            
            }
	        else
	        	 $snippet = search_excerpt($keys, $doc->body);    
            
            if (trim($snippet) == '...') {
              $snippet = '';
            }

Step3: adapt apachesolr.module

  $form['apachesolr_path'] = array(
    '#type' => 'textfield',
    '#title' => t('Solr path'),
    '#default_value' => variable_get('apachesolr_path', '/solr'),
    '#description' => t('Path that identifies the Solr request handler to be used. Leave this as /solr for now.'),
    );
  $form['apachesolr_solrhighlighting'] = array(
    '#type' => 'checkbox',
    '#title' => t('Enable highlighting by solr'),
    '#default_value' => variable_get('apachesolr_solrhighlighting', false),
    '#description' => t('Highlighting and text excerpt is done using solr.'),
   );
  
  $options = array();
  foreach (array(100, 150, 200, 250, 300, 400, 500, 600) as $option) {
    $options[$option] = $option;
  }
  $form['apachesolr_textsnippetlength'] = array(
    '#type' => 'select',
    '#title' => t('Length of text snippet in the result'),
    '#default_value' => variable_get('apachesolr_textsnippetlength', 100),
    '#options' => $options,
    '#description' => t('The number of characters the text snippet in the search result will be.'),
    );

No, the attached SolrPHPClient is slightly modified to support different search request handler.

Because I have implemented a spellchecker too, I'm not able to export this modification as "standalone". I have attached a patch in the diff form including the spellchecker modifications. Its for the latest 6.0 version.

Did someone test the code? I think, the highlighting feature of solr is very nice, we should use it. As you can see in my code, only 4 new parameter are required.

$params['hl'] = 'true';
$params['hl.fragsize']= variable_get('apachesolr_textsnippetlength', 100);
$params['hl.simple.pre'] = '';
$params['hl.simple.post'] = '';

Additionally the fragment size can be defined to a different lenght than 100 chars.

What may I do to help, that this feature is added to the dev version?

I'm working on this issue again. I will provide a standalone patch for the latest dev 6.0 version. The highlighting feature of solr is highly preferable compared to the drupal search_excerpt().

I'll provide a patch soon and hope for testers

Hello!

I have some problems with the highlighting. Unfortunately we save the whole html code of the body field in the solr database.

As I found out, the highlighting feature needs a stored field and will not work with "text" which is not stored.

I know, that we strip out all HTML Tags with the solr.HTMLStripWhitespaceTokenizerFactory Filter. But in the stored field, the HTML Code still remains. Its only stripped out for indexing.

This leads to the result, that the fragments contain crippled html code, because they are based on te body field.

Has anyone an idea, how to circumvent this? The highlighting feature has no possibility to remove all html tags.

By the way: Why do we store the full html code of the body in the index? Wouldn't it be better to strip all HTML tags out before we pass the whole thing to solr?

Ok! I found an easy solution for the point decribed above.

This is a standalone integration of the solr highlighting feature. It comes with a seperate menu for configuration.

I wanted to highlight words also in the search result title. But unfortunately all html tags are escaped:

"My title" will be displayed as "My <strong>title</strong>" and not as "My title".

If anyone nows an easy solution to inculde html code in search result titles, give me a short feedback. I'll implement highlighting in titles too.

It looks like you have "reinvented" the proposal I already implemented.

Hi JacobSingh

I will review your patch in the next two days and give you a feedback on this.

The main reason, why I implemented solr highlighting was the following:

- In the drupal core search it is not possible, to set the length of the search snippet. Its fixed to 100 chars. I needed about 300 chars, because I mainly index large pieces of text. When I searched for a solution for this in core search, I found out, that a lot of user would like to specify the length of the search snippets. That's the reason why I implemented a backend admin optioon. I think this should be configurable somewhere.

For me it doesn't matter, if the default values are in solrconfig.xml. But I think an admin page is more the drupal way than external config files.

To Point 2:
I agree. If you activate solr, you should use also the solr highlighting.

3) I don't understand here, what you mean

4) I agree

You you can mention me as a participant, that would be nice. I'm not able to commit, but I try to make patches and module extensions here and there for different drupal modules.

Maybe you could also review my implementation of the "Did you mean" functionality here:
http://drupal.org/node/303937