Support apachesolr_multilingual

I applied the patched, deleted the attachments and search index and re-indexed it all. The body of the search result is still "..." if the "Limit search to user's language by default" setting is enabled.

@wmostrey:
Have a look at #899560: apachesolr_do_query() has to take &$params by reference. Maybe you have to apply this patch as well to apachesolr itself.

Hi Markus, I'm using the latest 6.x-2.x-dev versions so I already have that patch applied.

OK, I'll review Robert's patch as soon as possible.

I tried this patch again with the newest 6.x-2.x-dev versions of apachesolr, _attachments and _multilingual but the result is still the same:

The body of the search result is still "..." if the "Limit search to user's language by default" setting is enabled. If that option is disabled then the correct snippet is shown.

@wmostrey: see #915626: hook modify_query clobbers hl.fl from previous modules

I already tested that one and it has no effect on this problem.

I did not test the patch right now but read the code of it and apachesolr_attachments. It should work ...

@wmostrey:
Just to be sure:
- did you re-index after you applied the patches?
- is apachesolr_multilingual working for nodes like expected? Are there highlighted snippets in the search result if you filtered by language?

@robertDouglas:
You assume that attachments are of the same language as the node. This should be at least configurable. What if it's a multilingual attachment ;-)
We should add a switch to optionally add the tika part to the generated solrconf.xml.

I re-index after each patch yes. And it's working perfectly for nodes with both the option disabled or enabled.

"Limit search to user's language by default" enabled:

"Limit search to user's language by default" disabled:

I tried Markus' new 6.x-2 patch from #915626: hook modify_query clobbers hl.fl from previous modules but that doesn't change anything related to this issue.

This might be related? #917188: Search results won't return body or teaser just "..."

(more descripteve title)

What is the status about this currently? I seen that attachments are stored in dynamic ss_ fields, so support should be there "nearly" OOTB ( the tika part is missing in the config though )

  <requestHandler name="/extract/tika" class="org.apache.solr.handler.extraction.ExtractingRequestHandler" startup="lazy">

    <lst name="defaults">
    </lst>
    <!-- This path only extracts - never updates -->
    <lst name="invariants">
      <bool name="extractOnly">true</bool>
    </lst>
  </requestHandler>

So that one should be added if tika is supported / wished.

Anyhow, i see the ss_file.. fields using the analy field tool but i dont find any files ( fields are filled ). I did not apply any of the above patches yet. Iam using the current 2.x dev.

I will dive into this tommorow and provide a patch if need.

Well in my case, the problem is rather clear. Attachments ( and users ) dont have an language entry, so language=de will filter all of them out.
I guess this all comes down to my option "limit only to the current language".

But there is no reason why my obvious german attachments and users (well users is argueable though, would be which language their user profile is in or maybe neutral?).

So:
- can we finetune the the index method so somehting like "neutral" is inserted, if the language cannot be detected? Otherwise this content will not be searchable at all (not by any language), as language is empty.

UPDATE:

Well interesting

    if (!$language) {
      // Language neutral
      $language = variable_get('apachesolr_multilingual_map_language_neutral', '');
      if ($language) {
        $document->language = $node->language = $language;
      }
    }

According to this there should neve be a content without a language then, as my neutral language is set to german. apachesolr_attachment does not define a update_index hook, so there is no way they overlap / override eachother.

There is only the implementation of hook_apachesolr_document_handlers for adding those attachments. Could the namespace be a problem here?

Looking at apachesolr_multilingual_apachesolr_update_index no namespaces are excluded, but iam not sure, i remember you have to register for the namespace, in that case apachesolr_attachments

Unfortunately I don't have a working setup for atachements at the moment.

@EugenMayer: What hppens if you configure "Map language neutral:" at "/admin/settings/apachesolr/multilingual" and re-index?

It was configured Markus, so there actually should be no content having "no language" actually.

@mkalkbrenner Mapping language neutral and rebuilding doesn't have an effect on this issue (see #10).

I managed to fix this by adding a "language" property directly on the Apache_Solr_Document instance.
2 ways to go: add

$document->language = $node->language;

in apachesolr_attachments.admin.inc line.255

Or, which I did as I needed to grab some data from another node during the indexing of documents, by adding an additional document extraction handler with hook_apachesolr_document_handlers():

/**
 *  Implementation of hook_apachesolr_document_handlers().
 */
function xyz_apachesolr_document_handlers( $type, $namespace )
{
    if ( $type == 'node' && $namespace == 'apachesolr_attachments' ) {
        return array( 'additional_solr_file_indexer' );
    }
}


/**
 *  Extract files for Tika differently
 */
function additional_solr_file_indexer( $node, $namespace )
{
    $documents = array();
    
    //      in my case there's a content type 'document' referenced by the nodes being indexed with that field
    if ( is_array( $node->field_documents ))
    {
        foreach ( $node->field_documents as $doc )
        {
            if ( empty( $doc[ 'nid' ])) continue;
            
            //      here's the node holding the file field
            $node_doc = node_load( $doc[ 'nid' ]);
            
            //      This part is taken from apachesolr_attachments_add_documents()
            //      @see    apachesolr_attachments.admin.inc @line.225
            $document = FALSE;
            $hash = apachesolr_site_hash();
            $files = apachesolr_attachments_get_indexable_files( $node_doc );
            
            // Since there is no notification for an attachment being unassociated with a
            // node (but that action will trigger it to be indexed again), we check for
            // fids that were added before but no longer present on this node.
            
            $fids = array();
            $result = db_query( "SELECT fid FROM {apachesolr_attachments_files} WHERE nid = %d", $node_doc->nid );
            while ( $row = db_fetch_array( $result )) {
                $fids[ $row[ 'fid' ]] = $row[ 'fid' ];
            }
            
            //      Find deleted files.
            $missing_fids = array_diff_key( $fids, $files );
            if ( $missing_fids ) {
                db_query( "UPDATE {apachesolr_attachments_files} SET removed = 1 WHERE fid IN ( ". db_placeholders( $missing_fids ) ." )", $missing_fids );
            }
            $new_files = array_diff_key( $files, $fids );
            
            //      Add new files.
            foreach( $new_files as $file ) {
                db_query( "INSERT INTO {apachesolr_attachments_files} ( fid, nid, removed, sha1 ) VALUES ( %d, %d, 0, '' )", $file->fid, $node_doc->nid );
            }
            
            foreach( $files as $file )
            {
                $text = apachesolr_attachments_get_attachment_text( $file );
                if ( $text )
                {
                    $document = new Apache_Solr_Document();
                    
                    //      Add the language
                    $document->language = $node->language;
                    
                    $document->id = apachesolr_document_id( $file->fid .'-'. $node->nid, 'file' );
                    $document->url = file_create_url( $file->filepath );
                    $document->path = $file->filepath;
                    $document->hash = $hash;
                    $document->entity = 'file';
                    $document->site = url( NULL, array( 'absolute' => TRUE ));
                                        
                    //      edit: Used node_doc->nid here
                    //$document->nid = $node->nid;
                    $document->nid = $node_doc->nid;
                    
                    $document->title = $file->filename;
                    $document->created = apachesolr_date_iso( $file->timestamp );
                    $document->changed = $document->created;
                    $document->status = $node->status;
                    $document->sticky = $node->sticky;
                    $document->promote = $node->promote;
                    $document->uid = $node->uid;
                    $document->name = $node->name;
                    $document->body = apachesolr_clean_text( $file->description ) .' '. $text;
                    $document->ss_filemime = $file->filemime;
                    
                    //      Here I needed to take the title of the node document
                    //$document->ss_file_node_title = apachesolr_clean_text( $node->title );
                    $document->ss_file_node_title = apachesolr_clean_text( $node_doc->title );
                    
                    $document->ss_file_node_url = url( 'node/' . $node->nid, array( 'absolute' => TRUE ));
                    apachesolr_add_taxonomy_to_document( $document, $node );
                    if ( module_exists( 'apachesolr_nodeaccess' )) {
                        apachesolr_nodeaccess_apachesolr_update_index( $document, $node, $namespace );
                    }
                    drupal_alter( 'apachesolr_attachment_index', $document, $node, $file );
                    $documents[] = $document;
                }
                else {
                  watchdog( 'Apache Solr Attachments', 'Could not extract any indexable text from %filepath', array( '%filepath' => $file->filepath ), WATCHDOG_WARNING );
                }
            }
        }
    }
    
    /**
    //      debug
    if ( !empty( $documents ))
    {
        watchdog(
            'xyz',
            'MATCH additional_solr_file_indexer() - $documents : <pre>'. print_r( $documents, 1 ) .'</pre>',
            NULL,
            WATCHDOG_DEBUG
        );
    }
    /**/
    
    return $documents;
}

I added this example in case anybody else used a node type dedicated to documents as weel, in which case there would be a bit of extra work in hook_apachesolr_process_results() and / or search-result.tpl.php to render the "attached to" to whatever is appropriate.

It's apachesolr_attachments that needs to be patched to support apachesolr_multilingual.

Attached patch will fix this. Apply the patch, delete the index and index again. With the patch applied everything works as expected, including the "Limit search to current language by default" option.

This is now actually a duplicate of #884606: Integrate apachesolr_attachments with apachesolr_multilingual

I am using version 6.x-3.x-dev (should I create a new issue for this?)
I seem to have this error without the apachesolr_attachments also.

After indexing my nodes on my apachesolr the search results (and snippets) are correct for all queries.
After running cron the snippets for some search terms are empty, while other are correct.
It's always the same search-terms that get empty snippets from this cache clearing .
When I switch back to dutch/english... from german, "lorem" gets the correct snippets.

http://drupal.org/node/961888 did not solve my problems.
Neither did the patch above comment # 1
If i disable the cache clearing in the cron hook the snippets appear and there's no problem (except that the cache is not cleared and rebuilt :)

I am using the following modules in a 4 language website setup:
Apachesolr ajax, Apachesolr autocomplete , Apachesolr framework , Apachesolr multilingual , Apachesolr node access , Apachesolr search , Apachesolr statistics.

Limit to user language is on and disabling this makes snippets showing correctly again.
I don't have the apachesolr attachements module enabled.