Problem/Motivation

I need to facet on fields that are semantically equal, but have different field names. There are cases where it isn't practical or possible to re-use fields. In those cases one can use the field_mappings_alter hook and define a name callback to normalize different field names into a common field name for the different fields that are actually semantically equal. This should cause the various Drupal fields to be indexed into one Solr field, thus making it possible to facet elegantly across fields.

However, this fails because the function that returns information about what node fields should be indexed only supports one instance of a given field name per entity type. In strict Drupal terms this isn't necessarily wrong, but can lead to surprising results when the Apachesolr module allows for the field_mappings_alter hook to change field indexing keys.

Proposed resolution

I've worked out a fix for this issue and I'm working on a patch soon to be submitted. The fix simply lets field names stack on top of each other, indexing all instances of a given field name instead of indexing only the last one added.

Remaining tasks

Review is needed. I don't think it should break other functionality, neither does it alter any part of the API.

User interface changes

N/A

API changes

N/A

Original report by remimikalsen

N/A

Comments

remimikalsen’s picture

Patch as promised.

The two patches do the same thing, but one is more compact as it doesn't fix new spacing to account for added loops.

It's easier to read the patch without spaces, but the one with spaces is more correct.

R

nick_vh’s picture

Status: Active » Needs review

Let's see if this breaks any of our tests.

Thanks for all the hard work!

nick_vh’s picture

Would you be able to show us two code snippets? A before and after (as in, something that would fail without the patch and something that would work with the patch)

Thanks!

remimikalsen’s picture

It's hard to show an "after" snippet, because the result of the change is only visible in the effectively indexed Solr fields. However, I have this code I can show to enlighten the problem further.

The first piece of code is adapted from a custom module I developed that implements the field_mappings_alter hook. It basically takes semantically equal fields and translates them to a commonly named Solr field. An example usage would be where different fields only use and display a sub-set of a "common taxonomy" based on criteria we can't cover when re-using fields across content types.

/**
 * Implements hook_apachesolr_field_mappings_alter()
 */
function apachesolr_normalize_fields_apachesolr_field_mappings_alter(&$mappings, $entity_type) {

  // Enable indexing for text fields
  $mappings['my_text'] = array(
    'indexing_callback' => 'apachesolr_fields_default_indexing_callback',
    'map callback' => '',
    'index_type' => 'string',
    'facets' => TRUE,
    'facet missing allowed' => TRUE,
    'dependency plugins' => array('bundle', 'role'),
    'hierarchy callback' => FALSE,
    'name callback' => '',
    'facet mincount allowed' => FALSE,
    'multiple' => FALSE,
  );    
    
  $mappings['per-field']['field_educode_articles'] = $mappings['my_text'];
  $mappings['per-field']['field_educode_articles']['name callback'] = 'apachesolr_normalize_fields_nameconvert';
  $mappings['per-field']['field_educode_courses'] = $mappings['my_text'];
  $mappings['per-field']['field_educode_courses']['name callback'] = 'apachesolr_normalize_fields_nameconvert';  
}

/**
* Callback that generates a sub-part of a Solr field name from a Drupal field name
*   - simple test implementation that drops everything after the underscore
*        eg.: field_educode_courses >> field_ecucode
*               field_educode_articles >> field_educode
*/
function apachesolr_normalize_fields_nameconvert ($field) {
    $name = explode('_', $field['field_name']);
    unset($name[count($name)-1]);
    return implode('_', $name); 
}

Without the patch, only one of the two mapped fields are indexed, because of this line of code in apachesolr.module:

    $fields[$entity_type][apachesolr_index_key($row)] = $row;

The field that is indexed depends on what field is processed last. Changing this line of code to:

    $fields[$entity_type][apachesolr_index_key($row)][] = $row;

will "stack" the different fields on top of each other; thus needing a "loop" to process the stacked $rows (3 places in the code).

Was this what you had in mind when asking for snippets?

R

nick_vh’s picture

That was exactly what I wanted to see. I am going to dive deeper in to this problem and hopefully report back with possible concerns.
If I understand correctly, this does not break any API function, it just fixes one right?

remimikalsen’s picture

You understand correctly, no changes in the API have been done, nor should the change impact the API, and it only fixes this one issue.

R

nick_vh’s picture

Version: 7.x-1.x-dev » 6.x-3.x-dev
Status: Needs review » Patch (to be ported)

Committed to D7. Welcome on the committers list!

pwolanin’s picture

Wait, what happened to my comment? I think there is too much code duplication here.

pwolanin’s picture

oops, nevermind - confusing 2 issues.

remimikalsen’s picture

Thanks Nick! I'll hopefully be using this module heavily in the time to come, so this might not be the last you hear from me. And if you don't, it just means everything works :)

R

nick_vh’s picture

So I guess I have that I hope not to hear from you? ;-)

Always welcome in the issue queue!

nick_vh’s picture

Status: Patch (to be ported) » Closed (fixed)

Committed to D6

David_Rothstein’s picture

FYI, this does seem to have been an API change, due to:

@@ -1915,7 +1919,7 @@ function apachesolr_entity_fields($entity_type = 'node') {
         // Only add to the $fields array if some instances are displayed for the search index.
         if (!empty($row['bundles'])) {
           // Use the Solr index key as the array key.
-          $fields[$entity_type][apachesolr_index_key($row)] = $row;
+          $fields[$entity_type][apachesolr_index_key($row)][] = $row;

Example: #1838854: Current version of Apache Solr Field Collection doesn't work with current Apachesolr

nick_vh’s picture

Well, given that the module was just in RC or even still in beta back then, I don't see any reason to add a compatibility again. Are you ok with that?

David_Rothstein’s picture

Yup, fine with me. (I just tracked the change down to this issue and figured I'd mention it since the comments above specifically discussed whether or not this constituted an API change.)