Add support for CCK Decimal fields and Numeric Facets

To be up front, I don't really know what I am doing here so I'm brain dumping what I have done so far.

From a look at the database, the CCK field type is 'number_decimal'. Checking views, this is handled by views_handler_filter_numeric and views_handler_field_numeric. Following the other examples of folks who added support for various CCK and other content, I did the following.

1) Added to the apachesolr_views_views_handlers hook. Specifically:

      'apachesolr_views_handler_field_number_decimal' => array(
        'parent' => 'views_handler_field_numeric',
      ),
      'apachesolr_views_handler_filter_number_decimal' => array(
        'parent' => 'views_handler_filter_numeric',
      ),

2) Added to the apachesolr_views_views_data hook. I followed some of the other patches and added to the bottom of the foreach (apachesolr_cck_fields() as $name => $field).

    elseif ($field['field_type'] == 'number_decimal') {
      $data['apachesolr'][apachesolr_index_key($field)] = array(
        'title' => t($field['label']),
        'help' => t('CCK Mapping for @fieldname', array('@fieldname' => $field['field_name'])),
        'field' => array(
          'numeric' => TRUE,
          'handler' => 'apachesolr_views_handler_field_number_decimal',
        ),
        'filter' => array(
          'handler' => 'apachesolr_views_handler_filter_number_decimal',
          'cck_field' => $field,
        ),
      );
    }

3) So far, it feels like I was on the right track. It feels like this field needed its own handler, specifically to define the number as a float. So now I created the file apachesolr_views_handler_field_number_decimal.inc. It looks like this:

class apachesolr_views_handler_field_number_decimal extends views_handler_field_numeric {
  function construct() {
    parent::construct();
    $this->definition['float'] = TRUE;
  }
  function query() {
    $this->query->add_field($this->real_field);
  }
}

I'm not sure about the above. This is where I start to go off into the weeds.

4) I created the filter as apachesolr_views_handler_filter_number_decimal.inc. This is what it looks like.

class apachesolr_views_handler_filter_number_decimal extends views_handler_filter_numeric {
  /**
   * Override query to do our thing
   */
  function query() {
    $this->query->add_filter($this->real_field, $this->value);
  }
}

And that's where I am. It's not working, and I'm not sure why. I'm going to keep working on it but anyone who might come across this and could chime in, I'd certainly appreciate it.

Thanks for the reply Scott. No errors anywhere, and I have been at a bit of a loss as to where to debug. Your comment was very helpful. I do see "facet.field=fs_ap_saleprice" in the queryString, so it looks like that part might be OK. I know that field is in the index as I looked directly in the Solr interface and it works 100% on the 'normal' non-views Apachesolr facets.

In terms of what's wrong, the facets in my numeric facet block in the solr view come back with no added arguments to facet on. The rest of my facets are on taxonomy terms, so I might see mysite.com/taxonomy/term/241,95. I don't see any added arguments to facet on for the price/decimal block.

I noticed that I wasn't returning anything in the field handler, so now it looks like this:

class apachesolr_views_handler_field_number_decimal extends views_handler_field_numeric {
  // Set to float so it knows we are dealing with a decimal
  function construct() {
    parent::construct();
    $this->definition['float'] = TRUE;
  }
  function query() {
    $this->query->add_field($this->real_field);
  }
  function render($doc) {
    if (!empty($doc->{$this->real_field})) {
      return check_plain($doc->{$this->real_field});
    }
  }
}

That didn't do it either. I'm still working through things and will keep posting what I find. Thanks again.

OK getting closer. In the 'default' Solr query, it has something that looks like this:

&fq=fs_ap_saleprice:[100 TO 200]

That's the facet for my field, showing values between 100 and 200. In my apachesolr views query, I have the &facet.field=fs_ap_saleprice but I have no &fq at all. The root of the problem at this point is that when my other facet block module does a $new_query->get_path(), it's not coming back with any facet arguments.

I wonder if #592594: CCK arguments not integrating properly with facet blocks has anything to do with it.

OK, let's take a step back.

My CCK field is number_decimal. My facets are trying to narrow down a specific range of values, such as "show me all nodes where the CCK number field is between 100 and 200". In practical terms, I'm trying to facet on price between $100 and $200.

The same view is currently enabled for faceting on taxonomy terms and author UID. Each of those two are specified by arguments now, such as "Apache Solr: Taxonomy terms" and "Apache Solr: Author Uid". Both of these work well. They are being output by a row style of Node so there are no fields in use within the view.

Specifically related to what I am trying to accomplish, it strikes me that to facet on these number_decimal values, they would also have to be present as an argument. Is this true?

If yes, to accomplish what I am looking for, at a minimum I would need:

1) An argument handler for number_decimal fields
2) A filter handler for number_decimal fields

I would not need the field handler, but it would be a nice to have perhaps for someone else.

As of now, I know what's failing - the apachesolr_cck_fields call never comes back with the field. It checks on an object that looks like this:

stdClass Object
(
    [field_name] => field_saleprice
    [multiple] => 0
    [field_type] => number_decimal
    [widget_type] => number
    [label] => Sale
)

That fails its check for widget type, so I guess I have to add a mapping.

Yep, that worked. Woo hoo. I'm further along now that I can actually get the CCK fields to work with :)

Yes, the goal is to enable a facet block to allow for faceting by price. In this case, price is a CCK decimal field. I created a new facet block for this purpose based on the Apachesolr Ubercart module which did that for Ubercart. In my case the block/facet code is almost exactly the same except I am using a CCK field instead of a Ubercart field.

I'm slowly getting it. The CCK field needs to be named in part_of_facet_block so that returns correctly, and then the add_filter also needs to poke in the correct value for the facet argument.

Thanks Scott - I have that part now. I'm manually adding my facet field into that section so it returns to get_path correctly. I'll add something else to make it more dynamic but for now that part seems to be OK.

I also had to rename my Solr field from fs_ap_saleprice to fs_cck_field_saleprice which greatly helped match it up with the code in this module.

Now I'm getting the facet block links to come up, but the links look like this (where 241,95 are already faceted taxonomy terms):
/taxonomy/term/241,95/all/[0 TO 100]

All is being dropped in there because this comes back with argument position 2:

$path_parts[$argument->position + 1] = $path_part;

The [0 to 100] is the correct search term, where it would look like this in the query: &fq=fs_cck_field_saleprice:[100 TO 200]

The problem is that Solr burps on it with this kind of error:

Error report HTTP Status 500 - For input string: "[200 TO 300]"#012#012java.lang.NumberFormatException: For input string: "[200 TO 300]"#012#011at sun.misc.FloatingDecimal.readJavaFormatString(Unknown Source)#012#011at java.lang.Float.parseFloat(Unknown Source)#012#011at org.apache.solr.util.NumberUtils.float2sortableStr(NumberUtils.java:79)#012#011at org.apache.solr.schema.SortableFloatField.toInternal(SortableFloatField.java:49)#012#011at org.apache.solr.schema.FieldType$DefaultAnalyzer$1.incrementToken(FieldType.java:320)#012#011at org.apache.lucene.analysis.CachingTokenFilter.fillCache(CachingTokenFilter.java:87)#012#011at org.apache.lucene.analysis.CachingTokenFilter.incrementToken(CachingTokenFilter.java:61)#012#011at org.apache.lucene.queryParser.QueryParser.getFieldQuery(QueryParser.java:599)#012#011at org.apache.solr.search.SolrQueryParser.getFieldQuery(SolrQueryParser.java:153)#012#011at org.apache.lucene.queryParser.QueryParser.getFieldQuery(QueryParser.java:741)#012#011at org.apache.lucene.queryParser.QueryParser.Term(QueryParser.java:1584)#012#011at org.apache.lucene.queryParser.QueryParser.Clause(QueryParser.java:1337)#012#011at org.apache.lucene.queryParser.QueryParser.Query(QueryParser.java:1265)#012#011at org.apache.lucene.queryParser.QueryParser.TopLevelQuery(QueryParser.java:1254)#012#011at org.apache.lucene.queryParser.QueryParser.parse(QueryParser.java:200)#012#011at org.apache.solr.search.LuceneQParser.parse(LuceneQParserPlugin.java:78)#012#011at org.apache.solr.search.QParser.getQuery(QParser.java:131)#012#011at org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:103)#012#011at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:174)#012#011a

So I'm figuring I need to rewrite the URL similar to how you are displaying /241,95 for taxonomy terms, but the search query is actually &fq=tid:241&fq=tid:95.

I'm close - the argument is correct, it's just adding quotes around it now. By the time I hit the rebuild_fq function, my argument looks like "[100 to 200]" instead of just [100 to 200]. Looking for what adds the quotes now.

My new part_of_facet_block function looks like this. For whatever reason, 'tid' does not come back so I leave that one hardcoded.

  protected function part_of_facet_block($argument) {
    $facet_arguments = array(
      'tid',
    );
    foreach (apachesolr_get_enabled_facets() as $module => $module_facets) {
      foreach($module_facets as $delta => $facet_field) {
        $facet_arguments[] = $facet_field;
      }
    } 
    // we use field instead of real_field
    // because we want these specific ones defined in
    // apachesolr_views.views.inc
    return (in_array($argument->field, $facet_arguments));
  }

On to my real problem - my module creates the facet search looks like this:

$facet = '['.$from_price.' TO '.$to_price.']';

which outputs something like [100 TO 200]. This runs into the function escape_term, which adds quotes to it because it has spaces in it.

It seems like [100 to 200] is a valid thing to search on as it comes right out of apachesolr_facet_block:
http://drupalcontrib.org/api/function/apachesolr_facet_block/6

where it has $facet = '[* TO *]';

What's happening specifically is I am inside of the function escape_term and falling into this:

    if (strpos($term, ' ') !== FALSE) {
      return Drupal_Apache_Solr_Service::phrase($term);
    }

Is this behavior correct? Or should it allow for spaces in there? I added this above it which now allows it to work.

    if (strpos($term, ' TO ') !== FALSE) {
      return $term;
    }

OK I have arrived at a final solution. It now works for me quite well and allows for faceting on CCK decimal fields. I have a few questions -

1) I'm unsure of how to interpret the escape_term issue from post #10 of this thread. My workaround solves my particular issue but it may not solve it for others that go down this path.

2) How do you feel about the reworked part_of_facet_block code from #10? It seems like just about every contributed patch for apachesolr views in the issue queue has to address that problem. Regardless of the rest of the CCK decimal support, perhaps something like that might be a candidate for a commit to the main module.

3) Would you like me to roll a patch? Either for consideration as part of the core module or to assist others who may want to do something similar. I don't believe anything I did is specific to my custom module and should generally apply to any CCK decimal fields. I'm also happy to write up the specifics of how to integrate this with a custom decimal/price facet. There were a few tricky parts, such as making sure to name the solr index field correctly.

Thanks for chiming in, you really did help me - especially #2. That cut down a lot of time figuring out how to look at the query.

OK, here's the first patch. All it does is to 'fix' the part_of_facet_block function. I'll roll a second patch for the CCK decimal support but this one is more universal. I left the hardcoded tid in there since right now it won't work without it.

Here's the second patch. This is the argument handler for CCK Decimal fields, and it includes the patch from #13 as well. That's about 20 hours for 23 lines of code. Not very efficient on my part!

There are directions for making this work properly.

1) You need to first make sure your CCK field is indexed. Decimal fields are not indexed by default, so this requires a hook in a custom module. It's VERY IMPORTANT that you name the Solr field fs_cck_field_FIELDNAME, where FIELDNAME is your CCK field name. In my case, my CCK field name is field_saleprice, so the Solr field is fs_cck_field_saleprice. Here's my implementation of this.

/**
 * Implementation of hook_apachesolr_update_index().
 *
 * This adds our CCK numeric sale price field to the Solr index.
 */
function mycustommodule_apachesolr_update_index(&$document, $node) {
  if($node->type == 'items') {
    if(!empty($node->field_saleprice[0]['value'])) {
      $document->fs_cck_field_saleprice = $node->field_saleprice[0]['value'];
    }
  }
}

2) Rebuild your Solr index. Then visit the Solr administration page (either via /admin/reports/apachesolr or directly to Solr) to make sure your field is in the index.

3) In the view you created for apachesolr_views, you should now see the CCK decimal fields in the arguments section. Add the corresponding field as an argument. It's VERY IMPORTANT again that you select 'Display all values' as the action to take if argument is not present. Otherwise you are doomed to troubleshooting and wondering why it isn't working.

4) Create a facet block for this. This gets back to a custom module. In my case, I based my module almost exactly on the Apachesolr Ubercart module. This consisted of removing all of their additional facets except for one, which is my fs_cck_field_saleprice field. Step #1 for indexing as listed above is also implemented in this module in a slightly different way. I also changed the ksort() to a natsort() since the price terms in the facet are mixed up otherwise.
http://drupal.org/project/apachesolr_ubercart

5) Enable your new facet block module on your view page.

6) Profit, er Facet! Or both.

Here's a re-roll of the patch against the new 6.x-1.x-dev, dated July 11, and as of commit #369072. You aren't going to get rid of me without a review and possible commit :) I'm going to keep porting this to the latest version as it's mission critical for my site and a facet based on price.

Thanks for the work on the new release. It's nice to move to the 2.x series of Apachesolr.

#17 I used this argument support to create facet/filter blocks based on code here:
http://drupal.org/project/apachesolr_ubercart

That's a great example of writing numeric facets for Solr. I did not release my numeric/price facet because it has some specific stuff for my site, but 95% of the code came from Apachesolr Ubercart's price facet.

Part of the patch in #16 was committed in #19, specifically the fix to the facets in the query. Otherwise, the remainder of that patch works for me 'out of the box' with views. The arguments show up in views for me with no other changes.

My argument was a requirement to facet based on numeric values, which I use to facet on a CCK decimal field that is indexed in Solr. In my case the numeric field represents monetary value and it's used to refine price so you can do things like facet on all content between 100 and 200. The argument needed to be in the view or there's nothing to match up against the Solr index.

I'd still like to see the full patch added to the core apachesolr_views module if possible. I'm never going to get away from that feature and I'm managing the patch on my own.

@dstuart - as the active module maintainer, how do you feel about possibly committing a patch for this? I have rolled it up a few times previously (as have others above). This is required functionality for me so I am left managing diffs and reconciling against any other changes to the module. At this point it's a fairly small change as some of the other initial requirements have been committed in #19.

I'm happy to re-roll a testable patch against the current 6.x-1.x-dev if you would consider reviewing it. Right now #27 seems to be what I'd need. Thanks for considering it - having it in the main module would make my life a lot easier.

Thanks, you need #27 and the fix from #37 - the field name changed since the original patch.

My approach is working for me and continues to do so, but I was interested in tinker's statement about storing decimal fields as floats. I have not researched that and have no information either way on what would be the best way to store it. I'd love to have some feedback here as I may have this wrong.

In all cases, this issue is one of four separate patches I am maintaining against the apachesolr_views module (all of which are in the issue queue, none of them committed yet.) So I'd be more than happy to collaborate on coming up with a final patch so we can get this put in.

As an aside and in case anyone is interested, the other patches I require are:
#992598: Enabling "Apache Solr Views Integration" causes duplicates in Views configuration screen "Add fields", patch from #7
#1108252: Provide filter for Published / Unpublished nodes (status), patch from #1
#1211276: Taxonomy filter adds duplicates, patch from #5