How to use Lucene Api?

Hi Bill.

Sorry for the lack of documentation. This module was build on my free time outside of work, so priority #1 was getting the code stable. Extensive documentation is in the works, but I need to take a deep breath before I embark on that adventure. I promise that it will be available soon.

Seth Cohn of CommonPlaces e-Solutions, LLC actually did something very similar to what you are trying to accomplish using the module's Facet API. It really showed some flexibility in the API that I never intended, but the result was being able to use facets as "tabs". It is really too much to describe here, but a module based off of his code should actually be available fairly soon. What I would recommend first is to implementing hook_luceneapi_document_alter() to add the fields to the index. For example, after adding the jobs field to the index, you will be able to search for them using the following query: jobs:"Job you are looking for". I would also recommend adding fields for City and State. Take a look at luceneapi_node_luceneapi_facet() as well as README.txt for a brief tutorial on the facet API. Again, more detailed documentation is on the way.

In terms of views hooks, that is also a separate module that is being explored, but Search Lucene API has an abstraction layer to Zend's query API, so you can create and manipulate queries programmatically. This would mesh very well with the various views hooks, although it has not been implemented. Any contributions in this area would be greatly appreciated.

Indexing files requires being able to parse the files into text first. Many times this requires external programs, but if you are trying to parse MS 2007 documents, the ZF components actually have document parsers built in. Once you extract the text, you can it to the index as a field via hook_luceneapi_document_alter().

The next step is to get some solid documentation online, and then we will determine the best way to start dialogue with the Drupal community. I am working with other developers who are currently implementing the API to contribute real world solutions making this module more than just a content indexer. I am sorry that I cannot be of more help right now, but check back in a little bit as the documentation is coming.

Hi Bill.

Here are a couple of examples to get you going. First, let's start with adding the field to the document. In the example, I am going to assume that the CCK field is stored in $node->field_location[0]['value'] and we want to add it to the Lucene field location. It is fairly simple to add it via hook_luceneapi_document_alter().

<?php
/**
 * Implementation of hook_luceneapi_document_alter().
 */
function mymodule_luceneapi_document_alter($doc, $node, $module, $type) {
  // bail if we are not dealing with nodes
  if ('node' != $type) {
    return;
  }

  // add the field to the document as a keyword.  Feel free to use another field type.
  luceneapi_field_add($doc, 'keyword', 'location', $node->field_location[0]['value']);
}

The following example creates a facet to be able to filter by location using wildcards.

<?php

/**
 * Implementation of hook_luceneapi_facet().
 */
function mymodule_luceneapi_facet($op, $module, $type) {
  // bail if we are not dealing with nodes
  if ('node' != $type) {
    return;
  }

  switch ($op) {
    // name of the facet, will show up in /admin/settings/luceneapi_node
    case 'name':
      return 'Location';

    // define the facet's form element, just a simple FAPI array
    case 'facet':
      $form = array();
      $form['location'] = array(
        '#type' => 'textfield',
        '#title' => t('Location'),
        '#size' => 30,
        '#default_value' => luceneapi_facet_value('location', ''),
        '#description' => t('Enter a location, wildcards allowed.'),
      );
      return $form;

    // callback function, a facet handler to process the value submitted by the user
    case 'callback':
      return array('mymodule_facet_handler' => array());
  }
}

/**
 * Adds query to filter by location.
 */
function mymodule_facet_handler() {
  // gets facet value passed by user
  $location = luceneapi_facet_value('location', '');

  // returns subquery to filter by location, in this case it is a wildcard query
  // NOTE: see luceneapi.query.inc for available query types and parameter definitions
  return luceneapi_query_get('wildcard', $location, 'location');
}

There are a bunch of different ways to do the custom form you are talking about, but you could always pass the search query to search_data($keys, 'luceneapi_node');. It will search the contents field by default, but you can change the default field to location by calling Zend_Search_Lucene::setDefaultSearchField('location'); beforehand. Since you are looking to add extra data via a custom SQL query, hook_luceneapi_result_alter() is what you are looking for. You can get the node ID from $result, execute your query, and then add the data back to $result. Since it is passed by reference, the array will be passed to the theme function and displayed. If you want to limit the results to the content type xyz, you can always append a subquery in hook_luceneapi_query_alter() to do the filtering.

/**
 * Implementation of hook_luceneapi_query_alter().
 */
function mymodule_luceneapi_query_alter($query, $module, $type) {
  if ('node' == $type) {
    $term_subquery = luceneapi_query_get('term', 'xyz', 'type');
    luceneapi_add_subquery($query, $term_subquery, 'required');
  }
}

As you can probably see, the luceneapi_query_get() function is just a factory function for the Zend_Search_Lucene_Search_Query objects. The advantage is that it handles exceptions gracefully and sends errors to watchdog so you don't have to. See the luceneapi_throw_error() docblock for the other benefits. To get the error messages piped to the screen in your development environment, set the error mode to Debug in /admin/settings/luceneapi. If the hooks aren't what you are looking for, feel free to build a custom query and pass it to $index->find() as you are a Zend guy. I came over to Drupal from the Zend Framework, so I understand wanting to program using straight Zend. Although there is an abstraction layer, I really tried to keep it close to Zend since they did an amazing job with their API. The advantages of doing things through the abstraction layer are graceful error handling as mentioned above, and consistent UTF8 handling without you having to think about it.

Hope this helps,
Chris

Chris and I just reviewed the code required to implement faceted tabs (identical to the way Drupal puts tabs into search now) using LuceneAPI, and it's pretty trivial (once we figured out a method that worked, which took some brainstorming initially), so I'll whip up a sample contrib module for this and post it in a new issue. Lucene Extras, Chris' planned place for this sort of contrib module will get a real release of it sooner or later.

A task has been set up at #498788: Document the hell out of the 2.x API.

Suppose I want to treat a CCK field as a facet (seems like the simplest use case), what is the hook for that? document_alter is for an external document, right? What about a node? Thanks for a very, very intriguing module....

This sample code worked for me, with one exception (below).

When I added 'location' as a CCK field to blog (could be any content) type, the facet search successfully searched on keywords with facet values in the location field only.

(What would be nice is an example with multiterm (OR-ed) values, not just a single string....)

The exception: luceneapi_add_subquery should be luceneapi_subquery_add -- although I left that function commented out, since it looks like it's already handled by the contributed luceneapi_node_module.

More confusing terminology. The sentence:

... [Lucene] will search the contents field by default ...

applies to Lucene fields, not to Drupal fields, which is confusing enough. Even more confusingly, luceneapi_node.module indexes Drupal fields, but only those built into nodes, like author and change date, and not CCK fields specific to content type. And even more confusingly, contents:foo will find "foo" anywhere in content, including CCK fields -- but won't display the hit that way.

Not a knock on the software at all or even the documentation! It looks to be like we're dealing with an intrinsically confusing situation here.

* * *

Am I right in thinking that hook_luceneapi_document_alter applies to a "document" from Lucene's perspective, and that therefore a Drupal node is a "document"? Or does the hook only apply to external documents?

@#6:

I don't understand, but as it's stated in #3 doesn't work for me (I'm working with 5.x branch).

Is there some modifications to apply to run it correctly?

Thanks a lot

The call in #3:

luceneapi_field_add($doc, 'keyword', 'location', $node->field_location[0]['value']);

should induce a re-index?

Maybe this is the reason for no search results when a "location" field is filled with a value.

Hi Alex.

This hook gets invoked when a document is being added to the index. After you implement this hook, you have to reindex and run cron. Also, I made a mistake in the above post. You should probably use "text" or "unstored" instead of "keyword". "keyword" fields are best used for things like node ID's or exact matches as opposed to textual data.

Hope this helps,
Chris

This thread only works with the 1.0 API, and is not applicable to 2.0.

Hi everyone,

I'm very intrigued by the possibilities of the Search Lucene API module and saw a reference to a contributed "Lucene Extras" module in Comment #4 above. Has the "Lucene Extras" module been released? I'm interested in trying out the faceted tabs feature....

Cheers,
Ben

P.S. I re-opened this issue because of the reference to Comment #4... if you'd rather that I open a new support request, just let me know. :-) Thanks!

Hi Ben.

Unfortunately the module never got off the ground for a number of reasons. The guy who came up with the cool faceted tabs stuff no longer works with me, but I can try to dig up some old code to point you in the right direction. I am going to close this issue and open a new one at #649458: How do I display facets as tabs?.

Thanks for bringing this up,
Chris

Hi cpliakas,

How can I display result by content type?

Thanks.