Integrate with Views front-end

yeah this is what I've been looking for...Anyone can help us here..Thanks

Is there anyway people can assist with this effort -- test -- code -- architecture?

you should try an alternate approach and see if speed is better. Try running the views query first, getting the set of nids, and then sending that to solr as nid:1,4,65,777,899 etc.

PS make sure and benchmark it somehow so you know which approach is actually faster.

Please feel free to open a support request in the views issue queue, or ask in the views developers group on g.d.o. Earl and the team are very helpful to developers writing Views support modules, and this one is a very interesting case that might have ramifications for other integration modules.

I guess this is it just good for views 2 D5. Nice job!

critical = blocking issue for 1.0 release.

Hmm, how about an actual patch to test? Or is the patch from http://drupal.org/node/254565#comment-900025 still valid?

Thanks!

Hmm... that patch is 74kB! Could you reduce it to patch *just* what you are trying to patch? =)

If HEAD is somehow out of sync with the DRUPAL-6--1 branch (is it?) then please patch against the DRUPAL-6--1 branch...

Hmm... I did look at it, but I admit that my previous comment was because I *just* saw a bunch of stuff at the top that's not really views-related. I think that we should handle code simplification seaparately from this patch.

Looking at it in detail, I think my opinion (only putting in my $.02 worth) stands... for example: the first hunk deals with putting duplicate code in function apachesolr_search_get_params(). To me this is cleanup and not views-related. Also, some parts of the patch are trying to patch old code (by now) or stuff that's active in other issues... (like adding the form to clear the index)

I'm just saying this needs a *little* bit of work so we don't mix issues!

And BTW, great work =) (I need it, so that's why I want it tested and commited) =)

@drunken monkey: if there are refactorings and cleanups that we could work on separately that'd be a good thing.

I've got a suspicion that the first iteration of this module is going to have to include a patch to views as part of the download.

Also, the views patch isn't linked to from this issue anywhere. If we care about views integration here we need to care for that patch as well.

"So should I at first delete the refactoring functions and just copy the code, and then post a patch for refactoring?" Yes, please.

Then go ahead and write the README as suggested. We'll commit your SoC work with the patch and have an ongoing lobby effort to get the patch into Views.

This patch does a few things all in one, contains a module apachesolr_views, modifies apachesolr files, and contains a patch itself (a patch created by a patch).

modifications to files:
* apachesolrlang.module
* apachesolr.module

adds the module:
* apachesolr_views

creates the patch:
* apachesolr_views/views_variable_query_class.patch
=> this patch is similar (but outdated) to the patch in the views issue

Therefore to install this patch involves patching twice:
* apply the merge patch to apachesolr module
* go into contrib/apachesolr_views and apply the newly created views patch
* download the latest views patch from #293841: Allow different back-ends for Views to views module and apply
* enable the apachesolr_views module

Could the apachesolr_views module be placed in its own drupal project or combined with apachesolr? It would make it easier to submit issues or patches to the module.

I tried out this module and tried using the apachesolr filter or arguments in a view, however it did not return the results I expected.

Thanks
Audrey

I'm wondering what the status is of this. It's something I could really use and would be happy to help out with if need be.

Seems like the views patch is near to be commited, #293841: Allow different back-ends for Views, but this patch is really old, so it would need to be re-rolled.

cool!
Which is the right patch to use? Your link is broken.

The module is pretty much empty, why not just do that as views integration for the main module?

+1 for doing it as views integration on the main module. Thank you for tending this patch for so long. I don't expect that we'll actually commit it until the views patch is in, and in a release, so please keep holding out a bit longer =)

@robertDouglas - given that this will likely require some specific version of views to work, I think we might want to keep such integration code as a separate module that can do its own checking via hook_requirements.

Won't that be taken care of by the views api itself?

+/**
+ * Implementation of hook_views_api().
+ */
+function apachesolr_views_views_api() {
+  return array('api' => '2.0');
+}

That's the only code that would be added to apachesolr.module, and I assume that the patched version of Views would have a version greater than 2.0. If that assumption isn't true, we'd need it in a separate module.

Good question, also see http://drupal.org/node/293841#comment-1281926

Anyway we are elaborating if there is better way to re-create existing views fields without having to reproduce all the code.

subscribing

subscribing

We can help test this, look for some feedback soon!

subscribe

subscribing... looks like i'm going to be doing some solr work in the near future.

I think this and any other really substantial sub-projects should probably get their own project that can be maintained independently. I suggested this to Moshe for the Solr-VotingAPI integration, and I think the same for this. Moshe understood well the motivation, since he at one point had hosted many /contrib projects under OG, but eventaully pushed them all out to separate projects.

subscribing

I suggest the following change

Index: apachesolr/apachesolr_views_query.inc
--- apachesolr/apachesolr_views_query.inc
+++ apachesolr/apachesolr_views_query.inc
 class apachesolr_views_query extends views_plugin_query {
*/
public function build() {
  if (empty($this->_keys)) {
    - $this->_query = FALSE;
    - return;
   + $this->_keys = '*';
  }
else if (count($this->_keys)) {
$this->_query = $this->_keys[0];

Allows you to construct a View without any search terms. ++ ya?

related #358166: Search for just facet(s)

#53 I think opens up a ton of great possibilities for creating views with solr backend -- definitely should be part of the main patch!

There is the bug in the current patch, needs this small fix (too lazy to reroll right now)

@@ -162,7 +165,7 @@ function apachesolr_views_data() {
       ),
     ),
     'filter' => array(
-      'handler' => 'views_handler_filter_apachesolr_type',
+      'handler' => 'apachesolr_views_handler_filter_type',
     ),

#53 can be updated as well to actually remove the *. so it looks like this

public function build() {
  if (empty($this->_keys)) {
    
  }
else if (count($this->_keys)) {
  $this->_query = $this->_keys[0];

and it works as well.

Need to study the behavior more to really dig into this.

This is a re-roll that fixes paging

Of note, it doesn't use pager_query any more (YAY!!!). Thanks to Earl's views code I was able to make the pager_query() stuff go away. Working on a patch for #345606: Unsed global $pager_total to do the same.

This patch fixes

• comments 53 and 55 - allow for no search keys and fix the hook_views_handlers() definition that m3avrck and i found
• fixes pagers so they work
• removes dreaded pager_query("SELECT %d", $params['rows'], 0, NULL, $total);

m3avrck pointed out to me that I removed the hook invocation of apachesolr_process_results. This was done on purpose.

Reasons

1. Views does all this magical stuff for us already. No need to duplicate it
2. The few modules that implement that hook expect the $items to be in a specific form, for the search results page. That form isn't the same form as the Views result

in order to get filters to work i did a lil magic in the apachesolr_views_query->execute() function

public function execute(&$view) {
    $view->result = array();
    $start_time = views_microtime();
    $query = $this->_query_object;
    $params = $this->_params;
    $solr = $this->_solr;
    
    // build in the filters
    foreach ($this->_keys as $key) {
      if ($type_end = strpos($key, ":")) {
        $value = substr($key, 0, $type_end);
        $type = substr($key, $type_end + 1);
        drupal_set_message("$type $value");
        $query->add_field($type, $value);
      }
    }
    if (empty($query)) {
      return;
    }

Working with Scott, here is an updated class that fixes ApacheSolr type to work like the node type selection when using the node backend.

/* $Id$ */

/**
 * Class for filtering by type.
 */
class apachesolr_views_handler_filter_type extends views_handler_filter_in_operator {
  
  public function get_value_options() {
    if (!isset($this->value_options)) {
      $this->value_title = t('Node type');
      $types = node_get_types();
      foreach ($types as $type => $info) {
        $options[$type] = $info->name;
      }
      $this->value_options = $options;
    }
  }

  public function query() {
    // to work around bug when keyword is not present
    $this->query->add_term('');
    
    if (empty($this->value)) {
      // Add term that will yield no results
      $this->query->add_term('nid', '-1');
    }
    else {
      $not = $this->operator == 'in' ? '' : 'NOT ';
      
      foreach ($this->value as $type) {
        $this->query->add_term($type, $not . 'type');
      }
    }
  }
}

phew ok this is a pretty significant update. Items that changed

• removed the add_terms function and replace with add_facet(). Its more clear and it provides a non 'strpos' method to add them to the solr query
• Load up the params specified in the apachesolr admin interface with the exception of the query field settings
• A new snippet field
• A new search score field and sort. Now by default Solr sorts by score, but you could add in other sorts. So say you wanted score, created as your sorts in that order, thus ties for score will be sorted by created
• significant rewrite of the db class to conform more closely with Views

Of note this patch doesn't make use of Solr_Base_Query. The apachesolr_views_query is analogous to it. As Solr_Base_Query handles the parsing of the urls, application of facets, and returning the query keys etc. So does apachesolr_views_query. They both are wrappers around Drupal_Apache_Solr_Service

doh! forgot to change the argument handler. And I wanted to add the node_view() style plugin for this..

After I applied all the patches mentioned, I dont see a view type of "Apache solr" when I try to create a new view. Any ideas on what might be wrong?

Did you apply the Views patch: http://drupal.org/node/293841#comment-1294986? and clear your Views cache? as plugins are cached.

After talking with both RobertDouglass and pwolanin we came to the conclusion that this is probably best implemented seperatally in a module elsewhere. I am in the progress of creating that project and rolling a -dev release.

The big thing we talked about was creating a way for the Views to plug into the facet blocks that work so well. My solution is to create an interface that defines a set of functions. Then all the facet_blocks have to do is to assume that query object implements that interface and away it goes. So this patch contains that as well as the needed changes to the function names in all places within the module.

It also removes the duplicate factory function. apachesolr_drupal_query and apachesolr_current_query are both labeled as factory functions, both really do the same thing except apachesolr_current_query is really used as a static cache. So thats what i did, made it a static cache for all those modules that call it.

I have tested this both with my newer Views implementation and with the standard Apachesolr and it works cleanly. It will also provide a way to properly define "apachesolr_query_class" with an interface by using instanceof operator.

All good things (tm).

This considerably smaller patch needs a review.

Heres the project
http://drupal.org/project/apachesolr_views

Yes, i have applied the views patch and cleared the views cache. still dont see the apache solr as a view type. any other way for me to debug ?

Ok i have done my first commit to the module. It provides really interesting stuff, and with this patch on this issue, it will actually work facet blocks provide that the facets you want are configured either has exposed filters or arguments. If they aren't set, clicking the facet link wont work. Its pretty neat, need to check out the CVS HEAD of course as this is still experimental.

updated patch with against the new Beta

I was able to get the apache solr view module from CVS. And now able to add a view. However I get a error when trying to preview the view. Attached the error in my next comment

Any ideas?

That can happen for a multitude of reasons. Views is telling you "Something went wrong."

If you create a page, give it a path and visit it, you will actually see the error. I get that alot when I forget to start up ApacheSolr

Quick comment:

function add_facet()

would be more accurately named

add_filter()

or even

add_filter_query()

Since we use this for node access and other sorts of added fq params that are unrelated to facets.

I think the change to apachesolr_drupal_query() may break the _nodeaccess module - the idea behind the current implementation is that you get a query object that can be manipulated as a subquery without it having the current search parameters filled in.

@75 I agree. According to http://wiki.apache.org/solr/CommonQueryParameters#head-6522ef80f22d0e50d..., fq stands for filter query

Not sure I understand #76 but I will play with that as its very important to us. There is a very small change to the nodeaccess implementation.

There is no change to apachesolr_drupal_query() there is a change to apachesolr_current_query(). They both, prior to this patch, returned the exact same thing, and took in the exact same parameters (This speaks loudly to this change btw. We are all getting confused on which function really does what and which one is really a factory function). But perhaps removing the static array() cache and replacing it with just a single static cache is the issue you are seeing?

Anyways, i will investigate, marking as need work until i confirm node_access works. And i probably need add in some more hand holding so its easier for people to get this up and running.

ok this is an updated patch taking both in notes about 75 and 76. node_access was indeed broken. This patch fixes it up.

This does touch a lot of code but mostly in minor ways, function name changing. With the expection of de-coupling apachesolr_drupal_query() and apachesolr_current_query(). My current Views implementation works with this patch

plz review

Missed the new date block. Heres the fix

patch looks basically fine - possibly needs a little cleanup in the doxygen for the interface (facet -> filter).

Obviously this will restrict a little our ability to refactor the query class if it must must conform to the interface - for instance, the handling of sorts is a bit ugly now compared to other parts of the code.

I agree sorts are messy. You can through the Views interface set up sorts and maybe that becomes the basis for the new sort setup in the query class? The way Solr_Base_Query works is by string, maybe basing it off what drunkenmonkey did for the Views interface is a cleaner way to go? Its fairly straight forward just adds adds the sort to its internal $params array.

we should probably refactor the query class to hold the arrray of params? we would need get/set/add_param() kind of methods - and I guess filters would be a special case in terms of having a separate set of methods since we are manipulating those so widely?

Not sure refactoring it that way would add anything. I think something as simple as maintaining a $sort_string property would do you just fine. That solves this problem.

In the apachesolr Views project I maintain a params array because there are three discrete stages (build, alter, execute). This allows me to semantically line up with how Views builds its queries. The standard search probably doesn't need it.

Of course with all this being said, I'm not sure how to get the facet sort blocks working with my implementation. It might have to be an alter hook that catches something in the url and changes the query (probably the same variable that the standard search uses). Probably end up being a lil hackish i think, but will make it work i hope. That is the one remaining thing.

I get this error when trying to access the view,

Fatal error: Call to undefined method views_query::add_filter() in /home/9a0440c5/web/public/modules/apachesolr_views/handlers/apachesolr_views_handler_filter_type.inc on line 50

http://desitrack.com/test/Business

I am using the latest code from apachesolr_views module. Also have applied the views patched as mentioned. What am I missing? thanks in advance.

dww reminded me today that we do want a better API for handling sort fields so that we can, at least, use field aliases. Also, solr does support sorting on multiple fields.

So, I may commit this, but likely we will want to alter the interface soon.

committed attached to 6.x (some code comment changes). needs follow-up for sort re-working, etc.

As already mentioned in #76 this patch breaks sub queries added via hook apachesolr_modify_query.

function foo_apachesolr_modify_query(&$query, &$params) {
  $sub_query = apachesolr_drupal_query();
  $sub_query->add_filter('type', 'page');
  $sub_query->add_filter('type', 'story');
  $query->add_subquery($sub_query, /* concat fields operator */ 'OR', /* concat query/subquery operator */ 'AND');
}

Before that patch a modification to the query object was not stored using apachesolr_current_query but applied directly before searching via solr.
Now the modification is stored which causes every link in facets to contain the sub query. For example a facet link for term id 27 looks like this (simplified):

filters=tid:27 (type:page OR type:story)

Now two bad things happen if you click such a link:
1. Hypothetically speaking, the sub query gets duplicated because the filters parameter will be parsed when creating the query object on the result page and become part of the main query. The original sub query will also be added via the hook.
2. You end up with an error message because parsing filters messes up the original sub query. The property fields of the query object looks like this after parse_filters():

array(2) {
  [19]=>
  array(2) {
    ["#name"]=>
    string(4) "type"
    ["#value"]=>
    string(6) "story)"
  }
  [0]=>
  array(2) {
    ["#name"]=>
    string(3) "tid"
    ["#value"]=>
    string(1) "5"
  }
}

Type "page" is gone and "story" became "story)". This ends up in a solr request containing fq=type:story) which causes a ParseException.

Does this work?

function foo_apachesolr_modify_query(&$query, &$params) {
  $query->add_filter('type', '(page or story'));
}

I do have mixed feelings about sub_queries. And perhaps this is really what subqueries are for (being totally separate from the actual query), and that has value.

perhaps in the _clone() method, we remove any subqueries?

Ok here ya go. I realized my error. I moved this call: apachesolr_current_query($query); to far down through the process. This lil patch should fix it. Plz test it

Looks like a reasonable fix.

@ #90
I had the same idea. Unfortunately it didn't solve the issue.

You removed cloning from apachesolr_current_query(). So it doesn't matter if you store the query before or after apachesolr_modify_query hook. You always modify the same query object passed via reference.

Here's a small patch that gets sub queries to work again in my setup. But it reintroduces cloning so I don't know which side effects this might have.

In fact, it should probably clone it both on the set and the return

yep pwolanin is right, needs to clone on both set and return.

I got lazy...

Here's a combined patch - please test so we can get this fixed asap.

helps if the variables have the right name.

this is working for me well enough, so I committed it, but please confirm and marked fixed.

Works here as well.

Nice feature thanks all. Much easier to theme Apache Solr search result page with Views theming. Subscribing.

Personal note: Latest updates are posted on Apache Solr Views module project page at http://drupal.org/project/apachesolr_views