Configurable limit to the Result Set Limit

Hi scotjam.

Thanks for breaking them up, but I think they can all stay in this thread as well. I am still recovering from Drupalcon Paris, so it will take me a couple of days to get my feet back on the ground :-). I promise I will respond to these request in the next few days, as I think they are all important.

Thanks again for posting,
Chris

scotjam,

Let me address the issues one at a time.

1) I think this is a good request. Look for it to be added in the near future.

2) Unfortunately this is not possible because of the way search results are found by the Zend Search Lucene component. First, it is important to understand exactly what this setting does. Let's say that you have an index of 50,000 documents, and you execute a search that would normally match 20,000. If the result set limit is set to 1,000, Zend Search Lucene will stop searching after the 1,000th document is matched. Therefore, we have no way of knowing how many items would have been matched otherwise, and it also means that there could be better matches elsewhere in the index that the search never sees. In other words, the first 1,000 documents are matched, not the best 1,000. Since Zend Search Lucene is a fully integrated search engine library, it needs some mechanism to limit score calculation for larger indexes, otherwise page load time would be unbearable. This is the limitation that makes projects like Apache Solr and Acquia Search more attractive for larger websites, but the integrated nature of Search Lucene API makes it more attractive to smaller or medium sized sites because of it's simplicity.

Depending on what you are searching for, there could be some alternatives. For example, if you are searching a single term, you could use the luceneapi_termdocs_get() function to return how many documents contain the term. It is a very efficient and inexpensive query, and it is not subject to the result set limit setting. However, this solution won't work for user generated queries since they could be more complex than straight terms, and it doesn't take node access into account since it is more of a low level index query.

3) If you want to get the total number of results matched in the query, it is actually stored in the $pager_total_items global variable. For example, in a custom function you can get the total by using the following code snippet:

<?php

function myfunction() {
  global $pager_total_items;
  $lucene_total = $pager_total_items[0];
}

However, the total will never exceed the Result Set Limit if one is set. In the scenario described in #2, $lucene_total would be 1,000 and not 20,000.

I hope this clears things up, and thanks again for the good feature requests.
~Chris

Feature added in commit #268798.