Design a better all-purpose query builder and API

Currently, my custom query builder fits to a minimal need (using only q and fq parameters). The first thing to do to get a better query builder is also handling the bf, bq then facets parameters with high level functions.

With this, API would be nicer to use, and code centralized into the query builder. Note that I did separated the params handling in a class that the query builder extends, I don't think it's the right way to do it.

I started sometime ago a type based query builder (a class for 'term', for 'type:term', etc.. handling boost, fuzzyness, distance modifier nd other stuff like this in specialized classes) but it started to be a big API, I stopped doing it this way because of existing PECL packages, I did not want to reinvent the wheel, and this was kind of really heavy (I think bad for performance also).

Another stuff to add is the a full cache construction for final query when calling the getRawOutput() method, and cache invalidating when modifying stuff in it, but it's not that easy to handle. The goal here was for me (which is not a need for 'apachesolr' module) the ability to serialize and store the full query object into database, load it in the future and run the query without the whole query construction phase.

And a new version.

Usage:

// Instanciate the object
$params = array('fl' => 'nid,score', 'op' => 'OR', 'rows' => 10);
$query = new AssistantSolrQuery($params);

// Set some q parameters
$query->q->addQuery("test"); // Add the term "test"
$query->q->addQuery("test suite", "title"); // Add the phrase "test suite" to field "title"

// Some fq parameters
$query->fq->addQuery("page", "type"); // Add the filter "type" with value "page"

// Also works with 'bq'
// In case a non supported query string is given, it throws an SolrQueryException instance

// Then get full rebuild param array including well formated query strings
$params = $query->getParams();

// Then use 'apachesolr' API to query:
try {
  $solr = apachesolr_get_solr();
  $response = $solr->search($params['q'], 0, $limit, $params);
  // Do stuff with response
}
catch (Exception $e) {
  // In some cases, apachesolr module leave some exceptions uncaught
  // Which should be resolved by the way
}

This is the code how I actually use it with my custom module.

Reminder from the original thread, ideas were:

• Need a more abstracted API
• Export CCK fields handling to hooks implementation, in a custom file (but not another module)
• Build a new query constructor object
• More customization options for params array, and generalized construction functions/object for this
• Better error handling for core query handling (try not to throw exception to submodules)
• Other stuff ?

I don't know if you are still working on the module, but I had to do a lot of improvements to my custom Solr query API, and I also does read a lot more about Solr/Lucene syntax, and other stuff.

My code is not perfect, but is really convenient. You might wan't to take a look on it. It's growing and coming more accurate. I heavily use OOP code.

New stuff:

• Accurate terms escaping based on the real Lucene syntax special chars specification
• Term abstraction, with prohibit or require operators support, as well as per term or per phrase boost handling
• Sub-query handling, with a clean AND and OR
• Multiple term per field support (using the term abstraction if needed)
• Preliminary Filter Query (fq), and natural Query (q) and Filter Query (fq) support

Still missing, but useless right now for me:

• Boost Functions (bf) and q.alt support
• Fuzziness and Roaming support
• Better boost support (current is based on some abitrary values that plays nicely with my usage)
• Facets supports

Parts of code are still ugly (those I don't really use actually), all missing stuff can be added using some API workarrounds without modifying the code (facts supports, better boost handling).

Code attached. Feel free to use/report bugs/contribute, copyright goes to me, licence is GPL, it will commited in a new module soon whatever happens.

New version, more abstract, base on a "Statement" and a "StatementCollection" abstraction. Every statement can be used in combination with require or prohibit operators and/or fuzzy and boost operators (I don't handle the furry/roaming operators yet).

A statement per definition is a collection (using parenthesis if contains more than one element), each collection have an operator (AND, OR, or default which is ' ' which means let apache solr uses the default operator).

A query is a collection of statements or a collection of collection.

A statement is a term or a field filter, a field filter is a collection of terms prefixed with the field name. Each term is a statement. A collection is a statement, a field filter is a statement, etc, etc.

I do real abstraction of what is the Lucene syntax -not handling Solr specialties right now- (in a really simplified manner), but it's much more efficient (lower runtime), much more clean designed (heavy usage of oop abstraction/inheritance), and also is lighter in term of code base (much more maintenable too).

The problem with this approach is I rewrote the full code and had to get rid of qf support (which will come back as soon as I really need it). Also separated syntax handling and glue with the apachesolr module in two files.

See attached files.

I must warn you that I'm currently working on this code, for my custom module. It's moving fast, evolutions and bugfixes every day. It will continue so until next week.
Once this delay passed, I will commit a new module on drupal.org that contains it. This does not close the door to any eventual code moves between my custom module and yours. If want to integrate some pieces, I'll be open to any code refactor if it ensures that my code goes more generic and can help you building a strong API oriented module I could use myself as base for my module.

Whats your module going to do that you are going to add to drupal.org? What does it do that Apache Solr Search Integration doesn't ? Or Apache Solr Views doesn't?

It's a views integration like module, but using a really simple UI (a simple filter builder UI for end users). It automatically feeds node reference fields using the user configured filters in order to do automatic or semi-automatic input.

The full node reference filling process is based on a task queue (jobqueue module like, but custom we needed to have customized the code here) through a PHP CLI deamon that runs and wake up quite often (DataSync like, but much more lighter).

In fact, to ensure the best reactivity we do not use the cron at all. Our PHP CLI daemon uses posix_* functions and handles well SIG* signals (we never experienced any defuncts at all, opposite to DataSync module which seems to have some problem to maintain their own daemon alive through time on some environements).

We are pushing the module usability to a really simple UI (D7UX like if you prefer saying this way) and trying to accomplish the best performance we could have.
And, we are also going to do nice other stuff arround this, can't tell you more right now, we must focus on basic features.

The fact I contacted you is the better is your API, the lighter is my module (and the stronger is your API, more my module will be maintenable, less code, better encapsulation, and this would avoid to commit an extremely large code base).

Edit: And I would complete this (almost full) description that this is not a duplicate in anyway of what's existing right now in the stable pool of modules. The fact is views is like some guerilla warfare we don't need here and that isn't user friendly at all. We also needed to handle sometimes complexe Lucene/SolR query with a nice OOP API (easier to develop, and fair enough in performances) that your API do not provides.

Re-edit: Date of release is getting clearer, we will begin our validation phase and finalization process within the next week.

@robertDouglass, it uses the same schema, I use your API to use SolR, the only fancy stuff I do with SolR is complex queries in fact!

EDIT: not that complex in fact, but playing with q and fq a lot, and () and operators and ^ and later ~

I may look deeper in the SolR config someday, but when our sysadmins will have the time to study more solr.

GREAT MODULE!

Thanks for this cool API. I started and deleted my own code because I wanted to add some FunctionQuery to the fq parameter.

Please update any progress! This sounds very very promising!

Just in case: http://drupal.org/project/assistant && http://drupal.org/project/assistant_ref && http://drupal.org/project/assistant_search are three apachesolr based helper modules that allow a higher level of refinement to searches (it does not replace the legacy apachesolr module in any way).

EDIT: @robertDouglass If you test those modules and see any part of code you would wan't to use in your own module, I'm ready to ear about and share it. The more core is shared, better will be the API for both modules.

I don't know if this is really the good issue to tell it, but refering to those issues: #664896: Automatic CCK introspection, #664866: Multiple indexes for one CCK field and #751004: Flag node for re-indexing if the 'exclude' setting is changed for one of its fields, it seems that the CCK part of apachesolr module has a lot of edge case problems and glitches. Might it be better to export all the CCK handling part to an optional module with a better and cleaner API?

EDIT: Please type the right module human readable names of Assistant* modules, which are Solace API, Solace Node Reference and Solace Search, "assistant" is the old codename (sadly remains the the codebase name and project canonical name).

No patch present.

Marked #578346: Nice example query class from typo3 duplicate.

Marked #898860: A low-level chaining API for query building duplicate (nice link to github code there: http://github.com/technosophos/SolrAPI)

I did some work to allowing chaining for certain method calls on the query object. We should get any really substantial API changes in soon (before RC). I think looking at what's needed for effective Views integration is likely to be the main driver for changes at this point.

Hello, I checked quickly the github code, seems a good high level API for the actual apachesolr module use case.

My own code is an API on a lower query construction level, I think both can be used together. My code is a OOP representation of terms, sentences and expression which allows to build complex object tree that will finally spawn the real query by casting it to string.

It has evolved since, you can find a more up to date version in the Solace module suite code, it may be interesting that you get it, it's probably the most interesting part of the whole module :)

@pounard - yes, your query builder seems more focused on the case of correct client-side query construction, a need we've totally tried to avoid by using dismax/edismax.

Yup, my original use case was the need to create complex queries, with subqueries, playing with q, bq, bf, and fq simultaneously, which I did quite well using lucene syntax instead of solr syntax (q=!{lucene}
).

This kind of complex query could be über powerful when using Views backend (especially fq could be used for basic query, and filters then goes to q or facets).

I understand that this issue here is to try and get a decent API working first and then use that to eventually get to a useful-to-end-users UI (query builder). So, I am subscribing in order to know when there is anything ready for testing.

Thanx for your work on this Robbert, Peter and Pierre ;)

PS: are you aiming for something similar to http://drupal.org/project/apachesolr_facetbuilder perhaps?

No, this has NOTHING to do with end users. It's an API-level discussion. Query building for end-users == Views.

Aha I see. Thanx for taking the time to reply Peter.

marking postponed until we see if any of this is needed for Views.

making closed for now