For example, I know what [[test is in my content, but neither contents:"\[\[test", nor contents:\[\[test, nor contents:[[test find it -- what is found is test as if the brackets were thrown away in the indexing process. But that makes no sense, since the lucene documentation explains how to escape them, so why filter them out?

Comments

cpliakas’s picture

cpliakas commented
Component: Documentation » Code
Assigned: Unassigned » cpliakas
Category: support » bug

Hi shunting.

I will look into this. The indexing is handled by the selected analyzer class in the Zend Framework, so I will test it out to see exactly where the characters are being filtered out. Let me get back to you on this one, because I don't have a good answer for you right now.

Thanks for pointing this out,
Chris

shunting’s picture

shunting commented

You're welcome.

The characters I am especially interested in are colon (:) and square brackets ([]).

You will note the interesting use case that, since Drupal stores content exactly as entered, and not as transformed for presentation into HTML, lots and lots of input filters are feeding square-bracketed markup to be tokenized and transformed. It would be nice to be able to search on those, eh? Since they're the closest thing we have to truly semantic markup in content right now (as opposed to whatever is being done with fielded data...

FWIW, exactly what you paid for it, there's a stop list of some sort, or the definition of what word boundaries are. Make those administratively configurable, maybe? Or at least in an include file in the module?

cpliakas’s picture

cpliakas commented

I like that idea. Support for stopwords would be fairly easy to add, I just need to figure out how to integrate it with the analyzer selection option that will be added sometime in the future (i.e. standard, english word stemming, etc.). I still have to track down where the boundaries are set.

shunting’s picture

shunting commented

I think that what I'm really looking for is a new tokenizer; I poked around the lucene source code for not a long time, and (a) the file layout is opaque to me (and can we add new tokenizers in Drupal file layout, say in contribs???) and (b) the objects are even more opaque to me (complete inexperience -- I'm used to seeing functions and config files somewhere....)

So, maybe what I'm asking for is not an easy thing. OTOH, it seems to me that supporting a use case that searches literally the only semantic markup there is in the Drupal world has to be a good thing.

There's an XML lucene library, yes? Perhaps that would offer some clues on tokenizers.

cpliakas’s picture

cpliakas commented

I added functionality to add custom analyzers via hook_luceneapi_analyzer() in commit #237532. You can therefore select and use any custom tokenizers you choose. In fact, the English word stemmer is a third party stemming solution written by Kenneth Katzgrau.

shunting’s picture

shunting commented

[rubs hands...] ;-)

I have a really stupid, newbie question. How do I grab the current state of the module? I want to start working with it.

(The stemmers, I take it, all live in the lucence part of the file layout, and not the drupal part?)

cpliakas’s picture

cpliakas commented

No problem. I confirmed that all changes made it into last nights development snapshot, so you can get the changes from the dev release on the project page. In terms of the English stemmer, you are correct in that it is bundled with the ZF components. You are able to select which analyzer is used in the Admin -> Site Configuration -> Search Lucene API admin page.

There are still features being added to the facet section, but the API is in place and is working out well. I think it can be extended to add the advanced features similar to the Faceted Search module. I also added two pages to the documentation last night, so I am hoping to get a rough draft of the API documentation on line by the end of the week.

cpliakas’s picture

cpliakas commented
Assigned: cpliakas » Unassigned
Status: Active » Closed (works as designed)

shunting,

Sorry for letting this sit for so long. I have looked into the issue further, and it seems that the parser correctly catches escaped characters. For example, the query nid:[10 To 20] searches nodes from ID 10 to 20, whereas nid:\[10 To 20\] searches the terms \10, TO, and \20. However, what I found is that the analyzer strips punctuation out. Search Lucene API does have the hook hook_luceneapi_analyzer() to add custom analyzers, and one would have to be created to index and search punctuation. I hope this satisfies your initial support request. I am marking the issue as "be design" because the ZF components are working as intended. The status change is directed towards my assumption that this was a bug.

Thank you very much for pointing this out and getting me to look further into the Zend_Search_Lucene component,
Chris