I would like to introduce a new module which will interact as an user interface for stemmers. This module is currently not released and I like to get some feed back before.
Appended is the so called "stemmer_api" in its initial state.
Please check if this module seems useful to you and send comments.

CommentFileSizeAuthor
stemmer_api.tgz11.37 KBschildi

Comments

jhodgdon’s picture

jhodgdon
she/her
Primary language English
commented

Can you explain in a few words what the purpose of the module is? Porter Stemmer doesn't need a user interface, as it uses a pre-defined algorithm. So all you can do really is turn it on or off, which you do by enabling/disabling the module.

schildi’s picture

schildi commented

Of course the algorithm itself is not effected, but

  • what you can do is check results of the stemming process on the fly. I had/have some case where searching led not to the results expected and then it might be useful to see if it has to do with stemming.
  • stop words are currently not handled by most Drupal stemmers what results in a bloated index. Lucene search offers to use a list of stop words, but core search don't. You can use a hard coded list of stop words delivered with the module and/or can user give a tool to set up this list.
  • and since all words are stemmed before handed over to the the search/index process the list of stop word should be managed by the stemmer and NOT by the search module.
  • the list of exceptions is expected to grow - at least in german there is currently no list for kind of porter stemmers. So users will add these exceptions as needed.

      Thanks for your question

jhodgdon’s picture

jhodgdon
she/her
Primary language English
commented

Hmmm...

Being able to check the results of stemming on the fly -- sounds like material for a completely separate module, as it's independent of whether you are using stemming or some other kind of preprocessing.

What do you mean by "stop words" resulting in a "bloated index"?

schildi’s picture

schildi commented

In every language you have a number of words which don't need to be indexed because they do not really contribute a certain amount of information.

In the above sentence these stop words might be:
in every you have a of which "don't" to be because they do not really certain

That means, most words can be wiped out before indexing the sentence. Remaining word are then:
language number words need indexed contribute amount information

This will reduce the resulting index about 50-60 percent and results in quicker responses.

By the way: It IS a module, but what I want to know is whether to set it up as a own module or - when developers of other stemmers signal me that there is no need for these functionalities - to deliver it together with the existing "de_stemmer" only.

jhodgdon’s picture

jhodgdon
she/her
Primary language English
commented

I see. It seems to me that stop words are not related to stemming. Stemming makes words match linguistically-related words, and what you are talking about is eliminating things like "the", "that", etc. in English (I think).

So I think that should be a separate module, since it is a separate function. There actually might already be a module that does this on drupal.org, so I would do a comprehensive search before creating a new module.

Good luck with your project!

jhodgdon’s picture

jhodgdon
she/her
Primary language English
commented
Status: Active » Fixed

BUT... be careful about stop words actually! The core Drupal Search module uses the search index when it does a search for a phrase. So if you elmiinate stop words, you have to make sure that phrases still work correctly. See:
http://drupal.org/node/1202#comment-40156

Anyway, I think we can mark this issue "fixed" now? Please feel free to re-open if you have more questions related to Porter Stemmer.

schildi’s picture

schildi commented

It seems to me that stop words are not related to stemming. Stemming makes words match linguistically-related words

Not really. There is no one to one relation between linguistic stems and the results of a stemming process according to a porter algorithm. The latter only aims to reduce words to a distinct form.

Phrases: Currently all Drupal stemmers are implementing hook_search_preprocess. Text to be indexed as well as search phrases undergo the same procedure and so there should be no issue.
I just verified this by searching for a phrase Schloss in Bensberg which contains the stop word "in". I got hits for the original phrase like Schloss in Bensberg as well as for Schloss Bensberg.

From my point of view using stop words is a procedure bound tight to indexing. For technical reasons (stop words could be changed by the stemming algorithm) they should be handled before stemming applies. So it looks logical to implement it as part of the stemmer in hook_search_preprocess.

Regards

Close issue

jhodgdon’s picture

jhodgdon
she/her
Primary language English
commented

Searching by phrase: You need to put the phrase in quotes, like "Schloss in Bensberg", to tell Drupal to search for the phrase, and you should only get nodes that have that exact phrase in them.

I agree, stop words should be applied before stemming, but I don't agree that it should be part of the same module as stemming. It should be a separate module with its own hook_search_preprocess.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.