Search improvements

+1 This contribution makes substantial improvements to all aspects of search, transforming the core search module from a very limited tool to a refined one.

Comments:

• With the ranking of results, it might be useful to provide settings in the admin settings page to tweak the score applied to specific parameters.
• Could the scoring distinguish between fields that the term occurs in (e.g., title score is higher than body)?

A couple of issues/comments:

• The results of a 'user search' look somewhat dull: it is repeating each username (using Xtemplate).
• Using the admin search forms (both admin/node/search and admin/user/search) result in a 'Call to undefined function'.

Committed to HEAD. This is _big_ IMO. Thanks a bunch Steven.

First off, the changes to search are amazing and work quite nicely.

This is a feature request. I did notice that noise word support has been taken out. I would like to make a request for noise words to come back.

As we all know a search for "the" on an english language site will return every single node - and a search for "the monkey" will likely return more results than just "monkey". Perhaps not the best use of processor time and bandwidth.

I think a good approach would be to:
1) Still index every word of every node.
2) Strip out noise words from queries before the sql query is built during search.
3) Return the results.

By indexing every word it would still allow for future improvements like allowing users to insist that a noise word be included in the search (e.g. +the monkey)

andre

Regarding Noise words:

Perhaps one of the reasons noise words were not used by users was that they didn't actually work. See http://drupal.org/node/11636

As for what words to use etc. This should be a user defined option as it was in the previous version of search. The addition of a page to the Handbook might help users choose their words wisely. Perhaps something along the lines of http://drupal.org/node/1202

As for how to go about removing the noise words: My thinking was that if the words are removed from the keys prior to the search query being built it would reduce an overly complicated SQL statement filled with exceptions.

Overall the pseudo code may be something like:

create an array of noiseword patterns pulled from some kind of storage (e.g. a table or field originally populated by an admin configuration page for noisewords)
//array might look something like this
//$noisewordpatterns[0] = '/noiseword0/'
//$noisewordpatterns[1] = '/noiseword1/'
//...
//$nosiewordpatterns[n] = '/noisewordn/'

define the replacement values
// in this case it need only be $noisewordreplacement = '';

do a preg_replace on a $keys string
// $keys = preg_replace($noisewordpatterns, $noisewordreplacement, $keys);

optionally add some code to out put a message to the user which if any noisewords were not included in the search
// output may be something like explode($matchednoisewords)." are common words that were not included in your search";

build the sql query based on the new values in $keys

How the noise words are stored and the admin interface for the words are a design choice.

I also figure that this approach would allow users to use an operator like '+' to indicate that they know its a common word but would still like it included in the results. (e.g. "+the monkey" ala Google) Since "+the" would not match anything $noisewordpatterns it would NOT be removed from $keys in the suggest method above.
Then it would only be a matter of a quick str_replace("+", "", $keys) - just priror to building the search query

I wish I was ready to contribute a patch of my own, but I'm still learning the drupal api and hooks.

Any thoughts?

andre

Before reintroducing noise words I want to evaluate the current search module improvements. I didn't like the 'noise words' feature to begin with, and chances are the new search rating/ranking makes it (partly) redundant.

I would certainly rank 'noise words' as a lower priority than the search features that are currently being introduced, but regardless of that noise words really do skew results.

And after scanning at the ranking code in the search module, I can see cases where noise words can further skew the results. For example

<h2>A list appart</h2> or
<a href="contact">Contact Be Circle</a> or
<stong>What to do in the event of fire</strong>

Will all bump words that need no help (in the last example 6 out of the 8 words that don't need help get improved scores).

Then again how noise words are indexed is a different feature request. i.e. Still index noise words, but DO NOT assign greater weight/score to them.

In my particular installation i'm likely to hard code a solution to remove noise words from searches for my own site - (mainly becuase my company Be Circle has a noise word right in the title and it shows up a great deal in my site - and any search that includes the word "be" is going to skew the results in ways that won't be useful for my visitors.)

Still, IMO noise words, as a feature, has many benefits to users that care to implement it. And while 99% of drupal sites didn't implement them, that 1% might be annoyed that a feature they use has been removed.

I'll leave it at that. After all this is just a feature request. I will try to provide a patches of my own when I am capable.

andre

Regarding these patches:

Wildcards do not appear to be working (in my installation or on drupal.org)

e.g. search for drup* does not return results

andre

Regarding these patches:

Wildcards do not appear to be working (in my installation or on drupal.org)

e.g. search for drup* does not return results

andre

Committed to HEAD! Thanks.

This really caught my eye, as I'm working on a bilingual en/ja website, and search is really the achilles' heel. I've discovered a simple patch to the standard search.module that allows it to search Japanese content, but this would be much better. Any specific tips on hooking in, say, Namazu to handle searching?

It also occurs to me that noise-word handling could be treated as a different kind of lexical analyzer, although I have no idea exactly how that might be coded up. But I can imagine an array of search-augmenting plugins that handle noise words in different languages using the same interface.

The fact that the version of this issue was changed has caused confusion.

adamrice: The 4.5 branch is in a bugfix-only state. All new features and enhancements go on the HEAD branch. Many search improvements have already been committed to the CVS HEAD branch... perhaps you could try it out and see if it fits your needs? It sounds like your questions would be more appropriate for the forum or drupal-support list.

Is anybody maintaining a version of this patch for the 4.5.x series? I know it can't go in the main distro but it'd be nice if somebody more familiar with how it works could update it for the latest 4.5.x release. Most of it applies except for a couple of lines in node.module.

You know what, I had to get it working anyway.

Here's a patch for this against 4.5.1

Be warned, if any modules implement the _search hook, it's probably wrong (such as the event module).

I just commented it out, does this mean I can't search events? Or do I need to copy the event_search from CVS?

Could you be a little more specific, I'd gladly add them to the patch.

I searched the cvs logs for anything commited to the Drupal repository from when dries initially checked in your changes until now and manually added them to the code. Some of those were good changes, thanks for warning me.

Here's the updated patch. Seems to work well, let me know if anything important is missing.

Hi pyromanfo, thanks for the patch!

Has to be applied with 'patch -R -p0 < search_4.5.1_0.patch'. After recreating the DB tables all data was lost but search works fine;)

What about a 4.5.2 version?

I can't seem to find any confirmation of this anywhere, but from what I can see 4.5.2 already includes the contents of this patch. It's not in the release notes but it seems to have everything this patch does.

I downloaded the 4.5.2 tarball from the download section here and everything from this patch seems to be already in there. The new database structure, changes to node.module, comment.module, search.module and common.inc are all actually in the code. If it wasn't meant to be in there, okay, but the contents of the 4.5.2 search.module is pretty much exactly what the 4.5.1 search.module with this patch looks like.

Yeah, that would be it. The patch was backwards and I had no idea which version I was talking about.

So here's the 4.5.2 patch for the search improvments.

Whoops, uploaded wrong file. This one should work.

today i upgrade a site from 4.5.2 to the cvs version, but now i gor a error with the search:

Compilation failed: characters with values > 255 are not yet supported in classes at offset 52 in /var/www/floor/modules/search.module on line 257.

does one one now's what is causing this?

greetings

MM

where can i find that? my pages are hosted at www.digitalus.nl.

thanks for the help

MM

Digitalus is not a good host for Drupal. I have been hosting on them for a year, but moved. because of all sorts of issues, which they refused to fix.

offtopic: what kind of problems. i have good experience with them and i was planing to move a onther site also toi them so i could use drupal. ( i can't at thew present host of that site)

How can I apply this search patch to my Drupal installation? My current search doesn't even work and this sounds like it's much better than the original search module included in (non-CVS) 4.5.2.

Was all this wonderous tweeking included in the standard release 4.6 search.module?
or is (some of) it still a patch?