On our swedish site we have been using fckeditor to input text and the fckeditor has been htmlencoding (å => å, ä => ä, ...) the input on save. luceneapi_node seem to break words on this encoding when indexing and when searching for words with international characters, matching fails.

I have hacked together that works on our specific site by converting the value in lucenefield_field_add using
$value = html_entity_decode(luceneapi_strip_tags($value), ENT_QUOTES, 'UTF-8');

I guess this is not a great solution performance wise, and I would love to hear another take on this issue.

CommentFileSizeAuthor
#4 luceneapi-658568-4.patch2.4 KBcpliakas

Comments

cpliakas’s picture

cpliakas commented
Status: Active » Postponed (maintainer needs more info)

Hi auth.

Thanks for posting. We use fckeditor on a number of our sites, and unfortunately I don't think there is a single one that it hasn't caused issues on :-\. In terms of your solution, thanks for submitting it. I wouldn't modify the Search Lucene API code, though. If possible, try putting this fix in a hook_nodeapi() implementation similar to the following:

/**
 * Implementation of hook_nodeapi().
 */
function mymodule_nodeapi(&$node, $op, $a3 = NULL, $a4 = NULL) {
  if ('load' == $op && !$a3 && !$a4) {
    $node->body = html_entity_decode($node->body, ENT_QUOTES, 'UTF-8');
  }
}

I haven't tested this code and the $op may need to be changed, but this way it separates things from Search Lucene API so you still have an upgrade path. Also, have you tested the core search with fckeditor, and does it work? After looking at the code my guess is that it does not, but I would be curious to confirm that.

My first inclination is to say that if the core search does not work either, then it should be the responsibility of the fckeditor module to prepare it's text for indexing. If fckeditor changes what it does in the future, I don't want to have to release a new version Search Lucene API to accommodate what it changed. However, I could be persuaded otherwise.

Thanks again for posting, and please feel free to post other findings or a better solution to the temporary fix I posted in this response.
~Chris

auth’s picture

auth commented

Thanks for the reply, and the suggestion to put the code outside for a cleaner upgrade path.

Regarding core search handling I have found it to handle html encoded & auml; and the likes by converting it to UTF-8 during indexing while splitting the words, and therefor not breaking up htmlencoded words. Look at the function search_simplify at line 305 in search.module.

From waht I have seen this is something luceneapi could do as well, but again, I am not aware of the performance implications this would add.

I will try to implement _nodeapi on $op == update_index to not mess with the output on the site. I will post my findings here.

cpliakas’s picture

cpliakas commented
Version: 6.x-2.x-dev » 6.x-2.0-rc2
Status: Postponed (maintainer needs more info) » Active
Issue tags: +6.x-2.0

Hi auth.

Thanks for the pointer to the file in the core search module. Since the core search handles this properly, Search Lucene API should as well. Marking this bug report as active, and tagging to be fixed in the 2.0 stable version, which is is next expected release. I also see that the Apache Solr Search Integration project takes the same approach as your solution, so I think I am going to run with it.

Thanks again for reporting your findings,
Chris

cpliakas’s picture

cpliakas commented
Status: Active » Needs review
StatusFileSize
new luceneapi-658568-4.patch2.4 KB

Here is a patch that applies the fix posted above. It also adds an API function luceneapi_html_prepare() so it can be used in other locations as well.

cpliakas’s picture

cpliakas commented
Status: Needs review » Fixed

Applied in commit #310902.

Status: Fixed » Closed (fixed)
Issue tags: -6.x-2.0

Automatically closed -- issue fixed for 2 weeks with no activity.

Issue tags: +6.x-2.0

Restoring issue tags, see #2125755: System messages removed all issue tags during D7 upgrade.