The _autotag_search_field function is using preg_split to split the fields into words, which it then checks to see if they match terms or synonyms in the selected vocabularies. However, preg_split doesn't handle utf-8 correctly. Any accented characters are being filtered out. A word like "cumpleaños" is being split into "cumple" and "os". The strtolower function being used to set all characters to lowercase, apparently also does not handle utf-8.

I have found one possible solution, which is as follows:

Replace this line:
$words_including_small = preg_split('/[\ `!"£$%^&*()_\-+={\[}\]:;@\'~#<,>.?\/|\\\]/', strtolower($field), -1, PREG_SPLIT_NO_EMPTY);

with these 2 lines:

mb_regex_encoding( "utf-8" );
$words_including_small = mb_split('\W', mb_strtolower($field,'UTF-8'));

There is also one other call to strtolower lower down in this function which should also be replaced with mb_strtolower.

Comments

sdrycroft’s picture

Version: 6.x-1.27 » 6.x-2.0

Again, thanks for the input Steve. You're definitely right, I've had the issue of diacritics reported before, but thought that it had been fixed - clearly not. I should actually be using drupal_strtolower, which in turn uses mb_strtolower. Another alternative, would be to use the function search_index_split, although this would add a dependency on the Search module.

sdrycroft’s picture

Version: 6.x-2.0 » 7.x-3.0
Issue summary: View changes
Status: Active » Fixed

I believe this has been fixed in the 7.x version of the module. If users are still experiencing this in Drupal 7, please re-open this issue with clear examples of how to reproduce.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.