The _autotag_search_field function is using preg_split to split the fields into words, which it then checks to see if they match terms or synonyms in the selected vocabularies. However, preg_split doesn't handle utf-8 correctly. Any accented characters are being filtered out. A word like "cumpleaños" is being split into "cumple" and "os". The strtolower function being used to set all characters to lowercase, apparently also does not handle utf-8.
I have found one possible solution, which is as follows:
Replace this line:
$words_including_small = preg_split('/[\ `!"£$%^&*()_\-+={\[}\]:;@\'~#<,>.?\/|\\\]/', strtolower($field), -1, PREG_SPLIT_NO_EMPTY);
with these 2 lines:
mb_regex_encoding( "utf-8" );
$words_including_small = mb_split('\W', mb_strtolower($field,'UTF-8'));There is also one other call to strtolower lower down in this function which should also be replaced with mb_strtolower.
Comments
Comment #1
sdrycroft commentedAgain, thanks for the input Steve. You're definitely right, I've had the issue of diacritics reported before, but thought that it had been fixed - clearly not. I should actually be using drupal_strtolower, which in turn uses mb_strtolower. Another alternative, would be to use the function search_index_split, although this would add a dependency on the Search module.
Comment #2
sdrycroft commentedI believe this has been fixed in the 7.x version of the module. If users are still experiencing this in Drupal 7, please re-open this issue with clear examples of how to reproduce.