Search multibytes language

Some more thoughts and pointers (but no fixes):

• You should check whether both charsets match; if your database uses ISO-8859-1 and the data you want to search for is provided in gb2312, then such search might not yield (m)any results.

  I think that is why searching for 这是一个非常好的软件。 on drupal.org yields no result. It is stored in the database as è¿æ¯ä¸ä¸ªé常好ç软件ã.

  Maybe use this page as a starting point.

• Nonetheless, I suspect that the search module is being "dumb" as well. At this point, I think it is best to put some debug statements in the search module (or maybe install/enable the devel.module) to check what search-related SQL queries are being invoked.
• By the way, is there any particular reason you can't use UTF-8? The reason I'm asking is because PHP is designed for ISO-8859-1, a subset of UTF-8. Some functions are "UTF-8 aware" but for gb2312 support, you probably need PHP's multi-byte string support.

A Japanese user had similar problems with search.module. Aside from encoding issues, the main problem was that like Chinese, Japanese does not use spaces, so entire sentences are being indexed rather than words. Unless I'm mistaken, Drupal only searches for identical word matches unless you explicitly tell it not to, using wildcards.

For Japanese, there is a utility called kakasi which splits sentences into words (actually I believe its main purpose is Kanji-to-Kana conversion, but it also does the splitting). Perhaps there is a similar utility for Chinese?

Another problem is the call to strtolower to convert the search string to lowercase. It seems PHP automatically assumes a Latin-1 character set, and thus converts a bunch of bytes in the 128-255 range (which are used for non-ASCII UTF8 encoding) to their Latin-1 uppercase/lowercase equivalent, e.g. Á (0xC1) to á (0xE1).

I checked it using the following script:

  for ($i = 32; $i < 256; ++$i) {
    $a = chr($i);
    if ((strtolower($a) != $a) || (strtoupper($a) != $a)) print "$i ";
  }

The results probably depend on your system's language/locale settings, but over here I get a significant amount of conversions in the upper ASCII range.

This is not really a major problem, because the indexed words are lowercased too. It simply means that data in the search index is in a somewhat encoded form. On a rare occasion, it could cause extra unrelated search results, though often the strtolowered characters are either invalid, or located in a completely different character range, making overlap with other unrelated words very unlikely.

In fact, Chinese and Japanese characters seem mostly lowercase-safe. However Russian for example gets completely mangled.