There is no current way to transliterate all known UTF-8 characters into its ASCII form, so i made a transliteration file, in replacement of i18n-ascii.txt. It is based on the Text::Unidecode Perl library, which i converted into .ini format leaving out the transliterations that are matching ctype_punct and only leaving in transliterations that are matching with ctype_print.

The file can be used as is, or if needed remove the parts not used by your site.

Here is the conversion method used to convert the Perl library's conversion table into .ini format:

header('Content-type: text/plain; charset=utf-8');
$cdata = array();
print "; global transliteration\n[default]\n";

function unichr($num) {
  if ($num < 128) {
    $utf = chr($num);
  }
  else if ($num < 2048) {
    $utf = chr(($num >> 6) + 192) . chr(($num & 63) + 128);
  }
  else if ($num < 65536) {
    $utf = chr(($num >> 12) + 224) . chr((($num >> 6) & 63) + 128) . chr(($num & 63) + 128);
  } 
  else if ($num < 2097152) {
    $utf = chr(($num >> 18) + 240) . chr((($num >> 12) & 63) + 128) . chr((($num >> 6) & 63) + 128) . chr(($num & 63) + 128);
  }
  return $utf;
}

for ($i=0; $i<256; $i++) {
  $fcode = str_repeat('0', 2-strlen(dechex($i))) . dechex($i);
  $fname = 'Unidecode/x'. $fcode .'.pm';

  if (is_file($fname)) {
    $data = file_get_contents($fname);
    preg_match('#\s*=\s*\[((?:(?!\];).)*?)\];#sS', $data, $matches);
    $replacements = preg_replace('/{\s*,\s*}/', '{,}', array_pop($matches));
    
    $cdata[$i] = preg_replace_callback(array('!qq{((?:[^}]|\\})*?)}!', "!['\"]([^']*?)['\"]!"), create_function(
               '$matches',
               '$char = array_pop($matches);'.
    'return trim(stripcslashes($char), "[ \n\t\r\0\x0b\\\/]");'),
      preg_split('/,\s+/', $replacements));

    foreach ($cdata[$i] as $code=>$charmap) {
      $char = unichr(($i << 8) + $code);
      if (preg_match('/^\S*$/', $charmap) && $code < 256 && trim($charmap) && trim($char) && ctype_print($charmap) && !ctype_punct($charmap) && $char != $charmap) {
        print $char .' = "'. $charmap ."\"\n";
      }
    }
  }
}
CommentFileSizeAuthor
i18n-ascii-full.txt469.86 KBPoetro

Comments

greggles’s picture

Wow, that's great!

So, I guess the answer would be to provide this file in place of the i18n-ascii.example.txt file, right?

Or maybe to provide it in addition so that people don't just use that massive file (potential performance issue?).

Poetro’s picture

I would provide it in addition, because this could really be a performance issue, as this file is really HUGE, and parsing and reading the file can take a bit of a time not to mention bulk update of paths.

greggles’s picture

I just realized that this file is about a half a megabyte which is indeed quite large. What do you think about just adding a page to the handbook and INSTALL.txt that says "if you need more characters or help building a character file for your language see handbook/page/x" which would then talk about downloading/using this file?

Poetro’s picture

Placing a link in install.txt and adding the link to the handbook is a fine idea. For most sites this huge file might be an overkill, but it is great to get guidelines or copy parts from it.

greggles’s picture

Title: Full transliteration for the whole Unicode character set » help users with a full transliteration for the whole Unicode character set
Status: Needs review » Fixed

Great, I created a help page at http://drupal.org/node/185664 and added some text to the INSTALL.txt which refers to it. If you have any feedback about updates to the handbook page you can leave them as a comment there or here.

Thanks for your assistance with this!

Anonymous’s picture

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for two weeks with no activity.