Closed (fixed)
Project:
Pathauto
Version:
5.x-2.x-dev
Component:
Miscellaneous
Priority:
Normal
Category:
Feature request
Assigned:
Unassigned
Reporter:
Created:
16 Oct 2007 at 11:25 UTC
Updated:
12 Nov 2007 at 22:42 UTC
There is no current way to transliterate all known UTF-8 characters into its ASCII form, so i made a transliteration file, in replacement of i18n-ascii.txt. It is based on the Text::Unidecode Perl library, which i converted into .ini format leaving out the transliterations that are matching ctype_punct and only leaving in transliterations that are matching with ctype_print.
The file can be used as is, or if needed remove the parts not used by your site.
Here is the conversion method used to convert the Perl library's conversion table into .ini format:
header('Content-type: text/plain; charset=utf-8');
$cdata = array();
print "; global transliteration\n[default]\n";
function unichr($num) {
if ($num < 128) {
$utf = chr($num);
}
else if ($num < 2048) {
$utf = chr(($num >> 6) + 192) . chr(($num & 63) + 128);
}
else if ($num < 65536) {
$utf = chr(($num >> 12) + 224) . chr((($num >> 6) & 63) + 128) . chr(($num & 63) + 128);
}
else if ($num < 2097152) {
$utf = chr(($num >> 18) + 240) . chr((($num >> 12) & 63) + 128) . chr((($num >> 6) & 63) + 128) . chr(($num & 63) + 128);
}
return $utf;
}
for ($i=0; $i<256; $i++) {
$fcode = str_repeat('0', 2-strlen(dechex($i))) . dechex($i);
$fname = 'Unidecode/x'. $fcode .'.pm';
if (is_file($fname)) {
$data = file_get_contents($fname);
preg_match('#\s*=\s*\[((?:(?!\];).)*?)\];#sS', $data, $matches);
$replacements = preg_replace('/{\s*,\s*}/', '{,}', array_pop($matches));
$cdata[$i] = preg_replace_callback(array('!qq{((?:[^}]|\\})*?)}!', "['\"]!"), create_function(
'$matches',
'$char = array_pop($matches);'.
'return trim(stripcslashes($char), "[ \n\t\r\0\x0b\\\/]");'),
preg_split('/,\s+/', $replacements));
foreach ($cdata[$i] as $code=>$charmap) {
$char = unichr(($i << 8) + $code);
if (preg_match('/^\S*$/', $charmap) && $code < 256 && trim($charmap) && trim($char) && ctype_print($charmap) && !ctype_punct($charmap) && $char != $charmap) {
print $char .' = "'. $charmap ."\"\n";
}
}
}
}
| Comment | File | Size | Author |
|---|---|---|---|
| i18n-ascii-full.txt | 469.86 KB | Poetro |
Comments
Comment #1
gregglesWow, that's great!
So, I guess the answer would be to provide this file in place of the i18n-ascii.example.txt file, right?
Or maybe to provide it in addition so that people don't just use that massive file (potential performance issue?).
Comment #2
Poetro commentedI would provide it in addition, because this could really be a performance issue, as this file is really HUGE, and parsing and reading the file can take a bit of a time not to mention bulk update of paths.
Comment #3
gregglesI just realized that this file is about a half a megabyte which is indeed quite large. What do you think about just adding a page to the handbook and INSTALL.txt that says "if you need more characters or help building a character file for your language see handbook/page/x" which would then talk about downloading/using this file?
Comment #4
Poetro commentedPlacing a link in install.txt and adding the link to the handbook is a fine idea. For most sites this huge file might be an overkill, but it is great to get guidelines or copy parts from it.
Comment #5
gregglesGreat, I created a help page at http://drupal.org/node/185664 and added some text to the INSTALL.txt which refers to it. If you have any feedback about updates to the handbook page you can leave them as a comment there or here.
Thanks for your assistance with this!
Comment #6
(not verified) commentedAutomatically closed -- issue fixed for two weeks with no activity.