Verify transliteration data sources and their quality, and potentially eliminate maintenance

As a note to whoever takes this on: Attach any scripts you used to do your testing/comparison to this issue please! We probably will want to be able to run the "take the ICU data out of PECL and put it into a format we can use" later on.

add link to pecl class

I am working on this.

So. I have created a MONSTER!! Well, actually it's a script that will read in all these data sources and compare them.

I ended up not doing the 'intl' PECL extension. The reason is that its transform operations are based on knowing exactly which scripts you want to transliterate, which is fine for some use cases but not this one. I don't want to become a Unicode/language expert and figure out which of their 257 possible transformations would be good to apply, and I am not even sure that these include a complete list.

I also found a couple more data sources:
a) unicode.org has XML-based transliteration data, which seems to be what the 257 transformations in the ICU project are based on, so again not useful unless you want to figure out which specific transforms to use... These can be downloaded at http://cldr.unicode.org/index/downloads if you are interested, and there is also some documentation there.

b) The JUnidecode project, which claims to be based on the Perl Text::Unidecode module (see above) with some additional data. http://www.ippatsuman.com/projects/junidecode/index.html

Anyway... With that said [and I'll update the issue summary], I have done an analysis of all of this data. Attached is the script I used and the CSV output of the differences. There is one differences file that has all differences; the other omits the lines where our data set had a value and one or more of the other data sets was missing a value.

I'm not sure what to do with this information though... It looks like there are some real differences between our data set and the others (like 30,000 of them!), and a few cases where there is data provided by one of the other data sets that is missing from ours. But it is difficult to evaluate whether our data or the other data sets is better.

So I'd be inclined to mark this as "won't fix", after all... thoughts?

This is about all I can do...

Revamped issue summary.

Oh and I just want to note about the script in #5 -- I'm sure it could be made cleaner and more efficient. My goal was to get this done, not make beautiful efficient code. It runs in a few seconds, so I think it's "good enough" for now (plus full of nice documentation and I think correct at least).

RE #8 - What I meant was that for this exercise, I was not able to generate tables to compare to the other data sources from the 'intl' extension with the time I had available for this process.

And #9 is not relevant to this issue, which is about improving (maybe) the character-by-character tables, not about doing transliteration in a totally different way. Please file a different issue to discuss using 'intl' and doing transliteration better if you want to discuss that.

By the way, what is this "CLDR" data you are talking about -- is that unicode.org under a different name or a different data source not listed above? And it looks like that sqlite database is a database of unicode character bitmaps, not transliteration (or maybe it includes both?)... if so, not relevant for this purpose...

I am returning to this issue... and did a few checks on this data in #5 today:

a) I checked over the places where we had case differences, by going to unicode.org and looking at the characters. There were actually only a few, and our data turned out to be correct, while Midgard, CPan, and Node.js had incorrect data:
0x014a,"NG" vs. "ng" (upper case is correct)
0x014b,"ng" vs. "NG" (upper case is correct)
0x1e83,"w" vs. "W" (lower case is correct)

There were a couple of other inconsequential case differences, like whether the TM symbol should be transliterated as (tm) or (TM), but I do not think these matter much if at all. So we do not need to patch our data for this, and the only other data set that had the correct values was JUnidecode.

b) There are several character ranges where other data sets have data and ours has none:

- 0x0a01 - 0x0a7f - Gurmukhi (a South Asian language) - Midgard has data
- 0x0a80 - 0x0aff - Gujarati (South Asian) - Midgard
- 0x0b00 - 7f - Oriya (South Asian) - Midgard
- 0x0b80 - ff - Tamil (South Asian) - Midgard
- 0x20b0-20ba - currency symbols - JUnidecode
- 0x2100-214f - letter-like symbols - JUnidecode
etc.

I think we should probably go ahead and adopt the data from these other sources where we can, and I can make a patch... it seems to me that if Midgard or one of the other projects has some way to transliterate the characters, it would be better to adopt those than to have nothing? I checked on licensing:
- Mediawiki is GPL
- The CPAN module is GPL
- I cannot find a license statement about Midgard or the node.js stuff, so we would need to ask them I guess? Oh, they're by the same person (Henri Bergius, henribergius, http://drupal.org/user/1885584) and he commented on the original Transliteration issue and wants to collaborate. :) I'll send off a query.
- JUnidecode is BSD (is that compatible?).

c) There are also a ton of places where we have differences between our choices and other data sets' choices... Those will be much harder to sort out!

Meanwhile, I updated my script a bit and have generated a new "differences" file (with fewer false positives), so attaching those.

So... Regarding #8 above commenting about using the intl package...

The issue at hand here is whether we can generate a better set of tables for the simple lookup-based transliteration that we are currently doing in Core (and which is similar to what many other projects are doing, see issue summary). Using the intl class to do transliteration better (thinking about context etc.) is a separate issue, beyond the scope of this issue.

So for this issue... What I'm wondering is whether we can mine the intl data somehow to get a better set of "the generic way to transliterate character X to US-ASCII", and also if we can mine it somehow to get a better set of "language-specific overrides of the transliteration of character X to US-ASCII" -- both of these are data sets we need in Core for the PHPTransliteration class(es) that we have.

The problem I had in trying to do that is that intl doesn't have a concept of "generic" transliteration. It has instead a concept of "transform text from one representation to another", including upper to lower case, etc. So what I don't know is which of their 257 supported transformation rules (of which 52 of them are something -> Latin), plus other transformations you can do (such as "separate out accents"), would be appropriate to apply to generate generic and/or language-specific tables for our classes.

What I've done so far is to use these operations:
NFD
Cyrillic-Latin
Greek-Latin
Hangul-Latin
Hiragana-Latin
Katakana-Latin
NFD
[:Nonspacing Mark:] Remove
NFC

Note: NFD means "separate accents from characters". Later on, the non-spacing mark thing removes the accents completely, and then NFC means re-compose everything.

Does anyone have any suggestions about the other 47 rules -- which ones should be applied to get as complete a set of characters as possible? And are there other rules I don't even know about? Attached is a list of the (something) -> Latin rules that I generated from print_r(Transliterator::listIDs()); followed by a grep... I can't really find any documentation on these rules though... any thoughts?

RE #12 - I am inclined to agree on that -- leave the differences alone. The main objective here I think would be to get *something* for as many characters as possible, and if there are errors in the data, and people are using the transliteration classes and find them, then we can patch Core to fix the errors.

RE #15 - that is a subtle distinction. Obviously we want to transliterate to Latin and then from there to ASCII, right? And of course there are language differences (that is why we have language overrides)...

But what I'm getting at is even the question of if we actually knew the language of some particular text (because normally we do, after all, in Drupal), how would we generate the tables for that language in intl? Which of the 52 something-to-Latin rules would be used, and which other transformations would also need to be applied? There doesn't seem to be any documentation for which rules to use for which language... not that I found anyway. Do you know of some?

RE #17 -- That is overwhelming... And the page you referenced doesn't tell me how to figure out which of the transforms in the intl package should be used for each language or each range of characters -- it seems to be more about how these transforms are constructed? Anyway, I guess we could probably go to unicode.org and systematically go through their tables of character codes, figure out what each script is called, and see if there is a corresponding transform in intl that would transform it.

I guess we could also just ask our specific-language experts from localize.drupal.org to provide transliteration tables for their scripts and skip intl. That actually might be a good idea... which would also involve having a list from unicode.org anyway.

I'll work on this a bit...

I finally found this "roadmap" document:
http://www.unicode.org/roadmaps/bmp/
which at least gives some clue as to which characters are in which range of Unicode (as opposed to starting at the Chart pages http://www.unicode.org/charts/ and clicking through each script to figure out the character range).

So, I went through the "roadmap" Unicode table, and picked out transforms from intl that seemed to correspond to the scripts listed there. A new readdata.php script is attached, and a new differences file.

Some notes on differences:

a) There are more case differences. Aside from conventions (our data is mostly taking the convention that if the result is a capital AE we present it as Ae, and other data sets might use AE or Ae depending), it looks like the intl data set is more accurate than ours (at least by looking at the actual Unicode characters for some specific characters). So we should patch our data set to fix the case problems. There are about 20 that should be fixed.

b) The transforms do provide some additional transform data that we don't have, such as Greek/Coptic characters in the 0x03xx range, Latin phonetic extensions in the 0x1dxx range, etc.

c) I also noticed that in some cases where both Midgard and intl provide data we don't have, the intl data is more accurate. For instance, there are some "enclosed alphanumerics" in the 0x24xx range:
http://www.unicode.org/charts/PDF/U2460.pdf
and for example, Midgard says that 0x24a0 should be transliterated as "A", but intl has a more accurate (at least so I would gather from the bitmap chart) "(e)". This makes me suspicious of using the Midgard data, unless we can figure out where it came from.

d) I would also be suspicious of any data for the CJK range of 0x33xx - 0x9fxx, because presumably various Chinese languages, Japanese, and possibly Korean would have different transliterations of the same character. That would seem to be a place where we could generate language-specific transliterations (possibly from intl) and should probably leave the generic data tables blank. The Midgard data has a bunch of results in that range that appear to be Chinese (just guessing) -- I think we shouldn't adopt them.

e) We should also not have anything in our tables from the UTF-16 and "private use" ranges between 0xd8 and 0xf8.

I'll see if I can make a patch for some of this stuff. And here are the updated script and output, if anyone is interested. If you have suggestions for additional intl rules that should be included to get a more complete data set, I'm receptive!

I am reasonably certain that we are not doing anything sane for those characters that are more than 4 bytes at this time.

I'll look into this... and I realized I forgot to assign this issue to myself, but I'm obviously working on it. :)

Regarding characters outside of 4-byte Unicode... It looks like some good test cases would be, for instance, the Gothic alphabet:
http://en.wikipedia.org/wiki/Gothic_alphabet
So I tried adding these characters to the Transliteration test, and I got a database error in the testing system:

General error: 1366 Incorrect string value: '\xF0\x90\x8C\xB0\xF0\x90...' for column 'message' at row 1: INSERT INTO {simpletest} (test_id, test_class, status, message, message_group, function, line, file) VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1, :db_insert_placeholder_2, :db_insert_placeholder_3, :db_insert_placeholder_4, :db_insert_placeholder_5, :db_insert_placeholder_6, :db_insert_placeholder_7); Array ( [:db_insert_placeholder_0] => 2 [:db_insert_placeholder_1] => Drupal\system\Tests\Transliteration\TransliterationTest [:db_insert_placeholder_2] => fail [:db_insert_placeholder_3] => [:db_insert_placeholder_3] =>

Ah, it looks like MySQL does not support characters outside the Basic plane of Unicode:
http://dev.mysql.com/doc/refman/5.0/en/charset-unicode.html

I'll do something else for the test then.

I got the test to work. And it passes!

Which is to say that 5-byte characters are working correctly for transliteration (correctly == returning "unknown character").

Here's a patch for the test for the 5-byte character, which should perhaps be on a different issue, but I'm attaching it here so it doesn't get lost, and it should be committed.

After this it's back to improving the tables for the 4-and-fewer byte characters.

RE #28 - @Damien Tournoud - I am not sure if we can even add rules for 5-byte characters, except I guess language overrides, which I guess we could test. Good idea if that is what you are suggesting. (Is that what you were suggesting?)

OK then, the patch above needs work then. I'll probably have time to make a new patch tomorrow.

Language overrides for multi-byte characters do actually work. I'm moving this to another issue though:
#1858376: Provide tests and documentation for long Unicode characters
and setting this issue back to "active" since we still need patches for the actual transliteration data sources. Damien: thanks very much for bringing this up!

So, back to where I left off at #20.

Regarding our current data set:
- It isn't consistently using capitalization for cases where a single upper-case character is transliterated into two ASCII characters -- sometimes it uses AA and sometimes Aa.
- Sometimes it has NULL in there and sometimes '' to indicate an untransliterated character.
- As I mentioned in #20, the intl data set appears to be more accurate about capitalization and characters in general than our data set where they differ.

So here's a first patch, which substitutes the output of intl where it differs from our data. And the PHP script I used to make the patch (though note that the x00 file came out wrong so I deleted that one from the patch). The patch definitely needs to be checked over...

Dang. Scratch that patch, some of the differences are just lower-case vs. upper-case in the array keys. Let's try this one...

Sorry, that patch has a problem too. Back to the drawing board (tomorrow probably).

One more try.

Ugh. Well this will need some more work:
- Some outputs are printing out with invalid PHP syntax (quote escaping problems)
- Some files without data in some areas are not printing out the rest as ''.

More work to be done: The missing transliterations need to be output as NULL and not ''. The transliteration code treats these differently (if it's '' that gets returned, but if it's NULL it returns the "unknown character", which defaults to "?"). I'm not sure whether or not this distinction is followed well in our current data files or not -- there seems to be a mix of '' and NULL in the files, and I'm not sure whether the '' in there really means "this character should be translated as an empty character" or if it means "we don't know".

OK! I think I have a viable patch this time I think. I reviewed the changes (I suggest applying the patch and using
git diff --color-words --unified=0
to review) and they all look like reasonable substitutions and additions to me. This will at least get us in synch with a lot of the intl data set. See attached script (function read_intl_data()) for details of how the intl data set was generated, and for how the updated data files were generated.

Someone please review this? :)

Unrelated test failure:
Two entities loaded by uid without caring about property translatability. EntityTranslationTest.php Drupal\system\Tests\Entity\EntityTranslationTest->testMultilingualProperties()

#43: 1823454-intldata-43.patch queued for re-testing.

#43: 1823454-intldata-43.patch queued for re-testing.

sun, amateescu, and Damien: You are the co-maintainers of the Transliteration system... Can one of you please review this patch so we can get our data set for Transliteration updated? Thanks!

There's also a patch on
#1858376: Provide tests and documentation for long Unicode characters
that needs a review. That one is simpler -- it adds a test and updates documentation on how to transliterate long Unicode characters.

New issue component!

#43: 1823454-intldata-43.patch queued for re-testing.

This patch still needs a review by the other maintainers of the Transliteration system, or at least someone. Please?

#43: 1823454-intldata-43.patch queued for re-testing.

I'm temporarily assigning this issue to Damien Tournoud... Could you review the patch? It would be good to get this into Core sometime. If you do not wish to review it, maybe you can assign it to one of the other people who volunteered to maintain the transliteration system (amateescu or sun)?

Weird. I never noticed that... or else it got corrupted at some point after I uploaded it... who knows. Anyway, I'll see about making a new patch. I'll have to start over with the script, as it's been 7 months now since I originally posted the patch. :(

OK, here's a new patch and a new script (the script only added one comment from the previous version). The patch appears to be fine at this time but I'll look after I upload it and see what it looks like... that was weird and I'm sure the patch didn't have that problem when I uploaded it in December.

Anyway. Some review notes:

a) I suggest applying the patch and using

git diff --color-words --unified=0

to review the differences.

b) There are notes in the script about how it was generated. A summary:
- The script reads in all the existing Drupal transliteration data from core/lib/Drupal/Component/Transliteration/data to form the "drupal" data set.
- The script uses the PECL "intl" package to generate a (hopefully) canonical set of transliteration data.
- To use "intl", you have to specify a set of transliterations to do. The ones that were applied are listed in the body of the read_intl_data() function in the script.
- All characters of interest are run through this transliteration individually, to create the "intl" data set.
- Then the script goes through character by character and checks for differences. If the "intl" data set has data where the "drupal" set doesn't, or if "intl" has different data from "drupal", the "intl" data is taken; otherwise we keep the "drupal" data. This logic is in the patch_drupal() function.

After using the script to make data files, I just copied them to the core/lib/Drupal/Component/Transliteration/data directory and used git diff to make a patch. Just to prove I'm not insane, I made a screen shot of the beginning of the uploaded patch file after I uploaded it. It definitely doesn't have all those weird characters in it currently.

I'll update the issue summary because it's out of date. Meanwhile, here's the patch and script.

update issue summary with additional data sets and notes

Issue summary is updated!

RE #58 - thanks for finally offering some suggestions. I made up that list of which transformations to do myself and didn't know about (or notice?) 'Any-Latin'. Are there any others I have missed? And if we do Any-Latin, do we need all the others? The documentation for the existing transforms is really not all that illuminating...

I am assuming this should go at the end of the list of foo->Latin transforms, right before the Latin->ASCII transform... I can try it out and see if it makes much of a difference.

OK, here's a patch and interdiff with the Any-Latin transformation added (new script attached also; the only difference is the addition of that Any-Latin line).

It added many more characters to the data set, so on the theory that providing transliterations for characters is better than not providing transliterations, this seems to be an improvement.

Just as a note on #62 - We need to be careful on native speakers submitting patches to the core character-by-character data tables. If the additions are specific to one language (such as conventions for o-with-umlaut becoming just o or oe depending on the language), then they should go into language data files, not character data files. We'll have to watch that as patches are submitted -- we can support both types of data files.

I am not sure about adding the script... Are there other scripts in there that are meant for component maintainers?

Filed #2097587: Follow-up: Add transliteration data maintenance script to scripts directory