Include bartfeenstra/cldr to parse Unicode CLDR number patterns

The patch depends on #1531974: Make currencies Ctools exportables. It adds Currency::format(), which accepts an amount and a format string. The format servers as the output value and the amount is rendered into it, after which some replacements are performed for 'tokens' (currency sign, code, and number) and to clean up unused characters from the format. It was deliberately not tied to the proposed CurrencyLocale system, so it serves as an API function for custom formats as well.

The previous patch still contained a line of debugging code.

Improved comments.

Cleanup, code comment love.

FML. 04 had debugging code...

Lalalaaaa.

+++ b/includes/Currency.inc
@@ -60,4 +60,68 @@ class Currency extends CurrencyBaseAbstract {
+   *     digits left, it will be suffixed by all of them.

Should be "prefixed".

We can do three things to make minor units work better:

1. Formats are tied to specific currencies they should work with.
2. Formats are tied to any currency that has a particular minor unit value: formats get a "minor unit" setting and they can only be used for currencies with the same minor unit.
3. Formats can be used for any currency: for this, we need to know the separation between the parts of formats for major and minor units, either dynamically, or through a format setting (such as a token, or by splitting the format in a format_major and a format_minor). This way we can render each unit away from the separator, starting at the separator (major: from the separator to the left, minor: from the separator to the right). This way any format can be used for any currency.

We might also want to remove every character between redundant hashes and not just periods or commas.

- Commenting issue fixed.
- Adds a SEPARATOR token to formats, which indicates the separation between the major and minor parts. This means formats will now work with every currency, but they should ideally have three replaceable characters (zeroes or hashes) on the right side of the separator to accommodate currencies with three decimals. The most flexible solution would be three hashes, since the format would then work for currencies with zero, one, two, or three decimals.
- Currency::format() now renders the major and minor parts individually, from the separator outward.
- Currency::format() no longer renders digits for which there are no more placeholders.
- Added tests.

See the test's assert messages to get an impression of possible formatting options.

+++ b/includes/Currency.inc
@@ -60,4 +60,135 @@ class Currency extends CurrencyBaseAbstract {
+      $output[$section] = preg_replace('/(\D*)#/', '', $output[$section]);

This should probably done for the entire rendered amount and not just for per major/minor section, because now a major format that ends in a comma or period will keep that comma or period even if there are no decimals to render.

Alan, thanks a lot for the review and the link to the Unicode number patterns documentation!

About the latest patch

I added support for negatives, but it's limited to a minus sign. Bracketed negatives don't really seem to exist. According to the docs there should be an entirely separate format for negatives, brackets or not.

Just on mute point that may not be applicable, is 1000000000 parsed as 1000000,000 or 1,000,000,000 using the format string #,##0?

It will be parsed as 0,000. The reason for that is that I didn't want to assume how groupings worked; I believe I saw some formats that used alternating groups of 2 and 3 digits. The idea was to force formats to have enough placeholders to accommodate the largest possible amount.

About the docs

1. ¤ can be replaced by either the sign or the code. For site administrators to create formats, this would require an extra configuration option to switch between sign and code. However, I've seen custom formats that used both signs and codes, so we might want to keep replacing ¤ by the sign, and keep a custom token for the code.
2. If a single apostrophe needs to be inserted as a literal character, should it be escaped by surrounding it by two other single apostrophes? If this is true, what should happen to single apostrophes that do not have a matching second apostrophe? This sounds overly complex compared to escaping by prefixing.

3. You need to modify the patterns when: The negative pattern doesn't simply add a minus sign.

   Does this mean adding a minus sign for negative amounts is the default behavior? If so, how should the sign be added?

4. Periods and commas are placeholders too. This would remove the need for a special SEPARATOR token, as introduced in the latest patch, but it introduces the requirement for replacement values to be configured.

In short, if we go with this formatting, I suggest we implement the currency general purpose format, and, at least to start with, without support for short numbers. To this, we add a currency code placeholder. However, the standard looks a bit fuzzy and open to implementation.

This is a common requirement in many systems especially in the commerce sector, and is seen in many of the cldr formats. I have lost the scripts I used last year to extract unique currency formats, but from memory I pulled out about 2 or 3 formats from the XML files that used brackets instead of a minus character. I found no other minus markers when parsing the files and no formats that used brackets for anything else, but if you want to handle two format strings as per the docs, no objections from me!

I was just saying in terms of formatting, there is no such concept as bracketed negatives. They're just negatives, and the system couldn't care less how exactly they're formatted.

2) Not sure what you mean with that. Escaping is used in any context that uses characters with a special meaning, so it applies to currency formatting as well, because what if you want to display a literal comma and not use it as a decimal separator placeholder?

3) I'm not sure what you mean here either. Is there a default behavior when there is no format for negative amounts, or isn't there?

4) Right. I understood that and meant that my solution seems more flexible to me. I haven't been able to think of a situation in which the system wants to know what characters are used as separators. The unicode standard needs to know them for replacement, by my approach worked fine without replacements, so I don't see the added value.

//Edit: are you on IRC? It might be easier discussing a few things there than here in the queue :)

Format? Maybe you are talking about something completely different. In regards to formatting, open MS Excel, click a cell, select currency type, and two of the four options are for bracketed negatives. Screenshot.

I know bracketing is a form of negative amount notation, but from the system's point of view, it doesn't matter how a negative is rendered. It should take care of supporting different locales, but not care about how those locales work. Just a bit of miscommunication :)

2) I opted to just go with the CLDR guidelines for consistency's sake.

3) Good, thanks! Would you have a link to a document that says that, so we can put it in our own documentation?

4) Like 2, I went with the CLDR guidelines here as well. This allows us to parse the CLDR XML files and generate format configurations based on the locale data they contains.

Per primary locale, en, es and ar were the most diverse:

ar: ¤ #0.00;¤ #0.00-;¤ #,##0.00;¤ #,##0.00-;¤#0.00
en: ¤#,##0.00;(¤#,##0.00);#,##0.00 ¤;¤ #,##,##0.00;¤ #0.00
es: #,##0.00 ¤;¤#,##0.00;¤-#,##0.00;¤ #,##0.00;¤ -#,##0.00;(¤ #,##0.00)

Why do those locales have more than two patterns? http://cldr.unicode.org/translation/number-patterns doesn't mention this.

The attached patch tries to conform to CLDR as much as possible. (Possible) differences:
- Two custom [XXX] and [999] tokens for the currency code and number.
- Every pattern is required to consist of a positive and negative fragment, separated by a semicolon. There is no default negative pattern yet.
- Every pattern fragment is required to have a period for a decimal separator.
- Every pattern is required to have a 00 decimal placeholder anywhere on the right side of the decimal separator.

Sorry, haven't looked at the code but seems strange to have a minor placeholder "00" for currencies like the Laos Kips where 1,000,000 LAK is only 125 USD.

From http://cldr.unicode.org/translation/number-patterns (first row of the second table):

Note: the number of decimals will be set by the program to whatever is appropriate for the currency, so do not change them; keep exactly 2 decimals.

I interpreted this as keeping two zeroes, which will be replaced by however many decimals the used currency has.

I wonder if the formatter shouldn't be part of the code, something like this maybe:

$formatter = new CurrencyFormatter(Currency $currency);
$formatter->format();

Which outputs the stock standard UNICODE formats, and if someone wants to fancy formats, you could extend the class to supply whatever you want. The above two lines could be wrapped into a factory method that initiates a globally defined formatter in case someone wants something special, falling back to the default CurrencyFormatter if the class isn't found.

I'm thinking about integration with external systems / third party modules, and supplying anything other than a raw formatted code would probably break the integration as these wouldn't expect HTML.

The idea is to let every piece of code do its own thing. This issue is literally just about parsing CLDR number patterns. Locale delegation, which will try to find the correct pattern (either stock Unicode CLDR or custom/overridden) based on available locale parameters, will be added in another issue.
Also, parsing a pattern requires a currency, but the pattern itself does not require a specific currency (e.g. should work with any currency), so passing on a Currency to CurrencyFormat for other reasons than parsing a single pattern only limits CurrencyFormat IMHO.

I just sent you an email through your contact form with my Skype handle.

-$fragments = preg_split("#(?<!');(?!')#", $pattern);
+$fragments = explode(';', $pattern);

This won't work, because it will split on escaped semicolons as well.

I don't think you can use this.

$sign = (int) $amount < 0;

Good catch. Your example isn't quite right, because amounts will always be integers anyway, but the casting is indeed wrong.

This patch adds support for default patterns for negative amounts, and fixes the type casting issue raised in #21. It also updates tests pattern formatting using every currency available in the system that has a minor unit set (and for this, it adds minor units to some currencies).

For some reason I didn't find this during my first search for existing code, but apparently PHP *can* parse CLDR number patterns using its NumberFormatter class. I haven't checked closely yet, but we can at least make it use custom patterns, and I believe we can even set custom replacement characters (meaning PHP will let us override the defaults it finds in the locale data it has).

We will also need to find out how common this extension is. Based on a quick search it seems most operating systems don't ship with ICU (a requirement for the Intl extension), so using the extension might limit Currency to people managing their server OS themselves.

A few recent findings/realizations:

• Hindi uses one 3-digit group. All other groups are indeed 2 digits.
• We probably want to support other digit character sets than Arabic (there are Hindi, Tamil, etc), as long as they have ten characters for 0-9.
• Unlike some other parsers (of which I could only find a few after hours of searching), I'd like to continue support 'unusual' characters for theming purposes, as opposed to
  

  Note: I think we already test patterns, to make sure that they are sane, eg no "###-hi mom-##0".

  (quote from http://site.icu-project.org/design/number-parsing)

• I'm thinking we might want to develop the parser code on Github for increased exposure. There are probably too few people here within the Drupal community with enough knowledge to pull this off, and a good PHP CLDR (currency) number pattern parser is usable for tons of other projects. Using Composer we can easily embed the code in Currency.module, which would deal with all the other Drupal-specific stuff we're planning to add.

I took the liberty of putting this on Github already: https://github.com/bartfeenstra/cldr. It's released under the MIT, so we can easily commit the code to the d.o repo.

The code is a massive rework of the patches posted here. It does not only support currencies, but every number format described by CLDR, except scientific notation (currency amounts, decimals, percentages), and it even does integers. Supporting integers actually makes testing and maintenance easier. It has PHPUnit tests and Composer integration.

The attached patch imports bartfeenstra/cldr using Composer. This ensures that people using Currency don't have to install Composer themselves. To update the code (whenever bartfeenstra/cldr is updated), simply run composer update and commit the changes.

Err, where did /vendor/bartfeenstra/* go?

See #1868952: Update bartfeenstra/cldr.