Drupal (PHP) client requirements vs. localization packaging

Yeah, our issue is that we do need to generate these files as static packages on ftp.drupal.org. If we need to have both gzip and non-gzip support, that can be painful in terms of the number of files. We have 5600 releases parsed now, and if assumed 100 languages (only), we'd need to generate 5600*100 = 560.000 files already. If we generate both gzip and non-gzip .po files, that is a million files :)

What we can do is to:

- require zlib (but Joomla did not have that much luck for it) -- we can cross-check with the plugin manager people, they might require zlib as well
- optionally use zlib -- not as suggested above, but as in we automatically download stuff if there is zlib, otherwise we just have manual instructions for users to download
- look whether PHP has better stream support for gzip compressed HTTP streams and let ftp.drupal.org (which we access over HTTP anyway) to gzip on the fly in the web server -- kkaefer believed that PHP might have better support on the stream level even if no standalone zlib support
- generate both versions of the packages as discussed above -- a million files :)
- just generate .po files and optimize the crap out of it (as in leave out location comments, newlines, untranslated strings)

I'll check how much we can save by this ".po optimization".

Was testing this "compression" by leaving out extra stuff thing on the latest Finnish translation:

$msgcat --use-first *.po -o fi-xl.pox
$msgcat --use-first *.po | msgattrib --no-fuzzy --translated -o fi-m.pox
$msgcat --use-first *.po | msgattrib --no-fuzzy --translated --no-location -o fi-s.pox
$msgcat --use-first *.po | msgattrib --no-fuzzy --translated --no-location --no-wrap -o fi-xs.pox

The results are:

716015 - fi-xl.pox
714169 - fi-m.pox
567729 - fi-s.pox
552405 - fi-xs.pox

Not including untranslated stuff and location comments gets us around 20% savings. As seen, we could still save 3% more if we also leave out all the wrapping at column 80. Location comments are only ever used to orient people and file names and line numbers don't help that much anyway.

The actual savings of course are highly dependent on the amount of strings translated. Not including untranslated could be a huge win for teams starting up and projects just partially translated.

To estimate the load on the server, I've counted up the source stings:

$select sum(char_length(value)) from l10n_community_string;
+-------------------------+
| sum(char_length(value)) |
+-------------------------+
|                 8191897 | 
+-------------------------+
1 row in set (0.27 sec)

This is the total length of all source strings. So when translated, if we assume the translations are same length (they are not, but we can use this assumption for now), the data is about 16MB per language. That also assumes single-byte UTF-8 chars, which fit some language teams, but far from all.

Then we need to add up the "Gettext overhead", things like msgid "..." and msgstr "...", which I've estimated to be around 30 per string. Rough estimate, we can probably get it down by not using as much whitespace, etc. But 150.000 (number of strings) * 30 would be 4MB.

So one language (assuming the above) would total up at 20MB .po file data uncompressed (with the above "minimalization" techniques, so we transport as few info as possible). That summed up for 100 languages would be 2000MB =~ 2GB.

Ok, so the current thinking is that 20MB in total per language might not look scary. We'd do best (in terms of low barrier to entry on the PHP setup side in Drupal) with on the fly HTTP gzip. The webserver can still cache the gzipped files, and we could serve them to people who have zlib, but do not send them to people without zlib. This would the be set up on ftp.drupal.org via HTTP.

I'll try to point Narayan and Gerhard on this issue, so we get feedback soon. If on the fly gzip does not fly, we can still generate the packages twice in two formats or just do it uncompressed. It would again be 20MB uncompressed per language with Drupal core or big modules like Ubercart with around 500k per language.

[14:25] GaborHojtsy: JacobSingh: hey can you answer a quick question? how does Plugin Manager get away with possibly no support on the client for unpackaging a plugin?
[14:25] JacobSingh: It doesn't really, but Archive_Tar is (AFAIK) the best solution out there

Looks like we don't have any ultimate wisdom after all. So what about hosting those .po files uncompressed then?

Talked to Narayan. He is saying we should be fine using simple .po files and compressing "on the fly" if the client supports it. But this later thing he needs to check with OSU OSL, our hosting service. For now we will assume uncompressed .po files then.

Well, we assume no gz now, and not sure we should preplan the initial structure for a possible gz option later :) Otherwise I totally agree.

Thanks for all your input, file/packaging structure implemented in #594570: Packaging module for l.d.o (l10n_packager). Marking this fixed. Also given that this is not going to be a Drupal 7 core feature, moving to the server queue, where it was implemented.