Export suggestions in .po files

Given #570648: Add mass moderations screen for suggestions (+ user filtering) in place, I'm not sure teams need suggestions imported, since they can clean up their suggestions and import their translations right away.

BTW the reason we did not implement suggestion export originally is that one string can have many suggestions but *valid* .po files can only contain one translation per string. So there is no straightforward way to generate actually editable .po files (which PoEdit and similar tools can handle) of the suggestions. For pure export/import, theoretically, the l10n_server does not care about this invalidity.

Well, we need to attract more contributors. If we don't, people translating a module directly on localize.drupal.org cannot get the resulting .po file until the review team does its job.

I suggest we add a "export suggestion" select box:

* Ignore suggestions
* Export the latest suggestion of non translated strings
* Export the latest suggestion of all strings

A smart default would be "Export the latest suggestion of non translated strings".

The idea is to limit the import/export facility to translation teams and pregenerate packages of strings for the module releases. Otherwise generating .po files on the file on user request sounds like a non-scalable operation. Killes specifically asked us to not let mass-PHP-generated exports happen in protection of the underlying infrastructure.

That said, we can always export those .po files with the latest suggestions for untranslated strings posing as translations but marked #fuzzy. That would let the import software on a Drupal site possibly skip the fuzzy strings. Since the CVS procedure did not have a review cycle and we are adding that here, this would be similar to the previous CVS resulting .po files, which did not even had the "under review" strings marked in a standard way.

Just marked #478664: Need Export suggestions into one large po file from @podarok as a duplicate. Quoting:

I couldn't find possibility for exporting all suggested(unapproved) translation for feature offline reviewing

Editing via online interface not so usable as poedit(or any) for example and will be helpfull to export only suggested with feature import over

The gettext PO format does not allow for what you request. On the l10n_server, one string can have any number of suggestions, so exporting all suggestions would not be possible with gettext PO which only allows one translation per string to be exported (and does not have the notion of suggestions). Do you have any better idea on how to segment the suggestions?

Also, when you import .po files for strings which have existing translations/suggestions, the text you import via the .po file is treated as suggestions, so you'd get even more suggestions after the import is done. Maybe we need to have a refined workflow here.

To be clear, I understand editing and picking the proper suggestions outside the l10n_server might be more convenient for some, but I don't see a way to implement that properly.

In my way i did a small SQL script for getting all the strings from database(csv) and after that reformatted it by hands into po file for importing into cvs.drupal.org
With a small change into script I can get all "suggestions" and put it into csv file
Possibly a good solution will be import-export via csv, not *.po ?

SELECT `value`,`translation`
FROM YOUR_DBNAME.l10n_community_string join YOUR_DBNAME.l10n_community_translation
on YOUR_DBNAME.l10n_community_string.sid = YOUR_DBNAME.l10n_community_translation.sid

Result can be stored as CSV and easy converted to po file that contains all translated and suggested strings.

After importing that file into new location - all translated strings would be marked as duplicates but all suggestions - as suggestions(If You select that option)

Retitled, since importing stuff from .po files is already happening, and strings marked with #fuzzy will always end up as suggestions while others will depend on existing suggestions and translations as well the permission level of the user. Over time I came to think that exporting suggestions would be useful (especially with the new UI patch from #563128: Rework the translation UI moving suggestions more to the same level as translations).

+1 for exporting suggestions.

After translating a number of strings via localize.d.o I'm now unable to use my own suggestions unless I copy paste them individually to a po file :(

Since in many cases there is only one suggestion, couldn't we add something like "Include English originals, translations and first suggestions" as type?

+1 for exporting suggestions too

Sounds like #700766: Implement per project or core/module permissions becomes more important :-). I'm not a fan of exporting suggestions... what should happen if there are 20 for one string!? Which one should win? Also clutters all Drupal installations and we will only result in complains about the "bad translation quality" of unapproved strings... strange request.

--1

I was utterly shocked to discover that the po export didn't include suggestions (for untranslated strings) - marked as fuzzy. To me, this is a rudimentary feature of any translation system. (I'm trying to resist using all caps.)

+1 for exporting suggestions.

PS! I do agree that using export as the general procedure to get a translation for normal users (non-translators) probably isn't scalable.

+1 for exporting suggestions.

It seems that we have the same pitfall on french group :
many good will, but with validation processes very separate in time many members couldn't simply use the localize as it could be (since waiting suggestions are not usable outside localize if we can't export them...).

So, i suggest this kind of more basic exporting capability to continue using .po :

"Exporting my suggestions" : where "my" is the user requesting the export or "Exporting suggestions of @username" (if we've seen a good work even if not approved yet : we could want to use it partially or not)

With a sorting option filter like :
- radio/select :
-me
-username (text)

- radio/select :
- approved suggestions first then my (or username) suggestions (if no approved suggestions found)
- my (or username) suggestions first then approved suggestions (even if approved suggestions found)
- only my (or username) suggestions (without approved suggestions)

- checkbox for :
if the string is empty get the most recent suggestion found (from other members)
(or the suggestions by desc votes : see below)

Pros :
This filter is covering big parts of needs and by the way .po is still usable.
Localize members could contribute and use the work done without having to wait for approval.
Export could be usable as some kind of a basis version uptodate for locale work on translation.

Votes : giving a vote ability on suggestions to the members imo could be great.
We read translations and give our version, if we read something good : it could be cool to mention it for the others (and making a popularity sorting of suggestions by this way and a more easy task for a moderator at validation time).

Not sure if it's possible or totally inconceivable.

Regards,
Denis

For the de - group this would be a nice feature since we have ~10.000 Suggestions awaiting approval.

Checking the suggestions with tools like poedit is much faster than online.

Suggestion for multiple suggestions:
Just add a prefix to the original String like [Suggestion 1] and the mark #fuzzy;

The translator adds a prefix like [approved] to the choosen translation (i'm not sure if all would be able / willing to edit the .po file in a texteditor)

in the Import drop all Strings with [Suggestion xxx] as prefix, only import the [approved] String.

This is compatible to the gettext - format ; maybe other possibilities exist to stuff it into the gettext format in an other way.

Best Thomas Zahreddin

What should happen if there are 20 suggestions for one string!? Which one should be in the exported PO file and how should the export evaluate what suggestion is the best?

As I know you can only have one translation per source string and not 20 in PO files. And then we have only one string - the importing back with auto-approval will decline all other 19 suggestions that may be much better than the one that has been exported.

"in the Import drop all Strings with [Suggestion xxx] as prefix, only import the [approved] String."

this should happen, is this so vague?

This means the suggestions can stay in the translation server as long as nobody deletes them.

I suppose i don't get your message - or didn't you read mine?

Import is OT here... we talk about Export and otherwise who'd like to edit a PO file if he use PoEdit??? Usability nightmare... and some non UTF8 aware editors may clutter UTF8 letters... I've seen this very often in past...

I thought your process describes a review process for the reason of an UI that is not that user-friendly (or slow) as you wish it should be.

1. Export an English string having 20 German suggestions with all suggestions from l.d.o (Impossible with multiple suggestions)
2. Review the suggestions with poEdit (Impossible, you can only have *one* suggestion per English string in PO files as I know)
3. Import the reviewed string back with auto-approval, kill all other 19 suggestions to this one string.

Maybe I misunderstood something, but this is how I understand something impossible with PO files, but requested here as feature request.

maybe we have a different understandig of the word prefix:

you can also say:

Do a string concatenation like:
$out_to_pofile="\"msgid\"."\"[Suggestion_$counter]"."$original_string\"";
$out_to_pofile="\"msgstr\""."\"[Suggestion_$counter]"."$suggestion[$counter]\"";

That's what i mean with a prefix - you get the idea now?

since for each suggestion a unique msgid is created (and removed during imort) this is compatible to the format of gettext.

Please read this and just think about for 30 seconds before you answer.

Instead of coming up with hacks/workarounds on our own end, I'd rather try and use an existing tool in the .po format. Looking up what's available, the "obsolete entry" (http://www.gnu.org/software/gettext/manual/gettext.html#Obsolete-Entries) markers seem to be able to do what we want: include msgid/msgtxt pairs without them being in the flow of the document and disrupting the editor when not needed.

I've tried this sample file with PoEdit, and the suggestions did not interfere with the translation flow. The comments do point out they are there, but unfortunately I did not find a way in PoEdit to make it show the "obsolete strings". According to the Gettext docs quoted above, a respectful PO editor should be able to display / manage these :|

msgid ""
msgstr ""
"Project-Id-Version: Drupal 6.0\n"
"POT-Creation-Date: \n"
"PO-Revision-Date: 2007-12-30 18:18-0000\n"
"Last-Translator: ....\n"
"Language-Team: .....\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=utf-8\n"
"Content-Transfer-Encoding: 8bit\n"
"Plural-Forms: nplurals=2; plural=(n != 1);\n"

# With 2 suggestions.
msgid "First attempt, untranslated"
msgstr ""

#~ msgid "First attempt, untranslated"
#~ msgstr "one suggestion"

#~ msgid "First attempt, untranslated"
#~ msgstr "another suggestion"

# With 1 suggestion.
msgid "Second attempt, translated"
msgstr "This is the translation"

#~ msgid "Second attempt, translated"
#~ msgstr "some suggestion suggestion"

@Thomas / @hass: what are your testing results with files in this format? Let's try to make this work with a format as close to gettext as possible instead of making up our own hacks IMHO.

I also don't like the hacks from thomas and do not plan to edit po files by hand nor break TM features. I have no good idea how to solve this export issue and therefore I'm not working on or with it. The obsolete strings sound like a great idea, but have also never seen them in an ui. If they are context sensitive shown it would be good.

But if 2.x is in production we have the highlighter and this is not beatable... :-)

Gabor thanks for your suggestion,

my tests with poedit, virtaal, gtranslator and OmegaT: they take lines starting with # as what they are meant - comments. So for a regular translator not willing to edit the file in a texteditor these lines / translation are not accessible.

Only poedit show the comments # With 1 suggestion. and # With 2 suggestion. -> to place the alternatives there makes them accessible via the comment field inside poedit.

This was the backgroud for my suggestion to use some prefix for the strings - it's ugly - i know, but i did not found a better solution.

;-) I hope someone in the queue has a better idea, than i.

thanks to hass for his comment.

Here is a simple first step, committing this one in to help the situation a bit. Exports a comment representation of the "has at least one suggestion flag" in this form:

#: mysoftware.po:0
# has suggestion on localization server
msgid "Download translations"
msgstr ""

Marking needs work since this does not yet actually export the suggestions.

Ok, here's an update to actually export suggestions. The logic is as follows:

- you need to check a checkbox to request to get suggestions included

- for translated strings, the suggestions show up as comments above the translation in the .po file (no interference with the translation, the translation memory, etc; but makes it possible to edit and retrieve via standard .po editors)

- for untranslated strings, the latest suggestion (by submission time) is included as a fuzzy translation (if reimported, it does not approve the translation, unless the fuzzy flag is removed beforehand), if any other suggestions are active, see previous point

I've just created a project/release with this sample .po file and added some suggestions/translations.

msgid "My software"
msgstr ""
msgid "Welcome to My software"
msgstr ""
msgid "Download translations"
msgstr ""
msgid "1 downloads"
msgid_plural "@count downloads"
msgstr[0] ""
msgstr[1] ""

When I've exported with the suggestions included, this is what I get:

#: drupo.txt:0
# This suggestion from the localization server:
# Máj szoftver
msgid "My software"
msgstr "Szoftverem"

#: drupo.txt:0
msgid "Welcome to My software"
msgstr ""

#: drupo.txt:0
# Suggestions from the localization server:
# Fordítások beszerzése
# Fordítások letöltése
#, fuzzy
msgid "Download translations"
msgstr "Fordítás letöltés"

#: drupo.txt:0
# Suggestions from the localization server:
# 1 letöltés / @count letöltés
#, fuzzy
msgid "1 downloads"
msgid_plural "@count downloads"
msgstr[0] "1 beszerzés"
msgstr[1] "@count beszerzés"

The first string has an active translation and a suggestion which is then in the comment. The second string has no translations or suggestions. The third one did not have a translation, so the latest suggestion is added as a fuzzy translation and the other suggestions are there as comments (one per line). The last string is similar, only has 2 suggestions, the latest is included as fuzzy translation, the other one in the comments.

Note that since once suggestion is included per line in the comments, if you have multiline strings/translations, you need to figure that out yourself. Such is life I'd say. Unfortunately .po does not have any standard for suggestions as such (this is extensively talked about above), so all we can do is to work around this a bit and include as many info as possible in editable/vieweable portions of the file.

Feedback on the format/behavior welcome.

I just like this structure; Goba has shown us this live on his own machine and it just looks good.

Looks good. I'd only suggest a minor text change.

+++ l10n_community/export.inc	24 Jun 2010 14:25:49 -0000
@@ -305,12 +308,42 @@ function _l10n_community_export_string_f
+        $output .= count($export_string['suggestions']) > 1 ? "# Suggestions from the localization server:\n" : "# This suggestion from the localization server:\n";

Change the singular case to: This IS A suggestion from the localization server

@helmo: Well, the "This suggestion from the localization server" wording can show up in two cases: if there was a translation and we print a single suggestion above that in a comment and when there was only one suggestion and no translation and we print the suggestion in the string (in this case, we also mark it with "#, fuzzy", the gettext standard for non-final translations). So "This IS A suggestion from the localization server" can be misleading in the first case, since then the string is a translation but we also have a suggestion additionally. We can break these two cases apart and have different wording. What do people think?

What about "This suggestion comes from the localization server"?

so the latest suggestion is added as a fuzzy translation and the other suggestions are there as comments (one per line).

if we can combine with #722758: Mark contributors language level would be more better.
and add a box let them choose which users suggestion they preferred.

- adding a START TAG and END TAG to separate suggestion. more easy to build a modules to parse the strings
(#, fuzzy may not always there when it unflag.)

- adding username before suggestion

#: drupo.txt:0
# Suggestions from the localization server:
# [USER-Droplet-NAME] 1 letöltés / @count letöltés
# END Suggestions from the localization server
#, fuzzy
msgid "1 downloads"
msgid_plural "@count downloads"
msgstr[0] "1 beszerzés"
msgstr[1] "@count beszerzés"

Changed wording to:

        $output .= count($export_string['suggestions']) > 1 ? "# Suggestions on the localization server:\n" : "# Suggestion on the localization server:\n";

Committing this and hoping to get more feedback and actual use cases from people using it live.

"All releases merged" exporting same suggestions into fuzzy. Drupal has 10 more version that export 10 times same suggestions.
should we avoid it ?

Ok, can you give an example?

Export Hungarian translations
- "All releases merged"
- All in one file
- Include suggestions (as fuzzy if untranslated, in comment if translated)

one of example from exported .po

Preview before submitting

Suggestions on the localization server:
Előnézet beküldés előtt
Előnézet beküldés előtt
Előnézet beküldés előtt
Előnézet beküldés előtt
Előnézet beküldés előtt

Right, looks like we have other problems there too. If we see a suggestion, additional location info is not collected. However because of the inclusion of suggestions, location information can also appear multiple times. So we need to ensure unique collection of location information and suggestions. The speediest way to do this seems to be to use array keys versus values (instead of array_unique(array_merge())-ing). The attached patch also simplifies and centralizes some repeated code. Can you take a look and help test? Thanks!

tested. works well.

#33: export-duplication.patch queued for re-testing.

Run testing in my own environment and it ran fine with the patch. Committed. Thanks for the review/test!

Also deployed to localize.drupal.org. Hope all is well :)

well - is an understatement ;-)

it's realy a big helper and time-saver!

many thanks for the good work!