Import duplicate check broken [#999092]

Comment	File	Size	Author
#9	d6uk.zip English Ukrainian translation	867.23 KB	podarok
#6	languageicons-6.x-1.2.uk_.po	63 bytes	podarok
	equal_strings_is_suggestion.png	48.93 KB	podarok

Comment #1

Gábor Hojtsy

he/him

Hungarian

Hungary

CreditAttribution: Gábor Hojtsy commented 15 December 2010 at 16:04

Status:

Active

» Postponed (maintainer needs more info)

Well, looks like the diff (shown with the last string) shows that there is some difference with the last char. Also, I tried to reproduce this by searching for "aluminium" in Kazakh (that's the language, right?), and no strings were found. Can you provide reproducible examples?

Log in or register to post comments

Comment #2

podarok

🇺🇦 he/him/his

Ukrainian

Ukraine

CreditAttribution: podarok commented 15 December 2010 at 16:12

2Gábor Hojtsy
It is Ukrainian
here is a link http://localize.drupal.org/translate/languages/uk/translate?project=&sta...

The strings are really equal !

Log in or register to post comments

Comment #3

podarok

🇺🇦 he/him/his

Ukrainian

Ukraine

CreditAttribution: podarok commented 15 December 2010 at 16:15

Status:

Postponed (maintainer needs more info)

» Active

Log in or register to post comments

Comment #4

Gábor Hojtsy

he/him

Hungarian

Hungary

CreditAttribution: Gábor Hojtsy commented 15 December 2010 at 16:24

Title:	Equal strings @is suggestion@ in database l.d.o	» Import duplicate check broken
Project:	Drupal.org site moderators	» Localization server
Version:		» 6.x-2.x-dev
Component:	Localize.drupal.org	» Import

That is interesting. The server should not accept strings which are same as previous ones. If you try to copy (with the pencil+arrow button) a suggestion and try to submit it unedited, you'll get "1 duplicate translation not saved" as feedback. If you copy more, the counter will increase. I tried this with Hungarian and Ukranian on the page you linked and it worked like this. Looks like something is not working the same way on import. Hum.

Log in or register to post comments

Comment #5

Gábor Hojtsy

he/him

Hungarian

Hungary

CreditAttribution: Gábor Hojtsy commented 15 December 2010 at 16:33

Status:

Active

» Postponed (maintainer needs more info)

Looking at this, the normal form submission runs via l10n_community_translate_form_submit(), which uses

$tid = l10n_community_add_suggestion($sid, l10n_community_pack_string($options['value']), $langcode, $user->uid);

Here, l10n_community_pack_string() is just \0 delimited strings, if multiple.

The import uses _l10n_community_import_one_string(), which does this:

$tid = l10n_community_add_suggestion($sid, $value['msgstr'], $langcode, $uid);

Here, $value['msgstr'] is again either a single string, or a \0 delimited list of strings, if plural.

So looks like if we can reproduce this with .po file imports, we should be able to do that with web based editing as well (which I could not). Can you provide me with the recipe to reproduce?

Log in or register to post comments

Comment #6

podarok

🇺🇦 he/him/his

Ukrainian

Ukraine

CreditAttribution: podarok commented 15 December 2010 at 17:32

File	Size
languageicons-6.x-1.2.uk_.po	63 bytes

the main trouble is that I do not know when and how that strings suggested into database...
it was done via po file import possibly

I can try to reproduce it...
I have a duplicated account at l.d.o for testing and have to try to play with po files...

Just tested

For security reasons, your upload has been renamed to languageicons-6.x-1.2.uk_.po.
The translation was successfully imported.
2 duplicate translations not saved

Tested with different options and their combinations

Attribute import to: 
 Yourself (podarok_auto)

Attribute import to: 
 Multiple contributors

and

Store as approved translations 
 Store as suggestions needing approval, possibly discussion

request from server is as usual

For security reasons, your upload has been renamed to languageicons-6.x-1.2.uk_.po.
The translation was successfully imported.
2 duplicate translations not saved

and nothing changed in http://localize.drupal.org/translate/languages/uk/translate?project=&sta...
suggestion was not added(that is good! but we still do not know where is a bug...)

Possibly it is a bug with xml-rpc from l10n_client?
cause as I see - suggested strings can be viewed at queue only after a some time(via cron? job_queue?), not at the same time as they were send from l10n_client

Log in or register to post comments

Comment #7

podarok

🇺🇦 he/him/his

Ukrainian

Ukraine

CreditAttribution: podarok commented 15 December 2010 at 17:33

Status:

Postponed (maintainer needs more info)

» Active

Log in or register to post comments

Comment #8

podarok

🇺🇦 he/him/his

Ukrainian

Ukraine

CreditAttribution: podarok commented 15 December 2010 at 17:36

possibly we have to store into database not only a user and time but a way too

f.e: suggested from (array('po', 'web', 'xmlrpc', 'other'))

for debugging purpose

Log in or register to post comments

Comment #9

podarok

🇺🇦 he/him/his

Ukrainian

Ukraine

CreditAttribution: podarok commented 15 December 2010 at 17:54

File	Size
d6uk.zip English Ukrainian translation	867.23 KB

he he
looks like it can be reproduced

first time when I imports a large po file (attached) server tells me one words

The translation was successfully imported.
514 suggestions added, 13113 duplicate translations not saved, 3142 source strings not found; their translations were ignored

the options

Attribute import to: 
 Yourself (podarok)
Status:  
 Store as suggestions needing approval, possibly discussion

second time for the same file

The translation was successfully imported.
153 suggestions added, 13113 duplicate translations not saved, 3142 source strings not found; their translations were ignored

the options

Attribute import to: 
 Yourself (podarok)
Status:  
 Store as suggestions needing approval, possibly discussion

third time

The translation was successfully imported.
133 suggestions added, 13133 duplicate translations not saved, 3142 source strings not found; their translations were ignored

the options

Attribute import to: 
 Yourself (podarok)
Status:  
 Store as suggestions needing approval, possibly discussion

Wow!!!
There is a bug! after the first time every next time suggested strings have to be zero!!! But system stores random strings?!

My po file was generated from one of my live sites(Drupal 6) via export (/uk/admin/build/translate/export) all the strings

Log in or register to post comments

Comment #10

podarok

🇺🇦 he/him/his

Ukrainian

Ukraine

CreditAttribution: podarok commented 15 December 2010 at 17:56

and this screen after my "games" is changed
http://localize.drupal.org/translate/languages/uk/translate?project=&sta...
8(((
were 3 pages, now 5 pages

Log in or register to post comments

Comment #11

Gábor Hojtsy

he/him

Hungarian

Hungary

CreditAttribution: Gábor Hojtsy commented 15 December 2010 at 18:24

Project:	Localization server	» Drupal.org site moderators
Version:	6.x-2.x-dev	»
Component:	Import	» Localize.drupal.org

Ok, so my new suspect is database replication. My understanding from the above is that as the new records appear in the replicated DB, you get to see less string accepted (although I don't see why the duplicate count stays the same). How fast did you perform these 3 imports?

Log in or register to post comments

Comment #12

podarok

🇺🇦 he/him/his

Ukrainian

Ukraine

CreditAttribution: podarok commented 15 December 2010 at 18:28

How fast did you perform these 3 imports?

around a 10sec-5minutes between each one (copy past text request into this topic and waiting for 2 MB upload (10-30 seconds))

Log in or register to post comments

Comment #13

podarok

🇺🇦 he/him/his

Ukrainian

Ukraine

CreditAttribution: podarok commented 15 December 2010 at 18:50

if there is an issue with replication(or any) we need a code for clearing such duplications from database, cause in this issue my hands just in a seconds already imported into l.d.o database near a 800 duplicated suggestions 8((((

possibly top translation groups have this issue too...

2Gábor Hojtsy
Do I need to create another issue for that?

Log in or register to post comments

Comment #14

Gábor Hojtsy

he/him

Hungarian

Hungary

CreditAttribution: Gábor Hojtsy commented 16 December 2010 at 14:22

Uhm, looked at our database settings, we do not actually use replication. So even more mysterious. Don't know what to think yet then. I'll look at debugging this maybe later this week with much smaller .po files.

Log in or register to post comments

Comment #15

Gábor Hojtsy

he/him

Hungarian

Hungary

CreditAttribution: Gábor Hojtsy commented 22 December 2010 at 19:03

For #8, I'm working on refactoring history handling for strings, which among other things should provide exact information on how the string got into the database (even who submitted it when the attribution was set different to the submitter). I'm working on that at #1005436: Refactor history storage / handling but do not expect immediate results. (That will not solve the issue, but since you mentioned that tangent, let me cross-link to that issue, so you are aware).

Log in or register to post comments

Comment #16

Gábor Hojtsy

he/him

Hungarian

Hungary

CreditAttribution: Gábor Hojtsy commented 11 March 2011 at 18:09

Project:	Drupal.org site moderators	» Localization server
Version:		» 6.x-3.x-dev
Component:	Localize.drupal.org	» Code
Status:	Active	» Postponed (maintainer needs more info)

We have string storage information for quite a while now. Are you still experiencing this issue? I have not heard from *anybody* else that they've seen this.

Log in or register to post comments

Comment #17

Gábor Hojtsy

he/him

Hungarian

Hungary

CreditAttribution: Gábor Hojtsy commented 23 March 2011 at 16:43

Could it be that #991724: force to Unix line endings is the reason for this?

Log in or register to post comments

Comment #18

droplet CreditAttribution: droplet commented 23 March 2011 at 19:06

@Gábor,
it is not the same problem as I reported on #991724. (endings problem only (99.x%) only happened on multiple line strings)

I've seen such problem before, all suggestion are older around / before 2010. maybe buggy on previous version.
if you are remember when we implement #622500: Add hashkey to strings and translations for quicker comparison, you found that some duplicated strings.

Guess it should look into DB to see what happening.

Log in or register to post comments

Comment #19

podarok

🇺🇦 he/him/his

Ukrainian

Ukraine

CreditAttribution: podarok commented 24 March 2011 at 08:24

2Gabor
How can I test it?
I can`t test it on localize.drupal.org - its production

Log in or register to post comments

Comment #20

Gábor Hojtsy

he/him

Hungarian

Hungary

CreditAttribution: Gábor Hojtsy commented 24 March 2011 at 08:43

@podarok: you can join the test team and use that to test. Please do use minimal .po files to try and reproduce. DO NOT use .po files with hundreds of strings, if you can reproduce with a couple of strings.