Error 503 when uploading da.po

Can you please try to break the file in two and try to import it that way? 1.2Mb is kind of a biggie to import. You get this 503 error, when the underlying request times out. It is not necessarily due to the file not being imported. So first it would be best to check whether the file was indeed imported anyway. That would help you skip moderating out the multiple imports later :)

CAn you attach one of the files here?

Which browser do you use? Can you try another one?

Ok, apparently the problem is that the upload is interrupted server-side when uploading this file.

Also, no string was uploaded when trying this. I suspect there was an out-of-memory error in php.

The import is using the (backported) Drupal 7 import code (to support contexts for Drupal 7 translations). That is not building any growing data structure of any kind (that I remember of) while reading the .po file, since all found strings are immediately saved into the database.

hm, I switched on the error logging, but that doesn't help. It also seems that I was wrong the first time, there are now quite a few translations in there...

Narayan made some config changes again, I am not sure they help for everybody.

The real problem is that the import takes so long: Narayan's upload of the da.po file took over 2 hours to complete.

This is probably because it does a lot of inserts into the SQL tables.

I think what we need is a queuing system which accepts the uploaded files and processes them in order. So the user feedback would be fast.

We should also make sure that we do not accept identical suggestions for strings.

So the diagnosis is that when only one import is running, it is relatively fast, but when multiples are running, they need to wait on each other, INSERT by INSERT, so both PHP processes are bogged down? So we should line them up instead in a queue?

If we do this, we need to somehow figure out how we provide feedback. If a .po file was malformed or it only contained (mostly) duplicate strings, then the user should know. Should we send out an email report of the import once done?

I am not sure if it really matters if there are several imports at once. They create more load of course. I think this is why Narayan suggested to have only one at a time.

Oh, and a email report would be nice, yes.

I guess the header of the file could be still checked during the upload.

Moving to infra issue.

We should also make sure that we do not accept identical suggestions for strings.

Ok, just deployed a patch for this. Now strings submitted via the web UI, imported from .po files or submitted via the remote interface are first trimmed, then applied the leading/trailing whitespace from the source string, then checked for duplicity. Only if they contain anything but whitespace AND do not equal any of the active translations or suggestions for the same source string do we save the translation/suggestion.

The .po import issue still needs to be solved. I've noticed that even entering single translations takes quite some time. We might want to revisit the indexes on the l10n_community_translations table and remove any that we don't need, which would speed up inserts considerably.

The .po import issue still needs to be solved. I've noticed that even entering single translations takes quite some time. We might want to revisit the indexes on the l10n_community_translations table and remove any that we don't need, which would speed up inserts considerably.

To clarify this, we have three types of input: UI translation, imports and remote submissions via l10n_client. These can happen concurrently. Even if we solve the concurrency of multiple imports at the same time, we still need to deal with UI imports and remote submissions concurring with the imports. UI submissions are small (at most 30 strings at once), and remote submissions are tiny (at most 1 submission at once :), but given all the languages, this can give us quite a write concurrency issue.

For queueing, I suggest the small but useful job_queue module. Authored by drumm.

The .po imports were sped up significantly by code level changes, and we have not seen such issues in the past few months, so I'm closing this issue.

Moving to new queue (sorry for issue-bombing).