Missing original strings for some modules (was Missing module release data)

Automatically closed -- issue fixed for 2 weeks with no activity.

I did some more investigation in the watchdog logs. I find regular issues with tar.gz files not being properly available (when downloaded they do not appear to be in the tar.gz format), but then l10n_project nicely recovers from this by removing the file and coming back to it later (when it parses it correctly). When the targz fails, it will not go on to try and save strings for the given release, so it might or might not be related. Opened issue at #681344: l10n_project download checking is bugos. That might also catch partially downloaded files though, which tar possibly still uncompresses.

Obviously we could try to prevent this by adding some "graceful rollback" code:

- after a release is parsed, look at whether we had any string relations stored (simplified version of above query)
- if not, restore the last_parsed date, so l10n_project will automatically come around to parse it again
- try to prevent this happens in loops forever via a counter on the release record maybe and only attempt 3 or 5 times at most maybe

While this could solve the issue, I'd like to find the real cause of the problem, so we are not wasting computing cycles for doing stuff we throw away quickly after that.

The incomplete source strings may be interesting yet: #642252: Missing English originals in ImageCache 6.x-2.0-beta10 or #593310: custom_breadcrumbs 6.x-1.5 parsing error

Or a bad release, which is contains only a css, and the licence.txt: http://drupal.org/node/244161 The last one is no l10n_server bug, but in so much relevant, that sends the planned system into an endless loop.

UPDATE: I did not read through the counter part first properly.

Given that I've deployed #622500: Add hashkey to strings and translations for quicker comparison earlier today and given #681344: l10n_project download checking is bugos, we now have better download checking (which already caught a faulty release), I've reset the above 403 (404 by today) releases, so they will be parsed again on upcoming crons. We have 10 releases parsed per cron (with crons twice per hour), so this backlog will be processed in about 20 hours. I'll look at the watchdog regularly.

These are actually completely normal. All of these are either themes (.mobi, B7, etc) or not actually Drupal conformant code (Knowledgetree integration 5.x-2.0). The list expanded a bit with the second round of 10 items but with only 2 items which also are in the same realm. So looks like we do not have reliably reproducible problems here (yet).

These 5 projects have been unpublished.

For example, FCKeditor - WYSIWYG HTML editor 5.x-2.3-rc2 contains a lot of strings.

codefilter-6.x-1.0 no strings

admin_menu 6.x-3.0-alpha4 - only 17 strings instead of 76.

Token 6.x-1.13 only 42 (correct: 132)

diff-6.x-2.0

print-6.x-1.12 - No strings found. Important, because this is the current release.

Ok, I've reset parsing of the mentioned token, textsize, views, print and memcache releases. I did not find the mentioned diff module release on l.d.o. A later release is present.

There are no strings in the 7.x-1.0-alpha3 release of Insert to export.

Dozens of missing strings in Ubercart 2.3 and hundreds in 2.4. UC 2.4 has around 900 string less than UC 2.2/2.3 at l.d.o.

Wojtha, Thomas: All right, dropped all strings for both modules (don't worry, your translations are kept), and parsing of the packages should run again in at most an hour. Thanks for the report!

Zoltan: Reset jQuery UI 1.4, so it will be reparsed similarly. Did not find a project called "Insert to export". Are you sure that is the right module name? Do you have the short name of the module (the drupal.org URL that is). Thanks!

Ok, reset that release. Thanks for the info.

Reset again. Now will look closer at what happens with it.

Marked #1017802: Releases broken as a duplicate of this issue.

1. In the translations interface I can't find the latest releases for Better Exposed Filters release 6.x-1.0 (only 6.x-1.0 beta1 shows up).
2. Views release 6.x-3.0-alpha 3 seems to be broken. Only 66 strings shows up, when alpha1 have 1 245 strings and alpha2 have 1 350 strings.
3. ImageCache release 6.x-2.0-beta10 is broken. Only 43 strings shows up, should be 131 strings. With the new release of ImageCache 6.x-2.0-beta12 all strings shows up again

no strings for xmlsitemap 6.x-2.0-beta2:
http://localize.drupal.org/translate/languages/de/translate?project=xmls...

I worked on #1063290: Make parsing messages shorter and more informative and more importantly on #1062120: Add count of source strings to releases and packager so we know have per-release counts of source strings in the database. This will be possible to be used to identify broken releases, but I'd be happy to have a general fix for the issue instead of building tools for the reactive more we are in. Anyway there were 503 releases where no source strings were identified. Note that most of those are valid, they are themes and other projects where people did not care to use any translation API. Well, well. Anyway, I set all these releases back to be re-parsed, so we have all these in the queue now for being looked at again. Now with better reporting and source strings stats tracking, we'll hopefully be able to find why this is happening. It frustrates me as well, let me assure you.

All missing strings seems to be parsed quite a while ago and I know there has been a lot of development on the localization server.

1. Has there been any comments on missing strings from "new" releases?
2. With a new parse, has this solved the issue for the affected release?
3. Maybe this issue is not valid any longer with new releases? If so, then a new parse of all projects and releases might solve this (yes, I know this would take a very long time).

Here is some data about one broken and the previous same branch ok release for imagecache and views:

The views one shows that the recorded strings are all over the place, BUT when a file is covered, it is covered 100%. The imagecache one shows similar information, the slight difference in number of strings recorded is IMHO due to changes in the release, not because the file was not parsed 100%. So it looks like a file is either included or it is entirely missing, but its not the first few files which are parsed, all files are recorded, so again it is not about the parser timing out.

I've also looked into whether any line information is stored for the given releases, but no line information is stored either. So looking at the code, this should go wrong somewhere between l10n_drupal_save_file() and l10n_drupal_save_string(). The later would not store line and string information if the file it belongs to is unknown. From this, I've realized we don't use the Drupal APIs properly, so submitted and committed #1064542: Use db_last_insert_id instead of custom code, which might help with this issue.

We should devise some queries to find the affected releases systematically, so we can hit regeneration of them.

hi,

first: thanks very much for all your effort in the project localization server.

i pick the example google_analytics version 6.x-3.2

it was released on 7th of February:
http://drupal.org/node/1053974

it had no strings on 13th of Feb.
see #45

it was parsed also on 17th of February:
with 111 strings
http://localize.drupal.org/translate/projects/google_analytics/releases/...

=> conclusion: yes, this happens also for new releases.

Hi,
Revisioning 6.x-3.12 is the active version for Drupal 6, and there are no strings for it on the localization server.

Admin 6.x-1-.0-beta5 - 1 string is missing.

Mollom 7.x-1.1 - 0 strings

Privatemsg 6.x-1.5 - 0 strings
Quick Tabs 6.x-3.0 - 0 strings
XML sitemap 6.x-2.0-beta3 - 0 strings

tagging-6.x-2.5. One string in missing.

Please, review #1267466: Start over packages without the administer localization server permission for the temporary solution.

Devel 7.x-1.2 (cca 150 instead of 336)

Thanks to Zoltán Balogh we put in a self-service workaround for now, see http://localize.drupal.org/node/4124. I still think it would be best to fix the root cause, since this should not require manual labour, neither in tracking these down, neither rescanning them. It should work right away... Anyway, we should have a manual way to work around this issue for now until we figure out why is this happening... Thanks again Zoltán!

All mentioned releases between #52 and #65 has been updated. In a similar situation please use the Missing strings? link below the releases combo. Its shown only, after you filtered the project and the release. If you don't have enough permission to start over a release, then ask your team admin, or contact me.

maybe this is a hint:
http://localize.drupal.org/translate/projects/better_exposed_filters/rel...
-> 3 releases

but in the user interface
http://localize.drupal.org/translate/languages/de/translate?project=bef&...

only 1 release!

So any ideas?

@Thomas_Zahreddin,
It's different issue, here's the patch:
#1380682: Fix filter to show enabled project only