Missing original strings for some modules (was Missing module release data)

Simply reparsing this module made the string available. Not sure what happens, but we do experience intermittent issues with modules not fully parsed / stored. Not sure whether it is an issue with the parsing or the database storage, but given that I cannot reproduce this by just setting the last_parsed time to 0 and running cron again (therefore reusing the same stack again to redo the task), it is probably a temporary issue. A similar issue was reported at #593310: custom_breadcrumbs 6.x-1.5 parsing error. Although I'm setting this to fixed, I'll try to be alert and look for this happening again (not that I can tell whether a module was completely parsed, when our watchdog says the cron runs completed fine).

There are other issues open for this problem, such as

#680700: There are no strings in the 5.x-1.2 release of Moleskine to export.,
#673122: There are no strings in the 6.x-1.3 release of Translation template extractor to export.,
#672562: There are no strings in the 6.x-1.0-beta7 release of Integrated Metatags to export.,
#662448: There are no strings in the release of Video Filter to export,
#662468: There are no strings in the 5.x-1.3 release of getID3() to export.,
#652620: There are no strings in the 6.x-1.1 release of Browscap to export.,
#627856: Backup and Migrate: No strings to export,
#649290: There are no strings in the 5.x-1.2 release of Devel to export..

And another one #642252: Missing English originals in ImageCache 6.x-2.0-beta10

I did some more investigation in the watchdog logs. I find regular issues with tar.gz files not being properly available (when downloaded they do not appear to be in the tar.gz format), but then l10n_project nicely recovers from this by removing the file and coming back to it later (when it parses it correctly). When the targz fails, it will not go on to try and save strings for the given release, so it might or might not be related. Opened issue at #681344: l10n_project download checking is bugos. That might also catch partially downloaded files though, which tar possibly still uncompresses.

Ha, here is a complete list of releases which are in the database but have no strings associated with them. Pretty impressive. Doh. Columns are project+version, release date and parse date. Looks like the date of parsing has no relation to no-string releases (and we are dealing with an extensive problem here, so good idea not to submit an issue for each of these :).

Code to produce this table was:

  $releases = db_query('SELECT p.title as name, r.title, r.file_date, r.last_parsed FROM l10n_community_project p INNER JOIN l10n_community_release r ON p.pid = r.pid LEFT JOIN l10n_community_line l ON r.rid = l.rid WHERE l.rid IS NULL ORDER BY name');
  while ($row = db_fetch_object($releases)) {
    printf("%-60s | %14s | %14s\n", $row->name .' '. $row->title, format_date($row->file_date), format_date($row->last_parsed));
  }

The key is "l.rid IS NULL", so we know that the given release has no line-string association related to it.

Obviously we could try to prevent this by adding some "graceful rollback" code:

- after a release is parsed, look at whether we had any string relations stored (simplified version of above query)
- if not, restore the last_parsed date, so l10n_project will automatically come around to parse it again
- try to prevent this happens in loops forever via a counter on the release record maybe and only attempt 3 or 5 times at most maybe

While this could solve the issue, I'd like to find the real cause of the problem, so we are not wasting computing cycles for doing stuff we throw away quickly after that.

BTW for those interested, the above TXT contains 403 releases listed, that means about 4.5% of all releases on l.d.o are broken (assuming these releases should contain at least one translatable string, which might not be the case for all of them, but most probably for almost all of them).

The incomplete source strings may be interesting yet: #642252: Missing English originals in ImageCache 6.x-2.0-beta10 or #593310: custom_breadcrumbs 6.x-1.5 parsing error

Or a bad release, which is contains only a css, and the licence.txt: http://drupal.org/node/244161 The last one is no l10n_server bug, but in so much relevant, that sends the planned system into an endless loop.

UPDATE: I did not read through the counter part first properly.

There is a patch at #622500: Add hashkey to strings and translations for quicker comparison which can improve the speed of new release parsing (and .po imports!) by 15 times(!) in my local testing. If we are dealing with a timeout issue here, that can be a savior :)

(I've already committed and deployed #681344: l10n_project download checking is bugos).

Given that I've deployed #622500: Add hashkey to strings and translations for quicker comparison earlier today and given #681344: l10n_project download checking is bugos, we now have better download checking (which already caught a faulty release), I've reset the above 403 (404 by today) releases, so they will be parsed again on upcoming crons. We have 10 releases parsed per cron (with crons twice per hour), so this backlog will be processed in about 20 hours. I'll look at the watchdog regularly.

This is not yet our silver bullet. Of the first 10 releases parsed, 6 got back to the same broken state where they have last_parsed != 0 but no strings:

+---------------------------+---------+------------+-------------+
| name                      | title   | file_date  | last_parsed |
+---------------------------+---------+------------+-------------+
| .mobi                     | 5.x-1.1 | 1176784507 |  1263303006 | 
| B7                        | 5.x-1.2 | 1170266408 |  1263303006 | 
| Fancy                     | 5.x-1.0 | 1168989602 |  1263303006 | 
| itheme                    | 5.x-1.5 | 1181274908 |  1263303008 | 
| KnowledgeTree integration | 5.x-2.0 | 1194633014 |  1263303009 | 
| Rezina                    | 5.x-1.2 | 1176828010 |  1263303007 | 
+---------------------------+---------+------------+-------------+

At least 4 of them passed the parsing fine. I keep monitoring the situation.

These are actually completely normal. All of these are either themes (.mobi, B7, etc) or not actually Drupal conformant code (Knowledgetree integration 5.x-2.0). The list expanded a bit with the second round of 10 items but with only 2 items which also are in the same realm. So looks like we do not have reliably reproducible problems here (yet).

After all releases are now parsed again, we have 82 not having any strings but parsed:

+------------------------------------------+------------------+------------+-------------+-------+
| name                                     | title            | file_date  | last_parsed | rid   |
+------------------------------------------+------------------+------------+-------------+-------+
| .mobi                                    | 5.x-1.1          | 1176784507 |  1263303006 |  2066 | 
| Abessive                                 | 5.x-1.3          | 1234798505 |  1263335408 |  5984 | 
| Absynthe                                 | 6.x-1.0          | 1237368903 |  1263335409 |  9130 | 
| Activity Map                             | 6.x-1.2          | 1250194532 |  1263340208 | 10688 | 
| Area                                     | 6.x-0.3          | 1249048212 |  1263339011 |  7762 | 
| Austere                                  | 5.x-2.0          | 1219074602 |  1263315008 |  5410 | 
| Austere                                  | 5.x-3.0          | 1219089303 |  1263315008 |  5412 | 
| B7                                       | 5.x-1.2          | 1170266408 |  1263303006 |   888 | 
| Black Mamba                              | 6.x-1.3          | 1253346309 |  1263351018 | 12272 | 
| Calendar Systems                         | 5.x-0.2          | 1250174732 |  1263340208 | 10650 | 
| Calendar Systems                         | 5.x-1.0-alpha1   | 1251467119 |  1263342611 | 11298 | 
| Certificate Login                        | 5.x-3.0          | 1220900104 |  1263317408 |  6046 | 
| Certificate Login                        | 6.x-2.1          | 1230572407 |  1263328207 |  6056 | 
| CHAT Users and Design Integrated (BoWoB) | 6.x-8.1          | 1251823814 |  1263343809 | 11490 | 
| CHAT Users and Design Integrated (BoWoB) | 5.x-8.2          | 1251916812 |  1263343810 | 11554 | 
| Clean A                                  | 6.x-1.0-beta2    | 1254751275 |  1263357019 | 12946 | 
| Cleanfolio                               | 6.x-2.0          | 1260888318 |  1263372612 | 16482 | 
| Colorfulness                             | 5.x-3.4          | 1217343305 |  1263313810 |  7620 | 
| Dance Template                           | 6.x-1.0          | 1256913931 |  1263364214 | 14118 | 
| Database Scripts                         | 5.x-1.1          | 1235341513 |  1263335408 |  6132 | 
| Densite                                  | 6.x-1.5          | 1245954919 |  1263339010 |  9902 | 
| DRAN - Drupal Analysis Tool              | 5.x-1.0-beta3    | 1215651612 |  1263311409 |  4890 | 
| Drush                                    | 5.x-2.0-alpha2   | 1239851136 |  1263336608 |  2476 | 
| Drush                                    | All-Versions-2.0 | 1244516117 |  1263336609 |  2478 | 
| Drush                                    | 7.x-2.0-alpha2   | 1239850899 |  1263336608 |  2480 | 
| Drush                                    | All-Versions-2.1 | 1256568650 |  1263361821 | 13934 | 
| Drush Make                               | 6.x-1.0-beta1    | 1252216846 |  1263347413 | 11701 | 
| Drush Make                               | 6.x-1.0-beta2    | 1254178841 |  1263354613 | 12678 | 
| Drush Make                               | 6.x-2.0-beta1    | 1256691654 |  1263361828 | 13998 | 
| Drush Make                               | 6.x-2.0-beta2    | 1257571820 |  1263364217 | 14540 | 
| Drush Make                               | 6.x-2.0-beta3    | 1259567121 |  1263369013 | 15642 | 
| Drush Make                               | 6.x-2.0-beta4    | 1259632838 |  1263369014 | 15692 | 
| Drush Make                               | 6.x-2.0-beta5    | 1259638519 |  1263371412 | 15702 | 
| Drush User                               | 6.x-1.0-beta1    | 1260831321 |  1263371425 | 16468 | 
| Drush User                               | 6.x-1.0-beta2    | 1260893118 |  1263372612 | 16490 | 
| Drush User                               | 6.x-1.0-beta3    | 1261089024 |  1263372616 | 16640 | 
| Drush User                               | 6.x-1.0-beta4    | 1261435818 |  1263372616 | 16774 | 
| Embeddable Blank Theme                   | 5.x-1.0          | 1220937004 |  1263317408 |  6064 | 
| Fancy                                    | 5.x-1.0          | 1168989602 |  1263303006 |  1190 | 
| Fast File Transfer with X-send file      | 5.x-1.1          | 1253719356 |  1263353415 | 12442 | 
| FCKeditor - WYSIWYG HTML editor          | 5.x-2.3-rc2      | 1251981624 |  1263347411 | 11588 | 
| Features Clone (for Drush)               | 6.x-1.0-rc1      | 1262738407 |  1263375017 | 17300 | 
| Google Translate Links                   | 5.x-1.7          | 1254229529 |  1263354615 | 12696 | 
| Google Translate Links                   | 6.x-1.5          | 1254229530 |  1263354615 | 12698 | 
| Group Admin                              | 6.x-1.1          | 1253717162 |  1263353415 | 12432 | 
| HostSlave                                | 5.x-0.1          | 1233882624 |  1263333009 |   628 | 
| HostSlave                                | 6.x-0.1          | 1233882624 |  1263335407 |   630 | 
| HWCTravel                                | 5.x-1.1          | 1231065023 |  1263328208 |  6138 | 
| id-Facta                                 | 6.x-1.1          | 1238420445 |  1263335409 |  9224 | 
| Imageset                                 | 6.x-1.5          | 1254183633 |  1263354614 | 12684 | 
| Industrial                               | 5.x-1.0          | 1219533613 |  1263315008 |  5924 | 
| itheme                                   | 5.x-1.5          | 1181274908 |  1263303008 |  2396 | 
| KnowledgeTree integration                | 5.x-2.0          | 1194633014 |  1263303009 |  7496 | 
| Magwood                                  | 5.x-1.0          | 1250004068 |  1263340207 | 10538 | 
| Markdown Editor for BUEditor             | 5.x-1.0-beta3    | 1233479125 |  1263333009 |  6558 | 
| Mobile by Mobify                         | 5.x-1.5          | 1256072748 |  1263360612 | 13646 | 
| Mobile by Mobify                         | 6.x-1.10         | 1256073941 |  1263360612 | 13658 | 
| Mobile by Mobify                         | 6.x-1.11         | 1256075174 |  1263360613 | 13666 | 
| Mobile by Mobify                         | 6.x-1.12         | 1256077862 |  1263360613 | 13670 | 
| PageEar                                  | 5.x-2.0-rc5      | 1252075562 |  1263347413 | 11636 | 
| PageEar                                  | 5.x-2.1-rc9      | 1258024268 |  1263365421 | 14788 | 
| PageEar                                  | 5.x-2.1-rc10     | 1258130791 |  1263365422 | 14840 | 
| Painted Wall                             | 7.x-1.0-beta2    | 1255836643 |  1263360612 | 13488 | 
| PHPUnit                                  | 5.x-0.1-alpha1   | 1235719529 |  1263335409 |   644 | 
| PHPUnit                                  | 5.x-1.0          | 1243409736 |  1263336609 |   646 | 
| Publish Content                          | 5.x-1.0          | 1214256019 |  1263311408 |  5488 | 
| Purple_Beauty                            | 5.x-1.2          | 1201881009 |  1263304207 |  2726 | 
| Purple_Beauty                            | 6.x-1.0          | 1249697748 |  1263340207 | 10380 | 
| Quran                                    | 6.x-3.0          | 1221312932 |  1263318607 |  4886 | 
| Recipe                                   | 6.x-1.0-beta1    | 1208722811 |  1263307808 |  4856 | 
| Rezina                                   | 5.x-1.2          | 1176828010 |  1263303007 |  2068 | 
| simplehtmldom API                        | 5.x-1.11         | 1251515473 |  1263343808 | 11342 | 
| Skyroots                                 | 6.x-1.2          | 1204709110 |  1263306606 |  7838 | 
| Spaces 3 demo                            | 6.x-1.0-alpha1   | 1261526762 |  1263375016 | 16808 | 
| Test client setup                        | 6.x-2.1          | 1247167939 |  1263339010 | 10308 | 
| Ubercart Product Power Tools             | 6.x-1.8          | 1253805969 |  1263353420 | 12520 | 
| Ubuntu Drupal                            | 6.x-5.0          | 1260035770 |  1263371422 | 16066 | 
| User Import                              | 5.x-2.10         | 1253448373 |  1263351021 | 12314 | 
| Video chat                               | 5.x-1.0          | 1249344975 |  1263340207 |  7408 | 
| Wiki installation profile                | 5.x-2.0-beta1    | 1200625504 |  1263304207 |  7516 | 
| Zen nineSixty (960 Grid system)          | 6.x-2.1          | 1252080135 |  1263347413 | 11638 | 
| Zen nineSixty (960 Grid system)          | 6.x-3.0          | 1252084265 |  1263347413 | 11644 | 
+------------------------------------------+------------------+------------+-------------+-------+
82 rows in set (0.00 sec)

As said yesterday, it looks like these are mostly themes not using t(), stub projects and/or projects which do not have Drupal conformant code.

There also seems to be a "constant backlog" of these 5 projects which does not seem to be possible to download or are not handled due to some other error (I'm not seeing any issues in cron about these and newly coming releases are parsed nicely):

+--------------+---------+------------+-------------+-------+
| name         | title   | file_date  | last_parsed | rid   |
+--------------+---------+------------+-------------+-------+
| A Blog Theme | 5.x-1.1 | 1250824804 |           0 | 10990 | 
| Darsch       | 6.x-1.1 | 1225878006 |           0 |  8424 | 
| Mystique     | 6.x-1.1 | 1255497338 |           0 | 13302 | 
| Pixel        | 5.x-1.2 | 1255148486 |           0 | 13138 | 
| Pixel        | 5.x-1.1 | 1255148487 |           0 | 13140 | 
+--------------+---------+------------+-------------+-------+
5 rows in set (0.06 sec)

So unless there are some suspicious projects in the above listings, I'd consider this "mostly fixed for now". I'll keep monitoring what happens with these 5 unparsed projects. Let me know if any of the 82 above should have strings still, in which case we identified a recurring bug :)

These 5 projects have been unpublished.

So given that parent projects for the 5 releases were unpublished, our release downloader will never attempt to download these. Unfortunately our release backlog counter still counts these as being in the queue, so I'm modifying its query to only count releases for projects which are published. Committing this one and deploying on l.d.o.

For example, FCKeditor - WYSIWYG HTML editor 5.x-2.3-rc2 contains a lot of strings.

Hm, that is not entirely good news (not that I'm surprised). Will take a stab at trying to kick these into rescanning again, and see what comes out of that.

wysiwyg 6.x-2.1 - one string

admin_menu 6.x-3.0-alpha4 - only 17 strings instead of 76.

I've kicked off a once-more-run for the no-string releases once again. Currently we had 242 releases parsed but without releases. We verified before that 82 were fine to stay the same, so I assume they'll get parsed and seen again to not contain text.

Also marked wysiwyg-6.x-2.1.tar.gz and admin_menu-6.x-3.0-alpha4.tar.gz for regeneration individually.

Token 6.x-1.13 only 42 (correct: 132)

textsize-6.x-1.8

diff-6.x-2.0

Too many strings! The views-6.x-2.11 contains 1284 strings in the package, but on l.d.o this release contains 1459. I checked some untranslated strings, but they are really not part of this release.

print-6.x-1.12 - No strings found. Important, because this is the current release.

Memcache API and Integration 6.x-1.6 - no strings

Ok, I've reset parsing of the mentioned token, textsize, views, print and memcache releases. I did not find the mentioned diff module release on l.d.o. A later release is present.

There are no strings in the 6.x-1.4 release of jQuery UI to export.

There are no strings in the 7.x-1.0-alpha3 release of Insert to export.

changed the issue title.

Dozens of missing strings in Ubercart 2.3 and hundreds in 2.4. UC 2.4 has around 900 string less than UC 2.2/2.3 at l.d.o.

for Ubercart 6.x-2.4 ~ 1700 strings are missing …

Wojtha, Thomas: All right, dropped all strings for both modules (don't worry, your translations are kept), and parsing of the packages should run again in at most an hour. Thanks for the report!

Zoltan: Reset jQuery UI 1.4, so it will be reparsed similarly. Did not find a project called "Insert to export". Are you sure that is the right module name? Do you have the short name of the module (the drupal.org URL that is). Thanks!

Ok, reset that release. Thanks for the info.

@Gábor: UC 2.4 has now only 2 strings at l.d.o. Re-parsing at friday probably didn't processed correctly :-(

http://localize.drupal.org/translate/languages/cs/translate?project=uber...

Reset again. Now will look closer at what happens with it.

@Gábor: UC 2.4 is finally 100% parsed, Thomas_Zahreddin was right - 1700 strings were missing. Do you know what's happening? Why are sometimes some releases of some modules parsed uncompletely?

Marked #1017802: Releases broken as a duplicate of this issue.

1. In the translations interface I can't find the latest releases for Better Exposed Filters release 6.x-1.0 (only 6.x-1.0 beta1 shows up).
2. Views release 6.x-3.0-alpha 3 seems to be broken. Only 66 strings shows up, when alpha1 have 1 245 strings and alpha2 have 1 350 strings.
3. ImageCache release 6.x-2.0-beta10 is broken. Only 43 strings shows up, should be 131 strings. With the new release of ImageCache 6.x-2.0-beta12 all strings shows up again

The Memcache API and Integration 7.x-1.0-beta2 and the 7.x-1.0-beta3 contains only 1 string, instead of 19.

no strings for xmlsitemap 6.x-2.0-beta2:
http://localize.drupal.org/translate/languages/de/translate?project=xmls...

all strings for Google Analytics 6.x-3.2 are missing:
http://localize.drupal.org/translate/languages/de/translate?project=goog...

version 6.x-3.1 has around 160 strings

I worked on #1063290: Make parsing messages shorter and more informative and more importantly on #1062120: Add count of source strings to releases and packager so we know have per-release counts of source strings in the database. This will be possible to be used to identify broken releases, but I'd be happy to have a general fix for the issue instead of building tools for the reactive more we are in. Anyway there were 503 releases where no source strings were identified. Note that most of those are valid, they are themes and other projects where people did not care to use any translation API. Well, well. Anyway, I set all these releases back to be re-parsed, so we have all these in the queue now for being looked at again. Now with better reporting and source strings stats tracking, we'll hopefully be able to find why this is happening. It frustrates me as well, let me assure you.

Ok, I've also built entirely new release summary pages for localize.drupal.org in #1063550: Add better release information for project inspection which should help us a LOT in identifying and tracking down these issues. Let's take a look at the affected modules/releases:

http://localize.drupal.org/translate/projects/views/releases
http://localize.drupal.org/translate/projects/memcache/releases
http://localize.drupal.org/translate/projects/imagecache/releases

What's stunning and telling us a lot is that (a) all the same files are parsed and (b) source code warnings are identified in them, its just that the strings are not saved. Its pretty interesting. So it does not look like a timeout issue or somesuch, because it actually looks like the files are parsed entirely (at least telling from the same warnings founds for them). The files themselves are recorded the first thing when a file is started to be parsed, and and the releases we find lacking strings the file records themselves are actually stored entirely. Any ideas?

All missing strings seems to be parsed quite a while ago and I know there has been a lot of development on the localization server.

1. Has there been any comments on missing strings from "new" releases?
2. With a new parse, has this solved the issue for the affected release?
3. Maybe this issue is not valid any longer with new releases? If so, then a new parse of all projects and releases might solve this (yes, I know this would take a very long time).

Well, the memcache project for example, had a badly parsed release in 2010 Sept and 2010 Oct as well. Not exactly too far away. I don't think the root case is solved for this bug yet.

Here is some data about one broken and the previous same branch ok release for imagecache and views:

mysql> select f.fid,f.location, count(*) from l10n_server_file f inner join l10n_server_line l on f.fid = l.fid inner join l10n_server_string s on s.sid = l.sid where f.rid = 10920 group by f.location;
+-------+------------------------------------+----------+
| fid   | location                           | count(*) |
+-------+------------------------------------+----------+
| 78580 | imagecache/imagecache_ui.pages.inc |       60 | 
+-------+------------------------------------+----------+
1 row in set (0.01 sec)

mysql> select f.fid,f.location, count(*) from l10n_server_file f inner join l10n_server_line l on f.fid = l.fid inner join l10n_server_string s on s.sid = l.sid where f.rid = 1284 group by f.location;
+-------+------------------------------------+----------+
| fid   | location                           | count(*) |
+-------+------------------------------------+----------+
| 44344 | imagecache/imagecache.info         |        2 | 
| 44340 | imagecache/imagecache.install      |       20 | 
| 44336 | imagecache/imagecache.module       |       29 | 
| 44332 | imagecache/imagecache_actions.inc  |       62 | 
| 44346 | imagecache/imagecache_ui.info      |        3 | 
| 44338 | imagecache/imagecache_ui.module    |        6 | 
| 44334 | imagecache/imagecache_ui.pages.inc |       64 | 
+-------+------------------------------------+----------+
7 rows in set (0.00 sec)

mysql> select f.fid,f.location, count(*) from l10n_server_file f inner join l10n_server_line l on f.fid = l.fid inner join l10n_server_string s on s.sid = l.sid where f.rid = 26478 group by f.location;
+--------+---------------------------------------------------------------+----------+
| fid    | location                                                      | count(*) |
+--------+---------------------------------------------------------------+----------+
| 379378 | views/handlers/views_handler_area.inc                         |        6 | 
| 380203 | views/modules/taxonomy/views_handler_field_term_link_edit.inc |        2 | 
| 380413 | views/plugins/views_plugin_argument_default_fixed.inc         |        1 | 
| 380483 | views/plugins/views_plugin_exposed_form.inc                   |       15 | 
| 380488 | views/plugins/views_plugin_exposed_form_basic.inc             |        1 | 
| 380493 | views/plugins/views_plugin_exposed_form_input_required.inc    |        4 | 
| 380498 | views/plugins/views_plugin_pager.inc                          |        1 | 
| 380503 | views/plugins/views_plugin_pager_full.inc                     |       24 | 
| 380508 | views/plugins/views_plugin_pager_mini.inc                     |        2 | 
| 380513 | views/plugins/views_plugin_pager_none.inc                     |        4 | 
| 380518 | views/plugins/views_plugin_pager_some.inc                     |        6 | 
| 380528 | views/plugins/views_plugin_query_default.inc                  |        6 | 
+--------+---------------------------------------------------------------+----------+
12 rows in set (0.00 sec)

mysql> select f.fid,f.location, count(*) from l10n_server_file f inner join l10n_server_line l on f.fid = l.fid inner join l10n_server_string s on s.sid = l.sid where f.rid = 18208 group by f.location;
+--------+-------------------------------------------------------------------------+----------+
| fid    | location                                                                | count(*) |
+--------+-------------------------------------------------------------------------+----------+
| 226630 | views/docs/docs.php                                                     |       21 | 
| 226632 | views/handlers/views_handler_area.inc                                   |        6 | 
| 226636 | views/handlers/views_handler_argument.inc                               |       26 | 
| 226638 | views/handlers/views_handler_argument_date.inc                          |        3 | 
| 226644 | views/handlers/views_handler_argument_many_to_one.inc                   |        8 | 
| 226646 | views/handlers/views_handler_argument_null.inc                          |        2 | 
| 226648 | views/handlers/views_handler_argument_numeric.inc                       |        7 | 
| 226650 | views/handlers/views_handler_argument_string.inc                        |       22 | 
| 226652 | views/handlers/views_handler_field.inc                                  |       54 | 
| 226654 | views/handlers/views_handler_field_boolean.inc                          |       12 | 
| 226656 | views/handlers/views_handler_field_counter.inc                          |        2 | 
| 226660 | views/handlers/views_handler_field_date.inc                             |        9 | 
| 226666 | views/handlers/views_handler_field_numeric.inc                          |       12 | 
| 226668 | views/handlers/views_handler_field_prerender_list.inc                   |        5 | 
| 226670 | views/handlers/views_handler_field_url.inc                              |        1 | 
| 226672 | views/handlers/views_handler_filter.inc                                 |       20 | 
| 226674 | views/handlers/views_handler_filter_boolean_operator.inc                |       10 | 
| 226678 | views/handlers/views_handler_filter_date.inc                            |        6 | 
| 226680 | views/handlers/views_handler_filter_equality.inc                        |        3 | 
| 226686 | views/handlers/views_handler_filter_in_operator.inc                     |       17 | 
| 226688 | views/handlers/views_handler_filter_many_to_one.inc                     |       13 | 
| 226690 | views/handlers/views_handler_filter_numeric.inc                         |       27 | 
| 226692 | views/handlers/views_handler_filter_string.inc                          |       32 | 
| 226694 | views/handlers/views_handler_relationship.inc                           |        6 | 
| 226696 | views/handlers/views_handler_sort.inc                                   |        8 | 
| 226698 | views/handlers/views_handler_sort_date.inc                              |        8 | 
| 226700 | views/handlers/views_handler_sort_formula.inc                           |        1 | 
| 226708 | views/includes/admin.inc                                                |      252 | 
| 226710 | views/includes/ajax.inc                                                 |        2 | 
| 226712 | views/includes/analyze.inc                                              |        2 | 
| 226716 | views/includes/cache.inc                                                |        2 | 
| 226718 | views/includes/convert.inc                                              |       19 | 
| 226720 | views/includes/form.inc                                                 |        1 | 
| 226722 | views/includes/handlers.inc                                             |        8 | 
| 226724 | views/includes/plugins.inc                                              |       57 | 
| 226728 | views/includes/view.inc                                                 |       37 | 
| 226730 | views/js/ajax.js                                                        |        2 | 
| 226738 | views/js/tabs.js                                                        |        2 | 
| 226740 | views/modules/aggregator.views.inc                                      |       33 | 
| 226806 | views/modules/aggregator/views_handler_field_aggregator_category.inc    |        2 | 
| 226810 | views/modules/aggregator/views_handler_field_aggregator_title_link.inc  |        1 | 
| 226814 | views/modules/aggregator/views_plugin_row_aggregator_rss.inc            |        5 | 
| 226742 | views/modules/book.views.inc                                            |       15 | 
| 226746 | views/modules/comment.views.inc                                         |       78 | 
| 226816 | views/modules/comment/views_handler_argument_comment_user_uid.inc       |        2 | 
| 226818 | views/modules/comment/views_handler_field_comment.inc                   |        2 | 
| 226822 | views/modules/comment/views_handler_field_comment_link.inc              |        2 | 
| 226824 | views/modules/comment/views_handler_field_comment_link_delete.inc       |        1 | 
| 226826 | views/modules/comment/views_handler_field_comment_link_edit.inc         |        1 | 
| 226828 | views/modules/comment/views_handler_field_comment_link_reply.inc        |        1 | 
| 226830 | views/modules/comment/views_handler_field_comment_node_link.inc         |        1 | 
| 226832 | views/modules/comment/views_handler_field_comment_username.inc          |        1 | 
| 226838 | views/modules/comment/views_handler_field_node_comment.inc              |        3 | 
| 226840 | views/modules/comment/views_handler_field_node_new_comments.inc         |        2 | 
| 226846 | views/modules/comment/views_handler_filter_node_comment.inc             |        3 | 
| 226856 | views/modules/comment/views_plugin_row_comment_view.inc                 |        1 | 
| 226752 | views/modules/contact.views.inc                                         |        2 | 
| 226858 | views/modules/contact/views_handler_field_contact_link.inc              |        7 | 
| 226756 | views/modules/locale.views.inc                                          |       34 | 
| 226862 | views/modules/locale/views_handler_argument_locale_group.inc            |        1 | 
| 226864 | views/modules/locale/views_handler_argument_locale_language.inc         |        1 | 
| 226870 | views/modules/locale/views_handler_field_locale_link_edit.inc           |        2 | 
| 226872 | views/modules/locale/views_handler_filter_locale_group.inc              |        1 | 
| 226874 | views/modules/locale/views_handler_filter_locale_language.inc           |        4 | 
| 226876 | views/modules/locale/views_handler_filter_locale_version.inc            |        2 | 
| 226758 | views/modules/node.views.inc                                            |      106 | 
| 226878 | views/modules/node/views_handler_argument_dates_various.inc             |        1 | 
| 226880 | views/modules/node/views_handler_argument_node_language.inc             |        1 | 
| 226884 | views/modules/node/views_handler_argument_node_type.inc                 |        1 | 
| 226888 | views/modules/node/views_handler_field_history_user_timestamp.inc       |        1 | 
| 226890 | views/modules/node/views_handler_field_node.inc                         |        2 | 
| 226892 | views/modules/node/views_handler_field_node_link.inc                    |        2 | 
| 226894 | views/modules/node/views_handler_field_node_link_delete.inc             |        1 | 
| 226896 | views/modules/node/views_handler_field_node_link_edit.inc               |        1 | 
| 226898 | views/modules/node/views_handler_field_node_revision_link_delete.inc    |        1 | 
| 226900 | views/modules/node/views_handler_field_node_revision_link_revert.inc    |        1 | 
| 226904 | views/modules/node/views_handler_filter_history_user_timestamp.inc      |        2 | 
| 226910 | views/modules/node/views_handler_filter_node_type.inc                   |        1 | 
| 226914 | views/modules/node/views_plugin_argument_validate_node.inc              |        6 | 
| 226916 | views/modules/node/views_plugin_row_node_rss.inc                        |        6 | 
| 226918 | views/modules/node/views_plugin_row_node_view.inc                       |        5 | 
| 226764 | views/modules/poll.views.inc                                            |        4 | 
| 226766 | views/modules/profile.views.inc                                         |       10 | 
| 226922 | views/modules/profile/views_handler_field_profile_list.inc              |        1 | 
| 226770 | views/modules/search.views.inc                                          |       13 | 
| 226928 | views/modules/search/views_handler_field_search_score.inc               |        6 | 
| 226930 | views/modules/search/views_handler_filter_search.inc                    |        6 | 
| 226934 | views/modules/search/views_plugin_row_search_view.inc                   |        1 | 
| 226776 | views/modules/statistics.views.inc                                      |       26 | 
| 226936 | views/modules/statistics/views_handler_field_accesslog_path.inc         |        1 | 
| 226782 | views/modules/system.views.inc                                          |       20 | 
| 226938 | views/modules/system/views_handler_argument_file_fid.inc                |        1 | 
| 226940 | views/modules/system/views_handler_field_file.inc                       |        2 | 
| 226784 | views/modules/taxonomy.views.inc                                        |       49 | 
| 226946 | views/modules/taxonomy/views_handler_argument_taxonomy.inc              |        1 | 
| 226948 | views/modules/taxonomy/views_handler_argument_term_node_tid.inc         |        4 | 
| 226950 | views/modules/taxonomy/views_handler_argument_term_node_tid_depth.inc   |        7 | 
| 226954 | views/modules/taxonomy/views_handler_argument_vocabulary_vid.inc        |        1 | 
| 226956 | views/modules/taxonomy/views_handler_field_taxonomy.inc                 |        2 | 
| 226958 | views/modules/taxonomy/views_handler_field_term_node_tid.inc            |        7 | 
| 226960 | views/modules/taxonomy/views_handler_filter_term_node_tid.inc           |       12 | 
| 226962 | views/modules/taxonomy/views_handler_filter_term_node_tid_depth.inc     |        3 | 
| 226966 | views/modules/taxonomy/views_handler_relationship_node_term_data.inc    |        2 | 
| 226968 | views/modules/taxonomy/views_plugin_argument_validate_taxonomy_term.inc |        9 | 
| 226790 | views/modules/translation.views.inc                                     |       25 | 
| 226970 | views/modules/translation/views_handler_argument_node_language.inc      |        1 | 
| 226978 | views/modules/translation/views_handler_filter_node_language.inc        |        4 | 
| 226980 | views/modules/translation/views_handler_filter_node_tnid.inc            |        3 | 
| 226984 | views/modules/translation/views_handler_relationship_translation.inc    |        5 | 
| 226792 | views/modules/upload.views.inc                                          |       19 | 
| 226986 | views/modules/upload/views_handler_field_upload_description.inc         |        1 | 
| 226988 | views/modules/upload/views_handler_field_upload_fid.inc                 |        8 | 
| 226990 | views/modules/upload/views_handler_filter_upload_fid.inc                |        1 | 
| 226796 | views/modules/user.views.inc                                            |       44 | 
| 226992 | views/modules/user/views_handler_argument_user_uid.inc                  |        1 | 
| 226996 | views/modules/user/views_handler_field_user.inc                         |        2 | 
| 227000 | views/modules/user/views_handler_field_user_link.inc                    |        2 | 
| 227002 | views/modules/user/views_handler_field_user_link_delete.inc             |        1 | 
| 227004 | views/modules/user/views_handler_field_user_link_edit.inc               |        1 | 
| 227006 | views/modules/user/views_handler_field_user_mail.inc                    |        4 | 
| 227008 | views/modules/user/views_handler_field_user_name.inc                    |        3 | 
| 227012 | views/modules/user/views_handler_field_user_roles.inc                   |        2 | 
| 227014 | views/modules/user/views_handler_filter_user_current.inc                |        1 | 
| 227016 | views/modules/user/views_handler_filter_user_name.inc                   |        3 | 
| 227022 | views/modules/user/views_plugin_argument_default_user.inc               |        1 | 
| 227024 | views/modules/user/views_plugin_argument_validate_user.inc              |        7 | 
| 226800 | views/modules/views.views.inc                                           |       11 | 
| 227026 | views/plugins/views_plugin_access.inc                                   |        1 | 
| 227028 | views/plugins/views_plugin_access_none.inc                              |        1 | 
| 227030 | views/plugins/views_plugin_access_perm.inc                              |        2 | 
| 227032 | views/plugins/views_plugin_access_role.inc                              |        5 | 
| 227034 | views/plugins/views_plugin_argument_default.inc                         |        1 | 
| 227036 | views/plugins/views_plugin_argument_default_fixed.inc                   |        1 | 
| 227038 | views/plugins/views_plugin_argument_default_php.inc                     |        2 | 
| 227040 | views/plugins/views_plugin_argument_validate.inc                        |        1 | 
| 227044 | views/plugins/views_plugin_argument_validate_php.inc                    |        2 | 
| 227046 | views/plugins/views_plugin_cache.inc                                    |        1 | 
| 227048 | views/plugins/views_plugin_cache_none.inc                               |        1 | 
| 227050 | views/plugins/views_plugin_cache_time.inc                               |        5 | 
| 227052 | views/plugins/views_plugin_display.inc                                  |      161 | 
| 227054 | views/plugins/views_plugin_display_attachment.inc                       |       36 | 
| 227056 | views/plugins/views_plugin_display_block.inc                            |       16 | 
| 227060 | views/plugins/views_plugin_display_feed.inc                             |        9 | 
| 227062 | views/plugins/views_plugin_display_page.inc                             |       45 | 
| 227064 | views/plugins/views_plugin_exposed_form.inc                             |       15 | 
| 227066 | views/plugins/views_plugin_exposed_form_basic.inc                       |        1 | 
| 227068 | views/plugins/views_plugin_exposed_form_input_required.inc              |        4 | 
| 227070 | views/plugins/views_plugin_pager.inc                                    |        1 | 
| 227072 | views/plugins/views_plugin_pager_full.inc                               |       24 | 
| 227074 | views/plugins/views_plugin_pager_mini.inc                               |        2 | 
| 227076 | views/plugins/views_plugin_pager_none.inc                               |        4 | 
| 227078 | views/plugins/views_plugin_pager_some.inc                               |        6 | 
| 227082 | views/plugins/views_plugin_query_default.inc                            |        6 | 
| 227084 | views/plugins/views_plugin_row.inc                                      |        2 | 
| 227086 | views/plugins/views_plugin_row_fields.inc                               |        6 | 
| 227088 | views/plugins/views_plugin_style.inc                                    |        4 | 
| 227092 | views/plugins/views_plugin_style_grid.inc                               |        5 | 
| 227094 | views/plugins/views_plugin_style_list.inc                               |        3 | 
| 227096 | views/plugins/views_plugin_style_rss.inc                                |        3 | 
| 227098 | views/plugins/views_plugin_style_summary.inc                            |        3 | 
| 227100 | views/plugins/views_plugin_style_summary_unformatted.inc                |        2 | 
| 227102 | views/plugins/views_plugin_style_table.inc                              |       13 | 
| 227142 | views/theme/theme.inc                                                   |       10 | 
| 227108 | views/theme/views-ui-edit-item.tpl.php                                  |        2 | 
| 227110 | views/theme/views-ui-edit-tab.tpl.php                                   |        1 | 
| 227112 | views/theme/views-ui-edit-view.tpl.php                                  |        5 | 
| 227114 | views/theme/views-ui-list-views.tpl.php                                 |        3 | 
| 226626 | views/views.info                                                        |        2 | 
| 226620 | views/views.module                                                      |       11 | 
| 227146 | views/views_export/views_export.info                                    |        3 | 
| 227144 | views/views_export/views_export.module                                  |       13 | 
| 226628 | views/views_ui.info                                                     |        3 | 
| 226622 | views/views_ui.module                                                   |       17 | 
+--------+-------------------------------------------------------------------------+----------+
173 rows in set (0.03 sec)

The views one shows that the recorded strings are all over the place, BUT when a file is covered, it is covered 100%. The imagecache one shows similar information, the slight difference in number of strings recorded is IMHO due to changes in the release, not because the file was not parsed 100%. So it looks like a file is either included or it is entirely missing, but its not the first few files which are parsed, all files are recorded, so again it is not about the parser timing out.

I've also looked into whether any line information is stored for the given releases, but no line information is stored either. So looking at the code, this should go wrong somewhere between l10n_drupal_save_file() and l10n_drupal_save_string(). The later would not store line and string information if the file it belongs to is unknown. From this, I've realized we don't use the Drupal APIs properly, so submitted and committed #1064542: Use db_last_insert_id instead of custom code, which might help with this issue.

We should devise some queries to find the affected releases systematically, so we can hit regeneration of them.

Ok, I devised this query to look for releases which might be broken (ie. they have less than 2/3rds of strings compared to the general average for the given project). Unfortunately most of these are false positives, eg. install profiles which have core and non-core packages, which greatly vary in size, or mostly just modules exploded from Drupal 5 to 6 to 7 in number of strings. If someone is interested to dig this data, I'm happy to provide a non-limited result set, but this is a sample:

mysql> select p.uri, x.title, avg(r.sid_count) average, x.sid_count from l10n_server_release r inner join l10n_server_project p on r.pid = p.pid inner join l10n_server_release x on r.pid = x.pid group by r.pid having x.sid_count < average * 2/3 limit 40;
+-----------------+---------------+-----------+-----------+
| uri             | title         | average   | sid_count |
+-----------------+---------------+-----------+-----------+
| bookmarks       | 5.x-1.0       |   82.7500 |        34 | 
| gallery         | 5.x-1.0       |  432.3333 |       269 | 
| privatemsg      | 5.x-1.8       |  252.5000 |        97 | 
| project         | 5.x-0.1-beta  |  388.7500 |       230 | 
| recipe          | 5.x-1.0       |  177.9091 |       102 | 
| subscriptions   | 5.x-1.9       |  249.1111 |        85 | 
| weather         | 5.x-1.1       |  192.1786 |        73 | 
| api             | 5.x-1.0       |  102.2500 |        65 | 
| webform         | 5.x-1.10      |  466.1563 |       268 | 
| captcha         | 5.x-1.1       |  143.2500 |        24 | 
| nodewords       | 5.x-1.13      |  166.6667 |        64 | 
| simpletest      | 5.x-1.2       |   98.1667 |        32 | 
| print           | 5.x-1.2       |  184.0000 |        32 | 
| taxonomy_block  | 6.x-1.0-beta1 |   15.6667 |        10 | 
| week            | 5.x-1.2       |   50.6000 |        15 | 
| fckeditor       | 5.x-1.3-beta  |  274.0556 |        51 | 
| filebrowser     | 5.x-1.1       |   52.0000 |        22 | 
| creativecommons | 6.x-1.0-beta2 |  134.2500 |        68 | 
| spreadfirefox   | 5.x-0.1       |    3.3333 |         1 | 
| taxonomy_xml    | 5.x-1.0       |   93.0000 |        26 | 
| location        | 5.x-2.7       |  450.6000 |       204 | 
| blogtheme       | 5.x-1.2       |   30.0000 |         2 | 
| rsvp            | 5.x-1.8       |  401.8333 |       129 | 
| slash           | 5.x-1.1       |   10.5000 |         2 | 
| adsense         | 5.x-1.6       |  360.6667 |       176 | 
| user_badges     | 5.x-1.4       |  105.0000 |        66 | 
| userpoints      | 5.x-2.16      |  190.3750 |        94 | 
| quiz            | 6.x-2.2       |  733.9667 |       256 | 
| browscap        | 5.x-1.0       |   24.7500 |         2 | 
| signup          | 5.x-1.1       |  322.6667 |       119 | 
| votingapi       | 5.x-1.6       |   69.0000 |        34 | 
| comment_upload  | 5.x-0.1       |   24.5000 |        16 | 
| send            | 5.x-1.1       |  108.2000 |        64 | 
| family          | 5.x-1.4       |  147.3750 |        67 | 
| views           | 5.x-1.6       | 1024.6429 |       488 | 
| lineage         | 5.x-1.5       |   11.6667 |         3 | 
| urllist         | 5.x-1.0       |   23.0000 |         6 | 
| db_maintenance  | 5.x-1.2       |   33.0000 |        21 | 
| cck             | 5.x-1.10      |  378.3000 |       251 | 
| systeminfo      | 5.x-1.1       |  176.8333 |        76 | 
+-----------------+---------------+-----------+-----------+
40 rows in set (0.32 sec)

hi,

first: thanks very much for all your effort in the project localization server.

i pick the example google_analytics version 6.x-3.2

it was released on 7th of February:
http://drupal.org/node/1053974

it had no strings on 13th of Feb.
see #45

it was parsed also on 17th of February:
with 111 strings
http://localize.drupal.org/translate/projects/google_analytics/releases/...

=> conclusion: yes, this happens also for new releases.

Video filter 6.x-2.8 is empty.

LoginToboggan 6.x-1.9. Please re-scan this release.

Admin 6.x-1-.0-beta5 - 1 string is missing.

I think, the parsed 7.x releases of the Pathauto module is bad. The scanned strings does not match with the reality. I founded more than 30 differences.

Mollom 7.x-1.1 - 0 strings

webform 7.x-3.10 and 7.x-3.11. Please re-scan this releases.

Privatemsg 6.x-1.5 - 0 strings
Quick Tabs 6.x-3.0 - 0 strings
XML sitemap 6.x-2.0-beta3 - 0 strings

webform 7.x-3.10 and 7.x-3.11. Please re-scan this releases again. Currently both contains 0 strings.

tagging-6.x-2.5. One string in missing.

Please, review #1267466: Start over packages without the administer localization server permission for the temporary solution.

Admin 6.x-2.0 - 5 strings are missing.

Devel 7.x-1.2 (cca 150 instead of 336)

quova 7.x-1.0

Thanks to Zoltán Balogh we put in a self-service workaround for now, see http://localize.drupal.org/node/4124. I still think it would be best to fix the root cause, since this should not require manual labour, neither in tracking these down, neither rescanning them. It should work right away... Anyway, we should have a manual way to work around this issue for now until we figure out why is this happening... Thanks again Zoltán!

rightly parsed but no strings ??

Why not add a strings count checker or CRON run every night to check it and re-parse it automatically

maybe add directly to l10n_drupal_drupalorg_parse_package before unlink the unzip folder/files

All mentioned releases between #52 and #65 has been updated. In a similar situation please use the Missing strings? link below the releases combo. Its shown only, after you filtered the project and the release. If you don't have enough permission to start over a release, then ask your team admin, or contact me.

For role_delegation localize server knows
http://localize.drupal.org/translate/projects/role_delegation/releases
only version 7.x-1.0 but not version 7.x-1.1
http://drupal.org/project/role_delegation from june 2011 :-(

any ideas?

maybe this is a hint:
http://localize.drupal.org/translate/projects/better_exposed_filters/rel...
-> 3 releases

but in the user interface
http://localize.drupal.org/translate/languages/de/translate?project=bef&...

only 1 release!

So any ideas?

ahh.

Better Exposed Filters moved from bef to better_exposed_filters. "Better Exposed Filters" matched 2 results in DB but only return the first one.

http://localize.drupal.org/translate/languages/de/translate?project=bett...

@Thomas_Zahreddin,
It's different issue, here's the patch:
#1380682: Fix filter to show enabled project only