Localize.drupal.org release parsing broken since 2010 Christmas

Huh, I don' have time to look at this immediately but assume this might have got broken with the .zip packaging that was rolled out on drupal.org. Possibly the ways we should collect this data changed. Doh. Moving to the infrastructure queue.

Ok, looked this up in the database (although this is also visible on the UI). Basically, localize.drupal.org now has 29 thousand(!!!) releases waiting in the queue for being parsed. Uh, oh. Of course these are all due to #11416: Please provide *.zip downloads., which filled in the release file list with .zip files for each release. Localize.drupal.org tries to pick up the top 10 releases to parse on each cron run, but all of those are .zip files now, so none of them can be extracted by tar (unsurprisingly). Therefore parsing of releases is not happening. It last happened on December 24th, before .zip packaging was rolled out:

mysql> select from_unixtime(max(last_parsed)) from l10n_server_release;
+---------------------------------+
| from_unixtime(max(last_parsed)) |
+---------------------------------+
| 2010-12-24 20:46:24             | 
+---------------------------------+
1 row in set (0.02 sec)

Since all packages will be still provided in tar.gz as well, our script should just ignore all .zip files for releases. Looks like there was no schema change (good), so we need to look up the filename by pattern (bad). So change our query to filter for tar.gz files only. Here is a patch I think would do the trick, but not sure about how slow it is going to be :| Maybe its better to do the filtering outside the query. Here is another patch for that too.

All-in-all, I'll need to manually remove all .zip file entries once this new code is reviewed, committed and deployed, since the new code will not have .zip files carried over anymore, but the old ones need to be cleared out.

Patches got lost somehow, trying to attach again.

Sorry, I didn't realize you were directly querying the d.o {files} table. The release history XML was altered in a backwards compatible way over at #652566: Support multiple download files per release in release history XML...

Anyway, yeah. Project* is struggling with the lack of an 'extension' column in {files}, too. We do various little tricks to parse the filename ourselves in PHP.

If you're JOINing on {project_release_file} already, perhaps the new weight column from #1006346: Add a {project_release_file}.weight column could help you. The .tar.gz for each release is (at least for now) always going to have the lightest weight. In fact, the main .tar.gz for a given release is always going to have weight 0, and the .zip will be weight 1. So, you could filter on that if you wanted. I don't know how often this query runs, but probably the LIKE would be more reliable than trusting weight, since we might change weights in the future (e.g. if the "put zip first" crowd wins that argument). Or just filter in PHP, sure. Either way seems fine, but I don't know enough about how this query is used to have a more definitive answer.

Ok, I committed the second patch and deployed that on localize.drupal.org. Also run this query:

mysql> delete from l10n_server_release where download_link like '%.zip';
Query OK, 28820 rows affected (0.39 sec)

Now it should be all up to the cron jobs to complete parsing the 223 outstanding releases that were not yet parsed due the days passed. This cron runs 4 times every hour and parses 10 releases at once. So handles 40 per hour, which means we should be done in 5-6 hours with catching up.

Ok, Drupal 7.0 RC4 is now parsed and available and the parsing process seems to run great again. I sped it up by kicking the cron job more often, so we only have 100 releases to go in the queue now. Should be all done in a couple of hours.

Great, glad to hear it! Sorry for the trouble .zip caused l.d.o -- definitely wasn't intentional...

Cheers,
-Derek

@Magnus: yes, that one is actually a totally different issue. Can you open another issue about this? Thanks!