No new releases since June 12

It's not only Rules 7.x-2.0-beta2 that is missing. I searched back for all new projects and couldn't find any new release on LDO after 06/12/2011 - 21:17.

Yes, it seems that the generation of new translation versions or parsing new module versions is not functioning right now. I update the priority of this issue.

We've had issues with one of our different cron processes hanging on localize.drupal.org. Looking into it.

Some bad release was apparently holding things up. We are now running cron more often and the backlog should be caught up with later today.

The backlog has been processed, I believe this to be fixed.

I believe it is not fixed yet. #1218156: German translation packages missing for very many releases lists several releases publised after 12.6 and there is still no download (at least 8h ago) when I tried to update the translations... Can you check again, please?

100% not fixed. No german drupal 7.4 download... Reopening.

This is also the case for French and Norwegian, but I suppose it's more of a general thing for all languages.

@Gabor: I've increased l10n_packager_release_limit to 300 in settings.local.php in order to speed up processing. Let's see what happens.

I think there is something wrong with the logic which decides which files to re-package:

For Drupal core, there were PO files for 6.14 repackaged as recently as 2 days ago. This clearly doesn't make sense and there has to be a bug somewhere. Also, a lot of alpha and beta releases get packaged, is this intentional?

Yes, the packaging "queue" logic needs to be rehauled for the increased release/language number. It was crafted by Jose at a time when we had a lot less projects/releases and especially languages. I'm happy to give guidance to people who want to help with working on this (I'm busy with Drupal 8 stuff, Dries asked me to prioritize that ahead). Until we have people to help out fixing the logic or I get more time allocation from my employer (I tried to ask for that several times), I don't have better options for you but to wait around.

Thankfully we have all the data sitting in the database to write logic to have a real queue for packaging which prioritizes project based on usage, real translation changes, etc. Right now the packaging logic is pretty dumb and takes cycles looking at whether it needs to regenerate things, which end up not being regenerated. It does not work off of the "change stream" and is not aware of project usage (priority) data, that was added to the server way later compared to packaging.

Here is a sampling of recent log entries for .po file generation. They note how much time it took, how many releases it looked at and how many actual .po files it ended up generating. All the releases and languages it looks at but figures out need no .po file generation are of course in vain and should not happen with a better logic (an actual queue system):

116730.58 ms for 300 releases/15 files.
132342.41 ms for 300 releases/14 files.
111673.63 ms for 300 releases/3 files.
112738.22 ms for 300 releases/10 files.
115098.38 ms for 300 releases/10 files.
80199.81 ms for 200 releases/24 files.
39189.33 ms for 100 releases/8 files.
38125.84 ms for 100 releases/0 files.
37849.72 ms for 100 releases/12 files.
37122.1 ms for 100 releases/3 files.
37661.35 ms for 100 releases/8 files.

(Note that it changed to 200 and 300 as killes reconfigured it). Because we almost have 100 languages, where the packaging script look at 300 releases, it would ideally at least generate 300 files (at least one language should have changes for each release, if we are working off of a changelog). Once again, the packaging code currently does not take into account the change information and the priorities for projects, it sort of "wonders around" (only takes into account its own information about when .po files were generated, and when it looked at a release).

Now that killes increased the limit to 300 releases, theoretically it could swipe through all the releases in 80 cron runs. We run these packaging crons every 5 minutes, so that is 400 minutes, which is still over 6 hours. That probably does not sound too bad, however, with the previous 100 settings, it was then supposedly 3 times as much (18+ hours), and we have seen it not actually regenerating as soon either. So I suspect there is a possibility for some other bugs too above the inefficiency of the generation process that is demonstrated with the log messages above.

Help welcome!

There are about 23k releases that are enabled on l.d.o, nearly 5k are beta releases and 2.5k are alpha. There are another 1.4k release candidates. I can see why these should be packaged.

In any case, packaging of alpha/beta/rc should stop after the release was made. I agree that this is probably difficult to capture in logic. An UI on ld.o might help though, I didn't find a place where I could disable individual releases. I also think that releases that aren't marked as current should be stopped to be packaged.

I considered to disable packaging for old releases in the database, but I want to discuss this with Gabor first.

I have manually created a drupal-7.4.de.po using the admin UI on l.d.o.

Note: this messes up the file's permissions on www1 to it is probably to be avoided if you don't have root access there.

Yes, drupal.org already has this info on whether a releases is current or not. Currently, that information is not propagated to l.d.o, that would be a great project for someone to take on (once again I can provide some support as to how to do that, probably best to consult someone else from the project module people too). Also, RCs, alphas, etc. do need to have .po files generated at least until their release comes out, so that people can work on their translations and test them in real environments.

Seeing that there were no problems so far I've increased the # of checked releases to 500. That's probably as much as we can do on the current hardware.

I will also disable parsing for alpha/beta/RCs that are older than 3 months if I can figure out how to.

CREATE TABLE l10n_old_stuff (a INT NOT NULL NULL AUTO_INCREMENT, PRIMARY KEY (a), KEY(rid)) SELECT r.rid FROM l10n_packager_release pr LEFT JOIN l10n_server_release r ON pr.rid = r.rid WHERE (r.title like '%-beta%' OR r.title like '%-alpha%' OR r.title like '%-rc%') and from_unixtime(r.file_date) < '2010-04-15' ;

update l10n_packager_release r inner join l10n_old_stuff o on r.rid = o.rid set r.status = 0 ;

This is what I intend to tun. It would set about 3.1k releases to "inactiv". Objections?

We can keep that extra table around for a bit so that we can revert this in case it causes more troubles.

Ah, and I'll add the 3k D5 releases as well.

Since this is reversible, I went ahead and executed it:

update l10n_packager_release r inner join l10n_disabled_releases_1202848 o on r.rid = o.rid set r.status = 0 ;
Query OK, 5751 rows affected (0.04 sec)
Rows matched: 5751 Changed: 5751 Warnings: 0

I had the date wrong when I disabled old RCs etc. Ran it again with the more current date: 10k extra disabled.

As a test, I've changed the query that retrieves releases that need packaging. It was ordered by pr.checked, I changed it to r.weight..

When I viewed the result in mysql CLI it made much more sense to be, ie things like core, views, cck near the top. The former query would have unknown stuff on top.

the original query would also have new releases on top so I changed it back.

I've then set the variable l10n_packager_update to one day. As I understand it, this should ensure that each release only gets processed once per day. This seems to have helped a great deal, now we are getting lots of releases processed.

However, we seem to be limited by the # of files now:
55601.93 ms for 53 releases/100 files.

Not sure where I can increase that and if I should.

found it, increased to 200.

increased to 1000

increased the packager update intervall to once every 3 days.

This means there are currently 8790 packages in the queue.

Great, thanks for investigating this. On ordering by weight, that could possibly starve the lower priority projects/releases depending on other factors, so those might never get translation files created. I think ideally we'd have a queue which gets (a) new or newly parsed releases added (b) releases added when people modify translations for them. So that only new stuff and modified stuff based on actual actions gets into the queue. Right now actual changes are not considered. Looks like you are getting to a much better situation by just tweaking the numbers, so we might be able to postpone making the release queueing better then, just wanted to jot this down.

Increased # of files to 2000.

8036 releases pending.

I thing the most urgent thing to do is to ensure that outdated releases do not get rebuilt over and over again for no gain. We still have over 42k active releases from 6.4k enabled projects.

1565937.76 ms for 500 releases/1772 files.

7683 releases pending.

Also, we seem to package install profiles separately. I would think that this should not be needed since all the individual components should already have been packaged.

e.g. https://drupal.org/project/displaysuite has a number of _huge_ translation files that in all likelyood won't really be used.

And they take a lot of time to package.

@killes: Theoretically people can use those big .po files to download and import on their system (which will not time out if they have gettextapi.module enabled). This might indeed be entirely theoretic.

Wel, i wouldn't mind to provide these files, but we should generate them in a different way, ie by concatenating the files of the individual modules.

Another problem with these files is that they change very often. I've disabled translation for all the old displaysuite releases.

Certain authors seem also to believe that they need to make a new release for each two-line change...

I've seen 160 releases from 3 branches for 1 module...

So you said after 6 hours all files of all projects should be generated. I have updated a few tranlations 8h ago when i stil had 3 missing translation packages. Checked again now and they are still missing. Here is te list:

Insert 7.x-1.1
Pathauto 7.x-1.0-rc2
Imce 7.x-1.4

Yes, there is still a problem. The query I gave above returned less and less results till about 7k, now it is back up at nearly 8k and very few packages are made per run.

I think the query run is wrong, here is one that I believe is more correct.

SELECT p.uri, from_unixtime(pr.checked), from_unixtime(pr.updated) FROM l10n_server_release r INNER JOIN l10n_server_project p ON r.pid = p.pid LEFT JOIN l10n_packager_release pr ON pr.rid = r.rid WHERE pr.status IS NULL OR (pr.status = 1 AND (pr.checked < (unix_timestamp(now()) - 3*86400 ) OR (pr.updated < unix_timestamp(now()) - 3*86400 ) AND pr.checked < unix_timestamp(now()) - 3*86400 )) order by pr.checked

Now running that query with a 1 day delay. A lot more releases get packaged again.

Also, there seems to be a but, some releases show up twice:

+--------+-------+----------------+------------+------------+--------+-----------------------------+
| rid | pid | title | checked | updated | status | uri |
+--------+-------+----------------+------------+------------+--------+-----------------------------+
| 235104 | 24584 | 7.x-1.0-beta2 | 1310665626 | NULL | 1 | commerce_kickstart |
| 235094 | 24584 | 7.x-1.0-beta2 | 1310665627 | NULL | 1 | commerce_kickstart |

All releases seem to be processed, but the ones mentioned as missing are still not there:

Insert 7.x-1.1 is marked as checked today but not updated (NULL)

Same for pathauto and imce.

I don't really understand how this works, so I'll keep my fingers off it.

The last Pathauto release (7.x-1.0-rc2) still isn't generated (at least for French language).

Is there some way I can help you ? By providing more examples of what is processed and what is not, for example ?

Modules posted in #30 are all still not available...

Is there any way I can help ?

Still no clue about how to solve the issue ?

I'm trying to understand the cause of that, but I do not have enough information.

SELECT r.*, p.*
FROM l10n_server_project p
INNER JOIN l10n_server_release r ON p.pid = r.pid
WHERE p.connector_module = 'l10n_project_drupalorg' AND p.uri = 'pathauto'
ORDER BY r.file_date ASC

Can anybody who have access to l.d.o DB public this query result here?

Moving this issue to the new l.d.o project.

I think #1480256: Project beididp not listed on l.d.o is related to this issue

I guess cron is down again.
Every package is: Up to date as of: 2012-Mar-08

Hmz, any news on this issue ?

This should be fixed as of yesterday.

How often are existing files updated? We made some large changes to the german translation and these is still 5th march, but the changes have been made in the last 2 weeks.

It looks like this issue has been finally fixed. Closing.

Still active, e.g. only one release at l.d.o for Better Exposed Filters.

Marked #1461100: missing releases of better exposed filters dev modules on l.d.o as a duplicate of this issue.

The main issue here was the (re-)packaging as I know. Not about missing versions in the dropdown filters.

Lots of translations still aren't available. But I can download fresh Drupal 5 translations... Anything thing we can do to help?

Could a message be posted on l.d.o explaining the current situation (linking here)?

If I would know the causes of the issues, I'd be happy to update everybody. The current situation for me is that I'm both busy with work and free time Drupal commitments (do 2+1/2 talks at DrupalCon Munich, leading a huge Drupal 8 multilingual initiative sprint there and working on stuff for that, etc.), so I don't have time to look into this in any capacity for an indefinite time. We need others who are interested enough to solve l.d.o problems and feature requests to take these on. @sebcorbin is occasionally working on updating the site to Drupal 7, which should help working on a relatively newer code base vs. building/fixing off an old Drupal core version, but he needs help in that too. We need active contributors to the site infrastructure itself.

Thank you Gabor for your post Who loves localize.drupal.org? New site maintainers needed!

I think the most recent comments of this issue are in fact instances of the problem I reported a while ago #1195110: Some modules show empty translation sets for given releases, which I believe has something to do with the same version number used for module releases targeting different versions of core.

The same trouble is now, 19.01.2014
There are no new files for latest drupal cores

@podarok: that would be due to #2124931: Update l.d.o syncing for d.o update of project data not related to prior issues.

This is now fixed, but please open another issue if you find that no new releases are being parsed.

\o/

Thanks, SebCorbin!