Package details not show on release page

I inspected the project_package_local_release_item table in the DB, and it looks like this is a problem with the table being populated.

Query on a Spark release node:

SELECT * FROM project_package_local_release_item WHERE package_nid = 1901600;

package_nid	item_nid
1901600	0
1901600	405854
1901600	999344
1901600	1044008
1901600	1460550
1901600	1650068
1901600	1662780
1901600	1672954
1901600	1722476
1901600	1733788
1901600	1736460
1901600	1738460
1901600	1738468
1901600	1751522
1901600	1799508
1901600	1802928
1901600	1860642
1901600	1873988
1901600	1898494
1901600	1901356

Query on agov:

SELECT * FROM project_package_local_release_item WHERE package_nid = 1890706;

package_nid	item_nid
1890706	0

Thanks to the sleuthing of Dave Reid: http://drupalcode.org/project/drupalorg.git?a=search&h=refs%2Fheads%2F6....

We know that this is probably a bug in http://drupalcode.org/project/drupalorg.git/blob?f=drupalorg_project/plu.... It's doing some kinda funky thing with Drush and saving package info to a temp dir or... I dunno.

Halp! :)

I looked into this. The problem that we have is that the generated metadata-contrib.json file contains multiple JSON objects smushed together. For example, a test run on Commerce Kickstart gives us:

https://gist.github.com/damz/cc1b78fc7e9f9f66465d

I assume that this is because of recursive runs of Drush Make (ie. the chosen module is checked out, and it contains itself a make file that gets processed separately). I assume this is a Drupal.org drush bug / limitation. Reassigning there.

We cannot prevent Drush from spawning a new process for recursive processing, so I see two options:

• We pass a directory instead of a file to --drupal-org-log-package-metadata. In DrupalorgProjectPackageReleaseDistro, we load all the *.json files in this directory and recursively merge them.
• We make sure to frame our JSON exports (for example by a \n character, which is a common framing boundary for multiple JSON data in a stream). In this case, we probably also want to lock the file (with flock) during the write operation, as writes are only guaranteed to be atomic up to PIPE_BUF characters. In In DrupalorgProjectPackageReleaseDistro, we decode each JSON object from the stream and recursively merge them.

Thanks for investigating, DamZ! I've always hated recursive drush make. ;) Yeah, I don't think drupalorg_drush and its communication with the packaging script was designed to properly handle this case. Ugh.

On very quick skim of the issue and based on my spotty memory of how all this crap works, I think the directory and separate files per drush invocation will be cleaner, but probably more work. Trying to get a bunch of concurrent drush makes to lock the same .json file and properly smoosh the output together in a sane way seems like more complication than just letting each drush make write its own file. Although, then the packaging script has to parse all the .json files and merge the results. I dunno.

I can't really justify spending a ton of time cleaning this up right now with the D7 upgrade so far behind. But, I'm now aware of this and will be happy to help answer questions (such as I can) and provide advice if anyone is motivated to work on this in the meanwhile...

Thanks/sorry,
-Derek

p.s.
- Just marked #1794854: Dependencies included with sub make files are not properly shown on D.O duplicate with this -- even though that's older, this issue as more info.
- Also, I'm staring to use a new 'Drupal.org distribution packaging' tag since 'drupal.org distribution blockers' is no longer really a valid name, but it'd be nice to track all issues related to d.o distro packaging across all projects (since some of the problems *are* in drupalorg_drush, some are in drush itself, some are infra, etc).