Fix how drupalorg_drush handles release history XML and propagates data to the packaging script

Problem/Motivation

Currently, drush make --drupal-org requires that *everything* in the projects array in the .make file corresponds with a valid release node on d.o. If we can't find a release node (e.g. no releases for the project yet at all), we consider it a fatal build error. Furthermore, it will currently blindly pull the latest recommended release if the project has history but we still can't find the right release node.

Also, since the data currently being propagated to the packaging script is so minimal, the packaging script has to do an extra SQL query on every distro it builds to get data that drush make already has.

Proposed resolution

• We should be smarter about how we search for release nodes in the release history XML. If we can't find a release node, it shouldn't be a fatal error. Given #1432312: Set project-level 'version' attribute automatically if a Git 'branch' or 'tag' attribute is defined. we can always look for a top-level version attribute, and if we have that, we request the history for exactly that version.
• Never fall back to find the latest recommended release for a given project/branch.
• If we can't find a release node, just remember the download attributes we used (url, revision, branch, etc)
• Instead of writing out a package_contents.txt file with just release nids, we'll write out a package_contents.json file with a nested array of data about each project and what we found. If we've got a release node, we include that nid. Otherwise, we include the download attributes we have. For example, the PHP array for the data would look something like this:
  

  array(
    'coder' => array(
      'nid' => 124124,
    ),
    'devel' => array(
      'url' => 'git://git.drupal.org/project/devel.git',
      'revision' => 'c2048dfe1d8caf9aa2608641f4f4fec413e4c46b',
      'branch' => '6.x-4.x',
    ),
  );
  

  In this imaginary example, we have a release node for coder (could be official or -dev, doesn't matter), but there's some revision on a new branch of devel that doesn't even have a -dev release node yet so we just explain WTF was checked out. The resulting package_contents.json file would therefore look something like this:

  {"coder":{"nid":124124},"devel":{"url":"git:\/\/git.drupal.org\/project\/devel.git","revision":"c2048dfe1d8caf9aa2608641f4f4fec413e4c46b","branch":"6.x-4.x"}}
  

• Fix the packaging script to parse this .json file and use that to a) avoid the query it's doing to find project shortnames for all the release node IDs (since we'll already have those as the top-level array keys) and then figure out what contents to display on the distro release node based on the data for each project (instead of assuming they're just release node ids).

Remaining tasks

• Fix drupalorg_drush to be smarter about how it searches for release nodes from the release history XML for the projects in the .make file.
• Have it construct the array of project => release/Git id/whatever data for everything it built, and write that out to the package_contents.json file.
• Fix the packaging script to read/parse this .json file, and store the right stuff in the project_package tables as appropriate (needs a separate issue)

User interface changes

When something is packaged into a distribution on d.o that doesn't have a release node we can associate with it, we'll just display that on the distro release node as if it was external code and print a URL (and the revision/branch whatever we know).

API changes

When the --drupal-org-log-package-items-to-file is set, a package_contents.json file will be written with a full nested array of info about projects instead of the old package_contents.txt file that just contained a newline-separated list of release nid integers.

Original report by dww

I'm nearly positive there's some infra or webmaster issue related to this bug, but I can't find it now...

I just discovered that if you're building a .make file and pointing it at Git stuff that doesn't correspond to a release node, drupalorg_drush just blindly pulls the release nid of the latest official release from the recommended branch. So, for example, if you do this:

projects[update_test_module][type] = module
projects[update_test_module][download][type] = git
projects[update_test_module][download][branch] = 7.x-4.x

You end up with:

Could not locate update_test_module version 7.x-4.x-dev, downloading   [warning]
latest stable version
update_test_module cloned from                                              [ok]
git://git.drupal.org/project/update_test_module.git.
Checked out branch 7.x-4.x.                                                 [ok]

And the package-contents.txt file contains: 683972 which is update_test_module 7.x-1.3.

I'm not 100% sure what we should do in this case, but we definitely shouldn't do what's happening here. ;)