Support multiple download files per release in release history XML

Very very closely related to #555362: Support multiple download links in the available updates report (.tar.gz + .zip packages like on drupal.org). I guess we could call this the sister issue for the Project* changes for the same problem. I spelled out some suggestions about how it could look/work over there. It just needs testing with existing update.module clients to make sure the new XML schema doesn't horribly break the code already out there in the wild. If it does horribly break stuff, then we get to introduce a new "XML schema version" concept, and host new files at a new URL, e.g. updates.drupal.org/release-history-[XML_schema_version]/[project_name]/[core_version] or updates.drupal.org/release-history/[project_name]/[core_version]/[XML_schema_version] or something.

In terms of the actual question here (how to distinguish between the different file types) at this point there's no metadata field in {project_release_file} that distinguishes them by type. I'm a bit reluctant to add one, since it'd be sort of hard to do it in a way that made sense in general which wasn't very specifically tied to d.o's own use of this system. We just use fid right now. If anything, we'd add a simple integer weight/delta field so you could reorder the files on the release node (hrm, thought I already opened an issue about this -- I guess not, just mentioned it in comments in lots of others). Regardless, I suggest we just list all the files for a given release inside the XML for that release (as explained at #555362) and then for now, drush just parses the filenames to tell -core vs. -no-core vs. nothing.

It's broken now. That's why you can't find it. Search for "// TODO: This is broken for N files per release node" in http://drupalcode.org/viewvc/drupal/contributions/modules/project/releas... -- this issue is about adding all N files associated with a release node to the release history XML inside a single <release> directive. Right now, it's fubar.

Having made a bunch of noise about this, I feel obliged to investigate helping here. ;-)

Will check out /drupal/contributions/modules/project/release/ and see if I think I'm capable of making a meaningful contribution. I'm not a great coder, but I have my moments... perhaps I can at least make a start. =)

Officially expanding the scope of this to handle zip vs. tar.gz, not just core vs no-core vs. make-file for packaged distros. This is blocking progress on #555362: Support multiple download links in the available updates report (.tar.gz + .zip packages like on drupal.org) and #11416: Please provide *.zip downloads.. I hope to get the green light to start working on this in the very near future, so assigning myself.

Switching this to myself.

This looks pretty straight forward, except for the items below:

A) There is currently no label stored with the file info in {project_release_file} to indicate what type of file it is, or how it should be labeled. The release script could just inspect file names and file extensions to get this information, but I wonder if we should think about finding another way to handle this since this would touch both the release packaging and the update history scripts each time a change is made, and we would have to special case how profiles are handling the the release XML from how regular modules/themes are handled. (this is related somewhat to the second item)

B) To update the xml creation script, a separate query would need to be added to fetch items from the project_release_file table instead of using the JOIN since that creates a separate node in the XML for each file. This isn't a huge deal, but over at #555362: Support multiple download links in the available updates report (.tar.gz + .zip packages like on drupal.org) I filed a sample schema for the release XML that keeps backwards compatibility with the update status module in D6 (haven't tested D7 yet). That schema leaves the current tar.gz file in place in addition to the new downloads section for files. We would need a way to determine what the default file should be for each release, this could be hard coded, and use the same detection method as in A, or this too could be a column in project_release_file, that indicates the default file for each release. This could even be extended someday to allow a UI for picking the default file for a release, to be used in download tables, and the release-xml. If such a column were added, it's value would probably be set in the release packaging scripts.

This could even be extended to include an enabled column in project_release_file for which files to include/list, but I feel that is getting too far outside of the scope of this issue. Right now I am just looking for feedback on the method for labeling each file (parse filename, store in DB, or ____) and how to select the default file for the backwards compatibility items in the release XML.

Re: A) Yeah, that's an interesting problem. So far, the closest thing to a "solution" to this is the -ext views token from project/release/views/handlers/project_release_handler_field_files.inc. It works, but it's a bit of a hack. And it certainly wouldn't work for the view of files in a packaged distro (e.g. at http://drupal.org/node/945570 ).

That said, it's not obvious to me that the server is the right place to be deciding what the labels should be in clients. I still think it's better for each client to decide what label is most appropriate for itself. Conceptually, it seems wrong for the server to be doing this logic and sticking this hint into the (already bloated) release history XML. The server can't know what label makes sense for all different clients, so I'm not sure it should even try.

Re: B) First step, see #645712: Enhance project_release_handler_field_files to allow sorting and limiting # of files displayed ;)

Eventually, we might want to add a {project_release_file}.weight column for this, but for now, sorting by fid, filename, filesize, or timestamp should work for 95% of the cases.

The {upload} table has a weight, which is why you can reorder files that you attach to nodes via the upload.module. We might need to add this weight column and consider a UI for this.

That said, for the initial pass, I'd just say to order by the fid. The smallest fid for a release node is the default (as far as the release history XML is concerned). The packaging script can be sure to insert the .tar.gz versions of things into {files} first so that they have the lower fid. That's probably all we need for our purposes here.

I was gonna post this on #555362: Support multiple download links in the available updates report (.tar.gz + .zip packages like on drupal.org) but it makes more sense here.

C) Sounds like the simplest plan is to remove all descriptive data from the section of each release. I was gonna suggest adding a <format> element to each file, but if we're just parsing that from the filename on our end, it makes sense to just leave that to the clients. Just to clarify, that would make a module look something like: (Note that I removed the <text> element from each file)

<release>
  <name>project 6.x-2.1</name>
  <version>6.x-2.1</version>
  <version_major>2</version_major>
  <version_patch>1</version_patch>
  <status>published</status>
  <release_link>http://default/node/315</release_link>
  <download_link>http://default/./sites/default/files/project/Copernicus.dmg</download_link>
  <date>1289721492</date>
  <mdhash>2da41c489f5585485df145d34d3d2650</mdhash>
  <filesize>5165903</filesize>
  <files>
    <file>
      <url>http://ftp.drupal.org/files/projects/drupal-7.x-dev.tar.gz</url>
      <mdhash>6331b74cf71a09f699ff5a23d1207926</mdhash>
      <filesize>1995307</filesize>
      <filedate>1250813342</filedate>
    </file>
    <file>
      <url>http://ftp.drupal.org/files/projects/drupal-7.x-dev.zip</url>
      <mdhash>6331b74cf71a09f699ff5a23d1209999</mdhash>
      <filesize>1999666</filesize>
      <filedate>1250813345</filedate>
    </file>
  </files>
  <terms>
    <term><name>Release type</name><value>New features</value></term>
    <term><name>Release type</name><value>Bug fixes</value></term>
  </terms>
</release>

And profiles would look something like:

<release>
  <name>drupalorg_testing 6.x-1.0-alpha6</name> 
  <version>6.x-1.0-alpha6</version> 
  <tag>DRUPAL-6--1-0-ALPHA6</tag> 
  <version_major>1</version_major> 
  <version_patch>0</version_patch> 
  <version_extra>alpha6</version_extra> 
  <status>published</status> 
  <release_link>http://drupal.org/node/945570</release_link> 
  <download_link>http://ftp.drupal.org/files/projects/drupalorg_testing-6.x-1.0-alpha6-core.tar.gz</download_link> 
  <date>1287450998</date> 
  <mdhash>96dafd4ce930e7ff3a40a5ad9cc32629</mdhash> 
  <filesize>3499033</filesize>
  <files>
    <file>
      <url>http://ftp.drupal.org/files/projects/drupalorg_testing-6.x-1.0-alpha6-core.tar.gz</url>
      <mdhash>96dafd4ce930e7ff3a40a5ad9cc32629</mdhash>
      <filesize>3499033</filesize>
      <filedate>1287450998</filedate>
    </file>
    <file>
      <url>http://ftp.drupal.org/files/projects/drupalorg_testing-6.x-1.0-alpha6-core.zip</url>
      <mdhash>96dafd4ce930e7ff3a40a5ad9cc32629</mdhash>
      <filesize>3499033</filesize>
      <filedate>1287450998</filedate>
    </file>
    <file>
      <url>http://ftp.drupal.org/files/projects/drupalorg_testing-6.x-1.0-alpha6-no-core.tar.gz</url>
      <mdhash>c22f10955f6a5291e39165ade232f77c</mdhash>
      <filesize>2399783</filesize>
      <filedate>1287450995</filedate>
    </file>
    <file>
      <url>http://ftp.drupal.org/files/projects/drupalorg_testing-6.x-1.0-alpha6-no-core.zip</url>
      <mdhash>c22f10955f6a5291e39165ade232f77c</mdhash>
      <filesize>2399783</filesize>
      <filedate>1287450995</filedate>
    </file>
    <file>
      <url>http://ftp.drupal.org/files/projects/drupalorg_testing-6.x-1.0-alpha6.tar.gz</url>
      <mdhash>fa4e4b00955e56e7658fa84dbb26060d</mdhash>
      <filesize>32638</filesize>
      <filedate>1287450992</filedate>
    </file>
    <file>
      <url>http://ftp.drupal.org/files/projects/drupalorg_testing-6.x-1.0-alpha6.zip</url>
      <mdhash>fa4e4b00955e56e7658fa84dbb26060d</mdhash>
      <filesize>32638</filesize>
      <filedate>1287450992</filedate>
    </file>
  </files>
  <terms>
    <term><name>Release type</name><value>New features</value></term>
    <term><name>Release type</name><value>Bug fixes</value></term>
  </terms>
</release>

This doesn't really solve the problem for profiles (which could now have up to 6 files per release), but we're not doing anything a client couldn't handle just as easily. (and we're keeping the size of the release XML down as well)

Yeah, that looks great. Sorry the original proposals in some of these issues weren't particularly consistent or coherent. I hadn't really pondered them much, and was just getting the basic idea across. ;) Thanks for being careful and systematic about it now that we're actually working on the implementation.

Cheers,
-Derek

Here's a start at a patch that generates XML identical to the first sample in #10.

Here's a sample generated by the script against a somewhat stale d.o DB with multiple files per release: http://d6.project.drupal.org/release-history/drupalorg_testing/6.x (releases from http://d6.project.drupal.org/node/131901/release)

See also the sample for releases with a single file: http://d6.project.drupal.org/release-history/drupal/6.x

Do we want to leave off the <files> section for releases that only have a single file?

Excellent, thanks! I think for consistency, we should always include <files> so newer clients can always parse and use that, instead of having to special case for the 1st file.

@moshe: I don't quite follow #16. What's wrong with the XML in #10 for your needs? If you want the server to do filename heuristics to add a <file-type> or whatever to each element in the <files> array, why can't drush do the same? Can you please be more specific about your concern and/or your proposal? Thanks!

So what is this attribute called and what are the allowed values?

After a very fruitful IRC discussion with moshe, dmitrig01 and mikey_p, I now agree we should do this parsing at the server. I pointed out that the exact same logic holds for why it's useful to have all the elements of the version string split out into their own attributes instead of just saying "you've got the version string, if you want something else, parse it!". Anyway, we've converged on the following:

<archive_type>tar.gz|zip</archive_type>  # or tar.bz2 someday, whatever
<variant>full|projects|profile-only</variant>  # optional, only there for packaged distros

variant could have also been "core|no-core|?" to directly match the filenames, but:
- what do we call the "profile-only" variant since there's nothing in the filename in this case?
- these names aren't very helpful or self-documenting (I know, we should have done a better job naming the files themselves).

So, we decided to give the variants some more useful (and hopefully self-documenting) values, and we can figure out later if we want to rename the files to match.

Anyway, here's an example of what the XML should look like:

  <files>
    <file>
      <url>http://ftp.drupal.org/files/projects/drupalorg_testing-6.x-1.0-alpha6-core.tar.gz</url>
      <mdhash>96dafd4ce930e7ff3a40a5ad9cc32629</mdhash>
      <filesize>3499033</filesize>
      <filedate>1287450998</filedate>
      <archive_type>tar.gz</archive_type>
      <variant>full</variant>
    </file>
    <file>
      <url>http://ftp.drupal.org/files/projects/drupalorg_testing-6.x-1.0-alpha6-core.zip</url>
      <mdhash>96dafd4ce930e7ff3a40a5ad9cc32629</mdhash>
      <filesize>3499033</filesize>
      <filedate>1287450998</filedate>
      <archive_type>zip</archive_type>
      <variant>full</variant>
    </file>
    <file>
      <url>http://ftp.drupal.org/files/projects/drupalorg_testing-6.x-1.0-alpha6-no-core.tar.gz</url>
      <mdhash>c22f10955f6a5291e39165ade232f77c</mdhash>
      <filesize>2399783</filesize>
      <filedate>1287450995</filedate>
      <archive_type>tar.gz</archive_type>
      <variant>projects</variant>
    </file>
    <file>
      <url>http://ftp.drupal.org/files/projects/drupalorg_testing-6.x-1.0-alpha6-no-core.zip</url>
      <mdhash>c22f10955f6a5291e39165ade232f77c</mdhash>
      <filesize>2399783</filesize>
      <filedate>1287450995</filedate>
      <archive_type>zip</archive_type>
      <variant>projects</variant>
    </file>
    <file>
      <url>http://ftp.drupal.org/files/projects/drupalorg_testing-6.x-1.0-alpha6.tar.gz</url>
      <mdhash>fa4e4b00955e56e7658fa84dbb26060d</mdhash>
      <filesize>32638</filesize>
      <filedate>1287450992</filedate>
      <archive_type>tar.gz</archive_type>
      <variant>profile-only</variant>
  </file>
    <file>
      <url>http://ftp.drupal.org/files/projects/drupalorg_testing-6.x-1.0-alpha6.zip</url>
      <mdhash>fa4e4b00955e56e7658fa84dbb26060d</mdhash>
      <filesize>32638</filesize>
      <filedate>1287450992</filedate>
      <archive_type>zip</archive_type>
      <variant>profile-only</variant>
    </file>
  </files>

I'll be working on this in the next few days, so assigning to myself...

Thanks,
-Derek

p.s. It strikes me a bit weird and redundant to have attributes inside <file> that begin with "file". So maybe it should be:

  <files>
    <file>
      <url>http://ftp.drupal.org/files/projects/drupalorg_testing-6.x-1.0-alpha6-core.tar.gz</url>
      <mdhash>96dafd4ce930e7ff3a40a5ad9cc32629</mdhash>
      <size>3499033</size>
      <date>1287450998</date>
      <archive_type>tar.gz</archive_type>
      <variant>full</variant>
    </file>
    ...
  </files>

Not sure if that's going to break anything or not... ;) We'll see.

This works for me -- +1

Well, ugh. Now that I'm actually writing the code for this, the whole <variants> thing totally sucks. We have no notion of anything like that concept in project_release already. It's strictly a d.o-specific thing that's handled by the packaging script. So, not only is there no clean way to just find the variant from the DB (e.g. another column in {project_release_file}) there's not even a "clean" way to hack it by parsing the filename, since we have no generic way to know if we even want to be adding a <variant> or not. :( So, this patch would probably work (haven't tested it yet) on d.o, but it's now got some stupid d.o-specific hacks to search for a taxonomy term on the project node called "Installation profiles". :( I could hard-code the tid if that made anyone happier. ;) Then it's even more d.o-centric. :(

Do we *really* need <variant>? Can we live without? I guess we can call this script a d.o-specific thing and other sites running this script will need to customize for their needs, but I'd rather not. :(

This looks like it should do the job. I applied that patch to d6.project.drupal.org and ran a couple of files, here's some sample output from the patch in #24:

Profile with variants: http://d6.project.drupal.org/release-history/drupalorg_testing/6.x
Project without variants: http://d6.project.drupal.org/release-history/drupal/6.x

I also wanted to point out that the whole bit with the default file being $files[0] is a little fragile as well, should the packaging script ever change the order that they insert files into the DB, that could break as well. Otherwise, I'd say its as close to RTBC as I think it's going to get without refactoring the DB schema, packaging and release scripts all at the same time.

@moshe: That's a lovely idea. However, I'm moving it into a separate issue, since I don't have the bandwidth to deal with a change of that scale right now. I'm in total firefighting mode on multiple fronts, and can't afford to do everything cleanly and elegantly. So, I'm happy to proceed with #24 as-is, and then we can clean it up later over at:
#1003764: Add an alter hook invoked while generating release-history XML files

@mikey_p: Thanks for the testing and review! I'll get this in ASAP.

Thanks,
-Derek

Just for the record D6 chokes pretty badly when there is a <date> in both the <release> and the <files><file> sections.

I'm also concerned about filesize and md5hash elements having the same issues. URL isn't duplicated anywhere else.

Argh, right. I already knew that. ;) Here's a new patch making the attributes inside the new <files> section unique from the existing tag names. I think this'll work just fine on legacy clients.

I also added links to #1003764: Add an alter hook invoked while generating release-history XML files in the @todo comments about the d.o-specific hack.

Trivial re-roll based on #1006246: Improve logic for filename parsing involving .tar extensions.

Committed #30 to HEAD of project.

This is now live, and the release history XML files have all been rebuilt. For example:

http://updates.drupal.org/release-history/drupalorg_testing/6.x

Whoops, I didn't notice that this broke the ordering of the release history XML files. :( This is untested, but should do. I'll try it out now.

Except version_extra_weight should sort the same way as the other version numbers, it doesn't behave like drupal weights... Thanks, mikey_p! ;)

Committed, deployed, and re-generating all release history XML files now...