Make fails for library archives with url querystring values

Ugh, yeah. That's a big problem. ;) Especially since folks are trying to use this library for distributions being packaged on d.o: http://drupal.org/node/1433402 (that's basically an example release of nodestream).

I think this patch should provide a test that works if the extraction was working correctly.

Right, here's a possible fix for this. With this in place the rather naive extraction code is able to work on these 'sample' download URLs in the test.

This still seems really fragile to me, we've replaced the super amazing drush make download code with with the really basic stuff in drush core, so I still think this is going to bite someone later without warning as the make api version is still 2 even though it's clearly not always compatible.

There has been another bug with "?" in the URL:

libraries[jquery.cycle][download][url] = http://malsup.com/jquery/cycle/release/jquery.cycle.zip?v2.99

But that one seemed related to the "get" download type. Anyway, the issue is #1432526: External URL not recorded for jQuery cycle

When I build the test makefile locally (from #3), the devel directory is empty (and thus the tests are failing).

@Steven Jones:
- Thanks for jumping in. Quick skim of the patch and tests looks good, although it'd be nice to look closely at what drush_make itself was doing and see if we can benefit from any wisdom that might be in there.

- The intention is that make remains compatible, even as it moves into drush core. There are still bugs, and we're finding and resolving them. Changing the API version isn't going to make it any more stable or better. But yes, I agree that some functionality has been lost, and we're doing what we can to ensure drush core can "catch up" with how drush make used to work.

@juan_g:
- That bug had nothing to do with the ? in the URL.

Cheers,
-Derek

argh, sorry for the x-post.

@jhedstrom Odd, is there any clue as to what part of the process is failing?

@Steven Jones it's hard to say--on some builds it works, on others the directory is empty. Even with --debug on, I don't see any immediate reason. My guess is that this indicates some weird behavior with drush's internal GET cache, perhaps it behaves oddly with the url querystring values.

Another indicator of weird behavior (but may be an unrelated issue due to download type = file) is that the message is output twice for each project.

jhedstrom@hyperion:/tmp/foo$ /usr/local/bin/drush --nocolor make /home/jhedstrom/work/contributions/utilities/drush/tests/makefiles/file-extract.make --no-core -y --md5=print
Make new site in the current directory? (y/n): y
 >> admin_menu downloaded from                                                                                   [ok]
http://ftp.drupal.org/files/projects/admin_menu-6.x-1.0-beta.zip?param=value.
 >> devel downloaded from http://ftp.drupal.org/files/projects/devel-6.x-1.0.tar.gz?param=value.                 [ok]
 >> devel downloaded from http://ftp.drupal.org/files/projects/devel-6.x-1.0.tar.gz?param=value.                 [ok]
 >> admin_menu downloaded from                                                                                   [ok]
http://ftp.drupal.org/files/projects/admin_menu-6.x-1.0-beta.zip?param=value.
600fe82fba63b87c6d5de21c9e269a8a
jhedstrom@hyperion:/tmp/foo$ ls sites/all/modules/devel/
devel.css              devel.module               devel_themer.info      loader-little.gif
devel_generate.inc     devel_node_access.info     devel_themer.install   macro.info
devel_generate.info    devel_node_access.install  devel_themer.js        macro.module
devel_generate.module  devel_node_access.module   devel_themer.module    po
devel.info             devel-rtl.css              generate               README_devel_node_access.txt
devel.install          devel_themer.css           jquery-ui-drag.min.js  README.txt
devel.js               devel_themer_ie_fix.css    LICENSE.txt            themer-bar.png
jhedstrom@hyperion:/tmp/foo$ /usr/local/bin/drush --nocolor make /home/jhedstrom/work/contributions/utilities/drush/tests/makefiles/file-extract.make --no-core -y --md5=print
Make new site in the current directory? (y/n): y
 >> admin_menu downloaded from                                                                                   [ok]
http://ftp.drupal.org/files/projects/admin_menu-6.x-1.0-beta.zip?param=value.
 >> devel downloaded from http://ftp.drupal.org/files/projects/devel-6.x-1.0.tar.gz?param=value.                 [ok]
 >> devel downloaded from http://ftp.drupal.org/files/projects/devel-6.x-1.0.tar.gz?param=value.                 [ok]
 >> admin_menu downloaded from                                                                                   [ok]
http://ftp.drupal.org/files/projects/admin_menu-6.x-1.0-beta.zip?param=value.
600fe82fba63b87c6d5de21c9e269a8a
jhedstrom@hyperion:/tmp/foo$ /usr/local/bin/drush --nocolor make /home/jhedstrom/work/contributions/utilities/drush/tests/makefiles/file-extract.make --no-core -y --md5=print
Make new site in the current directory? (y/n): y
 >> devel downloaded from http://ftp.drupal.org/files/projects/devel-6.x-1.0.tar.gz?param=value.                 [ok]
 >> admin_menu downloaded from                                                                                   [ok]
http://ftp.drupal.org/files/projects/admin_menu-6.x-1.0-beta.zip?param=value.
 >> devel downloaded from http://ftp.drupal.org/files/projects/devel-6.x-1.0.tar.gz?param=value.                 [ok]
8f5a2a16d28aa0195b2baf073ebd3869
jhedstrom@hyperion:/tmp/foo$ ls sites/all/modules/devel/
devel.css              devel.module               devel_themer.info      loader-little.gif
devel_generate.inc     devel_node_access.info     devel_themer.install   macro.info
devel_generate.info    devel_node_access.install  devel_themer.js        macro.module
devel_generate.module  devel_node_access.module   devel_themer.module    po
devel.info             devel-rtl.css              generate               README_devel_node_access.txt
devel.install          devel_themer.css           jquery-ui-drag.min.js  README.txt
devel.js               devel_themer_ie_fix.css    LICENSE.txt            themer-bar.png
jhedstrom@hyperion:/tmp/foo$ rm -rf *
jhedstrom@hyperion:/tmp/foo$ /usr/local/bin/drush --nocolor make /home/jhedstrom/work/contributions/utilities/drush/tests/makefiles/file-extract.make --no-core -y --md5=print
Make new site in the current directory? (y/n): y
 >> admin_menu downloaded from                                                                                   [ok]
http://ftp.drupal.org/files/projects/admin_menu-6.x-1.0-beta.zip?param=value.
 >> devel downloaded from http://ftp.drupal.org/files/projects/devel-6.x-1.0.tar.gz?param=value.                 [ok]
48bed6e3c80cea64088e4da4a75ca9a2
jhedstrom@hyperion:/tmp/foo$ ls sites/all/modules/devel/
jhedstrom@hyperion:/tmp/foo$ 

Given that we seem to have had a 'release candidate' since this issue was raised I'm worried that Drush 5 is going to be released with this bug, so I'm bumping to critical because IMHO drush make has been utterly broken.

@jhedstrom I'll try to have a look at what's going on.

I'm no longer able to duplicate the earlier test failures, and the solution in #3 looks good to me. Committed. Thanks!

Related issue: #1447558: Make fails for library archives with no file "extension"