Regression for recursive makefiles.

Sounds like a bug, but with our download cache 'fetching ctools 52 times' should be no slower than fetching it once (unless you grab dev snapshot).

Would one solution be to allow a flag that would prevent drush from attempting to recursively build the make files if it is set?

for instance:
project[project_name][build_recurseive] = 0
?

My attempt at a patch. (This is against the 5.x branch...)

We are using Panapoly as a part of a project. We include panopoly by including it's drupal-org makefile: http://drupalcode.org/project/panopoly.git/blob_plain/e32b9a5624b767ffff...
It references panopoly_core that again references panels via a make-file:
http://drupalcode.org/project/panopoly_core.git/blob/6179a979555af746ebd...

Now, we've just developed a patch we would like to have applied when panels is pulled in. But, if we specify the patch in our top-level makefile, the project is pulled down, patched, and then completely overwritten when drush reaches the panopoly_core makefile. This also happens if we copy the entire definition of the panopoly_core's dependency on panels (including its patches).

A nice solution would be if it where possible to define an "amendment" to a project in a parent makefile saying "should you ever encounter a dependency on panels, please make sure to apply this patch"

We could attempt to apply the patches after the build has run. But, as we're using Aegir in our development process, and it is pretty important for us that we can fire off a single drush make command and have the entire project build without having to go back in to patch things after the build.

Changing status, as there is a patch.

as patch author, I agree, this patch doesn't solve the pretty major regression that needs to be fixed.

#7
I've worked this through and this is what I found out.
Drush make traverses through the top level make file Bottom to top. So you should put your overwrites at the top.
It also has a concurrency set to 4 so your overwrites must be at least 4 modules away in your make file. So you need something like this.

projects['my_overrides'] = 3.5
projects['module1'] = 1.0
projects['module2'] = 1.0
projects['module3'] = 1.0
projects['panopoly_core'] = 1.1

Another idea?

#11sounds like a pretty big gotcha. needs documentation, if it can't be improved in code.

#7 @danquah
This one can be solved by including a patch for the panopoly make file in your master repo that removes the project from the make file and having the panopoly_core project in your master repo as you want it.

I encountered a different problem. Patches from recursive make file don't get applied... So only patches in the top level patch file get included. This means if you do something like have a drupal-org.make with your patches and a build.make which downloads core and the profile, it then finds and runs the drupal-org.make file, but doesn't apply any patches contained...

#14 @andrewbelcher

this works for me... unless it is a very recent regression

I encountered a different problem. Patches from recursive make file don't get applied... So only patches in the top level patch file get included. This means if you do something like have a drupal-org.make with your patches and a build.make which downloads core and the profile, it then finds and runs the drupal-org.make file, but doesn't apply any patches contained...

I guess you can infer from #11 that what happens is that the make files parsing+downloading+patching might happen in parallel and the last one prevails...

We hit this bug while developing OpenAtrium 2 which is based on the Panopoly submodules.
From OpenAtrium's makefile it is impossible to override or patch a dependency in panopoly_core's makefile.
The third-party dependency is downloaded and patched by us but then downloaded again and simply overwritten by panopoly_core's makefile.
This is a major problem for distributions but also large projects like #7, therefore marking major.
A build_recursive flag that allows to switch recursion off wouldn't fix the real problem, but at least would work as a stop-gap.

@pancho, working on a distro of my own, I figured it best to patch panopoly_core's makefile. See https://github.com/barraponto/aeconsulta/blob/master/aeconsulta.make#L17 and https://github.com/barraponto/aeconsulta/blob/master/patches/field_group...

@andrewbelcher we had the patching problem for a while. Upgraded drush to 5.9, and that particular problem went away.

But now we seem to sometimes - and randomly - get the wrong version of a module. I think this is owing to the concurrency issue that @redndahead mentions: but the module (views) is separated from the features with makefiles that download incorrect versions by quite a number of lines. What could be happening is that some of the concurrent threads are just tied up downloading the feature's makefile projects, so sometimes they just take much longer than the main makefile.

Regression is definitely important, but having a (command-line?) flag to disable recursion would from the sounds of it be really useful too!

If you review the help for make command, you will see --concurrency option. Set that to 1. Do we have any recursion issue here that still happens with --concurrency=1?

Yes, I saw that option. But is there not a performance hit with --concurrency set to 1? Otherwise, why not remove it from drush altogether?

It seems weird to require people to turn off concurrency because of e.g. recursion, yet not give them the option to turn off recursion because of e.g. concurrency.

@moshe:
Unfortunately, I'm still having the issue here with both drush 5.9 and --concurrency=1.
I'm gonna make sure others at OpenAtrium can reproduce this issue and will then turn back.

[edit:]
To make it clear: we still have the situation that one git checkout replaces the other of the exactly same revision. The other aspects work fine with --concurrency=1.

For those who want to switch off recursion altogether, I've put a proposal on #1989174: Prevent makefile recursion through per-project properties or a command-line kill-switch that rolls @joelcollinsdc's code in with a command-line option and tests. That allows this ticket to concentrate on the regression.

If you look at my post on #11 you can get past this by putting your own features in between panopoly and the feature that overrides the modules in there. You can see this make file for an example. https://github.com/gsbitse/gsb_public/blob/master/gsb_public.make

In my case I create an overrides module that all it does is have a make file that I use to override any modules I need. You can see that here: https://github.com/gsbitse/gsb_make_overrides/blob/master/gsb_make_overr...

The key is to make sure there are 4 modules in between the panopoly line in the make file and your overrides module. This is usually easy to do. Works everytime.

This issue was marked closed (won't fix) because Drush has moved to Github.

If desired, you may copy this bug to our Github project and then post a link here to the new issue. Please also change the status of this issue to closed (duplicate).

Please ask support questions on Drupal Answers.