See #1152908: Remove calls to drush_backend_invoke() (Call to undefined function drush_backend_invoke() in features.drush.inc line 214). If Mark's latest patch is committed, then features is broken for drush-4. If it is not committed, then features is broken for drush-5. Module writers already need to have separate versions for each major version of Drupal; it seems severe to require them to have multiple versions of their drush commands as well. I should have argued this more at the sprint.

The new drush_invoke_process is equivalent to the old drush_invoke_sitealias_args. If We put back drush_invoke_sitealias_args as a simple function that calls through to drush_invoke_process, and mark it as deprecated, then it would be possible for a module such as features to use that function and remain compatible with both drush-4 and drush-5.

If we don't do this, then we need to add support for *.drush-5.inc for commandfile loading (and make sure that drush-5 and later ignores *.drush.inc if there is any similarly-named *.drush-INTEGER.inc available), pronto.

Comments

moshe weitzman’s picture

If we don't do this, then we need to add support for *.drush-5.inc for commandfile loading (and make sure that drush-5 and later ignores *.drush.inc if there is any similarly-named *.drush-INTEGER.inc available), pronto.

I prefer this road, than keep around legacy code from a prior version. Needs discussion.

greg.1.anderson’s picture

Title: Bring back drush_invoke_sitealias_args to help ease the transition of drush-supporting modules from drush-4 to drush-5 » Add support for *.drush-5.inc, etc., to allow module developers to provide different code for different versions of drush.

I have to admit that there might be other differences between drush-4 and drush-5 that may get to be more and more difficult to track; perhaps it is better to allow / force developers to keep multiple versions of their drush commandfiles around. Perhaps we should put support for drush-4.inc into drush-4.5 as well to keep things uniform.

msonnabaum’s picture

Very glad to see this. As I was writing the new feature patch it dawned on me that this will be a big problem. I'm fine with the separate command files, but I could also see declaring api compatibility in the command hook (I know we've talked about that before), or even adding the 4 or 5 to the callback name.

Either way, we need to come up with something. I'm also in favor of doing this as opposed to trying to maintain compatibility in code.

moshe weitzman’s picture

I'll note that it is technically possible for module developers to do this today. Have a look at http://api.drush.org/api/function/_drush_add_commandfiles/5.x and focus on the $load_test_inc stuff. A module author can ship with two commandfiles and only load one of them by also shipping with two COMMANDFILE.drush.load.inc files with simple hook_drush_load() implementations (e.g. return DRUSH_VERSION == 5). I think Aegir uses this hook.

Apart from this strategy being obtuse, I don't think module developers will be too happy about shipping with two commandfiles. The commandfiles can share common code, but we might be causing too much aggravation. Not sure what other solution we can offer.

Declaring compatibility from within the command declaration is no good. We just can't load incompatible code. If we do, we'll run into duplicate function problems once the compatible commandfile tries to load.

moshe weitzman’s picture

Copying a valuable comment by by Greg.1.Anderson from #1175446: Best practice for upgrading shared drush to next major version that belongs here:

The largest issue we need to handle is Drupal modules that implement drush support. A Drupal module already needs to support multiple versions of core (ideally, anyway); it is a burden to also have to support drush 4 + drush 5 x Drupal 6 + Drupal 7. We should be very careful about breaking backwards compatibility for the command records and the most important APIs used by drush.

However, I agree that it should be possible to support multiple versions of drush on a system, and currently it is not. If the features module wants to use drush-5 functions, it forces anyone who wants to use drush + features to upgrade to drush-5; this in turn causes problems if the same user has other modules that are still on drush-4, if drush-5 breaks backwards compatibility for them.

To make this better, we'd need to do a couple of things.

First of all, we should modify commandfile searching so that paths are filtered out if they contain the pattern /drushNNN/, where NNN is any integer that does not equal the current drush major version number. This will allow users to make drush4 and drush5 folders inside of their .drush folder, etc.; this is better than putting in a drush major version in the command record, as it gives users the ability to organise their drush commands w/out modifying the commandfile (if the author has not correctly set up version checking in their commandfiles). This same facility should be used when searching for aliases and drushrc.php files (need to add searching for drushrc.php?), so that people can divide up their aliases and options by drush verison if necessary.

Additionally, it would also be a good idea if there was a commandfile "exception list", so a user could tell drush to ignore a certain module that was broken. For example, if module x did not work with drush 5, you could copy its x.drush.inc file to $HOME/.drush/drush5/x.drush.inc, fix it there (submit a patch to the issue queue for module x), and then add x to the blacklist in your drushrc.php file for drush-4.

Currenlty, %drush-script is ignored if remote-host is missing or localhost. It should be respected when present, even for local dispatches.

We should indeed implement these sorts of forward-thinking features and backport them to drush-4, so people will have a good management strategy for moving up from drush-4 to drush-5. I still think that we need to be cautious about breaking backwards compatibility due to the large number of interdependencies and permutations between different modules and different versions of Drupal core.

Has anyone done a survey of what can break in a drush-3 to drush-4 upgrade? I tend to think these versions are fairly similar, but it would probably be a good idea to maintain a document (in drush topic, perhaps) that enumerated the API changes and main 'gotchas' for each major drush version upgrade. A bit of a pain to maintain, but in theory, once drush-MAJOR.0 is released, the list from MAJOR-1 to MAJOR incompatibilities should never change. I'll see what I can do about writing something up for the drush-3 to 4 transition; me and my friend "grep" are going to do some research. :)

greg.1.anderson’s picture

Thanks. Regarding the research on function availability in different versions of drush, this is already available at api.drush.ws; for example, see http://api.drush.ws/api/function/drush_backend_invoke/4.x. Perhaps we could do more to point users to the right function to use in the next version of drush when a function is deprecated or removed.

tim.plunkett’s picture

Subscribe.

moshe weitzman’s picture

Priority: Normal » Critical
moshe weitzman’s picture

I think the plan outlined in #5 is solid. Anyone have further thoughts on this. Anyone willing to take it on? This issue is marked critical for good reason.

greg.1.anderson’s picture

Assigned: Unassigned » greg.1.anderson

Willing, but busy for a week or two; will be a squeak to get this in for BADCamp. Wouldn't mind if someone else wants to take it.

moshe weitzman’s picture

Status: Active » Needs review
StatusFileSize
new2.49 KB

Seems like the attached patch the first paragraph of the spec in #5. Just adds some drush4, drush5, etc to the $nomask for drush_scan_directory().

When we unify aliasfile searching with commandfile searching, this should apply to aliasfiles as well.

greg.1.anderson’s picture

Status: Needs review » Needs work
StatusFileSize
new5.56 KB

Enhanced version of #11 attached. Code should be good, but needs docs -- those are coming shortly.

greg.1.anderson’s picture

Status: Needs work » Needs review
StatusFileSize
new7.59 KB

Now with docs.

New features:

drushrc.php still works as before; drush5rc.php (where 5 == DRUSH_MAJOR_VERSION) is also loaded, if it exists. The version-specific file can appear anywhere the regular config file does. This feature is useful for end users, who can separate version-specific configuration directives from their general-purpose drush options.

drush4/mycommand.drush.inc (where 4 == any number other than DRUSH_MAJOR_VERSION) is now ignored, same as in #11, above. This is useful for end-users, who can separate their version-specific drush commands into $HOME/.drush/drush4 and $HOME/.drush/drush5.

mycommand.drush5.inc (where 5 == DRUSH_MAJOR_VERSION) is now loaded. This is useful for module developers, who can provide mycommand.drush4.inc and mycommand.drush5.inc. (n.b. no previous version of drush will load either of these files, which is good; drush 4.5 would load drush5/mycommand.drush.inc.)

alias files work the same as above (drush5/mysite.aliases.drushrc.php and mysite.aliases.drush5rc.php both recognized) for consistency. This is useful for users who want to put in version-specific alias options. Maybe this will be rare.

greg.1.anderson’s picture

This adds a --ignored-modules flag so that you can skip *.drush.inc files in modules that have drush commands that are incompatible with the version of drush that you are using. Only commandfiles in drupal modules are ignored, so you can move fixed drush commandfiles to $HOME/.drush/drushN if you'd like.

greg.1.anderson’s picture

StatusFileSize
new9.81 KB

And here is the patch.

moshe weitzman’s picture

Status: Needs review » Needs work
StatusFileSize
new15.22 KB

Nice work!

Attached is same patch plus unit tests for version specific config files and commandfiles. The commandfile one is not ready yet. I'll keep hacking on it, unless someone feels inspired to work on it today.

greg.1.anderson’s picture

StatusFileSize
new0 bytes

Here's a start on the 4.x branch. I think the code is working okay (only lightly tested), but the test cases are broken -- ran out of time and have to stop here.

greg.1.anderson’s picture

Whoops, I screwed up the patch above and then did a git reset --hard HEAD. Is there a way I can find the commit hash to get my work back? :p

Edit: Nope, no commits, so git reflog is of no hope. Oh well, I'll have it back with ~15 minutes of work.

greg.1.anderson’s picture

StatusFileSize
new12.82 KB

Re-made the patch above (this time using a branch :p); also adjusted the test case so that it runs on drush-4. Didn't add any new tests, though.

moshe weitzman’s picture

@Greg - not sure how I should proceed. Is my patch in #16 the latest for drush5? I'm guessing that you didn't try to fix the commandfile test, did you?

greg.1.anderson’s picture

I didn't try your tests on drush5 -- I presumed that you were going to continue here. I did make whatever tests you had in the drush5 branch work in the drush4 branch, so perhaps this work might be helpful in master too if your tests are broken there -- I guess I misread #16 the first time. I thought that the tests you committed worked, and you had more to write, but that isn't what you said. So yeah, I guess I must have fixed the non-working commandfile test.

Want me to roll a new patch for master?

greg.1.anderson’s picture

I looked at this a little bit, and my test for drush-4 fails in master (moved from contextTest in 4.x to contextTest in master; not commandTest). Hm.

greg.1.anderson’s picture

I note on test case philosophy: I think the addition of drush unit tests is a good thing, but a unit test should never call any bootstrap code. Drush can only bootstrap Drupal (and itself) once, whereas unit tests should be executable in any order, over and over again, etc.

So, tests requiring a bootstrap should be done as functional test. Unit tests should only test functions (god, that's so confusing) with no side effects.

moshe weitzman’s picture

FYI, each UnitTest test should be executed in isolation. See 'runTestsInSeparateProcesses' in the doxygen in Drush_UnitTestCase.

I will look into the test fail on drush5. i'm not immediately sure whats different about commandfile loading between the two versions.

moshe weitzman’s picture

Status: Needs work » Patch (to be ported)

I fixed up the commandfile test and committed #16 to master. I think my commandfile test changes should be backportable to drush4.

Assigning to Greg for backport. See commit 54d1ca9aeeef91da9049f1b379aaa268fa814def

moshe weitzman’s picture

Oops. Also needs my next commit which is only 1 line: 5e7573d

Update: changed again in 539283b

moshe weitzman’s picture

Status: Patch (to be ported) » Fixed

Committed to drush4 without commandfile test since that uses a UnitTest which is not available on drush4.

moshe weitzman’s picture

Issue tags: +Needs change record

Needs change notification

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

moshe weitzman’s picture

Issue tags: -Needs change record