Summary for drush 8 users with global drush phar

This issue is linked from the Drupal release note for users using drush 8 phar plus "pm-uprade". Hence adding this summary to help people the current status without having to read the whole history.

There are references in the issue comments that this scenario is "fragile" and "happens to work". These refer to the risk of problem in future Drupal minor releases D8.5+. This risk is being managed in #2906637: [META] Drush and core compatibility is fragile. The intention is that "drush pm-upgrade" will continue to work until a suitable and equally easy alternative is available. However this will require a lot of work, so please contribute if you can.

Problem/Motivation

After #2712647: Update Symfony components to ~3.2...
On Drupal 8.4.x with drush master you can install. On Drupal 8.3.x with drush 8.x you can install.
On Drupal 8.4.x with drush 8.x you get:

PHP Fatal error:  Declaration of Drush\Command\DrushInputAdapter::hasParameterOption($values) must be compatible with Symfony\Component\Console\Input\InputInterface::hasParameterOption($values, $onlyParams = false) in /Volumes/devdisk/dev/drush5/lib/Drush/Command/DrushInputAdapter.php on line 27

On Drupal 8.3.x with drush master you get:

PHP Fatal error:  Class Symfony\Component\Console\Style\SymfonyStyle contains 4 abstract methods and must therefore be declared abstract or implement the remaining methods (Symfony\Component\Console\Output\OutputInterface::isQuiet, Symfony\Component\Console\Output\OutputInterface::isVerbose, Symfony\Component\Console\Output\OutputInterface::isVeryVerbose, ...) in /Volumes/devdisk/dev/sites/drupal8alt.dev/vendor/symfony/console/Style/SymfonyStyle.php on line 33

Current status

Update: Drush 8.1.15 works with Drupal 8.3.x and Drupal 8.4.x whether drush is installed globally or site-locally.

Drush 8.1.12 site-local Drush + Drupal 8.3.x: Supported
Drush 8.1.12 site-local Drush + Drupal 8.4.x: Does not work (See https://github.com/drush-ops/drush/pull/2800)
Drush 8.1.12 phar + Drupal 8.3.x: Supported **
Drush 8.1.12 phar + Drupal 8.4.x: Works (Not officially supported) **

Drush 9.x site-local Drush + Drupal 8.3.x: Supported
Drush 9.x site-local Drush + Drupal 8.4.x: Supported
Drush 9.x global install + Drupal 8.3.x: Possible (See workaround below) **
Drush 9.x global install + Drupal 8.4.x: Works (Not officially supported, cgr install recommended) **

** These configurations run with two autoloaders: one from Drush, and one from Drupal. This is a fragile configuration that can be broken over time.

Workaround to create a global install of Drush 9 that works with Drupal 8.3.x and Drupal 8.4.x (not officially supported):

$ mkdir my-global-drush
$ cd my-global-drush
$ composer init
$ composer config prefer-stable 1
$ composer config minimum-stability dev
$ composer require symfony/console:~2.7 drush/drush:~9

See also: https://github.com/drush-ops/drush/pull/2799

Proposed resolution

Those who wish to use `drush pm-update` & c. can continue to use a global install of Drush 8.1.12 or later. This will work for the Drupal 8.3.x to 8.4.0 update. This configuration is not officially supported for Drush 8.4.x and later, any may stop working, depending on changes made to the dependencies in Drupal.

Those who are using a site-local Drush should upgrade to Drush 9 before Drupal 8.4.0 is released. Composer will indicate conflicts between Symfony dependencies if this is tried; the solution is to change the constraint on the project's Drush version from `^8` to `^9`.

**Update:** You can also install site-local Drush 8.1.15+ with Drupal 8.4.x.

Remaining tasks

Plan for universal site-local Drush + Composer use prior to upgrading to Symfony 4, as the global Drush will not support this.

User interface changes

API changes

Data model changes

CommentFileSizeAuthor
#9 correct_version.png20.27 KBxjm
#8 Status_report___admin.png94.95 KBxjm
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

alexpott created an issue. See original summary.

alexpott’s picture

So this was caused by https://github.com/drush-ops/drush/issues/2724 if you are on 9.0.0-beta1 Drush master can install both 8.3.x and 8.4.x

moshe weitzman’s picture

I think that Drush Master should work with Drupal 8.3. I'm perplexed by the Symfony Style error you posted above. That looks to me like Symfony is not compatible with itself?

alexpott’s picture

@moshe weitzman - the drush install I'm testing against is a global install of Drush not one done via composer require in your Drupal root. Drush 9.0.0-beta1 does work against 8.3.x.

I guess 8.3.x's Symfony and Drush's symfony clash.

moshe weitzman’s picture

Global install of Drush is no longer supported because you get errors exactly like the one described here. The install docs already don't mention it anymore. Each project needs to name Drush as a dependency. Its fine if Drush8 sites stick with the global install and Drupal 8.3-. For Drupal 8.4, they are going to need to use Drush9 as a dependency.

catch’s picture

@moshe we could still revert https://github.com/drush-ops/drush/issues/2724 until 8.4.0 is released, so that composer require drush/drush ~9 works on both 8.3.x and 8.4.x in the meantime.

xjm’s picture

Presumably, when we tag 8.4.0, about 150,000 sites will try to run drush up from Drupal 8.3.whatever. What happens to them?

Some significant fraction of those sites are probably using drush but not using composer. What happens to them?

xjm’s picture

Issue summary: View changes
FileSize
94.95 KB

Well, I mean, we don't need to wonder. I just installed 8.3.2 from the tarball, using my existing clone of Drush pinned to 8.1.10, and then did:
drush up 8.4.x-dev

It successfully downloaded the updated codebase, then blew up trying to run database updates:

Fatal error: Declaration of Drush\Command\DrushInputAdapter::hasParameterOption($values) must be compatible with Symfony\Component\Console\Input\InputInterface::hasParameterOption($values, $onlyParams = false) in /Users/xjm/git/drush/lib/Drush/Command/DrushInputAdapter.php on line 27

(and etc...)

Error: Call to undefined method Symfony\Component\HttpKernel\Event\PostResponseEvent::setDispatcher() in Drupal\Component\EventDispatcher\ContainerAwareEventDispatcher->dispatch() (line 94 of /Applications/MAMP/htdocs/drupal-8.3.2/core/lib/Drupal/Component/EventDispatcher/ContainerAwareEventDispatcher.php).
Drush command terminated abnormally due to an unrecoverable error. 

Which also means that if I did not see the barf from Drush, I might think my site was updated normally, but database updates did not run:

So, ta-da, data integrity problems using the mechanism for updating core that I and everyone else have used for more than a decade.

xjm’s picture

Issue summary: View changes
FileSize
20.27 KB

Just reiterating that I used drush up like I always have and my site appeared to be updated:

But database updates did not run:

moshe weitzman’s picture

Thats unfortunate but nothing is really broken IMO. Since the beginning of time, you have to watch for output when you run a CLI command. In this case, your CLI command warned you that it did not complete. As you noted Drupal's status report and Drush's equivalent (core-requirements) will tell you that DB updates are pending.

An update to Drush9 is mandatory when updating to Drupal 8.4. Once you have updated both, you can run updatedb to clear the pending updates. At this time, you can still use a global drush and not use a Composer managed project. A global Drush is now barely supported and is no longer mentioned in the install docs.

A dependency manager like Composer exists to prevent this very problem. Sites which neglect to use one have to manually care for their dependencies. C'est la vie.

moshe weitzman’s picture

I'd be thankful if we could mention the mandatory Drush upgrade on the release node and other communications. I'm happy to write a blurb for it.

cilefen’s picture

Issue tags: +8.4.0 release notes

If this can't be fixed as such, should we go ahead and close it?

cilefen’s picture

Presumably, when we tag 8.4.0, about 150,000 sites will try to run drush up from Drupal 8.3.whatever. What happens to them?

But drush up is in fact broken when trying to upgrade to 8.4.x. Are there mitigations? Could drush be patched to not allow that action? Plus which, as mentioned there is a never composer contingent.

moshe weitzman’s picture

Yes, I think drush 8 could be patched if someone cares to submit it. It would be nasty Drupal-8.4-specific code, but oh well. We'd want to merge and release a new Drush8 soon, since this only benefits folks who actually install the new Drush8 release before upgrading to Drupal 8.4.

cilefen’s picture

Being 8.4 is in some months, that could be more than a few. https://github.com/drush-ops/drush/issues/2777

xjm’s picture

#15 makes me sad but definitely seems like a needed measure until we see if we can fix it in a more user-friendly way.

@moshe weitzman:

I'd be thankful if we could mention the mandatory Drush upgrade on the release node and other communications. I'm happy to write a blurb for it.

How does requiring a Drush 9 upgrade help sites that already installed from tarball and are not managed from composer?

moshe weitzman’s picture

Whats your point? What would you like to see happen?

FWIW, the thing that kicked off this issue is that Drupal decided to rely on a new major version of Symfony during its minor release. There is no way to do that without disrupting the Drupal's tools ecosystem. I'm not complaining about it, but getting a bit weary of "defending" Drush.

xjm’s picture

My concern is that Drupal needs to continue to support tarball downloads for the duration of the Drupal 8 cycle at least, but the sudden rapid changes to Drush without BC are disrupting existing sites' workflows, and Drush now explicitly states that it doesn't support the primary supported way for installing Drupal, from a tarball, without composer.

I wanted to throw my laptop through the window when I had to update Drush to support Drupal 8.3.0, tried to update to master and had everything break dramatically, went back to 8.1.10, but then still had all of my scripts breaking because I had to change --dburl to --db-url and etc. All of that just so I could try to reproduce a critical update bug that we thought (and confirmed) was being caused by an intermediate change in Drush.

But my personal frustration is not the real concern for me; the real concern for me is that Drush is dropping support for existing sites. If we do go with #15, there will inevitably be sites who only update when relevant security releases for them come out (which is exactly what I did when maintaining sites) but they will be totally screwed when updating from like 8.3.1 to 8.4.3 or whatever, because suddenly their normal maintenance scripts break and then they have to deploy an untested drush update or update their sites by hand, which no one does or even knows how to do correctly because drush has been the best practice for 10 years.

xjm’s picture

FWIW, the thing that kicked off this issue is that Drupal decided to rely on a new major version of Symfony during its minor release. There is no way to do that without disrupting the Drupal's tools ecosystem.

Yeah, we held off as long as we could, but Symfony 2.8 support is ending in Nov. 2018 well before we will be ready to release Drupal 9. So we need to update to Symfony 3 by either 8.4.0 (Oct. 2017) or 8.5.0 (March 2018).

greg.1.anderson’s picture

Supporting Symfony 2.x and 3.x simultaneously in Drush will be painful -- perhaps even impossible -- but I'll give it a shot. Hopefully Drupal will not update to Symfony 4 prior to the release of Drupal 9, as I don't think we'll be supporting three versions of Symfony in Drush 8. It should also be everyone's expectation that Composer should be the only way to update a site to Drupal 9. I don't see the pm-* code being updated to allow this.

I had to change --dburl to --db-url and etc.

Drush has been using --db-url since Drush before Drush 5.x. Are there issues in the Drush queue for the b/c breaks & the above-mentioned critical update bug?

greg.1.anderson’s picture

Here's the a-few-minutes-older-me realizing that this is a Very Hard Problem for a global Drush. I made some comments about why in the Drush queue. We've been down this road before in Drush-land, but for a while I was thinking only about the site-local Drush, which of course is not helpful for the pm-* code, which requires a global Drush.

xjm’s picture

Drupal 8.4.0 is scheduled for October 4, and after that Drupal 8.3.x versions are, officially speaking, not supported. There is a month "grace period" where we do not disclose security issues affecting 8.3.x (since contrib and custom code needs time to make small updates), so in a way 8.3.x has passive security support until the next security window on November 15.

In an ideal world, we would provide a version of Drush that supported upgrading from security releases like Drupal 8.3.1 or 8.2.7 or 8.1.7 until at least the first 8.4.x security release. Probably until 8.5.0 would be best to protect the stragglers who haven't had a sec vulnerability affect them yet. Looking at the breakdown of how many sites are still on what D8 releases (summarizing the releases that have a significant number of sites still reporting):

  • 1.7k are on 8.1.7 (httpoxy) and 1.9k are on 8.1.8 (bugfix release following it that fixed it).
  • 2K sites are on 8.1.10 (SA-CORE-2016-004, critical vulns without published mitigations
  • Fewer than 1K sites are on any other 8.1.x version.
  • These many sites are still on 8.2.x releases:
    8.2.8	4,233
    8.2.7	7,809
    8.2.6	7,779
    8.2.5	5,073
    8.2.4	3,257
    8.2.3	3,368
    8.2.2	1,161
    8.2.1	1,987
  • 4K sites are on 8.3.0
  • 14K sites are on 8.3.1
  • 46K sites are on 8.3.2

So I think supporting Symfony 2.8 in Drush only until Drupal 8.5.0 and then dropping support for it entirely would meet the needs of the majority of sites.

To avoid supporting Symfony 2.8 forever, we would announce in the 8.4.0 communications that:

  • An update to Drush 8.1.11 (or whatever) is needed to update from 8.3.x to 8.4.x
  • An update to Drush 9.0.0 is recommended for sites managed with composer
  • A future version of Drush, 8.2.0, will be released after Drupal 8.5.0, so sites must update to that later version after 8.5.0 or Drush will no longer support updating. (And then do what cilefen's PR did, just for significantly older versions).
xjm’s picture

Drush has been using --db-url since Drush before Drush 5.x. Are there issues in the Drush queue for the b/c breaks & the above-mentioned critical update bug?

For the critical, yes, it was fixed in Drush 8.1.10 (don't have the PR handy) and we added a composer requirement for 8.1.10 for 8.3.0+ sites (but that doesn't help sites not using composer).

For the BC breaks mid-major release, I just read the docs and changed them, and was annoyed because my version of Drush was not older than whenever I installed it in January, I've used these scripts since before 8.0.0 with the (apparently deprecated) versions of many different flags for many commands, and the BC break happened in a patch release of Drush 8.x. So my fault that my scripts were old I guess, but it was painful to have them break in whack-a-mole ways when I was responding to criticals with 8.3.0.

greg.1.anderson’s picture

In an ideal world, we would provide a version of Drush that supported upgrading from security releases like Drupal 8.3.1 or 8.2.7 or 8.1.7 until at least the first 8.4.x security release.

In order to do this, we need to support Symfony 2.x and 3.x at the same time in Drush. This is Hard. No promises here. In an ideal world, we get this to work.

A future version of Drush, 8.2.0, will be released after Drupal 8.5.0, so sites must update to that later version after 8.5.0 or Drush will no longer support updating.

We can't assume that the Drush pm-* code will continue to work forever. That code is under passive maintenance only. It shouldn't be assumed that there will be any upgrade path other than Composer to any version of Drupal containing Symfony 4.

(And then do what cilefen's PR did, just for significantly older versions).

We can't change old versions of Drush; we can only change the latest Drush on the 8.x branch. Once we know in advance that future versions of Drupal will not be updatable by Drush, then we can make the change, but folks are going to have to keep updating on the Drush 8.x line or be caught unawares.

it was painful to have them break in whack-a-mole ways when I was responding to criticals

Noted. We try to maintain backwards compatibility for Drush within major version releases. I can't take any action here without more details though. I don't specifically recall anyone removing obsolete stuff mid-8.x. We typically only do that on master.

catch’s picture

We discussed this a bit on https://www.drupal.org/node/2874827

My understanding is drush with symfony 2 should work with 8.4.x, but drush with symfony 3 won't work with 8.3.x, so my suggestion would be to revert that PR and update to Symfony 3 when 8.4.0 is out.

There might be something broke that way 'round but that's where I got to testing different combinations several times.

greg.1.anderson’s picture

My understanding is drush with symfony 2 should work with 8.4.x

I'm not sure if this is true.

drush with symfony 3 won't work with 8.3.x

I'm pretty sure this is true.

suggestion would be to revert that PR

I'm in favor of that...

and update to Symfony 3 when 8.4.0 is out.

... but it only buys us some time; it's not a solution in and of itself.

I've started a proof-of-concept to attempt to support Symfony 2 and Symfony 3 simultaneously in Drush 8.x here: https://github.com/drush-ops/drush/pull/2787. It's not working yet. Anyone who loves the global Drush is encouraged to collaborate.

catch’s picture

#26: it's not necessarily going to be 100% true that drush with symfony 2 is fine with core on symfony 3, but it worked well in my testing and it's definitely better than vice versa.

... but it only buys us some time; it's not a solution in and of itself.

Not sure I agree:

1. Run drush as it is
2. Upgrade site to 8.4.0
3. Upgrade drush to latest version.

This allows you to update one thing at a time. You'll only run into issues if you have 8.3.x and 8.4.x sites on the same server, but since we drop support for minor versions when the new one is out, that's not really supported.

greg.1.anderson’s picture

3. Upgrade drush to latest version.

By latest version, do you mean Drush 9? I think it's probably possible to have a stable release of Drush 9 available by October. If this strategy also means that the official policy of Drupal 8.4.x is that all sites must be managed by Composer (n.b. Drush 9 does not contain the pm-* commands), then I think this is a viable plan. Note also that you won't necessarily have to run into problems with Drupal 8.3.x and Drupal 8.4.x on the same server, as each copy of Drupal has its own copy of Drush. You could even make Drush 8.x your global Drush if you didn't want to put a site-local Drush in your 8.3.x sites.

If this is what you mean, then I can firmly get behind this plan. If you mean something else, well, it might still work, but I'm not yet sure.

greg.1.anderson’s picture

I should clarify that in #28, I am also assuming that the upgrade to Symfony 3 is postponed until 8.5.x. It seems to me that it would be a bad idea to do a major Symfony upgrade in a patch release.

catch’s picture

3. Upgrade drush to latest version.
3. Upgrade drush to a tagged release using Symfony 3

Does that clarify?

I am also assuming that the upgrade to Symfony 3 is postponed until 8.5.x. It seems to me that it would be a bad idea to do a major Symfony upgrade in a patch release.

8.3.x has Symfony 2, 8.4.x has Symfony 3, no patch release involved so I'm not sure I understand?

xjm’s picture

Regarding:

If this strategy also means that the official policy of Drupal 8.4.x is that all sites must be managed by Composer

This is most definitely not going to be the case. Per the DA, 60% of Drupal core installations are from tarball, and therefore not managed by Composer. We can't and won't drop support for the primary supported way to install Drupal in the middle of a major release. Support for tarball installations (and their upgrade paths) will need to continue at least as long as Drupal 8 is supported.

catch’s picture

Going to type out the steps in #27 more explicitly:

1. drush master reverts to Symfony 2.* for now, since Symfony 2* code is compatible with Symfony 3 (in 8.4.x), whereas the reverse is not the case.
2. Drupal 8.4.0 is released
3. Since neither core nor drush needs to support older minor versions of core, drush master can then update to Symfony 3*
4. Sites will need to update to Drupal 8.4.* prior to updating to whichever drush release has Symfony 3 in it.

The advantages of this:

- both core and drush can run Symfony 3* as soon as core as a tagged release for 8.4.x

- site owners only need to update one thing at a time, there's no need to sort out a local drush or similar as a condition of updating to 8.4.x

- while 8.4.x is in development, a global drush install can be used for both 8.3.x and 8.4.x.

greg.1.anderson’s picture

First, let's clarify a point on the title of the issue so that we are clear on the scope of what we are discussing here:

Drush 8.x doesn't install Drupal 8.4.x and Drush master doesn't install Drupal 8.3.x

It is true that Drush 8.x does not install any version of Drupal that includes Symfony 3. This is by design, and will remain this way unless https://github.com/drush-ops/drush/pull/2787 moves forward. This is still up in the air.

Drush master does install Drupal 8.4.x and Drupal 8.3.x. This is proved by the test suite, which tests both branches. (The 8.4.x tests are failing under php 7 right now, but that's a separate issue.)

We can't really "revert" Drush master to Symfony 2.x. The version constraints for the Symfony component in Drush look like:

"symfony/console": "~2.7|^3",

Note that we cannot remove the `|^3` without breaking Drush with Drupal 8.4.x. The only supported way to use Drush 9 is via a site-local Drush, with one Drush per Drupal, included in Drupal's composer.json (or preferably in the project composer.json using drupal-composer/drupal-project. The global installation method is unofficially supported, but only for latest-to-latest (Drush master with Drupal 8.4.x currently.)

#32 is advocating support for the global installation method in Drush 9 across multiple versions of Drupal. I have not recently experimented with which combinations of this does and does not work; however, using Composer like this is generally unmaintainable, as explained by some of the posts linked to in https://github.com/drush-ops/drush/pull/2787. If anyone wanted to try this unsupported configuration, start with:

$ mkdir my-global-drush
$ cd my-global-drush
$ composer require symfony/console:~2.7 drush/drush

Then add my-global-drush/vendor/bin to your $PATH and see how far that gets you. I haven't tried it; I have my doubts, as explained above.

#31 is my primary concern here. If we accept the current Drush contraints as given, then there is no version of Drush that will upgrade from Drupal 8.3.x to Drupal 8.4.x via the pm-* methods. Note that once you are on Drush master, pm-* is gone, so we must have 100% of the Drupal users on Composer by the point in time that Drush 9 becomes the only supported version. Failing that, we need a maintainer for the pm-* code (perhaps as a Drush 9 contrib extension), or maybe a separate pm-updatecode tool that runs drush updatedb for database updates.

xjm’s picture

The only supported way to use Drush 9 is via a site-local Drush, with one Drush per Drupal, included in Drupal's composer.json (or preferably in the project composer.json using drupal-composer/drupal-project.

Yes, I've read this above. This means that Drush has decided to drop support for 60% of Drupal 8 sites.

moshe weitzman’s picture

This means that Drush has decided to drop support for 60% of Drupal 8 sites.

I tried to let this go without comment but I can't.

Drush did so in a new major version per semver conventions. This was announced all the way back in October 2016. The plan stated there is for Drush8 to continue serving these tarball folks (and drush 6 and drush7 users) for a long time. It was Drupal core that upgraded to Symfony3 during a minor release. This broke Drush8, and any other non-trivial script that works with Drupal. The stated reason is "We had to upgrade before Nov 2018." Er, thats a long time from now. You still have 18 months to train up folks on Composer, or release a new major version (Drupal 9), or ask Symfony to extend even further. Or you can do nothing - thats fine too. But lets not throw shade at Drush.

greg.1.anderson’s picture

Today I backported #2724 to Drush 8.x, and Moshe released Drush 8.1.12. This clears up the problem with Drush 8.x and Drupal 8.4.x.

Summary of the current state of the site-install command:

Drush 8.1.12 phar + Drupal 8.3.x: Works
Drush 8.1.12 phar + Drupal 8.4.x: Works
Drush 9.x site-local Drush + Drupal 8.3.x: Works
Drush 9.x site-local Drush + Drupal 8.4.x: Works

There is no Drush 9 phar, and the global installation option for Drush 9 is minimally supported, as mentioned above. To create a global install of Drush 9:

$ mkdir my-global-drush
$ cd my-global-drush
$ composer init
$ composer config prefer-stable 1
$ composer config minimum-stability dev
$ composer require symfony/console:~2.7 drush/drush

If you do this, and put vendor/bin in your $PATH, then the global Drush 9 install will also work with both Drupal 8.3.x and Drupal 8.4.x -- at least for the time being.

As the non-working permutations (e.g. global install of Drupal 9 out-of-the-box with Drupal 8.3.x) are not supported, I suggest that this issue be closed / fixed (Drush 8.3.x) or closed / works as designed (Drush master).

xjm’s picture

@moshe weitzman:

Er, thats a long time from now. You still have 18 months to train up folks on Composer, or release a new major version (Drupal 9), or ask Symfony to extend even further. Or you can do nothing - thats fine too. But lets not throw shade at Drush.

I don't mean to throw shade at Drush, although I am really concerned about the stated policy of no longer supporting the sites that need Drush most. This is a shared problem for Drush and core.

Having full Drush support for the Symfony 3 update and solving all the problems was a prerequisite for having Symfony 3 committed to Drupal. I was told by the framework managers that all the Drush problems were solved. I don't know how they got unsolved (catch implied it was a later commit that broke it), or if updating from Drupal 8.3.x to 8.4.x was never tested, or if the connection wasn't made that requiring Drush 9 wouldn't fly because Drush 9 essentially doesn't support most Drupal 8 sites.

And, unfortunately, it's not simply a matter of "training people on Composer". There is deep concern about composer as the primary means of installing Drupal from many audiences, strong enough that people pick other solutions instead of Drupal. Composer is the bee's knees -- for PHP devs and devops people like you and me who can support it -- but for a significant chunk of our current market, it's actually a barrier. Not because they're not "trained". This is offtopic here, but it is a big risk for Drupal right now and finding a way to support both audiences is a problem that's going to take a lot of resources to solve.

18 months is not a long time in a six-month release cycle. As I said, we need to have Symfony 3 in this minor, or the next. The notion of releasing Drupal 9 next year is quite far-fetched when 90% of Drupal sites are still on Drupal 7 and the migration path isn't even done yet.

xjm’s picture

@greg.1.anderson, that's great news! I'll try to test it later today.

So is the plan that Drush 8 will continue to be fully supported for as long as Drupal 8 is supported?

greg.1.anderson’s picture

I was told by the framework managers that all the Drush problems were solved.

Drush 8 has never officially supported Symfony 3. The current status of Drush 8 and Drupal 8.4.x is "happens to work". With the Symfony folks working on Symfony 4, it is likely that it will continue to work. However, we need to do more testing with contrib modules. The decision to formally support Symfony 3 with the global Drush (cross-linking Symfony 2 and Symfony 3) has not been made.

As long as you are using a system with two autoloaders, you can never claim that the Drush problems are "solved". In a two-autoloader environment, any patch release of any component that appears in both autoloaders can cause a catastrophic failure. See The Trouble with Two Autoloaders. Note that changes to non-Symfony components can also potentially cause problems.

There are three ways to avoid two autoloaders:

1. Use a site-local Drush (current strategy for both Drush 9 and Drupal Console)
2. Put a cli in core
3. Postpone loading the autoloader until the Drupal site is identified (https://github.com/drush-ops/drush/pull/2787; does not work yet)

The last option is the most difficult. Drush 5 had no autoloader, but obviously does not support Drupal 8. Drush 8 only uses a few classes during the early bootstrap, and isn't under active development. Drush 9, on the other hand, uses quite a few more external libraries, and uses them from the earliest stages. It is therefore much riskier to use the global install method with Drush 9 (currently happens to work, not supported), and it would be a ton of work to do something like #3 in Drush master or Drupal Console.

greg.1.anderson’s picture

Issue summary: View changes

Updated issue summary with current status of which configurations are supported, and which happen to work.

Chi’s picture

As long as you are using a system with two autoloaders, you can never claim that the Drush problems are "solved".

What if instead of two auto-loaders we use two separate Drush instances managed by a single launcher? I've just made a very quick experiment and it seems to work pretty well. I've managed to run successfully drush si on Drupal 7.52, Drupal 8.3.2 and Drupal 8.4.0-dev. Theoretically it should work even for Drupal 6 installations.

greg.1.anderson’s picture

@chi Your idea is not without merit; however, note that in your experimental project above, Drush is running with two autoloaders.

The more typical solution along these lines is to install a site-local Drush with every Drupal site, and then use either a global Drush or a global shim or launcher to select the specific version of Drush you wish to use. This corresponds to option 1 in #39.

Chi’s picture

@greg.1.anderson, I agree local installation is a preferable way to install Drush on Drupal 8 projects. But we still have an issue with supporting sites managed without Composer by some reason.

greg.1.anderson’s picture

@chi True, that. However, long-term viable solutions to that problem should avoid two autoloaders per #39. We already have a short-term 'happens to work' solution with Drush 8.1.12. #41 does not improve on this.

Chi’s picture

@greg.1.anderson, would this help to relieve Drush from having to support two major Symfony versions within a single release?

greg.1.anderson’s picture

@chi The primary supported installation method for Drush is as a dependency of a Drupal site. Drush master supports Drush 8.3.x and earlier, and Drush 8.4.x and later, so removing the `~2.7|` from the Symfony components is a non-starter. While we could potentially consider changing the rules of what is supported and what is not supported, note that supporting two versions of Symfony (at least Symfony 2 + Symfony 3, which are relatively similar) as part of a site-local command is only mildly inconvenient, and completely safe, as there is only one autoloader. In contrast, solutions such as #41 that run Drush + Drupal with two autoloaders are not safe, as they can potentially break any time dependencies are updated in either component. Breakage can even happen with dependencies that have a patch release that preserves backwards compatibility of its public API. For this reason, #41 cannot be considered a significant improvement over the current situation.

Using a shim or launcher in a configuration that gets us down to a single autoloader could be a potential solution, though.

greg.1.anderson’s picture

Issue summary: View changes

I updated the issue summery to clarify unresolved questions vis-a-vis supported configurations moving forward.

xjm’s picture

Thanks @greg.1.anderson and @Chi for your help with this.

greg.1.anderson’s picture

Issue summary: View changes

Update issue summary with current status, proposed resolution and remaining tasks.

bojanz’s picture

And, unfortunately, it's not simply a matter of "training people on Composer". There is deep concern about composer as the primary means of installing Drupal from many audiences, strong enough that people pick other solutions instead of Drupal.

But not the Drush audience, which is already using a command line tool to manage their site. I believe that has guided the Drush 9 decisions.

Or are we claiming that there's a significant portion of CLI users that refuse to use one specific CLI (Composer)?

ressa’s picture

I am part of the Drush audience, I use Drush all the time and love it -- it allows me to be very productive and perform tasks in the blink of an eye, or automate tedious tasks.

But I am not very comfortable with using Composer, as a site builder with modest programming experience. Comparing Drush to Composer is like comparing apples to oranges, they have very different difficulty levels. This enlightening article by Jeff Geerling describes very well how Composer is not as easy to use as Drush:

Composer and Drupal are still strange bedfellows

ressa’s picture

I can confirm that with the latest phar (non-Composer) version of Drush (8.1.12) you can drush dl Drupal 8.4.x. Thank you so much @greg.1.anderson!

Install Drush

$ wget http://files.drush.org/drush.phar;
$ php drush.phar core-status;
$ chmod +x drush.phar;
$ sudo mv drush.phar /usr/local/bin/drush;
$ drush init;

Download Drupal

$ drush dl --select drupal;

Result

Choose one of the available releases for drupal:
 [0]  :  Cancel                                               
 [1]  :  8.4.x-dev  -  2017-Jun-20  -  Development            
 [2]  :  8.3.3      -  2017-Jun-07  -  Supported, Recommended 
 [3]  :  8.3.1      -  2017-Apr-19  -  Security
$ drush status | grep version

Result

Drupal version: 8.4.0-dev
Drush version: 8.1.12
xjm’s picture

Status: Active » Fixed

With Drush 8.1.12 I was able to:

  1. drush dl drupal to download drupal 8.3.5.
  2. Install the site with drush si and administer it.
  3. drush up drupal-8.4.x-dev
  4. But then I still got:
    Error: Call to undefined method                                      [error]
    Symfony\Component\HttpKernel\Event\PostResponseEvent::setDispatcher()
    in
    Drupal\Component\EventDispatcher\ContainerAwareEventDispatcher->dispatch()
    (line 94 of
    /Applications/MAMP/htdocs/drupal-8.3.5/core/lib/Drupal/Component/EventDispatcher/ContainerAwareEventDispatcher.php)
    #0
    /Applications/MAMP/htdocs/drupal-8.3.5/vendor/symfony/http-kernel/HttpKernel.php(88):
    Drupal\Component\EventDispatcher\ContainerAwareEventDispatcher->dispatch('kernel.terminat...',
    Object(Symfony\Component\HttpKernel\Event\PostResponseEvent))
    #1
    /Applications/MAMP/htdocs/drupal-8.3.5/vendor/stack/builder/src/Stack/StackedHttpKernel.php(32):
    Symfony\Component\HttpKernel\HttpKernel->terminate(Object(Symfony\Component\HttpFoundation\Request),
    Object(Symfony\Component\HttpFoundation\Response))
    #2
    /Applications/MAMP/htdocs/drupal-8.3.5/core/lib/Drupal/Core/DrupalKernel.php(634):
    Stack\StackedHttpKernel->terminate(Object(Symfony\Component\HttpFoundation\Request),
    Object(Symfony\Component\HttpFoundation\Response))
    #3 /Users/xjm/git/drush/lib/Drush/Boot/DrupalBoot8.php(198):
    Drupal\Core\DrupalKernel->terminate(Object(Symfony\Component\HttpFoundation\Request),
    Object(Symfony\Component\HttpFoundation\Response))
    #4 /Users/xjm/git/drush/includes/preflight.inc(75):
    Drush\Boot\DrupalBoot8->terminate()
    #5 /Users/xjm/git/drush/drush.php(12): drush_main()
    #6 {main}.
    Error: Call to undefined method Symfony\Component\HttpKernel\Event\PostResponseEvent::setDispatcher() in Drupal\Component\EventDispatcher\ContainerAwareEventDispatcher->dispatch() (line 94 of /Applications/MAMP/htdocs/drupal-8.3.5/core/lib/Drupal/Component/EventDispatcher/ContainerAwareEventDispatcher.php).
    Drush command terminated abnormally due to an unrecoverable error.   [error]
  5. In this case the database updates appear to have all run successfully and I can continue to use Drush on the 8.4.x-dev site, but unsure if the error above might have consequences for a more complicated site than my test site.

So I think/hope this is fixed, but does anyone know of a PR related to the above error?

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

alison’s picture

For the record, I give a resounding +1 and "omg me too" to Comment #51 from @ressa.

Not expecting or asking anyone to change roadmaps, I just think it's an important point for everybody to keep in mind when explaining/advocating/otherwise communicating about these stuffs.

And yes, yes I am up at 3:20am because I've been trying to update a vanilla drupal-composer/drupal-project instance to Drupal 8.4.0-alpha1 for over 3.5 hours, and I'm extremely not-new to Drupal/Drush/PHP/Unix -- but I'm quite new to composer (I'm a Drupal/WP-then-PHP dev, not vice versa). I mean that's not even true, I've been using composer on and off for like 8 months, and I'm sure it'd be easier if I used it all the time, but still, on and off for 8 months, and it's almost as painful as it was at the beginning -- I sorta feel like I'm sorta getting it, and nnnnnnope lololol football yanked away *again.*

Anyway, like I said, not looking for roadmap changes, just want to chime in and hopefully help clarify what drush vs. composer is like.

Also, thanks for all the explanations and updates :)

alison’s picture

One more comment -- it's a little OT but the issue is closed anyway, and I want to share JIC anyone lands on this thread for the same reason I did...

Here's my (work in progress!) codebase, with the changes I had to make (after creating a drupal-composer/drupal-project project), to upgrade to Drupal 8.4.0-alpha1. The vvanilla branch is without my config files, or any Pantheon stuff:
https://github.com/alisonjo2786/vanilla2/tree/vvanilla

greg.1.anderson’s picture

More work in this area is continuing at https://github.com/drush-ops/drush/pull/2843. Note that this is for the Drush master branch, which does not have the pm-* commands. Once that PR has settled, we may return to https://github.com/drush-ops/drush/pull/2787 to get better pm-* functionality across Drupal minor releases.

Regarding #53, this is likely to be a continuing problem e.g. with the 8.4.x -> 8.5.x update. There's still some lead time before that happens; perhaps 2787 will bring some improvements in this area.

AdamPS’s picture

Although this issue is closed, it is referenced from the main Core release note so is likely to get quite a few readers.

In #31 @xjm reports that

Per the DA, 60% of Drupal core installations are from tarball, and therefore not managed by Composer

On the other hand, the issue summary states

Those who wish to use `drush pm-update` & c. can continue to use a global install of Drush 8.1.12 or later. This will work for the Drupal 8.3.x to 8.4.0 update. This configuration is not officially supported for Drush 8.4.x and later, any may stop working, depending on changes made to the dependencies in Drupal.

NB That's my bold on "Drush" which is presumably a typo and should state Drupal.

So I'm hoping that the italic statement is not true - please can an expert confirm? I suggest that these "changes made to the dependencies in Drupal" need to be managed so that `drush pm-update` does not suddenly stop working.

I am aware that composer is the future, but I suggest that any proposal to drop support for pm-update should be done in a managed way with sufficient notice to enable users to migrate. It's going to be a mess if D8.4 gets released and many 1000s of users suddenly discover that D8.4 is not compatible with pm-update and at the same time D8.3 is not supported.

moshe weitzman’s picture

pm-update is already dropped in Drush9. If you try to use it, you get a brief message with tips on converting your project to be Composer built. Drush9 is the only version that is supported with Drupal 8.4.

I'm especially interested in concrete proposals (and help) in getting the word out.

AdamPS’s picture

@moshe weitzman Thanks for a prompt reply. I was already aware that Drush9 would not include pm-update. No quibbles about dropping a feature in a major version.

I'd like to take this chance to thank you for the huge amount of hard work that you do on drush, and also for your huge tolerance for all the times that people blame drush for situations where it is not really at fault.

Drush9 is the only version that is supported with Drupal 8.4.

For me and the other maybe 10000+ users of pm-update, this part is a problem. I'm not saying that I blame drush - I think it's a problem that the whole Drupal community needs to work together on. Even if we could guarantee to get an email into the inbox of every one of these people right now, I suggest that this is not enough notice before D8.4 goes out.

My personal experience is that migrating to composer is not trivial. I experimented with it for 10s of hours, found various issues and stopped as I had ran out of time. This documentation runs to many pages which suggests the composer approach is much more difficult than the one-liner "drush pm-upgrade", especially for people who have never used composer.

As an example, I have a cron nightly script that runs "d pm-update --security-only" on tests site and if there were updates, sends an email to testers to tell them to get to work ASAP. From a quick glance I couldn't see a "--security-only" option when using composer.

As a second example, I recall contrib module issues where the mere act of switching to composer caused install to fail due to a bad composer.json - not detected because the module maintainer doesn't use composer yet. There may yet be a few more of these to be found on lesser-used modules.

It feels that having this discussion on a closed issue is not necessarily the best place. Here are my concrete suggestions

  1. Raise a major issue against D8.4 beta. It seems that a significant proportion of Drupal users will find that "drush pm-update", the tool they currently use to update Drupal and modules - including security updates - will no longer be supported. I suggest that there is a good case to make a determined effort to keep pm-update working at least until D8.5 and ideally longer. As I understand it, the "unfavourable changes in Drupal dependencies" might not actually occur, and perhaps could even be deliberately avoided for a period of time.
  2. As soon as possible release a new version of drush v8 that flags a prominent warning when run against D8 to inform users that they will soon need to migrate to drush v9. Users <8.1.12 will have to upgrade drush anyway for D8.4 so would get the message even if they didn't read the D8.4 release note.
  3. Use the D8.4 release note to warn users that they will soon need to migrate to drush v9 - I suggest alongside the part that documents the need to upgrade to drush 8.1.12.
  4. Use drush issues on github to work towards making the experience much easier for users migrating to composer. I suggest this could be a combination of improved documentation plus enhancements to drush. I would like to see "drush migrate-to-composer" and "drush cm-update" (also cm-download) as wrappers to composer giving a familiar user experience to the pm family. At this point hopefully the issues that I had will go away, and there would be a lot less resistance to composer in the Drupal community.

What are you thoughts on these suggestions?

cilefen’s picture

Some notes:

Thanks!

moshe weitzman’s picture

I just want to clarify what I mean by "Drush8+Drupal8.4+ is unsupported". I expect things to work, at least initially. You are welcome to use it. But over time, this is untenable. Folks are welcome to submit PRs to Drush8. They will get reviewed.

1. You suggest opening a Drupal issue but the rest of the paragraph is about Drush changes.
2. Makes sense. Please propose something via a PR.
3. OK
4. A tool that helps migrate to composer is at https://www.drupal.org/project/composer_generate. Feel free to help with that. A link to it is shown when you run drush pm-update and similar commands in drush9. Lets continue wrappers discussion at https://github.com/drush-ops/drush/issues/2614

alison’s picture

I thought that Drupal 8.4.x works fine with the latest stable Drush 8. In fact, I'm positive that it does, at least right now, b/c that's what I'm using, and I was advised to use Drush 8 with 8.4.x -- last I checked (a few weeks ago, so admittedly "out of date") there wasn't a stable release of Drush 9 anyway.

Has this changed...?

(I'm using Drush 8.1.12 + Drupal 8.4.0-beta1)

alison’s picture

@moshe weitzman -- composer_generate is for Drupal 7.x.

AdamPS’s picture

Thanks everyone for the responses #61-64

Clarification of my intended role here - I am a part-time D8 developer in a small firm. Unfortunately I don't have a lot of time to propose patches or write documentation. I spotted this issue by chance.

I am writing here I believe on behalf of a substantial number of similar D8 users who are as yet completely unaware of the potential problem. Many sites are still on D7, but some like me have decided to make the jump to D8 and in the process contribute in a small way.

As per #2845379: Provide optional composer integration but don't force users to understand how to use composer please "Do not leave non-experts behind:". In other words, "drush pm-update" should continue to be supported until the composer based solution is equally easy. GUI would be great, but drush cm-update cm-site-install etc should be fine.

Also drush 9 is now only beta. Surely there ought to be plenty of time (6 months) between the first stable release of drush 9 and the time when it becomes the only usable version (for D8). People need to test drush 9 and integrate it into their system. It's not convenient to be forced to do this in the middle of a major customer deadline.

It's great to hear that drush 8 works with D8.4 now. So rewording my 1-4 here are my suggestions:

  1. Drush 8 should continue to work until at least D8.5. If not, then a lot of non-export users may suddenly find "drush pm-update" doesn't work when they are trying to apply a security patch. Options: remain insecure or shut down the website. This will be extremely bad for the reputation of Drupal 8. I'm not clear how/where to raise this. It needs Drupal and Drush developers to avoid making any change that would break it. That seems like it will need an automatic test.
  2. I raised a git-hub issue as a starting point. Hopefully someone can make a PR.
  3. Suggest that the D8.4 release note warn that drush 9 will be required in D8.5
  4. Propose that #2845379: Provide optional composer integration but don't force users to understand how to use composer is made a blocking issue for D8.5.
alison’s picture

I am writing here I believe on behalf of a substantial number of similar D8 users who are as yet completely unaware of the potential problem. Many sites are still on D7, but some like me have decided to make the jump to D8 and in the process contribute in a small way.

Yes! PS Drupal 8.4 already increments the symfony version, which seems like plenty of version changing to me (not to mention all the *Drupal* structural changes).

I support the "do not leave non-experts behind," though the wording bugs me -- we're calling Drush users non-experts, that's pretty douchey of "us," but whatever, that's the way the issue was worded all along, and I don't care too much about the wording as long as the thing happens.

I think #65 suggestions 1-4 are great!!

ressa’s picture

AdamPS’s picture

AdamPS’s picture

@moshe weitzman, @greg.1.anderson

I would be grateful if a drush expert could comment in #2906637: [META] Drush and core compatibility is fragile to explain what changes to Drupal would cause "drush 8 + D8.x" to stop working - and hence how to keep it working for long enough to put in place an alternative. Thanks!

AdamPS’s picture

Issue summary: View changes

Updated the summary to give guidance to users coming here from the link in the D8.4 release note.

xjm’s picture

That summary update is helpful; thanks @AdamPS!

JvE’s picture

Note that Drush 9 is not stable but beta.
It also has a lot of command and parameter changes so expect to have to rewrite any build/test/deploy scripts using drush when you upgrade.

GaëlG’s picture

And some features are not yet implemented in Drush 9:
https://github.com/drush-ops/drush/projects/1#card-4996611

cilefen’s picture

Drush and the upgrade are being discussed on this issue.

zet’s picture

After a fresh new install of Drupal Commerce 2 project with composer, which now is coming with Drupal 8.4.0, I try to install drush too. I tried with /without versions , like dev-master, ^9.0 . None works.
-------------------

composer require drush/drush:dev-master
./composer.json has been updated
> DrupalProject\composer\ScriptHandler::checkComposerVersion
Loading composer repositories with package information                                                            Updating dependencies (including require-dev)         Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - Installation request for drush/drush dev-master -> satisfiable by drush/drush[dev-master].
    - drush/drush dev-master requires webflo/drupal-finder ^1.0 -> satisfiable by webflo/drupal-finder[1.0.0] but these conflict with your requirements or minimum-stability.


Installation failed, reverting ./composer.json to its original content.

---------------------------

composer require drush/drush
Using version ^8.1 for drush/drush
./composer.json has been updated
> DrupalProject\composer\ScriptHandler::checkComposerVersion
Loading composer repositories with package information                                                            Updating dependencies (including require-dev)         Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - Conclusion: don't install drush/drush 8.1.15
    - Conclusion: don't install drush/drush 8.1.14
    - Conclusion: don't install drush/drush 8.1.13
    - Conclusion: don't install drush/drush 8.1.12
    - Conclusion: don't install drush/drush 8.1.11
    - Conclusion: don't install drush/drush 8.1.10
    - Conclusion: don't install drush/drush 8.x-dev
    - Conclusion: don't install drush/drush 8.8.x-dev
    - Conclusion: remove drupal/core 8.4.0
    - Installation request for phpdocumentor/reflection-docblock (locked at 3.2.2) -> satisfiable by phpdocumentor/reflection-docblock[3.2.2].
    - Installation request for symfony/cache (locked at v3.3.10) -> satisfiable by symfony/cache[v3.3.10].
    - Conclusion: don't install drupal/core 8.4.0
    - drush/drush 8.1.0 conflicts with drupal/core[8.4.0].
    - drush/drush 8.1.1 conflicts with drupal/core[8.4.0].
    - drush/drush 8.1.2 conflicts with drupal/core[8.4.0].
    - drush/drush 8.1.3 conflicts with drupal/core[8.4.0].
    - drush/drush 8.1.4 conflicts with drupal/core[8.4.0].
    - drush/drush 8.1.5 conflicts with drupal/core[8.4.0].
    - drush/drush 8.1.6 conflicts with drupal/core[8.4.0].
    - drush/drush 8.1.7 conflicts with drupal/core[8.4.0].
    - drush/drush 8.1.8 conflicts with drupal/core[8.4.0].
    - drush/drush 8.1.9 conflicts with drupal/core[8.4.0].
    - Installation request for drupal/core (locked at 8.4.0, required as ~8.4) -> satisfiable by drupal/core[8.4.0].
    - Installation request for drush/drush ^8.1 -> satisfiable by drush/drush[8.1.0, 8.1.1, 8.1.10, 8.1.11, 8.1.12, 8.1.13, 8.1.14, 8.1.15, 8.1.2, 8.1.3, 8.1.4, 8.1.5, 8.1.6, 8.1.7, 8.1.8, 8.1.9, 8.8.x-dev, 8.x-dev].


Installation failed, reverting ./composer.json to its original content.
benjifisher’s picture

Issue summary: View changes

I added a couple of updates to the issue description now that Drush 8.1.15 has been released. I stopped short of removing the parts that are now obsolete.

@zet, please forgive this shameless plug: I have a blog post describing how I upgraded to Drupal 8.4.0 and Drush 9.

bisu_drupal’s picture

Hello
after install drush using composer in Ubuntu i m getting
Could not open input file: /var/www/html/drush/drush.php error

Any one can help me ?
Thanks