[META] Improve Drupal's use of Composer, part 1

Nice. Is there an issue that properly captures the thought below, without polluting another thread?

Relevant excerpt:
#1612910-34: [policy, no patch] Switch to Semantic Versioning for Drupal contrib extensions (modules, themes, etc)

EDIT: It's a whole other line of discussion whether the huge drupal core (essentially many interoperable modules with their own API's) can actually share a single semver schema. Maybe that's the root of the bigger question that we can't quite wrap our minds around (myself included), because we recognize it's such a drastic step. Taken to its logical extension (and this is definitely a thought-in-progress), Drupal core is really just a meta-package with a bunch of component libraries, each with their own independantly evolving API. Maybe a contrib module doesn't really want to set a dependency on a version of the monolithic drupal core meta-package, but instead on a specific version of watchdog or views or admin_overlay or user. That might be the only discussion that's possible to have in a meaningful way...

Related: #586146-16: [policy, no patch] Decide if and how to implement semantic versioning for Drupal 8.x

EDIT: Meh. Opened a new issue :)
#2018019: Split core modules into separate repos with independently versioned API's

Updated issue summary.

Added a link to a test I made to validate Drupal 8's use of Composer. https://github.com/paul-m/d8-create-project

And another one for composer require: https://github.com/paul-m/d8-require-drupal

Added beta evaluation.

Added some behaviors we want to support.

Please discuss these. :-)

Added some behaviors we want to support.

Please discuss these. :-)

For me the big one is composer require. Composer is a dependency manager—if I can't use it to add dependencies it's not really much use to me. I would love to see that feature make it in.

Added this test: https://travis-ci.org/paul-m/d8-drupal-require

Looks like this might just be working:

$ composer create-project drupal/drupal drupal 8.0.*@dev --no-progress
Installing drupal/drupal (8.0.x-dev a9f2eead745dc3df58313c360eca58712318fb06)
  - Installing drupal/drupal (8.0.x-dev a9f2eea)
    Cloning a9f2eead745dc3df58313c360eca58712318fb06

Created project in drupal
Loading composer repositories with package information
Installing dependencies (including require-dev)
  - Installing composer/installers (v1.0.21)
    Downloading

  - Installing masterminds/html5 (2.1.1)
    Downloading

  - Installing react/promise (v2.2.0)
    Downloading

  - Installing guzzlehttp/streams (3.0.0)
    Downloading

  - Installing guzzlehttp/ringphp (1.1.0)
    Downloading

  - Installing guzzlehttp/guzzle (5.3.0)
    Downloading

  - Installing symfony/dom-crawler (v2.7.0)
    Loading from cache

  - Installing symfony/css-selector (v2.6.9)
    Loading from cache

  - Installing symfony/browser-kit (v2.7.0)
    Loading from cache

  - Installing fabpot/goutte (v2.0.4)
    Downloading

  - Installing symfony/process (v2.6.9)
    Loading from cache

  - Installing symfony/event-dispatcher (v2.6.9)
    Loading from cache

  - Installing behat/mink (v1.6.1)
    Downloading

  - Installing behat/mink-browserkit-driver (v1.2.0)
    Downloading

  - Installing behat/mink-goutte-driver (v1.1.0)
    Downloading

  - Installing doctrine/lexer (v1.0.1)
    Downloading

  - Installing egulias/email-validator (1.2.8)
    Downloading

  - Installing psr/log (1.0.0)
    Downloading

  - Installing symfony/debug (v2.7.0)
    Loading from cache

  - Installing symfony/http-foundation (v2.6.9)
    Loading from cache

  - Installing symfony/http-kernel (v2.6.9)
    Loading from cache

  - Installing stack/builder (v1.0.3)
    Downloading

  - Installing mikey179/vfsstream (v1.5.0)
    Downloading

  - Installing zendframework/zend-stdlib (2.4.2)
    Downloading

  - Installing zendframework/zend-escaper (2.4.2)
    Downloading

  - Installing zendframework/zend-feed (2.4.2)
    Downloading

  - Installing phpunit/php-file-iterator (1.4.0)
    Loading from cache

  - Installing doctrine/instantiator (1.0.4)
    Downloading

  - Installing sebastian/recursion-context (1.0.0)
    Downloading

  - Installing sebastian/exporter (1.2.0)
    Downloading

  - Installing sebastian/diff (1.3.0)
    Downloading

  - Installing sebastian/comparator (1.1.1)
    Downloading

  - Installing phpdocumentor/reflection-docblock (2.0.4)
    Loading from cache

  - Installing phpspec/prophecy (v1.4.1)
    Loading from cache

  - Installing sebastian/version (1.0.5)
    Downloading

  - Installing sebastian/environment (1.2.2)
    Downloading

  - Installing phpunit/php-text-template (1.2.0)
    Downloading

  - Installing phpunit/php-token-stream (1.4.1)
    Downloading

  - Installing phpunit/php-code-coverage (2.1.3)
    Downloading

  - Installing sebastian/global-state (1.0.0)
    Downloading

  - Installing symfony/yaml (v2.6.9)
    Loading from cache

  - Installing phpunit/phpunit-mock-objects (2.3.3)
    Loading from cache

  - Installing phpunit/php-timer (1.0.5)
    Downloading

  - Installing phpunit/phpunit (4.6.10)
    Downloading

  - Installing easyrdf/easyrdf (0.9.1)
    Downloading

  - Installing symfony/routing (v2.6.9)
    Loading from cache

  - Installing symfony-cmf/routing (1.3.0)
    Downloading

  - Installing doctrine/annotations (v1.2.4)
    Downloading

  - Installing doctrine/collections (v1.3.0)
    Downloading

  - Installing doctrine/cache (v1.4.1)
    Downloading

  - Installing doctrine/inflector (v1.0.1)
    Downloading

  - Installing doctrine/common (v2.4.2)
    Downloading

  - Installing twig/twig (v1.18.1)
    Downloading

  - Installing symfony/translation (v2.7.0)
    Loading from cache

  - Installing symfony/validator (v2.6.9)
    Loading from cache

  - Installing symfony/serializer (v2.6.9)
    Loading from cache

  - Installing symfony/dependency-injection (v2.6.9)
    Loading from cache

  - Installing symfony/class-loader (v2.6.9)
    Loading from cache

  - Installing sdboyer/gliph (0.1.8)
    Downloading

  - Installing drupal/core (8.0.0-beta11)
    Loading from cache

behat/mink suggests installing behat/mink-selenium2-driver (slow, but JS-enabled driver for any app (requires Selenium2))
behat/mink suggests installing behat/mink-zombie-driver (fast and JS-enabled headless driver for any app (requires node.js))
symfony/http-kernel suggests installing symfony/config ()
symfony/http-kernel suggests installing symfony/console ()
symfony/http-kernel suggests installing symfony/finder ()
symfony/http-kernel suggests installing symfony/var-dumper ()
zendframework/zend-stdlib suggests installing zendframework/zend-eventmanager (To support aggregate hydrator usage)
zendframework/zend-stdlib suggests installing zendframework/zend-serializer (Zend\Serializer component)
zendframework/zend-stdlib suggests installing zendframework/zend-servicemanager (To support hydrator plugin manager usage)
zendframework/zend-stdlib suggests installing zendframework/zend-filter (To support naming strategy hydrator usage)
zendframework/zend-feed suggests installing zendframework/zend-cache (Zend\Cache component)
zendframework/zend-feed suggests installing zendframework/zend-db (Zend\Db component)
zendframework/zend-feed suggests installing zendframework/zend-http (Zend\Http for PubSubHubbub, and optionally for use with Zend\Feed\Reader)
zendframework/zend-feed suggests installing zendframework/zend-servicemanager (Zend\ServiceManager component, for default/recommended ExtensionManager implementations)
zendframework/zend-feed suggests installing zendframework/zend-validator (Zend\Validator component)
phpdocumentor/reflection-docblock suggests installing dflydev/markdown (~1.0)
phpdocumentor/reflection-docblock suggests installing erusev/parsedown (~1.0)
phpunit/php-code-coverage suggests installing ext-xdebug (>=2.2.1)
sebastian/global-state suggests installing ext-uopz (*)
phpunit/phpunit suggests installing phpunit/php-invoker (~1.1)
easyrdf/easyrdf suggests installing ml/json-ld (~1.0)
symfony/routing suggests installing symfony/config (For using the all-in-one router or any loader)
symfony/routing suggests installing symfony/expression-language (For using expression matching)
symfony/translation suggests installing symfony/config ()
symfony/validator suggests installing symfony/intl ()
symfony/validator suggests installing symfony/config ()
symfony/validator suggests installing symfony/property-access (For using the 2.4 Validator API)
symfony/validator suggests installing symfony/expression-language (For using the 2.4 Expression validator)
symfony/dependency-injection suggests installing symfony/config ()
symfony/dependency-injection suggests installing symfony/proxy-manager-bridge (Generate service proxies to lazy load them)
Writing lock file
Generating autoload files
Do you want to remove the existing VCS (.git, .svn..) history? [Y,n]? n

I have run this through travis again and it gets the code ok, but results in two vendor directories '/vendor' and '/core/vendor' which is causing issues.

https://travis-ci.org/timmillwood/d8-create-project

I have opened a new issue to sort out the duplicate vendor directory problem.

#2500295: 'composer install' creates two vendor directories

Made this: #2536576: Add composer_dependency module, have it help users figure out how to install Composer dependencies

Ladies and gentlemen, after #2380389: Use a single vendor directory in the root, we have: https://travis-ci.org/paul-m/d8-drupal-require

(Spoiler: It's passing.)

We can now say composer require crell/api-problem and it works.

https://travis-ci.org/paul-m/d8-create-project still fails, but actually looks solvable.

https://travis-ci.org/paul-m/d8-create-project fails because the package on packagist is outdated. Just wait a few days and it will work.

Here's a test that proves that you can (minimally) use core as a dependency of your own project: https://travis-ci.org/paul-m/d8-require-drupal

Added new child issue: #2609568: Have wikimedia/composer-merge-plugin manage contrib dependencies.

Super easy.

Has anybody looked at how much work it would be for Drupal to discover modules and themes in the vendor directory (if it's possible at all)?

That would certainly reduce the amount of odd things we have to do to support proper composer use.

Discovery is one problem, but we ship web accessible assets with our modules/themes and deny access to the vendor folder completely.

Careful that in 8.1-beta2, with zendframework/zend-stdlib 2.7.3, zend-hydrator was still being used, but as of zend-stdlib 3.0.0, it has been deprecated and removed as dependency from zend-stdlib (no longer installed by default).

from stdlib 2.7.3:

    "require": {
        "php": ">=5.5",
        "zendframework/zend-hydrator": "~1.0"
    },

That means if some code is using classes from the hydrator, the dependency will need to be added manually to the require in composer or the classes will need to be reworked.

https://api.drupal.org/api/drupal/vendor!zendframework!zend-hydrator!src...

The result -- if you download the 8.1-beta2 tarball, zend-hydrator is included; if you use composer to get drupal8, it will not be included.

more info:
https://github.com/drupal-composer/drupal-project
https://bojanz.wordpress.com/2015/09/18/d8-composer-definitive-intro/

## 3.0.0 - 2016-02-03

### Added

- [#51](https://github.com/zendframework/zend-stdlib/pull/51) adds PHP 7 as a
  supported PHP version.
- [#51](https://github.com/zendframework/zend-stdlib/pull/51) adds a migration
  document from v2 to v3. Hint: if you use hydrators, you need to be using
  zend-hydrator instead!
- [#51](https://github.com/zendframework/zend-stdlib/pull/51) adds automated
  documentation builds to gh-pages.

### Deprecated

- Nothing.

### Removed

- [#33](https://github.com/zendframework/zend-stdlib/pull/33) - removed
  deprecated classes
  - *All Hydrator classes* see #22.
  - `Zend\Stdlib\CallbackHandler` see #35
- #37: Missing Children when looking at recent comments(https://github.com/zendframework/zend-stdlib/pull/37) - removed
  deprecated classes and polyfills:
  - `Zend\Stdlib\DateTime`; this had been deprecated since 2.5, and only
    existed as a polyfill for the `createFromISO8601()` support, now standard
    in all PHP versions we support.
  - `Zend\Stdlib\Exception\InvalidCallbackException`, which was unused since #33.
  - `Zend\Stdlib\Guard\GuardUtils`, which duplicated `Zend\Stdlib\Guard\AllGuardsTrait`
    to allow usage with pre-PHP 5.4 versions.
  - `src/compatibility/autoload.php`, which has been dprecated since 2.5.
- #37: Missing Children when looking at recent comments(https://github.com/zendframework/zend-stdlib/pull/37) - removed
  unneeded dependencies:
  - zend-config (used only in testing ArrayUtils, and the test was redundant)
  - zend-serializer (no longer used)
- [#51](https://github.com/zendframework/zend-stdlib/pull/51) removes the
  documentation for hydrators, as those are part of the zend-hydrator
  component.

### Fixed

- Nothing.

Hello!

We just noticed that all 8.1.x tags are broken in https://github.com/drupal/core repository.
We did not know where to report this, so posted at https://github.com/drupal/core/issues/1

Also, I'm adding this comment to attract some attention :)

Tags other than 8.1.x look good.

UPD: the issue was fixed by Ryan Aslett.

the composer utility and drupal require conflicting PHP settings:

I recently ran into a problem with D8.x image upload that was only fixed by the PHP setting:

allow_url_fopen = Off

which is as documented on drupal.org for D8.x PHP settings requirements (see link):

https://www.drupal.org/docs/7/system-requirements/php#shortlist

Meanwhile the PHP setting above disables the composer utility which throws a specific error message stating that:

allow_url_fopen = On

must be set for it to operate (see composer error text):

The "https://packages.drupal.org/8/packages.json" file could not be downloaded: allow_url_fopen must be enabled in php.ini (https:// wrapper is disabled in the server configuration by allow_url_fopen=0
failed to open stream: no suitable wrapper could be found)
https://packages.drupal.org/8 could not be fully loaded, package information was loaded from the local cache and may be out of date
The "https://packagist.org/packages.json" file could not be downloaded: allow_url_fopen must be enabled in php.ini (https:// wrapper is disabled in the server configuration by allow_url_fopen=0
failed to open stream: no suitable wrapper could be found)
https://packagist.org could not be fully loaded, package information was loaded from the local cache and may be out of date

@DrupalWebsiteBuilder : Drupals recommendation for allow_url_fopen = Off is correct for web based php. However, composer is a command line php application, and as such you would typically have a completely separate php.ini for command line applications. If you have one configuration for both CLI php and webserver php, then you will likely encounter conflicts such as this.

Related issue: #2906637: [META] Drush and core compatibility is fragile

There is increased urgency for needing composer because "drush pm-upgrade" is in danger of stopping working in D8.

Heads up: #2958021: Proposal: Composer Support in Core initiative

In the first snippet in the description there's written $ cd path/to/drupal/ whereas in the second snippet there's written $ cd drupalroot/. Is there supposed to be a drupalroot folder inside path/to/drupal? I know these folder names are placeholders. But shouldn't it be $ cd path/to/drupal/ in both snippets to avoid confusion?

Is there an issue yet for fixing the problem whereby Drupal contrib projects that have submodules can't be updated painlessly?

E.g. I am on acsf 2.48.0, which is not the latest version, and 'composer update drupal/acsf' says:

> Nothing to install or update

'composer why-not drupal/acsf 2.51.0' gives the answer:

drupal/acsf_duplication  2.48.0  requires  drupal/acsf (self.version)
drupal/acsf_theme        2.48.0  requires  drupal/acsf (self.version)
drupal/acsf_variables    2.48.0  requires  drupal/acsf (self.version)

There are circular dependencies between the main module & its submodules, which AFAIK are created by d.org's composer packaging process.

That means that if you don't know about this, you are headdesking because Composer is refusing to install a new version.

If you DO know about this, you have to name ALL submodules when doing a composer update:

> composer update drupal/acsf drupal/acsf_theme drupal/acsf_variables

This is a) a pain, and b) counter-intuitive, and c) ruins the point of Composer being able to manage dependencies.

Doubt very much it is considered a software problem. Suggest to start with `composer prohibits` (why-not) when updating to find out whether there are things blocking the update. Suggest to use --with-dependencies --dry-run on `composer update` to find out what it will do. In the case that only its submodules are effected you can use --with-dependencies. Otherwise advice to whitelist them, possibly with acsf*.

Checked replacement of default autoloader with composer one brings +5-8% of RPS for me

Also https://twitter.com/nmdmatt/status/1328902315164766208 states similar

I am going to close this and open a new Meta, a part 2, for the following reasons

• The problem statement in the issue summary was completed some time ago.
• The discussion here has slowed down since 2016.
• The majority of child issues are fixed
• The title uses 'improve' and here that is open ended. I'd rather give people credit for the work done so far and have a following issue.
• Possibly easier to see what else there is to do from a new starting point.

I will move the child issues that are still open to the new meta, #3525703: [meta] Improve Drupal's use of Composer, round 2.