Remove the "replace" section from core/composer.json

Is the plan here to do this for Drupal 10 only, or to backport it to 9.5 (and 9.4?) as well?

The benefit of backporting to Drupal 9 is that for modules that are deprecated in Drupal 9 core (hal, quickedit), if the site plans to use the contrib module in Drupal 10, they could first composer require in that contrib module with their codebase still on Drupal 9, and then update to Drupal 10 in a following step, rather than needing to update to Drupal 10 first before being able to bring in the contrib module.

However, I'm not clear on whether backporting this to Drupal 9 would break BC or be disruptive in any way.

Yep, we can't install hal on 9.4 #3294977: The module says to be compatible with Drupal 9.4, but it's not possible to install it with Composer

Sounds like nothing depends on this anymore based on the IS, but we should probably still at least have a CR to inform developers.

Also I guess we should mention it in the patch release notes if we're backporting it to a patch release, if for no other reason than it makes the modules installable.

Added a CR and release note, I think this is ready to go.

Just for the record: There is no issue using contrib hal. Just call it the right way: drupal/hal-hal in version of drupal still providing it. When removed the contrib should get the name drupal/hal

Not convinced removing replace is the right thing to do, looks like a drupalism. Maybe removing the module from the replace at the time of deprecating better serves the naming conflict.

Just to be clear, this won't prevent core modules without a contrib counterpart from being installed correctly, will it? Or make Composer try to install contrib Views instead of core views or something? Or screw with the versioning for core modules?

One way to find out is to commit it and see what happens...

Saving issue credits.

I've committed this to 10.1.x, 10.0.x, and 9.5.x, and published the CR.

It should likely be backported to 9.4 as well, but I've held off on backporting it yet until we make sure there are no unexpected side effects from this change. Marking PTBP to clarify that; it can be RTBC again when we've made sure nothing goes wrong.

Its worth noting this will have a positive impact on composer update times as theres still an open issue on composer to prevent replaces causing package loads unnecessarily (https://github.com/composer/composer/pull/9619)

Good news indeed!

COMPOSER_ROOT_VERSION=9.5.x-dev composer require drupal/hal:1.0.1
./composer.json has been updated
Running composer update drupal/hal
No patches supplied.
Loading composer repositories with package information
Updating dependencies
Lock file operations: 1 install, 0 updates, 0 removals
  - Locking drupal/hal (1.0.1)
Writing lock file
Installing dependencies from lock file (including require-dev)
Package operations: 1 install, 0 updates, 0 removals
  - Downloading drupal/hal (1.0.1)
No patches supplied.
Gathering patches for dependencies. This might take a minute.
  - Installing drupal/hal (1.0.1): Extracting archive

So, yep can install drupal/hal from contrib on 9.5 now.

Note that the COMPOSER_ROOT_VERSION bit is an artifact of how I do core development, I have a composer.json outside the git clone that includes stuff I don't want in my composer.json inside the git clone (e.g. drush, contrib third-party libs). In that file I have drupal/core: self.version, so I use that constant to set the core version

So on that basis I think this is good to go into 9.4

So part of my hope for manually testing this would be that people test things weird and edgecase things using 9.5.x:

1. Make sure composer require drupal/views doesn't give you the old contrib version of views
2. Make sure composer require drupal/jsonapi doesn't give you the obsolete, contrib version of JSON:API
3. Make sure you get the correct versions of core modules and previously-core-modules when they're dependencies of dependencies

Etc.

If core no longer replaces its core components then third-party packages that require those components will now have composer trying to install subsplits to vendor? Interesting...
At the same time, perhaps out of scope, perhaps as a fix, there's now an opportunity to require all core components after declaring accompanying composer path repositories and have composer symlink to the core paths. (EDIT: guess this would only be a thing for drupal/recommended-project and drupal/drupal.)

Checking #25:

1. Make sure composer require drupal/views doesn't give you the old contrib version of views

composer require drupal/views
Info from https://repo.packagist.org: #StandWithUkraine

In PackageDiscoveryTrait.php line 366:
                                                                               
  Could not find package drupal/views. It was however found via repository se  
  arch, which indicates a consistency issue with the repository. 

This is correct - Drupal.org endpoint doesn't supply it because it maintains its own list of which modules are and aren't in core (which is not derived from the replace section).

2. Make sure composer require drupal/jsonapi doesn't give you the obsolete, contrib version of JSON:API

composer require drupal/jsonapi
Using version ^2.4 for drupal/jsonapi
./composer.json has been updated
Running composer update drupal/jsonapi
> Drupal\Composer\Composer::ensureComposerVersion
Loading composer repositories with package information
Updating dependencies
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - Root composer.json requires drupal/jsonapi ^2.4 -> satisfiable by drupal/jsonapi[2.4.0, 2.x-dev].
    - drupal/jsonapi[2.4.0, ..., 2.x-dev] require drupal/core ^8.5.11 -> satisfiable by drupal/core[8.5.11, ..., 8.9.x-dev] from composer repo (https://repo.packagist.org) but drupal/core[9.5.x-dev] from path repo (core) has higher repository priority. The packages from the higher priority repository do not match your constraint and are therefore not installable. That repository is canonical so the lower priority repo's packages are not installable. See https://getcomposer.org/repoprio for details and assistance.

Use the option --with-all-dependencies (-W) to allow upgrades, downgrades and removals for packages currently locked to specific versions.
You can also try re-running composer require with an explicit version constraint, e.g. "composer require drupal/jsonapi:*" to figure out if any version is installable, or "composer require drupal/jsonapi:^2.1" if you know which you need.

Installation failed, reverting ./composer.json and ./composer.lock to their original content.

The error message is nearly unreadable, but it doesn't let you download it due to the version mismatches, which is what we want.

3. Make sure you get the correct versions of core modules and previously-core-modules when they're dependencies of dependencies.

Wasn't sure exactly what this meant, but tried the following:

composer require drupal/quickedit
Using version ^1.0 for drupal/quickedit
./composer.json has been updated
Running composer update drupal/quickedit

All smooth.

I went looking for contrib modules that depend on core modules that are already deprecated with a contrib version (hal, aggregator, quickedit), and not only is it slim pickings, but some of those modules that do exist aren't compatible with Drupal 9 yet, or the dependency was only in an older version of a module and has already been removed. However I managed to find a couple:

better_normalizers, which depends on hal, and and shows what happens with an outdated dependency decaration:

composer require drupal/better_normalizers
Using version ^1.0@beta for drupal/better_normalizers
./composer.json has been updated
Running composer update drupal/better_normalizers
> Drupal\Composer\Composer::ensureComposerVersion
Loading composer repositories with package information
Updating dependencies
Lock file operations: 1 install, 0 updates, 0 removals
  - Locking drupal/better_normalizers (1.0.0-beta4)
Writing lock file
Installing dependencies from lock file (including require-dev)
Package operations: 1 install, 0 updates, 0 removals
  - Installing drupal/better_normalizers (1.0.0-beta4): Extracting archive

This is not pulling in HAL or even mentioning it, but that much is fine because d.o doesn't convert this into a dependency on the contrib version yet while it's in core.

But there's a further problem that better_normalizers requirement is wrong:

dependencies:
  - drupal:hal (>=8.3.x)

Contrib HAL is 1.0.x or 2.0.x doesn't meet that requirement.

You can install HAL separately anyway after installing drupal/better_normalizers, because the .info.yml requirement is ignored by composer (because Drupal.org ignores it).

composer require drupal/hal:^1.0
./composer.json has been updated
Running composer update drupal/hal
> Drupal\Composer\Composer::ensureComposerVersion
Loading composer repositories with package information
Updating dependencies
Lock file operations: 1 install, 0 updates, 0 removals
  - Locking drupal/hal (1.0.1)
Writing lock file
Installing dependencies from lock file (including require-dev)
Package operations: 1 install, 0 updates, 0 removals
  - Installing drupal/hal (1.0.1): Extracting archive

Then, if you try to install better_normalizers, you can't, because HAL 1.0.1 is lower than 8.x-3.x in core's version checking and core's own requirements checks reject that combination. But this means better_normalizers needs an upstream bug report about that dependency declaration because it's just wrong at this point.

Then I found fixed_block_content, which specifies a dependency on drupal:hal

composer require drupal/fixed_block_content
Using version ^1.1 for drupal/fixed_block_content
./composer.json has been updated
Running composer update drupal/fixed_block_content
> Drupal\Composer\Composer::ensureComposerVersion
Loading composer repositories with package information
Updating dependencies
Lock file operations: 1 install, 0 updates, 0 removals
  - Locking drupal/fixed_block_content (1.1.0)
Writing lock file
Installing dependencies from lock file (including require-dev)
Package operations: 1 install, 0 updates, 0 removals
  - Downloading drupal/fixed_block_content (1.1.0)
  - Installing drupal/fixed_block_content (1.1.0): Extracting archive

Again - this installs without pulling in or mentioning HAL, because Drupal.org will not convert that requirement to a dependency on the contrib module for 9.5.x (where HAL is still in core).

However, if you explicitly require drupal/hal, you get it:

composer require drupal/hal:^1.0
./composer.json has been updated
Running composer update drupal/hal
> Drupal\Composer\Composer::ensureComposerVersion
Loading composer repositories with package information
Updating dependencies
Lock file operations: 1 install, 0 updates, 0 removals
  - Locking drupal/hal (1.0.1)
Writing lock file
Installing dependencies from lock file (including require-dev)
Package operations: 1 install, 0 updates, 0 removals
  - Installing drupal/hal (1.0.1): Extracting archive

And because there's no bogus version requirement, you can then install fixed_block_content with the contrib version of the HAL module.

So tl;dr is that if you explicitly require drupal/hal with composer you can get it, if it's a dependency of a contrib module and you install that contrib module on 9.5.x, it won't bring it in (but you've already got it in core). If a contrib module has a broken dependency declaration, then it's broken and needs fixing independently of the changes in core.

I think this is as good as we're going to get against 9.4.x/9.5.x - it's better if people get the contrib version via explicitly including it, and not just because they install a different contrib module with a dependency.

The remaining thing which would be worth opening a follow-up to track would be making sure that with 10.0.x, if you install a contrib module with a dependency on a HAL or aggregator, then you get the contrib version pulled in properly. For this however we need a real, published, 10.x version of a module with one of those dependencies to fully test this though I think - i.e. it needs to be against Drupal.org's conversion of .info.yml metadata into the packagist facade. But either way we need the replace section gone for any of this to work at all.

Sounds like we need at least two followups:

1. The remaining thing which would be worth opening a follow-up to track would be making sure that with 10.0.x, if you install a contrib module with a dependency on a HAL or aggregator, then you get the contrib version pulled in properly. For this however we need a real, published, 10.x version of a module with one of those dependencies to fully test this though I think - i.e. it needs to be against Drupal.org's conversion of .info.yml metadata into the packagist facade. But either way we need the replace section gone for any of this to work at all.

2. Bug report against fixed_block_content to fix its constraint.

Hopefully we can also distill some of #28 into troubleshooting suggestions that can be in the handbook and linked in the change record?

For #1 I opened #3299191: Ensure that modules moved to contrib in 10.0.x are installed as dependencies of other contrib modules.

I found an existing issue against better_normalizers reporting the same issue with a patch: #3294780: Drupal 9.4 compatibility with Hal module not in core anymore. (fixed_block_content is fine).

I've added a troubleshooting section to https://www.drupal.org/docs/core-modules-and-themes/deprecated-and-obsol...

Also removed the composer commands from that page because they're provided by the project pages, and out of two examples, one was outdated - no point duplicating information that's better and automatically provided elsewhere.

Linked to handbook page from the CR.

Thanks @catch; those docs look good.

Committed to 9.4.x. Thanks!

Sorry to post to this fixed topic but I'm not sure where else to put this. It looks like the colorbox_media_video module is still referencing drupal/media:* as a composer dependency and that's blocking an update to Drupal 9.4.4. Composer is trying to fetch drupal/media from the Drupal core 8.x-dev.

I've opened an issue here: https://www.drupal.org/project/colorbox_media_video/issues/3300783

Is there a better place to report this?

Therer are still some modules like "jsonapi_include" using the "self" version. This breaks installing 9.4.4 when using i.e. "jsonapi_include"

It uses "jsonapi:jsonapi" as dependency rather than "drupal:jsonapi"

Request to revert this mistake.

There are several contributed projects having problems with this, but they are all due to bugs in how the modules themselves declare dependencies:

#3301066: Module can not be used with Drupal 9.4.4 and #3300807: Won't install on Drupal 9.4.4 for examples.

We could roll back from 9.4.x and re-commit to a later patch release if we wanted to, but either way once the contrib bugs are fixed they should be fine again.

This broke components: #3116405: Warnings generated when using an optimized autoload file with Composer 1.10 / Composer 2

@Eric_A called it. :-)

Suggest closing here and working on #3116405: Warnings generated when using an optimized autoload file with Composer 1.10 / Composer 2 as the bug report.

Possibly related #3301254: Not be able to update to Drupal 9.4.4 because some Drupal 9 components require Drupal 8 components

As far as I know the core bug with core components have been fixed, and the remaining mistakes are in contributed modules that have outdated, incorrect dependency references. Restoring fixed status.

Several more:

1. ckeditor_mentions → #3304251: Module can't be installed on Drupal core >=9.4.4 due to composer dependency
2. clever_reach → #3304566: Update composer.json for Drupal >=9.4.4 compatibility
3. smsframework → #3304584: Update composer.json for Drupal >=9.4.4 compatibility

IMHO this was a disruptive change that should have shipped in 9.5.x. But it's true that it'd be painful at any time, so better sooner than later!

Sorry for posting here but i cannot find any open issue that explains what to do with contrib modules that have a requirement on drupal:ckeditor but this must be a manifestation of the same problem?

The guidance here does not seem to help:

https://www.drupal.org/docs/core-modules-and-themes/deprecated-and-obsol...

Do contrib modules need to add a bogus requirement on the contrib 'ckeditor' module in our composer.json files? (Drutopia started before CKeditor got into core, so that legacy has protected all those sites, which is why i'm running into this months late.) But wait— does that mean the contrib module get enabled instead of core? It must. My head hurts.

Again, these are modules with composer.json files that do NOT mention any ckeditor module, because we know it's in core, but do list drupal:ckeditor as a requirement, because we do require core's ckeditor to be enabled. And having such a contrib module means Drupal core won't update! #3319209: Change in Drupal core can cause WYSIWYG Linebreaks' dependency on CKEditor to block upgrades to Drupal core past 9.4.3

My guess is there may be a fair number of sites still being held back to 9.4.3 because figuring out it's happening at all is pretty subtle.

And that maybe something in Drupal's translation of .info into composer requirements could fix this?

Never mind. Composer was lying to me all over the place.

If you are stuck on Drupal 9.4.3 here is how to find out why.

1. Edit your composer.json file to have an explicit requirement on "drupal/core-recommended": "9.4.8",
2. Run composer update
3. The last problem it lists will give the module that has the old "replace" approach that needs to be fixed.

This change is still causing problems.

 9.5.x > composer require drupal/colorbutton
Using version ^1.5 for drupal/colorbutton
./composer.json has been updated
Running composer update drupal/colorbutton
> Drupal\Composer\Composer::ensureComposerVersion
Loading composer repositories with package information
Updating dependencies
Lock file operations: 3 installs, 0 updates, 0 removals
  - Locking drupal/ckeditor (1.0.1)
  - Locking drupal/colorbutton (1.5.0)
  - Locking drupal/panelbutton (1.5.0)
Writing lock file
Installing dependencies from lock file (including require-dev)
Package operations: 3 installs, 0 updates, 0 removals
  - Downloading drupal/ckeditor (1.0.1)
  - Downloading drupal/panelbutton (1.5.0)
  - Downloading drupal/colorbutton (1.5.0)
  - Installing drupal/ckeditor (1.0.1): Extracting archive
  - Installing drupal/panelbutton (1.5.0): Extracting archive
  - Installing drupal/colorbutton (1.5.0): Extracting archive

I'm on Drupal 9.5.x and there is no reason that I should be installing drupal/ckeditor (1.0.1) on 9.5.x. I don't want to move to the contrib module yet.

In order for colorbutton to be 10 and 9.5 compatible it needs a dependency on CkEditor 4. What is the module supposed to do?

FYI, this also happened on one of our websites after an upgrade to Drupal 9.5, which caused CKEditor 4 to break, because of an incompatibility with advagg, which expects CKEditor JS path to be the one from core: #3321831: CKeditor skin CSS 404 after "Enable preprocess on all JS". We did not want to use the one from contrib, but there was no way to force it to use the module from Core, that I could tell.

And the migration to CKEditor 5 is still a no-go for this particular website, because there are some installed CKEditor plugins that are still not compatible with the new version.