[backport] PHP Fatal error in 8.9.x / 9.0.x upgrade path

Oy.

1. Reproduced the error in IS mannully on my local with the steps below:

   $ composer create-project drupal/recommended-project:8.8.4 -vvv
   $ cd recommended-project/
   $ composer require drupal/core-composer-scaffold:8.9.x-dev -vvv
   

2. Furthermore, confirmed that composer update does not work as expected on my local as well.

   $ composer create-project drupal/recommended-project:8.8.4 recommended-project2 -vvv
   $ cd recommended-project2/
   $ composer update -vvv
   

   Excpected version of drupal/core after running the commands above, is 8.8.5, but the actual version is 8.9.x-dev

3. No idea how to test the patch manually, But as the test-only patch in #7 reproduced the error exactly descripted in IS. and the patch in #8 fixed the error. I'd give a general +1 to the patch,
4. A tip for backporting the patch to 8.9.x: in order to get assertStringContains() working, the trait PhpunitCompatibilityTrait should be used in ScaffoldUpgradeTest.

+++ b/composer/Plugin/Scaffold/Operations/ScaffoldFileCollection.php
@@ -141,6 +141,33 @@ public function process(IOInterface $io, ScaffoldOptions $scaffold_options) {
+   *
+   * @deprecated. Only called when upgrading from the Core Composer Scaffold
+   *   plugin version 8.8.x.; remove in Drupal 9.1.x.
+   */

We should mark this @internal, we can't really deprecate it because it still has to be called and there's no way remove those calls until the new release. It will also need to be retained until Drupal 10 (i.e. remove in Drupal 10.0.0) - since we theoretically allow updates from Drupal 8.8 to Drupal 9.LAST.

The 8.9-dev issue that @jungle saw has its own issue at #3134648: [backport, needs scheduling] Don't pin the composer/installers version in drupal/core-recommended.

Thanks for the update. Only looked at the interdiff and noticed a tiny nitpick:

+++ b/composer/Plugin/Scaffold/Operations/ScaffoldFileCollection.php
@@ -154,8 +154,9 @@ public function processScaffoldFiles(IOInterface $io, ScaffoldOptions $scaffold_
+   *   version 8.8.x. due to a bug in the plugin and handler classes. Do not use

Nitpick in interdiff:

Change "8.8.x." to "8.8.x" (remove period at end).

Confirmed that #14 fixes the nitpick in #13 but leaving as "Needs review" as I don't understand the patch well enough to review better. :)

+++ b/core/tests/Drupal/Tests/Composer/Plugin/Scaffold/Functional/ScaffoldUpgradeTest.php
@@ -66,13 +67,25 @@ public function testScaffoldUpgrade() {
+    // Using 'mustExec' was giving a strange binary string here.
+    exec("composer --working-dir=$sut require drupal/core-composer-scaffold:$testVersion 2>&1", $output, $status);

Does the --no-ansi switch to composer require help? This should remove the colour encoding which might be the source of the binary string.

The binary string thing happens because PHPUnit does

            // Match for most non printable chars somewhat taking multibyte chars into account
            if (\preg_match('/[^\x09-\x0d\x1b\x20-\xff]/', $value)) {
                return 'Binary String: 0x' . \bin2hex($value);
            }

in \SebastianBergmann\Exporter\Exporter::recursiveExport(). See https://github.com/sebastianbergmann/phpunit/issues/736 for more.

Yeah, and I think --no-ansi should remove those non printable characters from the output - we use it in many other composer command lines in build tests.

Discussed with @greg.1.anderson on Slack after trying to manually reproduce; the only way to do so is have a git repository containing the updated plugin, as you have to remove the old directory from vendor first before switching to a path-based repository, and then the bug isn't triggered. Then I realised that this is exactly what the test does, and that passes, so the 9.0.x patch is RTBC. We could do with an 8.9.x patch to match though, as the one in #14 needs updating to include the --no-ansi switch.

Committed/pushed to 9.1.x and cherry-picked to 9.0.x

The 8.9.x backport is still with DrupalCI, so waiting for green there.

Committed 0423e3b and pushed to 8.9.x. Thanks!