[meta] Ensure compatibility of Drupal 9 with PHP 8.1 (as it evolves)

Failures with reflection could use to extend core's wrapper created in #3156542: \ReflectionParameter::getClass() is deprecated in PHP 8.0

composer deprecation notice should be fixed in https://github.com/justinrainbow/json-schema/pull/666 and then propagated to composer, filed https://github.com/composer/composer/issues/9981

EDIT composer compatibility issue https://github.com/composer/composer/issues/9718 and another fix is https://github.com/justinrainbow/json-schema/pull/657

PHP Deprecated: Declaration of Drupal\Core\Template\Attribute::jsonSerialize() should be compatible with JsonSerializable::jsonSerialize(): mixed in /var/www/html/core/lib/Drupal/Core/Template/Attribute.php on line 372 could wait when core will require PHP 8 to add "mixed" to result of \Drupal\Core\Template\Attribute::jsonSerialize()

Ref https://www.php.net/manual/en/language.types.declarations.php#language.t...

Lots of Deprecated: Declaration of Doctrine\Common\Reflection\StaticReflectionClass::getName() should be compatible with ReflectionClass::getName(): string in /var/www/html/vendor/doctrine/reflection/lib/Doctrine/Common/Reflection/StaticReflectionClass.php on line 28 require to file issues to doctrine

Here's a start that allows some unit and kernel tests to run without error.

Symfony's session stuff is going to be fun and the Doctrine reflection stuff will be fun again - especially now that it is deprecated.

This is another reason to move forward with #3180351: doctrine/reflection is abandoned

Trying this locally via early support in ddev (https://github.com/drud/ddev-contrib/pull/141), the patch helps but then I am seeing a strange error with the autoloader after logging in - there is no reason I can see that this class can't be found:

NOTICE: PHP message: Error: Class "Drupal\Core\Logger\RfcLogLevel" not found in /var/www/html/drupal/core/includes/errors.inc on line 26 #0 /var/www/html/drupal/core/includes/errors.inc(60): drupal_error_levels()

Maybe I should just wait for a more stable release.

edit: turns out this was likely user error in my container setup and not a PHP or Drupal problem

Alpha 2 coming tomorrow

There's https://github.com/spiral/attributes a piolifill to use native attributes

behat/mink and masterminds/html5 both trigger PHP deprecations, so upstream will need fixes for those, or we need to work around them. We also need to catch PHP deprecations the same way we catch and pass on userland deprecations in TestHttpClientMiddleware otherwise any functional test that triggers a deprecation in the test site results in a hard fail.

I've skipped a number of PHP deprecations for the time being until we track down the source of each and fix them. Locally this gets some functional tests running, and even a few passing.

1) Drupal\Tests\media\Kernel\MediaEmbedFilterTest::testAccessUnpublished with data set "user can access embedded media" (true, true, Drupal\Core\Cache\CacheableMetadata Object (...), array(array('media/filter.caption')))
PHPUnit\Framework\Exception: Segmentation fault (core dumped)

I'm sure this will be fun to debug!

There's actually some tools built into drupalci to help with segfaults.

We build the php containers with the debug symbols, and whenever a segfault is detected, a GDB script runs and produces a backtrace of *where* in php the segfault occurred. Usually this is helpful for pushing things back upstream to PHP (we found so many with our testsuite that they delayed php7 by a couple of months to fix them all)

I would try submitting a patch with just that one test to see if you can cause it to segfault on the testbots, and there should be some data in the artifacts directory with details as to what caused the segfault.

+++ b/core/tests/Drupal/Tests/Listeners/DeprecationListenerTrait.php
@@ -126,6 +126,15 @@ public static function getSkippedDeprecations() {
+      // PHP 8.1.
+      'strtolower(): Passing null to parameter #1 ($string) of type string is deprecated',
+      'stripos(): Passing null to parameter #1 ($haystack) of type string is deprecated',
+      'strlen(): Passing null to parameter #1 ($string) of type string is deprecated',
+      'preg_match_all(): Passing null to parameter #2 ($subject) of type string is deprecated',
+      'mb_strlen(): Passing null to parameter #1 ($string) of type string is deprecated',
+      'DOMElement::setAttribute(): Passing null to parameter #2 ($value) of type string is deprecated',
+      'DOMImplementation::createDocument(): Passing null to parameter #2 ($qualifiedName) of type string is deprecated',

I think we need to actually fix these. At least we have in the past.

We totally do, I was just trying to get closer to a green run first without those getting in the way, then hopefully we can pick these off one by one as we do with other deprecations.

Added all the deprecations from the last run.

The bulk of this looks like this is going to be quite a laborious task of adding checks or casts around values passed to numerous PHP builtin functions :(

Additionally, any tests that use mikey179/vfsstream are failing with a deprecation warning due to https://github.com/bovigo/vfsStream/issues/252 - but upgrading that to 1.x-dev brings with it its own deprecation:

Using the "org\bovigo\vfs\vfsStreamWrapper" class is deprecated since version 1.7 and will be removed in version 2, use "bovigo\vfs\StreamWrapper" instead.

Finally, some uses of Prophecy also seems to be generating some deprecation warnings which then turn into a crash, e.g. $this->prophesize(MarkupInterface::class):

Return type of Double\MarkupInterface\P1::jsonSerialize() should either be compatible with JsonSerializable::jsonSerialize(): mixed, or the #[ReturnTypeWillChange] attribute should be used to temporarily suppress the notice

PHP Warning:  Uncaught Error: Object of class UnwindExit could not be converted to string in /var/www/html/drupal/vendor/phpspec/prophecy/src/Prophecy/Doubler/Generator/ClassCreator.php(49) : eval()'d code:2

PHP Fatal error:  During inheritance of JsonSerializable: Uncaught  in /var/www/html/drupal/vendor/phpspec/prophecy/src/Prophecy/Doubler/Generator/ClassCreator.php(49) : eval()'d code on line 2

Upgrading Twig for https://github.com/twigphp/Twig/issues/3534 helps a lot.

Another option is setting for phpunit https://github.com/twigphp/Twig/issues/3534#issuecomment-859866066

convertDeprecationsToExceptions="false"

With Symfony 4.4.26 we can drop the HttpFoundation workarounds: https://github.com/symfony/symfony/pull/41495

I guess we might need to sprinkle our own code with #[ReturnTypeWillChange] to start with.

Note this is an xpost with #26...

@longwave I'm not sure the best to get green is to skip all the errors. That's not how we've done it before. On the other issues like this we've worked through each error till we get each test green.

Upgrading symfony/http-foundation and masterminds/html5 fixes quite a bit too. We should do the same as Twig and Symfony and us the PHP annotation to indicate the return type will change - i.e. #[\ReturnTypeWillChange].

The patch attached has a green \Drupal\Tests\system\Functional\Module\InstallUninstallTest

Opened https://github.com/symfony/symfony/pull/42074 for one deprecation.
We need to open a PR against Mink too.

@alexpott because there are so many deprecations my intent was to get it green with skips (or at least some numbers reported back!) and then try to fix them one by one in spinoff issues, a bit like we have with PHPUnit deprecations - otherwise I feel this patch is going to become enormous very quickly

I think we need to ask for the PHP 8.1 fix for bovigo\vfs to be backported to the version we're on - or at least more compatible with it...

Drupal\Tests\Component\Gettext\PoStreamWriterTest::testCloseException
Error: Class "bovigo\vfs\vfsStreamFile" not found

... that's going to break a lot of contrib.

@longwave I think it is fine for this patch to get enormous - it's a scope of work and not much of it is worth carving into separate issues really. I'm not convinced that lots of little issues for each string cast or ?? '' in a string function is helpful. Things like the syslog config on install fix - that definitely should be it's own issue.

The bovigo\vfs issues are a bad find/replace on my part, pushed another set of fixes for this that should now be complete - worth spinning this off to its own issue and getting it committed?

@longwave we need a bovigo\vfs release before we should do that. What would be good is to update our 9.3.x dependencies now... because that'd get us a couple of fixes.

Oh yeah, 1.6.7 doesn't include the new namespaces. https://github.com/bovigo/vfsStream/issues/252#issuecomment-836191861 notes there will be a new 1.6.x or 1.7.x later this year.

Some of the exception changes overlap with #3209619: [Symfony 6] Passing null as $message to Symfony exception constructors is deprecated, pass an empty string instead but I think there might be subtle BC breaks:

-  public function __construct(CacheableDependencyInterface $cacheability, $message = NULL, \Exception $previous = NULL, $code = 0) {
+  public function __construct(CacheableDependencyInterface $cacheability, $message = '', \Exception $previous = NULL, $code = 0) {

What if someone throws CacheableAccessDeniedHttpException with no message, and later catches it and checks $message === NULL?

Re #35 I think think we're going to have that kind of problem. Imo checking for a message === NULL is just not supported. Use empty(). It's great that #3209619: [Symfony 6] Passing null as $message to Symfony exception constructors is deprecated, pass an empty string instead exists to discuss this already. The NULL exception message is one obvious issue to break out.

Adding some info about dependency updates - more to be added.

Adding more upstream issues.

The remaining fails are:

• Drupal\BuildTests\Composer\Template\ComposerProjectTemplatesTest - due to using development vendor code - will be resolved once Twig is released
• Drupal\Tests\Component\Datetime\DateTimePlusTest and Drupal\Tests\Core\Datetime\DrupalDateTimeTest - due to a PHP bug - https://bugs.php.net/bug.php?id=81273

Now it's time to create some Drupal issues for the runtime fixes here.

To sidestep the Guzzle issue, we can use a shim. Guzzle doesn't use use function or prefix its function calls with \ - so we can override http_build_query() in the GuzzleHttp namespace, and intercept and rewrite the calls as necessary. Something like:

namespace GuzzleHttp;

function http_build_query($data, $numeric_prefix, $arg_separator, $encoding_type) {
    \http_build_query($data, is_null($numeric_prefix) ? '' : $numeric_prefix, $arg_separator, $encoding_type);
}

Looking at https://github.com/php/php-src/blob/php-8.1.0beta1/UPGRADING#L486-L489

ReflectionProperty::setAccessible() and ReflectionMethod::setAccessible()
no longer have an effect. Properties and methods are always considered
accessible through reflection.
RFC: https://wiki.php.net/rfc/make-reflection-setaccessible-no-op

there's current core usage

core9$ git grep setAccessible |wc -l
160

I bet it needs separate issue as mostly tests are affected

@andypost we can't change that until we require minimum PHP 8.1 though?

Additionally pecl yaml extension fails to build with PHP 8.1 so d-org infra can't upgrade to beta

Filed pr to fix that https://github.com/php/pecl-file_formats-yaml/pull/60

Added #3212021: [PHP 8.1] Serializable interface is deprecated to child tasks

SF issue about Countable https://github.com/symfony/symfony/pull/42260

#3224592: \Drupal\path_alias\AliasManager::cacheClear() can cause deprecations on PHP 8.1 and when set to NULL it tries to flush the cache but that's not possible has landed.

Prophecy has a PR for 8.1 compatibility - https://github.com/phpspec/prophecy/pull/533 - I've applied it as a patch via our composer phpunit update commands for now and it is working.

Better linking

Beta 2 announced https://www.php.net/archive/2021.php#2021-08-05-2

Both #3180351: doctrine/reflection is abandoned and #3224414: Installing the syslog module uses its configuration before it is written have landed.

Updating issue summary with the info about what would prevent an install on PHP 8.1 due to PHP requirements. Note that these upper constraints don't affect drupal core dev because set the platform in the root composer json but it will affect anything built from core recommended and the other project templates.

SF tracking changes in https://github.com/symfony/symfony/issues/41552

Filed #3230562: Update dependencies for Drupal 9.3 as a lots of updates arrived

Merged in the Symfony updates fwiw.

Updating list of Drupal issues.

Prophecy got a new release
https://github.com/phpspec/prophecy/releases/tag/1.14.0

Created #3232571: Update dependencies for 9.3.x to address #74

Merged 9.3.x - there's been some progress. For our dependencies:

• we're still using dev versions of behat/mink behat/mink-selenium2-driver mikey179/vfsstream - these need releases
• we have a replacement class for laminas/laminas-feed

#3232571: Update dependencies for 9.3.x has landed and with that the 2 problems with Prophecy have been fixed.

The 2 twig fixes have had a release and landed in the vendor directory.

Making the why-not command correct.

PHP 8.1 RC2 is out. See: https://www.php.net/archive/2021.php#2021-09-16-1
Could we get a testbot run with 8.1 RC2?

@daffie it's already rc2 - https://www.drupal.org/pift-ci-job/2181317

I've updated the issue summary to point to the github issues asking for a PHP 8.1 compatible stable release for behat/mink, behat/mink-selenium2-driver and mikey179/vfsstream .

#3212021: [PHP 8.1] Serializable interface is deprecated has landed.

Child issue to be fixed:

• #3236769: PoItem causes deprecation errors on PHP 8.1
• #3236284: Passing request headers to string functions causes deprecations in PHP 8.1
• #3236796: Fix PHP 8.1 deprecation errors in Drupal\Core\Datetime classes
• #3233010: drupal_phpunit_contrib_extension_directory_roots() passes NULL to file_exists() causing deprecations in PHP 8.1
• #3232699: SQLite schema introspection implementation struggles with NULL default values
• #3232695: Only process values in config entity query conditions for operators that have values
• #3224941: Remove usage of setAccessible() when core requires PHP 8.1
• #3236798: \Drupal\Core\Datetime\DateFormatter::formatInterval() causes deprecations in PHP 8.1

@daffie thanks for updating the issue summary. 3224941 doesn't need fixing yet - it's open for when core's minimum PHP version is PHP 8.1 - feels quite a way off :)

#3224421: [PHP 8.1] Add a shim to Guzzle 6 for PHP 8.1 compatibility has landed.

The last commit https://git.drupalcode.org/project/drupal/-/merge_requests/937/diffs?com... is to work out how and if #3238452: Exception messages must default to an empty string not a NULL for PHP 8.1 is actually necessary.

#3236769: PoItem causes deprecation errors on PHP 8.1, #3233010: drupal_phpunit_contrib_extension_directory_roots() passes NULL to file_exists() causing deprecations in PHP 8.1 and #3232695: Only process values in config entity query conditions for operators that have values have landed.

#3238452: Exception messages must default to an empty string not a NULL for PHP 8.1 has landed.

Filed 2 new patches - it makes my watchdog clean

- #3238941: \Drupal\big_pipe\Render\BigPipe::splitHtmlOnPlaceholders() causes deprecation errors on PHP 8.1
- #3238942: \Drupal\Core\Routing\RedirectDestination::get() causes deprecation errors on PHP 8.1

#3238942: \Drupal\Core\Routing\RedirectDestination::get() causes deprecation errors on PHP 8.1 has landed and created: #3239270: Update dependencies for 9.3 for vfsStream.

#3239295: Passing NULL to the $replace parameter of str_replace() and preg_replace() triggers a deprecation in PHP 8.1, #3239294: Passing NULL to the limit argument of preg_split triggers a deprecation in PHP 8.1 and #3239313: \Drupal\views\Controller\ViewAjaxController causes deprecations on PHP 8.1 have landed.

moved to done #3239287: Fix \Drupal\Core\Extension\ModuleDependencyMessageTrait to not cause deprecations in PHP 8.1

filed
- #3239552: Improve hash() calculation for PHP 8.1
- #3239553: \Symfony\Component\Routing\Route::getRequirement() causes deprecation errors on PHP 8.1 when it returns NULL

Added #3239556: Fix return type of \Drupal\Tests\migrate\Kernel\TestFilterIterator::accept()

added #3239558: Properly use state counter in \Drupal\search_embedded_form\Form\SearchEmbeddedForm

Fixed
#3239292: Missing configuration in KernelTests causes deprecations in PHP 8.1
#3239552: Improve hash() calculation for PHP 8.1
#3239556: Fix return type of \Drupal\Tests\migrate\Kernel\TestFilterIterator::accept()
#3239558: Properly use state counter in \Drupal\search_embedded_form\Form\SearchEmbeddedForm
#3239287: Fix \Drupal\Core\Extension\ModuleDependencyMessageTrait to not cause deprecations in PHP 8.1

Filed upgrade for SF (has some 8.1 fixes) and the rest #3239772: Update dependencies for Drupal 9.3

Fixed

• #3239553: \Symfony\Component\Routing\Route::getRequirement() causes deprecation errors on PHP 8.1 when it returns NULL
• #3239746: \Drupal\Core\Flood\MemoryBackend uses array keys that cause deprecations in PHP 8.1
• #3239762: \Drupal\Core\Template\AttributeString::__toString() can cause deprecations in PHP 8.1

moved to done

• #3239710: \Drupal\Core\Menu\StaticMenuLinkOverrides::loadOverride() causes a deprecation on PHP 8.1
• #3239761: Fix MimeTypeTest to prevent deprecations in PHP 8.1 and fix \Drupal\Core\StreamWrapper\PrivateStream::basePath() documentation

The biggest change has done #3232673: \Drupal\Core\Entity\EntityInterface::label() can return a NULL

PHPCS also should create new release soon as milestone finished https://github.com/squizlabs/PHP_CodeSniffer/milestone/26

Lots of follow-ups files by Alex

Meantime I find this 2 issues are priority as prevents lots of noise running tests

- #3240191: \Drupal\KernelTests\KernelTestBase::bootKernel() causing deprecations in PHP 8.1
- #3240192: \Drupal\user\AccountForm::buildEntity() causing deprecations in PHP 8.1

PHP has released PHP 8.1 RC3. See: https://www.php.net/archive/2021.php#2021-09-30-1

We are getting a number of explode() warnings: "Exception: Deprecated function: explode(): Passing null to parameter #2 ($string) of type string is deprecated Drupal\comment\CommentViewBuilder->buildComponents()() (Line: 127)"

Could we do a testbot run with PHP 8.1-RC3.

Just queued test run on updated images (labels are not deployed yet)

00:27:50 php -v

00:27:51 PHP 8.1.0RC3 (cli) (built: Oct  1 2021 18:22:46) (NTS)
00:27:51 Copyright (c) The PHP Group
00:27:51 Zend Engine v4.1.0RC3, Copyright (c) Zend Technologies
00:27:51     with Zend OPcache v8.1.0RC3, Copyright (c), by Zend Technologies

@daffie this new failure will gone after #3240167: \Drupal\comment\CommentStorage::getMaxThread() and \Drupal\comment\Entity\Comment::getThread() cause deprecations on PHP 8.1

Filed #3240360: Clean-up usage of http-client in \Drupal\Tests\views\Functional\ViewAjaxTest::testNonAjaxViewViaAjax()

Filed #3240361: \Drupal\user\Entity\User::checkExistingPassword() causes deprecations on PHP 8.1

Next blocker #3240456: Allow E_DEPRECATED deprecations to be skipped so we can test on PHP 8.1

These issue have landed. Could we do a reroll? Hopefully with a testbot with RC3

• #3240364: \Drupal\Tests\system\Functional\Pager\PagerTest::testMultiplePagers() causes deprecations on PHP 8.1
• #3240167: \Drupal\comment\CommentStorage::getMaxThread() and \Drupal\comment\Entity\Comment::getThread() cause deprecations on PHP 8.1
• #3240180: \Drupal\user\Entity\User::getEmail() can return NULL causing deprecations in PHP 8.1
• #3240455: Fix Drupal\Tests\content_translation\Functional\ContentTranslationSettingsTest so it does not trigger deprecations on PHP 8.1
• #3240361: \Drupal\user\Entity\User::checkExistingPassword() causes deprecations on PHP 8.1
• #3240456: Allow E_DEPRECATED deprecations to be skipped so we can test on PHP 8.1
• #3240182: \Drupal\dblog\Controller\DbLogController::createLink() can cause deprecations on PHP 8.1
• #3240228: _filter_url_trim() can cause deprecations on PHP 8.1
• #3240362: \Drupal\tour\TipPluginBase::getLocation() causes deprecations on PHP 8.1
• #3240174: \Drupal\config_translation\FormElement\Textarea() can cause deprecations on PHP 8.1

7 new child issues filed

and 2 more
- #3240955: \Drupal\media\Plugin\Filter\MediaEmbed::applyPerEmbedMediaOverrides() triggers deprecations in PHP 8.1
- #3240958: template_preprocess_locale_translation_last_check() causes deprecations on PHP 8.1

I *think* that we now have sub issues for every single change in the meta merge request. The only thing we're missing is a Mink and a MinkSelenium2Driver upgrade issue. The Mink maintainer has said that there will be a release soon as I've resolved their blockers for that.

Updating outstanding list.

#3241296: core/tests/Drupal/KernelTests/Core/Path/PathValidatorTest.php triggers deprecations on PHP 8.1, #3241267: Drupal\Tests\Core\Session\WriteSafeSessionHandlerTest::testOtherMethods() fails due to a type error on PHP 8.1 and #3241268: core/modules/node/tests/modules/node_test/node_test.module cause deprecations on PHP 8.1 have landed.

#3241280: Fix PHP 8.1 deprecations caused by views code and #3241306: core/tests/Drupal/Tests/Core/Database/ConnectionTest.php fails on PHP 8.1 have landed.

Only 3 issues left!

Oops there are 4... forgot to make this a parent of one... #3241272: core/modules/system/tests/src/Functional/FileTransfer/TestFileTransfer.php causes deprecations on PHP 8.1

There's phpcs release compatible with 8.1

https://github.com/squizlabs/PHP_CodeSniffer/releases/tag/3.6.1

@andypost let's wait for behat/* until we create a dependency update.

The last issue is RTBC, updated IS with https://github.com/squizlabs/PHP_CodeSniffer/releases/tag/3.6.1

I filed another child issue #3242889: Update dependencies for 9.3.x

we could re-purpose it for other updates but I get locally still 2 deprecation notices

Green at last.

We are done! PHP 8.1 testing has been enabled for 9.3.x on commit.

All child issues that are required for PHP 8.1 compatibility are fixed, therefore, we can move this meta to fixed as it is complete.

🎉 great work, thanks a lot to all involved!

I think daily or weekly run of CI 8.1 could be enabled to prevent regressions

One more child added #3244533: usleep() usage triggers deprecations on PHP 8.1

After latest upgrade to 9.3.7 with php 8.1, got the strange error mentioned in #12
NOTICE: PHP message: Error: Class "Drupal\Core\Logger\RfcLogLevel" not found in ...

Did composer install after removing `core` && `vendor` folders as well. same issue.

EDIT: Fixed this by restarting `service php8.1-fpm reload`

I just ran into the error message about RfcLogLevel too. See #12.

In my case a Commerce Migrate functional test was working on 9.4.x on one ddev environemnt and failing in another ddev environment The difference was that the working one was using composer 2, composer_version: "2" in .ddev/config.yaml and the failing one had no configuration value for composer_version. Once I

Fatal error with PHP 8.1 and Drupal 9.3.12:

Uncaught TypeError: htmlspecialchars() expects parameter 1 to be string, array given in Drupal\Component\Utility\Html::escape() (line 424 of /app/web/core/lib/Drupal/Component/Utility/Html.php)

If the field value is empty (''), then following twig template code will triggle that fatal error.

{% set body = {
  '#type': 'processed_text',
  '#text': paragraph.field_body.value,
  '#format': paragraph.field_body.format,
} %}

{{ body }}

@Mingsong try {{ paragraph.field_body.processed }}

Thanks @Lee Rowlands for prompt response.

Yes, using 'processed' instead of the render array will avoid the fatal PHP error.