Please consider all contributors to #2701829: Extension objects should not implement \Serializable for credit.
Actually there also may be other classes implementing Serializable interface. The \Drupal\views\ViewExecutable is just the first known one to cause the trouble.
Problem/Motivation
There is a PHP bug in the unserialize() function - https://bugs.php.net/bug.php?id=66052 . Precisely in nested unserialization.
When one serialized string contains object which implements Serializable interface, and this object calls another unserialize() with string containing references (serialized as r:NUMBER;), these references are substituted by values from the first string upon unserialization, not from the current one. This behavior leads to corrupted data in runtime and to fatal errors in some cases, and that data may accidentally be saved to persistent storage if not validated explicitly.
The ViewExecutable::execute() method is known to call unserialize() during its job. Examples of Fatals caused by this nested unserialization: #2750463: Fatal error: Call to a member function getFieldStorageDefinition() on array, #2701829: Extension objects should not implement \Serializable. There may be more. (Other code from ViewExecutable::unserialize() was not tested by me.)
A simple test case, which can be run in /devel/php form on any D8 dev site using the content view (/admin/content):
$field_manager = \Drupal::service('entity_field.manager');
// Get the definition of node's nid field, for example. Only get it not from
// the field manager directly, but from the item data definition. It should be
// the same base field definition object (the field and item definitions refer
// to each other).
$nid_definition = $field_manager->getBaseFieldDefinitions('node')['nid']
->getItemDefinition()
->getFieldDefinition();
dpm($nid_definition, 'Expected:');
// Load and execute the view.
$view_entity = \Drupal\views\Entity\View::load('content');
$view = \Drupal::service('views.executable')->get($view_entity);
$view->execute('page_1');
// Reset static cache...
$field_manager->useCaches(FALSE);
// ... but leave possibility to use database cache.
$field_manager->useCaches(TRUE);
// Magic line.
unserialize(serialize(['SOMETHING UNEXPECTED', $view]));
// The same code as in the beginning of the test.
$nid_definition = $field_manager->getBaseFieldDefinitions('node')['nid']
->getItemDefinition()
->getFieldDefinition();
dpm($nid_definition, 'After nested unserialization:');
Proposed resolution
As the bug is still not fixed in PHP, we need to provide conditions to avoid its appearance in Drupal.
I propose to implement some kind of lazy execution in ViewExecutable, if that possible. Ideally, unserialize() method should simply assign values to object properties.
For impatient I'm attaching a patch with a very rough workaround for a limited use case - it covers only ViewExecutable class and the Database cache backend. But it may slow down your project, so don't use it on production.
Remaining tasks
- Examine other classes with methods
unserialize()and__wakeup().
| Comment | File | Size | Author |
|---|---|---|---|
| #88 | 2849674-88.patch | 7.44 KB | lendude |
| #88 | interdiff-2849674-80-88.txt | 663 bytes | lendude |
| #80 | 2849674-80.patch | 7.3 KB | lendude |
| #80 | 2849674-80-TEST_ONLY.patch | 2.03 KB | lendude |
| #80 | interdiff-2849674-76-80.txt | 626 bytes | lendude |
Comments
Comment #2
mxh commentedAs already written in https://www.drupal.org/node/2750463#comment-12081985, I suspect that the "trigger" of the problem is calling the
serialize()function inside theViewExecutable::serialize()method.Attaching a patch which replaces the
Serializableimplementation by an alternative using magic methods.By this, I also removed the usage of DependencySerializationTrait, because it seems that it had no use anyway (
__sleep()and__wakeup()aren't used anymore when a class implements the\Serializableinterface). I haven't found any other point where this might be relevant, thus removed.Comment #3
mxh commentedComment #4
geek-merlinCode looks straightforward to me.
OK storage ID is serialized, but what about the other injected services (routeProvider) and data (user, viewsData)? My gut feeling is, add viewsData and the IDs of user and routeProvider.
Comment #5
mxh commentedYes, while
__wakeup()already loads the user data, it's still missing theviews.views_dataandrouter.route_providerservice. Seems that this was already broken before. Writing new patch...Comment #6
mxh commentedNow attaching all necessary services on the executable, plus added circular referencing from storage to executable just like in the constructor method.
Needs some manual serialization tests for proper reviewing.
Comment #7
geek-merlinCan we thoroughly assume these objects can not change? Use cases for this might sound esoteric but what can be thought will be done... Storing the IDs imho does not cost us much, are are there disadvantages?
Comment #8
geek-merlinComment #9
mxh commentedAgree, except for the current user object. Since the executable makes some access checks on it, it should be ensured that the object actually represents the current user.
Comment #10
manuel garcia commentedComment #11
geek-merlinImho to the contrary. There are legitimate use cases where the view user is *not* the current user. I remember working on a D7 commerce issue where the payment gateway made a callback, bootstrapped as anonymous user, and had to execute a view for the transaction's user. That's an ideal case for executing a serialized view for another user with their access context. What do you think?
EDIT: And another viewpoint: If we don't respect the original user, the user property of the view does not make any sense and it should simply use the global current user.
Comment #12
mxh commentedDisagree with the following arguments:
userproperty to be the current user. Other objects which use the executable object would adopt this expectation. Exceptional use cases should handle the change of this behavior on their own.It does make sense, since it's there to let other objects being able to use the object, without the need of dependency injection or global container usage.
Comment #13
geek-merlinHmm...
> The whole executable object expects the user property to be the current user.
Is there any evidence for that? I see evidence for the opposite, besides the user property, read comments in e.g. node_query_node_access_alter().
> A user who is able to use unserialized view executables, ... might then bypass access restrictions.
To unserialize arbitrary objects you already need execute-php, which is far stronger. So i see no privilege escalation potential here.
But hey no problem, we can give the component maintainer both patches to decide. It's anyway up to them.
Comment #14
mxh commentedDocumentation of the
ViewExecutablepropertyuser,::__construct()as well as::getUser():Besides there's also no
setUser()method which allows to change the user object, this should indicate that the executable view always expects the current(ly logged in) user.Sorry, I might have not explained it correctly. I meant the permission to use / access executables which have been serialized before. The permission to access unserialized data doesn't require a permission to unserialize.
Yes, and since there might be something different which needs to be done before it's RTBC, I'd wait for a response from them. But feel free to provide an alternative patch which contains the solution you'd prefer.
Comment #15
mxh commentedI've read the comments in
node_query_node_access_alter(), but I'm not sure which comment is related to this issue and would indicate a different behavior for the ViewExecutable. Could you please point to a concrete example / comment?Comment #16
geek-merlinRegarding node_query_node_access_alter(), i meant that it is prepared for non-current-user queries:
But in fact i did not check that all ViewExecutable is prepared for that. Well, if not we can add a feature issue for that.
But important for this is the objection you stated about permissions:
Is there a user permission to "use any serialized view" in any *.permissions.yml? I did not get that if.
Comment #17
mxh commentedI brought in the term "permission" but that wasn't intended to focus the discussion on permission definitions. Although there doesn't exist a concrete permission in core about using any serialized view, there might be use cases for it.
The problem I see here is the general possibility that a user might get valid access to a view, which might have been serialized before by another user (e.g. maybe guest or admin). The
ViewExecutableclass provides the current user object, and this object must not differ from\Drupal::currentUser()IMO.Comment #18
mxh commentedI'd like to have at least the error itself getting fixed in core, thus changing status to try gaining some attention of the maintainers.
Please review the patch and give feedback whether it solves (at least partially) the serialization problem for you.
What the serialization process should include like it has been discusses above, may be of course discussed further, although it's not really part of this issue.
Comment #19
podarok#9 fixed issue with site stability for 8.3.5 core. It applied without the need to reroll.
We had an issue when cold cache page was loading, but cached page was throwing error 500
Comment #21
mxh commentedI have manually queued #9 for re-testing for 8.4.x, the test now passes (again).
@podarok thanks for your feedback. I'd like to have additional feedback from other people as well though, since this issue seems to be a complicated one and addresses multiple symptoms of this problem. Thus I'm switching to needs review (again :).
Comment #22
mxh commentedComment #23
andypostComment #24
mxh commented@andypost I think #2750463: Fatal error: Call to a member function getFieldStorageDefinition() on array is strongly related to this issue. I originally created the patch for this issue, thus the source of the problem would be the Views module there as well. Feel free to also change to proper status on this one. I still might be wrong of course.
Comment #25
andypost@mxh This looks proper fix, also I attached related issues that affected by the change
Why Serializable removed?
I think it needs inline comment explaining why we can't use DST here to prevent future errors
Comment #26
podarokThere are cases when Drupal::request() returns NULL
Which causes the
fatal error: Argument 1 passed to Drupal\views\ViewExecutable::setRequest() must be an instance of Symfony\Component\HttpFoundation\Request, null given
Comment #27
mxh commented@andypost Is following comment in from patch #9 wrong (or maybe misplaced)? :
Can you tell a concrete scenario? Drush / testing maybe? What should happen in case the request is missing? Not sure if the view is able to work without the request object, need to check.
Comment #28
podarokI've changed the line to get rid of https://www.drupal.org/node/2849674#comment-12164812
Comment #29
podarok@mxh
See docroot/vendor/symfony/http-foundation/RequestStack.php:61
Comment #30
podarok@mxh
Attaching workaround applied over #9
It is helping me to fix issue with request null so far
Comment #32
podarokNope
After hard testing of https://www.drupal.org/node/2849674#comment-12164887 found
PHP Fatal error: Call to undefined function Drupal\views\Plugin\views\query\db_and() in docroot/core/modules/views/src/Plugin/views/query/Sql.php
So the issue not only with request here
Comment #33
mxh commentedHow do you test / which unit test is affected? I'd like to reproduce.
Comment #34
andypostbtw Sometimes it happens, when there's no request see #2650434: Clearing cache via UI in translated language resets config translation of field labels to default language
But I'm not sure that ViewExecutable require request to make some job
Comment #35
andypost@podarok can you dump backtrace to get when it happens, and why there's no request?
Comment #36
podarok@andypost
Without memcache error is another
https://www.drupal.org/node/2849674#comment-12164892
Comment #37
andypostSo indeed request lost in page cache deserialization
#32 tells about missing
db_and()- that means again kernel is not bootstraped properly (includes/database.inc is not loaded yet)So all this points to wrong config for site
Comment #38
mxh commentedWrapped
\Drupal::request()call with an if (addresses #26), placed the explanation why\Serializableis not being used to the class description (addresses #25).Comment #39
mxh commentedComment #40
podarok@mxh not enough, in case if request is empty would be nice to create it from Globals as in https://www.drupal.org/node/2849674#comment-12164887
Comment #41
mxh commentedIn which case would this happen? I'd assume that, since
index.php/update.phpetc. already creates from globals: When\Drupal::request()isn't returning any request, there won't be any, except some other component of the system is either misconfigured or behaving wrong by changingRequestStack::requests.Whatever, this issue isn't actually addressing how the
ViewExecutableshould behave on deserialization. It's about solving the deserialization bug itself, which patch from #38 apparently does. Feel free to create and discuss another issue regarding howViewExecutableobjects should behave during (de-)serialization.Comment #42
podarok> In which case would this happen?
I'd be a GOD when catched this issue, but no luck so far
Atm to get this issue I have to apachebench the page with a view embedded via reference to get cache broken and to get errors from my above messages
Also memcached version way less stable in comparison to DB one...
Comment #43
mxh commentedFeel free to vote wich one you'd prefer (atm I'd still go for #38).
Just a side note: The exception added in #30 (680.diff) would never be thrown, because Request::createFromGlobals would always return an object.
Comment #44
lendudeThis is the patch in #38 merged with @andyposts test from #2701829: Extension objects should not implement \Serializable, both updated/cleaned up a bit.
Since this issue was split off from #2701829: Extension objects should not implement \Serializable, it feel a bit weird to close that as a duplicate, but it does feel like both are working on the same issue at the moment. But since there is a bit more activity here, lets start here.
Comment #45
andypostYay! Let's get this in and re-purpose #2701829: Extension objects should not implement \Serializable to it's original intent to get rid of legacy code
Comment #47
lendudeKnown random fail, #2828143: Stop tests like LocaleConfigTranslationImportTest from failing if l.d.o becomes unavailable
Comment #48
catchCould we add an @see to https://bugs.php.net/bug.php?id=66052 as well as the link to here?
Also I wonder whether we could add a kernel or unit test similar to the test case in the issue summary? Relying on batch triggering this seems a bit indirect in the current test, although probably fine to have both.
Comment #49
lendude@catch yes that sounds like much better explicit coverage for this.
Here we go, the test case from the IS converted to a kernel test. Plus a @see to the php bug
Comment #51
andypost#48 addressed so back to RTBC
Comment #54
catchCommitted/pushed to 8.5.x and cherry-picked to 8.4.x. Thanks!
Comment #57
catchI've reverted this for now, due to #2898721: Random segfault currently in FileFieldWidgetTest::testMultiValuedWidget(). This is based on probability, not sure if this issue was the commit that did it, queueing some test runs to see if we can flush it out here.
Comment #60
tacituseu commentedAnd here they are:
8.4.x https://www.drupal.org/pift-ci-job/729321
8.5.x https://www.drupal.org/pift-ci-job/729861
otherwise testbots are stable after the revert.
Comment #61
lendudeDon't mess with \Drupal in a test ......
Has to be something like this right? This has no other overlap with FileFieldWidgetTest::testMultiValuedWidget()
Comment #63
tacituseu commentedDid some testing and it turns out it isn't even the 'patch' part that is causing this, the added test coverage is enough to trigger it (see: pift-ci-job/731299)
Comment #64
lendude@tacituseu yeah I figured the same thing, so lets strip out some more stuff that might carry over into another test.....also turns out isn't needed to make the test green.
Comment #66
tacituseu commentedSo adding
user_batch_action_test_actionis enough to trigger it. (see pift-ci-job/731329)Comment #68
tacituseu commentedRunning out of ideas, but noticed couple of nits:
1.
there should be no comma after "user"
2.
can't return bool if called with
$return_as_objectset to TRUEComment #69
tim.plunkett68.1
There can be a trailing commas in annotations now
Comment #70
xjmNote that we changed the patch testing default branch to PHP 7 (see #2607222: [policy, no patch] Default to PHP 7 for Drupal core patch testing) so patches to untangle that segfault will need to be manually queued against the 5.6 environment.
Comment #71
xjmFrom @vaplas in #2898721: Random segfault currently in FileFieldWidgetTest::testMultiValuedWidget():
Comment #72
tacituseu commentedMaybe commit this with kernel test only, and leave figuring out why the
Actiontest segfaults to a follow-up.@Lendude found in the testing issue that it could be trimmed even further, and the sheer fact of registering the test module is enough to trigger it (pift-ci-job/732110).
Comment #73
tacituseu commented@xjm: quote in #71 is from @vaplas.
Comment #74
lendudeFor those interested, we are running a ton of tests in #2879048: Ignore: patch testing issue for #2919863
Note worthy stuff:
- Going overboard on garbage collection makes the problem go away https://www.drupal.org/node/2879048#comment-12202416
- Just adding an empty module will break that test, but changing the info file slightly makes it pass https://www.drupal.org/node/2879048#comment-12202738
So the problem doesn't really seem to be with this patch, it seems to be more FileFieldWidgetTest being really unstable.
I'd be for #72. Remove the batch test for now and only add the kerneltest, that gives good coverage for this change and should be stable.
Comment #75
xjmRe: #73: Oops, fixed, thanks @tacituseu.
I agree with #72; let's split the action test coverage off into a separate followup issue where we can investigate and work around the segfault, rather than holding up a bugfix on a PHP 5.6 bug that (as far as I understand) won't surface under normal site operation.
Comment #76
lendudeHere we go, the fix with just the kernel test, slightly cleaned up.
Comment #77
andyposttest only patch
Comment #78
andypostSomething wrong with list of patches
Comment #79
tacituseu commentedLooks like the static cache clear is needed for kernel test to work, changed in #64.
Comment #80
lendude@andypost++, @tacituseu++ nice catch, completely failed to check that the test-only was still red!
Here we go, this should be better.
Comment #82
andypostLooks great! Just not clear why this kind of cache clear used
Why not use
clearCachedFieldDefinitions()?Comment #83
pingwin4eg@andypost because the
clearCachedFieldDefinitions()clears persistent cache (e.g. from DB). And in the test we need to get that serialized cache data.Comment #84
tim.plunkettI'd add a comment documenting that, so someone else doesn't try to refactor it.
Comment #85
berdirAnother approach to do that would be to do a set(..., NULL) on the container for that service, then the whole service is re-instantiated. This only works well if the service isn't access indirectly through another that is still instantiated, then you'd have to unset the whole dependency chain.
Agreed about adding a comment either way, so setting to needs work for that.
Comment #86
berdirActually doing that now ;)
Comment #87
xjmWe also still need a followup for that Action test. It should be a major (since it triggers a segfault) and we should mention it on the random fail meta.
Comment #88
lendudeCreated follow up for adding the full batch bulk update test coverage #2900399: Adding test coverage for batch bulk updates, the fix for the cause of the seg fault is being explored in #2842393: Discover why gc_disable() improves update.php stability
Extended a comment to mention
clearCachedFieldDefinitions, not sure about the wording, let me know if it needs some polishing.Comment #89
Anonymous (not verified) commentedUnstable part is moved to the separate issue. So back to RTBC.
Comment #90
catchSplitting the test off is good, although I have a feeling the eventual resolution to these gc/segfault issues is going to be 'require PHP 7' when that happens.
Committed/pushed to 8.5.x and cherry-picked to 8.4.x. Thanks!