Problem/Motivation

When using Drupal\Core\Entity\ContentEntityBase::toArray() to create an array of an entity, the resulting array cannot be used to create a new entity with entity_create().

$node = \Drupal\node\Entity\Node::load(4);
$data = $node->toArray();

unset($data[$node->getEntityType()->getKey('id')]);
unset($data[$node->getEntityType()->getKey('revision')]);
unset($data[$node->getEntityType()->getKey('uuid')]);

$node = entity_create('node', $data);
$node->save();

Doing so results in the following error:

Fatal error: Call to a member function getFieldStorageDefinition() on a non-object in core/lib/Drupal/Core/Entity/ContentEntityBase.php on line 972

Proposed resolution

Ensure that Drupal\Core\Entity\EntityInterface\ContentEntityStorageBase::doCreate() can accept a bundle from an array rather than a string.

Remaining tasks

  1. Test Patch
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

davidwbarratt’s picture

Title: Drupal\Core\Entity\ContentEntityBase::toArray does not output an array of values that can be resaved with entity_create » Drupal\Core\Entity\ContentEntityBase::toArray() does not output an array of values that can be resaved with entity_create()
Berdir’s picture

Status: Active » Postponed (maintainer needs more info)

What exactly did you do?

Blind guess is something with the bundle is not right.

davidwbarratt’s picture

Status: Postponed (maintainer needs more info) » Active

I'm working on a contrib module that exports entities as YAML and then imports them back in to another environment (I couldn't find a module like it, so I thought I'd write one).

Here's an over-simplified version of the problem:

$node = \Drupal\node\Entity\Node::load(1);
$data = $node->toArray();
$node = entity_create('node', $data);
$node->save();

When you execute $node->toArray() the resulting output is:

Array
(
    [nid] => Array
        (
            [0] => Array
                (
                    [value] => 1
                )

        )

    [uuid] => Array
        (
            [0] => Array
                (
                    [value] => 0d2e663e-3a9e-4bdf-b247-710fbad93b25
                )

        )

    [vid] => Array
        (
            [0] => Array
                (
                    [value] => 1
                )

        )

    [type] => Array
        (
            [0] => Array
                (
                    [target_id] => article
                )

        )

    [langcode] => Array
        (
            [0] => Array
                (
                    [value] => en
                )

        )

    [title] => Array
        (
            [0] => Array
                (
                    [value] => Some Article
                )

        )

    [uid] => Array
        (
            [0] => Array
                (
                    [target_id] => 1
                )

        )

    [status] => Array
        (
            [0] => Array
                (
                    [value] => 1
                )

        )

    [created] => Array
        (
            [0] => Array
                (
                    [value] => 1425082505
                )

        )

    [changed] => Array
        (
            [0] => Array
                (
                    [value] => 1425084323
                )

        )

    [promote] => Array
        (
            [0] => Array
                (
                    [value] => 1
                )

        )

    [sticky] => Array
        (
            [0] => Array
                (
                    [value] => 0
                )

        )

    [revision_timestamp] => Array
        (
            [0] => Array
                (
                    [value] => 1425082505
                )

        )

    [revision_uid] => Array
        (
            [0] => Array
                (
                    [target_id] => 1
                )

        )

    [revision_log] => Array
        (
            [0] => Array
                (
                    [value] => 
                )

        )

    [path] => Array
        (
        )

    [body] => Array
        (
            [0] => Array
                (
                    [value] => <p>Lorem ipsum dolor sit amet, ex pri amet legimus suscipiantur, wisi munere voluptua eu vel, dictas nusquam cotidieque eos ad. Agam summo pro ut. Postea latine reprehendunt cum id. Pri ut ignota dictas mentitum, eos mazim tollit accusam ei.</p>

<p>Eu duo facer commune, at vim deseruisse sententiae. Cu electram assentior est, propriae invidunt referrentur duo ei. Cu per erat nulla, ius perfecto consequat et. Ea dicit ignota suscipit quo.</p>

<p>Soleat scribentur vel ne, malis dicunt ad pri, nonumes blandit fastidii no per. Cu nec fabulas epicurei persequeris, facer solet in eos, id meis simul mel. Vel eius vivendo te. Per te eius vocent. Sed et consul alterum appetere. Ut ipsum noluisse partiendo mei.</p>

<p>Id legimus fastidii vituperata quo, ferri rationibus mea ad. In nec commune concludaturque, ex his nisl vocent. Dicit probatus mandamus mea ut. Te discere detraxit vim, ad albucius deleniti eam. No agam paulo his, quo ut illud causae noluisse, has denique referrentur te.</p>

<p>Abhorreant appellantur comprehensam et per, at mel graecis deserunt. Mel cu saperet deleniti repudiandae, iudicabit consectetuer per eu. Mea an ullum facilisis, ius te graeco commodo equidem. At pri persius invenire, mei ornatus tacimates persecuti ei. Mel ad erant admodum adipisci, facete contentiones te eos.</p>

                    [summary] => 
                    [format] => basic_html
                )

        )

    [comment] => Array
        (
            [0] => Array
                (
                    [status] => 2
                    [cid] => 0
                    [last_comment_timestamp] => 1425082548
                    [last_comment_name] => 
                    [last_comment_uid] => 1
                    [comment_count] => 0
                )

        )

    [field_image] => Array
        (
            [0] => Array
                (
                    [target_id] => 1
                    [alt] => 
                    [title] => 
                    [width] => 640
                    [height] => 640
                )

        )

    [field_tags] => Array
        (
            [0] => Array
                (
                    [target_id] => 1
                )

            [1] => Array
                (
                    [target_id] => 2
                )

        )

)

I believe the problem is, is that properties of the entity are in multivalue arrays, and entity_create() expects them to just be values.

I was thinking that when Drupal\Core\Entity\ContentEntityBase::toArray() creates the array, it should check to see if the value is a property or field and output the array accordingly.

Berdir’s picture

Everything is a field.

Yes, I'm pretty sure that it's the bundle detection that is flawed. ContentEntityStorageBase::doCreate() assumes that it's just a string. We can try to make that code more generic, but the easiest solution for you is probably just to special case that.

if ($node->getEntityType()->hasKey('bundle')) {
  $data[$node->getEntityType()->getKey('bundle')] = $node->bundle();
}

Untested, but that should do the trick and is generic.

davidwbarratt’s picture

Title: Drupal\Core\Entity\ContentEntityBase::toArray() does not output an array of values that can be resaved with entity_create() » Drupal\Core\Entity\EntityInterface\ContentEntityStorageBase::doCreate() assumes that the bundle is a string
Issue summary: View changes

#4,

That did the trick!

I actually changed it around so it would be on import rather than export:

$entity_type = \Drupal::entityManager()->getDefinition('node');

if ($entity_type->hasKey('bundle')) {
  $data[$entity_type->getKey('bundle')] = $data[$entity_type->getKey('bundle')][0]['target_id'];
}

You can see the whole thing here:
https://gist.github.com/davidbarratt/5aace2d13f54e511698c

davidwbarratt’s picture

Issue summary: View changes
davidwbarratt’s picture

Issue summary: View changes
Status: Active » Needs review
FileSize
873 bytes

Attached is a patch that fixes the problem. :)

davidwbarratt’s picture

FileSize
867 bytes

whoops... let's get rid of that whitespace.

Berdir’s picture

Issue tags: +Needs tests

That's not generic enough, the bundle is not always in a target_id property. EntityTest for example doesn't use a reference field, so it is in 'value'.

The best way might be to just reset() in a loop until we no longer have an array.

We also need tests.

Berdir’s picture

Btw, your code is flawed for the same reason. Mine works no matter how the bundle is stored, because it uses the API.

Also, you could also use the serialization module for this instead of your own implementation. See \Drupal\hal\Tests\EntityTest for examples. That allows entity and field types to customize the process.

Berdir’s picture

Status: Needs review » Needs work
davidwbarratt’s picture

You're right, I did make an assumption that it's always target_id, thanks for the information, I didn't understand that sometimes it's value.

However, while my code does make an assumption, it is not inherently incorrect.

The problem is, is that there is no API to determine if target_id or value should be used.

I looked into the Serialization module, that yes, that's basically what I was trying to do manually, so it's nice to see that there's an API for that. :)

However, the module suffers from the same problem. I rewrote my import/export test to use Serialization:
https://gist.github.com/davidbarratt/a1e805aef086d535b8c3

The export works perfectly, but when you try and import you get this error:

exception 'Drupal\Core\Entity\EntityStorageException' with message 'Missing bundle for entity type node' in core/lib/Drupal/Core/Entity/ContentEntityStorageBase.php:63

If you look in Drupal\serialization\Normalizer\EntityNormalizer::denormalize() you'll see this line:

  $type = $data [$bundle_key][0]['value'];
  $data[$bundle_key] = $type;

It looks like Drupal is making the same assumption I did, but in reverse. We either need to have an API to determine the bundle key from the array, or ensure that Drupal\Core\Entity\EntityInterface\ContentEntityStorageBase::doCreate() (and any other method that would get the method from the array) properly handles either case.

davidwbarratt’s picture

Status: Needs work » Needs review
Related issues: +#1892320: Add deserialize for JSON/AJAX
FileSize
1.7 KB

I've attached a patch that loops through the array until a non-array is reached. I've also removed the work-around that was added in #1892320: Add deserialize for JSON/AJAX.

I haven't added any tests yet (if needed), just wanted to make sure it looks good to everyone first.

davidwbarratt’s picture

FileSize
1.69 KB

damn you whitespace!

The last submitted patch, 13: create_bundle-2443165-13.patch, failed testing.

Status: Needs review » Needs work

The last submitted patch, 14: create_bundle-2443165-14.patch, failed testing.

davidwbarratt’s picture

Status: Needs work » Needs review
FileSize
3.6 KB

Needed to update the texts since there are two methods that are no longer called. Also, the $test_data remains the same throughout the test. :)

davidwbarratt’s picture

Does this still need tests or are the existing ones sufficient?

Berdir’s picture

Status: Needs review » Needs work
  1. +++ b/core/lib/Drupal/Core/Entity/ContentEntityStorageBase.php
    @@ -62,7 +62,14 @@ protected function doCreate(array $values) {
    +      if (is_array($bundle)) {
    +        do {
    +          $bundle = reset($bundle);
    +        } while (is_array($bundle));
    +      }
    

    That's a creative way of writing a normal while loop as a do/while.. :)

    We also need a comment here, something like this:

    The bundle could be a string or it might a field based structure. We can rely on it being a single string, so remove all wrapping arrays until we have that string.

    Yes, I I think we need explicit tests for this.

  2. +++ b/core/modules/serialization/src/Normalizer/EntityNormalizer.php
    @@ -50,14 +50,6 @@ public function denormalize($data, $class, $format = NULL, array $context = arra
    -    // The bundle property behaves differently from other entity properties.
    -    // i.e. the nested structure with a 'value' key does not work.
    -    if ($entity_type->hasKey('bundle')) {
    -      $bundle_key = $entity_type->getKey('bundle');
    -      $type = $data[$bundle_key][0]['value'];
    -      $data[$bundle_key] = $type;
    -    }
    -
    

    Grml. json serialization is crap.

    I recommend to use hal+json from the hal.module instead. Doesn't mean that we shouldn't fix this, of course.

davidwbarratt’s picture

FileSize
3.74 KB

I've updated the while() loop, but I couldn't figure out where to add the test. I actually don't see any tests for ContentEntityStorageBase. Am I missing something? Do we need to create a new test file?

davidwbarratt’s picture

Status: Needs work » Needs review
yched’s picture

Rerolled, fixed the comment wrapping and adjusted the wording a bit.

Other than that - a bit ugly indeed, but no better suggestion on my side :-/

davidwbarratt’s picture

Issue tags: -Needs tests
FileSize
6.15 KB

I've added some tests. There isn't really much to test since the Drupal\Core\Entity\EntityInterface does not have a __construct. I'm not sure how we can actually rely on an entity (content or config) implementing the constructor in the same way we are calling it. :/

Perhaps the class calling this class should provide an empty entity class? or perhaps the __construct should be added to the interface?

Then again, there is no way to enforce scalier values (string) in PHP 5.4 (that I am aware of)

mgifford’s picture

Status: Needs review » Needs work

Patch no longer applies.

andypost’s picture

Issue tags: +Needs reroll
kostyashupenko’s picture

Status: Needs work » Needs review
FileSize
5.56 KB

Rerolled patch #23

andypost’s picture

Issue tags: -Needs reroll
andypost’s picture

+++ b/core/modules/serialization/src/Normalizer/EntityNormalizer.php
@@ -50,14 +50,6 @@ public function denormalize($data, $class, $format = NULL, array $context = arra
-    // The bundle property behaves differently from other entity properties.
-    // i.e. the nested structure with a 'value' key does not work.
-    if ($entity_type->hasKey('bundle')) {
-      $bundle_key = $entity_type->getKey('bundle');
-      $type = $data[$bundle_key][0]['value'];
-      $data[$bundle_key] = $type;
-    }

lost in re-roll

Status: Needs review » Needs work

The last submitted patch, 26: 2443165-26.patch, failed testing.

HOG’s picture

Status: Needs review » Needs work

The last submitted patch, 30: create_bundle-2443165-30.patch, failed testing.

davidwbarratt’s picture

I don't understand the test failure...

davidwbarratt’s picture

Status: Needs work » Needs review

Hopefully kicking off the test bot to get an error message that makes sense.

davidwbarratt’s picture

I opened #2674428: Php Core dumps should be more visible to end users to hopefully get a better explanation of the failure.

tstoeckler’s picture

FWIW, I don't think this is a particularly good idea. If we want to support multiple possible inputs we should do so more explicitly. The current patch allows, for example Node::create('type' => [[[[[[[[[['article']]]]]]]]]]). These are the sort of things that - in my opinion - come to bite us years from now because this will hide some other obscure bug that would generate obscenely nested arrays.

Edit: fix example code

davidwbarratt’s picture

#36,

Right now, there isn't a way to serialize an entity, and then unserialize it without changing the bundle to a string. It's the only field that has this problem. :(

I don't really care how we fix this, but you should be able to execute the test code in the issue description and it should work without error.

tstoeckler’s picture

OK, sorry, maybe that was unclear.

I'm totally fine with supporting all of the below:

$property_value = 'article';
Node::create('type' => $property_value);
$field_item_value = ['value' => 'article'];
Node::create('type' => $field_item_value);
$field_item_list = [['value' => 'article']];
Node::create('type' => $field_item_list);

which - unless I am mistaken - is all the variants we support for all other fields, as well. But we IMO we should support those explicitly and not arbitrarily deeply nested arrays, which is what the current patch does.

But of course me having that opinion does not mean that it has to done that way :-)

Mixologic’s picture

If you click on the test results->view results on dispatcher->console log->console log full, there you will see the culprit(s):
https://dispatcher.drupalci.org/job/default/84494/consoleFull

00:17:26.259 FATAL Drupal\Tests\Core\Entity\ContentEntityStorageBaseTest: test runner returned a non-zero error code (2).

00:17:46.192 PHP Fatal error:  Call to a member function create() on a non-object in /var/www/html/core/modules/serialization/src/Normalizer/EntityNormalizer.php on line 88
00:17:46.209 Drupal\Tests\serialization\Unit\Normalizer\EntityNormalizerT   0 passes   1 fails      

00:17:46.561 FATAL Drupal\Tests\serialization\Unit\Normalizer\EntityNormalizerTest: test runner returned a non-zero error code (255).
00:17:46.563 Drupal\Tests\serialization\Unit\Normalizer\EntityNormalizerT   0 passes   1 fails     

Those are all essentially segmentation faults or something like it. Whatever you are doing in that test, its causing phpunit to crash, hard.

Sorry that the error is "PHPunit Test failed to complete " -> thats actually in core and is what the testbots tell you: http://cgit.drupalcode.org/drupal/tree/core/modules/simpletest/simpletes...

Version: 8.0.x-dev » 8.1.x-dev

Drupal 8.0.6 was released on April 6 and is the final bugfix release for the Drupal 8.0.x series. Drupal 8.0.x will not receive any further development aside from security fixes. Drupal 8.1.0-rc1 is now available and sites should prepare to update to 8.1.0.

Bug reports should be targeted against the 8.1.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.2.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

xjm’s picture

Issue tags: +D8 major triage deferred

The core committers and Field API maintainers discussed this issue today. We decided to defer triage since we did not really have consensus on whether the issue was a bug vs. a task and whether it was major vs. normal. Both Entity system maintainers considered it a bug despite being developer-facing only, and emphasized that it was a very common usecase for the API, so leaving at major to revisit later.

Version: 8.1.x-dev » 8.2.x-dev

Drupal 8.1.9 was released on September 7 and is the final bugfix release for the Drupal 8.1.x series. Drupal 8.1.x will not receive any further development aside from security fixes. Drupal 8.2.0-rc1 is now available and sites should prepare to upgrade to 8.2.0.

Bug reports should be targeted against the 8.2.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.6 was released on February 1, 2017 and is the final full bugfix release for the Drupal 8.2.x series. Drupal 8.2.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.3.0 on April 5, 2017. (Drupal 8.3.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.3.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

hkirsman’s picture

This once fixed a save error in Paragraphs module which I can't reproduce at the moment (will post if it comes up again). Must have been a very specific case. I can't apply the patch to Drupal 8.3 so could anybody clarify what's the status of this? Is it somehow fixed from. How serious is this bug? Thank you!

Version: 8.3.x-dev » 8.4.x-dev

Drupal 8.3.6 was released on August 2, 2017 and is the final full bugfix release for the Drupal 8.3.x series. Drupal 8.3.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.4.0 on October 4, 2017. (Drupal 8.4.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.4.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.4 was released on January 3, 2018 and is the final full bugfix release for the Drupal 8.4.x series. Drupal 8.4.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.5.0 on March 7, 2018. (Drupal 8.5.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.5.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.6 was released on August 1, 2018 and is the final bugfix release for the Drupal 8.5.x series. Drupal 8.5.x will not receive any further development aside from security fixes. Sites should prepare to update to 8.6.0 on September 5, 2018. (Drupal 8.6.0-rc1 is available for testing.)

Bug reports should be targeted against the 8.6.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

amateescu’s picture

I just bumped into this problem while working on another patch :/ It would be nice to have it fixed, major or not.

Here's a new patch, written from scratch to take into account @tstoeckler's feedback from #37.

The last submitted patch, 47: 2443165-47-test-only.patch, failed testing. View results

andypost’s picture

Version: 8.6.x-dev » 8.7.x-dev
Status: Needs review » Reviewed & tested by the community

Looks good to go

alexpott’s picture

Status: Reviewed & tested by the community » Needs work
  1.   /**
       * @covers ::create
       */
      public function testReCreate() {
        $storage = $this->container->get('entity_type.manager')->getStorage('entity_test');
    
        $values =  $storage->create(['type' => 'test_bundle'])->toArray();
        $entity = $storage->create($values);
        $this->assertEquals('test_bundle', $entity->bundle());
      }
    

    I think it is worthwhile to have a test something like the above to prove that we don't regress the reported bug in some other way.

  2. +++ b/core/lib/Drupal/Core/Entity/ContentEntityStorageBase.php
    @@ -85,7 +85,22 @@ protected function doCreate(array $values) {
    +      // Normalize the bundle value.
    +      $bundle_value = $values[$this->bundleKey];
    +      if (!is_array($bundle_value)) {
    +        // The bundle value is a scalar, use it as-is.
    +        $bundle = $bundle_value;
    +      }
    +      elseif (is_numeric(array_keys($bundle_value)[0])) {
    +        // The bundle value is a field item list array, keyed by delta.
    +        $bundle = reset($bundle_value[0]);
    +      }
    +      else {
    +        // The bundle value is a field item array, keyed by the field's main
    +        // property name.
    +        $bundle = reset($bundle_value);
    +      }
    

    Is it worth having a static method that does this in a more generic way. I.e. do we do something similar anywhere else?

amateescu’s picture

Status: Needs work » Reviewed & tested by the community
FileSize
3.05 KB
783 bytes

Thanks @alexpott for the review :)

Re #50:

1. That's a very good idea, added the test to the patch.

2. We have \Drupal\Core\Field\FieldInputValueNormalizerTrait::normalizeValue() with a purpose similar to the code from this patch, but that helper does a lot more work than we need because it is normalizing a field value in an array of field properties keyed by delta, and we just need the scalar value instead. The code here is also more performant because it doesn't require any knowledge about the definition of the bundle field in order to get its main property name.

However, to answer your initial question, FieldInputValueNormalizerTrait::normalizeValue() is the method that should be used in most cases.

alexpott’s picture

Status: Reviewed & tested by the community » Needs work
+++ b/core/lib/Drupal/Core/Entity/ContentEntityStorageBase.php
@@ -85,7 +85,22 @@ protected function doCreate(array $values) {
+      // Normalize the bundle value.

Let's add a code comment here to point out this is an optimised version of \Drupal\Core\Field\FieldInputValueNormalizerTrait::normalizeValue() so people don't do copy pasta unnecessarily.

amateescu’s picture

Status: Needs work » Reviewed & tested by the community
FileSize
3.2 KB
845 bytes

Sure thing, expanded the comment :)

alexpott’s picture

Status: Reviewed & tested by the community » Fixed

Committed and pushed c594d13e25 to 8.7.x and 44c3648423 to 8.6.x. Thanks!

Backported to 8.6.x because this is a bugfix that does not have an API impact. The original code path of providing a scalar value is unaffected.

  • alexpott committed c594d13 on 8.7.x
    Issue #2443165 by davidwbarratt, amateescu, HOG, kostyashupenko, yched,...

  • alexpott committed 44c3648 on 8.6.x
    Issue #2443165 by davidwbarratt, amateescu, HOG, kostyashupenko, yched,...

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.