Problem/Motivation
Especially for migrate it would be really useful to be able to copy the old changed time, for example from a note.
After #2428795: Translatable entity 'changed' timestamps are not working at all we have a solution for new entities, but we still need a solution for existing entities e.g. currently you can't resave an entity with changes without updating the changed timestamp.
Proposed resolution
Skip the automatic updating of the changed time when synchronizing entities.
Ever since #2985297: Generalize the concept of entity synchronization, all entities are synchronizable. And it already provides the exact semantics that we need. Quoting \Drupal\Core\Entity\SynchronizableInterface::isSyncing():
/**
* Returns whether this entity is being changed as part of a synchronization.
*
* If you are writing code that responds to a change in this entity (insert,
* update, delete, presave, etc.), and your code would result in a change to
* this entity itself, a configuration change (whether related to this entity,
* another entity, or non-entity configuration), you need to check and see if
* this entity change is part of a synchronization process, and skip executing
* your code if that is the case.
*
* For example, \Drupal\node\Entity\NodeType::postSave() adds the default body
* field to newly created node type configuration entities, which is a
* configuration change. You would not want this code to run during an import,
* because imported entities were already given the body field when they were
* originally created, and the imported configuration includes all of their
* currently-configured fields. On the other hand,
* \Drupal\field\Entity\FieldStorageConfig::preSave() and the methods it calls
* make sure that the storage tables are created or updated for the field
* storage configuration entity, which is not a configuration change, and it
* must be done whether due to an import or not. So, the first method should
* check $entity->isSyncing() and skip executing if it returns TRUE, and the
* second should not perform this check.
*
* @return bool
* TRUE if the configuration entity is being created, updated, or deleted
* through a synchronization process.
*/
public function isSyncing();
Remaining tasks
None.
User interface changes
None.
API changes
None.
| Comment | File | Size | Author |
|---|---|---|---|
| #140 | 2329253-10.x.patch | 3.22 KB | bobooon |
Issue fork drupal-2329253
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
Comments
Comment #1
dawehnerJust a rough idea.
Comment #2
berdirWondering if skipUpdate() as a method name would suffice, instead of setSkipUpdate()...
Comment #3
dawehnerI don't have a strong opinion here, but it feels like having skipUpdate() could also return the state whether it should be skipped.
Comment #4
tstoecklerI spent some time thinking about how Migrate would actually interact with this and I really wasn't pleased with adding any
ChangedItemspecific checks to the migration system. So I thought how we could make this more generic. I don't think the "skip-update" functionality is something that is generically applicable. Instead I think we should provide a Migrate-specific interface that various objects can optionally implement if they want to provide migrate-specific logic. The attached patch is a proof-of-concept of how I think that could work. The integration on migrate side could probably be solved a bit more elegantly, but I wanted to get some feedback before burning huge amounts of time on this.Thoughts?
Comment #5
tstoecklerAhem.
Comment #6
tstoecklerDiscussed this with @dawehner a bit. He wasn't particularly fond of making the Entity system aware of the Migration system, and that's a fair point. So we discussed adding a special "don't try to be smart" mode to the entity system itself, and have
ChangedItemuse that. The Migration system can then simply enable that mode.Here's a patch for that. I called the thing "preset mode". I'm not too happy with that name, but I couldn't come with anything better.
Comment #8
tstoecklerOh ViewUI....
Comment #9
tstoecklerShould probably default to FALSE.
I'd like to hear some thoughts on this, though, before re-rolling.
Especially naming suggestions!!!
Comment #10
swentel commentedCode looks fine and yes, should definitely default to FALSE.
I'm trying to come up with better names, but no clue either, we need some native english speakers in this issue :)
Comment #11
swentel commentedHow about readOnly ? Or maybe even locked ?
Comment #12
dawehnerWell, you can certainly still change the entity, so read only or locked kind of doesn't fit.
Comment #13
tstoecklerYeah, I had readOnlyMode when initially creating the patch but then changed it because of #12. Again, totally not happy with presetMode though. This is a tough one.
Comment #14
wim leersSeveral entity types already have the concept of "locking" already, so that'd also be problematic.
What about
freeze()(orsetFrozen()) andisFrozen()?I think the reason it's so hard to find a good name for this is that what this issue is after is partial immutability, and even then it's opt-in for individual fields, the developer cannot specify the partialness of the immutability: it's essentially a blind step "in good faith". Otherwise naming would be simple:
setImmutable().Due to the above, perhaps it is better to put the burden of dealing with this on Migrate?
Comment #15
tstoecklerOohh, I like the frozen metaphor. It makes it clear that the entity is not to participate in dynamic value assignment (i.e. in some way
frozen == !dynamic). People might still assume that frozen means that in fact nothing at all can change (which is not the case), but we can clarify that with documentation I think.Wim Leers++
Comment #16
tstoecklerHere's a patch for that. Just for the record: I like dedicated, small interfaces a lot more than huge, monolithic ones. That's not what's currently prevalent in core, though, so presumably this is less controversial than something like #5.
No interdiff because I created this from scratch.
Comment #17
wim leers@tstoeckler: actually, we have
EntityOwnerInterfaceandEntityChangedInterfacealready, so it's really not that controversial. You could then even makeEntityChangedInterfaceextendEntityFrozenInterface: that'd make it clear that freezability is required.Comment #18
dawehnerIt would be actually quite cool to just let ContentEntityInterface implement that interface, and maybe just call it on the interface conditionally. Note: This allows you to drop the stupid code on ViewUI!!
Comment #19
tstoecklerOk, adapted patch for that.
Comment #20
tstoecklerRe #18: Yes, good point! Thanks. Will fix in next patch.
Comment #21
wim leersWhy this optionality? Why not do
EntityChangedInterface extends EntityFrozenInterface?Comment #22
tstoeckler@Wim Leers: Maybe someone else has some input on this as well, but my opinion (!) is the following:
While the concrete implementation of the implementors of
EntityChangedInterfacedepends on and is related toEntityFrozenInterface, the interface itself does not. It is possible that someone provides agetChangedTime()method on their entity (perhapsConfigEntity?) that works completely different than the core implementations. Now, if there were anEntityChangedTrait(*cough* #2209971: Automatically provide a changed field definition *cough) I would certainly promote usingEntityFrozenTraitfrom there, but that - at least to me - is a different story, as traits are specifically about implemenation.Comment #23
tstoecklerBump
Comment #24
mkalkbrennerHmmm,
when #2428795: Translatable entity 'changed' timestamps are not working at all gets committed, the already existing function
ContentTranslationMetadataWrapperInterface::setChangedTime()will become functional again. I assume that this one could simply be used to solve the issue here.Alternatively something like
$entity->set('changed', $timestamp);should work as well.With that related patch applied the "automation" wihtin an instance of
ChangedItemwill act like this:This algorithm also works for non-translatable entities.
Comment #25
hchonovThe patch introduced in #2428795: Translatable entity 'changed' timestamps are not working at all by @mkalkbrenner works when setting the changed timestamps manually during migration on translations, but not when initially creating the entity.
I am attaching a small patch fixing this, which should be applied after the patch from @mkalkbrenner.
Comment #26
mkalkbrennerI see the problem with new entities but the proposed solution doesn't seem to work because it turns off the normal functionality for non-translatable entities which aren't new.
Comment #27
hchonovI missed that one.... An improved version is attached...
Comment #28
mkalkbrennerI think this one is the easier solution.
Comment #29
hchonovbut like this you remove the condition check if not translatable?
Comment #30
mkalkbrennerThis check was just a shortcut to skip the algorithm below for non-translatable entities.
To achieve that, the same shortcut has to be removed either.
Comment #31
mkalkbrennerI included #28 in #2428795: Translatable entity 'changed' timestamps are not working at all. Can you give it a try?
Comment #32
mkalkbrennerhttps://www.drupal.org/node/2453153#comment-10060032 will definitely fix this issue here without any additional patch required.
Comment #36
Anonymous (not verified) commentedAs @mkalkbrenner said, this approach was included in #2428795: Translatable entity 'changed' timestamps are not working at all. Can we close this issue?
Comment #37
hchonovThere it has been solved only for new entities, but we still need a solution for existing entities e.g. currently you can't resave an entity with changes without updating the changed timestamp.
Comment #38
Anonymous (not verified) commentedThank you for clarifying the issue state, @hchonov! IS update per #37.
At first it seems that this is a logical behavior :). But the possibility of freezing the value also makes sense. May we add some $is_frozen property to class? And maybe we can use this flag like hack in #2857843: Random fail in Drupal\KernelTests\Core\Entity\ContentEntityChangedTest::testChanged too.
Comment #39
hchonovGoing though the issue over again I like the solution from #19.
Comment #42
hchonovA patch on top of #2985297: Generalize the concept of entity synchronization. Release for testing as soon as the referenced issue is committed.
Comment #43
wim leersQuoting @hchonov in #2985297-28: Generalize the concept of entity synchronization:
That patch is now committed, so let's move this forward :)
Comment #44
hchonovI've queued the patch for testing. Let's see what happens :).
Comment #45
claudiu.cristeaSo, this patch is expected to solve the 'changed' field freeze in this way?
Then, even if will achieve the goal, the
->isSyncing()naming is not self-explanatory. We are just using other functionality (sync flag) to resolve this particular case. I think we should have a dedicated method/flag.What about adding a new property to ChangedItem, along with
value? Something likepreserve(bool, defaultFALSE). Then our code would look more meaningful:EDIT: PasswordItem also uses some additional properties to act on save.
EDIT2: Or even a new method: ChangedItem::setReadOnly($flag = TRUE); that sets the flag:
Comment #46
claudiu.cristeaTill this will land, I've built a tiny module that resolves the freeze of fields of type
changedwhen explicitly is required: https://www.drupal.org/project/preserve_changed. IMO, the way this is done$node->changed->preserve = TRUEis self-explanatory compared to$entity->isSyncing(TRUE);which has no meaning in this context.Comment #48
fenstratI'd agree with @claudiu.cristea that
->isSyncing()doesn't make much sense for this use case.ChangedItem::setReadOnly($flag = TRUE);probably makes the most sense.Comment #49
larowlanOver in #2803717: Allow 'syncing' content to be updated in content moderation without forcing the creation of a new revision which is also proposing using the
isSyncingflag I proposed something like so:Then we can move away from it being a binary flag?
Comment #50
sam152 commentedI like the
SynchronizableInterfaceapproach, since it means callers can rely on syncing semantics without having to inspect or understand if an entity type has aChangedItemfield or what that field is named. I've spun off #3057483: Better describe how SynchronizableInterface should be used for content entities to try and make it clear based on the interface docs that it's actually a good choice for this exact issue.Comment #51
amateescu commentedIMO, #50 is a good arguments against a solution specific to the
changedfield type..Comment #52
hchonovWe started using the syncing semantic to prevent creations of new revisions, updating specific states and so on. We just assume that when the entity is synching then it is not in the regular state of updating. There might be a lot of cases why the entity is in a synching state, but no matter the reason it is not going to be a regular user interaction.
In all that cases the changed field should not be updated, because if we skip creating a new revision then we'll be manipulating the real time of the change performed by the user.
Comment #53
sam152 commentedI was having a look at this. I think this will probably be easier to get in if the migration changes are split and moved into: #3052115: Mark an entity as 'syncing' during a migration update.
I think it makes sense add specific tests for migrations as well as tests that check
ChangedItemin isolation, so splitting will probably reduce some complexity.Comment #54
sam152 commentedStraight reroll, may look into #53 shortly.
Comment #56
sam152 commentedI went to go write some tests for this, but on the whole the testing for
ChangedItemseemed complicated and brittle, so I've logged #3064727: Remove the REQUEST_TIME constant from ChangedItem and CreatedItem and delete ChangedTestItem in favour of a mock programmable time service in the meantime.Comment #57
sam152 commentedPatch on top of #3064727: Remove the REQUEST_TIME constant from ChangedItem and CreatedItem and delete ChangedTestItem in favour of a mock programmable time service, which delegates the migration work to another issue and adds test coverage for
ChangedItem.Comment #59
hchonov@Sam152++, this totally makes sense. The patch is small, but that are two different issues. This separation also kind of shows that we want that to happen independent of the changed item.
Comment #61
firewaller commented+1
Comment #64
golddragon007 commentedWith this, we need to allow the user to be able to choose if the date needs to be changed on user cancellation (i.e. content reassignment for Anonymous user) or not.
Comment #65
norman.lolUI could be done from contrib then. I don't think the UI for this should be provided from core as this is nothing a normal editor needs.
Comment #66
norman.lolAh sorry, on user cancellation and content reassignment, yes of course, there it makes sense.
Comment #67
golddragon007 commentedI thought a global checkbox under the account settings or a maximum per content type and not on the entity edit page.
EDIT: Ehh you were faster.
Comment #68
golddragon007 commentedThe https://www.drupal.org/project/drupal/issues/3064727 was a duplicate of https://www.drupal.org/project/drupal/issues/2809515, so I've done an update after the second issue.
I've also extracted the base of this issue for easier updates later on (it's in the do not test patch).
Comment #69
golddragon007 commentedGreat I've uploaded a 0-byte file...
Comment #70
quietone commentedComment #71
dieterholvoet commentedI re-rolled the patch.
Comment #73
muthuraman-s commentedHi,
I am also facing a similar issue updating the 1k volume of existing content. But at the same time, I don't want to update the changed time. I am working on the Drupal 8.9 Version. any suggestion?
Comment #74
norman.lolAny suggestions are:
1. Read through this issue
2. Try out the patches
3. Provide feedback
Thank you
Comment #76
weekbeforenextRe-rolled the patch from #71 for 8.9.x.
Comment #77
dieterholvoet commentedRe-rolled the patch from #71 and committed to the issue fork.
Comment #78
codebymikey commentedRerolled #71 for 9.3.x.
Needed because of #3131281: Replace assertEqual() with assertEquals()
Also replaced a reference to
\Drupal::time()with$this->time()to improve testability - hopefully it doesn't break the test.Comment #79
wim leersTested the patch locally, it works perfectly. So I next went to review this in detail:
ChangedItemis using the time service instead of theREQUEST_TIMEglobal. That is necessary to allow for testability.👍 This is the only actual functional change!
It's very simple: it wrap the existing logic in this if-statement. The existing logic already ensured we only update the
changedtimestamp when updating.This addition ensures that we only update the
changedtimestamp when not syncing.The prime example of syncing (content) entities in Drupal core are migrations.
👍 This becomes obsolete thanks to
ChangedItemno longer usingREQUEST_TIME.👍 This is proving that what we aim to do is working.
Without the first piece of code, the last assertion here will fail: even when syncing, the
changedtimestamp will have changed!👍 This is the crucial piece in this dummy/test-only
Timeservice: it allows us to control time for testing purposes 🤓… this patch is perfect! 🤩
It is hugely disruptive for migrations that involve content that has been updated upstream — i.e. this patch is pretty much essential for any migration of a relatively big Drupal 7 site.
Let's get this committed!
Created a change record: https://www.drupal.org/node/3250104
Comment #80
heddnJust a few quick nits.
Do we really need to do loose comparison (==) or can we use strict (===)? I guess that isn't a change from previous code, so maybe we can ignore it.
Nit: copy/pasta.
Can we type hint this as an int?
Comment #81
berdir1. Content Entity API is not type safe, never use === with field values unless you explicitly cast them yourself.
Comment #82
wim leers@huzooka has been working on https://www.drupal.org/project/workbench_moderation_migrate, whose goal is to migrate all of that per-revision-moderation-state-metadata into Drupal core's
content_moderationmodule. Added tag for that. Without this fixed, that migration cannot be made to work correctly.Comment #85
rclemings commentedUnless I'm missing something, I think the patch in referenced issue #3052115, which marks an entity as "syncing" during a migration update, is needed for this patch to work (#78) in that use (i.e. when updating previously migrated nodes with e.g. "drush mim upgrade_d7_node_article --update").
With that caveat, it seems to work for me.
Comment #86
rclemings commentedIt also seems to me that the title for this issue (Allow the ChangedItem to skip updating when synchronizing) could be read to mean that we want to skip updating the entity in its entirety when syncing. In fact we just want to skip updating the "changed" field for that entity. right? I've revised the title to clarify.
Comment #87
damienmckennaThis and #3052115 have not been enough for my site to fix the problem. I opened #3280140 in migrate_tools as a sort of "meta" issue because I suspect a lot of people are running into this problem because of that module.
Comment #88
wim leersLinking issues from #87 :)
Comment #90
tstoecklerFixed 2. and 3. from #80. (1. can/should not be fixed per #81.)
Didn't quite grok what's going on with the merge request so posting an updated patch.
Comment #91
damienmckennaLooking into the tests, I believe some of this data was covered up by using incorrect values in the source database.
For example, in drupal7.php node 1 has the following in the "node" table:
For comparison, "node_revisions" has the following:
This data is consistent - the original creation was 1421727515, which corresponds to values in both "file_managed" and "taxonomy_index", and it was then updated at 1441032132.
However, after the migration runs the "created" and "changed" values are set to 1529615790, which come from "entity_translation" and "entity_translation_revision". That points to errors in the source data - the time values in "entity_translation" and "entity_translation_revision" should be the same as what "node" and "node_revisions" contain.
Comment #92
damienmckennaAn example of what would be necessary to fix the test coverage:
That would correct the expectations of what node 1's data should look like after migration.
Comment #93
damienmckennaCorrection - a single run migration wouldn't trigger this, the problem is when an existing entity is updated, and I don't think core's upgrade path cares about that. However, the discrepancy in the test data is still confusing.
Comment #94
damienmckennaI opened a separate issue for the data inconsistency problems: #3303925: D7 upgrade migration test shows incorrect node created, updated values are migrated
Comment #95
wim leers… but it should, shouldn't it? 🤔
Comment #96
junaidpvExactly same as #90, just re-rolled for 9.4.x
Comment #98
junaidpvAgain, re-rolled for 9.4.9
Comment #99
wim leersThank you, @junaidpv! 😊
What is in the way of an RTBC here? 😅 @heddn's nits from #80 were addressed by @tstoeckler in #90 … which means this actually has been ready for >6 months! 🫣
Comment #100
wim leersActually, looks like this needs a reroll for
10.1.x(this won't land in9.5.xmost likely).Comment #101
Gunjan Rao Naik commentedadded patch against #97 in 10.1.x
Comment #102
wim leersThanks!
Comment #103
longwavePHPStan is complaining about various errors in #101.
Comment #104
berdirIt's complaining because this replaces a lot of REQUEST_TIME usages, so the ignore file needs to be updated.
Comment #106
bhanu951 commentedGot hit again by this issue during migration. Worked on re-roll.
Test
core/tests/Drupal/KernelTests/TestTime.phpis missing in patch #101.Rerolled #101 against 11.x and added missing test.
Comment #107
mglamanThese errors need to be removed from the baseline to prevent failures.
Comment #108
bhanu951 commentedComment #109
smustgrave commentedSeeing a lot of changes for $this->value = REQUEST_TIME; and reading the proposed solution not super clear how they are connected. If that could please documented.
Comment #110
luenemannUpdating the proposed solution to document the changes of
$this->value = REQUEST_TIME;, see #79.Comment #111
needs-review-queue-bot commentedThe Needs Review Queue Bot tested this issue. It no longer applies to Drupal core. Therefore, this issue status is now "Needs work".
This does not mean that the patch needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.
Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.
Comment #112
wim leersRelated:#3052115: Mark an entity as 'syncing' during a migration update landed 👍
Comment #113
bhanu951 commentedSeems #111 is a false positive as patch applies cleanly against 11.x in #108.
Setting again for needs review
Comment #114
needs-review-queue-bot commentedThe Needs Review Queue Bot tested this issue. It no longer applies to Drupal core. Therefore, this issue status is now "Needs work".
This does not mean that the patch needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.
Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.
Comment #115
nod_patch failing to apply is legit. the green test was two weeks ago, requeued one that failed like the bot.
Comment #116
bhanu951 commentedRerolled #108 against latest 11.x Head.
Comment #118
smustgrave commentedTested by using
Verified date changed
Applied MR 4203
Verified date is the same.
Comment #119
longwaveThis seems out of scope, I don't think we need to change CreatedItem in this patch.
A lot of the test changes seem out of scope as well.
Comment #120
longwaveComment #121
bhanu951 commentedComment #122
casey commentedTo be able to update the change time during syncing the EntityChangedConstraintValidator must also be updated
Comment #123
casey commentedWrong patch
Comment #125
mikelutzComment #126
mikelutzUpdated the MR, fixed conflicts, and added in the change to the validator in the patch in #123 Still need to address @longwave's comments in #119
Comment #127
piotr pakulskiRe-rolling the #116 for 10.2.5
Comment #128
rgpublicProbably too late to say this, but as much as I understand the reasoning in #50, I still think a solution like in #45 would still make sense *additionally*. IMHO there should be a chain: isSyncing() should internally rely on sth. like #45. Even in the issue summary it says "f.e.(!) when migrating". Now, the solution is ONLY about migrating. What if I just want to skip updating the changed date? I don't really know what setSyncing() might do now or in the future. I still think there's a valid reason to just want to skip changing the changed date. With the solution outlined here (AFAIU) I would have to pretend/claim that I'm syncing sth, when in fact I'm not.
Comment #129
johnpitcairn commented@rgpublic: I totally agree.
We often find ourselves in the situation of wanting to update entity field values in a deploy hook. Changed assumptions, bad data entry, whatever.
Or admins need a way to generate and save a reference to an external api entity that failed to create (flaky api, whatever).
In both cases we don't want to mess with the changed timestamp (that just confuses editors), and we are not "syncing" a migration.
Comment #132
kksandr commented@longwave I opened a new merge request #8536, it has minimal changes.
All refactoring/optimization was removed, I also removed the validation omission during synchronization, this seemed unrelated to the problem.
But manually changing the request time instead of waiting in ChangedTestItem looked interesting, it might speed up testing a bit.
Comment #134
smustgrave commentedRan the test-only feature
Which shows the coverage
Applied a nitpicky change for the void return.
Issue summary appears fine to me
LGTM
Comment #135
quietone commentedI read the issue summary and the comments and the latest MR. The first thing I noted is that the proposed resolution is out or date, it no longer touches the time service. And then at the end of the comments a new MR was created that removed code. Unfortunately, I haven't found a way to compare 2 MR with the UI so it is difficult to confirm those changes.
There were discussion here of alternate ideas as well as wanting the solution to apply to more that just the 'changed' field. #45, #49. $51, #128, #129. Is a follow up needed for those? I am not sure.
I have updated the issue summary.
Leaving at RTBC
Comment #136
larowlanUpdating issue credits
Comment #138
larowlanCommitted to 11.x
Per https://www.drupal.org/about/core/policies/core-change-policies/allowed-... this isn't eligible for backport to 11.0.x.
Glad to see this one resolved.
Comment #140
bobooon commentedRe-rolled for 10.3.x
Comment #142
joelpittetThanks, @robbiehobby — the D10 patch worked great and completely bypassed the issue (good in our case)!
We encountered this because Link Checker bot modified some links on nodes, resulting in multiple revisions sharing the same changed timestamp (as expected from an automated bot). At the same time, we had multilingual support enabled (due to search_api_solr dependencies), and we were also running
d7_node_completemigrations.The root of the problem was the comparison
$this->value == $original_value, which assumes the values are different. In our case, up to six revisions shared the same changed timestamp!During migration, this caused confusion: the second revision would get the current timestamp, the third would revert to the original migrated timestamp, the fourth would get the current one again, and so on—because the
$originalrevision matched a previous one with the same changed value.