Block plugins need to be able to update their settings in multiple different storage implementations

#4 I haven't looked at Dashboard module internals, but pretty sure that also needs the same treatment.

Confirm, we would need to do the same, and we are not doing anything for this scenario (in this case I'd expect layout builder implementation to handle this for everyone using a SectionStorage plugin).

#0 Conversely, updates are not the only place that this logic needs to run, it also needs to run on presave of all these different config entity types to support shipped config too.

IMHO this will be complex enough that it could be left for a follow-up, and AFAIK we have no precedent of this. Looks like a related-but-not-the-same separate beast.

E.g. if node module implements a config schema change on node_type, the shipped node types config of contrib module X are not updated on installation of module X.

#8 @longwave mentioned in slack the possibility of generalising this to all plugins that have settings.

If we include shipped config on the scope, and per my comment above, wouldn't this be a general problem for any config that has an update to its structure/config schema, and not only plugins?

And I'd expect we would need some kind of tracking of the last hook_update_N/post_update where this shipped config was generated with, which might mean a huge change with no BC layer.

if node module implements a config schema change on node_type, the shipped node types config of contrib module X are not updated on installation of module X

We're starting to tackle that now with pre_save hooks (see issue below)

I've opened #3521618: Add generic interface + base class for upgrade paths that require config changes which is sort of similar to this issue but for config entity changes specifically.

WRT. this issue, Layout builder is going to be really tricky here for overridden entities. That config is stored as a blob in the database. It's a massive headache to update these and requires an entity save or some DB voodoo.
I wonder if instead we have something that applies default config upon loading the plugin? That way when a user edits an entity that has a block plugin in its overridden LB storage, it'll get those config changes applied and then when they save the changes will persist to the db?

> a just-in-time update on entity presave

This also has to take place on entity load, because we need to fix up data so it can be handled correctly by anything that wants to consume it - the renderer or whatever else needs the entity; it might never even be saved again without some kind of bulk/batch/queue mechanism. But doing all these checks on load likely comes with a performance penalty, especially if the amount of updates grows over time.

we already sort of have a pattern for just-in-time updates in the phpass module

Fascinating! 🤓

Ideally we'd have a way to detect which entities and revisions need updating

Indeed! That's where #3523841: Versioned Component config entities (SDC, JS: prop_field_definitions, block: default_setting, all: slots for fallback) + component instances refer to versions ⇒ less data to store per XB field row is part of the solution, or Experience Builder at least — to know when either a component has changed its "explicit input schema" (SDC: props defined in JSON Schema, block plugin: default configuration and config schema — these require an update path to be provided by the module that provides them), or when the way that a site decides to store data for an SDC (e.g. change which field type is used for storing images — this requires XB itself to figure out the update path — related: #3509115-8: [META] Plan to allow editing props and slots for in-use code components).

@longwave asked what I was going to ask in #16, and #17 kind of blows my mind 🤯😄

FYI: test coverage is under way at #3501708: Prove that it *will* be possible to apply block settings update paths (assuming #3521221 in core) to stored XB component trees in config/content, which should help vet whatever we come up here — because the test infrastructure that introduces should be trivially updatable to first expect component instance validation errors (and rendering failure) as they do now, but then after applying the update path infrastructure that this issue adds, they should expect success.

Interesting ideas, but as always, things get a bit more complicated with contrib, with the main challenge being release management/timing where an event invoked by core might be not be the right solution.

A practical example is when we deprecated and replaced the node_type condition plugin with the generic entity bundle plugin derivates. Core provided an upgrade path for block configs. pathauto also uses condition plugins to control which patterns apply to which entity. I did write an update function there, but there were still a bunch of people that did run into problems because they either had old default config in install profiles/modules, might have updated pathauto before core, or reimported fixed config after running updates (a surprisingly common issue).

A solution where a block plugin can trigger an update for all possibly afffected places is IMHO extremely hard, because it would require that people update all contrib and custom code at the same time as core introduces that new update, so those implementations actually get triggered. I think that's not realistic and more complex solutions like discovering those conversion handlers/plugins doesn't seem easier than each implementation writing its own update function. Maybe it could be useful to have some apis on plugins to deal with the conversion, but that won't handle a case when a whole plugin gets deprecated, I guess.

Also, what navigation does with block plugins isn't even config entities, just a single simple config file, so generic config entity based approaches wouldn't work there (fwiw, I think what navigation does is wrong, you shouldn't manage block config in simple config, but that's a different topic)

There is an additional hard problem in the case of interdependent entities: if content of entity type A depends on some content of entity type B, and both receive some kind of structure update using this mechanism, which order do the updates run in? Does the update code for entity type A have to deal with the old or new structure of entity type B? Perhaps this is a hypothetical issue for now but as @berdir mentioned contrib I am sure we will run into this case sooner or later, if what we are effectively trying to here is provide long term BC for data structures.

This surfaced again in Dashboard's #3557882: Support "placeholder" dashboard blocks.

Had a thought on how the just in time updates for blocks could work, kinda derived from the database updates system:

• Introduce a new attribute, say SettingsUpdate, which has one property schemaVersion and make the attribute applicable to methods
• Add schema_version as a property on the block_settings config schema, and set it as requiredKey: false
• On block load, run reflection on the the block plugin class looking for all methods with the SettingsUpdate attribute, and invoke any of those methods whose schemaVersion is greater than the schema_version stored in the block config, or run all SettingsUpdate methods if there is no schema_version stored in the block config
• For Layout Builder, instead of on block load, this might need to happen in SectionComponent::getPlugin()?
• As each method is run, the schema_version is updated to match
• Since historically block plugin settings haven't been updated a lot, it might be fine to keep the SystemUpdate methods around a long time, or if needed, could also introduce a similar mechanism to remove old methods and set a last removed update. For example, the last removed update could be a separate attribute applied on the block class

I think the check for and running outstanding SettingsUpdate methods would have to occur on every block load, so there might be a performance consideration.

Related: SDCs may gain new required props.

Canvas MUST solve tackle this. Issue: #3568602: Handle upgrading and rendering not yet upgraded component instances of components with new(ly) required props that include an example value.

I've opened draft MR 14553 as a PoC of #25.

Some notes:
Attribute class is ConfigUpdate instead of SettingsUpdate, but whatever, naming things is hard

• The block config update method (updateSchema) is called from BlockPluginTrait::setConfiguration, which runs every time a block plugin is instantiated
• updateSchema could possibly just be called from BlockPluginTrait::__construct instead, not sure
• An alternate thought I had to avoid iterating through all a block plugin's methods to find an attribute would be to have an attribute applied to the class instead, for example:
  

  #[ConfigUpdates([
    11400 => 'updateMethod1',
    11500 => 'updateMethod2',
    // ...
  ]
  class TestBlock extends BlockBase {
  
    public function updateMethod1(): void {
          // ...
    }
  
    public function updateMethod2(): void {
      // ...
    }
  }
  

  But I think the DX would be a little worse

In the MR, I have created a test block plugin test_block_display_message in test module block_test_config_update. It has one text property display_message, and when the block is viewed, it displays the content of that property, as well as a form (more on that in a bit).

Based on a property in keyvalue, the class implementing the block plugin is swapped and the block config schema is changed as well. There are two alternate classes:
TestDisplayMessageConfigUpdateBlock which adds a ConfigUpdate method and a second text property that gets displayed.
TestDisplayMessageConfigUpdate2Block which adds two ConfigUpdate methods and another text property that gets displayed. In addition, the setThirdDisplayMessage() config update method removes the config property introduced by TestDisplayMessageConfigUpdateBlock. The intent of the changeDisplayMessage() config update method is demonstrate that it does not run if the schema was already updated and saved as a higher version from TestDisplayMessageConfigUpdateBlock

So initially when the block is placed in any of the global block layout, navigation layout, LB defaults, or LB overrides, the contents of the 'display_message' property is shown along with a form that shows an "Update block config schema" button. When pressed, the button sets the keyvalue property to switch the block plugin class to TestDisplayMessageConfigUpdateBlock and flushes all caches. After that happens, the display of the block is updated to show the content of the display_message property as well as the added second_display_message property whose value was set in the setSecondDisplayMessage() config update method. In addition, the button has been changed to "Update block config schema again". If pressed, block plugin class is changed to TestDisplayMessageConfigUpdate2Block, which removes the second_display_message property, and adds and sets the third_display_message property. Note that none of these property changes are saved without explicitly saving the block.

Last couple notes:

• For a block display that is already cached, there would need to be a cache rebuild triggered in order for block displays to be updated. I think an update hook that updates block entities and saves the new configuration would make sure the changes are committed to storage for global block layout, and also trigger the cache rebuild that would update navigation and LB displays. But there would need to be a separate mechanism to save the changes in navigation and LB. (Though since Navigation is just one config object, that could happen in the update hook as well.)
• Finishing an automated test is TBD because it's a lot of steps to place the block in the global layout, Navigation, LB defaults and overrides, etc.
• I haven't done any profiling

Moving to NR to see if #29/MR 14553 is a good way to go, high level. Tagging for framework manager review since component is base system (should it be block?) and affects blocks/navigation/layout builder.

Some follow up thoughts from #29 to deal with Layout Builder components, in overrides mostly:

• To make it easier to find block schema versions when querying the serialized blob in the LB field tables, maybe instead of just an int, the schema version should be stored as a string like {base_plugin_id}:{version}
• There probably needs to be a registry of latest version per base plugin ID. This might be something that can be generated at block discovery time, but probably the way to do that would be to add an updates property to the Block plugin attribute (instead of the per method attribute), that would look like
  

  #[Block(
    id: "block_id",
    admin_label: new TranslatableMarkup("Block label"),
    updates: [
      11400 => 'updateMethod1',
      11500 => 'updateMethod2',
      // ...
    ]
  )]
  

• An initial schema version for all blocks might need to be set with an update method in BlockPluginTrait
• UI to update LB would query the LB field tables and look for {base_plugin_id}:{version} strings; compare those with the latest registered versions, then add entities that need updates to a queue, which could be run in a batch
• One thing that needs consideration is if content moderation is involved: would updates need to be run for just the default revision? Or all forward revisions beyond that? Or just the latest forward revision?
• Other thought about revisions: when viewing an old revision, the configuration update will take place on block load, so the revision being viewed in the future could possibly not look the same as it was originally

tear them apart

let block settings be something @block.plugin_settings.$plugin_id.$uuid or some such, those are config objects, block manager can read them, hook_update_N has one place to update them all

Stumbled upon this in a random test that links here ItemsPerPageTest so can that code be removed or todo updated?

Stumbled upon this in a random test that links here ItemsPerPageTest so can that code be removed or todo updated?

The current MR is just a PoC to get consensus on the approach/architecture. Details like that can be updated once we decide on how to proceed.

Gotcha, my bad

#36: No worries. It'll likely be a while before this lands, so I added a line to the Remaining tasks in the IS to clean up any code references to this issue.

Nice start, @godotislate! 👏 A few questions on the MR :)

So more thinking about what do about thousands or millions of content entities that have block config in them. I'm not too familiar with how Canvas works, so again using Layout Builder as the example:

I think trying to query for the schema version for a block plugin the serialized blob in LB overrides field table is probably a dead end. So I'm wondering instead if there should be an event that can be used in update hooks. Something like:

function my_module_update_12000(): void {
  \Drupal::service(EventDispatcherInterface::class)->dispatch(new PluginConfigUpdateEvent('block', 'my_block_plugin_id', {version}));
}

The Block module could have a subscriber that simply loads all block entities that use the block plugin and resaves them.
LB subscriber would run a query in overrides tables for the block ID, then add the entity to a queue. Queue would be processed during cron, or maybe a UI using the batch API. For defaults, it would load LB enabled displays, load the layout, then resave.

Some questions remain about how to handle forward vs default revisions, etc.