Problem/Motivation

The Locale module has an implementation of hook_modules_installed() that calls

locale_system_set_config_langcodes();

This function attempts to go through all of the active config and set a 'langcode' value on it. If you have strict config schema checking turned on, as it is when you are running a SimpleTest test, this currently fails if you try to install in a foreign language, because not every config schema has a 'langcode' value defined on it.

They should -- top-level config is supposed to either inherit from config_object or config_entity, but it doesn't all.

Without patching, currently when you try to set up a simpletest test that installs in Spanish, it fails with:

Drupal\Core\Config\Schema\SchemaIncompleteException: Schema errors for menu_ui.settings with the following errors: menu_ui.settings:langcode missing schema in Drupal\Core\Config\Testing\ConfigSchemaChecker->onConfigSave() (line 98 of /opt/www/fmarket/core/lib/Drupal/Core/Config/Testing/ConfigSchemaChecker.php).

The stack trace is, down to WebTestBase->setUp():

Drupal\Core\Config\Testing\ConfigSchemaChecker->onConfigSave(Object, 'config.save', Object)
Drupal\Component\EventDispatcher\ContainerAwareEventDispatcher->dispatch('config.save', Object)
Drupal\Core\Config\Config->save()
locale_system_set_config_langcodes()
locale_modules_installed(Array)
call_user_func_array('locale_modules_installed', Array)
Drupal\Core\Extension\ModuleHandler->invokeAll('modules_installed', Array)
Drupal\Core\Extension\ModuleInstaller->install(Array, )
_install_module_batch('menu_ui', 'Menu UI', Array)
call_user_func_array('_install_module_batch', Array)
_batch_process()
batch_process(Object, Object)
install_run_task(Array, Array)
install_run_tasks(Array)
install_drupal(Object, Array)
Drupal\simpletest\WebTestBase->doInstall(Array)
Drupal\simpletest\WebTestBase->setUp()

Note that the existing multilingual install tests don't use the standard install profile (which installs menu_ui, containing this menu_ui.settings config item), so they're not hitting this problem.

Proposed resolution

Audit config for which objects do not inherit from the config_object type and fix them. Write a test so it stays fixed.

Remaining tasks

Make a patch with a test.

User interface changes

None.

API changes

None.

Data model changes

Not really. Some config schema will change from mapping to config_object, but this will not introduce any backwards incompatibility changes.

Beta phase evaluation

Reference: https://www.drupal.org/core/beta-changes
Issue category Bug because some config schema are not following our standards according to https://www.drupal.org/node/2460751
Issue priority Major because incorrect config schemas are data model problems, and it is preventing installation within SimpleTest.
Prioritized changes It's a bug fix.
Disruption Not really. Some config schema will inherit a 'lancode' config key that they didn't already have, and no one should be using this key for anything else anyway as it's reserved. Only affects one core config top-level schema, plus a few in tests. Doesn't affect existing config/install or config/optional YML config files because 'langcode' is optional.
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

jhodgdon’s picture

Here's the failing test.

jhodgdon’s picture

Status: Active » Needs review

Status: Needs review » Needs work

The last submitted patch, 1: 2541800-failing-test.patch, failed testing.

jhodgdon’s picture

Status: Needs work » Needs review
FileSize
2.07 KB

Let's try this patch. The test failed on the bot the same as for me, so that's good. Here's a patch with the test... may break everything... let's see.

Status: Needs review » Needs work

The last submitted patch, 4: 2541800-with-test-3.patch, failed testing.

Gábor Hojtsy’s picture

Title: locale_system_set_config_langcodes() creates invalid configuration » Some config does not inherit from config_object, so locale_system_set_config_langcodes() results in schema errors
Issue summary: View changes
Issue tags: +sprint, +language-config

locale_system_set_config_langcodes() should only set the language code if the config entry exists for langcode.

The way the config system works if a config file does not have a langcode key, it has an assumed langcode of English. Lack of a langcode key does not indicate that there is no key in the file that may have human readable text (and therefore need a langcode associated, so we know what language it is in). So only *changing* langcode if one already exists does not fly because if one does not exist, it is implicitly English. In the Spanish case, we need to set it to Spanish, so the translation will be stored within the default config and not as a Spanish translation override.

The top level langcode key is reserved in configuration and people should use the config_object base type in their config schema to get one automatically. See https://www.drupal.org/node/2460751

Retitled accordingly. Updated issue summary accordingly.

Gábor Hojtsy’s picture

Title: Some config does not inherit from config_object, so locale_system_set_config_langcodes() results in schema errors » Some config do not inherit from config_object, so locale_system_set_config_langcodes() results in schema errors

Language :D

Gábor Hojtsy’s picture

Not sure if other objects are affected, but this would be the change at least for menu UI settings:

diff --git a/core/modules/menu_ui/config/schema/menu_ui.schema.yml b/core/modules/menu_ui/config/schema/menu_ui.schema.yml
index 19d3efc..69af435 100644
--- a/core/modules/menu_ui/config/schema/menu_ui.schema.yml
+++ b/core/modules/menu_ui/config/schema/menu_ui.schema.yml
@@ -1,7 +1,7 @@
 # Schema for configuration files of the Menu UI module.
 
 menu_ui.settings:
-  type: mapping
+  type: config_object
   label: 'Menu settings'
   mapping:
     override_parent_selector:
jhodgdon’s picture

Interesting. So that would be a better fix. It would be interesting to make a test profile that has every module enabled and see if this installation test would pass using that, after fixing Menu UI's config. That would smoke out any other config in config/install that doesn't inherit from config_object.

We should also document that at the top level you should always inherit from config_object. It's certainly something I wasn't aware of.

jhodgdon’s picture

Assigned: Unassigned » jhodgdon

I'll take this on...

jhodgdon’s picture

Status: Needs work » Needs review
FileSize
2.41 KB

OK. As a first step towards making sure Core has all good config that inherits from config_base, I make a slightly different test -- only difference it that it uses a new testing profile that enables all the modules. Here's a patch for that.

Next I'll see about fixing it by
- patching the core config as suggested in #8 until the test passes
- also making sure the docs are updated (may be a .api.php topic needs an update, or maybe just d.o docs).

Setting to needs review so we can confirm the "test-only" patch fails with the same error reported above (it did locally).

Status: Needs review » Needs work

The last submitted patch, 11: 2541800-failing-test-11.patch, failed testing.

jhodgdon’s picture

Status: Needs work » Needs review
FileSize
52.9 KB

Well that's interesting. Apparently menu_ui was the only problematic schema, and fixing its two config things to inherit from config_object made the test in #11 pass.

Just to make sure, I added all the config/install config items from the standard profile to the testing_everything profile, and ran the test again. Still passed. Here's a patch with the test, added schema for the "everything" profile, and the two lines of actual pass in menu_ui.schema.yml.

Also I checked and there are no specifics about mapping vs. config_object in the Config API topic on api.drupal.org, so I'll update the docs page under https://www.drupal.org/developing/api/8/configuration instead.

Gábor Hojtsy’s picture

Status: Needs review » Needs work

One definite problem with the fix:

+++ b/core/modules/menu_ui/config/schema/menu_ui.schema.yml
@@ -9,7 +9,7 @@ menu_ui.settings:
 node.type.*.third_party.menu_ui:
-  type: mapping
+  type: config_object

This should not be a config object, its not a top level config object, its a third party setting within a node type config entity.

Not sure about the testing everything profile. Config already has a test to enable all the modules and it checks config schema there too I believe (ConfigImportAllTest using the SchemaCheckTestTrait), it does not come with a foreign language, so this issue was not spotted by that test.

    // Now we have all configuration imported, test all of them for schema
    // conformance. Ensures all imported default configuration is valid when
    // all modules are enabled.
    $names = $this->container->get('config.storage')->listAll();
    /** @var \Drupal\Core\Config\TypedConfigManagerInterface $typed_config */
    $typed_config = $this->container->get('config.typed');
    foreach ($names as $name) {
      $config = $this->config($name);
      $this->assertConfigSchema($typed_config, $name, $config->get());
    }

jhodgdon’s picture

Hm... I wonder if in that existing test we could also assert that the top-level config objects either start with config_object or config_entity somehow?

jhodgdon’s picture

Talked to berdir and Gabor in IRC. Let's try this patch:
- Test patch: Adds to SchemaCheckTrait a few lines that verify that top-level schema has a langcode element on it... Hopefully this is OK, I am assuming this method only gets called for top-level schema? If not it will definitely fail. Let's see. As a note, we'd ideally want to make sure it's either config_entity or config_object but that is not really possible. config_object just adds langcode, so that is a minimal test.
- The patch with test just fixes the one menu_ui schema element as noted in #14.

No interdiff, it would be almost the whole patch, and this patch is small. So let's see if this test finds the bad spot, doesn't break anything else, and if the patch fixes it.

The last submitted patch, 16: 2541800-failing-test-16.patch, failed testing.

Status: Needs review » Needs work

The last submitted patch, 16: 2541800-with-test-16.patch, failed testing.

jhodgdon’s picture

whoops.
= vs. => in

        $errors[$error_key] = 'missing langcode';

I must not be awake yet. ;)

The last submitted patch, 19: 2541800-with-test-19.patch, failed testing.

Status: Needs review » Needs work

The last submitted patch, 19: 2541800-failing-test-19.patch, failed testing.

jhodgdon’s picture

Well. So apparently when you do

    $this->schema = $typed_config->create($data_definition, $config_data);

it only generates schema for data that exists. And langcode (as with any part of config) is optional. So, this failed miserably.

Trying another tactic...

The last submitted patch, 22: 2541800-with-test-22.patch, failed testing.

Status: Needs review » Needs work

The last submitted patch, 22: 2541800-failing-test-22.patch, failed testing.

jhodgdon’s picture

Status: Needs work » Needs review
FileSize
3.67 KB

OK, now we're getting somewhere. The failing test did catch the errors that it was intended to catch. But it also located a few config schema entries in test modules whose top level was 'mapping' instead of 'config_object', which is why the with-test-22 patch also failed some tests.

So here is a new patch that fixes these as well.

jhodgdon’s picture

Issue summary: View changes

Updating summary.

Gábor Hojtsy’s picture

I looked at the changed schema types. Schema types with multiple dots in them are always suspect, because they would usually be a config entity or internal type. Indeed, this was an internal type:

+++ b/core/modules/views/tests/modules/views_test_data/config/schema/views_test_data.views.schema.yml
@@ -39,7 +39,7 @@ views_test_data.tests:
 views.access.test_static:
-  type: mapping
+  type: config_object

This is not a top level type:

core/modules/views/config/schema/views.data_types.schema.yml: type: views.access.[%parent.type]

Gábor Hojtsy’s picture

Status: Needs review » Needs work
jhodgdon’s picture

Doh! Right. The others in the patch all came up in the test results. Shouldn't have done that one.

Same patch without that one bit.

Gábor Hojtsy’s picture

Status: Needs review » Reviewed & tested by the community

Looks good now, yay!

Status: Reviewed & tested by the community » Needs work

The last submitted patch, 29: 2541800-with-test-29.patch, failed testing.

Gábor Hojtsy’s picture

How did this not fail when this new(?) migration landed?

Schema key block.settings.statistics_popular_block:langcode failed with: missing schema Other MigrateStatisticsPopularBlockSettingsTest.php

Looks like a new missing thing found...

jhodgdon’s picture

Because this test wasn't in place yet. This patch hasn't been committed.

I'm not sure how to fix this. Here's the schema, which is in statistics.schema.yml:

block.settings.statistics_popular_block:
  type: block_settings
  label: 'Popular content block settings'
  mapping:
    top_day_num:
...

And block_settings is defined in core/config/schema.yml:

block_settings:
  type: mapping
  label: 'Block settings'
  mapping:
    id:
      type: string
      label: 'ID'
...

So it looks to me as though block_settings is not meant to be a top-level type. But for some reason, the statistics module is putting this block's settings as a top-level schema object, and using block_settings as a top-level type. Not sure what to do here, maybe just add a 'langcode' element to this manually?

If you look at what the settings are for this block, they are really not translatable (they are all just numbers).

Thoughts?

Berdir’s picture

So it looks to me as though block_settings is not meant to be a top-level type. But for some reason, the statistics module is putting this block's settings as a top-level schema object, and using block_settings as a top-level type. Not sure what to do here, maybe just add a 'langcode' element to this manually?

Isn't the part about "the statistics module is putting this block's settings as a top-level schema object" the problem here? That makes no sense, it shouldn't do that.

jhodgdon’s picture

Yeah, I agree, but I'm not sure how to fix it?

jhodgdon’s picture

Do we know which commit this came from and can we go back and reopen that issue? And give a bop on the head to whoever did that, reviewed it, and/or committed it? :)

Gábor Hojtsy’s picture

It only happens in *migrate* *tests* currently, that may mean its the migrations fault just as much.

Gábor Hojtsy’s picture

@jhodgdon: Looks like http://cgit.drupalcode.org/drupal/commit/?id=f541fae97c910e53e7fef93c2ac... migrates to the wrong place, it attempts to use this internal type as a top level config name. Either that should be rolled back or fixed.

jhodgdon’s picture

Status: Needs work » Reviewed & tested by the community

The other issue has been rolled back. Hitting retest.

alexpott’s picture

Status: Reviewed & tested by the community » Fixed

This might break a few contrib module tests - BUT - in a good way as this will mean that their schema is wrong. NIce work everyone. This issue addresses a major bug and is allowed per https://www.drupal.org/core/beta-changes. Committed 2d56224 and pushed to 8.0.x. Thanks!

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

benjy’s picture

This broke my tests for entity_print, fixed by changing the top level type from mapping to config_object in #2559883: Top level type should be config_object

Gábor Hojtsy’s picture

@benjy: looks like the tests succeeded then :)

Gábor Hojtsy’s picture

Issue tags: -sprint