Polish file format of configuration metadata (for translation)

Postponing this until the CMI i18n patch is in; no sense in holding up other features in the meantime.

#188 said "mixup of config names and $definition elements was very very confusing". #240 said "I don't think having two alternate syntaxes is going to make things easier. It will make things harder to understand. We need to agree on one syntax IMHO. "

Whether you call it .meta or .configuration, it's immaterial but -- get rid of .label and .type. It's insanely confusing and the amount of writing you need to do is so small it doesn't matter.

Some notes, just to add some more background information into the issue:
- While any of the proposed formats will work for the machine, the goal of this is to make metadata easier to read and write by module maintainers.
- We don't need to rush this since we can script the changes for any format so my suggestion is we try to write some more metadata files for complex cases (views, node types if we get them in the config system, etc..)
- There are two kinds of properties we need to get into the metadata:
a) Description of the configuration structure. So far we've got '.include' and '.list'.
b) The TypedData API definition of the element. So far: '.type', '.label', '.definition'.
(Thus being familiar with both, configuration structure for complex cases like views and the full typed data api specification will help understanding the problem).
- Since the goal of this is to make the format human-readable, I think side by side comparisons will be helpful for any format proposed.

As an example (for image.style.%.yml) :

name:
  .label: 'Machine name'
label:
  .label: 'Label'
  .type: text
effects:
  .label: 'Style effects'
  .list:
    .label: 'Image effect'
    weight:
      .label: 'Weight'
      .type: integer
    ieid:
      .label: 'IEID'
    name:
      .label: 'Machine name'
    data:
      .include: 'image.effect.[%parent.name]'
      .label: 'Data'

name:
  .definition:
     label: 'Machine name'
label:
  .definition:
     label: 'Label'
     type: text
effects:
  .definition:
     label: 'Style effects'
  .list:
    .definition:
       label: 'Image effect'
    weight:
      .definition:
        label: 'Weight'
        type: integer
    ieid:
      .definition:
        label: 'IEID'
    name:
      .definition:
        label: 'Machine name'
    data:
      .include: 'image.effect.[%parent.name]'
      .definition:
         label: 'Data'

.list and .include already mars the simplicity (but when had multilingual cared about anything being simple?) of just using .definition. If we can't have a single dot property, it doesn't mater. All multilingual API is extremely confusing, it was always so, apparently always will be, so please, carry on undeterred.

But, of course, noone will accept me won't fixing this so I reopen although I do not know why :/

@chx: the .include was requested / suggested by @yched due to complexities in how plugins are used in Drupal, it was not added for the fun of making this complex because it is multilingual.

I personally don't see a reason in terms of the format that we could not include the "include" element under a .definition (vs. besides it like it is now). It is a special syntax element that we need to explain either way.

The fundamental problem with the dot is that while I understand it was chosen as no config name ever can have a dot in it as CMI explode() on dot, it is also an extremely common character for marking a materialized path, even if you do not know that by name, for an example of everyday usage, consider something like Section 1.3.4 As I look at a meta file like that I want to create a reading from the root to the leaf driven by the dots arriving to complete nonsense and confusion. (Edit: of course CMI itself uses the dot to create such a path)

If you want (I am not sure whether anyone cares, noone but me complained about this in the mother issue but then again noone but me opened an issue against field_get_items either while everyone was just whining on IRC, twitter and blog posts so it might be that people are not happy they just don't want to fight) so if you want to, then consider using another character and maybe different characters for list and include and a different for definition. There are a lot of characters that can not occur in a filename especially on Windows : ? * < > " | / \. Slash for example makes an XPath-y expression which looks good to me. Maybe /include and /list and *definition? Or just prefix them with a star? Use the star instead of .list and use /include and >definition? There are a lot of possibilities.

Updated issue summary link related other polish followup for file location and naming.

Updated issue summary.

Incorporating existing concepts and syntaxes:

- Annotations (doesn't fly if we have to parse this as YAML though)
- Shell/command line

name:
  #label: 'Machine name'
label:
  #label: 'Label'
  #type: text
effects:
  #label: 'Style effects'
  *:
    #label: 'Image effect'
    weight:
      #label: 'Weight'
      #type: integer
    ieid:
      #label: 'IEID'
    name:
      #label: 'Machine name'
    data:
      #label: 'Data'
      < image.effect.[%parent.name]

- # annotation-alike comment to denote metadata properties.
- * wildcard to indicate a list.
- < to read in another file.

That said, the include/piping doesn't look exactly right to me.

Why is the include an include and not a non-primitive type instead?

 effects:
   #label: 'Style effects'
   *:
     #label: 'Image effect'
+    #type: image.effect.[name]
...
     name:
       #label: 'Machine name'
     data:
       #label: 'Data'
-      < image.effect.[%parent.name]

Abot the special character to use for marking properties I don't have any special preference, but the dot seems to be the only one 'safe' at this point. Only because there are no specs about how properties can and cannot be named in config files.

Btw, the '#' won't work because as it is used for yaml comments, it is filtered out by the parser.

Only because there are no specs about how properties can and cannot be named in config files. <= files. Files have restrictions. While you probably could create any filename programatically (perhaps not the slash on a Unix system) working with the is seriously inconvenient. I thought we had this discussion when the metadata files wanted to contain * in them...? We can file a separate issue to have config boink you over the head with a pretty exception if you want to name your config |<>|

@chx: I could see how file naming gets into question when the file name pattern is looked at, but how are file naming restrictions relevant for the keys inside the .yml files? I feel like I'm not getting some connection here.

The keys available to meta are the keys that are invalid to CMI and CMI keys are / can be filenames. Yes, it's possible to have separate restrictions for config($objectname)->get($keyname) $objectname and $keyname but let's avoid that at all costs.

Maybe the problem we have is that the configuration system doesn't define anything at all about filenames (config names) nor keynames within files so we are completely lost about what we can use for each (besides OS filename restrictions) ??

And I wouldn't ask for a harcoded validation, but a single documentation line telling us what is what.

So far I only know config names must be something that can be a filename with dots in it and that keynames are something that is split on dots to map it to an array.

So please, does anyone here know something I don't?. And please don't just tell me, but point me to the line withing Drupal core where that is documented.

Otherwise I am afraid the only real option we have for metadata keys, as ugly as it may be, are dots.

#17, we make our own rules. If we need to, we can add it to the doxygen of config(), Drupal\Core\Config\Config::get() etc.

#17, we make our own rules. If we need to,

Well, yes, good point :-)

So:
1. What is the char/chars we are going to save (aka forbid from config names) for marking up metadata names?
2. Are we discussing just a character or is there anything else in the format that needs fixing?

Here's a thought: what if instead of needing a meta file per config file (or set of files with the same structure), we make one meta file per module. And make its format:

CMI_OBJECT_NAME:
  CMI_KEY:
    DEFINITION_KEY: DEFINITION_VALUE

That way, it's a known 3-level structure, and we allow CMI_KEY to contain "." and "%" as needed, rather than needing the meta file to follow the CMI tree.

I'm attaching a patch to show what I mean. This replaces all the meta files from #1648930-284: Introduce configuration schema and use for translation to follow this suggestion.

Additionally, I'm attaching a "min" file that contains the bare minimum needed for identifying what is translatable. My point in doing this is that while full metadata may be cool in theory, it's also a DX pain, and maybe isn't worth introducing until we have real use cases for them (e.g., if we end up implementing a Validation API). Perhaps it makes sense to start with the minimum, and then progressively add more metadata entries in the file as we have reason to? Note that this "min" file doesn't include the 'label' values. We currently have these labels in the forms where the configuration is edited, so duplicating them in a meta file is another DX pain. However, I'd be all for re-adding them here, if we also make the forms use this info rather than duplicating it.

@effulgentsia: I'm curious how would this new file format apply to plugin data which can show up at any depth in a structure. Such as views plugin metadata, which I believe allows for that by design. Would you define it as many times with the same definition as it could appear in arbitrary nested depths in config?

As for the minimal format, well, it is not really minimal if you consider you only want to identify the translatable strings. You could cut it down even further and only define the translatable things (which don't need type since they should all be single strings). So in your minimal format everything is either a type: text or list: true. You can just as well include this as the "value" for the CMI key then and avoid one level of nesting, unless you invision other types of information would be relevant in the minimal format (which I doubt).

Anyway, I still remember the days when webchick and Dries were freaked out that Drupal 8 cannot be called a multilingual system if you cannot go and translate the site name. Well, even if we implement the metadata fully to how it is proposed in the current RTBC patch it would not be possible to do in core, because obviously we lack the form generation, validation and save code (and have no hope to have anything like that in D8). However, if we implement something like your proposed minimal format, we cannot even say core supports a contrib config translation UI in any useful way. Why?

1. Yes, it would identify translatable items, which should be extracted and translated on localize.drupal.org. It will not identify multilingual items, such as the site logo URL, which should not be extracted for localize.drupal.org but should be translatable in any case on a site (lots of logos have a tagline for example that should be translated). A contrib module's generated translation UI for the site information would include both the site name and site logo.

2. The text type information and CMI keys are far from enough to generate any form that we can honestly present to actual users. Having labels was a minimal support for a translation UI, but if we don't even have that, a contrib module's only choice would be to generate a form with bare CMI keys, no descriptions and no supporting context. Is that a shippable UI even in a contrib module? Well, I don't think it is. So the config translation contrib module would need to redefine all config metadata from the ground up again anyway (nesting, inclusion, etc) to define the labels for all of core at least. Also then contrib module authors will see no example to build on, so they will not define any supporting data for these translation UIs either, so this one single config translation contrib will need to define the metadata for other modules as well (and/or we will be back in the bridge module proliferation state where we are in Drupal 7, so you have views and i18n and then need views_i18n module to bridge the two). That is not even one step ahead of where i18n module was in Drupal 7. At least we could hopefully still inject the translations for CMI pieces with the CMI override system, if it stands the test of time that is (see #1859110: Circular dependencies and hidden bugs with configuration overrides).

Last summer I posted this summary of what i18n in Drupal 7 does to describe objects (metadata), inject translations (context and overrides) and generate translation forms, see http://groups.drupal.org/node/152929 - we have been explaining these over and over again in different discussions and meetings in Denver, Barcelona, Berkeley, etc. and it seems there is some collective understanding of the requirements when everybody is at one place but then after a while it fades and the "why the hell is this so complex" feeling takes hold and the requirements are ignored in further proposals.

The initial goal of the D8MI initiative was to not need a contrib module for config translation but that would have needed a whole rework of all configuration forms and workflows, that CMI did not even plan to touch. So given we had enough on our plate, we stepped back to a position where core would only provide supporting data for a contrib solution. If we step back from the most of the supporting data as well, then it is mostly all on contrib like in Drupal 7, and the only improvement we gain from CMI will be that the contrib module will have a little easier time to inject the overrides. (If the metadata is constrained to identifying translatables, it just plugs in a regression since exportables already identified their translatables with t() in Drupal 7).

Just in case someone wants to check documentation for inspiration, here are a few ideas and links:
- Yup, YAML spec supports metadata (see types and tags), http://www.yaml.org/spec/1.2/spec.html
- Unfortunately, the Symfony YAML parser does not, http://symfony.com/doc/2.0/components/yaml.html
- Interesting about mapping classes in YAML, http://docs.doctrine-project.org/en/2.0.x/reference/yaml-mapping.html
- Kwalify Schema definition for YAML, http://www.kuwata-lab.com/kwalify/
- Rx: Simple, Extensible Schemata http://rx.codesimply.com/

I like @effulgentsia's idea and proposal in #21, as it keeps the data together in one place, and thus, that $module.config.yml config metadata YAML file will live next to, e.g., $module.routing.yml, which is equally a YAML file that defines all static routes that belong to a module. So I think that makes sense from a DX perspective.

I wonder whether we could combine one of the ideas in #12 into that — treating the definition for a sub-key as a non-primitive 'type'; i.e., this:

image.style.%:
  name:
    type: machine_name
    label: Machine name
  label:
    type: text
    label: Label
  effects:
    type: list
  effects.%.name:
    type: machine_name
  effects.%.weight:
    type: integer
    label: Weight
  effects.%.data:
    type: 'image.effect.data.[effects.%.name]'

image.effect.data.image_crop:
...

I.e., instead of 'include', we define the entire element to be of a certain 'type'.

That said, we could as well do something like this:

image.style.%:
  name:
    type: machine_name
    label: Machine name
  label:
    type: text
    label: Label
  effects:
    type: list
  effects.%:
    type: image.effect

image.effect:
  name:
    type: machine_name
  weight:
    type: integer
    label: Weight
  data:
    type: 'image.effect.data.[name]'

image.effect.data.image_crop:
...

Well, I like the format in #24 too (the second one).

Only some concerns (that I don't think would be hard to fix) about it:
- By using types like that we are doing away with the TypedData API, which I wouldn't oppose because so far it has given us more trouble than benefits. So maybe it is a better idea after all.
- Not all the metadata for an object is in the same module. Another module may define additional image effects that are used in image styles. (The other patch prefetches all metadata names, not a problem, so we can do something similar parsing *all* the files in advance and merging all in a big array)
- Views is a bit harder than that with multiple levels of plugins (plugins that use plugins). But again I don't think this is a blocker, we can make them fit into this schema, just it would be good to see how they look like to prevent ugly surprises.)
- We may need a merging strategy for modules to add their bits into other module's metadata, just for the record (I wish we had node types into CMI to see an example of this).

That said I don't really have the time anymore to embark into another one like #1648930: Introduce configuration schema and use for translation so all I can help here with will be some reviews. But I may be able to take care of adding the localization part back, later, if you want to drop it from this one so we can focus on metadata format only.

With the form generation concept discarded in #1648930: Introduce configuration schema and use for translation, type and label and not relevant anymore and actual work is happening in #1861640: Provide config metadata solely for translation with the express goal of having a format that people can agree on out of the gate. No need for followups.

Updated issue summary.