Ignore configuration storage collections

I would also like to have this feature. Here's my case:

I have a Drupal 8 multisite with a dev/stage/production environments. Client is operating translations on production (both user interface translations and config translations). Meanwhile, development team is working on some other features and from time to time, a deployment is done.

The problem is that before each deployment we have to export all config translations from production and put them into code. Otherwise it will get overridden when new features are deployed.

Ideally, there should be a possibility to completely ignore config translations from sync.

Hello!
I created simple module like Config ignore to ignore whole collections.
https://www.drupal.org/project/config_ignore_collection

Thanks @lpeabody to provide solution to ignore collection.

I looked at config_ignore_collection and the argumentation in #4. But I am afraid that the conclusion is not exactly right.

It is entirely possible to filter which languages are available. One simply has to implement \Drupal\config_filter\Config\StorageFilterInterface::filterGetAllCollectionNames. The StorageComparer should not be interacted with and it is not a service because it should always give an accurate diff. The filter allows to set this as you need. That said with CMI 2.0 this would become significantly easier. see #2991683: Move configuration transformation API in \Drupal\Core\Config namespace.

I am leaving this feature request open in case someone would like to create a patch for it. I would accept a new filter plugin that only filters out entire languages so that the ignore filter can be kept relatively simple. (And maybe a third one that combines ignoring some config in some language.)

Damn, I should've come back to this ticket before quickly making a patch. But I'm on a deadline, and once I'm done I can refactor it to a new filter. And do it more efficiently using the filterGetAllCollectionNames instead of doing the check in matchConfig and activeRead.

For anyone who needs it: The patch from #8 works perfectly, even though still waiting for refactoring.

Yeah sorry about that, not sure when I'll have time to refactor it. Been busier than I'd hoped as of late. If someone else wants to take a crack at it, be my guest.

Hi @lpeabody

Thanks for the patches and digging into how config filters work.
Unfortunately though, I think you are not on the right track.

The conclusion based on your assumption is correct, but then you could go one step further and not do any filtering and instead just go all the way with changing the drush command. Drush in essence does the following on export:

• delete all from sync
• read all from active and write to sync
• loop over collections in sync and remove all in it
• loop over collections in active and write to corresponding collection in sync

If you want to ignore collections for the export all you have to do is skip the last two steps. This is what drupal console did in an early version and it was fixed because in general that is a bug.
The import then need messing with the active storage as you discovered.

But all that is not in line with how config_ignore works and won't work with the UI or drupal console.

Config Ignore works by filtering what is read from the sync store in such a way that it seems that there is no difference to what the active store contains.
So for ignoring all collections the filter should return the same collection names as the active store has. And when the filter is used on the collection it should filter all the read operations to return what the active storage has fo that collection.
And for ignoring on export the filter would not delete from collections and would not write anything when the filter is filtering a collection.
But that is quite advanced to get it right and needs good testing.
It will become much simpler with the changes we are trying to bring to core with CMI 2.0 But unfortunately I don't know how far off that is.

I am also looking for this feature. My use case is with Lingotek Translations. Our client is translating config on their stage server and we don't want to blow that away on deploys. So ignoring the config/language folder/config would be helpful.

I found this module https://www.drupal.org/project/config_ignore_collection however it doesn't not look well supported or maintained. This should be a feature of Config Ignore in my opinion. It would be great if the patch above could be updated to work the same way config ignore does.

Any update here?

This issue has come up again with placeholder text that is being translated after applying https://www.drupal.org/project/drupal/issues/2546212

Our client is translating on their stage environment. However, our deploys blow away that translation.

Ok, after a day of testing and some help via Drupal Slack from @lpeabody (Thank You!) I have this working. The key was the patch from #18 with the Drush patch from #17 and ensuring the Config Ignore Drush Commands module was turned on. I would say this is needed functionality of Config Ignore and this should be RTBC ASAP.

I ran into a strange issue with Config Split where having a language folder in my split folder prevented this from working. However, that feels like more of a problem with how Config Split and Config Ignore interact. Once possible solution would be to use the Config Split Ignore module. In my use case I just added the language folder to my git ignore.

We were using this patch at #17... with the latest update to 2.2 the install hook added with the patch conflicts.

So here's a new one.

I think joey-santiago accidentally forgot file src/Plugin/ConfigFilter/CollectionIgnore.php. Here is re-rolled patch.

I tried using the patch from #26 and drush patch from #17. It worked well on my local but failed on travis as the travis was using drush cst (config status) command to match configuration from db and active storage. Thus, I had to override the drush cst command as well. Here's the patch if someone needs the same functionality.

RE: #27 This is because the filter from #26 is wrong.
In general adding a drush command to config_ignore is the wrong approach, so if it is necessary then the patch is wrong.

As I wrote in #19 this is now easier if we do this in the 3.x version of config ignore. Or alternatively create a 2.x version of config_ignore_collection and use an event subscriber just as 3.x of config_ignore does now.
I can help review that in case.

RE: #28
Thanks for your inputs. I have tried to accomplish this in 3.x using the same event subscriber and implementing the logic as detailed in #19. However, I still had to override the Drush commands to use the export storage service for config export and status check as detailed in this change record. So that appropriate events could be dispatched. Also, I did override the config import to use 'config.import_transformer' to transform the source storage and hence dispatching the appropriate event.

While using the 3.x version, I found a few issues with array checks so I have applied them accordingly and it seems to be working fine for me now.

@bircher, I would be happy to have you review this.

updated patch

Incorporating patch from https://www.drupal.org/project/config_ignore/issues/3117694.

Ok I think since this is much simpler in 3.x and the issue is now tagged agains 3.x this is postponed until 3.x works again #3117694: Make all available tests pass

and RE #29
The drush commands are not needed because drush 10 dispatches the events and 3.x is only compatible with core 8.8+ and drush 10+.
If you use drush 8 or 9 or drupal core 8.7 none of this will work.

for 2.x the patch from #26 is wrong because it doesn't do what I wrote in #19:

So for ignoring all collections the filter should return the same collection names as the active store has. And when the filter is used on the collection it should filter all the read operations to return what the active storage has fo that collection.
And for ignoring on export the filter would not delete from collections and would not write anything when the filter is filtering a collection.

Took me a couple of hours, reading through the patches, how config_filter works. But I might have fixed it using the "config ignore" and "config filter" way.

By comparing the collection name in the source with the one that we might want to ignore, we can execute the same kind of comparison on active storage of that collection.

Fantastic! this is much more in line with how it can work.
However you can do it in a much more simple way:

Extend the base class from config filter and then;

+++ b/src/Plugin/ConfigFilter/CollectionIgnore.php
@@ -0,0 +1,128 @@
+    if (!$this->matchCollectionName()) {
+      return $data;
+    }

Do this in every read-related method.

Otherwise return what the active store returns for read operations, (don't delete or write to the active store! if you want to also filter on export you have to filter the delete and deleteAll and prevent the deletion when it is your ignored collection, and then filter on write by returning what you read from the source, but I would recommend doing this step by step).

The exception is filterGetAllCollectionNames where you need to array merge the ignored collections that exist in the active store. And most importantly the filterCreateCollection which you can copy from the IgnoreFilter.

And please write tests for it otherwise I don't know if the behaviour is that you expect.

I don't really get what you mean with how it can be more simple. At least, from my current understanding.

What I did was:
- I used the setup of the "IgnoreFilter" to see where they "check" on situations to prevent importing.
- There I overridden the FilterRead and FilterReadMultiple to incorporate my logic for reading from the active collection, if the collection should be ignored.
- Same for filterExists and filterListAll

So my questions are:
- Why implement delete if the IgnoreFilter does not?
- What can be simpler?

Maybe I am missing something...

I could extend the base class, but that would mean duplicate could on the __construct and filterCreateCollection in comparison to IgnoreFilter.
But sure, this would make it more "separate".

Tests... I suck at tests, for now. And have to find the time to do this.
It would fix a lot of deployment issues for us, so maybe the boss will sponsor some time to do this.

Yes you are right! Don't delete on export, that is another issue.

What can be simpler? well it depends on how you expect it to work, if there is config in the sync storage for a collection you ignore but it is not in the active store then it is going to be imported (as config ignore 2.x currently works) then it is probably simple enough. (you don't need the extra function to get the active storage in the collection because it already is since each collection will have its cloned filter.

But yes just copy the one line from filterCreateCollection and extending the base class is much better, because then you don't inherit any complexity that then gets hidden.

Also filterExists needs the check for the ignored collection.

I am trying to wrap my head around this:

What can be simpler? well it depends on how you expect it to work, if there is config in the sync storage for a collection you ignore but it is not in the active store then it is going to be imported (as config ignore 2.x currently works) then it is probably simple enough. (you don't need the extra function to get the active storage in the collection because it already is since each collection will have its cloned filter.

What can be simpler? well it depends on how you expect it to work, if there is config in the sync storage for a collection you ignore but it is not in the active store then it is going to be imported (as config ignore 2.x currently works) then it is probably simple enough.

This is what currrently seems to happen (needs tests ofcourse).

(you don't need the extra function to get the active storage in the collection because it already is since each collection will have its cloned filter.

Because as I understoond, the active storage that the IgnoreFilter uses, is the default collection. Because it always uses:

$container->get('config.storage'). 

So I still need to get the active collection storage right (for a language for instance)?

/**
   * Get the active storage belonging to the source collection.
   *
   * @return \Drupal\Core\Config\StorageInterface
   *   The Active Collection Storage.
   */
  protected function getActiveCollectionStorage() {
    return $this->active->createCollection($this->getSourceStorage()->getCollectionName());
  }

And I can only use $this->active->createCollection($this->getSourceStorage()->getCollectionName()), once setSourceStorage is called.
Tried to set it in the construct, but it gave an error because $this->getSourceStorage() was null.

I think that is another reason why you should not extend the IgnoreFilter and instead copy the things you need. It will make it very clear what is going on.
The create method instantiates the first and default filter. I passes the active storage form the container to the plugin. But then the filterCreateCollection method creates a new filter and it passes the active storage in the new collection to it. so your filter in the collection mode has $this->active also in the collection.

And the setSourceStorage is called by config_filter when the storage is set up (which it also is for each collection). so you can be sure that in any filter method the storage is set. (not in the constructor of course)

What I am trying to say is that this is ok:

  protected function getActiveCollectionStorage() {
    return $this->active;
  }

So you can just skip it. if you had tests for this functionality then you could just switch this out and see if the tests still pass...
Tests are a very useful development tool. Which have the added benefit of continually ensure that it works in the future thanks to the drupal CI.

Wow... I was typing again that I could not see it, writing the quoted text below and stuff.

You sure? I will try that out then.
Because my perception of it was that $this->active is the main storage (the sync folder itself) and not the collection store (language/en for instance). It is not changed when filterCreateCollection is called, $this->active in the default, and instantiated plugin for each collection is always the same right?

But then when I was adding the code examples to demonstrate what I meant... I saw it haha.

  /**
   * {@inheritdoc}
   */
  public function filterCreateCollection($collection) {
    return new static($this->configuration, $this->pluginId, $this->pluginDefinition, $this->active->createCollection($collection));
  }

$this->active->createCollection($collection) is passed as active, dit not see that haha. Changed it in the new patch.

Needs test still of course. But I think we can start doing this to wrap this up.

@jefuri thanks for the great #40 patch, but there is already config_ignore_update_8202 function (config_ignore.install) in 8.x-2.2 version, may you please check and update the patch?
https://git.drupalcode.org/project/config_ignore/-/blob/8.x-2.2/config_i...

Done! Nice catch @aponomarenko

Patch from #42 seems to work nicely!

There are some linting issues to be addressed, however. Please see https://www.drupal.org/pift-ci-job/1641520

Fixing PHPLint issue in patch #42, please review.

Thanks @mrinalini9, I can confirm that patch #45 works!

@mrinalini9, After applying this patch i added language.* in Configuration storage collections to ignore field but still all language collections are appearing in Synchronize page with n number of config under each collection and they are not showing below "The following configuration entities are ignored due to the Config Ignore Settings and therefore not displayed in the list above".

Thanks! It works for me.

+++ b/config_ignore.install
@@ -13,9 +13,17 @@ function config_ignore_update_8201() {
+  $config = \Drupal::configFactory()->getEditable('config_ignore.settings');

Why is this update function added under existing 8202, and what was previously 8202, now is 8203?

For folks that already had module, update hooks will not be executed correctly.

Good catch, I think it is because this patch started when there wasn't a update hook 8202, but during development of the patch it was.
I corrected it incorrectly, it should indeed be 8203.

Nice work!

A few comments:

1. +++ b/src/Plugin/ConfigFilter/CollectionIgnore.php
   @@ -0,0 +1,172 @@
   + *   storages = {"config.storage.sync", "config.storage.active"},
   

   I am not sure why we would add config.storage.active here.

2. +++ b/src/Plugin/ConfigFilter/CollectionIgnore.php
   @@ -0,0 +1,172 @@
   +    if (!$this->matchCollectionName()) {
   +      parent::filterExists($name, $exists);
   +    }
   ...
   +    if (!$this->matchCollectionName()) {
   +      parent::filterListAll($prefix, $data);
   +    }
   

   This is missing a return

3. +++ b/src/Plugin/ConfigFilter/CollectionIgnore.php
   @@ -0,0 +1,172 @@
   +    // Add active collection names as there could be ignored config in them.
   +    return array_merge($collections, $this->active->getAllCollectionNames());
   

   I think we should only add the collections from the active store that we also ignore, and then we can also de-duplicate them with array unique

-

-

Attaching the solution to get this working together with Support for export filtering via Drush - 2857247.
The previous patch at #54 & #56 doesn't work as expected, this one is the correct version instead.
the patch order is:
1) support-for-export-2857247-24.patch
2) config-ignore-collections-2973431-57.patch

Re-rolled the supplementary drush commands patch file to work with Drush 10, since the change from 9 to 10 introduced breaking changes.

Tested #52 with #53.1 + #53.2 (patch added). Worked nicely for my use case, expect for custom languages with language code containing dash. Since it worked with contrib languages with dash (e.g. zh-hant), I would not expect this niche use case to be solved in this issue.

#60 has some issues, created this new one to be applied over the 2.x-2.3.

@lpeabody,

rejected on: src/Form/Settings.php and on config/schema/config_ignore.schema.yml

--- src/Form/Settings.php
+++ src/Form/Settings.php
@@ -84,6 +94,9 @@ Examples: <ul>
     $config_ignore_settings_array = array_filter($config_ignore_settings_array);
     $config_ignore_settings_array = array_values($config_ignore_settings_array);
     $config_ignore_settings->set('ignored_config_entities', $config_ignore_settings_array);
+    $config_collection_storage_array = preg_split("[\n|\r]", $values['ignored_config_collections']);
+    $config_collection_storage_array = array_filter($config_collection_storage_array);
+    $config_ignore_settings->set('ignored_config_collections', $config_collection_storage_array);
     $config_ignore_settings->set('enable_export_filtering', $values['enable_export_filtering']);
     $config_ignore_settings->save();
     parent::submitForm($form, $form_state);

--- config/schema/config_ignore.schema.yml
+++ config/schema/config_ignore.schema.yml
@@ -7,6 +7,11 @@ config_ignore.settings:
       label: 'List of ignored configurations'
       sequence:
         type: string
+    ignored_config_collections:
+      type: sequence
+      label: 'List of ignored configuration storage collections'
+      sequence:
+        type: string
     enable_export_filtering:
       type: boolean
       label: 'Enable export filtering'

Also, I see the `CollectionIgnore` is added multiple times, so only added one instance in "src/Plugin/ConfigFilter/CollectionIgnore.php"

@lpeabody, silly question at this point, but shouldn't it be ignored as well on drush cex ?, since I see the collection is ignored on drush cim but unfortunately no on drush cex. either path #60 or # 61

never mind, I was missing 'enable_export_filtering: true' property in 'config_ignore.settings'. worked for me. thanks.

#60 applies when you've also applied the patch from https://www.drupal.org/project/config_ignore/issues/2857247
#61 does not apply when you have that patch, and it includes unrelated export-ignore stuff if you do not have that patch.

What is needed is a patch that applies to an unpatched config_ignore and does not include unrelated stuff that causes collisions with the export-ignore patch.

Unsure what u_tiwari is trying to accomplish in their PR, but it seems to be completely unrelated to this issue (it creates a `config_ignore_drush` submodule).

Also noting that none of the patches in here apply to 3.0-beta1. When upgrading core to 9.3.0, config_ignore on the latest 2.x branch ends up deleting a lot of user roles from config, generally causing a heap of issues.

It appears the only path forward is to:

1. Update to config_ignore 3.x
2. Port this patch to 3.x

What combination is working for others?
Is the approach to use the patch on comment #61 and upgrade config_ignore to 2.3?

Then use the following code in custom module to invoke that patch?

function hook_config_ignore_collections_alter(array &$collections) {
  $collections[] = 'language.*';
}

function hook_config_ignore_settings_alter(array &$settings) {
  $settings[] = 'language.negotiation';
  $settings[] = 'language.entity.*';
}

@jozzy_a I confirm that patch on #61 cleanly applies over config_ignore 2.3.

Once patch is applied, I configure the module via UI (admin/config/development/configuration/ignore):

Configuration entity names to ignore:

language.negotiation
language.entity.*

Configuration storage collections to ignore:
language.*

However, language collections still not ignored on "drush cim". Does it work for you?

@kmonty I agree that a patch for config_ignore 3.x is needed.

I have followed @rcodina instructions:

1. Add the patch v2-3 of #61 comment: applied correctly
2. Run drush updb without any error
3. Add language.* to the store collections to ignore, and language.negotiation and language.entity.* in entity names to ignore in the UI,
4. Add a new translation
5. Did drush cim

After the last step the translation is gone.

Updated #32 to work with latest config_ignore 3.x. It seems to work on a clean site install.

Added install config and fixed wrong update counter.

I tried adding some basic test coverage.

I've added static caching for collections and refactored the patch to match the logic of the getRules method.

While it's nice for a patch to not touch too much of the core functionality, I currently see a lot of duplication happening in the code. Maybe there are more elegant ways to achieve this? Or is it better to merge as is and to the refactoring later?

Patch doesn't apply for 8.x-3.0-beta1

sorry I just checked in new release i.e 8.x-3.0-beta1 there is no commit available https://www.drupal.org/project/config_ignore/issues/3226493 But, this commit is available in 8.x-3.0 branch. After applying https://www.drupal.org/project/config_ignore/issues/3226493 patch then we can apply the above patch.

Getting the below errors after applying the patch and trying to export config

 [warning] array_merge(): Expected parameter 1 to be an array, null given SplitFilter.php:161
 [warning] array_merge(): Expected parameter 1 to be an array, null given SplitFilter.php:162
 [warning] uksort() expects parameter 1 to be array, null given SplitFilter.php:172
 [warning] array_merge(): Expected parameter 1 to be an array, null given SplitFilter.php:161
 [warning] array_merge(): Expected parameter 1 to be an array, null given SplitFilter.php:162
 [warning] uksort() expects parameter 1 to be array, null given SplitFilter.php:172

Patch in #61 cleanly applies to config_ignore 2.x, and doesn't throw any errors

The patch in #79 worked for me with 8.x-3.0-beta1. Some notes:

1) I first applied #3226493-4: Clear Config Filter plugin cache to update from 2.x to 3.x (committed after beta1 was released), then the patch in #75. Otherwise, the update scripts conflicted, and the patch wouldn't apply.

2) The update script added in #75 sets the newly-added "config_ignore.settings:ignored_config_collections" config to an empty array. Since I am bundling this in a deployment that has an existing language configuration, I added an update script to my custom module and run it first. Example:

/**
 * Implements hook_update_dependencies().
 */
function my_module_update_dependencies() {
  $dependencies['my_module'][9401] = ['config_ignore' => 8302];
  return $dependencies;
}

/**
 * Sets Config Ignore's initial ignored collections.
 *
 * This must run after config_ignore_update_8302().
 */
function my_module_update_9401() {
  $config = \Drupal::configFactory()->getEditable('config_ignore.settings');
  $config->set('ignored_config_collections', [
    'language.*:my_module.settings',
    'language.*:system.site:name',
  ])->save();
}

#79 is working for me, but throws warnings when no collections have been added.

[warning] Invalid argument supplied for foreach() ConfigIgnoreEventSubscriber.php:391
[warning] Invalid argument supplied for foreach() ConfigIgnoreEventSubscriber.php:291

Re-roll for 8.x-3.0-beta1

LGTM!

Thanks for this patch! I also needed to figure out how to ignore translated configuration and #78 worked great. We are using 3.0-beta2.

Re-roll since another hook_update_N() got in already. I also adjusted the patch to respect previously configured settings instead of reseting them since now the update will get executed one more time for sites that had the previous patch.

Thank you for the patch!
It works for us so far.

The patch in #91 no longer applies to 3.x-dev and it's not an easy re-roll... the split of modes with Basic, Intermediate, and Advanced complicates this.

Yes the patch here needs to be re-worked. And I would argue the approach needs to be re-thought.

But if you look at the code in 3.x-rc1 you will see that it was refactored with this issue in mind.

To achieve the same purpose (ignoring config in certain collections), I think it would be easier to integrate them with the other patterns.

You know that now you can ignore patterns with their config name and optionally separated with a : also ignore just parts of it. If you leave the partial part out then it is essentially everything, or *.

I think we could do the same for collections. I don't know what the best separator would be. Let's assume for this argument that we settled on | as the separator, then imagine:
lang.de|config_name:some.part and if you omit the language part it is implied just * and if you want your config to be ignore just in the default collection you would write |config_name

So you could also ignore lang.*|* and ~lang.en|* which would mean ignore all languages except for English. This would just make it more difficult to then also ignore something but not everything in the english translations. But to be honest.. I think that is an edge case that we could be ok with. I don't think we need ~~ to exclude excludes... But we could in a different issue.

Made a quick patch for the RC version (without tests for now).

Now we can use | separator for collections.
For example: language.*|*

Hi
Yes the patch from #95 is precisely what I had in mind.

Tests should be pretty straightforward to write, one can just add more test cases to where all the other things are. They already have languages set up for this exact reason.

#95 Worked for me for 3.1 release.
Upgrading from 2.4 and after settings update it was pretty straightforward,

Thanks a lot!

Also confirming that #95 worked for me for 3.1 release.

As a note for others - you'll need to update your config_ignore.settings after applying the patch from #95.

Examples

• * (will ignore everything)
• ~webform.webform.contact (will force import for this configuration, even if ignored by a wildcard)
• user.mail:register_no_approval_required.body (will ignore the body of the no approval required email setting, but will not ignore other user.mail configuration.)
• language.*|* (will ignore all language collections)
• language.fr|* (will ignore all fr language collection)
• language.fr|field.field.* (will ignore all fr field translations)
• ~language.fr|field.field.media.file.field_media_file (will force import for certain field translation)

Changing the status of this issue to "Reviewed & tested by the community" per the Issue Status Field documentation.

Before this can be committed someone needs to add a couple of test cases here:

https://git.drupalcode.org/project/config_ignore/-/blob/8.x-3.x/tests/sr...

It can be a simple configuration with a couple of the language examples.

Added tests to the patch

Hide all outdated patches and made a PR

The latest patch worked for me.

The latest patch worked for me too!

this works for me

@gmustapha not sure why you readded patch #79, this is a patch for 2.x and 3.0-beta2 but we need a patch for 3.x already.
If you need a patch for the old version you can just use the older one (#79).

I have a use case that the patches in #95 and #101 don't resolve.

For example, I'd like to ignore the site name in language collections.

I expected something like this to work:

- language.*|system.site:name (doesn't work)

FYI:

- language.*|system.site (works, but it ignores too much)
- language.*|system.site.* (doesn't work, though the help text added in #95 suggests it should)

Thank you everyone for contributing to this long standing issue!
I was never really happy with how it worked in 2.x, but I am very happy to see that the architecture of 3.x makes this much simpler and yet much more flexible to configure.

I added and changed some stuff based on the great work from patch in #101 and I added a test for #108 which passes so, if that doesn't work then we need to find a better test and fix it as a bug in the next release.