As I've not been involved with CMI much, I'm not sure this has been discussed intensively already. If so feel free to close :-)

So with CMI, obviously the config is in files and database dumps won't work without the right config. While that's a direct consequence of the file-based configuration management system, it might have some not so obvious consequences to average users.

I mean, for a developer who is used to keep its config in version control, that's not a big deal. If you have an old dump, and it does not work with your config any more, you roll back and it works again. But wait - how do recover? You roll back and import your old config? Nope, you cannot import without having an active configuration for comparing in place afaik - so you'd actually have to violate the mantra of "do not touch the active config" and copy the staged config to your active config to make things working again. But worse - if you are not using version control you are basically screwed - or better said your dump is.

And this happens quicker as one might think. Consider you are dumping your database just to take a quick snapshot of the current content, then you are adding a field. Afterwards that small change you did not export to config yet, you want to revert to your content snapshot, but when you do you'll run into troubles. The database and your active config becomes out of sync - the config contains the newly created field, while the database does not know about the field and misses its table. I guess the site will throw quite some sql errors and the dump became disfunctionial.

Compared to Drupal 7 + features, where a db dump always contained the active config that's quite a change. Given that active config it was always possible to revert to the latest config/features, as there was a working state (database) to start over from. But with Drupal 8 a database dump alone is not a working snapshot of the site anymore, and might become disfunctional rather quickly.

To avoid any troubles in d8, you may *never* dump your database without keeping track or dumping the respective active config, such that you can restore it *with* the dump.

While that's not so problematic for people using Git or a similar VCS, even those could use some assistant tools that work like "drush sql-dump" & "drush sql-cli" before, i.e. drush really needs a dump and restore active config command I guess.

Moreover, I think that this might become an issue for not so technical users who are used to "taking snapshots" of their sites by dumping databases. Really that isn't enough any more, people will have to snapshot the database + the whole files directory or at least their active config.

As possible improvements I think we could
a) improve the tools (drush) and educate people very well, maybe improve the UI to offer a complete "site snapshot" (optionally covering files?) also.
or
b) rethink how active config is stored to avoid inconsistent states (again), i.e. move active config back to the db.... It's a step that seems like a step back and quite weird for a primarily file-based config system, but it wouldn't actually change much compared to now, solve the consistency problem and avoid the "do not hack your active config" mantra.

Comments

yched’s picture

In D8, a dump is the combination of "a db dump + the set of files in the active folder at the time the db was dumped", one without the other is useless.

Restoring means restoring the db and putting the config files *directly in the active folder*. This works just fine, I've been using a custom script to do this for quite a while now.
I'm pretty sure D8 "backup and migrate" module will provide similar functionality.

That sounds "by design" to me.

fago’s picture

That sounds "by design" to me.

That's ok, but imo we need to become way better in communicating this - then.

I must say, I've not been aware this until now - I did not care much as I just re-installed d8 whenever it broke. I mean I knew the system needs its active config to run, but not being able to use database dumps is another thing. This requires quite a change in people's workflow (a dump is not enough!), so it shouldn't be so easy to miss.

Also, I figured the regular dev-test-prod workflows need to be changed to accommodate for this. Pushing config in staging to prod is just fine, but getting the latest content back from prod works only if you get database+files. However, when working with big sites files are often quite large and developers often opt to just get the latest database, which in Drupal8's case must be syncing back the latest database + the latest active config to import content in a consistent state, i.e. active config has always be synced with your database.

A regular tool like pantheon or dev-cloud without d8 support would just sync back the database and break your dev if active config has been changed meanwhile.

anavarre’s picture

That's ok, but imo we need to become way better in communicating this - then.

+1 - With most CMS tools, it's common knowledge that restoring your codebase with any dump you'll find is usually enough to get back in business again. Drupal doings things differently is not a problem per se and no one will argue over the fantastic improvements that CMI is bringing to the mix, but it'll cause pain and frustration for sure if this architectural change is not correctly conveyed. I can totally see this also making the entry level higher for new drupalists.

@yched I'm curious, is your custom script available on github by any chance? It looks like a possible workflow would be to cut a tag each time you're doing a DB backup...and to name this tag with the name of your DB backup, to make sure you'll roll back to the correct configuration. Quite a change indeed and it'd need a flawless workflow for dev teams.

yched’s picture

@anavarre: just posted it at https://gist.github.com/yched/7369557
(but drush seems currently broken on D8...)

anavarre’s picture

Thanks! The latest drush build should hopefully work. I'll try and play with this early next week and will report back here.

anavarre’s picture

Okay, I played with the drush script and it did work as advertised. Thanks again for sharing.

On my end I came up with the (very) rough https://github.com/anavarre/drupal-8-tools/tree/master/cmi to be able to still snapshot the DB and CMI files when drush isn't installed or not working for any reason (dev build broken for instance).

catch’s picture

Title: Database dumps might become disfunctional » Properly document backup workflow that includes database + configuration
Category: Bug report » Task

Is this better as an issue title?

anavarre’s picture

@catch: I don't think it's only about properly documenting the backup workflow, though it's crucial.

IMHO there's a strong concern about the underlying design which currently forces you to have a valid CMI files backup along with your DB snapshot. I realize it's by design and I don't have any problem with this per se (just need to plan for it), but like I said in #3, the entry level for new drupalists as well as the numerous issues we can think of - from people manually doing a DB snapshot only and ending up with a corrupted website to those messing with their dev / stage / prod workflow - requires that we step back a little.

I don't have any idea if other Open Source communities have ended up with the same design (configuration in files) and what decisions they made, but it would worth checking. I also have a few open questions:

  • What, besides UUIDs, would cause a CMI YAML file to be corrupted?
  • Would making progress on #2121751: [META] Making configuration synchronisation work change anything for this issue?
  • Put aside efforts, time and frustration, is there absolutely NO way to rethink the tight coupling between a DB and its CMI files to allow for more flexibility and protect those who didn't RTFM?
dasjo’s picture

i understand that CMI is all about putting config in files and appreciate the awesome achievements made here.

still, for the average site building experience, i think it makes more sense to put active config back into the database.

as far as i can see, putting active config into the database, wouldn't make any difference for the strength of cmi. because we only need the config in files when we are able to version it, move it around etc. this is what the staging directory is all about.

besides "this is by design", i'd like to see reasoning of why we should have an active config in files when nobody should touch the config. are there any performance advantages that i'm not aware of?

anavarre’s picture

are there any performance advantages that i'm not aware of?

Not loading everything that's inside the variables table on each page request and lazy loading of CMI files seems to be the most significant performance improvements I can think of, currently.

While it's not necessarily a performance improvement per se, CMI also removes the need completely for tens of database tables.

damiankloip’s picture

This performance based too; as you do not want to loading and parsing YAML from a file every time you load a config object. That would be bad.

catch’s picture

@dasjo that's been discussed a fair bit, and beejeebus has a contrib project enabling it: http://drupal.org/project/config_db

There are not really any performance advantages to the active config being in files at all, and it also causes problems for multiple web server configurations (since you have to have the configuration on a shared filesystem). The only thing that mitigates this is caching of config - which can equally be done for configuration in the db.

xjm’s picture

Component: configuration system » documentation
Issue tags: -DX (Developer Experience) +Configuration system

Not so much a DX issue. We do need docs though -- moving to the correct queue.

fago’s picture

I still think that quite some people will be bitten by that, however having explained CMIs to d8 newcomers in a d8 training I've seen how having CMI files written to the disk automatically is key to the understanding. So that's a plus for having active config in files (while it would be more optimal in the db workflow-wise).

jhodgdon’s picture

Can someone please make a concise and clear issue summary here? Thanks!

mtift’s picture

It might be premature to document a suggested workflow before we have a working configuration system. Currently, we are trying to make one workflow possible, and that is outlined in the issue summary in #2121751: [META] Making configuration synchronisation work.

I am going to group together a number of these workflow/process/documentation issues in one meta issue.

(On a related note, we will be discussing workflow issues at an upcoming CMI meeting: http://drupal8cmi.org/upcoming-cmi-meetings. All are welcome!)

fago’s picture

Status: Active » Closed (works as designed)

Given #2161591: Change default active config from file storage to DB storage the issue is resolved now as everything keeps working as expected :-)