[META] Allow a site to be installed from existing configuration

installer itself needs config to bootstrap, for now we have variable bootstrap_without_db to skip db cache loading

Yes, this seems like it would be a very useful step in the installation process to offer the option to (1) import configuration or (2) install from scratch

"Import configuration" should be an install profile... no?

Eclipse

@Eclipse - that would be excellent.
It would meet the needs to those who need to deploy a site build locally to another server, for the initial deployment .
Not sure where to start...

I guess I'm not really sure what this option buys you over install and then sync. As Kris said, if you really want this in one step you can easily do an install profile.

Ok - sounds good.
Install and sync won't work at the moment due to some UUID conflicts.
I'll open separate issues where appropriate.

After our discussions at BADCamp, I've changed my mind on this one. I believe "won't fix" is the correct status, but I'll leave it for now.

Done

I'm also stil at the point of

Install and sync won't work at the moment due to some UUID conflicts.

It wasn't stated in that way, but in my book #2110615: Do not ship with default UUIDs was postponed on this issue.

There might very well be valid reasons why this is won't fix, but "discussions at BADCamp" are not a sufficient enough reason to close an issue. Could someone post some discussion items / thoughts / ... around that topic and the discussion at BADCamp here? Thanks, that would be great.

Just re-read my #10 and realized it could come off as a bit negative. Sorry, I'm not a native speaker and that was not my intention. In no way do I actually want to discredit those discussions at BADCamp and I also wasn't offended or put off by the decision to close this issue. It's just that I'd like to know what the current thinking is as to me this is still very valid and it's something I might have worked on in the future.

This is now part of the plan for #2121751: [META] Making configuration synchronisation work. Given that this is a key delivery for Drupal 8 this should block the beta.

Patch adds the ability to install Drupal using existing configuration!

There is a tick box on the DB page to enable this. In order to test the patch:

1. Install Drupal
2. Make configuration changes
3. Copy the active configuration directory somewhere safe
4. Destroy Drupal site db, files and settings.php
5. In a fresh settings.php add a line pointing to the "somewhere safe" for the staging directory eg $config_directories['staging']['path'] = '/somewhere/safe';
6. Install Drupal selecting the same profile and on the db settings page tick the checkbox to use existing configuration
7. Enjoy the win!

Hm, I'm wondering how is this supposed to work. An install profile may create default configuration as well as default content. Eg. the Spark distro does that. Or Commerce Kickstart. Is this process supposed to skip all the creation of those default config but not skip the creation of the default content? I am kind of confused what to expect from this process in the first place.

So this may work with some profiles only then if the profile does not create any default content only default configuration?

The profile has to written to settings php because once installation is complete is should never change. Remember that many profiles contain modules so the installation profile fundamentally changes what modules are available. My original version of this patch had a config profile but this will not work for the that reason. The installation profile all sites being synchronised through full config sync must match.

@sun: all right, that makes the reach of this feature pretty limited. Eg. if you look at the standard profile, it creates 3 content entities (1 menu link and 2 shortcuts) and updates DB values for other entities, eg. assign user 1 the admin role. So this feature could not even reproduce the standard profile. What is it designed for and even more so why is it critical then?

CMI should not be controlling the available modules on a site - this is a recipe for dependancy disaster. As far as I'm concerned the installation profile belongs in settings.php - yes we can explore not presenting people with the profile selection screen if they've got a staging directory with configuration configured in their settings.php but this is just an implementation detail and not important.

Yes content entity staging is important to but actually everything that is in the standard_install() function actually works with this. The plan here is implement a hook or perhaps an new install step to create content entities. This gets even more complicated when you think about default values for entity reference fields or default images. But this is an existing problem!

Why is this patch critical - because it is key expectation of Drupal 8 to be able to rebuild sites from configuration. We need to be able to lock which sites can be sync because there is no way you sure be able to take a full config dump of an unrelated site and just import that. But we need some way of building that matching site - the way we build Drupal sites is through the installer. Once we lock config sync to only work between sites with the same site UUID #2133325: Create a site UUID on install and only allow config sync between sites with the same UUID this patch becomes critical.

Also through implementing default UUIDS in all config provided by modules and exploring whether or not a creation hash might be a better alternative we've hit more brick walls of complexity than I care to really think about. See #2127573: Stop creating node, comment and custom block fields automatically and provide defaults in CMI and #2138559: Replace UUIDs in configuration entities with a creation ID.

@sun available does not equal installed. For me available is the list of installed and uninstalled modules returned by _system_rebuild_module_data(). Needing a working CMI to return this list is going to cause heaps of pain some of which we went through in order to get enabled modules in CMI. Additionally, if profile was to be in configuration it would be alterable on the basis of language, domain or whatever - and I think this is going to lead to even more unintended consequences.

With regards to the site UUID - perhaps we should called site configuration sync UUID because then it use is better defined. Obviously this would be alterable too BUT altering it will only affect what can be synced and does not have a runtime impact like installation profile.

What if we envisioned that [site sync UUID + profile] info as a "special item" ?
- it's needed internally for the "config sync" & "install from existing config" processes
- as such, it needs to live in a file in the same folder than the rest of the config .yml's and be moved / deployed alongside
- but strictly speaking it doesn't need to be an actual configuration item, reachable through the regular config API ?

If we made it a special sync.metadata file (possibly skipping the .yml extension, even ?) that only the above processes read, then it stays out of reach of unwanted changes / context overrides ?

So, yeah, that would be special casing a file in there, but IMO it makes sense since it's metadata needed for the system to work ?

@sun: I don't agree entity implementations (content/config) should be changed based on whether they can then be pushed with this system or not, that seems to be totally the wrong perspective. Your profile may also depend on modules that create/update content. Eg. user module creates 2 users on install. Several modules also create configuration from PHP on install and #2127573: Stop creating node, comment and custom block fields automatically and provide defaults in CMI decided they should continue to do so. So for the idea in this issue to work at all, looks like we'd need a hook_install_config() and a hook_install_other_things() that modules and profiles would implement instead of hook_install() (which would also be needed to satisfy @alexpott's separation of content/config in the install profile). Unless that is done, we cannot rebuild a site with proper "content" (eg. default users) and still not take any config that the modules would create. (I consider that sheer luck that user_install() only does "content" related things now).

All-in-all this seems to be a contrived use case to me still.

Updated summary to explain criticality and use case.

So I discussed this at length with @alexpott on IRC and my summary is as follows:

- Drupal 6/7 has this "download the DB dump and files and import/place locally to reproduce a remote site for development" system which works well
- many people expect CMI to allow you to just take the config and **install** a local site with that (note no DB dump, the DB is created from scratch) reproducing part of that staging environment practice

It took so long for me to get this because I'm not part of those many people I guess. Maybe the old staging practice ingrained for me so long that I don't see much value in a "freshly installed site copy".

It would certainly provide a tool for scenarios like drupal.org where we could freely hand out copies of the configuration behind drupal.org without the need to write a full install profile (and without handing out the content) for development of new features (at least some of them, given IMHO many requiring actual content to be present). One could say "but the setup of the site is what the install profile should be about", but the argument here is that if you have a site copy made like this, config changes are syncable back to the original version as-is without further trickery (due to the uuids matching, so you can tell what is new and/or changed).

So this is basically a new config feature that apparently "many people expect" and it would work similarly but somewhat simpler/more limited than the "pull down db and files" staging setup methodology that we are used to.

I think this is a great feature, but I don't understand why it is 'Critical' (e.g. we wouldn't release Drupal without it). I don't see any rationale for it in #1613424: [META] Allow a site to be installed from existing configuration either. I know this is a complicated problem, but I'd love to better understand why it is critical.

Added a note on "creation ids based on entity name" (the "why it doesn't work" 1. to 5. bullet points don't make sense without it)

Some questions from a person who generally doesn't do much core dev but wants to see CMI advance in this direction…

Requirement 2: the ability to determine if a configuration entity has been deleted and recreated OR just updated
This is especially important for fields. As updating will not destroy data but delete and create will cause a field to be purged.

Would an entity which has been deleted and then recreated have the same UUID? I figured entities were getting random UUIDs upon creation.

@Garrett Albright you are correct that UUIDs are generated when an entity is created unless it is being provided by default configuration - which now it should not be.

Discussed with @alexpott. While I still think this is a necessary part of making CMI feature-complete, we do not need to block beta on the feature. The patch does not change the public Configuration API in a way that will impact the majority usecase for contrib.

Meanwhile, two recommendations:

1. This functionality depends on some refactoring for import that, per @alexpott "we probably need to do anyway". I'd suggest splitting that off into its own issue, separate from the functionality.
2. Let's start by not including any UI options in the installer -- the installer is heavily scrutinized for usability. We could add the functionality first using settings.php or suchlike, and then work on the UI problem in a subsequent issue.

For me, this one is one of the most important features that I expected from CMI.
What I expect is being able of working with my team as follows, without any db dump involved:

1. git pull my repo (that contains my site and the staging config)
2. drush site-install (if avoidable, the better, but I guess this is quite difficult to implement then)
3. drush config-import --yes
4. Do whatever changes I need (actual work).
5. drush config-export --yes
6. Commit related config in the staging folder for my "business feature".

15: 1613424.15.patch queued for re-testing.

Per discussion on IRC with alexpott and xjm, postponed on #2004370: Batch the configuration synchronisation process, and probably on others at #2187339: [META-0] CMI path to beta

#2004370: Batch the configuration synchronisation process has been committed.
Perhaps this can be reconsidered now

@alex - If the parent issue is committed, then should this functionality now be complete?
[Will get a chance to give it a try soon]

@alanburke configuration sync works (https://twitter.com/alexpott/status/451704896975556608) but there is no support for spinning up a Drupal site from configuration atm.

Excellent - I really need to get back to more D8 testing!

Until we have a content deployment functionality in core we are going to struggle with doing this when there is configuration that depends on content. If you're managing a deployment between live and dev at least you can create all content on live and bring back for dev and then push the configuration - this is not possible during installation.

Discussed with @alexpott. This (and content deployment) could make good minor version features.

The following script allowed us to import from an existing site after a site install:

  $site_uuid = Yaml::decode(file_get_contents("$root/profiles/foo/default_config/system.site.yml"))['uuid'];
  \Drupal::config('system.site')->set('uuid', $site_uuid)->save();

  // Adapt the default language. Since we use english as default, this can stay static.
  $language_en_uuid = Yaml::decode(file_get_contents("$root/profiles/foo/default_config/language.entity.en.yml"))['uuid'];
  $language_en_id = Yaml::decode(file_get_contents("$root/profiles/foo/default_config/language.entity.en.yml"))['id'];
  $language_en_label = Yaml::decode(file_get_contents("$root/profiles/foo/default_config/language.entity.en.yml"))['label'];
  \Drupal::config('language.entity.en.yml')->set('uuid', $language_en_uuid)->save();
  \Drupal::config('language.entity.en.yml')->set('id', $language_en_id)->save();
  \Drupal::config('language.entity.en.yml')->set('label', $language_en_label)->save();

@alexpott created https://www.drupal.org/project/config_installer to allow that.
Still would be precious to have in core IMO, even in a minor release.

Just curious : is there still a reason for this to be marked "postponed" ?

@penyaskito's comment in #46 is exactly what I expected too, and was surprised to see it did not work when we started working on an actual D8 project with my colleague. Discussed this offline with @yched friday, and I am under the impression other developers not doing too much D8 core are expecting this too : it is clearly a direct application of the much touted benefits of CMI.

Yeah, I'm kind of surprised that a contrib module is necessary to do this. I was under the impression it was a key feature of the whole CMI initiative.

I guess it's too late to core-ify it now, though. So I'll be bold and reassign it to 8.1.x myself. alexpott, what do you say about getting your module into core for the next release?

I'm more than happy to get this into 8.1.x. I have thought for a long time that we should use contrib as a proving ground for core work. So why not this install profile.

Some thoughts:

• The trickiest thing is the autoloader issue. Since the install profile is never truly used. Although this might just be exposing a bug in the installer. See http://cgit.drupalcode.org/config_installer/tree/config_installer.profile
• The only core hack I had to do to make it easy to use was #2384853: Both configuration directories have to be specified in settings.php
• For some reason the form added after the database details is not styled correctly. I think this is because we're in a super weird state - a fully bootstrapped Drupal with only system and user installed.

The fact that it's an install profile that actually never used is indeed a bit weird (and the series of steps for a fresh install of a real profile and for the creation of a new site instance from an existing config are fairly different).

I get that it's probably the only way to make it work from contrib, but maybe if it was in core it would be doable directly in the installer ?

re #62 you would still need the unused install profile - or something completely different (which would probably require a complete core installer rewrite) if we did this in core.

I've created an issue for the theming issue I discovered - #2385545: Installer can generate unthemed pages.

I don't normally post 'me too!' responses, but this issue is actually the first problem I ran across on my first attempt at using Drupal 8 (for serious work). I know it's just my opinion (not worth much), but I don't get why CMI even exists without: "the ability to install two Drupal sites and sync configuration between them".

@Liam McDermott checkout https://www.drupal.org/project/config_installer

FYI, Drush has a new option on site-install command which provides this functionality. Needs testing and improvement. See https://github.com/drush-ops/drush/pull/1875

Here is a patch that just adds www.drupal.org/project/config_installer to core.

Perhaps we should mark it as experimental for 8.1.x? As it is not a module it can't be in the experimental package.

Fixing SingleVisibleProfileTest

Ah PHP 5.5 vs PHP 5.6

1. +++ b/core/profiles/config_installer/config_installer.info.yml
   @@ -0,0 +1,7 @@
   +# Un comment to completely remove profile selection.
   +#distribution: TRUE
   

   I think we don't need this for core any longer right?

2. +++ b/core/profiles/config_installer/config_installer.profile
   @@ -0,0 +1,266 @@
   +/**
   + * Need to do a manual include since this install profile never actually gets
   + * installed so therefore its code cannot be autoloaded.
   + */
   +include_once __DIR__ . '/src/Form/SiteConfigureForm.php';
   +include_once __DIR__ . '/src/Form/SyncConfigureForm.php';
   +include_once __DIR__ . '/src/Storage/SourceStorage.php';
   +
   

   Just an idea, could we here register the install profile in the autoloader instead?

3. +++ b/core/profiles/config_installer/config_installer.profile
   @@ -0,0 +1,266 @@
   +function config_installer_install_tasks_alter(&$tasks, $install_state) {
   +  $key = array_search('install_profile_modules', array_keys($tasks));
   

   IMHO a short oneliner what we do here would be nice: Add upload and install config task after the install modules task.

4. +++ b/core/profiles/config_installer/config_installer.profile
   @@ -0,0 +1,266 @@
   +      'function' => 'Drupal\config_installer\Form\SyncConfigureForm'
   ...
   +  $tasks['install_configure_form']['function'] = 'Drupal\config_installer\Form\SiteConfigureForm';
   

   Nitpick: We could use ...ConfigureForm::class

5. +++ b/core/profiles/config_installer/config_installer.profile
   @@ -0,0 +1,266 @@
   +      'file' => drupal_get_path('module', 'config') . '/config.admin.inc',
   

   Does that mean this install profile requires the config module to work?

6. +++ b/core/profiles/config_installer/config_installer.profile
   @@ -0,0 +1,266 @@
   +    drupal_set_message(\Drupal::translation()->translate('The configuration synchronization failed validation.'));
   ...
   +      drupal_set_message(\Drupal::translation()->translate('The configuration was imported with errors.'), 'warning');
   ...
   +    $message = \Drupal::translation()
   +      ->translate('An error occurred while processing %error_operation with arguments: @arguments', [
   ...
   +      drupal_set_message(\Drupal::translation()->translate('The configuration synchronization failed validation.'));
   

   Nitpick: Let's just use t()

7. +++ b/core/profiles/config_installer/config_installer.profile
   @@ -0,0 +1,266 @@
   +    if (!isset($context['results']['errors'])) {
   +      $context['results']['errors'] = [];
   +    }
   

   Just use $context['results'] += ['errors' => []], no reason to have 3 lines here.

8. +++ b/core/profiles/config_installer/src/Form/SiteConfigureForm.php
   @@ -0,0 +1,213 @@
   + * This is based on the install_configure_form provided by core.
   + *
   + * @see \Drupal\Core\Installer\Form\SiteConfigureForm
   + */
   +class SiteConfigureForm extends FormBase {
   

   Did you considered to extend from the other form and remove everything unneeded instead?

9. +++ b/core/profiles/config_installer/src/Form/SyncConfigureForm.php
   @@ -0,0 +1,131 @@
   +    $form['sync_directory'] = [
   +      '#type' => 'textfield',
   +      '#title' => $this->t('Synchronisation directory'),
   +      '#default_value' => config_get_config_directory(CONFIG_SYNC_DIRECTORY),
   +      '#maxlength' => 255,
   +      '#description' => $this->t('Path to the config directory you wish to import, can be relative to document root or an absolute path.'),
   +      '#required' => TRUE,
   +    ];
   +
   +    $form['import_tarball'] = [
   +      '#type' => 'file',
   +      '#title' => $this->t('Select your configuration export file'),
   +      '#description' => $this->t('If the sync directory is empty you can upload a configuration export file.'),
   +    ];
   

   I'm wondering whether the import_tarball should be the first choice instead.

10. +++ b/core/profiles/config_installer/src/Form/SyncConfigureForm.php
    @@ -0,0 +1,131 @@
    +          '@handbook_url' => 'http://drupal.org/server-permissions',
    

    Nitpick: Let's use :handbook_url, just for consistency

11. +++ b/core/profiles/config_installer/src/Form/SyncConfigureForm.php
    @@ -0,0 +1,131 @@
    +      if (count($sync->listAll()) === 0) {
    +        $form_state->setErrorByName('sync_directory', t('No file upload provided and the sync directory is empty'));
    +      }
    

    I'm wondering whether we could apply some better validation? Is there any more sane way to ensure that its a full export?

12. +++ b/core/profiles/config_installer/src/Tests/ConfigInstallerFrTarballTest.php
    @@ -0,0 +1,79 @@
    +class ConfigInstallerFrTarballTest extends ConfigInstallerTestBase {
    

    Fr? Is this supposed to just work on fridays? Seems like a fair feature

In short: when performing two identical operations (site install), the result should be the same (config files).

I'd say this is critical when using Drupal as an application (that needs to be reinstalled on deploy) and don't want to move around database dumps. Subscribing.

At the moment yml files contain state (they're environment/installation dependent).
It should be possible to install multiple times and have the same result each time (without needing a custom install profile that is not an install profile).

A lot of people are circumventing this by overriding the site's uuid but as Berdir mentions the risk for loosing data is huge if UUIDs are different.

I need to enable my own profile and import config from the repo to build a fresh site from code.

i.e. drush site-install my_profile --config-dir=PATH/TO/config/base --yes

How can I use config_installer to import the config? It is not possible to have two profiles enabled at the same time.

@marcelovani

Have you tried the following:
drush site-install config_installer --config-dir=PATH/TO/config/base --yes
If I understood it correctly, config_installer is used as a kind of wizzard that set's up the site with your configuration during the installation process. And as part of that it may realize that your site is acctually using the my_profile install profile & set's it up along with the configuration.
If you visit the site afterward ony my_profile is enabled because the config_installer was only used during installation.

install time = config_installer
website run time = my_profile
=> only one profile active at a given time

I think I figured out why it wasn't working and how config_installer knows that I want my_profile enabled.

Basically core.extension.yml is what defines which profile will be enabled.

I had an entry for standard: 0 which I changed to my_profile: 0.

Now it works.

I can only make it work using the UI. When using drush, I get an error saying that the folder config/base is empty.

Yep, this is a profile so it suppose to be used with UI only, is not it?

But it makes sense to have drush commend as well)

Looking at code http://cgit.drupalcode.org/config_installer/tree/config_installer.profil... I think we still depend on #2208429: Extension System, Part III: ExtensionList, ModuleExtensionList and ProfileExtensionList for proper detection

I finally figured out how to make it work using drush.

1. Add this to the settings.php: $config_directories['sync'] = '../config/sync';
2. Run the drush command for config_installer profile without the --config-dir option: drush si config_installer --yes

ps: config/sync folder should be on the same level as the docroot(web) folder

Just a small post-it about this issue actually one of my projects try to use full fonctionalities of CMI & Synchronisation config with simple composer commands. It's seems like config_installer but need to use and depend to composer.

I hope this project can help anyone to seen CMI works (Thanks to your hard work about config_installer).

Ok let's get this back on track.

config_installer is great and when setting the sync directory first drush has no issue installing a site with it. But I think for the MVP here, as the issue summary suggests, it would be enough to skip the UI part for selecting a tar to upload and just set existing_config when the sync directory is set and filled. (The service user form is still needed of course.)

WIP

missing config validation and fixes with multilingual

I added validation, but that one still needs to be tested, for some reason the multilingual tests crash on my system. Maybe there are other failures too.

Since the issues are related, I also added an interdiff to the patch in #2788777: Allow a site-specific profile to be installed from existing config.

Installing profiles that depend (in their install hooks and elsewhere) on modules that can be uninstalled is fundamentally at odds with installing a site from existing configuration.

Profiles that mark themselves as compatible could be re-installed more easily. #2788777: Allow a site-specific profile to be installed from existing config

So I think it makes sense postponing this issue on fixing the profile dependencies first.

Can anyone comment on how this is different from running drush si with the --config-dir argument? Is this essentially applying that same logic to Drupal core and/or the installation UI?

@Dane Powell from my testing with Lightning, the difference is that using --config-dir with drush si can cause the problem mentioned by you here: https://github.com/acquia/lightning/issues/387 whereas the patch in #2788777: Allow a site-specific profile to be installed from existing config doesn't seem to have that problem with Lightning, but has the downside of setting the same uuids on every Lightning site and only working with profiles.

I haven't figure out the patch in *this* issue yet though. Seems like a huge patch compared to others (maybe because of the tests) and modifies the installer UI. I haven't yet figured out the exact "drush si" command to use with this patch since all my installs are from command line.

#87 updated to 8.4.4

postponed on #2952888: Allow install profiles to require modules

It appears Drupal Console have figured out how to install a site from config.
https://weknowinc.com/blog/how-install-drupal-8-existing-configuration

@joshua.boltz it is not the same thing. In the linked blog post they are first installing a site with standard profile config and then importing existing config on top of that. Whereas in this issue we are installing site using the existing config.

Tagging for potential CMI 2.0, see #2957423: Proposal: Configuration Management 2.0 initiative.

#2788777: Allow a site-specific profile to be installed from existing config is now committed.

Made the issue into a meta and updated the description a bit. Feel free to improve it further.

How is this going on??
There are some options with Drupal Console.

Moved #2982052: Allow an install hook in profiles installing from configuration to be attached directly to #3008882: Roadmap for CMI 2
@bircher and I decided that #2982056: Add options to the installer UI which allow a user to upload a zip with configuration or specify a config directory might not be needed so punted that issue to 9.1.x and disconnected it from CMI 2.0.

Therefore this issue is done.