[meta] D8 Extension System: Discovery/Listing/Info

Current action plan

• #2185559: Extension System, Part I: SystemListing clean-up + performance; remove SystemListingInfo + drupal_system_listing()
• #2188661: Extension System, Part II: ExtensionDiscovery
• #2208429: Extension System, Part III: ExtensionList, ModuleExtensionList and ProfileExtensionList
• #2659940: Extension System, Part III: ThemeExtensionList and ThemeEngineExtensionList
• #2941757: Extension System, Part IV: Properly register all installed extensions/namespaces during container generation
• Extension System, Part V: ExtensionInfo (is this still needed?)
• #3023133: [PP-1] Extension System, Part V: Decouple profiles from modules

Objective

1. Work on some installer related issues revealed that HEAD contains plenty of insane call chains in code relating to discovering and listing extensions.
2. Some of these call chains are causing an infinite recursion when calling the wrong function at the wrong point in time, because calls are hopping through legacy procedural global state and OO code and back.
3. On top, ~40% of DrupalKernel consists of very complex code and undocumented logic that only exists in order to cope with the fragile legacy extension discovery/listing code.
4. We need to fix this before release.

Goal

• Replace the entire legacy code with dedicated Extension component classes.

Preface: Drupal's Extension Architecture (current)

1. There are 4 extension types: profile, module, theme, theme_engine.
2. Every extension type has a standard file extension; e.g.: bartik.info.yml
3. Extensions are selectively discovered by extension type; e.g.: 'theme'
   1. Extensions of a certain type are expected to be located in a type-specific subdirectory; e.g.: ./themes
   2. Extensions are supplied by (1) Drupal core (2) An installation profile (3) The developer/site builder/end-user.
   3. To discover extensions, the installation profile and its directory must be known; e.g.: /core/profiles/standard
   4. Extensions are searched in multiple filepaths: (1) /core/$type (2) /$profile/$type (3) /$type (4) /sites/$site/$type
   5. If the same extension is found in multiple locations, then the later location overrides the former — unless the later one is not compatible (whereas "compatible" is limited to a Drupal core major version compatibility check as of now).
   6. The list of available/discoverable extensions is not expected to change within a single request at regular runtime.
   7. Extensions are normally not discovered while serving regular page requests.
4. A list of enabled/installed extensions is stored in configuration instead.
   1. The current extension list and their filesystem locations are compiled into the service container.
   2. If the configured list of extensions is changed, the service container needs to be rebuilt.
5. Each extension is required to provide an $extension.info.yml file, which contains meta information to describe the extension (package).
   1. The meta info of an extension/package is normally not used while serving regular page requests.
   2. Info Exception #1: Themes require their full meta information in order to operate. Therefore, the full theme meta info is persistently cached and retrieved from cache on every request.
   3. Info Exception #2: Various administrative user interfaces need to output the human-readable label of a module and thus need to retrieve meta info.
   4. Info Exception #3: Various subsystems/services need to know the installed version of an extension.
      (Applies to contributed extensions only. Core has a fixed version constant defined in code. Custom extensions do not have a version, unless a git commit hash can be determined from .git.)

Inventory/Stock-Check

DrupalKernel::getModuleFileNames() → DrupalKernel::moduleData()

Manually (1) scans for all installation profiles to (2) retrieve the path of installed profile, so as to (3) discover all applicable modules.

The resulting module/filepath map is compiled into the %container.modules% parameter.
The %container.modules% data is used to register module namespaces.

Automatically invoked whenever the service container is rebuilt.
The service container is rebuilt either by Drupal via DrupalKernel::updateModules() or manually via rebuild.php.

drupal_get_filename()

Legacy. A static variable that maps an extension to a filesystem path.

Formerly primed by system_list() with module and theme data retrieved from persistent cache.
system_list() is only used for theme data today.
Neither DrupalKernel nor ModuleHandler are priming the paths for modules in HEAD, which causes any module/profile filename retrieval to be a cache miss.

On cache miss,

1. for type module, the %container.modules% parameter is retrieved via ModuleHandler::getModuleList(), if that service available.
2. a system.$type.files state record is looked up, if the state service is available, and even though that record is only written by system_rebuild_$type_data() (and not by SystemListingInfo).
3. otherwise, a full filesystem scan is performed for the requested extension type via SystemListingInfo. The results are statically cached, so as to only scan once per request.

drupal_get_path()

Legacy. return dirname(drupal_get_filename($type, $name));

system_rebuild_module_data()

Legacy.

1. Discovers all available modules (and installation profile(s))
2. parses + enhances meta info
3. allows already installed modules to alter meta info
4. manipulates status + weight of enabled modules
5. stores the result in a system.module.files state record.

The generated system.module.files state is consumed by drupal_get_filename().

system_rebuild_theme_data()

Legacy. Calls ThemeHandler::rebuildThemeData() to

1. Discover all available themes and theme engines
2. parse + enhance meta info
3. allow modules to alter meta info
4. apply basic parent/sub theme info inheritance logic.

Then proceeds and

5. manipulates status + weight of enabled themes
6. stores the theme data/info result in a system.theme.data state record
7. stores the filesystem paths in a system.theme.files state record.

The generated system.theme.data state is consumed by system_list().
The generated system.theme.files state is consumed by drupal_get_filename().

Except for the parent/sub theme info inheritance logic, the basic meta info operations performed for each extension are identical to system_rebuild_module_data().

The parent/sub theme info inheritance logic is incomplete, both system_list() and later on ThemeHandler::listInfo() are performing additional inheritance logic.

install_profile_info()

Legacy. Parses meta info for a particular installation profile, applies defaults, and ensures required base system module dependencies.

system_get_info()

Legacy.

1. For type module, force-invokes module discovery via system_rebuild_module_data() (once per request) and returns the meta info, if the module is enabled.
2. For type theme, retrieves the cached data from system_list(), if the theme is enabled.

ProjectInfo

Utility class to perform some specific processing and validations for additional information that is automatically added to extension meta .info.yml files by drupal.org packaging scripts.

system_list()

Legacy. Cache item wrapper. Primes drupal_get_filename() with theme and theme_engine filepaths. On cache miss:

1. Reads list of enabled themes from system.theme:enabled
2. Consumes system.theme.data state record or calls system_rebuild_theme_data() to rebuild it.
3. Calls ThemeHandler::getBaseThemes() to determine the base themes of each enabled theme based on meta info.
4. Sets the theme engine for each theme.

list_themes()

Legacy. → ThemeHandler::listInfo()

ThemeHandler::listInfo()

Retrieves list of enabled themes and copies meta info properties into theme file object properties.

Calls system_list(), or manually system_rebuild_theme_data() upon failure/exception.

system_list_reset()

Legacy/obsolete.

1. Resets static caches of system_list(), system_rebuild_module_data(), list_themes().
2. Clears persistent cache of system_list() and system_get_module_info()/ModuleInfo.
3. Deletes system.theme.data state record.

In short: All static and persistent extension meta info caches are cleared.

YamlDiscovery

Independent utility class to discover and parse YAML files in a given set of directories. Primarily used by Plugin Managers, like this:

$discovery = new YamlDiscovery('routing', $this->moduleHandler->getModuleDirectories());
foreach ($discovery->findAll() as $provider => $info) {
}

Since the extension directories are injected as a dependency, this info file discovery + parsing tool cannot be used to discover the extensions themselves.

Data sources

$settings['install_profile']

The name of the installation profile Drupal was installed with.
(Only the name, not its filesystem location.)

simpletest.settings:parent_profile

Only exists when a (web) test is executed by Simpletest: The name of the installation profile of the parent site (test runner).
(Only the name, not its filesystem location.)

Enables tests using a particular installation profile to also find extensions in the installation profile of the parent site. Too complex to summarize in a few words; to learn more, see #911354: Tests in profiles/[name]/modules cannot be run and cannot use a different profile for running tests

Used by DrupalKernel::moduleData() and SystemListingInfo.

system.module:enabled

The configured list of enabled/installed modules, sorted by weight.

system.theme:enabled

The configured list of enabled/installed themes.
(Theme engines are not included.)

Application environment conditions

Exactly two:

Regular runtime

$settings['install_profile'] must exist, extension discovery and resulting listing depends on it.

(Early) Installer

There is no profile yet, extension discovery only lists (1) core extensions and (2) installation profiles.

Previously, there was a 3rd condition of update.php, which was not able to assume that the installation profile of a previous major version of Drupal core still exists, but that case is obsolete and gone with Migrate in core.

Proposed solution

Clean Inventory

1. Extension\Discovery
2. Extension\List
3. Extension\Info

Drupal\Core\Extension\ExtensionDiscovery

Pretty much replaces SystemListingInfo as-is, but tailors its architecture, implementation, and API to its primary use-case.

Since the assumption that available/discoverable extensions are not supposed to change within a single request is sound/reasonable, this could be a static utility class or singleton.

Regardless of implementation, the results should be statically cached, so as to prevent multiple filesystem scans triggered by different services that would yield the exact same result within a single request.

The extension discovery process will return SplFileInfo objects instead of the custom anonymous objects that just happen to have $uri, $filename, and $name properties. If time permits, we will wrap (extend) SplFileInfo in an Extension class.

To profile/benchmark: A single + highly optimized filesystem scan for *.info.yml

1. could be a lot faster than separate scans per type, especially given that we almost always scan for all extension types in case extension discovery is triggered anyway
2. could leverage the new required type: [type] property in .info.yml files (even more so, if we'd require it to be defined on the first file line)
3. would allow us to get rid of hook_system_theme_info(), which allows modules to ship with themes and only exists for simpletest.

Drupal\Core\Extension\ExtensionList / ExtensionHandler

Replaces legacy system_list(), drupal_get_path(), drupal_get_filename().

Central service to lookup filesystem location of a particular extension.

Uses core extension configuration to determine enabled extensions.
Uses the (1) service container [for modules/profiles] and/or (2) state records to retrieve filesystem paths.

-drupal_get_path('module', $name);
+$extensionList->getPath('module', $name);
+$extensionList->getModulePath($name);

Allows the /admin/modules page and rebuild.php to explicitly clear state information and rebuild it via ExtensionDiscovery, but otherwise, the existing state records are always used for lookups, and any kind of automatic rebuilding is strictly prohibited.

Partially represented in the form of ModuleHandler and ThemeHandler already, whereas the list facility of ModuleHandler is actually just the %container.modules% service container parameter at regular runtime.

ThemeHandler has no direct equivalent to ModuleHandler::getModuleList(); i.e., just the map of extension names to filesystem locations.

There is neither a ThemeEngineHandler nor a ProfileHandler yet. We either need to add those, or especially because the extension list logic is the same for all, that asks for a base ExtensionList/ExtensionHandler.

Also:

// Check all enabled + supported core extensions for whether they have Fu.
$skills = array();
foreach ($extensionList->getTypes() as $type) {
  $skills[$type] = $extensionList->discover($type, 'fu.info.yml'); // might be YamlDiscovery
}
return $skills;

Drupal\Core\Extension\ExtensionInfo

Replaces legacy system_rebuild_module_data(), system_rebuild_theme_data(), system_get_info(), install_profile_info(), system_get_module_info(), ModuleInfo, system_list(), list_themes(), ThemeHandler::listInfo(), etc.pp.

Central service to retrieve, process, and prepare meta info of all core extension types, and also store them in state (probably à la CacheCollector, but no, that is state information.).

State invalidation follows the ExtensionList/ExtensionHandler concept; i.e., only when explicitly requested. — That said, invalidation will probably have to be synchronized with ExtensionList/ExtensionHandler, so I'm not sure whether this facility should not be an integral part of ExtensionList/ExtensionHandler instead of a separate class/service.

Because our time to release is very limited, and because we have exactly 4 extension types in core, and because all of this is strictly bound and tailored towards the extension types we have in core, the concrete proposal is to simply hard-code the necessary post-processing for each extension type into this class/service — i.e., no subclasses or domain objects for each extension type.
(An extensible extension system is D9 material.)

Data sources

1. -$settings['install_profile'] = 'standard';
   +$settings['install_profile']['standard'] = 'core/profiles/standard';
   

   The filesystem location of the installation profile is required in order to efficiently discover + rebuild extension lists.

   This setting is not supposed to be touched by site builders/developers either way, so the exact data type/value in settings.php should not matter.

   Additionally, this opens the door to add support for #1356276: Allow profiles to define a base/parent profile, potentially even post-8.0.0.

2. -$conf['simpletest.settings']['parent_profile'] = 'openatrium';
   +$settings['simpletest_parent_install_profile']['openatrium'] = 'profiles/openatrium';
   

   ...or completely obsolete with (1), because with that, reality is just this:

   $settings['install_profile']['openatrium'] = 'profiles/openatrium';
   $settings['install_profile']['standard'] = 'core/profiles/standard';
   

3. -system.module:enabled
   -system.theme:enabled
   +core.extension:modules
   +core.extension:themes
   

   → /core/config/core.extension.yml (← "core.extension" == Drupal\Core\Extension)

   Introducing core configuration. (restricted to critical base system components)

   Resolving a major circular dependency problem in the installer, tests, and elsewhere: System module cannot be installed through regular means, because System module itself supplies the system.module configuration file.

   Likewise, #2184387: Remove SchemaStorage discovered that System module defines some basic configuration schema data types that have to be available at all times. But yet, for concistency, DX, discovery, and code re-use purposes, it would be preferable to keep defining them in a YAML file like all other config schema files.

   → /core/config/schema/core.data_types.yml

   The idea is to turn "core" into extension and extension type itself.

   So as to allow it to be a regular data provider like any other extension type. The only exception is that there is only one core, so $extensionList->getPath('core', 'whatever') will always return "core" (the extension name is ignored).

   Starting from core services (core.services.yml), to core libraries (core.libraries.yml), to base config schema, and base system extension (default) configuration, core already is a data provider like any other extension:

   /core/config/schema
   /core/config
   /core/lib
   /core/tests
   /core/core.libraries.yml
   /core/core.services.yml
   

Blockers

The primary purpose of this meta issue is to (1) raise awareness of the (non-obvious) problem space, (2) discuss the concrete proposal presented here, and (3) come to an agreement ASAP.

Given that other issues required me to make myself familiar with all of this code/insanity, I'd be happy to volunteer on my own to code up these gems, but before doing so, I'd like to achieve at least some basic level of agreement. (The devil is in the details anyway.)