Pipeline

The Pipeline module allows splitting processes in steps and then running them via an orchestrator service. The main goal of this module is to make it easier to execute long processes without running out of memory or timing-out.

Architecture

Pipelines are defined as plugins of type @PipelinePipeline(). A pipeline consists in a number of pipeline steps. Each step is a plugin of type @PipelineStep().

Pipelines

A @PipelinePipeline() plugin should implement the PipelinePipelineInterface interface. The module provides a PipelinePipelinePluginBase abstract class as a starting point for pipeline plugins. It's recommended to extend this abstract when creating a new pipeline plugin.

A pipeline plugin is declaring a list of pipeline steps in its annotation, under
the annotation steps key. The steps will be executed in the order specified in the pipeline annotation. Each item from the list is a pipeline step plugin ID.

/**
 * @PipelinePipeline(
 *   id = "import_data",
 *   label = @Translation("Import geographical data"),
 *   steps = {
 *     "download_file",
 *     "normalize_values",
 *     "metric_to_inch_conversion",
 *     "store_data",
 *   },
 *)
 */

It's also possible to pass pipeline step configurations to the steps that are requiring to be configured. See how store_data step plugin is described:

/**
 * @PipelinePipeline(
 *   id = "import_data",
 *   label = @Translation("Import geographical data"),
 *   steps = {
 *     "download_file",
 *     "normalize_values",
 *     "metric_to_inch_conversion",
 *     "store_data" = {
 *       "extension" => "txt",
 *       "encoding" => "UTF-8",
 *       "line_endings" => "LF"
 *     },
 *   },
 *)
 */

Steps

A @PipelineStep() plugin should implement the PipelineStepInterface interface. The module provides a PipelineStepPluginBase abstract class as a starting point for pipeline step plugins. It's recommended to extend this abstract when creating a new step plugin.

The module allows different kind of steps, depending on the time needed for that step. For example a very intensive step should be defined to run as a batch process, while a good performing step should run in a single request. It's also possible to allow consecutive performing steps to run in the same request. Form a business logic perspective, a pipeline may need a step that requires user input. It's possible to define such step plugins, that are exposing a UI and waiting for the user input in order to execute their logic.

• Fast steps: These are steps that have no performance impact. These plugins will extend PipelineStepPluginBase. If there are consecutive steps of this kind, they will all run in the same request. In order to end the request after executing its logic, the step should implement also the PipelineStepWithResponseInterface interface. The response could be a HTML page or a redirect response. Depending on the business logic, the steps responding with a redirect may want to use one of the shipped traits:
  • PipelineStepWithRedirectResponseTrait,
  • PipelineStepWithClientRedirectResponseTrait.
• Intensive steps: These steps are performing intensive operations that cannot run in a single request, thus we need to run them as a batch process. Such step plugins should implement also PipelineStepWithBatchInterface interface and use the shipped PipelineStepWithBatchTrait trait.
• User input steps: Sometimes, a process (pipeline) need a step plugin that exposes a form to the user, waiting their input, before executing its logic. Such steps should implement PipelineStepWithFormInterface interface and make use of PipelineStepWithFormTrait to fulfill their logic.