Pipeline parameters UX proposal

[ptitzler]

Motivation

Users are looking for the ability to customize pipeline node inputs without the need to edit the pipeline in the pipeline editor.

Example:

• User creates a pipeline that executes one node. The node contains an input property (say a threshold) that impacts the output of the node.
• The user wants to run the pipeline multiple times (e.g. using the Elyra CLI or from the runtime environment's GUI) and use a different value for each run.

As of version 3.12 the user has to modify the pipeline file using the pipeline editor to accomplish the goal.

Pipeline parameters would enable us to provide the means to customize certain properties, which are evaluated at the time the pipeline is processed (exported/submitted for execution), which can happen outside the pipeline editor.

Example how pipeline parameters are exposed in the Kubeflow Central Dashboard:

Creating pipelines with parameters

Expose pipeline parameters in pipeline editor

• If the runtime supports pipeline parameters, the properties slide-out includes an additional tab (e.g. pipeline parameters).

---

Define pipeline parameters

• User can define a list of parameters.
• By default a pipeline does not include any parameters.
• A pipeline parameter definition comprises
  1. A unique key, identifying the parameter name, e.g. PARM_1
  2. An optional default value. A default value makes it possible to process ('submit'/'export') a pipeline using the pipeline editor without the need for the user to specify a custom value. (Refer to later section on usage.)
  3. A parameter type, which can be initially only 'string', with support for other primitive types added in the future. Typing allows for basic validation and is only applicable if applicable node properties are typed. For example, if a node property PROP1 is typed number, only compatible parameter can be assigned to that property.
  4. An attribute that defines whether a parameter value is required. If a parameter is required, a parameter default value must be defined or a user-provided value exist or the pipeline will not be processed by Elyra.

Runtime parameter definitions are persisted in the .pipeline file, e.g. as siblings of pipeline_defaults as shown in this mockup:

     "app_data": {
        "ui_data": {
          "comments": []
        },
        "version": 8,
        "properties": {
          "pipeline_parameters": [
              {
                "name": "PARM_1",
                "default_value": "default_value_1",
                ...
              }
           ],
          "pipeline_defaults": {
              ...
          }

---

Add 'pipeline parameter' as property value input option

• This only applies to pipeline editors for runtime environments that support pipeline parameters.
• Only node properties that serve as component inputs should accept parameters as input.
• For those properties the input selector list is extended to include a new source: pipeline parameter.

---

Assign pipeline parameter to property

• If 'pipeline parameter' is selected as property input source, the user can choose an entry from the list of available type-compatible parameters.

  

The pipeline editor stores the association between the property and the parameter name in the .pipeline file, as shown in the last line of this mockup:

            "component_parameters": {
              "dependencies": [],
              "include_subdirectories": "false",
              "outputs": [],
              "env_vars": [],
              "kubernetes_pod_annotations": [],
              "kubernetes_pod_labels": [],
              "kubernetes_secrets": [],
              "kubernetes_tolerations": [],
              "mounted_volumes": [],
              "algorithm": "...",
              "threshold": "<REF-TO-PARM_1>"

---

Processing pipelines with parameters

When Elyra processes a pipeline today the following happens:

• User invokes export or execute.

• User chooses target runtime environment.
  

• Pipeline is validated and then exported or executed.

  

The following applies only to runtime editors that support pipeline parameters.

The steps remain the same if a pipeline does not include any node properties that are configured to use pipeline parameters as input. If at least one pipeline parameter is assigned as an input, the process changes as follows:

• User invokes export or execute.

• User chooses target runtime environment.

• User assigns a custom value to each referenced pipeline parameter. The default is rendered as placeholder and an explicit or implicit value must be assigned if the parameter was defined as required.

  

• Pipeline is validated and then exported or executed.

CLI

• 'run'/'submit'/'export' commands are extended to allow for parameter value override (e.g. parm_1=custom_value_1)
• 'describe' command is extended to list pipeline parameters

Runtime env tooling

• parameters are exposed as runtime env specific 'run parameters' and can be customized

[ptitzler]

The Elyra KFP DSL code generator produces output like the following, if the user defines one or more pipeline parameters, such as parm_1 or some_other_parm:

def generated_pipeline(
    parm_1: str = "default value",
    some_other_parm: str,
):

    task_7a9 = (
        factory_635(input=parm_1)
    )

Since the pipeline parameter names are used as a Python variables, they must meet these rules. While it would be possible to sanitize user-provided parameter names, Kubeflow Pipelines would only be aware of the sanitized name. When interacting with Kubeflow Pipelines outside of Elyra (e.g. using the Central Dashboard) the user would have to look/use the sanitized name, which might look very different. For example, a user-provided parameter name of 9 lives of a cat might be sanitized to a9_lives_of_a_cat.

[kevin-bates]

Hi @ptitzler - Are you suggesting that the Elyra UI (and/or validation) enforce parameter name syntax so that there's only one interpretation? I think that makes a lot of sense but couldn't determine that from your comment. When read, it seems like more of a "heads up" statement/warning.

[ptitzler]

Yes, that would be my preferred approach! Since the pipeline editor is the only entity where parameter names can be defined, it should be straightforward to enforce the constraints. We already have precedence for doing that (label and annotation names come to mind), so the overall user experience would be consistent.