WIP: PipelineDefinition

PipelineDefinition reference page

In-progress design notes

2024-12-12: Pipeline Graph Path names

Discussion page

The current PipelineDefinition uses the name of the first PipelineElement in a Graph Path as the name of the Graph Path. This is an indirect, inflexible and non-obvious means of naming Graph Paths. If multiple Graph Paths start with the same PipelineElement, then the PipelineElement needs to appear more than once in the "elements" list and be renamed accordingly ... not elegant at all !

"graph": [
  "(TextReadFile          TextSummarize TextWriteFile)",
  "(TranslateTextReadFile TextTranslate TextWriteFile)"
],
"elements": [
  { "name":   "TranslateTextReadFile",  # unfortunate need to rename the PipelineElement to avoid a name re-use conflict
      "deploy": {
        "local": {
            "class_name": "TextReadFile",

The proposed PipelineDefinition uses a "path:" prefix to indicate that this is a Graph Path name and not an actual PipelineElement. A bonus is that a similar syntax can be used for other upcoming requirements for adding features to the Graph Path elements

"graph": [
  "(path:summarize TextReadFile TextSummarize TextWriteFile)",
  "(path:translate TextReadFile TextTranslate TextWriteFile)"
]

Future thoughts

• Part of the current Pipeline design / implementation (and similarly the PipelineDefinition) also applies to Actors and Services ... and should be refactored accordingly, e.g StateMachine (TBC), Streams, Parameters and the Protocol (declaration of method signature inputs / outputs)

• Remove the need to define all the PipelineElements details, by using code reflection

• Replace JSON with S-Expressions