Shift to a local palette structure

[bourdakos1]

This is a bit of an extreme proposal, but I think we should move from a "global" palette of components (every pipeline opened has access to all components) to a "local" palette of components (components are stored in the pipeline and only available to that pipeline).

The largest benefit from this change would be portability. Since all components would be defined in the pipeline file users wouldn't need to manually install any components when using a shared pipeline. Additionally, relative paths to components could be relative to the pipeline (similar to how notebook nodes work).

I would see a common pattern being having a repo of common utility components that you would want to use across most projects. This repo could be easily added as a catalog to any new pipeline. Any additional one off components could also be easily added with a couple clicks. Finally, custom components could be locally stored in the project and referred to with a relative path

thoughts?

(I know we will probably still need global, but I think it should be leaned on in moderation/not a best practice)

[ptitzler]

It's an interesting idea (creating a self-contained pipeline file with embedded components) but would create a glob of potential issues, such as

• a global palette would still be needed because the component specification has to come from somewhere when the pipeline is created
• every pipeline would include the complete component specifications (which can be large Python scripts, e.g. for Airflow, not just relatively small YAML files)
• the VPE would have to have the ability to refresh embedded specifications (as they will change over time)
• pipeline file copies would get out of synch over time (not a problem specific to the proposal but much more likely to happen)
• users would have to have the ability to choose wether to embed specifications or include them by reference (current behavior)
• there are many entities that consume component specifications, Elyra is just one of them. The embedded specification text therefore cannot be the single source of truth.

[bourdakos1]

every pipeline would include the complete component specifications (which can be large Python scripts, e.g. for Airflow, not just relatively small YAML files)

Oh, I didn't mean store the complete component specification. We would keep the same structure we currently use, just inside the pipeline file, something like:

{
  "catalog_type": "URL",
  "paths": [
    "https://raw.githubusercontent.com/kubeflow/pipelines/1.6.0/components/web/Download/component.yaml"
  ]
  ...
}