Elyra 4.0: breaking changes proposals

[ptitzler]

With the delivery of Elyra 4.0 (a major version increase) we have the opportunity to introduce breaking changes that enable us to

• remove functionality deemed no longer necessary (e.g. marked deprecated in earlier releases)
• improve existing functionality in a way that is no longer compatible with earlier releases

Please add proposals as responses below, providing:

• description/motivation
• impact (GUI, CLI, both)

[ptitzler]

description

Remove support for legacy connector API from the server back-end. The legacy API was marked deprecated in 3.7. #2514

impact

• Existing connectors that use the legacy API will stop working.
• Known impacted connectors (all other community-maintained connectors already use the new API):
  • https://github.com/elyra-ai/examples/tree/master/component-catalog-connectors/airflow-example-components-connector (The connector was marked deprecated and will therefore not be updated to use the new API.)

[kevin-bates]

description

The elyra-metadata tool has some odd-named commands and option defaults that I'd like to see changed in 4.0:

1. To update instances users must include --replace. I think --update would be more natural.
2. Likewise, I've always found the install sub-command to create instances to be unusual. Perhaps this could be renamed to create and we could let --replace (i.e., update) be its own sub-command so elyra-metadata update would be use to update and elyra-metadata create to create instance.
3. The --valid-only option is added when the user wants only valid metadata instances to be listed or exported - with the default being to include invalid instances. I think its more intuitive to use an --include-invalid flag so that the invalid instances are listed (or exported) .

impact

Docs would need to be updated, along with the CLI (obviously), but I think these changes would make the tool more intuitive. The other commands (list, remove, export, and (todo) import) all seem intuitive already.
Folks that have scripts would require updates.
We could explore making these changes prior to 4.0 with a deprecation cycle whereby the existing forms continue to work until removed in 4.0.

[ptitzler]

• +1 for # 3

• +1 for # 2 (elyra-metadata create & elyra-metadata update) + I've always been a fan of upserts. they tend to save a lot of time

• this would also impact existing Dockerfiles that install metadata

[ptitzler]

3.7 tasks:

• Metadata CLI: replace 'install' command with 'create' and 'update' commands #2563

4.0 tasks:

• Metadata CLI: Change 'list' command behavior #2562
• Remove support for elyra-metadata CLI command 'install' #2580

[ptitzler]

Delivered in #3143 and #3142

[ptitzler]

description

Every Elyra installation currently includes a set of system-owned runtime image configurations, which were included to make it easier for users to get started with pipelines. The proposal is to remove these RTIs from the Elyra build and provide (where possible) the means to add them in a way that enables users to remove them, without the need for an Elyra rebuild. For details see #2476

impact

• Elyra pip/conda installations do not include system-owned runtime configurations anymore. Users have to explicitly create one before they can run pipelines in any runtime environment. (Slight degradation of the up-and-running user experience)
• Elyra would provide the following pre-built container images:
  • elyra/elyra-slim - Elyra installation that does not include any predefined RTIs
  • elyra/elyra - Elyra installation that does include RTIs (same as before), but they are not system-owned
  • elyra/kf-notebook-slim - Elyra installation that does not include any predefined RTIs
  • elyra/kf-notebook - Elyra installation that does include RTIs (same as before), but they are not system-owned
    Note: elyra/elyra would extend elyra/elyra-slim, and elyra/kf-notebook extend elyra/kf-notebook-slim
• Documentation and examples would require updates
• VPE welcome screen would require updates (minor change)

[lresende]

Not having the concept of a pre-installed runtime image then make it hard to have users up and running out of the box as well, it also makes it more complicated to get some sort of examples that work out of the box.

And this is all because the user reported on #2476 that he needed the ability to delete these runtime-images? Then make them delectable? Maybe enabled by a configuration? And leave the example images there.

[kevin-bates]

Not having the concept of a pre-installed runtime image then make it hard to have users up and running out of the box as well, it also makes it more complicated to get some sort of examples that work out of the box.

We all agree about this. However, it seems like users, while following the documentation for how to work with the examples out of the box will first encounter an item that instructs them how to create a runtime image instance that references one of the several images we provide (per Patrick's comment). As a result, I don't think the copy/paste of a one-line command, in light of the rest of the Elyra's complexity, will introduce that much interference.

Then make them delectable?

It's not that simple. Making them deletable is not the issue. The issue is making them permanently deleted, because as soon as they upgrade/patch Elyra, they'd all come right back and you have a Groundhog Day syndrome on your hands. in addition, system-owned instances can only be supported with filesystem-based persistence. Not to mention the general confusion and hassle of maintaining this kind of behavior.

Maybe enabled by a configuration?

This is what @ptitzler mentions in #2476 (comment) where the system-owned-instance behavior is disabled (across all schemaspaces) via a configuration option. If we did this, users could capture the images they want to use by "modifying" the system-owned instance (prior to setting the configuration value) which then copies those instances in the user space (making them user-owned), then disable system-owned instances. Users can then "permanently delete" those images until the system-owned instances are enabled again - at which time they "reappear".

And leave the example images there.

Yes, this is the other issue. There's no way to clean up these instances. They will be present long after Elyra has been deleted. 👎

[ptitzler]

makes it more complicated to get some sort of examples that work out of the box.

Example pipelines are preconfigured and should therefore always work as is, whether or not runtime images are defined or not.

[kevin-bates]

Oh right, because they reference the image directly. Good point.

[ptitzler]

The other problem with system-owned instances is that they make day-to-day life unnecessarily harder. For example

• override a system owned instance (e.g. runtime-image, by changing the container image tag)
• elyra-metadata list the instances (the customized version appears in the list, the system-owned one does not)
  -elyra-metadata export the instances (this exports the customized instance, as expected)
• remove the customized instance
• try to elyra-metadata create the instance using the exported JSON. the operation fails because there is already a [system-owned] instance with that name. therefore the user has to use elyra-metadata update, which implies that they need to understand that there are these special instances that 'magically' appear even though they were hidden 'before' - see output of elyra-metadata list. (the upcoming elyra-metadata import command will behave the same and not update existing metadata by default)

All in all I believe system-owned instances cause more problems than they solve.

[ptitzler]

description

Users can migrate pipelines that were created using older Elyra releases to the latest release (7). Currently the VPE supports migration for pipelines as old as version 'X'. By now most of these pipelines (that are still in use) would have been already migrated by customers to a more recent version than 'X'. (If they weren't migrated that implies that the user is still using an older version of Elyra or is no longer using those pipelines.) We should therefore consider removing migration support for pipelines that are older than version 'Y' to reduce the amount of unused code.

The https://github.com/elyra-ai/examples repository still includes pipelines version 4. These pipelines (*) were intentionally not migrated because they utilize components that were deprecated and are no longer maintained. We should consider removing those pipelines in Elyra v4.0.

• TODO: identify suitable candidates for version X and Y (@ajbozarth any recommendations?)

impact

• (*) Remove https://github.com/elyra-ai/examples/tree/master/pipelines/airflow_component_examples

[ajbozarth]

There is a really large leap between v3 and v4 since we did not update the pipeline schema during the Elyra 2.X release cycle. Based on this timeline most user will be either on pipeline v7 if they're up to date on 3.X or pipeline v3 if they're using 1.X or 2.X, so we would not be able to remove the v3->v4 migration.

Looking at the migration code, v0->v1 is mostly clean up of un-versioned pipelines to match the v1 schema, v1->v2 handles the change to using pipeline relative paths, which the removal of would simplify the api due to the removal of the required callback function, and lastly v2->v3 is a no-op (we added script node support).

So realistically we would only to able to remove the v0->v3 migration with a release of Elyra 4.0 (but as shown above this would be helpful).

[ajbozarth]

Also as another idea for streamlining migration. We could do a forced version bump with 4.0, let say to v8. Then for 4.0 and on, users can migrate from either v3->v8 or v7->v8 and if they want to migrate anything else they should open a pipeline in the latest 3.x release. This would be very straight forward if we make v7->v8 a no-op and v3->v8 would be very small since the large majority of pipeline migration for v4->v7 is for kfp and airflow pipelines which did not exist before v4 (eg. for generic pipelines: v4->v5 is a no-op, v5->v6 deletes a value, and v6->v7 is also a no-op)

tl:dr v3->v4 is functionally the same as v3->v7

[ajbozarth]

ok so after all that mind dump here's my pitch of two options:

1. We remove support for v0->v3 migration and only support migrating v3 - v7 pipelines starting with 4.0

2. We simplify migration support and with the release of 4.0 we increment to pipeline v8 (assuming we don't bump the version before then) as a no-op. We then only support migration of v3->v8 (for 1.X and 2.X users) and v7->v8 (for 3.X users) and direct users to migrate any other versions (v0, v1, v2, v4, v5, and v6) using the latest Elyra 3.X.

   • This would also allow us to easily move migration to the server since v3->v8 would only consist of the code found in v3->v4 migration and the delete line in v5->v6, easily done on the backend.
   • We could then implement a policy of doing a no-op version bump with each major release and requiring user to use previous releases for mid release cycle migration (eg. v4-v7 in 3.X)

[ajbozarth]

I've updated #1923 with our decisions from today's meeting on this topic including the future versioning policy and our specific plans for 4.0

[ptitzler]

Updated #1190 with the latest information from above