Improvement to dataplane selection and configuration

[paullatzelsperger]

Feature Request

This is a story issue that spans across several subtasks and aims to improve the data plane capabilities in the aspects detailed here.

1. The transferType as key indicator

In the provider data plane, the transferType (sometimes referred to as "format") should dictate the physical destination of a data pipeline:

• in PULL transfers: the DataSink is inferred, e.g. HttpData-PULL means, the data is piped into a HttpDataSink
• in PUSH transfers: the DataSink is inferred, but additional data is required, e.g. AmazonS3-PUSH, the data is pushed into a S3BucketDataSink, but the bucket name, region, etc. must be provided by the consumer.

Note that the source of the data pipeline is always determined by the DataAddress that is associated with the Asset.

From this, we can derive the following requirements:

1. There must be a way to infer the data destination type from the transferType. I.e. the provider DP must be able to infer from HttpData-PULL that it must instantiate a HttpDataSink. This must be extensible.
2. The data plane registration only contains a list of explicit mappings transferType -> destinationType. To achieve that, the filtering based on the sourceType should be dropped, and there must be an explicit mapping instead of two disjoint lists ("allowedTransferType", "allowedDestinationTypes").
3. Changes to the data request: TransferRequestMessage#dataDestination should be optional. DataAddress#getType could be made optional.

2. Additional filtering when building the catalog

In the current implementation, the inclusion of a Distribution for an asset in the catalog is solely based on the physical capability of a data plane (determined by its corresponding DataPlaneInstance). That means, if there is a data plane, that can handle a certain format, it will be included in the catalog. In some situations, users may want to restrict the transfer of certain assets to specific data planes, for example due to security concerns. Note that this could cause some assets not to be included in the catalog at all!

To achieve that, the data plane selector is extended with a getCandidates() method, which evaluates dynamically at runtime the set of data planes that can satisfy a data offering.

3. Automatic mapping of the transferType

As stated before, it is necessary to infer the physical DataSink from the transferType. To achieve that, an extensible directory of transferType -> Class<DataSink> is to be provided.

Which Areas Would Be Affected?

Data Plane, Data Plane Selector

Why Is the Feature Desired?

compliance with DSP (transferType as sole arbiter), avoid invalid configurations in data plane registration

Solution Proposal

the following list of subtasks will realize this feature:

• Create dataplane selection improvements decision record with technical details #4148
• Give to the dataplane a way to know which transferTypes can handle #4150
• Add unique UUID to dataplane configuration for idempotent self-registration #4156
• Self Registration of Data Planes #2534
• Merge signaling and control api context #4175
• Move data plane selection endpoints from management to control api #4177
• Control Api for the dataplane to de-register itself at shutdown #4191
• API Security: enable authentication for (internal) REST APIs  #839
• Move API configuration classes into spi #4195
• Heartbeat from data-plane-selector to dataplane #4220
• Put select operation into DataPlaneClientFactory implementation #4203
• Remove data-plane-server launcher #4198
• Make data plane self registration unregister the instance #4238