docs: Add first draft of a Data Plane Signaling specification.

docs/developer/data-plane-signaling/data-plane-signaling-state-machine.puml

@@ -0,0 +1,64 @@
+'
+' Copyright (c) 2024 Fraunhofer Institute for Software and Systems Engineering ISST
+'
+' This program and the accompanying materials are made available under the
+' terms of the Apache License, Version 2.0 which is available at
+' https://www.apache.org/licenses/LICENSE-2.0
+'
+' SPDX-License-Identifier: Apache-2.0
+'
+' Contributors:
+' Florian Zimmer - Initial Version
+'
+
+@startuml
+!define DscClr(x) <color:#7b839a>x</color>
+
+skinparam WrapWidth 270
+
+state "STARTED" as started_push #e6ebfa;line:blue
+started_push: DscClr(//trying to complete transfer process//)
+
+state "STARTED" as started_pull #e6ebfa;line:blue
+started_pull: DscClr(DataFlowStartMessage<PULL>)
+
+state "RECEIVED" as received #e6ebfa;line:blue
+received : DscClr(DataFlowStartMessage<PUSH>)
+
+state "COMPLETED" as completed #e6ebfa;line:blue
+completed: DscClr(//trying to send TransferProcessCompleteRequest to callback address (CP)//)
+
+state "NOTIFIED" as notified #e6ebfa;line:blue
+notified : DscClr(TransferProcessFailRequest //or// TransferProcessCompleteRequest)
+
+state "FAILED" as failed #e6ebfa;line:blue
+failed : DscClr(//trying to send TransferProcessFailRequest to callback address (CP)//)
+
+state "SUSPENDED / TERMINATED" as suspended #f5cccd;line:red
+suspended : DscClr(DataFlowSuspendMessage //or// DataFlowTerminateMessage)
+
+[*] --> received : CP
+received -[dashed]-> started_push
+
+started_push -[dashed]-> failed
+started_push -[dashed]-> completed
+
+failed -[dashed]-> failed
+failed --> notified : DP
+
+completed -[dashed]-> completed
+completed --> notified : DP
+
+notified --> [*]
+
+[*] --> started_pull : CP
+started_pull --> [*]
+
+received --> suspended : CP
+started_push --> suspended : CP
+started_pull --> suspended : CP
+failed --> suspended : CP
+completed --> suspended : CP
+notified --> suspended : CP
+suspended --> [*]
+@enduml

docs/developer/data-plane-signaling/data-plane-signaling.md

@@ -39,25 +39,112 @@ The data plane access token may be renewable based on the capabilities of its as
 
 ### The Signaling Protocol
 
-All requests must support idempotent behavior. Data planes must therefore perform request de-duplication. After a data plane commits a request, it will return an ack to the control plane, which will transition the `TransferProcess` to its next state (e.g., `STARTED`, `SUSPENDED`, `TERMINATED`). If a successful ack is not received, the control plane will resend the request during a subsequent tick period.
+All requests must support idempotent behavior. Data planes must therefore perform request de-duplication. After a data plane commits a request, it will return an ack to the control plane, which will transition the `TransferProcess` to its next state. If a successful ack is not received, the control plane will resend the request during a subsequent tick period.
 
-##### 1. `START` 
+### States
 
-During the transfer process `STARTING` phase, a data plane will be selected by the default push and pull `DataFlowControllers`, which will then send a `DataFlowStartMessage` (renamed from `DataFlowRequest`) to the data plane. 
+The Data Plane Signaling protocol is represented by the following states:
 
-The control plane (`DataFlowController`) will record which data plane was selected for the transfer process so that it can properly route subsequent, start, stop, and terminate requests.
+- **RECEIVED:** The Data Plane received a [DataFlowStartMessage](#DataFlowStartMessage) with a `FlowType` of `PUSH`. It acknowledges receiving the message by responding with a [DataFlowResponseMessage](#DataFlowResponseMessage).
+- **STARTED:** In case of a `PULL` transfer, the EDR token was issued and provided to the Control Plane successfully using a [DataFlowResponseMessage](#DataFlowResponseMessage). In case of a `PUSH` transfer, the transfer process was initiated successfully and acknowledged by sending an empty [DataFlowResponseMessage](#DataFlowResponseMessage).
+- **COMPLETED:** A `PUSH` transfer was completed successfully. The Data Plane tries to notify the Control Plane about the outcome.
+- **FAILED:** The Data Plane failed to complete the transfer process and continuously tries to notify the Control Plane about the issue.
+- **NOTIFIED:** The Data Plane successfully notified the Control Plane about the outcome of the transfer, either being a success or failure.
+- **SUSPENDED:** The Control Plane sent a [DataFlowSuspendMessage](#DataFlowSuspendMessage) to the Data Plane, thus suspending the transfer process.
+- **TERMINATED:** The Control Plane sent a [DataFlowTerminateMessage](#DataFlowTerminateMessage) to the Data Plane, thus terminating the transfer process.
 
-For client pull transfers, the data plane will return a `DataAddress` and an access token. 
+##### State Machine
 
-If the data flow was previously `SUSPENDED`, the data plane may elect to return the same `DataAddress` or create a new one.
+The Data Plane Signaling state machine can be seen in the following diagram:
 
-##### 2. `SUSPEND` 
+![](data-plane-signaling-state-machine.png)
 
-During the transfer process `SUSPENDING` phase, the `DataFlowController` will send a `DataFlowSuspendMessage` to the data plane. The data plane will transition the data flow to the `SUSPENDED` state and invalidate the associated access token.
+### Message Types
 
-##### 3. `TERMINATE`
+The Data Plane Signaling supports the following message types, in order to allow for the Control Plane to fulfill a `TransferProcess` using a suitable Data Plane. A Data Plane gets selected based on their capabilities by a Data Plane Framework Selector. Furthermore, the Control Plane (`DataFlowController`) will record which Data Plane was selected for the transfer process so that it can properly route subsequent, start, stop, and terminate requests.
 
-During the transfer process `TERMINATING` phase, the `DataFlowController` will send a `DataFlowTerminateMessage` to the data plane. The data plane will transition the data flow to the `TERMINATED` state and invalidate the associated access token.
+#### DataFlowStartMessage
+
+| Message Type | Start |
+|---------------------|--------------------------------------------------------------------------|
+| **Sent by** | Control Plane |
+| **Resulting state** | `RECEIVED` or `STARTED` dependent on the `FlowType` (Push or Pull) |
+| **Response** | [DataFlowResponseMessage](#DataFlowResponseMessage) |
+| **Schema** | JSON Schema |
+| **Example** | [DataFlowStartMessage](message-types/examples/DataFlowStartMessage.json) |
+| **Diagram(s)** | ![](message-types/diagrams/dataFlowStartMessage.png) |
+
+Initiates the data flow by signaling the Data Plane to start the transfer process. Dependent on the `FlowType` the following things will happen:
+- `PUSH`: Data Plane acknowledges the request using a [DataFlowResponseMessage](#DataFlowResponseMessage) and tries to start the transfer process. Will notify Control Plane about the respective outcome after either success or failure in a later state.
+- `PULL`: Data Plane returns callback address to pull the data from and an access token using a [DataFlowResponseMessage](#DataFlowResponseMessage).
+
+Furthermore, if the data flow was previously `SUSPENDED`, the data plane may elect to return the prior used `DataAddress` or create a new one.
+
+#### DataFlowSuspendMessage
+
+| Message Type | Suspend |
+|---------------------|------------------------------------------------------------------------------|
+| **Sent by** | Control Plane |
+| **Resulting state** | `SUSPENDED` |
+| **Response** | - |
+| **Schema** | JSON Schema |
+| **Example** | [DataFlowSuspendMessage](message-types/examples/DataFlowSuspendMessage.json) |
+| **Diagram(s)** | ![](message-types/diagrams/dataFlowSuspendMessage.png) |
+
+Leads to the suspension of the data flow, thus stopping it and invalidating the associated access token.
+
+#### DataFlowTerminateMessage
+
+| Message Type | Terminate |
+|---------------------|----------------------------------------------------------------------------------|
+| **Sent by** | Control Plane |
+| **Resulting state** | `TERMINATED` |
+| **Response** | - |
+| **Schema** | JSON Schema |
+| **Example** | [DataFlowTerminateMessage](message-types/examples/DataFlowTerminateMessage.json) |
+| **Diagram(s)** | ![](message-types/diagrams/dataFlowTerminateMessage.png) |
+
+Leads to the termination of the data flow, thus stopping it and invalidating the associated access token.
+
+### Response Types
+
+The response messages are either sent as a direct response to an incoming message or as an indirect response at a later stage during the transfer process.
+
+#### DataFlowResponseMessage
+
+| Message Type | Response |
+|---------------------|--------------------------------------------------------------------------------|
+| **Sent by** | Data Plane |
+| **Resulting state** | - |
+| **Schema** | JSON Schema |
+| **Example** | [DataFlowResponseMessage](message-types/examples/DataFlowResponseMessage.json) |
+| **Diagram(s)** | ![](message-types/diagrams/dataFlowResponseMessage.png) |
+
+This message is a direct response message to a [DataFlowStartMessage](#DataFlowStartMessage) and acknowledges receiving it. Will be used to provide access token and callback address in case of a `PULL` transfer.
+
+#### TransferProcessCompleteRequest
+
+| Message Type | Notification |
+|---------------------|--------------------------|
+| **Sent by** | Data Plane |
+| **Resulting state** | `NOTIFIED` |
+| **Schema** | *None due to empty body* |
+| **Example** | - |
+| **Diagram(s)** | - |
+
+This message is not a direct response to an incoming message. It rather gets sent by the Data Plane if the already started transfer process was successfully completed. Successfully sending this message leads to the transfer state machine to proceed to the state `NOTIFIED` in case of a `PUSH` transfer.
+
+#### TransferProcessFailRequest
+
+| Message Type | Notification |
+|---------------------|--------------------------------------------------------------------------------------|
+| **Sent by** | Data Plane |
+| **Resulting state** | `NOTIFIED` |
+| **Schema** | JSON Schema |
+| **Example** | [TransferProcessFailRequest](message-types/examples/TransferProcessFailRequest.json) |
+| **Diagram(s)** | ![](message-types/diagrams/transferProcessFailRequest.png) |
+
+This message is not a direct response to an incoming message. It rather gets sent by the Data Plane if the already started transfer process encountered an error, thus failed to be fulfilled. Successfully sending this message leads to the transfer state machine to proceed to the state `NOTIFIED` in case of a `PUSH` transfer.
 
 ## II. Control Plane Refactoring
 

docs/developer/data-plane-signaling/message-types/diagrams/dataFlowResponseMessage.puml

@@ -0,0 +1,25 @@
+'
+' Copyright (c) 2024 Fraunhofer Institute for Software and Systems Engineering ISST
+'
+' This program and the accompanying materials are made available under the
+' terms of the Apache License, Version 2.0 which is available at
+' https://www.apache.org/licenses/LICENSE-2.0
+'
+' SPDX-License-Identifier: Apache-2.0
+'
+' Contributors:
+' Florian Zimmer - Initial Version
+'
+
+@startuml
+class DataFlowReponseMessage #e4ebfa ##75acfa {
+ @type: "DataFlowResponseMessage"
+}
+
+class DataAddress #e4ebfa ##75acfa {
+ @type: "DataAddress"
+ properties: Map<String, Object>
+}
+
+DataFlowReponseMessage "1" *-[#747c94]- "1" DataAddress : dataAddress
+@enduml

docs/developer/data-plane-signaling/message-types/diagrams/dataFlowStartMessage.puml

@@ -0,0 +1,39 @@
+'
+' Copyright (c) 2024 Fraunhofer Institute for Software and Systems Engineering ISST
+'
+' This program and the accompanying materials are made available under the
+' terms of the Apache License, Version 2.0 which is available at
+' https://www.apache.org/licenses/LICENSE-2.0
+'
+' SPDX-License-Identifier: Apache-2.0
+'
+' Contributors:
+' Florian Zimmer - Initial Version
+'
+
+@startuml
+class DataFlowStartMessage #e4ebfa ##75acfa {
+ @type: "DataFlowStartMessage"
+ id: String
+ processId: String
+ assetId: String
+ participantId: String
+ agreementId: String
+ callbackAddress: URI
+ properties: Map<String, String>
+ traceContext: Map<String, String>
+}
+
+class DataAddress #e4ebfa ##75acfa {
+ @type: "DataAddress"
+ properties: Map<String, Object>
+}
+
+enum FlowType #e4ebfa ##75acfa {
+ PUSH
+ PULL
+}
+
+DataFlowStartMessage "1" *-[#747c94]- "1" FlowType : flowType
+DataFlowStartMessage "1" *-[#747c94]- "2" DataAddress : sourceDataAddress\n destinationDataAddress
+@enduml

docs/developer/data-plane-signaling/message-types/diagrams/dataFlowSuspendMessage.puml

@@ -0,0 +1,19 @@
+'
+' Copyright (c) 2024 Fraunhofer Institute for Software and Systems Engineering ISST
+'
+' This program and the accompanying materials are made available under the
+' terms of the Apache License, Version 2.0 which is available at
+' https://www.apache.org/licenses/LICENSE-2.0
+'
+' SPDX-License-Identifier: Apache-2.0
+'
+' Contributors:
+' Florian Zimmer - Initial Version
+'
+
+@startuml
+class DataFlowSuspendMessage #e4ebfa ##75acfa {
+ @type: "DataFlowSuspendMessage"
+ reason: String
+}
+@enduml

docs/developer/data-plane-signaling/message-types/diagrams/dataFlowTerminateMessage.puml

@@ -0,0 +1,19 @@
+'
+' Copyright (c) 2024 Fraunhofer Institute for Software and Systems Engineering ISST
+'
+' This program and the accompanying materials are made available under the
+' terms of the Apache License, Version 2.0 which is available at
+' https://www.apache.org/licenses/LICENSE-2.0
+'
+' SPDX-License-Identifier: Apache-2.0
+'
+' Contributors:
+' Florian Zimmer - Initial Version
+'
+
+@startuml
+class DataFlowTerminateMessage #e4ebfa ##75acfa {
+ @type: "DataFlowTerminateMessage"
+ reason: String
+}
+@enduml

docs/developer/data-plane-signaling/message-types/diagrams/transferProcessFailRequest.puml

@@ -0,0 +1,18 @@
+'
+' Copyright (c) 2024 Fraunhofer Institute for Software and Systems Engineering ISST
+'
+' This program and the accompanying materials are made available under the
+' terms of the Apache License, Version 2.0 which is available at
+' https://www.apache.org/licenses/LICENSE-2.0
+'
+' SPDX-License-Identifier: Apache-2.0
+'
+' Contributors:
+' Florian Zimmer - Initial Version
+'
+
+@startuml
+class TransferProcessFailRequest #e4ebfa ##75acfa {
+ errorMessage: String
+}
+@enduml

docs/developer/data-plane-signaling/message-types/examples/DataFlowResponseMessage.json

@@ -0,0 +1,14 @@
+{
+ "@type": "https://w3id.org/edc/v0.0.1/ns/DataFlowResponseMessage",
+ "https://w3id.org/edc/v0.0.1/ns/dataAddress": {
+ "@type": "https://w3id.org/dspace/v0.8/DataAddress",
+ "https://w3id.org/dspace/v0.8/endpointType": "endpointType",
+ "https://w3id.org/dspace/v0.8/endpointProperties": [
+ {
+ "@type": "https://w3id.org/dspace/v0.8/EndpointProperty",
+ "https://w3id.org/dspace/v0.8/name": "name",
+ "https://w3id.org/dspace/v0.8/value": "value"
+ }
+ ]
+ }
+}

docs/developer/data-plane-signaling/message-types/examples/DataFlowStartMessage.json

@@ -0,0 +1,32 @@
+{
+ "@type": "https://w3id.org/edc/v0.0.1/ns/DataFlowStartMessage",
+ "https://w3id.org/edc/v0.0.1/ns/flowType": "PUSH",
+ "https://w3id.org/edc/v0.0.1/ns/agreementId": "agreementId",
+ "https://w3id.org/edc/v0.0.1/ns/processId": "processId",
+ "https://w3id.org/edc/v0.0.1/ns/datasetId": "datasetId",
+ "https://w3id.org/edc/v0.0.1/ns/properties": {},
+ "https://w3id.org/edc/v0.0.1/ns/callbackAddress": "http://localhost",
+ "https://w3id.org/edc/v0.0.1/ns/sourceDataAddress": {
+ "@type": "https://w3id.org/dspace/v0.8/DataAddress",
+ "https://w3id.org/dspace/v0.8/endpointType": "sourceType",
+ "https://w3id.org/dspace/v0.8/endpointProperties": [
+ {
+ "@type": "https://w3id.org/dspace/v0.8/EndpointProperty",
+ "https://w3id.org/dspace/v0.8/name": "name",
+ "https://w3id.org/dspace/v0.8/value": "value"
+ }
+ ]
+ },
+ "https://w3id.org/edc/v0.0.1/ns/participantId": "participantId",
+ "https://w3id.org/edc/v0.0.1/ns/destinationDataAddress": {
+ "@type": "https://w3id.org/dspace/v0.8/DataAddress",
+ "https://w3id.org/dspace/v0.8/endpointType": "endpointType",
+ "https://w3id.org/dspace/v0.8/endpointProperties": [
+ {
+ "@type": "https://w3id.org/dspace/v0.8/EndpointProperty",
+ "https://w3id.org/dspace/v0.8/name": "name",
+ "https://w3id.org/dspace/v0.8/value": "value"
+ }
+ ]
+ }
+}

docs/developer/data-plane-signaling/message-types/examples/DataFlowSuspendMessage.json

@@ -0,0 +1,4 @@
+{
+ "@type": "https://w3id.org/edc/v0.0.1/ns/DataFlowSuspendMessage",
+ "https://w3id.org/edc/v0.0.1/ns/reason": "reason"
+}

docs/developer/data-plane-signaling/message-types/examples/DataFlowTerminateMessage.json

@@ -0,0 +1,4 @@
+{
+ "@type": "https://w3id.org/edc/v0.0.1/ns/DataFlowTerminateMessage",
+ "https://w3id.org/edc/v0.0.1/ns/reason": "reason"
+}

docs/developer/data-plane-signaling/message-types/examples/TransferProcessFailRequest.json

@@ -0,0 +1,3 @@
+{
+ "errorMessage": "errorMessage"
+}