Intrinsic operations in the Connector SDK

[chillleader]

Is your feature request related to a problem? Please describe

The Connector SDK supports document handling capabilities since 8.7. However, the SDK is still missing a convenient way to apply transformations to documents in runtime without creating too much operational overhead.

Example use-case: The GitHub connector can support uploading files to GitHub via the API. GitHub API accepts Base64-encoded files.

Since Camunda now supports documents, I would like to be able to upload a Camunda document to GitHub, but for that I need to provide it in the Base64 format. I would like to do this in a low-code way that fits well into existing connectors machinery, and I would like to avoid creating a separate connector invocation to transform the document into Base64, as this will be overly complicated and will negate the benefits of seamless document mappings in the Connector SDK.

I therefore would like to have an option to add manipulations to the connector input variables that are executed in runtime, after the variable values are resolved and all document references are transformed to documents, but before the values are passed to the connector function.

Describe the solution you'd like

The concept of an intrinsic operation can be implemented to instruct the connector runtime to perform a manipulation on values before they are passed to the connector function.

An intrinsic operation is a JSON object that can be provided as a connector input, for example:

{
  "camunda.operation.name": "base64",
  "inputs": [
    {"camunda.document.type": "camunda", "documentId": "...", "storeId": "s3", "metadata": {}} 
  ]  
}

If the connector runtime receives this object as part of the job input, it will recognize the operation based on the presence of the marker field (camunda.operation.name) and execute the operation before the connector function object is invoked.

The connector can then bind the properties using connectorContext.bindVariables(InputType.class) - the result of the operation execution will be of type String for base64 content, but can be different for other operations.

This solution will also allow to retrofit existing connectors with document support without the need to modify the connector code - as long as an intrinsic operation transforms the document to a format that can be consumed by a legacy connector.

Further iterations may include FEEL functions to simplify the creation of operation objects for the most frequently used operations - for example, the example above can be simplified to to base64(myDocument), where to base64 is a custom FEEL function, and myDocument is a variable that contains the document reference.

Describe alternatives you've considered

• Ad-hoc connectors can be implemented for the given transformations, but this would not be a low-code solution and would require significant effort to implement every new kind of transformation.

• Since this solution mainly addresses documents, we have initially considered including the operation into the document reference structure. However, with a separate concept of an operation, we will have more benefits such as: 1) an ability to pass multiple inputs into the operation, e.g. merge two documents into one; 2) room for future extensions of the operation concept and potentially applying it to other values (not only documents). The main downside of the suggested solution where operations are first-class citizens rather than part of document references is the need to introduce a new abstraction and overall higher maintenance effort.

Additional context (🔒 limited access)

• This feature request evolved and branched off from the Connector SDK document support epic
• Initial design documentation

[chillleader]

Another use-case: https://github.com/camunda/team-connectors/issues/1004