feat: extend fail job with incident key

[skayliu]

Description

As this issue feature request descripted Proposal: Extend FailJobResponse and ThrowErrorResponse with incidentKey #16580 ,the zeebe engine should response a fail job with incident key.

Related issues

part of #16580

[skayliu]

Hey, @korthout Please take your time to check this feature when you are free.

[korthout]

Hi @skayliu 🙇 As always, thank you for contributing! I haven't reviewed the code because I'm not sure whether we want to proceed with the proposed solution.

Specifically, I expect that querying of incidents will become available through the upcoming unified Camunda 8 API, which will enable new ways to respond to incidents:

• [Root Epic] Architecture Streamlining step 5: One Camunda 8 API issues#715

I worry that the current approach would also be needed for all other RPCs (as incidents can be raised at any point in process execution). I prefer looking at alternative solutions.

@aleksander-dytko, Could you share your insights about the need to solve this problem in relation to upcoming changes in 8.6? Do we expect the incident API to be available in 8.6 already?

[aleksander-dytko]

Hi @korthout, thanks for mentioning this. With 8.6 we plan to have a unified Camunda 8 API with POST endpoint to search for incidents. It will be basically the same as we currently have with Operate API - Incidents. Users would still need to use 2 endpoints and look for the incidentKey to resolve incident.

I would propose to accept this contribution to simplify incident handling.

I worry that the current approach would also be needed for all other RPCs (as incidents can be raised at any point in process execution). I prefer looking at alternative solutions.

Could you give an example here @korthout ?

@skayliu thank you for the contribution 🙇

[korthout]

I worry that the current approach would also be needed for all other RPCs (as incidents can be raised at any point in process execution). I prefer looking at alternative solutions.

Could you give an example here @korthout ?

Most RPCs can result in one or several incidents on a process instance:

• Create a process instance
• Complete a job
• Publish a message
• Broadcast a signal
• Modify/Migrate a process instance
• etc

The same is true for one of our REST end points, but this will likely grow over time:

• Complete a user task

There can be many reasons why any of these would raise an incident:

• missing variables
• mistakes in input/output mappings, or any other FEEL expression
• unexpected conditions on sequence flows when leaving a gateway
• unknown process referenced by call activity
• etc

Clearly, users need some way to respond to incidents. I'm challenging that we should have this in our API as it will leak to all endpoints. Instead, I'd argue that we need a single specific way to respond to incidents:

• a query API to search for incidents;
• or, an API to register listeners for incidents, that notifies the listeners when new incidents are raised