Coordinator: Restart Conflation at Runtime

[fluentcrafter]

Description

Coordinator should expose an endpoint to restart confaltion from last finalized block at runtime. This allow opst/prover team to restart conflation during incidents without the need of restarting the coordinator.

Note

What happens if the Tracer plugin is updated and we need to update the configs as well? Just the RPC call won't be good enough.

Motivation

Faster/better incident reation

Tasks

• Implement JSON-RPC endpoint to restart conflation process on coordinator side

Acceptance criteria

• List tests that need to be implemented or checks to consider this task completed.

Risks

• These are risks that may prevent completion of the task as well as areas to pay special attention to

Remember to

• Add the documentation label in case there is an impact on the documentation
• Add priority and team labels
• Add Task for updating the Runbook or adding/updating existing metrics and alerts.

[fluentcrafter]

@FlorianHuc @AlexandreBelling, about the endpoint we discussed to restart conflation at runtime in production without restarting the coordinator, can you please remind me what scenarios/cases this would be used for? Note: I am not referring to the back testing endpoint.

I see 3 potential component bugs that would require the coordinator to restart the conflation:

• prover fix - we can just move the failed request to the {execution/compression/aggregation}/requests folder
• tracer fix - requires updating the coordinator config expected-traces-api-version = "beta-v4.0-rcXX", which requires a coordinator restart.
• shomei fix - requires updating the coordinator config state-manager.version = "2.3.0", which requires a coordinator restart.

So, just an endpoint like coordinator_restartConflation() won't fit the purpose. Maybe I misunderstood the case. Could you please elaborate a bit more on the case where you felt this need during production incidents?