Graceful Shutdown

[sfmskywalker]

Currently, when an ASP.NET Core application running Elsa Workflows shuts down, it may terminate workflows that are still executing. This can lead to inconsistencies, especially when workflows interact with external systems, databases, or queues.

This feature request proposes a graceful shutdown mechanism that allows active workflows to complete up to a commit point before the application stops while ensuring that no new workflows are started during the shutdown phase.

Use Case & Requirements

1. Ongoing Workflows Complete Before Shutdown

   • Elsa should detect when the application is shutting down.
   • Workflows that are currently executing should be allowed to finish their current activity but not proceed beyond a commit point.

2. New Workloads Should Not Be Accepted

   • MassTransit consumers should stop accepting new messages while allowing in-flight messages to complete.
   • Background jobs (e.g., Quartz.NET, Hangfire) should pause scheduling new work.
   • HTTP API endpoints for starting new workflows should return a service unavailable response or queue execution until restart.

3. Extensibility for 3rd-Party Modules

   • Introduce a global shutdown notification event that extensions can hook into.
   • Provide an interface like IShutdownHandler that modules can implement to clean up background tasks and other resources.

Proposed Implementation (by chatGPT)

• Introduce a shutdown token or state flag that Elsa’s execution engine checks before starting new activities.
• Implement a shutdown event dispatcher that:
  • Stops MassTransit consumers.
  • Pauses job schedulers.
  • Notifies registered IShutdownHandler implementations.
• Ensure the WorkflowRunner (or equivalent execution engine) respects shutdown signals.

Impact

This would:

• Improve reliability when shutting down applications running Elsa Workflows.
• Ensure workflows reach a natural stopping point before termination.
• Provide a consistent and extensible mechanism for module developers.

[CODEdire]

A couple other thoughts here:

• ASP.NET HealthChecks - Would be good to use this to also return an unhealthy state so load balancers can key off this and prevent API calls to the server shutting down
• ProtoActor - Passing around a state or ensuring that the node leaves the cluster when shutting down, so it does not try to handle actions or events around workflows. This also means passing on the cluster lead in the case that the current node is the elected lead. (Assuming there is a leader in the cluster).