Description
With GCS FT enabled using external Redis/Valkey, ensure existing workers reconnecting after a head-node repave are properly recognized and reconciled by the new head instead of being marked stale/dead due to an old session ID
Use case
With GCS FT (Fault Tolerance) enabled in Ray 2.55.0, using an external Redis/Valkey, ensure that existing workers remain valid and reusable when the GCS head node is repaved/recreated.
Current Behavior / Problem:
During a head-node repave, the new head may not recognize the existing workers as belonging to the current head/session. When the workers reconnect, they may still present an old session ID associated with the previous head.
The new head may interpret these workers as stale/invalid workers, causing them to be marked as dead. The autoscaler then removes those workers and provisions replacement workers.
This result in:
Unnecessary worker termination and recreation
Increased cluster recovery time after a head-node repave
Loss of healthy worker capacity
Proposed Enhancement
When GCS FT is enabled and the cluster is configured with external Redis/Valkey, the new head should be able to recover and reconcile existing worker session state after a head-node repave.
The worker's previous session ID should not automatically cause the worker to be considered stale merely because the head node has changed.
Instead, the new head should:
Recover the persisted cluster/worker state from external Redis/Valkey.
Identify workers that were previously registered with the cluster.
Accept workers reconnecting with the previous session ID when they can be validated as belonging to the same cluster.
Keep healthy workers in service rather than marking them dead.
Description
With GCS FT enabled using external Redis/Valkey, ensure existing workers reconnecting after a head-node repave are properly recognized and reconciled by the new head instead of being marked stale/dead due to an old session ID
Use case
With GCS FT (Fault Tolerance) enabled in Ray 2.55.0, using an external Redis/Valkey, ensure that existing workers remain valid and reusable when the GCS head node is repaved/recreated.
Current Behavior / Problem:
During a head-node repave, the new head may not recognize the existing workers as belonging to the current head/session. When the workers reconnect, they may still present an old session ID associated with the previous head.
The new head may interpret these workers as stale/invalid workers, causing them to be marked as dead. The autoscaler then removes those workers and provisions replacement workers.
This result in:
Unnecessary worker termination and recreation
Increased cluster recovery time after a head-node repave
Loss of healthy worker capacity
Proposed Enhancement
When GCS FT is enabled and the cluster is configured with external Redis/Valkey, the new head should be able to recover and reconcile existing worker session state after a head-node repave.
The worker's previous session ID should not automatically cause the worker to be considered stale merely because the head node has changed.
Instead, the new head should:
Recover the persisted cluster/worker state from external Redis/Valkey.
Identify workers that were previously registered with the cluster.
Accept workers reconnecting with the previous session ID when they can be validated as belonging to the same cluster.
Keep healthy workers in service rather than marking them dead.