Proper procedures for recovering EKS-A cluster from broken state.

[SunghoHong-gif]

I have a question about recovering EKS-A provisioned clusters from a broken state.

Suppose a cluster has failed machines in both the control plane and worker nodes, and these machines are assumed to be unrecoverable (=physically broken and have to add new baremetal machines). How should this be handled when we want to execute cluster upgrade?

example@example-admin:~$ kubectl get nodes
NAME                STATUS                        ROLES           AGE    VERSION
example-cp3-26       Ready                         control-plane   191d   v1.28.15
example-cp3-27       NotReady,SchedulingDisabled   control-plane   191d   v1.29.13
example-cp5-26       Ready                         control-plane   191d   v1.28.15
example-gpu-wk3-9    NotReady,SchedulingDisabled   <none>          191d   v1.29.13
example-gpu-wk5-11   Ready                         <none>          191d   v1.28.15

Are we expected to manually add healthy control plane and worker nodes to proceed with the cluster upgrade?
Or are we expected to re-provision the cluster from scratch and execute backups?

I’m trying to understand the intended recovery path when the cluster is in an unstable state and cannot be restored using the originally provisioned machines.

[sp1999]

Hey @SunghoHong-gif, thank you for raising this important question about EKS-A cluster recovery.

Looking at your cluster state, you have multiple nodes in NotReady and SchedulingDisabled status:

• example-cp3-27 (control-plane)
• example-gpu-wk3-9 (worker node)

For clusters with physically broken/unrecoverable machines in both control plane and worker nodes, there are two potential approaches:

1. Manual Recovery Path:

• Add new healthy control plane and worker nodes to replace the failed ones
• Remove the unhealthy nodes from the cluster
• Proceed with the cluster upgrade once the cluster reaches a stable state

2. Full Re-provisioning Path:

• If the cluster is in a severely unstable state
• Re-provision the cluster from scratch using backup/restore procedures

Before proceeding with either approach, it's critical to:

1. Perform a full cluster backup to preserve the latest stable state
2. Ensure you have sufficient healthy nodes to maintain quorum
3. Validate cluster health after adding new nodes

The specific approach would depend on:

• The extent of node failures
• Whether you can maintain control plane quorum
• Your cluster's backup status