etcd member removal fails in 3 node cluster

[ghohman-msft]

Environmental Info:
K3s Version:

k3s 1.33.7

Node(s) CPU architecture, OS, and Version:

n/a

Cluster Configuration:

3 etcd+cp nodes

Describe the bug:

When removing nodes from the cluster, after removing 2 nodes the cluster becomes unhealthy. ETCD shows issues with leader election in the logs.

this does not happen consistently, in about ~3/10 tests i have done i have been able to reproduce the issue. Log collection is somewhat difficult since this is done via automated test framework, but the end of the k3s logs show etcd stuck in a loop where leader election keeps failing.

Steps To Reproduce:

• Installed K3s
• set up 3 node cluster
• remove 2 nodes

Expected behavior:

k3s removes member node from etcd when node is removed from cluster.

Actual behavior:

After removal of 2 nodes, the cluster becomes unresponsive. k3s logs shows that ETCD gets stuck in a loop trying to elect members:

k3s[28091]: time="2026-01-23T23:51:40Z" level=info msg="Failed to test etcd connection: failed to get etcd status: rpc error: code = Unavailable desc = connection error: desc = \"transport: authentication handshake failed: context deadline exceeded\""
k3s[28091]: {"level":"info","ts":"2026-01-23T23:51:40.971015Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8adc34f2f139fc91 is starting a new election at term 5"}
k3s[28091]: {"level":"info","ts":"2026-01-23T23:51:40.971355Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8adc34f2f139fc91 became pre-candidate at term 5"}
k3s[28091]: {"level":"info","ts":"2026-01-23T23:51:40.971370Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8adc34f2f139fc91 received MsgPreVoteResp from 8adc34f2f139fc91 at term 5"}
k3s[28091]: {"level":"info","ts":"2026-01-23T23:51:40.971382Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8adc34f2f139fc91 [logterm: 5, index: 15800] sent MsgPreVote request to d082830930c9dd1c at term 5"}
k3s[28091]: {"level":"warn","ts":"2026-01-23T23:51:42.219939Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d082830930c9dd1c","rtt":"0s","error":"dial tcp 192.168.0.52:2380: connect: no route to host"}
k3s[28091]: {"level":"warn","ts":"2026-01-23T23:51:42.220105Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d082830930c9dd1c","rtt":"0s","error":"dial tcp 192.168.0.52:2380: connect: no route to host"}
k3s[28091]: time="2026-01-23T23:51:45Z" level=error msg="Failed to check local etcd status for learner management: rpc error: code = Unavailable desc = connection error: desc = \"transport: authentication handshake failed: context deadline exceeded\""
k3s[28091]: time="2026-01-23T23:51:45Z" level=info msg="Failed to test etcd connection: failed to get etcd status: rpc error: code = Unavailable desc = connection error: desc = \"transport: authentication handshake failed: context deadline exceeded\""
k3s[28091]: {"level":"info","ts":"2026-01-23T23:51:46.471187Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8adc34f2f139fc91 is starting a new election at term 5"}
k3s[28091]: {"level":"info","ts":"2026-01-23T23:51:46.471443Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8adc34f2f139fc91 became pre-candidate at term 5"}
k3s[28091]: {"level":"info","ts":"2026-01-23T23:51:46.471471Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8adc34f2f139fc91 received MsgPreVoteResp from 8adc34f2f139fc91 at term 5"}
k3s[28091]: {"level":"info","ts":"2026-01-23T23:51:46.471489Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8adc34f2f139fc91 [logterm: 5, index: 15800] sent MsgPreVote request to d082830930c9dd1c at term 5"}
k3s[28091]: {"level":"warn","ts":"2026-01-23T23:51:47.221072Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d082830930c9dd1c","rtt":"0s","error":"dial tcp 192.168.0.52:2380: connect: no route to host"}
k3s[28091]: {"level":"warn","ts":"2026-01-23T23:51:47.221226Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d082830930c9dd1c","rtt":"0s","error":"dial tcp 192.168.0.52:2380: connect: no route to host"}
k3s[28091]: time="2026-01-23T23:51:50Z" level=info msg="Failed to test etcd connection: failed to get etcd status: rpc error: code = Unavailable desc = connection error: desc = \"transport: authentication handshake failed: context deadline exceeded\""

Additional context / logs:

Since it seems there is only one "ghost" member being sent a vote request, I assume that one of the nodes was removed from etcd successfully as a member but the other remained. There was an issue on this previously: #12908, so I am wondering if it is possible there is an issue with the local k3s setup and if there are any better steps I could take to diagnose the issue?

[brandond]

Please document your process. Are you waiting until each member is successfully removed, and a new leader has been elected, before removing the next? Are you taking advantage of the member removal annotation? Are you stopping the node before or after removing it?

It is definitely possible to break etcd if you remove members unsafely.

[brandond]

Also note that 1.33 is still on etcd 3.5, I would recommend going to a newer minor to use etcd 3.6.

[ghohman-msft]

@brandond sorry for the late reply - I did some further testing on this. I did some further testing with the following flow for node removal:
Annotate node with etcd.k3s.cattle.io/remove=true -> wait 10 seconds -> call kubectl delete node <node name> --wait="false" -> remove linux VM hosting the k3s node -> wait for 60 seconds before moving to next node to allow time for etcd leader to rotate. With this flow I was able to get the tests to pass but I noticed that calling kubectl get nodes would show nodes that were previously removed. In the test setup, I only can access the logs from the last node but I see this entry:

m1 k3s[891]: time="2026-01-27T03:03:42Z" level=info msg="Removing etcd member from cluster due to remove annotation" address=192.168.0.52 name=m3-70b1f566
m1 k3s[891]: time="2026-01-27T03:03:42Z" level=info msg="etcd emember removed successfully" address=192.168.0.52 name=m3-70b1f566

Since this is a three-node setup i would expect to see a similar entry for m2 as well... I am working on getting a test environment set up so I can debug this manually/get more clarity on what is happening within the cluster, but I wanted to check as well if there was any official guidance/documentation for safe control plane node removal? My current thinking is that it is most likely that the issue lies in how we handle node cleanup in multi-node deployments.

[brandond]

You should also be able to just delete the node resource and then terminate the instance once the node is gone. The annotation just gets it out of the etcd cluster extra early.

Recent releases will block node deletion (by not removing the finalizer), or clear the annotation, if etcd rejects the member removal for whatever reason.

[ghohman-msft]

I've been able to get a 2 cluster setup locally to reproduce what I last tested (so using the annotation before removal). What I noticed is that during the node's removal, it goes into an unhealthy state with the following output in the k3s logs:

"msg":"server error","error":"the member has been permanently removed from the cluster"
"msg":"data-dir used by this member must be removed"

And halting the removal at this point leaves the cluster in the following state:

NAME                   STATUS                        ROLES                       AGE   VERSION
node2                   NotReady,SchedulingDisabled   control-plane,etcd,master   53m   v1.33.7+k3s-
node1                   Ready                         control-plane,etcd,master   73m   v1.33.7+k3s-