etcd keeps getting killed

[k82cn]

Bug report criteria

• This bug report is not security related, security issues should be disclosed privately via [email protected].
• This is not a support request or question, support requests or questions should be raised in the etcd discussion forums.
• You have read the etcd bug reporting guidelines.
• Existing open issues along with etcd frequently asked questions have been checked and this is not a duplicate.

What happened?

The etcd keep killed, so the other components also failed, e.g. kube-apiserver.

{"level":"info","ts":"2024-02-16T00:19:30.697042Z","caller":"embed/serve.go:250","msg":"serving client traffic securely","traffic":"grpc+http","address":"10.209.224.88:2379"}
{"level":"info","ts":"2024-02-16T00:19:30.697097Z","caller":"embed/serve.go:250","msg":"serving client traffic securely","traffic":"grpc+http","address":"127.0.0.1:2379"}
{"level":"info","ts":"2024-02-16T00:19:35.711928Z","caller":"osutil/interrupt_unix.go:64","msg":"received signal; shutting down","signal":"terminated"}
{"level":"info","ts":"2024-02-16T00:19:35.711985Z","caller":"embed/etcd.go:376","msg":"closing etcd server","name":"swx-forge04","data-dir":"/var/lib/etcd","advertise-peer-urls":["https://10.209.224.88:2380"],"advertise-client-urls":["https://10.209.224.88:2379"]}

What did you expect to happen?

etcd running as expected

How can we reproduce it (as minimally and precisely as possible)?

Run script at: https://github.com/openbce/installer . This script is simple; it prepares environments for kubeadm, and execute it.

Anything else we need to know?

No response

Etcd version (please run commands below)

IMAGE                                     TAG                 IMAGE ID            SIZE
registry.k8s.io/coredns/coredns           v1.11.1             cbb01a7bd410d       18.2MB
registry.k8s.io/etcd                      3.5.10-0            a0eed15eed449       56.6MB
registry.k8s.io/kube-apiserver            v1.29.2             8a9000f98a528       35.1MB
registry.k8s.io/kube-controller-manager   v1.29.2             138fb5a3a2e34       33.4MB
registry.k8s.io/kube-proxy                v1.29.2             9344fce2372f8       28.4MB
registry.k8s.io/kube-scheduler            v1.29.2             6fc5e6b7218c7       18.5MB
registry.k8s.io/pause                     3.8                 4873874c08efc       311kB
registry.k8s.io/pause                     3.9                 e6f1816883972       322kB

Etcd configuration (command line flags or environment variables)

paste your configuration here

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

$ etcdctl member list -w table
# paste output here

$ etcdctl --endpoints=<member list> endpoint status -w table
# paste output here

Relevant log output

No response

[k82cn]

xref: kubernetes/kubeadm#3024

[jmhbnz]

Hey @k82cn - As shown in the log files above, etcd is being sent a SIGTERM signal from an external source.

{"level":"info","ts":"2024-02-16T00:19:35.711928Z","caller":"osutil/interrupt_unix.go:64","msg":"received signal; shutting down","signal":"terminated"}

etcd is designed to shut down when receiving this signal.

You will need to investigate why etcd is being sent this signal on your systems. This could be an issue with your scripts, or your environment / operating system. Please refer to this explanation for how signals work.

[pacoxu]

To investigate why etcd receive the shutdown SIGTERM, you may see kubelet logs first as kubeadm started etcd using the static pod manifest configuration.