Pod receives API calls after SIGTERM

[klaushofrichter]

Hello,
I have a pod in a K3D cluster that receives SIGTERM because of a rolling update. The pod does not receive any more calls to /readyz or /livez after SIGTERM, but the LB seems to route API calls to the pod for about 3 to 4 seconds after SIGTERM. Is that expected? Is there some deterministic behavior? I appreciate any advice, thanks in advance!

• Klaus

[klaushofrichter]

Answering myself: I found this article clarifying the pod shutdown sequence and saying "The pod should largely ignore [SIGTERM], keep running, and keep serving new connections." - that somehow seems not to support "graceful" handling. Is there any best practice not involving preStop hooks? In my case, it's a NodeJS app, and there is full control over the application.

[iwilltry42]

Hi @klaushofrichter , this is general Kubernetes behavior and unrelated to K3d or k3d.
You can learn more about the pod lifecycle here:

• general lifecycle: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/
• terminating with grace: https://cloud.google.com/blog/products/containers-kubernetes/kubernetes-best-practices-terminating-with-grace
• https://medium.com/brexeng/kubernetes-rolling-update-and-termination-grace-periods-d922c6b84d88

[klaushofrichter]

Thanks @iwilltry42, these are good references. In hindsight, the Kubernetes behavior makes sense, but in my initial implementation, I was returning 500 on the API directly after SIGTERM, resulting in 500 returns coming through to the client for a few seconds. That's not great. I am now doing a two-stage shutdown in the app, delaying the 500 returns for a bit, but this seems to rely on a certain performance of the LB to not use the terminating pods anymore.

[iwilltry42]

Are you using LivenessProbes/ReadinessProbes in your deployment?
You may want to look into that in general 👍

[klaushofrichter]

Hello,
yes, /readyz and /livez are used. I don't see any more calls to the health APIs after SIGTERM but that's OK, Kubernetes should not need the probes after SIGTERM as the pod is going down anyway. The issue was that I expected no more calls from the LB to the worker APIs after SIGTERM, but that was a wrong assumption. The point now is to decide when it is safe to stop responding to API calls after SIGTERM. A "two-stage" approach works well for me: starting a timer waiting for API calls to end based on time estimation, and then doing the pods internal graceful shutdown while returning 500 on the worker APIs (which should not receive calls anymore), hopefully calling exit() before SIGKILL.

• Klaus