Pods not always creating logs

[DaniloGT]

Environmental Info:
K3s Version: v1.28.8+k3s1 (653dd61) | go version go1.21.8

Node(s) CPU architecture, OS, and Version: Linux orangepizero3 6.1.31-sun50iw9 #1.0.2 SMP Thu Sep 28 14:15:40 CST 2023 aarch64 aarch64 aarch64 GNU/Linux

Cluster Configuration: Just one node!

Describe the bug:

I'm creating some pods but the problem is that NOT ALWAYS the pods create logs files. Even with the same configuration, I have to kill a pod and wait that the deployment create a new one until I see logs.
I also checked /var/log/pods directly in the node and I get empty files
Example:

orangepi@orangepizero3:/var/log/pods/default_telegram-bot-554c8945cb-mrmfr_890c4f9c-284b-45eb-8e42-d55cbf2488b9/telegram-bot$ sudo cat 0.log 
orangepi@orangepizero3:/var/log/pods/default_telegram-bot-554c8945cb-mrmfr_890c4f9c-284b-45eb-8e42-d55cbf2488b9/telegram-bot$

Then I restart the pod many times, and..

orangepi@orangepizero3:/var/log/pods/default_telegram-bot-554c8945cb-cnxmb_4336c9ca-72a5-49cd-bde4-922f59aac445/telegram-bot$ sudo cat 0.log 
2024-05-03T15:04:15.346227804-04:00 stdout F 
2024-05-03T15:04:15.346329804-04:00 stdout F > [email protected] start
2024-05-03T15:04:15.346349553-04:00 stdout F > node app.js
2024-05-03T15:04:15.346356387-04:00 stdout F 
orangepi@orangepizero3:/var/log/pods/default_telegram-bot-554c8945cb-cnxmb_4336c9ca-72a5-49cd-bde4-922f59aac445/telegram-bot$ 

This happens while creating pods with kubectl apply or argocd

❯ k get po
NAME                                        READY   STATUS      RESTARTS   AGE
telegram-scheduler-5669dd7cc5-nfpfp         1/1     Running     0          15h
restart-telegram-scheduler-28578480-vtxs6   0/1     Completed   0          15h
telegram-bot-2-67f4d6f74f-96cnb             1/1     Running     0          2s
❯ k logs pods/telegram-bot-2-67f4d6f74f-96cnb -f

Steps To Reproduce:

• Installed K3s: --write-kubeconfig-mode=644 --tls-san=MYDNS
• Installed argocd
• Installed a private docker registry

Expected behavior:

Every time that a pod is created, the logs file should be created.

Actual behavior:

Pod logs are not consistent since many times they are not even created.

[brandond]

Every time that a pod is created, the logs file should be created.
Pod logs are not consistent since many times they are not even created.

That's not what you showed. You showed that the log files are being created, but contain no output. Can you replicate this with a test pods whose container is guaranteed to produce output on stdout or stderr?

It seems more likely that your containers just aren't writing any log output. I'm not aware of any issues with containerd where it fails to collect output from containerized processes.

Can you also confirm that you're using the embedded containerd, and not docker or some other container runtime?