coredns

[bpopovich44]

I am using rke2 1.24.7 and the dns works fine for a while, but then seems to bottleneck with coredns giving the following errors. It still works, but is intermittent. Ideas?

[WARNING] plugin/health: Local health request to "http://:8080/health" took more than 1s: 1.899644677s
[WARNING] plugin/health: Local health request to "http://:8080/health" took more than 1s: 1.899697325s
[WARNING] plugin/health: Local health request to "http://:8080/health" failed: Get "http://:8080/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
[WARNING] plugin/health: Local health request to "http://:8080/health" took more than 1s: 3.200311788s
[WARNING] plugin/health: Local health request to "http://:8080/health" took more than 1s: 2.000010172s
[WARNING] plugin/health: Local health request to "http://:8080/health" took more than 1s: 1.374516495s
[WARNING] plugin/health: Local health request to "http://:8080/health" took more than 1s: 1.300175671s
[WARNING] plugin/health: Local health request to "http://:8080/health" took more than 1s: 2.100442093s
[WARNING] plugin/health: Local health request to "http://:8080/health" took more than 1s: 1.799865368s
[WARNING] plugin/health: Local health request to "http://:8080/health" took more than 1s: 1.500314764s
[WARNING] plugin/health: Local health request to "http://:8080/health" failed: Get "http://:8080/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
[WARNING] plugin/health: Local health request to "http://:8080/health" took more than 1s: 2.600244392s
[WARNING] plugin/health: Local health request to "http://:8080/health" took more than 1s: 2.671504106s
[WARNING] plugin/health: Local health request to "http://:8080/health" took more than 1s: 1.585956884s
[WARNING] plugin/health: Local health request to "http://:8080/health" failed: Get "http://:8080/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
[WARNING] plugin/health: Local health request to "http://:8080/health" took more than 1s: 2.003437761s
[WARNING] plugin/health: Local health request to "http://:8080/health" took more than 1s: 2.999504985s
[WARNING] plugin/health: Local health request to "http://:8080/health" took more than 1s: 1.100054141s
[WARNING] plugin/health: Local health request to "http://:8080/health" took more than 1s: 1.100353923s
[WARNING] plugin/health: Local health request to "http://:8080/health" failed: Get "http://:8080/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
[WARNING] plugin/health: Local health request to "http://:8080/health" took more than 1s: 2.300393019s
[WARNING] plugin/health: Local health request to "http://:8080/health" took more than 1s: 2.499457076s
[WARNING] plugin/health: Local health request to "http://:8080/health" took more than 1s: 2.400130227s
[WARNING] plugin/health: Local health request to "http://:8080/health" took more than 1s: 1.500350949s
[WARNING] plugin/health: Local health request to "http://:8080/health" failed: Get "http://:8080/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
[WARNING] plugin/health: Local health request to "http://:8080/health" took more than 1s: 1.59925706s
[WARNING] plugin/health: Local health request to "http://:8080/health" took more than 1s: 1.399565328s
[WARNING] plugin/health: Local health request to "http://:8080/health" took more than 1s: 1.10005273s
[WARNING] plugin/health: Local health request to "http://:8080/health" took more than 1s: 1.300457729s
[WARNING] plugin/health: Local health request to "http://:8080/health" took more than 1s: 1.504547838s
[WARNING] plugin/health: Local health request to "http://:8080/health" took more than 1s: 1.701145327s

[brandond]

How much CPU time is the coredns pod using? What does kubectl top pod -n kube-system show? If you describe the coredns pod, is it being restarted for any reason?

[bpopovich44]

Hi Brandon,
Not sure if this is still open but still getting errors with DNS, but the logs are empty. I put in another request for help because not sure if this is still open.
Thank you

[brandond]

This is a discussion thread, it doesn't ever really close, although you can mark a comment as the answer. If you have a Suse/Rancher support contract, you can open a ticket - but this is just best-effort community assistance, and we don't always have time to respond.

I don't have any immediate suggestions for you other than checking for errors in other logs and ensuring that the node-local dns pods aren't also being throttled.

[bpopovich44]

No worries on the response, I know you have other stuff going on, but I appreciate your help.
I ended up deleting the NodeLocal DNS because it seemed to make things worse. I also noticed it didn't launch a pod which I believe it is supposed to.

[brandond]

It should result in creation of a daemonset that runs a coredns pod on each of the nodes. I do suspect that it might help with your load issue.

[bpopovich44]

Hey! So it turns out there was a bad NIC in one of the servers causing all the issues. I drained the node and pushed coredns to another worker node and it fixed the issues. Although the CPUs for coredns are still hitting about 250m, you definitely helped fix that issue so thank you. The coredns logs are empty. I really appreciate your help.