-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Failed to connect to kube-dns after setting up calico cni #8811
Comments
I even allowed communication between workers and masters for all ports from wildcard range |
Logs from
|
I also have the same version v3.27.3 and encountered the same problem |
@AlanduzzZ
Are you installing calico using the operator? What's your config and are you using dual stack? |
I found something. |
I seem to be having a similar issue. Going to try to add a node and see what happens, I believe I had all of this working earlier but I have gone from 4 nodes to 1 now. |
Adding a node solved my problem.. |
Actually the pods running on same nodes where coredns is running is able to resolve dns. I've multiple worker nodes and If I run all coredns pods on any of the node e.g |
i dont have any firewall rules, pretty vanilla setup.. for me if I only have one node, I get pods that can't do dns/TLS verification. |
@sqaisar |
Could one of you please help me out with some diags? What I'd like to know is:
|
@AlanduzzZ @sqaisar @mfreeman451 any new information on this issue that would help us to diagnose it? |
same. kubernetes 1.21, ubuntu 18-22. migrated from flannel to calico 3.28.0, |
I have the same problem, pods not able to resolve pods in same namespace. |
Glad more people are finally running into this.. |
For anyone saying that the problem on occurs when the pod is on a different node from coredns, please make sure you have allowed the necessary traffic in your security groups / cloud firewall rules for your method of installation: https://docs.tigera.io/calico/latest/getting-started/kubernetes/requirements#network-requirements Another thing to check would be what encapsulation mode you are using. |
For me, these problems were occuring in a private k8s/on-prem setting, so no cloud controller involved (k3s). |
@mfreeman451 it's still possible that an on-premise network could have a firewall or similar device in place that is preventing the necessary traffic, though if you're confident that isn't the case in your situation the next step is trace the packets to determine where they are getting dropped. |
Expected Behavior
Should be able to connect with kube-dns ip from any of the pods
Current Behavior
Fails to connect with kube-dns service ip from within the pod and because of this any operation that needs dns resolution fails inside any of the scheduled pods in the cluster.
Possible Solution
Steps to Reproduce (for bugs)
Context
I've setup kubernetes
v1.29
using kubeadm and using tiegra operator for setting up the calico versionv3.27.3
. All of thecalico-nodes
are running fine without any restarts and I don't see any obvious errors.Installation config
Nodes
All pods in
calico-system
namespaceService
kube-dns
From withing the pod I'm able to connect to any external public IP but the dns resolution fails because of timeout. the rest of pods ip allocation and everything works as expected.
Although I'm able to connect with other service IP's for example
I'm sure I've allowed all required ports from the firewall ingress/egress from all the k8s nodes that are listed here
https://docs.tigera.io/calico/latest/getting-started/kubernetes/requirements#network-requirements
Your Environment
v3.27.3
v1.29
Ubuntu 22.04
v1.29.0
SystemdCgroup = true
Systemd Groups are configured correctly BTWI'm new to setting up calico so I'd really appreciate all the help, I can shared the calico-node logs if that's required.
The text was updated successfully, but these errors were encountered: