Issue with Pod networking

[dalekurt]

I was able to complete the installation of Flux using task cluster:install

kubectl -n flux-system get pods -o wide
NAME                                          READY   STATUS    RESTARTS      AGE   IP            NODE       NOMINATED NODE   READINESS GATES
helm-controller-784f764bfb-7m8pm              0/1     Running   1 (15s ago)   45s   10.42.1.68    worker-1   <none>           <none>
image-automation-controller-b7c5b48f6-vszrr   0/1     Running   1 (5s ago)    35s   10.42.1.252   worker-1   <none>           <none>
image-reflector-controller-6658b59bc4-fs2lx   0/1     Running   0             27s   10.42.3.15    worker-3   <none>           <none>
kustomize-controller-fd48f87c4-kfbxz          0/1     Running   0             20s   10.42.2.207   worker-2   <none>           <none>
notification-controller-6b9b544755-6v8fx      0/1     Running   0             13s   10.42.3.78    worker-3   <none>           <none>
source-controller-8448d9bf67-52qzt            0/1     Running   0             5s    10.42.2.35    worker-2   <none>           <none>

Verified that the Flux components are running in the cluster

Then failed to verified all the resources on the cluster with task cluster:resources

task: [cluster:certificates] kubectl get certificates -A
error: the server doesn't have a resource type "certificates"
task: Failed to run task "cluster:resources": exit status 1

Anyone having a similar issue?

[onedr0p]

It doesn't look like flux is healthy, none of the pods are reporting ready/healthy 1/1

[dalekurt]

Looks like there is a communication issues

Events:
  Type     Reason     Age                     From     Message
  ----     ------     ----                    ----     -------
  Normal   Pulled     47m (x38 over 152m)     kubelet  Container image "ghcr.io/fluxcd/source-controller:v1.1.2" already present on machine
  Warning  Unhealthy  22m (x284 over 152m)    kubelet  Readiness probe failed: Get "http://10.42.2.35:9090/": dial tcp 10.42.2.35:9090: connect: connection refused
  Warning  BackOff    2m53s (x603 over 150m)  kubelet  Back-off restarting failed container manager in pod source-controller-8448d9bf67-52qzt_flux-system(3f9b9a9b-1707-4010-a3b5-c9b0c9f83ba0)

I'm looking into this further.

[onedr0p]

Looks like more gremlins, is you CNI healthy?

[dalekurt]

@onedr0p I nuked and recreated the cluster. Everything in kube-system looks healthy

kubectl get pods -n kube-system
NAME                               READY   STATUS    RESTARTS       AGE
cilium-8t5z5                       1/1     Running   0              11m
cilium-d7zmg                       1/1     Running   0              11m
cilium-m24pg                       1/1     Running   0              11m
cilium-operator-7779cbffcf-xpz2r   1/1     Running   1 (11m ago)    11m
cilium-plfks                       1/1     Running   0              11m
coredns-c996f6cfc-8rv4v            1/1     Running   0              11m
kube-vip-control-plane             1/1     Running   14 (11m ago)   12m

Here are the logs from one of the pods. Seeing an issue there

level=warning msg="Unable to install direct node route {Ifindex: 0 Dst: 10.42.3.0/24 Src: <nil> Gw: 172.16.30.23 Flags: [] Table: 0 Realm: 0}" error="route to destination 172.16.30.23 contains gateway 172.16.30.1, must be directly reachable" subsys=linux-datapath
level=warning msg="Unable to install direct node route {Ifindex: 0 Dst: 10.42.0.0/24 Src: <nil> Gw: 172.16.30.20 Flags: [] Table: 0 Realm: 0}" error="route to destination 172.16.30.20 contains gateway 172.16.30.1, must be directly reachable" subsys=linux-datapath
level=warning msg="Unable to install direct node route {Ifindex: 0 Dst: 10.42.1.0/24 Src: <nil> Gw: 172.16.30.21 Flags: [] Table: 0 Realm: 0}" error="route to destination 172.16.30.21 contains gateway 172.16.30.1, must be directly reachable" subsys=linux-datapath
level=warning msg="Unable to install direct node route {Ifindex: 0 Dst: 10.42.3.0/24 Src: <nil> Gw: 172.16.30.23 Flags: [] Table: 0 Realm: 0}" error="route to destination 172.16.30.23 contains gateway 172.16.30.1, must be directly reachable" subsys=linux-datapath
level=warning msg="Unable to install direct node route {Ifindex: 0 Dst: 10.42.0.0/24 Src: <nil> Gw: 172.16.30.20 Flags: [] Table: 0 Realm: 0}" error="route to destination 172.16.30.20 contains gateway 172.16.30.1, must be directly reachable" subsys=linux-datapath
level=warning msg="Unable to install direct node route {Ifindex: 0 Dst: 10.42.1.0/24 Src: <nil> Gw: 172.16.30.21 Flags: [] Table: 0 Realm: 0}" error="route to destination 172.16.30.21 contains gateway 172.16.30.1, must be directly reachable" subsys=linux-datapath
level=warning msg="Unable to install direct node route {Ifindex: 0 Dst: 10.42.3.0/24 Src: <nil> Gw: 172.16.30.23 Flags: [] Table: 0 Realm: 0}" error="route to destination 172.16.30.23 contains gateway 172.16.30.1, must be directly reachable" subsys=linux-datapath
level=info msg="Conntrack garbage collector interval recalculated" deleteRatio=0.00102996826171875 newInterval=11m15s subsys=map-ct

[dalekurt]

I found this after doing some Googling, I'm checking my network configuration

[dalekurt]

Does not look like a configuration issue on my router at the moment.

My route tables on a couple of my nodes in the cluster

worker-1

ubuntu@worker-1:~$ route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         _gateway        0.0.0.0         UG    0      0        0 eth0
10.42.1.0       worker-1        255.255.255.0   UG    0      0        0 cilium_host
worker-1        0.0.0.0         255.255.255.255 UH    0      0        0 cilium_host
10.42.1.166     0.0.0.0         255.255.255.255 UH    0      0        0 lxce06f980a9564
10.42.1.181     0.0.0.0         255.255.255.255 UH    0      0        0 lxc64bb53ac21b9
10.42.1.200     0.0.0.0         255.255.255.255 UH    0      0        0 lxc_health

worker-2

ubuntu@worker-2:~$ route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         _gateway        0.0.0.0         UG    0      0        0 eth0
10.42.2.0       worker-2        255.255.255.0   UG    0      0        0 cilium_host
worker-2        0.0.0.0         255.255.255.255 UH    0      0        0 cilium_host
10.42.2.55      0.0.0.0         255.255.255.255 UH    0      0        0 lxc_health
10.42.2.71      0.0.0.0         255.255.255.255 UH    0      0        0 lxc0fcd46f68a97
10.42.2.124     0.0.0.0         255.255.255.255 UH    0      0        0 lxc547e2c11c67e

[onedr0p]

I am not sure what is happening here :/ Are all the nodes on the same network? It's hard to debug without knowing the exact details of your setup.

[dalekurt]

All the nodes are on the same network. Yeah, I understand. I have been digging through some issues which others have experiences similar to my own versions cilium v1.14.3, kubernetes v1.28.2 (k3s)

[dalekurt]

@onedr0p I found someone who raised this issue, similar to what I'm experiencing

[onedr0p]

I wonder if changing the bootstrap_cilium_loadbalancer_mode to snat would help.
https://github.com/onedr0p/flux-cluster-template/blob/6428678fae57937bcdffd99d6b0a58eb4f65f9a8/bootstrap/vars/config.sample.yaml#L59-L63