Helm-chart - client pod not starting up

[dberardo-com]

Details

What steps did you take and what happened:

i have installed the helm chart from: https://github.com/angelnu/helm-charts/tree/main/charts/apps/pod-gateway following the instructions on: https://docs.k8s-at-home.com/guides/pod-gateway/

the chart installs correctly, but then by deploying the test pod from: https://docs.k8s-at-home.com/guides/pod-gateway/#test-deployment i run into a timeout from the sidebar init-container (see below)

What did you expect to happen:

expected an output similar to this: #15 (comment)

Anything else you would like to add:

using k3s, single-node cluster w/ flatten

Additional Information:

this is the output of the gateway-init sidecar container:

+ ip route
10.42.0.0/24 dev eth0 proto kernel scope link src 10.42.0.13 
10.42.0.0/16 via 10.42.0.1 dev eth0 
++ cut -d ' ' -f 1
+ K8S_DNS_IP=10.43.0.10
++ dig +short vpn-gateway-pod-gateway.pod-gateway.svc.cluster.local @10.43.0.10
+ GATEWAY_IP=';; connection timed out; no servers could be reached'

[dberardo-com]

UPDATE

ok i have found the exact same issue on the doc: https://docs.k8s-at-home.com/guides/pod-gateway/#routed-pod-fails-to-init

that solves the issue above, but i still get this error now

...


+ echo 'Get dynamic IP'
+ dhclient -v -cf /etc/dhclient.conf vxlan0
Internet Systems Consortium DHCP Client 4.4.3
Copyright 2004-2022 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/

/etc/dhclient.conf line 3: semicolon expected.
link-timeout 10;
              ^
/etc/dhclient.conf line 4: semicolon expected.
reboot 
 ^
Listening on LPF/vxlan0/52:22:39:e1:6b:ec
Sending on   LPF/vxlan0/52:22:39:e1:6b:ec
Sending on   Socket/fallback
DHCPDISCOVER on vxlan0 to 255.255.255.255 port 67 interval 1
DHCPDISCOVER on vxlan0 to 255.255.255.255 port 67 interval 2
DHCPDISCOVER on vxlan0 to 255.255.255.255 port 67 interval 2
DHCPDISCOVER on vxlan0 to 255.255.255.255 port 67 interval 1
DHCPDISCOVER on vxlan0 to 255.255.255.255 port 67 interval 2
DHCPDISCOVER on vxlan0 to 255.255.255.255 port 67 interval 2
DHCPDISCOVER on vxlan0 to 255.255.255.255 port 67 interval 1
DHCPDISCOVER on vxlan0 to 255.255.255.255 port 67 interval 2
DHCPDISCOVER on vxlan0 to 255.255.255.255 port 67 interval 2
DHCPDISCOVER on vxlan0 to 255.255.255.255 port 67 interval 2
DHCPDISCOVER on vxlan0 to 255.255.255.255 port 67 interval 1
DHCPDISCOVER on vxlan0 to 255.255.255.255 port 67 interval 2
DHCPDISCOVER on vxlan0 to 255.255.255.255 port 67 interval 2
DHCPDISCOVER on vxlan0 to 255.255.255.255 port 67 interval 2
DHCPDISCOVER on vxlan0 to 255.255.255.255 port 67 interval 2
DHCPDISCOVER on vxlan0 to 255.255.255.255 port 67 interval 2
DHCPDISCOVER on vxlan0 to 255.255.255.255 port 67 interval 2
DHCPDISCOVER on vxlan0 to 255.255.255.255 port 67 interval 1
No DHCPOFFERS received.
No working leases in persistent database - sleeping.
+ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0@if199954: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 47950 qdisc noqueue state UP group default 
    link/ether ae:b9:74:cb:db:43 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.42.0.14/24 brd 10.42.0.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::acb9:74ff:fecb:db43/64 scope link 
       valid_lft forever preferred_lft forever
5: vxlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 47900 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 52:22:39:e1:6b:ec brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5022:39ff:fee1:6bec/64 scope link 
       valid_lft forever preferred_lft forever
+ ip route
10.42.0.0/24 dev eth0 proto kernel scope link src 10.42.0.14 
10.42.0.0/16 via 10.42.0.1 dev eth0 
10.42.0.12 via 10.42.0.1 dev eth0 
10.43.0.0/16 via 10.42.0.1 dev eth0 
+ ping -c 1 172.16.0.1
ping: sendto: Network unreachable
PING 172.16.0.1 (172.16.0.1): 56 data bytes


---

[dberardo-com]

so the important missing part in the log above is the default route, looking aht the k8s@home doc:

The important part is that the default gateway and the DNS are set to 172.16.0.1 which is the default IP of the gateway POD in the vlxlan network. If this is the case then you are ready for the (optional) VPN setup.

and indeed this is not there:

• ip route
  10.42.0.0/24 dev eth0 proto kernel scope link src 10.42.0.14
  10.42.0.0/16 via 10.42.0.1 dev eth0
  10.42.0.12 via 10.42.0.1 dev eth0
  10.43.0.0/16 via 10.42.0.1 dev eth0

[dberardo-com]

when running this line manually: https://github.com/angelnu/pod-gateway/blob/main/bin/gateway_init.sh#L30

the script returns:

$ ip link add vxlan0 type vxlan id $VXLAN_ID dev eth0 dstport 0
RTNETLINK answers: File exists

---

it seems like the vxlan exists, but not the route:

• if i try to delete the vxlan before adding it, then the error from above disappears:

bash-5.1# ip route
10.42.0.0/24 dev eth0 proto kernel scope link src 10.42.0.19
10.42.0.0/16 via 10.42.0.1 dev eth0
10.42.0.12 via 10.42.0.1 dev eth0
10.43.0.0/16 via 10.42.0.1 dev eth0
bash-5.1# ip link del vxlan0
bash-5.1# ip link del vxlan0
Cannot find device "vxlan0"
bash-5.1# ip link add vxlan0 type vxlan id $VXLAN_ID dev eth0 dstport 0
bash-5.1# ip route
10.42.0.0/24 dev eth0 proto kernel scope link src 10.42.0.19
10.42.0.0/16 via 10.42.0.1 dev eth0
10.42.0.12 via 10.42.0.1 dev eth0
10.43.0.0/16 via 10.42.0.1 dev eth0

but as you can see from the last ip route, the default route is not applied ... how come ?

[arana198]

I'm not sure what your settings are but I spent months trying to get pod-gateway working and finally got it working. Here's my values.yaml

image:
  tag: v1.8.1

routed_namespaces:
 - mediaserver

settings:
  VPN_INTERFACE: "tun0"
  VPN_BLOCK_OTHER_TRAFFIC: true
  VPN_TRAFFIC_PORT: 443
  NOT_ROUTED_TO_GATEWAY_CIDRS: "10.0.0.0/8"

addons:
  vpn:
    enabled: true
    type: gluetun
    env:
      - name: VPN_SERVICE_PROVIDER
        value: "nordvpn"
      - name: VPN_TYPE
        value: "openvpn"
      - name: OPENVPN_PROTOCOL
        value: "tcp"
 
    securityContext:
      capabilities:
        add:
          - NET_ADMIN
          
    networkPolicy:
      enabled: false
      policyTypes:
        - Ingress
        - Egress
      ingress:
        - from:
            # Only allow ingress from K8S
            - ipBlock:
                cidr: 10.0.0.0/8
      egress:
        # Allow only VPN traffic to Internet
        - to:
          - ipBlock:
              cidr: 0.0.0.0/0
          ports:
            # VPN traffic port - change if your provider uses a different port
            - port: 443
              protocol: TCP
        - to:
            # Allow traffic within K8S - change if your K8S cluster uses a different CIDR
          - ipBlock:
              cidr: 10.0.0.0/8

[dberardo-com]

thanks for the quick response, are you using Calico or Flannel ?

[arana198]

I am using Flannel with Canal

[dberardo-com]

alright, in my case i am trying to setup the gateway first without any VPN, so using this:

settings:
  NOT_ROUTED_TO_GATEWAY_CIDRS: "10.42.0.0/16 10.43.0.0/16"

as per documentation.

but that seems not to work because the init container somehow does not manage to setup a default gateway within the container ... i wonder why. i am now trying with your setup, i.e. setting the image.tag value and the :

NOT_ROUTED_TO_GATEWAY_CIDRS: "10.0.0.0/8"

but i guest this will not make any difference. same goes for the VPN part, i guess that should not solve the core issue

---

indeed it didnt make any difference. do you also get this strange error/warning in the log?

For info, please visit https://www.isc.org/software/dhcp/

/etc/dhclient.conf line 3: semicolon expected.
link-timeout 10;
              ^
/etc/dhclient.conf line 4: semicolon expected.
reboot 

i cannot find a way to overwrite that file , but it seems that it is not supported: https://raspberrypi.stackexchange.com/questions/135208/semicolon-required-in-etc-dhcp-dhclient-conf

i think this might be the issue because that file ends with this:

interface "vxlan0"
{
  request subnet-mask,
          broadcast-address,
          routers;
          #domain-name-servers;
  require routers,
          subnet-mask;
          #domain-name-servers;
}

and perhaps this is why the interface/default route is not correctly created

---

here i found the same error reported by someone else: k8s-at-home/charts#1633 (comment)

[arana198]

Can you post your full config (values.yaml) file?

[dberardo-com]

Can you post your full config (values.yaml) file?

these are litterally the only values i have changed:

image:
  tag: v1.8.1
routed_namespaces:
- vpn
settings:
  NOT_ROUTED_TO_GATEWAY_CIDRS: 10.42.0.0/16 10.43.0.0/16

so the full values file is:

DNS: 172.16.0.1
DNSPolicy: None
addons:
  vpn:
    enabled: false
    networkPolicy:
      egress:
      - ports:
        - port: 1194
          protocol: UDP
        to:
        - ipBlock:
            cidr: 0.0.0.0/0
      - to:
        - ipBlock:
            cidr: 10.0.0.0/8
      enabled: true
    type: openvpn
clusterName: cluster.local
common:
  addons:
    codeserver:
      args:
      - --auth
      - none
      enabled: false
      env: {}
      git:
        deployKey: ""
        deployKeyBase64: ""
        deployKeySecret: ""
      image:
        pullPolicy: IfNotPresent
        repository: ghcr.io/coder/code-server
        tag: 4.10.0
      ingress:
        annotations: {}
        enabled: false
        hosts:
        - host: code.chart-example.local
          paths:
          - path: /
            pathType: Prefix
        labels: {}
        tls: []
      securityContext:
        runAsUser: 0
      service:
        annotations: {}
        enabled: true
        labels: {}
        ports:
          codeserver:
            enabled: true
            port: 12321
            protocol: TCP
            targetPort: 12321
        type: ClusterIP
      volumeMounts: []
      workingDir: ""
    netshoot:
      enabled: false
      env: {}
      image:
        pullPolicy: IfNotPresent
        repository: ghcr.io/nicolaka/netshoot
        tag: v0.9
      securityContext:
        capabilities:
          add:
          - NET_ADMIN
    vpn:
      additionalVolumeMounts: []
      args: []
      enabled: false
      env: {}
      gluetun:
        image:
          pullPolicy: IfNotPresent
          repository: docker.io/qmcgaw/gluetun
          tag: v3.32.0
      livenessProbe: {}
      networkPolicy:
        annotations: {}
        enabled: false
        labels: {}
        podSelectorLabels: {}
      scripts: {}
      securityContext:
        capabilities:
          add:
          - NET_ADMIN
          - SYS_MODULE
      type: gluetun
  affinity: {}
  args: []
  automountServiceAccountToken: true
  command: []
  configMaps:
    config:
      annotations: {}
      data: {}
      enabled: false
      labels: {}
  controller:
    annotations: {}
    cronjob:
      concurrencyPolicy: Forbid
      failedJobsHistory: 1
      schedule: '*/20 * * * *'
      startingDeadlineSeconds: 30
      successfulJobsHistory: 1
    enabled: true
    labels: {}
    replicas: 1
    revisionHistoryLimit: 3
    rollingUpdate: {}
    type: deployment
  dnsConfig: {}
  enableServiceLinks: true
  envFrom: []
  global:
    annotations: {}
    labels: {}
  hostAliases: []
  hostNetwork: false
  image: {}
  imagePullSecrets: []
  ingress:
    main:
      annotations: {}
      enabled: false
      hosts:
      - host: chart-example.local
        paths:
        - path: /
          pathType: Prefix
          service:
            name: null
            port: null
      labels: {}
      primary: true
      tls: []
  initContainers: {}
  lifecycle: {}
  nodeSelector: {}
  persistence:
    config:
      accessMode: ReadWriteOnce
      enabled: false
      readOnly: false
      retain: false
      size: 1Gi
      type: pvc
    shared:
      enabled: false
      mountPath: /shared
      type: emptyDir
  podAnnotations: {}
  podLabels: {}
  podSecurityContext: {}
  probes:
    liveness:
      custom: false
      enabled: true
      spec:
        failureThreshold: 3
        initialDelaySeconds: 0
        periodSeconds: 10
        timeoutSeconds: 1
      type: TCP
    readiness:
      custom: false
      enabled: true
      spec:
        failureThreshold: 3
        initialDelaySeconds: 0
        periodSeconds: 10
        timeoutSeconds: 1
      type: TCP
    startup:
      custom: false
      enabled: true
      spec:
        failureThreshold: 30
        initialDelaySeconds: 0
        periodSeconds: 5
        timeoutSeconds: 1
      type: TCP
  resources: {}
  route:
    main:
      annotations: {}
      enabled: false
      hostnames: []
      kind: HTTPRoute
      labels: {}
      parentRefs:
      - group: gateway.networking.k8s.io
        kind: Gateway
        name: null
        namespace: null
        sectionName: null
      rules:
      - backendRefs:
        - group: ""
          kind: Service
          name: null
          namespace: null
          port: null
          weight: 1
        matches:
        - path:
            type: PathPrefix
            value: /
  secrets:
    secret:
      annotations: {}
      enabled: false
      labels: {}
      stringData: {}
  securityContext: {}
  service:
    main:
      annotations: {}
      enabled: true
      ipFamilies: []
      labels: {}
      ports:
        http:
          enabled: true
          extraSelectorLabels: {}
          primary: true
          protocol: HTTP
      primary: true
      type: ClusterIP
  serviceAccount:
    annotations: {}
    create: false
    name: ""
  serviceMonitor:
    main:
      annotations: {}
      enabled: false
      endpoints:
      - interval: 1m
        path: /metrics
        port: http
        scheme: http
        scrapeTimeout: 10s
      labels: {}
      selector: {}
      serviceName: '{{ include "bjw-s.common.lib.chart.names.fullname" $ }}'
  sidecars: {}
  termination: {}
  tolerations: []
  topologySpreadConstraints: []
  volumeClaimTemplates: []
image:
  pullPolicy: IfNotPresent
  repository: ghcr.io/angelnu/pod-gateway
  tag: v1.8.1
publicPorts: null
routed_namespaces:
- vpn
settings:
  DNS_LOCAL_CIDRS: local
  NOT_ROUTED_TO_GATEWAY_CIDRS: 10.42.0.0/16 10.43.0.0/16
  VPN_BLOCK_OTHER_TRAFFIC: false
  VPN_INTERFACE: tun0
  VPN_LOCAL_CIDRS: 10.0.0.0/8 192.168.0.0/16
  VPN_TRAFFIC_PORT: 1194
  VXLAN_GATEWAY_FIRST_DYNAMIC_IP: 20
  VXLAN_ID: 42
  VXLAN_IP_NETWORK: 172.16.0
webhook:
  gatewayAnnotation: setGateway
  gatewayAnnotationValue: null
  gatewayDefault: true
  gatewayLabel: setGateway
  gatewayLabelValue: null
  image:
    pullPolicy: IfNotPresent
    repository: ghcr.io/angelnu/gateway-admision-controller
    tag: v3.8.0
  namespaceSelector:
    custom: {}
    label: routed-gateway
    type: label
  replicas: 1
  strategy:
    type: RollingUpdate

[arana198]

Just checking that your routed pods is not in the same namespace as pod-gateway? i.e pod-gateway is in different namespace to sonarr/radarr/any other routed pod

[dberardo-com]

Just checking that your routed pods is not in the same namespace as pod-gateway? i.e pod-gateway is in different namespace to sonarr/radarr/any other routed pod

done, i have followed this procedure: https://docs.k8s-at-home.com/guides/pod-gateway/#pod-gateway-helm-release

so i have 2 different namespaces "vpn" for the routed pod and "pod-gateway" for the gateway container (which is healty, up and running)

---

here the gateway namespace:

here the pod namespace: