Add configurable Backend Type for flannel

[michaelkebe]

Pull Request

What? (description)

Make the flannel backend type easily configurable.

I am no go developer, but I am willing to learn. Please give me hints what the next thing is to handle. The machine config need an additional parameter (https://www.talos.dev/v1.9/reference/configuration/v1alpha1/config/#Config.cluster.network.cni.flannel).

Why? (reasoning)

Currently there is no easy way to set the backend type to something else. Running Talos in a VMware environment with a VMXNET3 adapter needs the backend type host-gw.

There is a note in the VMware documentation, but it is broken.

Acceptance

Please use the following checklist:

• you linked an issue (if applicable)
• you included tests (if applicable)
• you ran conformance (make conformance)
• you formatted your code (make fmt)
• you linted your code (make lint)
• you generated documentation (make docs)
• you ran unit-tests (make unit-tests)

See make help for a description of the available targets.

[smira]

I think this will be half-baked solution, as backend settings are still not configurable.

Why can't you use upstream Flannel manifests and disable Talos's built-in Flannel? Anything bad on that path?

[smira]

Marked as draft, as it's certainly not ready to be merged.

[pit-hub]

@michaelkebe If you have issues with Node to Node communication in VMWare when using Geneve tunneling. I suggest you try Talos v1.10 alpha with EthernetConfig witch support low level networking config and setup tx-udp_tnl-segmentation off and tx-udp_tnl-csum-segmentation off

Or Standard Linux command line:

ethtool -K eth0 tx-udp_tnl-segmentation off tx-udp_tnl-csum-segmentation off

I'm currently using RKE2 with Cilium, in VMWare, and that fixed my problem for Node to Node communications within a Geneve networking.
See source info here: github issue: Document installation on vSphere #21801

The Talos EthernetConfig should be like this:

apiVersion: v1alpha1
kind: EthernetConfig
name: eth0 # Name of the link (interface).
# Configuration for Ethernet features.
features:
    tx-udp_tnl-segmentation: false
    tx-udp_tnl-csum-segmentation: false

[michaelkebe]

Thanks @pit-hub for your information.

Currently I use this Talos configuration patch:

cluster:
    network:
        podSubnets:
          - 10.36.0.0/16
        serviceSubnets:
          - 10.35.0.0/16
        # Deactivated default installation of flannel
        # See below "flannel-custom-installation"
        cni:
          name: none
    inlineManifests:
      - name: flannel-custom-installation
        # https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml
        # with configured backend (host-gw), because VMwares VMXNET adapter does not work without problems
        # (https://www.talos.dev/v1.9/talos-guides/install/virtualized-platforms/vmware/#validate-the-configuration-files)
        # and adjusted network (see cluster.network.podSubnets)
        contents: |-
          apiVersion: v1
          kind: Namespace
          metadata:
            labels:
              k8s-app: flannel
              pod-security.kubernetes.io/enforce: privileged
            name: kube-flannel
          ---
          apiVersion: v1
          kind: ServiceAccount
          metadata:
            labels:
              k8s-app: flannel
            name: flannel
            namespace: kube-flannel
          ---
          apiVersion: rbac.authorization.k8s.io/v1
          kind: ClusterRole
          metadata:
            labels:
              k8s-app: flannel
            name: flannel
          rules:
          - apiGroups:
            - ""
            resources:
            - pods
            verbs:
            - get
          - apiGroups:
            - ""
            resources:
            - nodes
            verbs:
            - get
            - list
            - watch
          - apiGroups:
            - ""
            resources:
            - nodes/status
            verbs:
            - patch
          ---
          apiVersion: rbac.authorization.k8s.io/v1
          kind: ClusterRoleBinding
          metadata:
            labels:
              k8s-app: flannel
            name: flannel
          roleRef:
            apiGroup: rbac.authorization.k8s.io
            kind: ClusterRole
            name: flannel
          subjects:
          - kind: ServiceAccount
            name: flannel
            namespace: kube-flannel
          ---
          apiVersion: v1
          data:
            cni-conf.json: |
              {
                "name": "cbr0",
                "cniVersion": "0.3.1",
                "plugins": [
                  {
                    "type": "flannel",
                    "delegate": {
                      "hairpinMode": true,
                      "isDefaultGateway": true
                    }
                  },
                  {
                    "type": "portmap",
                    "capabilities": {
                      "portMappings": true
                    }
                  }
                ]
              }
            net-conf.json: |
              {
                "Network": "10.36.0.0/16",
                "EnableNFTables": false,
                "Backend": {
                  "Type": "host-gw"
                }
              }
          kind: ConfigMap
          metadata:
            labels:
              app: flannel
              k8s-app: flannel
              tier: node
            name: kube-flannel-cfg
            namespace: kube-flannel
          ---
          apiVersion: apps/v1
          kind: DaemonSet
          metadata:
            labels:
              app: flannel
              k8s-app: flannel
              tier: node
            name: kube-flannel-ds
            namespace: kube-flannel
          spec:
            selector:
              matchLabels:
                app: flannel
                k8s-app: flannel
            template:
              metadata:
                labels:
                  app: flannel
                  k8s-app: flannel
                  tier: node
              spec:
                affinity:
                  nodeAffinity:
                    requiredDuringSchedulingIgnoredDuringExecution:
                      nodeSelectorTerms:
                      - matchExpressions:
                        - key: kubernetes.io/os
                          operator: In
                          values:
                          - linux
                containers:
                - args:
                  - --ip-masq
                  - --kube-subnet-mgr
                  command:
                  - /opt/bin/flanneld
                  env:
                  - name: POD_NAME
                    valueFrom:
                      fieldRef:
                        fieldPath: metadata.name
                  - name: POD_NAMESPACE
                    valueFrom:
                      fieldRef:
                        fieldPath: metadata.namespace
                  - name: EVENT_QUEUE_DEPTH
                    value: "5000"
                  image: ghcr.io/flannel-io/flannel:v0.26.4
                  name: kube-flannel
                  resources:
                    requests:
                      cpu: 100m
                      memory: 50Mi
                  securityContext:
                    capabilities:
                      add:
                      - NET_ADMIN
                      - NET_RAW
                    privileged: false
                  volumeMounts:
                  - mountPath: /run/flannel
                    name: run
                  - mountPath: /etc/kube-flannel/
                    name: flannel-cfg
                  - mountPath: /run/xtables.lock
                    name: xtables-lock
                hostNetwork: true
                initContainers:
                - args:
                  - -f
                  - /flannel
                  - /opt/cni/bin/flannel
                  command:
                  - cp
                  image: ghcr.io/flannel-io/flannel-cni-plugin:v1.6.2-flannel1
                  name: install-cni-plugin
                  volumeMounts:
                  - mountPath: /opt/cni/bin
                    name: cni-plugin
                - args:
                  - -f
                  - /etc/kube-flannel/cni-conf.json
                  - /etc/cni/net.d/10-flannel.conflist
                  command:
                  - cp
                  image: ghcr.io/flannel-io/flannel:v0.26.4
                  name: install-cni
                  volumeMounts:
                  - mountPath: /etc/cni/net.d
                    name: cni
                  - mountPath: /etc/kube-flannel/
                    name: flannel-cfg
                priorityClassName: system-node-critical
                serviceAccountName: flannel
                tolerations:
                - effect: NoSchedule
                  operator: Exists
                volumes:
                - hostPath:
                    path: /run/flannel
                  name: run
                - hostPath:
                    path: /opt/cni/bin
                  name: cni-plugin
                - hostPath:
                    path: /etc/cni/net.d
                  name: cni
                - configMap:
                    name: kube-flannel-cfg
                  name: flannel-cfg
                - hostPath:
                    path: /run/xtables.lock
                    type: FileOrCreate
                  name: xtables-lock

[github-actions]

This PR is stale because it has been open 45 days with no activity.