Encryption and private networking question

[giovannicandido]

Hello, thanks for the great project.
I have one question about network encryption.
I planing to install the k8s cluster in a untrusted network, without a private network option. The Virtual Private Servers I choose to run unfortunate don't have this option.
Do RKE2 has a network encryption option to configure?
I can create a mesh encrypted network between the hosts using tailscale, zerotier or pure wireguard, and configure the servers and clientes to listen to the private interface address, then setup my machine to connect to this VPN. I think this is the option I will proceed but I would like some advice on this.
I know that RKE1 has support for wave network which brings encryption, but I would like to setup a new cluster using containerd because is more future prof (kubernetes droping support for docker).
Any feedback I appreciate.
Thanks.

[manuelbuil]

Hey Giovanni, rke2 supports calico, cilium and canal ==> https://docs.rke2.io/install/network_options/#network-options. All of them have encryption features (e.g. using wireguard natively)

[giovannicandido]

Thanks, I will check it

[Torkolis]

Hey Giovanni, rke2 supports calico, cilium and canal ==> https://docs.rke2.io/install/network_options/#network-options. All of them have encryption features (e.g. using wireguard natively)

Hi I have another question about this. When I install an actual RKE2 with it's defaults, like canal, which has encryption features as you say, is this encryption enabled by default? Flannel uses vxlan as default backend as I have observed and read in the docs (https://docs.rke2.io/install/network_options/), which does not encrypt traffic by default as in https://rancher.com/docs/rancher/v2.6/en/faq/networking/cni-providers/#flannel. So I assume it is not encrypted by default, but please correct me if I'm wrong.

However RKE2 docs say, that I can create a helmchartconfig object

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-canal
  namespace: kube-system
spec:
  valuesContent: |-
    flannel:
      backend: "wireguard"

to enable wireguard backend instead of vxlan, which I have tried and it seems to have done something (it edited the file /etc/kube-flannel/net-conf.json in the kube-flannel container in rke2-canal pod) and though I don't know how exactly to test it I trust that it works as it should.
but I don't understand entirelly why the docs say "Starting with RKE2 v1.23 it is possible to use flannels wireguard backend ...", I have checked on RKE2 v1.22.9+rke2r1 cluster in the kube-flannel container
bash-4.2# /opt/bin/flanneld -version
and it says "v.0.17.0" same as on RKE2 v1.23.6+rke2r2, so why is the v1.22 not supposed to work too?

Another interesting fact is if I check logs of the kube-flannel container, I can still see some vxlan keywords but no wireguard keywords:

kubectl logs -n kube-system rke2-canal-v25kl -c kube-flannel
I0519 08:44:54.487957       1 main.go:205] CLI flags config: {etcdEndpoints:http://127.0.0.1:4001,http://127.0.0.1:2379 etcdPrefix:/coreos.com/network etcdKeyfile: etcdCertfile: etcdCAFile: etcdUsername: etcdPassword: version:false kubeSubnetMgr:true kubeApiUrl: kubeAnnotationPrefix:flannel.alpha.coreos.com kubeConfigFile: iface:[] ifaceRegex:[] ipMasq:true subnetFile:/run/flannel/subnet.env publicIP: publicIPv6: subnetLeaseRenewMargin:60 healthzIP:0.0.0.0 healthzPort:0 iptablesResyncSeconds:5 iptablesForwardRules:true netConfPath:/etc/kube-flannel/net-conf.json setNodeNetworkUnavailable:true}
W0519 08:44:54.488101       1 client_config.go:614] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0519 08:44:54.507231       1 kube.go:120] Waiting 10m0s for node controller to sync
I0519 08:44:54.507357       1 kube.go:378] Starting kube subnet manager
I0519 08:44:55.507657       1 kube.go:127] Node controller sync successful
I0519 08:44:55.507689       1 main.go:225] Created subnet manager: Kubernetes Subnet Manager - dockervm
I0519 08:44:55.507694       1 main.go:228] Installing signal handlers
I0519 08:44:55.507898       1 main.go:454] Found network config - Backend type: vxlan
I0519 08:44:55.507917       1 match.go:189] Determining IP address of default interface
I0519 08:44:55.508166       1 match.go:242] Using interface with name eth0 and address 172.27.102.164
I0519 08:44:55.508215       1 match.go:264] Defaulting external address to interface address (172.27.102.164)
I0519 08:44:55.508275       1 vxlan.go:138] VXLAN config: VNI=1 Port=0 GBP=false Learning=false DirectRouting=false
I0519 08:44:55.523021       1 kube.go:339] Setting NodeNetworkUnavailable
I0519 08:44:55.531613       1 main.go:403] Current network or subnet (10.42.0.0/16, 10.42.0.0/24) is not equal to previous one (0.0.0.0/0, 0.0.0.0/0), trying to recycle old iptables rules
I0519 08:44:55.555681       1 iptables.go:255] Deleting iptables rule: -s 0.0.0.0/0 -d 0.0.0.0/0 -m comment --comment flanneld masq -j RETURN
I0519 08:44:55.556598       1 iptables.go:255] Deleting iptables rule: -s 0.0.0.0/0 ! -d 224.0.0.0/4 -m comment --comment flanneld masq -j MASQUERADE --random-fully
I0519 08:44:55.557488       1 iptables.go:255] Deleting iptables rule: ! -s 0.0.0.0/0 -d 0.0.0.0/0 -m comment --comment flanneld masq -j RETURN
I0519 08:44:55.558477       1 iptables.go:255] Deleting iptables rule: ! -s 0.0.0.0/0 -d 0.0.0.0/0 -m comment --comment flanneld masq -j MASQUERADE --random-fully
I0519 08:44:55.559291       1 main.go:332] Setting up masking rules
I0519 08:44:55.559996       1 main.go:353] Changing default FORWARD chain policy to ACCEPT
I0519 08:44:55.560199       1 main.go:366] Wrote subnet file to /run/flannel/subnet.env
I0519 08:44:55.560237       1 main.go:370] Running backend.
I0519 08:44:55.560280       1 main.go:391] Waiting for all goroutines to exit
I0519 08:44:55.560316       1 vxlan_network.go:61] watching for new subnet leases
I0519 08:44:55.561503       1 iptables.go:231] Some iptables rules are missing; deleting and recreating rules
I0519 08:44:55.561537       1 iptables.go:255] Deleting iptables rule: -s 10.42.0.0/16 -d 10.42.0.0/16 -m comment --comment flanneld masq -j RETURN
I0519 08:44:55.561994       1 iptables.go:231] Some iptables rules are missing; deleting and recreating rules
I0519 08:44:55.562020       1 iptables.go:255] Deleting iptables rule: -s 10.42.0.0/16 -m comment --comment flanneld forward -j ACCEPT
I0519 08:44:55.562299       1 iptables.go:255] Deleting iptables rule: -s 10.42.0.0/16 ! -d 224.0.0.0/4 -m comment --comment flanneld masq -j MASQUERADE --random-fully
I0519 08:44:55.562989       1 iptables.go:255] Deleting iptables rule: ! -s 10.42.0.0/16 -d 10.42.0.0/24 -m comment --comment flanneld masq -j RETURN
I0519 08:44:55.563106       1 iptables.go:255] Deleting iptables rule: -d 10.42.0.0/16 -m comment --comment flanneld forward -j ACCEPT
I0519 08:44:55.563914       1 iptables.go:255] Deleting iptables rule: ! -s 10.42.0.0/16 -d 10.42.0.0/16 -m comment --comment flanneld masq -j MASQUERADE --random-fully
I0519 08:44:55.564207       1 iptables.go:243] Adding iptables rule: -s 10.42.0.0/16 -m comment --comment flanneld forward -j ACCEPT
I0519 08:44:55.564698       1 iptables.go:243] Adding iptables rule: -s 10.42.0.0/16 -d 10.42.0.0/16 -m comment --comment flanneld masq -j RETURN
I0519 08:44:55.565770       1 iptables.go:243] Adding iptables rule: -d 10.42.0.0/16 -m comment --comment flanneld forward -j ACCEPT
I0519 08:44:55.566286       1 iptables.go:243] Adding iptables rule: -s 10.42.0.0/16 ! -d 224.0.0.0/4 -m comment --comment flanneld masq -j MASQUERADE --random-fully
I0519 08:44:55.567930       1 iptables.go:243] Adding iptables rule: ! -s 10.42.0.0/16 -d 10.42.0.0/24 -m comment --comment flanneld masq -j RETURN
I0519 08:44:55.569626       1 iptables.go:243] Adding iptables rule: ! -s 10.42.0.0/16 -d 10.42.0.0/16 -m comment --comment flanneld masq -j MASQUERADE --random-fully

like that vxlan.go:138] VXLAN config: VNI=1 Port=0 GBP=false Learning=false DirectRouting=false
can I get an explanation for this?

[Torkolis]

Interesting, I think the issue with the logs is that after creating the helmchartconfig object (and after the job for the helm upgrade is executed) the daemonset is not restarted, I just restarted it manually and it looks like it now has different logs (with wireguard keywords in it):

kubectl logs -n kube-system rke2-canal-sxt5t -c kube-flannel
I0519 09:27:34.097878       1 main.go:205] CLI flags config: {etcdEndpoints:http://127.0.0.1:4001,http://127.0.0.1:2379 etcdPrefix:/coreos.com/network etcdKeyfile: etcdCertfile: etcdCAFile: etcdUsername: etcdPassword: version:false kubeSubnetMgr:true kubeApiUrl: kubeAnnotationPrefix:flannel.alpha.coreos.com kubeConfigFile: iface:[] ifaceRegex:[] ipMasq:true subnetFile:/run/flannel/subnet.env publicIP: publicIPv6: subnetLeaseRenewMargin:60 healthzIP:0.0.0.0 healthzPort:0 iptablesResyncSeconds:5 iptablesForwardRules:true netConfPath:/etc/kube-flannel/net-conf.json setNodeNetworkUnavailable:true}
W0519 09:27:34.097960       1 client_config.go:614] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0519 09:27:34.108023       1 kube.go:120] Waiting 10m0s for node controller to sync
I0519 09:27:34.108063       1 kube.go:378] Starting kube subnet manager
I0519 09:27:35.108650       1 kube.go:127] Node controller sync successful
I0519 09:27:35.110197       1 main.go:225] Created subnet manager: Kubernetes Subnet Manager - dockervm
I0519 09:27:35.110321       1 main.go:228] Installing signal handlers
I0519 09:27:35.111381       1 main.go:454] Found network config - Backend type: wireguard
I0519 09:27:35.111600       1 match.go:189] Determining IP address of default interface
I0519 09:27:35.113686       1 match.go:242] Using interface with name eth0 and address 172.27.102.164
I0519 09:27:35.113941       1 match.go:264] Defaulting external address to interface address (172.27.102.164)
I0519 09:27:35.152197       1 kube.go:339] Setting NodeNetworkUnavailable
I0519 09:27:35.161157       1 main.go:403] Current network or subnet (10.42.0.0/16, 10.42.0.0/24) is not equal to previous one (0.0.0.0/0, 0.0.0.0/0), trying to recycle old iptables rules
I0519 09:27:35.209973       1 iptables.go:255] Deleting iptables rule: -s 0.0.0.0/0 -d 0.0.0.0/0 -m comment --comment flanneld masq -j RETURN
I0519 09:27:35.211090       1 iptables.go:255] Deleting iptables rule: -s 0.0.0.0/0 ! -d 224.0.0.0/4 -m comment --comment flanneld masq -j MASQUERADE --random-fully
I0519 09:27:35.218846       1 iptables.go:255] Deleting iptables rule: ! -s 0.0.0.0/0 -d 0.0.0.0/0 -m comment --comment flanneld masq -j RETURN
I0519 09:27:35.222916       1 iptables.go:255] Deleting iptables rule: ! -s 0.0.0.0/0 -d 0.0.0.0/0 -m comment --comment flanneld masq -j MASQUERADE --random-fully
I0519 09:27:35.224107       1 main.go:332] Setting up masking rules
I0519 09:27:35.224864       1 main.go:353] Changing default FORWARD chain policy to ACCEPT
I0519 09:27:35.225222       1 main.go:366] Wrote subnet file to /run/flannel/subnet.env
I0519 09:27:35.225233       1 main.go:370] Running backend.
I0519 09:27:35.225251       1 main.go:391] Waiting for all goroutines to exit
I0519 09:27:35.225270       1 wireguard_network.go:74] Watching for new subnet leases

[manuelbuil]

Thanks for reporting this. Regarding wireguard backend, we need to update the docs.

Regarding the restart, HelmChartConfig is based on Helm. Helm will only restart what gets changed with the new config. In this case, what changes is the config and thus it does not restart everything automatically. Therefore, the manual restart of the daemonset, as you experienced, is needed

[manuelbuil]

And yes, you are right, it could be supported in some latest patch versions of 1.22. We wrote 1.23 directly because we thought it would be less confusing, as all patch versions of v1.23 support it. BTW, here is a doc update based on your report: #2959. You are welcome to review it :)