/etc/rancher/k3s directory does not exist, causing FileOrCreate hostpath mount for /etc/rancher/k3s/k3s.yaml to fail.

[andsens]

Is there an existing issue for this?

• I have searched the existing issues

What happened?

May 05 20:05:55 k8s-hyperv.local k3s[500]: E0505 20:05:55.541324     500 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/host-path/a3199835-e501-487b-aa5e-bea3986e7576-kube-config podName:a3199835-e501-487b-aa5e-bea3986e7576 nodeName:}" failed. No retries permitted until 2024-05-05 20:05:56.541315924 +0200 CEST m=+55.771315174 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-config" (UniqueName: "kubernetes.io/host-path/a3199835-e501-487b-aa5e-bea3986e7576-kube-config") pod "cilium-pzxwn" (UID: "a3199835-e501-487b-aa5e-bea3986e7576") : open /etc/rancher/k3s/k3s.yaml: no such file or directory
May 05 20:05:56 k8s-hyperv.local k3s[500]: E0505 20:05:56.547003     500 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/host-path/a3199835-e501-487b-aa5e-bea3986e7576-kube-config podName:a3199835-e501-487b-aa5e-bea3986e7576 nodeName:}" failed. No retries permitted until 2024-05-05 20:05:58.546991655 +0200 CEST m=+57.776990905 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-config" (UniqueName: "kubernetes.io/host-path/a3199835-e501-487b-aa5e-bea3986e7576-kube-config") pod "cilium-pzxwn" (UID: "a3199835-e501-487b-aa5e-bea3986e7576") : open /etc/rancher/k3s/k3s.yaml: no such file or directory
May 05 20:05:58 k8s-hyperv.local k3s[500]: E0505 20:05:58.558081     500 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/host-path/a3199835-e501-487b-aa5e-bea3986e7576-kube-config podName:a3199835-e501-487b-aa5e-bea3986e7576 nodeName:}" failed. No retries permitted until 2024-05-05 20:06:02.558070983 +0200 CEST m=+61.788070233 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-config" (UniqueName: "kubernetes.io/host-path/a3199835-e501-487b-aa5e-bea3986e7576-kube-config") pod "cilium-pzxwn" (UID: "a3199835-e501-487b-aa5e-bea3986e7576") : open /etc/rancher/k3s/k3s.yaml: no such file or directory
^C
k8s-hyperv% sudo mkdir /etc/rancher/k3s
[sudo] password for anders:
k8s-hyperv% journalctl -fu k3s-agent
May 05 20:05:54 k8s-hyperv.local k3s[500]: I0505 20:05:54.938108     500 reconciler_common.go:258] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3199835-e501-487b-aa5e-bea3986e7576-tmp\") pod \"cilium-pzxwn\" (UID: \"a3199835-e501-487b-aa5e-bea3986e7576\") " pod="kube-system/cilium-pzxwn"

The issue is in the helm chart and further up: https://github.com/cilium/cilium/blob/v1.15.4/install/kubernetes/cilium/templates/cilium-agent/daemonset.yaml#L841-L844

I don't quite understand how this isn't a problem on my master node. Do note that I am bootstrapping my nodes with INSTALL_K3S_SKIP_START=true and then spinning them up later.

Cilium Version

cilium-cli: v0.16.5 compiled with go1.22.2 on linux/amd64
cilium image (default): v1.15.4
cilium image (stable): v1.15.4
cilium image (running): unknown. Unable to obtain cilium version. Reason: Kubernetes cluster unreachable: Get "http://localhost:8080/version": dial tcp 127.0.0.1:8080: connect: connection refused

Kernel Version

Linux k8s-hyperv.local 6.1.0-20-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.85-1 (2024-04-11) x86_64 GNU/Linux

Kubernetes Version

Client Version: v1.29.4+k3s1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.4+k3s1

Regression

No response

Sysdump

No response

Relevant log output

No response

Anything else?

No response

Cilium Users Document

• Are you a user of Cilium? Please add yourself to the Users doc

Code of Conduct

• I agree to follow this project's Code of Conduct

[youngnick]

Thanks for this issue @andsens, but I'm not sure if Cilium can fix this for you. That setting sets where the kubeconfig is stored, and defaults to "~/.kube/config" as per

cilium/install/kubernetes/cilium/values.yaml

Line 36 in 9b3f9a8

# -- (string) Kubernetes config path

It seems like the method you are using to install Cilium on k3s is overriding that?

Per https://kubernetes.io/docs/concepts/storage/volumes/#hostpath-volume-types, the FileOrCreate mode does not create parent directories, sadly.

[andsens]

Agreed. Though I think it's more of a documentation issue in this case. The docs state that things should just work when spinning up k3s nodes, which is not true. A note regarding this would help a lot I think.
I also discovered that the default k8sServiceHost setting in helm is 127.0.0.1, which tripped me up even further. Adding a note telling the user to specify --set k8sServiceHost=<K8SAPI IP/HOSTNAME> if they intend to run multiple k3s nodes would help enormously.

[youngnick]

Thanks for that, yes, I agree that documentation updates would be ideal. As always, PRs are welcome (😄 ), otherwise I will assign this to the correct team and they will prioritize it accordingly.