[BUG] #1139

ashersilver · 2022-08-18T21:50:52Z

ashersilver
Aug 18, 2022

What did you do

Trying to get k3d cluster started, disable flannel and enable calico on k3s
(next step to install Istio and CN postgres once this is working. Need calico to permit greater consistency with k8s network policy)

How was the cluster created?

k3d cluster create my-cluster \
    --api-port 6550 \
    --servers 1 \
	--k3s-arg '--flannel-backend=none@server:*' \
    --volume "$(pwd)/calico.yaml:/var/lib/rancher/k3s/server/manifests/calico.yaml" \
    --agents=3 \
    --image rancher/k3s:v1.24.3-k3s1 \
    --port 443:443@loadbalancer \
    --wait --verbose

What did you do afterwards?
- k3d commands?
  kubectl get pods -n kube-system
- docker commands?
- OS operations (e.g. shutdown/reboot)?
  Restarted docker, rebooted, removed all old unused docker images, restarted, retried with 1 agent, no api-port etc.

What did you expect to happen

cluster to start with 1 server, 3 agents

Screenshots or terminal output

NAME                                       READY   STATUS                      RESTARTS   AGE
helm-install-traefik-crd-b92vr             0/1     ContainerCreating           0          17h
coredns-b96499967-sxdwv                    0/1     ContainerCreating           0          17h
calico-kube-controllers-555bc4b957-crwb4   0/1     ContainerCreating           0          17h
helm-install-traefik-74f72                 0/1     ContainerCreating           0          17h
local-path-provisioner-7b7dc8d6f5-zs2t7    0/1     ContainerCreating           0          17h
metrics-server-668d979685-sz5vx            0/1     ContainerCreating           0          17h
calico-node-v22rw                          0/1     Init:CreateContainerError   1          17h
calico-node-shbct                          0/1     Init:CreateContainerError   1          17h
calico-node-vxfv4                          0/1     Init:CreateContainerError   1          17h
calico-node-p4284                          0/1     Init:CreateContainerError   1          17h

Which OS & Architecture

macOS Monterrey 2020 16" MBP 16GB; Docker 4.11.1 with 9 GB, 3GB swap 6 processors

output of k3d runtime-info

arch: x86_64
cgroupdriver: cgroupfs
cgroupversion: "2"
endpoint: /var/run/docker.sock
filesystem: extfs
name: docker
os: Docker Desktop
ostype: linux
version: 20.10.17

Which version of `k3d`

1.24 k3s 1.24.3

output of k3d version

k3d version v5.4.4
k3s version v1.23.8-k3s1 (default)

Which version of docker

4.11.1 build 84025

output of docker version and docker info

Client:
 Cloud integration: v1.0.28
 Version:           20.10.17
 API version:       1.41
 Go version:        go1.17.11
 Git commit:        100c701
 Built:             Mon Jun  6 23:04:45 2022
 OS/Arch:           darwin/amd64
 Context:           default
 Experimental:      true

Server: Docker Desktop 4.11.1 (84025)
 Engine:
  Version:          20.10.17
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.17.11
  Git commit:       a89b842
  Built:            Mon Jun  6 23:01:23 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.6
  GitCommit:        10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
 runc:
  Version:          1.1.2
  GitCommit:        v1.1.2-0-ga916309
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.8.2)
  compose: Docker Compose (Docker Inc., v2.7.0)
  extension: Manages Docker extensions (Docker Inc., v0.2.8)
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc., 0.6.0)
  scan: Docker Scan (Docker Inc., v0.17.0)

Server:
 Containers: 2
  Running: 2
  Paused: 0
  Stopped: 0
 Images: 5
 Server Version: 20.10.17
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
 runc version: v1.1.2-0-ga916309
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.10.104-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 8.746GiB
 Name: docker-desktop
 ID: EQMW:E53W:T3Q4:FQYA:BR6P:AQWU:4CL5:LCS5:5SIH:ORXJ:DRWC:6I6Y
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5000
  127.0.0.0/8
 Live Restore Enabled: false

iwilltry42 · 2022-08-29T10:46:13Z

iwilltry42
Aug 29, 2022
Maintainer

Hi @ashersilver , thanks for starting this discussion!

Can you please provide some more info on why the calico-node pods are failing?
E.g. kubectl describe calico-node-v22rw?

5 replies

ashersilver Aug 31, 2022
Author

Ok I ran the following

kubectl describe pods calico-node-ktj25 -n kube-system:

The problem appears to be something related to shared volumes, - I saw something similar in another bug reported by another user here

Here's what I got its probably only the log entries at the end which are useful:

Name:                 calico-node-ktj25
Namespace:            kube-system
Priority:             2000001000
Priority Class Name:  system-node-critical
Node:                 k3d-my-cluster-agent-2/172.21.0.6
Start Time:           Tue, 30 Aug 2022 20:04:16 -0400
Labels:               controller-revision-hash=7d5459d554
                      k8s-app=calico-node
                      pod-template-generation=1
Annotations:          <none>
Status:               Pending
IP:                   172.21.0.6
IPs:
  IP:           172.21.0.6
Controlled By:  DaemonSet/calico-node
Init Containers:
  upgrade-ipam:
    Container ID:  containerd://d193125c61427bf763e0e9899315d34a969918aa017b7a3631482ae4e2711067
    Image:         docker.io/calico/cni:v3.24.0
    Image ID:      docker.io/calico/cni@sha256:3ed05c9504e4f7defa7eab60ac5594dd99460fea2a126ed650f6723c8c0a0193
    Port:          <none>
    Host Port:     <none>
    Command:
      /opt/cni/bin/calico-ipam
      -upgrade
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Tue, 30 Aug 2022 20:04:32 -0400
      Finished:     Tue, 30 Aug 2022 20:04:32 -0400
    Ready:          True
    Restart Count:  0
    Environment Variables from:
      kubernetes-services-endpoint  ConfigMap  Optional: true
    Environment:
      KUBERNETES_NODE_NAME:        (v1:spec.nodeName)
      CALICO_NETWORKING_BACKEND:  <set to the key 'calico_backend' of config map 'calico-config'>  Optional: false
    Mounts:
      /host/opt/cni/bin from cni-bin-dir (rw)
      /var/lib/cni/networks from host-local-net-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-zpc26 (ro)
  install-cni:
    Container ID:  containerd://6711954334e06ca1f4b4bad4056d4cbb2de23b0a446896a572286b442102a736
    Image:         docker.io/calico/cni:v3.24.0
    Image ID:      docker.io/calico/cni@sha256:3ed05c9504e4f7defa7eab60ac5594dd99460fea2a126ed650f6723c8c0a0193
    Port:          <none>
    Host Port:     <none>
    Command:
      /opt/cni/bin/install
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Tue, 30 Aug 2022 20:04:33 -0400
      Finished:     Tue, 30 Aug 2022 20:04:34 -0400
    Ready:          True
    Restart Count:  0
    Environment Variables from:
      kubernetes-services-endpoint  ConfigMap  Optional: true
    Environment:
      CNI_CONF_NAME:         10-calico.conflist
      CNI_NETWORK_CONFIG:    <set to the key 'cni_network_config' of config map 'calico-config'>  Optional: false
      KUBERNETES_NODE_NAME:   (v1:spec.nodeName)
      CNI_MTU:               <set to the key 'veth_mtu' of config map 'calico-config'>  Optional: false
      SLEEP:                 false
    Mounts:
      /host/etc/cni/net.d from cni-net-dir (rw)
      /host/opt/cni/bin from cni-bin-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-zpc26 (ro)
  mount-bpffs:
    Container ID:
    Image:         docker.io/calico/node:v3.24.0
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      calico-node
      -init
      -best-effort
    State:          Waiting
      Reason:       CreateContainerError
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /nodeproc from nodeproc (ro)
      /sys/fs from sys-fs (rw)
      /var/run/calico from var-run-calico (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-zpc26 (ro)
Containers:
  calico-node:
    Container ID:
    Image:          docker.io/calico/node:v3.24.0
    Image ID:
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:      250m
    Liveness:   exec [/bin/calico-node -felix-live -bird-live] delay=10s timeout=10s period=10s #success=1 #failure=6
    Readiness:  exec [/bin/calico-node -felix-ready -bird-ready] delay=0s timeout=10s period=10s #success=1 #failure=3
    Environment Variables from:
      kubernetes-services-endpoint  ConfigMap  Optional: true
    Environment:
      DATASTORE_TYPE:                     kubernetes
      WAIT_FOR_DATASTORE:                 true
      NODENAME:                            (v1:spec.nodeName)
      CALICO_NETWORKING_BACKEND:          <set to the key 'calico_backend' of config map 'calico-config'>  Optional: false
      CLUSTER_TYPE:                       k8s,bgp
      IP:                                 autodetect
      CALICO_IPV4POOL_IPIP:               Always
      CALICO_IPV4POOL_VXLAN:              Never
      CALICO_IPV6POOL_VXLAN:              Never
      FELIX_IPINIPMTU:                    <set to the key 'veth_mtu' of config map 'calico-config'>  Optional: false
      FELIX_VXLANMTU:                     <set to the key 'veth_mtu' of config map 'calico-config'>  Optional: false
      FELIX_WIREGUARDMTU:                 <set to the key 'veth_mtu' of config map 'calico-config'>  Optional: false
      CALICO_DISABLE_FILE_LOGGING:        true
      FELIX_DEFAULTENDPOINTTOHOSTACTION:  ACCEPT
      FELIX_IPV6SUPPORT:                  false
      FELIX_HEALTHENABLED:                true
    Mounts:
      /host/etc/cni/net.d from cni-net-dir (rw)
      /lib/modules from lib-modules (ro)
      /run/xtables.lock from xtables-lock (rw)
      /sys/fs/bpf from bpffs (rw)
      /var/lib/calico from var-lib-calico (rw)
      /var/log/calico/cni from cni-log-dir (ro)
      /var/run/calico from var-run-calico (rw)
      /var/run/nodeagent from policysync (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-zpc26 (ro)
Conditions:
  Type              Status
  Initialized       False
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  lib-modules:
    Type:          HostPath (bare host directory volume)
    Path:          /lib/modules
    HostPathType:
  var-run-calico:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/calico
    HostPathType:
  var-lib-calico:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/calico
    HostPathType:
  xtables-lock:
    Type:          HostPath (bare host directory volume)
    Path:          /run/xtables.lock
    HostPathType:  FileOrCreate
  sys-fs:
    Type:          HostPath (bare host directory volume)
    Path:          /sys/fs/
    HostPathType:  DirectoryOrCreate
  bpffs:
    Type:          HostPath (bare host directory volume)
    Path:          /sys/fs/bpf
    HostPathType:  Directory
  nodeproc:
    Type:          HostPath (bare host directory volume)
    Path:          /proc
    HostPathType:
  cni-bin-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /opt/cni/bin
    HostPathType:
  cni-net-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/cni/net.d
    HostPathType:
  cni-log-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/log/calico/cni
    HostPathType:
  host-local-net-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/cni/networks
    HostPathType:
  policysync:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/nodeagent
    HostPathType:  DirectoryOrCreate
  kube-api-access-zpc26:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 :NoSchedule op=Exists
                             :NoExecute op=Exists
                             CriticalAddonsOnly op=Exists
                             node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/network-unavailable:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists
                             node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                             node.kubernetes.io/unreachable:NoExecute op=Exists
                             node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  4m38s                 default-scheduler  Successfully assigned kube-system/calico-node-ktj25 to k3d-my-cluster-agent-2
  Normal   Pulling    4m36s                 kubelet            Pulling image "docker.io/calico/cni:v3.24.0"
  Normal   Pulled     4m22s                 kubelet            Successfully pulled image "docker.io/calico/cni:v3.24.0" in 14.465720615s
  Normal   Started    4m22s                 kubelet            Started container upgrade-ipam
  Normal   Created    4m22s                 kubelet            Created container upgrade-ipam
  Normal   Created    4m21s                 kubelet            Created container install-cni
  Normal   Pulled     4m21s                 kubelet            Container image "docker.io/calico/cni:v3.24.0" already present on machine
  Normal   Started    4m21s                 kubelet            Started container install-cni
  Normal   Pulling    4m19s                 kubelet            Pulling image "docker.io/calico/node:v3.24.0"
  Warning  Failed     4m9s                  kubelet            Error: failed to generate container "2a3dd748e92f2a3b179f412f8b07aaf3fbe8b203b5a569eb08cd8c8f9329e55c" spec: failed to generate spec: path "/sys/fs/" is mounted on "/sys" but it is not a shared mount
  Warning  Failed     4m9s                  kubelet            Error: failed to generate container "8550590fcd93cf46380aa24d24864f7513de32cdbcac6fad173234864611d9e9" spec: failed to generate spec: path "/sys/fs/" is mounted on "/sys" but it is not a shared mount
  Normal   Pulled     4m9s                  kubelet            Successfully pulled image "docker.io/calico/node:v3.24.0" in 9.279468774s
  Warning  Failed     3m57s                 kubelet            Error: failed to generate container "fb3e8677cd528b8c22d667c4043b46b32b228f655569f018e3bf640d85bdd164" spec: failed to generate spec: path "/sys/fs/" is mounted on "/sys" but it is not a shared mount
  Warning  Failed     3m42s                 kubelet            Error: failed to generate container "7326693341b0320f045ff360ce5f4204dd40924abeeb516f21fbb418b9a000a3" spec: failed to generate spec: path "/sys/fs/" is mounted on "/sys" but it is not a shared mount
  Warning  Failed     3m27s                 kubelet            Error: failed to generate container "5e71d3caf8de2cb7ff1de931c5e99d6325749452ae2ef5eada67ac3d623851c3" spec: failed to generate spec: path "/sys/fs/" is mounted on "/sys" but it is not a shared mount
  Warning  Failed     3m14s                 kubelet            Error: failed to generate container "8ded6f9e502e5f54cc557e06dcc4743b48d03c29e7e41d1412a6e24617c0a0f4" spec: failed to generate spec: path "/sys/fs/" is mounted on "/sys" but it is not a shared mount
  Warning  Failed     2m59s                 kubelet            Error: failed to generate container "c1f29d93c31f19c7ea3d0b0f0075b1a56de043808268cf97f75572fa8495f355" spec: failed to generate spec: path "/sys/fs/" is mounted on "/sys" but it is not a shared mount
  Warning  Failed     2m45s                 kubelet            Error: failed to generate container "c0a922786711bfcfa65f6f1b33ea1cb386077f6b0cf69136855029d484704f39" spec: failed to generate spec: path "/sys/fs/" is mounted on "/sys" but it is not a shared mount
  Normal   Pulled     2m33s (x8 over 4m9s)  kubelet            Container image "docker.io/calico/node:v3.24.0" already present on machine
`

iwilltry42 Aug 31, 2022
Maintainer

Thank you!
I was just wondering, what changed, since we have a frequently-used guide on using Calico here.
I searched the manifest linked there for the troublemaking sysfs mount and couldn't find it. Then I realized it's coming from the mount-bpffs container. Are you trying to run Calico in eBPF mode maybe?

iwilltry42 Aug 31, 2022
Maintainer

Maybe this helps? #479

ashersilver Aug 31, 2022
Author

Thanks I tried that, but it goes deeper .. that issue wi the sharing is /sys/fs and I tried the code

docker exec -it k3d-my-cluster-agent-2 mount bpffs /sys/fs/bpf -t bpf
docker exec -it k3d-my-cluster-agent-2 mount --make-shared /sys/fs/bpf

when I restart the container in Docker, and I log in via a shell using docker exec -it, it appears to succeed in sharing the /sys/fs/bpf folder r-x for all but the issue I am experiencing is actually for /sys/fs not being shared above.

I don't know cilium, but ultimately I am installing k3s with calico to be able to work with Istio ..

I have a horrible feeling it comes down to incompatibilities with MacOS and linux and the lack of support for servicectl (linux) in macOS (launchctl).

restarted all pods in sight after the new commands above... seems some progress...

Warning  Failed          3m43s                 kubelet  Error: failed to generate container "841b9ddabec46f4b19621988aa1674d534f757f5f45f1034112ece310b0e5299" spec: failed to generate spec: path "/sys/fs/" is mounted on "/sys" but it is not a shared mount
  Normal   Pulled          15s (x25 over 5m18s)  kubelet  Container image "docker.io/calico/node:v3.24.0" already present on machine

I suspect this might be easier on Ubuntu 20.4

GabeBolton Sep 15, 2022

If you're still working on this I've been running into this issue too.

I haven't fully solved it but I've gotten past it failing on path "/sys/fs/" is mounted on "/sys" but it is not a shared mount by adding --volume "/sys/fs/:/sys/fs/:shared@server:*" to the k3d command.

I am working on Ubuntu 22.04.1, not macOS, but I think that this problem is distinct from the /sys/fs/bpf requirement that #479 deals with, as you've said calico $\neq$ cilium.

Note that with podman I experienced some kubectl commands hanging which cleared up when I switched to docker.

Now the pod hangs at Init:0/3 with the event

  Warning  FailedCreatePodSandBox  3s (x9 over 109s)  kubelet            (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: unable to apply cgroup configuration: failed to write 1427: write /sys/fs/cgroup/kubepods/burstable/pod3f4b0971-e0c3-45b5-be01-62b97f7dfed5/4a0866b30209bf72a2c66c02ad5ca9ff2ee3ff0e336ea7a1219216a94a0fe722/cgroup.procs: no such file or directory: unknown

with changing pod ids each time. Do you get that error as well?

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] #1139

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment 5 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

[BUG] #1139

ashersilver Aug 18, 2022

What did you do

What did you expect to happen

Screenshots or terminal output

Which OS & Architecture

Which version of k3d

Which version of docker

Replies: 1 comment · 5 replies

iwilltry42 Aug 29, 2022 Maintainer

ashersilver Aug 31, 2022 Author

iwilltry42 Aug 31, 2022 Maintainer

iwilltry42 Aug 31, 2022 Maintainer

ashersilver Aug 31, 2022 Author

GabeBolton Sep 15, 2022

ashersilver
Aug 18, 2022

Which version of `k3d`

Replies: 1 comment 5 replies

iwilltry42
Aug 29, 2022
Maintainer

ashersilver Aug 31, 2022
Author

iwilltry42 Aug 31, 2022
Maintainer

iwilltry42 Aug 31, 2022
Maintainer

ashersilver Aug 31, 2022
Author