[support] Trouble getting started with the example docker-compose.yml file #3324

fubarhouse · 2021-05-13T00:05:12Z

fubarhouse
May 13, 2021

I'm having a lot of difficulty getting k3s running inside of docker-compose... Interestingly I can start a regular cluster on bare metal via kubeadm. Unfortunately I need some assistance because it's only refusing to run from inside the docker. I'll also note that I did fix this problem previously with the kernel parameter systemd.unified_cgroup_hierachy=false, however having been forced to reinstall the operating system, this appears to be doing nothing which is apparent from the container... (even though I can see the cgroups are not unified...

I've been trying to use this file:
https://github.com/k3s-io/k3s/blob/master/docker-compose.yml

Here's a bunch of information that could help.

$ neofetch

                   -`                    karl@karl 
                  .o+`                   --------- 
                 `ooo/                   OS: Arch Linux x86_64 
                `+oooo:                  Host: AORUS 15G YC 
               `+oooooo:                 Kernel: 5.12.2-arch1-1 
               -+oooooo+:                Uptime: 21 mins 
             `/:-:++oooo+:               Packages: 1476 (pacman) 
            `/++++/+++++++:              Shell: bash 5.1.4 
           `/++++++++++++++:             Resolution: 1920x1080 
          `/+++ooooooooooooo/`           DE: Plasma 5.21.5 
         ./ooosssso++osssssso+`          WM: bspwm 
        .oossssso-````/ossssss+`         Theme: Breeze Light [Plasma], Breeze [GTK2/3] 
       -osssssso.      :ssssssso.        Icons: Sweet-Mars [Plasma], Sweet-Mars [GTK2/3] 
      :osssssss/        osssso+++.       Terminal: termite 
     /ossssssss/        +ssssooo/-       Terminal Font: Monospace 9 
   `/ossssso+/:-        -:/+osssso+-     CPU: Intel i7-10870H (16) @ 5.000GHz 
  `+sso+:-`                 `.-/+oso:    GPU: NVIDIA GeForce RTX 3070 Mobile / Max-Q 8GB/16GB 
 `++:.                           `-/+/   GPU: Intel UHD Graphics 
 .`                                 `/   Memory: 5784MiB / 64183MiB

$ cat /etc/default/grub

# GRUB boot loader configuration

GRUB_DEFAULT="0"
GRUB_TIMEOUT="5"
GRUB_DISTRIBUTOR="Arch"
GRUB_CMDLINE_LINUX=""
GRUB_CMDLINE_LINUX_DEFAULT="loglevel=3 quiet systemd.unified_cgroup_hierachy=false systemd.legacy_systemd_cgroup_controller=1"

# Preload both GPT and MBR modules so that they are not missed
GRUB_PRELOAD_MODULES="part_gpt part_msdos"

# Uncomment to enable booting from LUKS encrypted devices
#GRUB_ENABLE_CRYPTODISK="y"

# Set to 'countdown' or 'hidden' to change timeout behavior,
# press ESC key to display menu.
GRUB_TIMEOUT_STYLE="menu"

# Uncomment to use basic console
GRUB_TERMINAL_INPUT="console"

# Uncomment to disable graphical terminal
#GRUB_TERMINAL_OUTPUT="console"

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
GRUB_GFXMODE="auto"

# Uncomment to allow the kernel use the same resolution used by grub
GRUB_GFXPAYLOAD_LINUX="keep"

# Uncomment if you want GRUB to pass to the Linux kernel the old parameter
# format "root=/dev/xxx" instead of "root=/dev/disk/by-uuid/xxx"
#GRUB_DISABLE_LINUX_UUID="true"

# Uncomment to disable generation of recovery mode menu entries
GRUB_DISABLE_RECOVERY="true"

# Uncomment and set to the desired menu colors.  Used by normal and wallpaper
# modes only.  Entries specified as foreground/background.
#GRUB_COLOR_NORMAL="light-blue/black"
#GRUB_COLOR_HIGHLIGHT="light-cyan/blue"

# Uncomment one of them for the gfx desired, a image background or a gfxtheme
#GRUB_BACKGROUND="/path/to/wallpaper"
#GRUB_THEME="/path/to/gfxtheme"

# Uncomment to get a beep at GRUB start
#GRUB_INIT_TUNE="480 440 1"

# Uncomment to make GRUB remember the last selection. This requires
# setting 'GRUB_DEFAULT=saved' above.
#GRUB_SAVEDEFAULT="true"

# Uncomment to disable submenus in boot menu
#GRUB_DISABLE_SUBMENU="y"

$ docker info

~ $ docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-tp-docker)

Server:
 Containers: 4
  Running: 0
  Paused: 0
  Stopped: 4
 Images: 77
 Server Version: 20.10.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: false
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8c906ff108ac28da23f69cc7b74f8e7a470d1df0.m
 runc version: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.12.2-arch1-1
 Operating System: Arch Linux
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 62.68GiB
 Name: karl
 ID: 6JDJ:I7MZ:IGCC:5FE2:3Z7K:ZS4L:ZITH:OZDM:XRUC:4RVJ:QODP:76FM
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: fubarhouse
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No kernel memory limit support
WARNING: No oom kill disable support

$ ll /sys/fs/cgroup/

total 0
drwxr-xr-x  2 root root 0 2021-05-13 09:42 -.mount
dr-xr-xr-x 19 root root 0 2021-05-13 09:42 .
drwxr-xr-x  7 root root 0 2021-05-13 09:27 ..
-r--r--r--  1 root root 0 2021-05-13 09:27 cgroup.controllers
-rw-r--r--  1 root root 0 2021-05-13 10:09 cgroup.max.depth
-rw-r--r--  1 root root 0 2021-05-13 10:09 cgroup.max.descendants
-rw-r--r--  1 root root 0 2021-05-13 09:27 cgroup.procs
-r--r--r--  1 root root 0 2021-05-13 10:09 cgroup.stat
-rw-r--r--  1 root root 0 2021-05-13 09:42 cgroup.subtree_control
-rw-r--r--  1 root root 0 2021-05-13 10:09 cgroup.threads
-rw-r--r--  1 root root 0 2021-05-13 10:09 cpu.pressure
-r--r--r--  1 root root 0 2021-05-13 10:09 cpu.stat
-r--r--r--  1 root root 0 2021-05-13 09:27 cpuset.cpus.effective
-r--r--r--  1 root root 0 2021-05-13 09:27 cpuset.mems.effective
drwxr-xr-x  2 root root 0 2021-05-13 09:28 dev-hugepages.mount
drwxr-xr-x  2 root root 0 2021-05-13 09:28 dev-mqueue.mount
drwxr-xr-x  2 root root 0 2021-05-13 09:42 init.scope
-rw-r--r--  1 root root 0 2021-05-13 10:09 io.cost.model
-rw-r--r--  1 root root 0 2021-05-13 10:09 io.cost.qos
-rw-r--r--  1 root root 0 2021-05-13 10:09 io.pressure
-r--r--r--  1 root root 0 2021-05-13 10:09 io.stat
-r--r--r--  1 root root 0 2021-05-13 10:09 memory.numa_stat
-rw-r--r--  1 root root 0 2021-05-13 10:09 memory.pressure
-r--r--r--  1 root root 0 2021-05-13 10:09 memory.stat
drwxr-xr-x  2 root root 0 2021-05-13 09:28 proc-sys-fs-binfmt_misc.mount
drwxr-xr-x  2 root root 0 2021-05-13 09:28 sys-fs-fuse-connections.mount
drwxr-xr-x  2 root root 0 2021-05-13 09:28 sys-kernel-config.mount
drwxr-xr-x  2 root root 0 2021-05-13 09:28 sys-kernel-debug.mount
drwxr-xr-x  2 root root 0 2021-05-13 09:28 sys-kernel-tracing.mount
drwxr-xr-x 58 root root 0 2021-05-13 09:57 system.slice
drwxr-xr-x  2 root root 0 2021-05-13 09:42 tmp-.mount_JoplinrPC5xO.mount
drwxr-xr-x  3 root root 0 2021-05-13 09:28 user.slice
drwxr-xr-x  2 root root 0 2021-05-13 09:42 var-lib-docker-overlay2-358067c6bd59448898698f9ce0a3becd062634c4ad5eed3a4acf1f94dfc191ae-merged.mount
drwxr-xr-x  2 root root 0 2021-05-13 09:42 var-lib-docker-overlay2-91e846613acf71b322a1f0897e07ed2bee02ddaaaffea53affa1a500ec634951-merged.mount
drwxr-xr-x  2 root root 0 2021-05-13 09:42 var-lib-docker-overlay2-c08917dc9cd48ca3923d6512240d2d15b14f5fa1a241b6dd268dab88c76df1f9-merged.mount
drwxr-xr-x  2 root root 0 2021-05-13 09:42 var-lib-docker-overlay2-dc72cc753da4bd5321445980964ea3e3a6c9721dfc24b39b413838ff908cb5db-merged.mount
drwxr-xr-x  2 root root 0 2021-05-13 09:42 var-lib-docker-overlay2-e8e279ab0551ddf02b1a1518591a83d7310fcc58ba6728addeb6f0b9ae74c921-merged.mount

$ docker-compose up -d; docker-compose logs -f server

Attaching to k3s_server_1
�[36mserver_1  |�[0m time="2021-05-12T23:31:31.940371057Z" level=info msg="Starting k3s v1.21.0+k3s1 (2705431d)"
�[36mserver_1  |�[0m time="2021-05-12T23:31:31.947839634Z" level=info msg="Configuring sqlite3 database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"
�[36mserver_1  |�[0m time="2021-05-12T23:31:31.947857983Z" level=info msg="Configuring database table schema and indexes, this may take a moment..."
�[36mserver_1  |�[0m time="2021-05-12T23:31:31.951173452Z" level=info msg="Database tables and indexes are up to date"
�[36mserver_1  |�[0m time="2021-05-12T23:31:31.951830630Z" level=info msg="Kine listening on unix://kine.sock"
�[36mserver_1  |�[0m time="2021-05-12T23:31:31.961001318Z" level=info msg="certificate CN=system:admin,O=system:masters signed by CN=k3s-client-ca@1620862291: notBefore=2021-05-12 23:31:31 +0000 UTC notAfter=2022-05-12 23:31:31 +0000 UTC"
�[36mserver_1  |�[0m time="2021-05-12T23:31:31.961581662Z" level=info msg="certificate CN=system:kube-controller-manager signed by CN=k3s-client-ca@1620862291: notBefore=2021-05-12 23:31:31 +0000 UTC notAfter=2022-05-12 23:31:31 +0000 UTC"
�[36mserver_1  |�[0m time="2021-05-12T23:31:31.962187995Z" level=info msg="certificate CN=system:kube-scheduler signed by CN=k3s-client-ca@1620862291: notBefore=2021-05-12 23:31:31 +0000 UTC notAfter=2022-05-12 23:31:31 +0000 UTC"
�[36mserver_1  |�[0m time="2021-05-12T23:31:31.962793860Z" level=info msg="certificate CN=kube-apiserver signed by CN=k3s-client-ca@1620862291: notBefore=2021-05-12 23:31:31 +0000 UTC notAfter=2022-05-12 23:31:31 +0000 UTC"
�[36mserver_1  |�[0m time="2021-05-12T23:31:31.963410863Z" level=info msg="certificate CN=system:kube-proxy signed by CN=k3s-client-ca@1620862291: notBefore=2021-05-12 23:31:31 +0000 UTC notAfter=2022-05-12 23:31:31 +0000 UTC"
�[36mserver_1  |�[0m time="2021-05-12T23:31:31.963912304Z" level=info msg="certificate CN=system:k3s-controller signed by CN=k3s-client-ca@1620862291: notBefore=2021-05-12 23:31:31 +0000 UTC notAfter=2022-05-12 23:31:31 +0000 UTC"
�[36mserver_1  |�[0m time="2021-05-12T23:31:31.964484418Z" level=info msg="certificate CN=cloud-controller-manager signed by CN=k3s-client-ca@1620862291: notBefore=2021-05-12 23:31:31 +0000 UTC notAfter=2022-05-12 23:31:31 +0000 UTC"
�[36mserver_1  |�[0m time="2021-05-12T23:31:31.965262406Z" level=info msg="certificate CN=kube-apiserver signed by CN=k3s-server-ca@1620862291: notBefore=2021-05-12 23:31:31 +0000 UTC notAfter=2022-05-12 23:31:31 +0000 UTC"
�[36mserver_1  |�[0m time="2021-05-12T23:31:31.966063041Z" level=info msg="certificate CN=system:auth-proxy signed by CN=k3s-request-header-ca@1620862291: notBefore=2021-05-12 23:31:31 +0000 UTC notAfter=2022-05-12 23:31:31 +0000 UTC"
�[36mserver_1  |�[0m time="2021-05-12T23:31:31.966842634Z" level=info msg="certificate CN=etcd-server signed by CN=etcd-server-ca@1620862291: notBefore=2021-05-12 23:31:31 +0000 UTC notAfter=2022-05-12 23:31:31 +0000 UTC"
�[36mserver_1  |�[0m time="2021-05-12T23:31:31.967277089Z" level=info msg="certificate CN=etcd-client signed by CN=etcd-server-ca@1620862291: notBefore=2021-05-12 23:31:31 +0000 UTC notAfter=2022-05-12 23:31:31 +0000 UTC"
�[36mserver_1  |�[0m time="2021-05-12T23:31:31.968057467Z" level=info msg="certificate CN=etcd-peer signed by CN=etcd-peer-ca@1620862291: notBefore=2021-05-12 23:31:31 +0000 UTC notAfter=2022-05-12 23:31:31 +0000 UTC"
�[36mserver_1  |�[0m time="2021-05-12T23:31:32.053322187Z" level=info msg="certificate CN=k3s,O=k3s signed by CN=k3s-server-ca@1620862291: notBefore=2021-05-12 23:31:31 +0000 UTC notAfter=2022-05-12 23:31:32 +0000 UTC"
�[36mserver_1  |�[0m time="2021-05-12T23:31:32.053663116Z" level=info msg="Active TLS secret  (ver=) (count 7): map[listener.cattle.io/cn-10.43.0.1:10.43.0.1 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-172.18.0.2:172.18.0.2 listener.cattle.io/cn-kubernetes:kubernetes listener.cattle.io/cn-kubernetes.default:kubernetes.default listener.cattle.io/cn-kubernetes.default.svc.cluster.local:kubernetes.default.svc.cluster.local listener.cattle.io/cn-localhost:localhost listener.cattle.io/fingerprint:SHA1=C07AD29EEDD589A181CE66297B7B89A304594043]"
�[36mserver_1  |�[0m time="2021-05-12T23:31:32.057008279Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --enable-admission-plugins=NodeRestriction --etcd-servers=unix://kine.sock --insecure-port=0 --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key"
�[36mserver_1  |�[0m Flag --insecure-port has been deprecated, This flag has no effect now and will be removed in v1.24.
�[36mserver_1  |�[0m I0512 23:31:32.058128       1 server.go:656] external host was not specified, using 172.18.0.2
�[36mserver_1  |�[0m I0512 23:31:32.058424       1 server.go:195] Version: v1.21.0+k3s1
�[36mserver_1  |�[0m I0512 23:31:32.401670       1 shared_informer.go:240] Waiting for caches to sync for node_authorizer
�[36mserver_1  |�[0m I0512 23:31:32.402460       1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
�[36mserver_1  |�[0m I0512 23:31:32.402468       1 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
�[36mserver_1  |�[0m I0512 23:31:32.403262       1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
�[36mserver_1  |�[0m I0512 23:31:32.403288       1 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
�[36mserver_1  |�[0m I0512 23:31:32.416685       1 instance.go:283] Using reconciler: lease
�[36mserver_1  |�[0m I0512 23:31:32.465637       1 rest.go:130] the default service ipfamily for this cluster is: IPv4
�[36mserver_1  |�[0m W0512 23:31:32.690688       1 genericapiserver.go:425] Skipping API node.k8s.io/v1alpha1 because it has no resources.
�[36mserver_1  |�[0m W0512 23:31:32.697341       1 genericapiserver.go:425] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
�[36mserver_1  |�[0m W0512 23:31:32.699889       1 genericapiserver.go:425] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.
�[36mserver_1  |�[0m W0512 23:31:32.703947       1 genericapiserver.go:425] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
�[36mserver_1  |�[0m W0512 23:31:32.705690       1 genericapiserver.go:425] Skipping API flowcontrol.apiserver.k8s.io/v1alpha1 because it has no resources.
�[36mserver_1  |�[0m W0512 23:31:32.708905       1 genericapiserver.go:425] Skipping API apps/v1beta2 because it has no resources.
�[36mserver_1  |�[0m W0512 23:31:32.708911       1 genericapiserver.go:425] Skipping API apps/v1beta1 because it has no resources.
�[36mserver_1  |�[0m I0512 23:31:32.722017       1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
�[36mserver_1  |�[0m I0512 23:31:32.722032       1 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
�[36mserver_1  |�[0m time="2021-05-12T23:31:32.730963928Z" level=info msg="Running kube-scheduler --address=127.0.0.1 --bind-address=127.0.0.1 --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --leader-elect=false --port=10251 --profiling=false --secure-port=0"
�[36mserver_1  |�[0m time="2021-05-12T23:31:32.731069847Z" level=info msg="Waiting for API server to become available"
�[36mserver_1  |�[0m time="2021-05-12T23:31:32.731621274Z" level=info msg="Running kube-controller-manager --address=127.0.0.1 --allocate-node-cidrs=true --bind-address=127.0.0.1 --cluster-cidr=10.42.0.0/16 --cluster-signing-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --leader-elect=false --port=10252 --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=0 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --use-service-account-credentials=true"
�[36mserver_1  |�[0m time="2021-05-12T23:31:32.732863271Z" level=info msg="Node token is available at /var/lib/rancher/k3s/server/token"
�[36mserver_1  |�[0m time="2021-05-12T23:31:32.732917846Z" level=info msg="To join node to cluster: k3s agent -s https://172.18.0.2:6443 -t ${NODE_TOKEN}"
�[36mserver_1  |�[0m time="2021-05-12T23:31:32.734040303Z" level=info msg="Wrote kubeconfig /output/kubeconfig.yaml"
�[36mserver_1  |�[0m time="2021-05-12T23:31:32.734228307Z" level=info msg="Run: k3s kubectl"
�[36mserver_1  |�[0m time="2021-05-12T23:31:32.743497922Z" level=info msg="Cluster-Http-Server 2021/05/12 23:31:32 http: TLS handshake error from 127.0.0.1:51914: remote error: tls: bad certificate"
�[36mserver_1  |�[0m time="2021-05-12T23:31:32.747025701Z" level=info msg="Cluster-Http-Server 2021/05/12 23:31:32 http: TLS handshake error from 127.0.0.1:51920: remote error: tls: bad certificate"
�[36mserver_1  |�[0m time="2021-05-12T23:31:32.754365960Z" level=info msg="certificate CN=cc9d29cc51b1 signed by CN=k3s-server-ca@1620862291: notBefore=2021-05-12 23:31:31 +0000 UTC notAfter=2022-05-12 23:31:32 +0000 UTC"
�[36mserver_1  |�[0m time="2021-05-12T23:31:32.756501133Z" level=info msg="certificate CN=system:node:cc9d29cc51b1,O=system:nodes signed by CN=k3s-client-ca@1620862291: notBefore=2021-05-12 23:31:31 +0000 UTC notAfter=2022-05-12 23:31:32 +0000 UTC"
�[36mserver_1  |�[0m time="2021-05-12T23:31:32.767396682Z" level=info msg="Module overlay was already loaded"
�[36mserver_1  |�[0m time="2021-05-12T23:31:32.767439044Z" level=info msg="Module nf_conntrack was already loaded"
�[36mserver_1  |�[0m time="2021-05-12T23:31:32.767447033Z" level=info msg="Module br_netfilter was already loaded"
�[36mserver_1  |�[0m time="2021-05-12T23:31:32.767456942Z" level=info msg="Module iptable_nat was already loaded"
�[36mserver_1  |�[0m time="2021-05-12T23:31:32.768294820Z" level=info msg="Logging containerd to /var/lib/rancher/k3s/agent/containerd/containerd.log"
�[36mserver_1  |�[0m time="2021-05-12T23:31:32.768456383Z" level=info msg="Running containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd"
�[36mserver_1  |�[0m time="2021-05-12T23:31:33.770294744Z" level=info msg="Containerd is now running"
�[36mserver_1  |�[0m time="2021-05-12T23:31:33.776035214Z" level=info msg="Connecting to proxy" url="wss://127.0.0.1:6443/v1-k3s/connect"
�[36mserver_1  |�[0m time="2021-05-12T23:31:33.777637160Z" level=info msg="Handling backend connection request [cc9d29cc51b1]"
�[36mserver_1  |�[0m time="2021-05-12T23:31:33.778301643Z" level=info msg="Running kubelet --address=0.0.0.0 --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=cgroupfs --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --cni-bin-dir=/bin --cni-conf-dir=/var/lib/rancher/k3s/agent/etc/cni/net.d --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --container-runtime=remote --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=cc9d29cc51b1 --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --node-labels= --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --read-only-port=0 --resolv-conf=/tmp/k3s-resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key"
�[36mserver_1  |�[0m time="2021-05-12T23:31:33.779001258Z" level=info msg="Running kube-proxy --cluster-cidr=10.42.0.0/16 --healthz-bind-address=127.0.0.1 --hostname-override=cc9d29cc51b1 --kubeconfig=/var/lib/rancher/k3s/agent/kubeproxy.kubeconfig --proxy-mode=iptables"
�[36mserver_1  |�[0m Flag --cloud-provider has been deprecated, will be removed in 1.23, in favor of removing cloud provider code from Kubelet.
�[36mserver_1  |�[0m Flag --cni-bin-dir has been deprecated, will be removed along with dockershim.
�[36mserver_1  |�[0m Flag --cni-conf-dir has been deprecated, will be removed along with dockershim.
�[36mserver_1  |�[0m Flag --containerd has been deprecated, This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.
�[36mserver_1  |�[0m W0512 23:31:33.779511       1 server.go:220] WARNING: all flags other than --config, --write-config-to, and --cleanup are deprecated. Please begin using a config file ASAP.
�[36mserver_1  |�[0m I0512 23:31:33.779896       1 server.go:436] "Kubelet version" kubeletVersion="v1.21.0+k3s1"
�[36mserver_1  |�[0m W0512 23:31:33.780282       1 proxier.go:653] Failed to read file /lib/modules/5.12.2-arch1-1/modules.builtin with error open /lib/modules/5.12.2-arch1-1/modules.builtin: no such file or directory. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
�[36mserver_1  |�[0m W0512 23:31:33.780779       1 proxier.go:663] Failed to load kernel module ip_vs with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
�[36mserver_1  |�[0m W0512 23:31:33.781134       1 proxier.go:663] Failed to load kernel module ip_vs_rr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
�[36mserver_1  |�[0m W0512 23:31:33.781406       1 proxier.go:663] Failed to load kernel module ip_vs_wrr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
�[36mserver_1  |�[0m W0512 23:31:33.781703       1 proxier.go:663] Failed to load kernel module ip_vs_sh with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
�[36mserver_1  |�[0m W0512 23:31:33.782015       1 proxier.go:663] Failed to load kernel module nf_conntrack with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
�[36mserver_1  |�[0m E0512 23:31:33.786343       1 node.go:161] Failed to retrieve node info: nodes "cc9d29cc51b1" is forbidden: User "system:kube-proxy" cannot get resource "nodes" in API group "" at the cluster scope
�[36mserver_1  |�[0m W0512 23:31:33.795902       1 manager.go:159] Cannot detect current cgroup on cgroup v2
�[36mserver_1  |�[0m I0512 23:31:33.795942       1 dynamic_cafile_content.go:167] Starting client-ca-bundle::/var/lib/rancher/k3s/agent/client-ca.crt
�[36mserver_1  |�[0m time="2021-05-12T23:31:33.945021545Z" level=info msg="Cluster-Http-Server 2021/05/12 23:31:33 http: TLS handshake error from 172.18.0.3:44914: remote error: tls: bad certificate"
�[36mserver_1  |�[0m time="2021-05-12T23:31:33.955969614Z" level=info msg="Cluster-Http-Server 2021/05/12 23:31:33 http: TLS handshake error from 172.18.0.3:44920: remote error: tls: bad certificate"
�[36mserver_1  |�[0m time="2021-05-12T23:31:33.965737514Z" level=error msg="runtime core not ready"
�[36mserver_1  |�[0m I0512 23:31:34.147815       1 dynamic_cafile_content.go:167] Starting request-header::/var/lib/rancher/k3s/server/tls/request-header-ca.crt
�[36mserver_1  |�[0m I0512 23:31:34.147816       1 dynamic_cafile_content.go:167] Starting client-ca-bundle::/var/lib/rancher/k3s/server/tls/client-ca.crt
�[36mserver_1  |�[0m I0512 23:31:34.148155       1 dynamic_serving_content.go:130] Starting serving-cert::/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt::/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key
�[36mserver_1  |�[0m I0512 23:31:34.148635       1 secure_serving.go:197] Serving securely on 127.0.0.1:6444
�[36mserver_1  |�[0m I0512 23:31:34.148851       1 apiservice_controller.go:97] Starting APIServiceRegistrationController
�[36mserver_1  |�[0m I0512 23:31:34.148884       1 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller
�[36mserver_1  |�[0m I0512 23:31:34.148914       1 tlsconfig.go:240] Starting DynamicServingCertificateController
�[36mserver_1  |�[0m I0512 23:31:34.148986       1 controller.go:83] Starting OpenAPI AggregationController
�[36mserver_1  |�[0m I0512 23:31:34.149094       1 available_controller.go:475] Starting AvailableConditionController
�[36mserver_1  |�[0m I0512 23:31:34.149122       1 cache.go:32] Waiting for caches to sync for AvailableConditionController controller
�[36mserver_1  |�[0m I0512 23:31:34.149175       1 autoregister_controller.go:141] Starting autoregister controller
�[36mserver_1  |�[0m I0512 23:31:34.149194       1 cache.go:32] Waiting for caches to sync for autoregister controller
�[36mserver_1  |�[0m I0512 23:31:34.149244       1 apf_controller.go:294] Starting API Priority and Fairness config controller
�[36mserver_1  |�[0m I0512 23:31:34.149351       1 customresource_discovery_controller.go:209] Starting DiscoveryController
�[36mserver_1  |�[0m I0512 23:31:34.149127       1 dynamic_serving_content.go:130] Starting aggregator-proxy-cert::/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt::/var/lib/rancher/k3s/server/tls/client-auth-proxy.key
�[36mserver_1  |�[0m I0512 23:31:34.149403       1 crdregistration_controller.go:111] Starting crd-autoregister controller
�[36mserver_1  |�[0m I0512 23:31:34.149412       1 shared_informer.go:240] Waiting for caches to sync for crd-autoregister
�[36mserver_1  |�[0m I0512 23:31:34.149759       1 apiapproval_controller.go:186] Starting KubernetesAPIApprovalPolicyConformantConditionController
�[36mserver_1  |�[0m I0512 23:31:34.149796       1 controller.go:86] Starting OpenAPI controller
�[36mserver_1  |�[0m I0512 23:31:34.149817       1 naming_controller.go:291] Starting NamingConditionController
�[36mserver_1  |�[0m I0512 23:31:34.149835       1 establishing_controller.go:76] Starting EstablishingController
�[36mserver_1  |�[0m I0512 23:31:34.149847       1 nonstructuralschema_controller.go:192] Starting NonStructuralSchemaConditionController
�[36mserver_1  |�[0m I0512 23:31:34.149911       1 crd_finalizer.go:266] Starting CRDFinalizer
�[36mserver_1  |�[0m I0512 23:31:34.149990       1 cluster_authentication_trust_controller.go:440] Starting cluster_authentication_trust_controller controller
�[36mserver_1  |�[0m I0512 23:31:34.149998       1 shared_informer.go:240] Waiting for caches to sync for cluster_authentication_trust_controller
�[36mserver_1  |�[0m I0512 23:31:34.150029       1 dynamic_cafile_content.go:167] Starting client-ca-bundle::/var/lib/rancher/k3s/server/tls/client-ca.crt
�[36mserver_1  |�[0m I0512 23:31:34.152212       1 dynamic_cafile_content.go:167] Starting request-header::/var/lib/rancher/k3s/server/tls/request-header-ca.crt
�[36mserver_1  |�[0m time="2021-05-12T23:31:34.158404656Z" level=info msg="Waiting for cloudcontroller rbac role to be created"
�[36mserver_1  |�[0m I0512 23:31:34.202731       1 shared_informer.go:247] Caches are synced for node_authorizer 
�[36mserver_1  |�[0m I0512 23:31:34.249879       1 cache.go:39] Caches are synced for APIServiceRegistrationController controller
�[36mserver_1  |�[0m I0512 23:31:34.249968       1 cache.go:39] Caches are synced for autoregister controller
�[36mserver_1  |�[0m I0512 23:31:34.249975       1 shared_informer.go:247] Caches are synced for crd-autoregister 
�[36mserver_1  |�[0m I0512 23:31:34.250004       1 apf_controller.go:299] Running API Priority and Fairness config worker
�[36mserver_1  |�[0m I0512 23:31:34.250098       1 cache.go:39] Caches are synced for AvailableConditionController controller
�[36mserver_1  |�[0m I0512 23:31:34.250137       1 shared_informer.go:247] Caches are synced for cluster_authentication_trust_controller 
�[36mserver_1  |�[0m I0512 23:31:34.259754       1 controller.go:611] quota admission added evaluator for: namespaces
�[36mserver_1  |�[0m E0512 23:31:34.275077       1 controller.go:151] Unable to perform initial Kubernetes service initialization: Service "kubernetes" is invalid: spec.clusterIPs: Invalid value: []string{"10.43.0.1"}: failed to allocated ip:10.43.0.1 with error:cannot allocate resources of type serviceipallocations at this time
�[36mserver_1  |�[0m E0512 23:31:34.276073       1 controller.go:156] Unable to remove old endpoints from kubernetes service: StorageError: key not found, Code: 1, Key: /registry/masterleases/172.18.0.2, ResourceVersion: 0, AdditionalErrorMsg: 
�[36mserver_1  |�[0m E0512 23:31:34.797621       1 node.go:161] Failed to retrieve node info: nodes "cc9d29cc51b1" is forbidden: User "system:kube-proxy" cannot get resource "nodes" in API group "" at the cluster scope
�[36mserver_1  |�[0m I0512 23:31:35.148030       1 controller.go:132] OpenAPI AggregationController: action for item : Nothing (removed from the queue).
�[36mserver_1  |�[0m I0512 23:31:35.148083       1 controller.go:132] OpenAPI AggregationController: action for item k8s_internal_local_delegation_chain_0000000000: Nothing (removed from the queue).
�[36mserver_1  |�[0m I0512 23:31:35.156801       1 storage_scheduling.go:132] created PriorityClass system-node-critical with value 2000001000
�[36mserver_1  |�[0m I0512 23:31:35.159131       1 storage_scheduling.go:132] created PriorityClass system-cluster-critical with value 2000000000
�[36mserver_1  |�[0m I0512 23:31:35.159142       1 storage_scheduling.go:148] all system priority classes are created successfully or already exist.
�[36mserver_1  |�[0m I0512 23:31:35.386124       1 controller.go:611] quota admission added evaluator for: roles.rbac.authorization.k8s.io
�[36mserver_1  |�[0m I0512 23:31:35.405079       1 controller.go:611] quota admission added evaluator for: rolebindings.rbac.authorization.k8s.io
�[36mserver_1  |�[0m W0512 23:31:35.494013       1 lease.go:233] Resetting endpoints for master service "kubernetes" to [172.18.0.2]
�[36mserver_1  |�[0m I0512 23:31:35.494645       1 controller.go:611] quota admission added evaluator for: endpoints
�[36mserver_1  |�[0m I0512 23:31:35.497378       1 controller.go:611] quota admission added evaluator for: endpointslices.discovery.k8s.io
�[36mserver_1  |�[0m time="2021-05-12T23:31:35.793556482Z" level=info msg="Waiting for node cc9d29cc51b1: nodes \"cc9d29cc51b1\" not found"
�[36mserver_1  |�[0m time="2021-05-12T23:31:36.165025765Z" level=info msg="Kube API server is now running"
�[36mserver_1  |�[0m time="2021-05-12T23:31:36.165116571Z" level=info msg="k3s is up and running"
�[36mserver_1  |�[0m Flag --address has been deprecated, see --bind-address instead.
�[36mserver_1  |�[0m I0512 23:31:36.171347       1 controllermanager.go:175] Version: v1.21.0+k3s1
�[36mserver_1  |�[0m I0512 23:31:36.172432       1 deprecated_insecure_serving.go:53] Serving insecurely on 127.0.0.1:10252
�[36mserver_1  |�[0m time="2021-05-12T23:31:36.175658557Z" level=info msg="Creating CRD addons.k3s.cattle.io"
�[36mserver_1  |�[0m W0512 23:31:36.180710       1 authorization.go:47] Authorization is disabled
�[36mserver_1  |�[0m W0512 23:31:36.180737       1 authentication.go:47] Authentication is disabled
�[36mserver_1  |�[0m I0512 23:31:36.180743       1 deprecated_insecure_serving.go:51] Serving healthz insecurely on 127.0.0.1:10251
�[36mserver_1  |�[0m time="2021-05-12T23:31:36.181860401Z" level=info msg="Creating CRD helmcharts.helm.cattle.io"
�[36mserver_1  |�[0m time="2021-05-12T23:31:36.184307122Z" level=info msg="Creating CRD helmchartconfigs.helm.cattle.io"
�[36mserver_1  |�[0m time="2021-05-12T23:31:36.189970433Z" level=info msg="Waiting for CRD helmchartconfigs.helm.cattle.io to become available"
�[36mserver_1  |�[0m time="2021-05-12T23:31:36.696182899Z" level=info msg="Done waiting for CRD helmchartconfigs.helm.cattle.io to become available"
�[36mserver_1  |�[0m time="2021-05-12T23:31:36.696235989Z" level=info msg="Waiting for CRD addons.k3s.cattle.io to become available"
�[36mserver_1  |�[0m I0512 23:31:36.739843       1 shared_informer.go:240] Waiting for caches to sync for tokens
�[36mserver_1  |�[0m I0512 23:31:36.749810       1 controller.go:611] quota admission added evaluator for: serviceaccounts
�[36mserver_1  |�[0m I0512 23:31:36.761114       1 controllermanager.go:574] Started "namespace"
�[36mserver_1  |�[0m I0512 23:31:36.761182       1 namespace_controller.go:200] Starting namespace controller
�[36mserver_1  |�[0m I0512 23:31:36.761192       1 shared_informer.go:240] Waiting for caches to sync for namespace
�[36mserver_1  |�[0m I0512 23:31:36.773087       1 controllermanager.go:574] Started "statefulset"
�[36mserver_1  |�[0m I0512 23:31:36.773249       1 stateful_set.go:146] Starting stateful set controller
�[36mserver_1  |�[0m I0512 23:31:36.773259       1 shared_informer.go:240] Waiting for caches to sync for stateful set
�[36mserver_1  |�[0m I0512 23:31:36.777449       1 controllermanager.go:574] Started "ttl"
�[36mserver_1  |�[0m I0512 23:31:36.777481       1 ttl_controller.go:121] Starting TTL controller
�[36mserver_1  |�[0m I0512 23:31:36.777486       1 shared_informer.go:240] Waiting for caches to sync for TTL
�[36mserver_1  |�[0m I0512 23:31:36.782061       1 node_ipam_controller.go:91] Sending events to api server.
�[36mserver_1  |�[0m I0512 23:31:36.840991       1 shared_informer.go:247] Caches are synced for tokens 
�[36mserver_1  |�[0m E0512 23:31:37.123064       1 node.go:161] Failed to retrieve node info: nodes "cc9d29cc51b1" not found
�[36mserver_1  |�[0m time="2021-05-12T23:31:37.199185259Z" level=info msg="Done waiting for CRD addons.k3s.cattle.io to become available"
�[36mserver_1  |�[0m time="2021-05-12T23:31:37.199199711Z" level=info msg="Waiting for CRD helmcharts.helm.cattle.io to become available"
�[36mserver_1  |�[0m time="2021-05-12T23:31:37.706693912Z" level=info msg="Done waiting for CRD helmcharts.helm.cattle.io to become available"
�[36mserver_1  |�[0m time="2021-05-12T23:31:37.721681648Z" level=info msg="Writing static file: /var/lib/rancher/k3s/server/static/charts/traefik-9.18.2.tgz"
�[36mserver_1  |�[0m time="2021-05-12T23:31:37.722108338Z" level=info msg="Writing static file: /var/lib/rancher/k3s/server/static/charts/traefik-crd-9.18.2.tgz"
�[36mserver_1  |�[0m time="2021-05-12T23:31:37.771967744Z" level=info msg="Failed to get existing traefik HelmChart" error="helmcharts.helm.cattle.io \"traefik\" not found"
�[36mserver_1  |�[0m time="2021-05-12T23:31:37.772126907Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/local-storage.yaml"
�[36mserver_1  |�[0m time="2021-05-12T23:31:37.772236148Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/auth-delegator.yaml"
�[36mserver_1  |�[0m time="2021-05-12T23:31:37.772311154Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-service.yaml"
�[36mserver_1  |�[0m time="2021-05-12T23:31:37.772385661Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/resource-reader.yaml"
�[36mserver_1  |�[0m time="2021-05-12T23:31:37.772466525Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/traefik.yaml"
�[36mserver_1  |�[0m time="2021-05-12T23:31:37.772542001Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/ccm.yaml"
�[36mserver_1  |�[0m time="2021-05-12T23:31:37.772614640Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/aggregated-metrics-reader.yaml"
�[36mserver_1  |�[0m time="2021-05-12T23:31:37.772684617Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/auth-reader.yaml"
�[36mserver_1  |�[0m time="2021-05-12T23:31:37.772753014Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/metrics-apiservice.yaml"
�[36mserver_1  |�[0m time="2021-05-12T23:31:37.772830501Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-deployment.yaml"
�[36mserver_1  |�[0m time="2021-05-12T23:31:37.772906121Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/rolebindings.yaml"
�[36mserver_1  |�[0m time="2021-05-12T23:31:37.773018653Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/coredns.yaml"
�[36mserver_1  |�[0m time="2021-05-12T23:31:37.797500817Z" level=info msg="Waiting for node cc9d29cc51b1: nodes \"cc9d29cc51b1\" not found"
�[36mserver_1  |�[0m time="2021-05-12T23:31:37.873682314Z" level=info msg="Starting k3s.cattle.io/v1, Kind=Addon controller"
�[36mserver_1  |�[0m time="2021-05-12T23:31:37.877852666Z" level=info msg="Waiting for control-plane node cc9d29cc51b1 startup: nodes \"cc9d29cc51b1\" not found"
�[36mserver_1  |�[0m time="2021-05-12T23:31:37.882185366Z" level=info msg="Cluster dns configmap has been set successfully"
�[36mserver_1  |�[0m time="2021-05-12T23:31:37.975673115Z" level=info msg="Starting helm.cattle.io/v1, Kind=HelmChart controller"
�[36mserver_1  |�[0m time="2021-05-12T23:31:37.975652366Z" level=info msg="Starting helm.cattle.io/v1, Kind=HelmChartConfig controller"
�[36mserver_1  |�[0m time="2021-05-12T23:31:37.975750874Z" level=info msg="Starting /v1, Kind=Service controller"
�[36mserver_1  |�[0m time="2021-05-12T23:31:37.975799085Z" level=info msg="Starting /v1, Kind=Endpoints controller"
�[36mserver_1  |�[0m time="2021-05-12T23:31:37.975713031Z" level=info msg="Starting batch/v1, Kind=Job controller"
�[36mserver_1  |�[0m time="2021-05-12T23:31:37.975836883Z" level=info msg="Starting /v1, Kind=Node controller"
�[36mserver_1  |�[0m time="2021-05-12T23:31:37.975823678Z" level=info msg="Starting /v1, Kind=Pod controller"
�[36mserver_1  |�[0m time="2021-05-12T23:31:38.282351071Z" level=info msg="Starting /v1, Kind=Secret controller"
�[36mserver_1  |�[0m time="2021-05-12T23:31:38.481097862Z" level=info msg="Active TLS secret k3s-serving (ver=230) (count 7): map[listener.cattle.io/cn-10.43.0.1:10.43.0.1 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-172.18.0.2:172.18.0.2 listener.cattle.io/cn-kubernetes:kubernetes listener.cattle.io/cn-kubernetes.default:kubernetes.default listener.cattle.io/cn-kubernetes.default.svc.cluster.local:kubernetes.default.svc.cluster.local listener.cattle.io/cn-localhost:localhost listener.cattle.io/fingerprint:SHA1=C07AD29EEDD589A181CE66297B7B89A304594043]"
�[36mserver_1  |�[0m time="2021-05-12T23:31:38.777791283Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
�[36mserver_1  |�[0m W0512 23:31:38.819157       1 info.go:53] Couldn't collect info from any of the files in "/etc/machine-id,/var/lib/dbus/machine-id"
�[36mserver_1  |�[0m I0512 23:31:38.835816       1 server.go:660] "--cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /"
�[36mserver_1  |�[0m I0512 23:31:38.836069       1 container_manager_linux.go:291] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
�[36mserver_1  |�[0m I0512 23:31:38.836228       1 container_manager_linux.go:296] "Creating Container Manager object based on Node Config" nodeConfig={RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:remote CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:cgroupfs KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: ReservedSystemCPUs: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:imagefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>}]} QOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalTopologyManagerScope:container ExperimentalCPUManagerReconcilePeriod:10s ExperimentalMemoryManagerPolicy:None ExperimentalMemoryManagerReservedMemory:[] ExperimentalPodPidsLimit:-1 EnforceCPULimits:true CPUCFSQuotaPeriod:100ms ExperimentalTopologyManagerPolicy:none Rootless:false}
�[36mserver_1  |�[0m I0512 23:31:38.836271       1 topology_manager.go:120] "Creating topology manager with policy per scope" topologyPolicyName="none" topologyScopeName="container"
�[36mserver_1  |�[0m I0512 23:31:38.836320       1 container_manager_linux.go:327] "Initializing Topology Manager" policy="none" scope="container"
�[36mserver_1  |�[0m I0512 23:31:38.836326       1 container_manager_linux.go:332] "Creating device plugin manager" devicePluginEnabled=true
�[36mserver_1  |�[0m I0512 23:31:38.836617       1 kubelet.go:275] "Adding static pod path" path="/var/lib/rancher/k3s/agent/pod-manifests"
�[36mserver_1  |�[0m I0512 23:31:38.836660       1 kubelet.go:286] "Watching apiserver"
�[36mserver_1  |�[0m I0512 23:31:38.836747       1 kubelet.go:465] "Kubelet client is not nil"
�[36mserver_1  |�[0m I0512 23:31:38.837353       1 kuberuntime_manager.go:222] "Container runtime initialized" containerRuntime="containerd" version="v1.4.4-k3s1" apiVersion="v1alpha2"
�[36mserver_1  |�[0m W0512 23:31:38.837641       1 probe.go:268] Flexvolume plugin directory at /usr/libexec/kubernetes/kubelet-plugins/volume/exec/ does not exist. Recreating.
�[36mserver_1  |�[0m I0512 23:31:38.838036       1 server.go:1191] "Started kubelet"
�[36mserver_1  |�[0m I0512 23:31:38.838070       1 server.go:149] "Starting to listen" address="0.0.0.0" port=10250
�[36mserver_1  |�[0m E0512 23:31:38.838396       1 cri_stats_provider.go:369] "Failed to get the info of the filesystem with mountpoint" err="unable to find data in memory cache" mountpoint="/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs"
�[36mserver_1  |�[0m E0512 23:31:38.838422       1 kubelet.go:1313] "Image garbage collection failed once. Stats initialization may not have completed yet" err="invalid capacity 0 on image filesystem"
�[36mserver_1  |�[0m I0512 23:31:38.838810       1 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
�[36mserver_1  |�[0m I0512 23:31:38.838861       1 volume_manager.go:271] "Starting Kubelet Volume Manager"
�[36mserver_1  |�[0m I0512 23:31:38.838897       1 desired_state_of_world_populator.go:141] "Desired state populator starts to run"
�[36mserver_1  |�[0m I0512 23:31:38.839030       1 server.go:405] "Adding debug handlers to kubelet server"
�[36mserver_1  |�[0m I0512 23:31:38.842576       1 kubelet.go:461] "Kubelet nodes not sync"
�[36mserver_1  |�[0m I0512 23:31:38.842588       1 kubelet.go:461] "Kubelet nodes not sync"
�[36mserver_1  |�[0m E0512 23:31:38.848963       1 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"cc9d29cc51b1\" not found" node="cc9d29cc51b1"
�[36mserver_1  |�[0m I0512 23:31:38.863267       1 kubelet_network_linux.go:56] "Initialized protocol iptables rules." protocol=IPv4
�[36mserver_1  |�[0m E0512 23:31:38.872520       1 kubelet_network_linux.go:167] "Failed to ensure SNAT rule for packets marked by KUBE-MARK-MASQ chain in nat table KUBE-POSTROUTING chain" err="error checking rule: exit status 2: ip6tables v1.8.5 (nf_tables): unknown option \"--random-fully\"\nTry `ip6tables -h' or 'ip6tables --help' for more information.\n"
�[36mserver_1  |�[0m I0512 23:31:38.872536       1 kubelet_network_linux.go:64] "Failed to initialize protocol iptables rules; some functionality may be missing." protocol=IPv6
�[36mserver_1  |�[0m I0512 23:31:38.872544       1 status_manager.go:157] "Starting to sync pod status with apiserver"
�[36mserver_1  |�[0m I0512 23:31:38.872554       1 kubelet.go:1853] "Starting kubelet main sync loop"
�[36mserver_1  |�[0m E0512 23:31:38.872615       1 kubelet.go:1877] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
�[36mserver_1  |�[0m E0512 23:31:38.939703       1 kubelet.go:2298] "Error getting node" err="node \"cc9d29cc51b1\" not found"
�[36mserver_1  |�[0m I0512 23:31:38.941448       1 kubelet_node_status.go:71] "Attempting to register node" node="cc9d29cc51b1"
�[36mserver_1  |�[0m time="2021-05-12T23:31:38.970555657Z" level=info msg="Cluster-Http-Server 2021/05/12 23:31:38 http: TLS handshake error from 172.18.0.3:44936: remote error: tls: bad certificate"
�[36mserver_1  |�[0m E0512 23:31:38.972743       1 kubelet.go:1877] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
�[36mserver_1  |�[0m E0512 23:31:39.040692       1 kubelet.go:2298] "Error getting node" err="node \"cc9d29cc51b1\" not found"
�[36mserver_1  |�[0m time="2021-05-12T23:31:39.079533587Z" level=info msg="Waiting for control-plane node cc9d29cc51b1 startup: nodes \"cc9d29cc51b1\" not found"
�[36mserver_1  |�[0m E0512 23:31:39.141619       1 kubelet.go:2298] "Error getting node" err="node \"cc9d29cc51b1\" not found"
�[36mserver_1  |�[0m time="2021-05-12T23:31:39.168330990Z" level=info msg="Running cloud-controller-manager with args --profiling=false"
�[36mserver_1  |�[0m E0512 23:31:39.173371       1 kubelet.go:1877] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
�[36mserver_1  |�[0m I0512 23:31:39.174283       1 controllermanager.go:142] Version: v1.21.0+k3s1
�[36mserver_1  |�[0m E0512 23:31:39.241792       1 kubelet.go:2298] "Error getting node" err="node \"cc9d29cc51b1\" not found"
�[36mserver_1  |�[0m I0512 23:31:39.248189       1 kubelet_node_status.go:74] "Successfully registered node" node="cc9d29cc51b1"
�[36mserver_1  |�[0m time="2021-05-12T23:31:39.252341579Z" level=warning msg="Unable to fetch coredns config map: configmap \"coredns\" not found"
�[36mserver_1  |�[0m time="2021-05-12T23:31:39.483259842Z" level=warning msg="Unable to fetch coredns config map: configmap \"coredns\" not found"
�[36mserver_1  |�[0m E0512 23:31:39.574357       1 kubelet.go:1877] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
�[36mserver_1  |�[0m time="2021-05-12T23:31:39.682085491Z" level=info msg="certificate CN=514ada0b2fb5 signed by CN=k3s-server-ca@1620862291: notBefore=2021-05-12 23:31:31 +0000 UTC notAfter=2022-05-12 23:31:39 +0000 UTC"
�[36mserver_1  |�[0m time="2021-05-12T23:31:39.804480236Z" level=info msg="Waiting for node cc9d29cc51b1 CIDR not assigned yet"
�[36mserver_1  |�[0m I0512 23:31:39.837800       1 node_controller.go:115] Sending events to api server.
�[36mserver_1  |�[0m I0512 23:31:39.837958       1 controllermanager.go:285] Started "cloud-node"
�[36mserver_1  |�[0m I0512 23:31:39.838058       1 node_controller.go:154] Waiting for informer caches to sync
�[36mserver_1  |�[0m I0512 23:31:39.844250       1 cpu_manager.go:199] "Starting CPU manager" policy="none"
�[36mserver_1  |�[0m I0512 23:31:39.844289       1 cpu_manager.go:200] "Reconciling" reconcilePeriod="10s"
�[36mserver_1  |�[0m I0512 23:31:39.844327       1 state_mem.go:36] "Initialized new in-memory state store"
�[36mserver_1  |�[0m I0512 23:31:39.848249       1 node_lifecycle_controller.go:76] Sending events to api server
�[36mserver_1  |�[0m I0512 23:31:39.848376       1 controllermanager.go:285] Started "cloud-node-lifecycle"
�[36mserver_1  |�[0m I0512 23:31:39.851527       1 policy_none.go:44] "None policy: Start"
�[36mserver_1  |�[0m time="2021-05-12T23:31:39.879471436Z" level=warning msg="Unable to fetch coredns config map: configmap \"coredns\" not found"
�[36mserver_1  |�[0m I0512 23:31:39.894251       1 controller.go:611] quota admission added evaluator for: addons.k3s.cattle.io
�[36mserver_1  |�[0m W0512 23:31:39.895001       1 watcher.go:95] Error while processing event ("/sys/fs/cgroup/kubepods": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/kubepods: no such file or directory
�[36mserver_1  |�[0m E0512 23:31:39.895032       1 node_container_manager_linux.go:57] "Failed to create cgroup" err="cannot enter cgroupv2 \"/sys/fs/cgroup/kubepods\" with domain controllers -- it is in an invalid state" cgroupName=[kubepods]
�[36mserver_1  |�[0m E0512 23:31:39.895088       1 kubelet.go:1391] "Failed to start ContainerManager" err="cannot enter cgroupv2 \"/sys/fs/cgroup/kubepods\" with domain controllers -- it is in an invalid state"
�[36mserver_1  |�[0m time="2021-05-12T23:39:54.614086743Z" level=info msg="Starting k3s v1.21.0+k3s1 (2705431d)"
�[36mserver_1  |�[0m time="2021-05-12T23:39:54.614359914Z" level=info msg="Cluster bootstrap already complete"
�[36mserver_1  |�[0m time="2021-05-12T23:39:54.630222884Z" level=info msg="Configuring sqlite3 database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"
�[36mserver_1  |�[0m time="2021-05-12T23:39:54.630250498Z" level=info msg="Configuring database table schema and indexes, this may take a moment..."
�[36mserver_1  |�[0m time="2021-05-12T23:39:54.630360480Z" level=info msg="Database tables and indexes are up to date"
�[36mserver_1  |�[0m time="2021-05-12T23:39:54.631185632Z" level=info msg="Kine listening on unix://kine.sock"
�[36mserver_1  |�[0m time="2021-05-12T23:39:54.631298180Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --enable-admission-plugins=NodeRestriction --etcd-servers=unix://kine.sock --insecure-port=0 --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key"
�[36mserver_1  |�[0m Flag --insecure-port has been deprecated, This flag has no effect now and will be removed in v1.24.
�[36mserver_1  |�[0m I0512 23:39:54.632218       1 server.go:656] external host was not specified, using 172.18.0.2
�[36mserver_1  |�[0m I0512 23:39:54.632415       1 server.go:195] Version: v1.21.0+k3s1
�[36mserver_1  |�[0m I0512 23:39:54.635656       1 shared_informer.go:240] Waiting for caches to sync for node_authorizer
�[36mserver_1  |�[0m I0512 23:39:54.636524       1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
�[36mserver_1  |�[0m I0512 23:39:54.636535       1 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
�[36mserver_1  |�[0m I0512 23:39:54.637737       1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
�[36mserver_1  |�[0m I0512 23:39:54.637749       1 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
�[36mserver_1  |�[0m I0512 23:39:54.656375       1 instance.go:283] Using reconciler: lease
�[36mserver_1  |�[0m I0512 23:39:54.692817       1 rest.go:130] the default service ipfamily for this cluster is: IPv4
�[36mserver_1  |�[0m W0512 23:39:54.987861       1 genericapiserver.go:425] Skipping API node.k8s.io/v1alpha1 because it has no resources.
�[36mserver_1  |�[0m W0512 23:39:54.999303       1 genericapiserver.go:425] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
�[36mserver_1  |�[0m W0512 23:39:55.002676       1 genericapiserver.go:425] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.
�[36mserver_1  |�[0m W0512 23:39:55.010113       1 genericapiserver.go:425] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
�[36mserver_1  |�[0m W0512 23:39:55.014217       1 genericapiserver.go:425] Skipping API flowcontrol.apiserver.k8s.io/v1alpha1 because it has no resources.
�[36mserver_1  |�[0m W0512 23:39:55.021776       1 genericapiserver.go:425] Skipping API apps/v1beta2 because it has no resources.
�[36mserver_1  |�[0m W0512 23:39:55.021795       1 genericapiserver.go:425] Skipping API apps/v1beta1 because it has no resources.
�[36mserver_1  |�[0m I0512 23:39:55.031805       1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
�[36mserver_1  |�[0m I0512 23:39:55.031819       1 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
�[36mserver_1  |�[0m time="2021-05-12T23:39:55.043927759Z" level=info msg="Running kube-scheduler --address=127.0.0.1 --bind-address=127.0.0.1 --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --leader-elect=false --port=10251 --profiling=false --secure-port=0"
�[36mserver_1  |�[0m time="2021-05-12T23:39:55.043913487Z" level=info msg="Waiting for API server to become available"
�[36mserver_1  |�[0m time="2021-05-12T23:39:55.044679975Z" level=info msg="Running kube-controller-manager --address=127.0.0.1 --allocate-node-cidrs=true --bind-address=127.0.0.1 --cluster-cidr=10.42.0.0/16 --cluster-signing-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --leader-elect=false --port=10252 --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=0 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --use-service-account-credentials=true"
�[36mserver_1  |�[0m time="2021-05-12T23:39:55.046374414Z" level=info msg="Node token is available at /var/lib/rancher/k3s/server/token"
�[36mserver_1  |�[0m time="2021-05-12T23:39:55.046415968Z" level=info msg="To join node to cluster: k3s agent -s https://172.18.0.2:6443 -t ${NODE_TOKEN}"
�[36mserver_1  |�[0m time="2021-05-12T23:39:55.047575266Z" level=info msg="Wrote kubeconfig /output/kubeconfig.yaml"
�[36mserver_1  |�[0m time="2021-05-12T23:39:55.047638866Z" level=info msg="Run: k3s kubectl"
�[36mserver_1  |�[0m time="2021-05-12T23:39:55.063916681Z" level=info msg="Cluster-Http-Server 2021/05/12 23:39:55 http: TLS handshake error from 127.0.0.1:51968: remote error: tls: bad certificate"
�[36mserver_1  |�[0m time="2021-05-12T23:39:55.068231444Z" level=info msg="Cluster-Http-Server 2021/05/12 23:39:55 http: TLS handshake error from 127.0.0.1:51974: remote error: tls: bad certificate"
�[36mserver_1  |�[0m time="2021-05-12T23:39:55.082785926Z" level=info msg="certificate CN=cc9d29cc51b1 signed by CN=k3s-server-ca@1620862291: notBefore=2021-05-12 23:31:31 +0000 UTC notAfter=2022-05-12 23:39:55 +0000 UTC"
�[36mserver_1  |�[0m time="2021-05-12T23:39:55.085758198Z" level=info msg="certificate CN=system:node:cc9d29cc51b1,O=system:nodes signed by CN=k3s-client-ca@1620862291: notBefore=2021-05-12 23:31:31 +0000 UTC notAfter=2022-05-12 23:39:55 +0000 UTC"
�[36mserver_1  |�[0m time="2021-05-12T23:39:55.113276404Z" level=info msg="Module overlay was already loaded"
�[36mserver_1  |�[0m time="2021-05-12T23:39:55.113350316Z" level=info msg="Module nf_conntrack was already loaded"
�[36mserver_1  |�[0m time="2021-05-12T23:39:55.113361433Z" level=info msg="Module br_netfilter was already loaded"
�[36mserver_1  |�[0m time="2021-05-12T23:39:55.113370594Z" level=info msg="Module iptable_nat was already loaded"
�[36mserver_1  |�[0m time="2021-05-12T23:39:55.114509277Z" level=info msg="Logging containerd to /var/lib/rancher/k3s/agent/containerd/containerd.log"
�[36mserver_1  |�[0m time="2021-05-12T23:39:55.114674638Z" level=info msg="Running containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd"
�[36mserver_1  |�[0m time="2021-05-12T23:39:56.115919559Z" level=info msg="Containerd is now running"
�[36mserver_1  |�[0m time="2021-05-12T23:39:56.123855314Z" level=info msg="Connecting to proxy" url="wss://127.0.0.1:6443/v1-k3s/connect"
�[36mserver_1  |�[0m time="2021-05-12T23:39:56.126001885Z" level=info msg="Handling backend connection request [cc9d29cc51b1]"
�[36mserver_1  |�[0m time="2021-05-12T23:39:56.126590126Z" level=info msg="Running kubelet --address=0.0.0.0 --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=cgroupfs --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --cni-bin-dir=/bin --cni-conf-dir=/var/lib/rancher/k3s/agent/etc/cni/net.d --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --container-runtime=remote --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=cc9d29cc51b1 --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --node-labels= --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --read-only-port=0 --resolv-conf=/tmp/k3s-resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key"
�[36mserver_1  |�[0m time="2021-05-12T23:39:56.127192426Z" level=info msg="Running kube-proxy --cluster-cidr=10.42.0.0/16 --healthz-bind-address=127.0.0.1 --hostname-override=cc9d29cc51b1 --kubeconfig=/var/lib/rancher/k3s/agent/kubeproxy.kubeconfig --proxy-mode=iptables"
�[36mserver_1  |�[0m Flag --cloud-provider has been deprecated, will be removed in 1.23, in favor of removing cloud provider code from Kubelet.
�[36mserver_1  |�[0m Flag --cni-bin-dir has been deprecated, will be removed along with dockershim.
�[36mserver_1  |�[0m Flag --cni-conf-dir has been deprecated, will be removed along with dockershim.
�[36mserver_1  |�[0m Flag --containerd has been deprecated, This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.
�[36mserver_1  |�[0m W0512 23:39:56.127473       1 server.go:220] WARNING: all flags other than --config, --write-config-to, and --cleanup are deprecated. Please begin using a config file ASAP.
�[36mserver_1  |�[0m W0512 23:39:56.128082       1 proxier.go:653] Failed to read file /lib/modules/5.12.2-arch1-1/modules.builtin with error open /lib/modules/5.12.2-arch1-1/modules.builtin: no such file or directory. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
�[36mserver_1  |�[0m I0512 23:39:56.128086       1 server.go:436] "Kubelet version" kubeletVersion="v1.21.0+k3s1"
�[36mserver_1  |�[0m W0512 23:39:56.128563       1 proxier.go:663] Failed to load kernel module ip_vs with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
�[36mserver_1  |�[0m W0512 23:39:56.128982       1 proxier.go:663] Failed to load kernel module ip_vs_rr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
�[36mserver_1  |�[0m W0512 23:39:56.129289       1 proxier.go:663] Failed to load kernel module ip_vs_wrr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
�[36mserver_1  |�[0m W0512 23:39:56.129676       1 proxier.go:663] Failed to load kernel module ip_vs_sh with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
�[36mserver_1  |�[0m W0512 23:39:56.129978       1 proxier.go:663] Failed to load kernel module nf_conntrack with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
�[36mserver_1  |�[0m E0512 23:39:56.135361       1 node.go:161] Failed to retrieve node info: nodes "cc9d29cc51b1" is forbidden: User "system:kube-proxy" cannot get resource "nodes" in API group "" at the cluster scope
�[36mserver_1  |�[0m I0512 23:39:56.148929       1 dynamic_cafile_content.go:167] Starting client-ca-bundle::/var/lib/rancher/k3s/agent/client-ca.crt
�[36mserver_1  |�[0m W0512 23:39:56.148951       1 manager.go:159] Cannot detect current cgroup on cgroup v2
�[36mserver_1  |�[0m I0512 23:39:57.233159       1 dynamic_cafile_content.go:167] Starting request-header::/var/lib/rancher/k3s/server/tls/request-header-ca.crt
�[36mserver_1  |�[0m I0512 23:39:57.233165       1 dynamic_cafile_content.go:167] Starting client-ca-bundle::/var/lib/rancher/k3s/server/tls/client-ca.crt
�[36mserver_1  |�[0m I0512 23:39:57.233403       1 dynamic_serving_content.go:130] Starting serving-cert::/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt::/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key
�[36mserver_1  |�[0m I0512 23:39:57.233854       1 secure_serving.go:197] Serving securely on 127.0.0.1:6444
�[36mserver_1  |�[0m I0512 23:39:57.233889       1 controller.go:83] Starting OpenAPI AggregationController
�[36mserver_1  |�[0m I0512 23:39:57.233910       1 tlsconfig.go:240] Starting DynamicServingCertificateController
�[36mserver_1  |�[0m I0512 23:39:57.233989       1 apf_controller.go:294] Starting API Priority and Fairness config controller
�[36mserver_1  |�[0m I0512 23:39:57.234046       1 apiservice_controller.go:97] Starting APIServiceRegistrationController
�[36mserver_1  |�[0m I0512 23:39:57.234063       1 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller
�[36mserver_1  |�[0m I0512 23:39:57.234069       1 available_controller.go:475] Starting AvailableConditionController
�[36mserver_1  |�[0m I0512 23:39:57.234097       1 cache.go:32] Waiting for caches to sync for AvailableConditionController controller
�[36mserver_1  |�[0m I0512 23:39:57.234096       1 autoregister_controller.go:141] Starting autoregister controller
�[36mserver_1  |�[0m I0512 23:39:57.234108       1 cache.go:32] Waiting for caches to sync for autoregister controller
�[36mserver_1  |�[0m I0512 23:39:57.234100       1 crdregistration_controller.go:111] Starting crd-autoregister controller
�[36mserver_1  |�[0m I0512 23:39:57.234129       1 shared_informer.go:240] Waiting for caches to sync for crd-autoregister
�[36mserver_1  |�[0m I0512 23:39:57.234274       1 establishing_controller.go:76] Starting EstablishingController
�[36mserver_1  |�[0m I0512 23:39:57.234288       1 customresource_discovery_controller.go:209] Starting DiscoveryController
�[36mserver_1  |�[0m I0512 23:39:57.234299       1 controller.go:86] Starting OpenAPI controller
�[36mserver_1  |�[0m I0512 23:39:57.234326       1 naming_controller.go:291] Starting NamingConditionController
�[36mserver_1  |�[0m I0512 23:39:57.234343       1 nonstructuralschema_controller.go:192] Starting NonStructuralSchemaConditionController
�[36mserver_1  |�[0m I0512 23:39:57.234361       1 apiapproval_controller.go:186] Starting KubernetesAPIApprovalPolicyConformantConditionController
�[36mserver_1  |�[0m I0512 23:39:57.234384       1 crd_finalizer.go:266] Starting CRDFinalizer
�[36mserver_1  |�[0m I0512 23:39:57.234443       1 cluster_authentication_trust_controller.go:440] Starting cluster_authentication_trust_controller controller
�[36mserver_1  |�[0m I0512 23:39:57.234452       1 shared_informer.go:240] Waiting for caches to sync for cluster_authentication_trust_controller
�[36mserver_1  |�[0m I0512 23:39:57.234502       1 dynamic_cafile_content.go:167] Starting client-ca-bundle::/var/lib/rancher/k3s/server/tls/client-ca.crt
�[36mserver_1  |�[0m I0512 23:39:57.234530       1 dynamic_cafile_content.go:167] Starting request-header::/var/lib/rancher/k3s/server/tls/request-header-ca.crt
�[36mserver_1  |�[0m I0512 23:39:57.234538       1 dynamic_serving_content.go:130] Starting aggregator-proxy-cert::/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt::/var/lib/rancher/k3s/server/tls/client-auth-proxy.key
�[36mserver_1  |�[0m time="2021-05-12T23:39:57.241482921Z" level=info msg="Running cloud-controller-manager with args --profiling=false"
�[36mserver_1  |�[0m I0512 23:39:57.244781       1 controllermanager.go:142] Version: v1.21.0+k3s1
�[36mserver_1  |�[0m E0512 23:39:57.254540       1 controller.go:156] Unable to remove old endpoints from kubernetes service: no master IPs were listed in storage, refusing to erase all endpoints for the kubernetes service
�[36mserver_1  |�[0m I0512 23:39:57.258173       1 node.go:172] Successfully retrieved node IP: 172.18.0.2
�[36mserver_1  |�[0m I0512 23:39:57.258188       1 server_others.go:141] Detected node IP 172.18.0.2
�[36mserver_1  |�[0m I0512 23:39:57.259875       1 server_others.go:207] kube-proxy running in dual-stack mode, IPv4-primary
�[36mserver_1  |�[0m I0512 23:39:57.259905       1 server_others.go:213] Using iptables Proxier.
�[36mserver_1  |�[0m I0512 23:39:57.259920       1 server_others.go:220] creating dualStackProxier for iptables.
�[36mserver_1  |�[0m W0512 23:39:57.259934       1 server_others.go:513] detect-local-mode set to ClusterCIDR, but no IPv6 cluster CIDR defined, , defaulting to no-op detect-local for IPv6
�[36mserver_1  |�[0m I0512 23:39:57.260268       1 server.go:643] Version: v1.21.0+k3s1
�[36mserver_1  |�[0m I0512 23:39:57.263655       1 conntrack.go:52] Setting nf_conntrack_max to 524288
�[36mserver_1  |�[0m I0512 23:39:57.263802       1 conntrack.go:83] Setting conntrack hashsize to 131072
�[36mserver_1  |�[0m E0512 23:39:57.263823       1 conntrack.go:85] failed to set conntrack hashsize to 131072: write /sys/module/nf_conntrack/parameters/hashsize: operation not supported
�[36mserver_1  |�[0m I0512 23:39:57.263870       1 conntrack.go:103] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
�[36mserver_1  |�[0m I0512 23:39:57.263903       1 conntrack.go:103] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
�[36mserver_1  |�[0m I0512 23:39:57.264005       1 config.go:315] Starting service config controller
�[36mserver_1  |�[0m I0512 23:39:57.264013       1 shared_informer.go:240] Waiting for caches to sync for service config
�[36mserver_1  |�[0m I0512 23:39:57.264031       1 config.go:224] Starting endpoint slice config controller
�[36mserver_1  |�[0m I0512 23:39:57.264040       1 shared_informer.go:240] Waiting for caches to sync for endpoint slice config
�[36mserver_1  |�[0m I0512 23:39:57.334113       1 apf_controller.go:299] Running API Priority and Fairness config worker
�[36mserver_1  |�[0m I0512 23:39:57.334127       1 cache.go:39] Caches are synced for AvailableConditionController controller
�[36mserver_1  |�[0m I0512 23:39:57.334114       1 cache.go:39] Caches are synced for APIServiceRegistrationController controller
�[36mserver_1  |�[0m I0512 23:39:57.334271       1 cache.go:39] Caches are synced for autoregister controller
�[36mserver_1  |�[0m I0512 23:39:57.334276       1 shared_informer.go:247] Caches are synced for crd-autoregister 
�[36mserver_1  |�[0m I0512 23:39:57.334658       1 shared_informer.go:247] Caches are synced for cluster_authentication_trust_controller 
�[36mserver_1  |�[0m I0512 23:39:57.335803       1 shared_informer.go:247] Caches are synced for node_authorizer 
�[36mserver_1  |�[0m I0512 23:39:57.364533       1 shared_informer.go:247] Caches are synced for endpoint slice config 
�[36mserver_1  |�[0m I0512 23:39:57.364559       1 shared_informer.go:247] Caches are synced for service config 
�[36mserver_1  |�[0m E0512 23:39:57.488086       1 proxier.go:1612] "Failed to execute iptables-restore" err="exit status 2 (ip6tables-restore v1.8.5 (nf_tables): unknown option \"--random-fully\"\nError occurred at line: 18\nTry `ip6tables-restore -h' or 'ip6tables-restore --help' for more information.\n)"
�[36mserver_1  |�[0m I0512 23:39:57.488101       1 proxier.go:859] "Sync failed" retryingTime="30s"
�[36mserver_1  |�[0m I0512 23:39:58.233873       1 controller.go:132] OpenAPI AggregationController: action for item : Nothing (removed from the queue).
�[36mserver_1  |�[0m I0512 23:39:58.233913       1 controller.go:132] OpenAPI AggregationController: action for item k8s_internal_local_delegation_chain_0000000000: Nothing (removed from the queue).
�[36mserver_1  |�[0m I0512 23:39:58.238909       1 storage_scheduling.go:148] all system priority classes are created successfully or already exist.
�[36mserver_1  |�[0m time="2021-05-12T23:39:59.139754666Z" level=info msg="Waiting for node cc9d29cc51b1 CIDR not assigned yet"
�[36mserver_1  |�[0m time="2021-05-12T23:39:59.243375540Z" level=info msg="Kube API server is now running"
�[36mserver_1  |�[0m time="2021-05-12T23:39:59.243449999Z" level=info msg="k3s is up and running"
�[36mserver_1  |�[0m Flag --address has been deprecated, see --bind-address instead.
�[36mserver_1  |�[0m I0512 23:39:59.246101       1 controllermanager.go:175] Version: v1.21.0+k3s1
�[36mserver_1  |�[0m I0512 23:39:59.246519       1 deprecated_insecure_serving.go:53] Serving insecurely on 127.0.0.1:10252
�[36mserver_1  |�[0m W0512 23:39:59.256049       1 authorization.go:47] Authorization is disabled
�[36mserver_1  |�[0m W0512 23:39:59.256062       1 authentication.go:47] Authentication is disabled
�[36mserver_1  |�[0m I0512 23:39:59.256069       1 deprecated_insecure_serving.go:51] Serving healthz insecurely on 127.0.0.1:10251
�[36mserver_1  |�[0m time="2021-05-12T23:39:59.263111272Z" level=info msg="Writing static file: /var/lib/rancher/k3s/server/static/charts/traefik-9.18.2.tgz"
�[36mserver_1  |�[0m time="2021-05-12T23:39:59.263532942Z" level=info msg="Writing static file: /var/lib/rancher/k3s/server/static/charts/traefik-crd-9.18.2.tgz"
�[36mserver_1  |�[0m time="2021-05-12T23:39:59.287140641Z" level=info msg="Failed to get existing traefik HelmChart" error="helmcharts.helm.cattle.io \"traefik\" not found"
�[36mserver_1  |�[0m time="2021-05-12T23:39:59.287224791Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-deployment.yaml"
�[36mserver_1  |�[0m time="2021-05-12T23:39:59.287399993Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/resource-reader.yaml"
�[36mserver_1  |�[0m time="2021-05-12T23:39:59.287488435Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/traefik.yaml"
�[36mserver_1  |�[0m time="2021-05-12T23:39:59.287619754Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/local-storage.yaml"
�[36mserver_1  |�[0m time="2021-05-12T23:39:59.287786777Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/coredns.yaml"
�[36mserver_1  |�[0m time="2021-05-12T23:39:59.287901999Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/aggregated-metrics-reader.yaml"
�[36mserver_1  |�[0m time="2021-05-12T23:39:59.288035817Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/auth-delegator.yaml"
�[36mserver_1  |�[0m time="2021-05-12T23:39:59.288141183Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/auth-reader.yaml"
�[36mserver_1  |�[0m time="2021-05-12T23:39:59.288219435Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/metrics-apiservice.yaml"
�[36mserver_1  |�[0m time="2021-05-12T23:39:59.288305048Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-service.yaml"
�[36mserver_1  |�[0m time="2021-05-12T23:39:59.288397200Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/rolebindings.yaml"
�[36mserver_1  |�[0m time="2021-05-12T23:39:59.288481628Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/ccm.yaml"
�[36mserver_1  |�[0m time="2021-05-12T23:39:59.388749947Z" level=info msg="Starting k3s.cattle.io/v1, Kind=Addon controller"
�[36mserver_1  |�[0m time="2021-05-12T23:39:59.391920809Z" level=info msg="Cluster dns configmap already exists"
�[36mserver_1  |�[0m time="2021-05-12T23:39:59.398358931Z" level=info msg="Control-plane role label has been set successfully on node: cc9d29cc51b1"
�[36mserver_1  |�[0m I0512 23:39:59.403917       1 controller.go:611] quota admission added evaluator for: addons.k3s.cattle.io
�[36mserver_1  |�[0m I0512 23:39:59.415352       1 controller.go:611] quota admission added evaluator for: serviceaccounts
�[36mserver_1  |�[0m I0512 23:39:59.445768       1 controller.go:611] quota admission added evaluator for: deployments.apps
�[36mserver_1  |�[0m time="2021-05-12T23:39:59.489995815Z" level=info msg="Starting batch/v1, Kind=Job controller"
�[36mserver_1  |�[0m time="2021-05-12T23:39:59.489999179Z" level=info msg="Starting /v1, Kind=Node controller"
�[36mserver_1  |�[0m time="2021-05-12T23:39:59.489993880Z" level=info msg="Starting helm.cattle.io/v1, Kind=HelmChartConfig controller"
�[36mserver_1  |�[0m time="2021-05-12T23:39:59.490012570Z" level=info msg="Starting helm.cattle.io/v1, Kind=HelmChart controller"
�[36mserver_1  |�[0m time="2021-05-12T23:39:59.490000679Z" level=info msg="Starting /v1, Kind=Service controller"
�[36mserver_1  |�[0m time="2021-05-12T23:39:59.490021042Z" level=info msg="Starting /v1, Kind=Pod controller"
�[36mserver_1  |�[0m time="2021-05-12T23:39:59.490025715Z" level=info msg="Starting /v1, Kind=Endpoints controller"
�[36mserver_1  |�[0m time="2021-05-12T23:39:59.492511910Z" level=info msg="Updated coredns node hosts entry [172.18.0.2 cc9d29cc51b1]"
�[36mserver_1  |�[0m I0512 23:39:59.510865       1 controller.go:611] quota admission added evaluator for: rolebindings.rbac.authorization.k8s.io
�[36mserver_1  |�[0m I0512 23:39:59.701154       1 controller.go:611] quota admission added evaluator for: helmcharts.helm.cattle.io
�[36mserver_1  |�[0m I0512 23:39:59.722066       1 controller.go:611] quota admission added evaluator for: jobs.batch
�[36mserver_1  |�[0m time="2021-05-12T23:39:59.833679960Z" level=info msg="Starting /v1, Kind=Secret controller"
�[36mserver_1  |�[0m I0512 23:39:59.911142       1 shared_informer.go:240] Waiting for caches to sync for tokens
�[36mserver_1  |�[0m I0512 23:39:59.912057       1 node_controller.go:115] Sending events to api server.
�[36mserver_1  |�[0m I0512 23:39:59.912134       1 controllermanager.go:285] Started "cloud-node"
�[36mserver_1  |�[0m I0512 23:39:59.912229       1 node_controller.go:154] Waiting for informer caches to sync
�[36mserver_1  |�[0m I0512 23:39:59.913214       1 node_lifecycle_controller.go:76] Sending events to api server
�[36mserver_1  |�[0m I0512 23:39:59.913236       1 controllermanager.go:285] Started "cloud-node-lifecycle"
�[36mserver_1  |�[0m I0512 23:39:59.919947       1 controllermanager.go:574] Started "clusterrole-aggregation"
�[36mserver_1  |�[0m I0512 23:39:59.920049       1 clusterroleaggregation_controller.go:194] Starting ClusterRoleAggregator
�[36mserver_1  |�[0m I0512 23:39:59.920056       1 shared_informer.go:240] Waiting for caches to sync for ClusterRoleAggregator
�[36mserver_1  |�[0m I0512 23:39:59.925914       1 controllermanager.go:574] Started "ttl-after-finished"
�[36mserver_1  |�[0m I0512 23:39:59.926001       1 ttlafterfinished_controller.go:109] Starting TTL after finished controller
�[36mserver_1  |�[0m I0512 23:39:59.926008       1 shared_informer.go:240] Waiting for caches to sync for TTL after finished
�[36mserver_1  |�[0m I0512 23:39:59.930686       1 controllermanager.go:574] Started "deployment"
�[36mserver_1  |�[0m I0512 23:39:59.930784       1 deployment_controller.go:153] "Starting controller" controller="deployment"
�[36mserver_1  |�[0m I0512 23:39:59.930792       1 shared_informer.go:240] Waiting for caches to sync for deployment
�[36mserver_1  |�[0m I0512 23:39:59.935520       1 controllermanager.go:574] Started "csrapproving"
�[36mserver_1  |�[0m I0512 23:39:59.935535       1 certificate_controller.go:118] Starting certificate controller "csrapproving"
�[36mserver_1  |�[0m I0512 23:39:59.935543       1 shared_informer.go:240] Waiting for caches to sync for certificate-csrapproving
�[36mserver_1  |�[0m I0512 23:39:59.937436       1 node_lifecycle_controller.go:377] Sending events to api server.
�[36mserver_1  |�[0m I0512 23:39:59.937603       1 taint_manager.go:163] "Sending events to api server"
�[36mserver_1  |�[0m I0512 23:39:59.937673       1 node_lifecycle_controller.go:505] Controller will reconcile labels.
�[36mserver_1  |�[0m I0512 23:39:59.937716       1 controllermanager.go:574] Started "nodelifecycle"
�[36mserver_1  |�[0m W0512 23:39:59.937727       1 controllermanager.go:553] "route" is disabled
�[36mserver_1  |�[0m I0512 23:39:59.937827       1 node_lifecycle_controller.go:539] Starting node controller
�[36mserver_1  |�[0m I0512 23:39:59.937836       1 shared_informer.go:240] Waiting for caches to sync for taint
�[36mserver_1  |�[0m I0512 23:39:59.949097       1 controllermanager.go:574] Started "horizontalpodautoscaling"
�[36mserver_1  |�[0m I0512 23:39:59.949202       1 horizontal.go:169] Starting HPA controller
�[36mserver_1  |�[0m I0512 23:39:59.949212       1 shared_informer.go:240] Waiting for caches to sync for HPA
�[36mserver_1  |�[0m I0512 23:39:59.954603       1 controllermanager.go:574] Started "cronjob"
�[36mserver_1  |�[0m I0512 23:39:59.954697       1 cronjob_controllerv2.go:125] Starting cronjob controller v2
�[36mserver_1  |�[0m I0512 23:39:59.954706       1 shared_informer.go:240] Waiting for caches to sync for cronjob
�[36mserver_1  |�[0m I0512 23:39:59.960022       1 controllermanager.go:574] Started "ephemeral-volume"
�[36mserver_1  |�[0m I0512 23:39:59.960126       1 controller.go:170] Starting ephemeral volume controller
�[36mserver_1  |�[0m I0512 23:39:59.960135       1 shared_informer.go:240] Waiting for caches to sync for ephemeral
�[36mserver_1  |�[0m I0512 23:39:59.965166       1 controllermanager.go:574] Started "pv-protection"
�[36mserver_1  |�[0m I0512 23:39:59.965213       1 pv_protection_controller.go:83] Starting PV protection controller
�[36mserver_1  |�[0m I0512 23:39:59.965222       1 shared_informer.go:240] Waiting for caches to sync for PV protection
�[36mserver_1  |�[0m I0512 23:39:59.970391       1 controllermanager.go:574] Started "endpointslicemirroring"
�[36mserver_1  |�[0m I0512 23:39:59.970436       1 endpointslicemirroring_controller.go:211] Starting EndpointSliceMirroring controller
�[36mserver_1  |�[0m I0512 23:39:59.970444       1 shared_informer.go:240] Waiting for caches to sync for endpoint_slice_mirroring
�[36mserver_1  |�[0m I0512 23:40:00.012095       1 shared_informer.go:247] Caches are synced for tokens 
�[36mserver_1  |�[0m I0512 23:40:00.012314       1 node_controller.go:390] Initializing node cc9d29cc51b1 with cloud provider
�[36mserver_1  |�[0m time="2021-05-12T23:40:00.012383390Z" level=info msg="Couldn't find node internal ip annotation or label on node cc9d29cc51b1"
�[36mserver_1  |�[0m time="2021-05-12T23:40:00.012415871Z" level=info msg="Couldn't find node hostname annotation or label on node cc9d29cc51b1"
�[36mserver_1  |�[0m time="2021-05-12T23:40:00.013725161Z" level=info msg="Couldn't find node internal ip annotation or label on node cc9d29cc51b1"
�[36mserver_1  |�[0m time="2021-05-12T23:40:00.013744857Z" level=info msg="Couldn't find node hostname annotation or label on node cc9d29cc51b1"
�[36mserver_1  |�[0m I0512 23:40:00.020604       1 node_controller.go:454] Successfully initialized node cc9d29cc51b1 with cloud provider
�[36mserver_1  |�[0m I0512 23:40:00.021383       1 event.go:291] "Event occurred" object="cc9d29cc51b1" kind="Node" apiVersion="v1" type="Normal" reason="Synced" message="Node synced successfully"
�[36mserver_1  |�[0m I0512 23:40:00.114886       1 controllermanager.go:574] Started "replicaset"
�[36mserver_1  |�[0m I0512 23:40:00.114936       1 replica_set.go:182] Starting replicaset controller
�[36mserver_1  |�[0m I0512 23:40:00.114970       1 shared_informer.go:240] Waiting for caches to sync for ReplicaSet
�[36mserver_1  |�[0m I0512 23:40:00.164791       1 certificate_controller.go:118] Starting certificate controller "csrsigning-kubelet-serving"
�[36mserver_1  |�[0m I0512 23:40:00.164806       1 shared_informer.go:240] Waiting for caches to sync for certificate-csrsigning-kubelet-serving
�[36mserver_1  |�[0m I0512 23:40:00.164826       1 dynamic_serving_content.go:130] Starting csr-controller::/var/lib/rancher/k3s/server/tls/client-ca.crt::/var/lib/rancher/k3s/server/tls/client-ca.key
�[36mserver_1  |�[0m I0512 23:40:00.165119       1 certificate_controller.go:118] Starting certificate controller "csrsigning-kubelet-client"
�[36mserver_1  |�[0m I0512 23:40:00.165131       1 shared_informer.go:240] Waiting for caches to sync for certificate-csrsigning-kubelet-client
�[36mserver_1  |�[0m I0512 23:40:00.165132       1 dynamic_serving_content.go:130] Starting csr-controller::/var/lib/rancher/k3s/server/tls/client-ca.crt::/var/lib/rancher/k3s/server/tls/client-ca.key
�[36mserver_1  |�[0m I0512 23:40:00.165415       1 certificate_controller.go:118] Starting certificate controller "csrsigning-kube-apiserver-client"
�[36mserver_1  |�[0m I0512 23:40:00.165426       1 shared_informer.go:240] Waiting for caches to sync for certificate-csrsigning-kube-apiserver-client
�[36mserver_1  |�[0m I0512 23:40:00.165440       1 dynamic_serving_content.go:130] Starting csr-controller::/var/lib/rancher/k3s/server/tls/client-ca.crt::/var/lib/rancher/k3s/server/tls/client-ca.key
�[36mserver_1  |�[0m I0512 23:40:00.165577       1 controllermanager.go:574] Started "csrsigning"
�[36mserver_1  |�[0m I0512 23:40:00.165608       1 certificate_controller.go:118] Starting certificate controller "csrsigning-legacy-unknown"
�[36mserver_1  |�[0m I0512 23:40:00.165614       1 shared_informer.go:240] Waiting for caches to sync for certificate-csrsigning-legacy-unknown
�[36mserver_1  |�[0m I0512 23:40:00.165626       1 dynamic_serving_content.go:130] Starting csr-controller::/var/lib/rancher/k3s/server/tls/client-ca.crt::/var/lib/rancher/k3s/server/tls/client-ca.key
�[36mserver_1  |�[0m I0512 23:40:00.214239       1 controllermanager.go:574] Started "csrcleaner"
�[36mserver_1  |�[0m I0512 23:40:00.214288       1 cleaner.go:82] Starting CSR cleaner controller
�[36mserver_1  |�[0m E0512 23:40:00.373171       1 namespaced_resources_deleter.go:161] unable to get all supported resources from server: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request
�[36mserver_1  |�[0m I0512 23:40:00.373253       1 controllermanager.go:574] Started "namespace"
�[36mserver_1  |�[0m I0512 23:40:00.373292       1 namespace_controller.go:200] Starting namespace controller
�[36mserver_1  |�[0m I0512 23:40:00.373298       1 shared_informer.go:240] Waiting for caches to sync for namespace
�[36mserver_1  |�[0m I0512 23:40:00.515172       1 controllermanager.go:574] Started "job"
�[36mserver_1  |�[0m I0512 23:40:00.515230       1 job_controller.go:150] Starting job controller
�[36mserver_1  |�[0m I0512 23:40:00.515239       1 shared_informer.go:240] Waiting for caches to sync for job
�[36mserver_1  |�[0m I0512 23:40:00.666437       1 controllermanager.go:574] Started "persistentvolume-expander"
�[36mserver_1  |�[0m I0512 23:40:00.666495       1 expand_controller.go:327] Starting expand controller
�[36mserver_1  |�[0m I0512 23:40:00.666504       1 shared_informer.go:240] Waiting for caches to sync for expand
�[36mserver_1  |�[0m I0512 23:40:00.815103       1 controllermanager.go:574] Started "root-ca-cert-publisher"
�[36mserver_1  |�[0m W0512 23:40:00.815117       1 controllermanager.go:553] "cloud-node-lifecycle" is disabled
�[36mserver_1  |�[0m I0512 23:40:00.815152       1 publisher.go:102] Starting root CA certificate configmap publisher
�[36mserver_1  |�[0m I0512 23:40:00.815162       1 shared_informer.go:240] Waiting for caches to sync for crt configmap
�[36mserver_1  |�[0m I0512 23:40:00.967987       1 controllermanager.go:574] Started "persistentvolume-binder"
�[36mserver_1  |�[0m I0512 23:40:00.968052       1 pv_controller_base.go:308] Starting persistent volume controller
�[36mserver_1  |�[0m I0512 23:40:00.968058       1 shared_informer.go:240] Waiting for caches to sync for persistent volume
�[36mserver_1  |�[0m I0512 23:40:01.115987       1 controllermanager.go:574] Started "endpoint"
�[36mserver_1  |�[0m I0512 23:40:01.116067       1 endpoints_controller.go:189] Starting endpoint controller
�[36mserver_1  |�[0m I0512 23:40:01.116072       1 shared_informer.go:240] Waiting for caches to sync for endpoint
�[36mserver_1  |�[0m time="2021-05-12T23:40:01.126925433Z" level=info msg="Stopped tunnel to 127.0.0.1:6443"
�[36mserver_1  |�[0m time="2021-05-12T23:40:01.127018067Z" level=info msg="Connecting to proxy" url="wss://172.18.0.2:6443/v1-k3s/connect"
�[36mserver_1  |�[0m time="2021-05-12T23:40:01.127037866Z" level=info msg="Proxy done" err="context canceled" url="wss://127.0.0.1:6443/v1-k3s/connect"
�[36mserver_1  |�[0m time="2021-05-12T23:40:01.127123550Z" level=info msg="error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF"
�[36mserver_1  |�[0m time="2021-05-12T23:40:01.129375045Z" level=info msg="Handling backend connection request [cc9d29cc51b1]"
�[36mserver_1  |�[0m time="2021-05-12T23:40:01.146283967Z" level=info msg="Waiting for node cc9d29cc51b1 CIDR not assigned yet"
�[36mserver_1  |�[0m W0512 23:40:01.159242       1 info.go:53] Couldn't collect info from any of the files in "/etc/machine-id,/var/lib/dbus/machine-id"
�[36mserver_1  |�[0m I0512 23:40:01.159982       1 server.go:660] "--cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /"
�[36mserver_1  |�[0m I0512 23:40:01.160191       1 container_manager_linux.go:291] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
�[36mserver_1  |�[0m I0512 23:40:01.160271       1 container_manager_linux.go:296] "Creating Container Manager object based on Node Config" nodeConfig={RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:remote CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:cgroupfs KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: ReservedSystemCPUs: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:imagefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>}]} QOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalTopologyManagerScope:container ExperimentalCPUManagerReconcilePeriod:10s ExperimentalMemoryManagerPolicy:None ExperimentalMemoryManagerReservedMemory:[] ExperimentalPodPidsLimit:-1 EnforceCPULimits:true CPUCFSQuotaPeriod:100ms ExperimentalTopologyManagerPolicy:none Rootless:false}
�[36mserver_1  |�[0m I0512 23:40:01.160295       1 topology_manager.go:120] "Creating topology manager with policy per scope" topologyPolicyName="none" topologyScopeName="container"
�[36mserver_1  |�[0m I0512 23:40:01.160305       1 container_manager_linux.go:327] "Initializing Topology Manager" policy="none" scope="container"
�[36mserver_1  |�[0m I0512 23:40:01.160311       1 container_manager_linux.go:332] "Creating device plugin manager" devicePluginEnabled=true
�[36mserver_1  |�[0m I0512 23:40:01.160437       1 kubelet.go:275] "Adding static pod path" path="/var/lib/rancher/k3s/agent/pod-manifests"
�[36mserver_1  |�[0m I0512 23:40:01.160455       1 kubelet.go:286] "Watching apiserver"
�[36mserver_1  |�[0m I0512 23:40:01.160515       1 kubelet.go:465] "Kubelet client is not nil"
�[36mserver_1  |�[0m I0512 23:40:01.160996       1 kuberuntime_manager.go:222] "Container runtime initialized" containerRuntime="containerd" version="v1.4.4-k3s1" apiVersion="v1alpha2"
�[36mserver_1  |�[0m I0512 23:40:01.161263       1 server.go:1191] "Started kubelet"
�[36mserver_1  |�[0m I0512 23:40:01.161312       1 server.go:149] "Starting to listen" address="0.0.0.0" port=10250
�[36mserver_1  |�[0m E0512 23:40:01.161777       1 cri_stats_provider.go:369] "Failed to get the info of the filesystem with mountpoint" err="unable to find data in memory cache" mountpoint="/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs"
�[36mserver_1  |�[0m E0512 23:40:01.161802       1 kubelet.go:1313] "Image garbage collection failed once. Stats initialization may not have completed yet" err="invalid capacity 0 on image filesystem"
�[36mserver_1  |�[0m I0512 23:40:01.161988       1 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
�[36mserver_1  |�[0m I0512 23:40:01.162178       1 volume_manager.go:271] "Starting Kubelet Volume Manager"
�[36mserver_1  |�[0m I0512 23:40:01.162233       1 server.go:405] "Adding debug handlers to kubelet server"
�[36mserver_1  |�[0m I0512 23:40:01.162236       1 desired_state_of_world_populator.go:141] "Desired state populator starts to run"
�[36mserver_1  |�[0m I0512 23:40:01.167557       1 cpu_manager.go:199] "Starting CPU manager" policy="none"
�[36mserver_1  |�[0m I0512 23:40:01.167585       1 cpu_manager.go:200] "Reconciling" reconcilePeriod="10s"
�[36mserver_1  |�[0m I0512 23:40:01.167608       1 state_mem.go:36] "Initialized new in-memory state store"
�[36mserver_1  |�[0m I0512 23:40:01.167791       1 state_mem.go:88] "Updated default CPUSet" cpuSet=""
�[36mserver_1  |�[0m I0512 23:40:01.167806       1 state_mem.go:96] "Updated CPUSet assignments" assignments=map[]
�[36mserver_1  |�[0m I0512 23:40:01.167812       1 policy_none.go:44] "None policy: Start"
�[36mserver_1  |�[0m I0512 23:40:01.171726       1 controller.go:611] quota admission added evaluator for: leases.coordination.k8s.io
�[36mserver_1  |�[0m I0512 23:40:01.175559       1 kubelet_network_linux.go:56] "Initialized protocol iptables rules." protocol=IPv4
�[36mserver_1  |�[0m E0512 23:40:01.187483       1 kubelet_network_linux.go:167] "Failed to ensure SNAT rule for packets marked by KUBE-MARK-MASQ chain in nat table KUBE-POSTROUTING chain" err="error checking rule: exit status 2: ip6tables v1.8.5 (nf_tables): unknown option \"--random-fully\"\nTry `ip6tables -h' or 'ip6tables --help' for more information.\n"
�[36mserver_1  |�[0m I0512 23:40:01.187499       1 kubelet_network_linux.go:64] "Failed to initialize protocol iptables rules; some functionality may be missing." protocol=IPv6
�[36mserver_1  |�[0m I0512 23:40:01.187509       1 status_manager.go:157] "Starting to sync pod status with apiserver"
�[36mserver_1  |�[0m I0512 23:40:01.187523       1 kubelet.go:1853] "Starting kubelet main sync loop"
�[36mserver_1  |�[0m E0512 23:40:01.187577       1 kubelet.go:1877] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
�[36mserver_1  |�[0m E0512 23:40:01.192442       1 node_container_manager_linux.go:57] "Failed to create cgroup" err="cannot enter cgroupv2 \"/sys/fs/cgroup/kubepods\" with domain controllers -- it is in an invalid state" cgroupName=[kubepods]
�[36mserver_1  |�[0m E0512 23:40:01.192458       1 kubelet.go:1391] "Failed to start ContainerManager" err="cannot enter cgroupv2 \"/sys/fs/cgroup/kubepods\" with domain controllers -- it is in an invalid state"
�[36mserver_1  |�[0m time="2021-05-12T23:40:12.468653940Z" level=info msg="Starting k3s v1.21.0+k3s1 (2705431d)"
�[36mserver_1  |�[0m time="2021-05-12T23:40:12.468873446Z" level=info msg="Cluster bootstrap already complete"
�[36mserver_1  |�[0m time="2021-05-12T23:40:12.485813651Z" level=info msg="Configuring sqlite3 database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"
�[36mserver_1  |�[0m time="2021-05-12T23:40:12.485845923Z" level=info msg="Configuring database table schema and indexes, this may take a moment..."
�[36mserver_1  |�[0m time="2021-05-12T23:40:12.485928195Z" level=info msg="Database tables and indexes are up to date"
�[36mserver_1  |�[0m time="2021-05-12T23:40:12.486790516Z" level=info msg="Kine listening on unix://kine.sock"
�[36mserver_1  |�[0m time="2021-05-12T23:40:12.486902136Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --enable-admission-plugins=NodeRestriction --etcd-servers=unix://kine.sock --insecure-port=0 --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key"
�[36mserver_1  |�[0m Flag --insecure-port has been deprecated, This flag has no effect now and will be removed in v1.24.
�[36mserver_1  |�[0m I0512 23:40:12.487876       1 server.go:656] external host was not specified, using 172.18.0.2
�[36mserver_1  |�[0m I0512 23:40:12.488119       1 server.go:195] Version: v1.21.0+k3s1
�[36mserver_1  |�[0m I0512 23:40:12.491430       1 shared_informer.go:240] Waiting for caches to sync for node_authorizer
�[36mserver_1  |�[0m I0512 23:40:12.492528       1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
�[36mserver_1  |�[0m I0512 23:40:12.492541       1 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
�[36mserver_1  |�[0m I0512 23:40:12.493839       1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
�[36mserver_1  |�[0m I0512 23:40:12.493852       1 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
�[36mserver_1  |�[0m I0512 23:40:12.513512       1 instance.go:283] Using reconciler: lease
�[36mserver_1  |�[0m I0512 23:40:12.554781       1 rest.go:130] the default service ipfamily for this cluster is: IPv4
�[36mserver_1  |�[0m W0512 23:40:12.862008       1 genericapiserver.go:425] Skipping API node.k8s.io/v1alpha1 because it has no resources.
�[36mserver_1  |�[0m W0512 23:40:12.871007       1 genericapiserver.go:425] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
�[36mserver_1  |�[0m W0512 23:40:12.875345       1 genericapiserver.go:425] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.
�[36mserver_1  |�[0m W0512 23:40:12.884748       1 genericapiserver.go:425] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
�[36mserver_1  |�[0m W0512 23:40:12.888051       1 genericapiserver.go:425] Skipping API flowcontrol.apiserver.k8s.io/v1alpha1 because it has no resources.
�[36mserver_1  |�[0m W0512 23:40:12.895025       1 genericapiserver.go:425] Skipping API apps/v1beta2 because it has no resources.
�[36mserver_1  |�[0m W0512 23:40:12.895038       1 genericapiserver.go:425] Skipping API apps/v1beta1 because it has no resources.
�[36mserver_1  |�[0m I0512 23:40:12.904694       1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
�[36mserver_1  |�[0m I0512 23:40:12.904711       1 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
�[36mserver_1  |�[0m time="2021-05-12T23:40:12.916676617Z" level=info msg="Running kube-scheduler --address=127.0.0.1 --bind-address=127.0.0.1 --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --leader-elect=false --port=10251 --profiling=false --secure-port=0"
�[36mserver_1  |�[0m time="2021-05-12T23:40:12.916771576Z" level=info msg="Waiting for API server to become available"
�[36mserver_1  |�[0m time="2021-05-12T23:40:12.917552935Z" level=info msg="Running kube-controller-manager --address=127.0.0.1 --allocate-node-cidrs=true --bind-address=127.0.0.1 --cluster-cidr=10.42.0.0/16 --cluster-signing-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --leader-elect=false --port=10252 --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=0 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --use-service-account-credentials=true"
�[36mserver_1  |�[0m time="2021-05-12T23:40:12.918966584Z" level=info msg="Node token is available at /var/lib/rancher/k3s/server/token"
�[36mserver_1  |�[0m time="2021-05-12T23:40:12.918985762Z" level=info msg="To join node to cluster: k3s agent -s https://172.18.0.2:6443 -t ${NODE_TOKEN}"
�[36mserver_1  |�[0m time="2021-05-12T23:40:12.919842172Z" level=info msg="Wrote kubeconfig /output/kubeconfig.yaml"
�[36mserver_1  |�[0m time="2021-05-12T23:40:12.919896083Z" level=info msg="Run: k3s kubectl"
�[36mserver_1  |�[0m time="2021-05-12T23:40:12.935009711Z" level=info msg="Cluster-Http-Server 2021/05/12 23:40:12 http: TLS handshake error from 127.0.0.1:52024: remote error: tls: bad certificate"
�[36mserver_1  |�[0m time="2021-05-12T23:40:12.938637106Z" level=info msg="Cluster-Http-Server 2021/05/12 23:40:12 http: TLS handshake error from 127.0.0.1:52030: remote error: tls: bad certificate"
�[36mserver_1  |�[0m time="2021-05-12T23:40:12.947501474Z" level=info msg="certificate CN=cc9d29cc51b1 signed by CN=k3s-server-ca@1620862291: notBefore=2021-05-12 23:31:31 +0000 UTC notAfter=2022-05-12 23:40:12 +0000 UTC"
�[36mserver_1  |�[0m time="2021-05-12T23:40:12.949766086Z" level=info msg="certificate CN=system:node:cc9d29cc51b1,O=system:nodes signed by CN=k3s-client-ca@1620862291: notBefore=2021-05-12 23:31:31 +0000 UTC notAfter=2022-05-12 23:40:12 +0000 UTC"
�[36mserver_1  |�[0m time="2021-05-12T23:40:12.961034842Z" level=info msg="Module overlay was already loaded"
�[36mserver_1  |�[0m time="2021-05-12T23:40:12.961155554Z" level=info msg="Module nf_conntrack was already loaded"
�[36mserver_1  |�[0m time="2021-05-12T23:40:12.961171081Z" level=info msg="Module br_netfilter was already loaded"
�[36mserver_1  |�[0m time="2021-05-12T23:40:12.961195300Z" level=info msg="Module iptable_nat was already loaded"
�[36mserver_1  |�[0m time="2021-05-12T23:40:12.963031918Z" level=info msg="Logging containerd to /var/lib/rancher/k3s/agent/containerd/containerd.log"
�[36mserver_1  |�[0m time="2021-05-12T23:40:12.963280446Z" level=info msg="Running containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd"
�[36mserver_1  |�[0m time="2021-05-12T23:40:13.970265240Z" level=info msg="Containerd is now running"
�[36mserver_1  |�[0m time="2021-05-12T23:40:13.978283492Z" level=info msg="Connecting to proxy" url="wss://127.0.0.1:6443/v1-k3s/connect"
�[36mserver_1  |�[0m time="2021-05-12T23:40:13.980646230Z" level=info msg="Handling backend connection request [cc9d29cc51b1]"
�[36mserver_1  |�[0m time="2021-05-12T23:40:13.981209724Z" level=info msg="Running kubelet --address=0.0.0.0 --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=cgroupfs --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --cni-bin-dir=/bin --cni-conf-dir=/var/lib/rancher/k3s/agent/etc/cni/net.d --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --container-runtime=remote --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=cc9d29cc51b1 --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --node-labels= --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --read-only-port=0 --resolv-conf=/tmp/k3s-resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key"
�[36mserver_1  |�[0m time="2021-05-12T23:40:13.981946681Z" level=info msg="Running kube-proxy --cluster-cidr=10.42.0.0/16 --healthz-bind-address=127.0.0.1 --hostname-override=cc9d29cc51b1 --kubeconfig=/var/lib/rancher/k3s/agent/kubeproxy.kubeconfig --proxy-mode=iptables"
�[36mserver_1  |�[0m Flag --cloud-provider has been deprecated, will be removed in 1.23, in favor of removing cloud provider code from Kubelet.
�[36mserver_1  |�[0m Flag --cni-bin-dir has been deprecated, will be removed along with dockershim.
�[36mserver_1  |�[0m Flag --cni-conf-dir has been deprecated, will be removed along with dockershim.
�[36mserver_1  |�[0m Flag --containerd has been deprecated, This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.
�[36mserver_1  |�[0m W0512 23:40:13.982269       1 server.go:220] WARNING: all flags other than --config, --write-config-to, and --cleanup are deprecated. Please begin using a config file ASAP.
�[36mserver_1  |�[0m I0512 23:40:13.982615       1 server.go:436] "Kubelet version" kubeletVersion="v1.21.0+k3s1"
�[36mserver_1  |�[0m W0512 23:40:13.983144       1 proxier.go:653] Failed to read file /lib/modules/5.12.2-arch1-1/modules.builtin with error open /lib/modules/5.12.2-arch1-1/modules.builtin: no such file or directory. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
�[36mserver_1  |�[0m W0512 23:40:13.983549       1 proxier.go:663] Failed to load kernel module ip_vs with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
�[36mserver_1  |�[0m W0512 23:40:13.983914       1 proxier.go:663] Failed to load kernel module ip_vs_rr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
�[36mserver_1  |�[0m W0512 23:40:13.984279       1 proxier.go:663] Failed to load kernel module ip_vs_wrr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
�[36mserver_1  |�[0m W0512 23:40:13.984758       1 proxier.go:663] Failed to load kernel module ip_vs_sh with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
�[36mserver_1  |�[0m W0512 23:40:13.985067       1 proxier.go:663] Failed to load kernel module nf_conntrack with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
�[36mserver_1  |�[0m E0512 23:40:13.991873       1 node.go:161] Failed to retrieve node info: nodes "cc9d29cc51b1" is forbidden: User "system:kube-proxy" cannot get resource "nodes" in API group "" at the cluster scope
�[36mserver_1  |�[0m W0512 23:40:14.002120       1 manager.go:159] Cannot detect current cgroup on cgroup v2
�[36mserver_1  |�[0m I0512 23:40:14.002176       1 dynamic_cafile_content.go:167] Starting client-ca-bundle::/var/lib/rancher/k3s/agent/client-ca.crt
�[36mserver_1  |�[0m E0512 23:40:15.153711       1 node.go:161] Failed to retrieve node info: nodes "cc9d29cc51b1" is forbidden: User "system:kube-proxy" cannot get resource "nodes" in API group "" at the cluster scope
�[36mserver_1  |�[0m I0512 23:40:15.197013       1 dynamic_cafile_content.go:167] Starting request-header::/var/lib/rancher/k3s/server/tls/request-header-ca.crt
�[36mserver_1  |�[0m I0512 23:40:15.197034       1 dynamic_cafile_content.go:167] Starting client-ca-bundle::/var/lib/rancher/k3s/server/tls/client-ca.crt
�[36mserver_1  |�[0m I0512 23:40:15.197302       1 dynamic_serving_content.go:130] Starting serving-cert::/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt::/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key
�[36mserver_1  |�[0m I0512 23:40:15.197741       1 secure_serving.go:197] Serving securely on 127.0.0.1:6444
�[36mserver_1  |�[0m I0512 23:40:15.197799       1 autoregister_controller.go:141] Starting autoregister controller
�[36mserver_1  |�[0m I0512 23:40:15.197805       1 cache.go:32] Waiting for caches to sync for autoregister controller
�[36mserver_1  |�[0m I0512 23:40:15.197824       1 apf_controller.go:294] Starting API Priority and Fairness config controller
�[36mserver_1  |�[0m I0512 23:40:15.197853       1 controller.go:83] Starting OpenAPI AggregationController
�[36mserver_1  |�[0m I0512 23:40:15.197872       1 customresource_discovery_controller.go:209] Starting DiscoveryController
�[36mserver_1  |�[0m I0512 23:40:15.197906       1 controller.go:86] Starting OpenAPI controller
�[36mserver_1  |�[0m I0512 23:40:15.197923       1 tlsconfig.go:240] Starting DynamicServingCertificateController
�[36mserver_1  |�[0m I0512 23:40:15.197944       1 establishing_controller.go:76] Starting EstablishingController
�[36mserver_1  |�[0m I0512 23:40:15.197967       1 nonstructuralschema_controller.go:192] Starting NonStructuralSchemaConditionController
�[36mserver_1  |�[0m I0512 23:40:15.198012       1 apiapproval_controller.go:186] Starting KubernetesAPIApprovalPolicyConformantConditionController
�[36mserver_1  |�[0m I0512 23:40:15.198066       1 crd_finalizer.go:266] Starting CRDFinalizer
�[36mserver_1  |�[0m I0512 23:40:15.198099       1 crdregistration_controller.go:111] Starting crd-autoregister controller
�[36mserver_1  |�[0m I0512 23:40:15.198112       1 shared_informer.go:240] Waiting for caches to sync for crd-autoregister
�[36mserver_1  |�[0m I0512 23:40:15.197929       1 naming_controller.go:291] Starting NamingConditionController
�[36mserver_1  |�[0m I0512 23:40:15.198086       1 apiservice_controller.go:97] Starting APIServiceRegistrationController
�[36mserver_1  |�[0m I0512 23:40:15.198421       1 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller
�[36mserver_1  |�[0m I0512 23:40:15.198473       1 cluster_authentication_trust_controller.go:440] Starting cluster_authentication_trust_controller controller
�[36mserver_1  |�[0m I0512 23:40:15.198485       1 shared_informer.go:240] Waiting for caches to sync for cluster_authentication_trust_controller
�[36mserver_1  |�[0m I0512 23:40:15.198550       1 dynamic_cafile_content.go:167] Starting client-ca-bundle::/var/lib/rancher/k3s/server/tls/client-ca.crt
�[36mserver_1  |�[0m I0512 23:40:15.198562       1 dynamic_serving_content.go:130] Starting aggregator-proxy-cert::/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt::/var/lib/rancher/k3s/server/tls/client-auth-proxy.key
�[36mserver_1  |�[0m I0512 23:40:15.198573       1 dynamic_cafile_content.go:167] Starting request-header::/var/lib/rancher/k3s/server/tls/request-header-ca.crt
�[36mserver_1  |�[0m I0512 23:40:15.198664       1 available_controller.go:475] Starting AvailableConditionController
�[36mserver_1  |�[0m I0512 23:40:15.198700       1 cache.go:32] Waiting for caches to sync for AvailableConditionController controller
�[36mserver_1  |�[0m time="2021-05-12T23:40:15.210347639Z" level=info msg="Running cloud-controller-manager with args --profiling=false"
�[36mserver_1  |�[0m I0512 23:40:15.213794       1 controllermanager.go:142] Version: v1.21.0+k3s1
�[36mserver_1  |�[0m E0512 23:40:15.230560       1 controller.go:156] Unable to remove old endpoints from kubernetes service: no master IPs were listed in storage, refusing to erase all endpoints for the kubernetes service
�[36mserver_1  |�[0m I0512 23:40:15.291491       1 shared_informer.go:247] Caches are synced for node_authorizer 
�[36mserver_1  |�[0m I0512 23:40:15.298748       1 cache.go:39] Caches are synced for AvailableConditionController controller
�[36mserver_1  |�[0m I0512 23:40:15.298783       1 cache.go:39] Caches are synced for APIServiceRegistrationController controller
�[36mserver_1  |�[0m I0512 23:40:15.298920       1 shared_informer.go:247] Caches are synced for cluster_authentication_trust_controller 
�[36mserver_1  |�[0m I0512 23:40:15.298937       1 shared_informer.go:247] Caches are synced for crd-autoregister 
�[36mserver_1  |�[0m I0512 23:40:15.298944       1 apf_controller.go:299] Running API Priority and Fairness config worker
�[36mserver_1  |�[0m I0512 23:40:15.298947       1 cache.go:39] Caches are synced for autoregister controller
�[36mserver_1  |�[0m I0512 23:40:16.197233       1 controller.go:132] OpenAPI AggregationController: action for item : Nothing (removed from the queue).
�[36mserver_1  |�[0m I0512 23:40:16.197256       1 controller.go:132] OpenAPI AggregationController: action for item k8s_internal_local_delegation_chain_0000000000: Nothing (removed from the queue).
�[36mserver_1  |�[0m I0512 23:40:16.200701       1 storage_scheduling.go:148] all system priority classes are created successfully or already exist.
�[36mserver_1  |�[0m W0512 23:40:16.952253       1 handler_proxy.go:102] no RequestInfo found in the context
�[36mserver_1  |�[0m E0512 23:40:16.952343       1 controller.go:116] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
�[36mserver_1  |�[0m , Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
�[36mserver_1  |�[0m I0512 23:40:16.952350       1 controller.go:129] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
�[36mserver_1  |�[0m time="2021-05-12T23:40:16.995129015Z" level=info msg="Waiting for node cc9d29cc51b1 CIDR not assigned yet"
�[36mserver_1  |�[0m time="2021-05-12T23:40:17.206993453Z" level=info msg="Kube API server is now running"
�[36mserver_1  |�[0m time="2021-05-12T23:40:17.207079050Z" level=info msg="k3s is up and running"
�[36mserver_1  |�[0m Flag --address has been deprecated, see --bind-address instead.
�[36mserver_1  |�[0m I0512 23:40:17.209646       1 controllermanager.go:175] Version: v1.21.0+k3s1
�[36mserver_1  |�[0m I0512 23:40:17.209928       1 deprecated_insecure_serving.go:53] Serving insecurely on 127.0.0.1:10252
�[36mserver_1  |�[0m W0512 23:40:17.219056       1 authorization.go:47] Authorization is disabled
�[36mserver_1  |�[0m W0512 23:40:17.219111       1 authentication.go:47] Authentication is disabled
�[36mserver_1  |�[0m I0512 23:40:17.219121       1 deprecated_insecure_serving.go:51] Serving healthz insecurely on 127.0.0.1:10251
�[36mserver_1  |�[0m time="2021-05-12T23:40:17.224219692Z" level=info msg="Writing static file: /var/lib/rancher/k3s/server/static/charts/traefik-9.18.2.tgz"
�[36mserver_1  |�[0m time="2021-05-12T23:40:17.224411638Z" level=info msg="Writing static file: /var/lib/rancher/k3s/server/static/charts/traefik-crd-9.18.2.tgz"
�[36mserver_1  |�[0m time="2021-05-12T23:40:17.249188330Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/coredns.yaml"
�[36mserver_1  |�[0m time="2021-05-12T23:40:17.249485379Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/local-storage.yaml"
�[36mserver_1  |�[0m time="2021-05-12T23:40:17.249639037Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-deployment.yaml"
�[36mserver_1  |�[0m time="2021-05-12T23:40:17.249734598Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-service.yaml"
�[36mserver_1  |�[0m time="2021-05-12T23:40:17.249897926Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/resource-reader.yaml"
�[36mserver_1  |�[0m time="2021-05-12T23:40:17.250035917Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/traefik.yaml"
�[36mserver_1  |�[0m time="2021-05-12T23:40:17.250166210Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/ccm.yaml"
�[36mserver_1  |�[0m time="2021-05-12T23:40:17.250254727Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/auth-delegator.yaml"
�[36mserver_1  |�[0m time="2021-05-12T23:40:17.250344650Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/auth-reader.yaml"
�[36mserver_1  |�[0m time="2021-05-12T23:40:17.250437817Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/metrics-apiservice.yaml"
�[36mserver_1  |�[0m time="2021-05-12T23:40:17.250536442Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/rolebindings.yaml"
�[36mserver_1  |�[0m time="2021-05-12T23:40:17.250625616Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/aggregated-metrics-reader.yaml"
�[36mserver_1  |�[0m I0512 23:40:17.347828       1 node.go:172] Successfully retrieved node IP: 172.18.0.2
�[36mserver_1  |�[0m I0512 23:40:17.347847       1 server_others.go:141] Detected node IP 172.18.0.2
�[36mserver_1  |�[0m I0512 23:40:17.349755       1 server_others.go:207] kube-proxy running in dual-stack mode, IPv4-primary
�[36mserver_1  |�[0m I0512 23:40:17.349774       1 server_others.go:213] Using iptables Proxier.
�[36mserver_1  |�[0m I0512 23:40:17.349784       1 server_others.go:220] creating dualStackProxier for iptables.
�[36mserver_1  |�[0m W0512 23:40:17.349812       1 server_others.go:513] detect-local-mode set to ClusterCIDR, but no IPv6 cluster CIDR defined, , defaulting to no-op detect-local for IPv6
�[36mserver_1  |�[0m I0512 23:40:17.350167       1 server.go:643] Version: v1.21.0+k3s1
�[36mserver_1  |�[0m time="2021-05-12T23:40:17.350859169Z" level=info msg="Starting k3s.cattle.io/v1, Kind=Addon controller"
�[36mserver_1  |�[0m time="2021-05-12T23:40:17.353760702Z" level=info msg="Cluster dns configmap already exists"
�[36mserver_1  |�[0m I0512 23:40:17.354804       1 conntrack.go:52] Setting nf_conntrack_max to 524288
�[36mserver_1  |�[0m I0512 23:40:17.355017       1 conntrack.go:83] Setting conntrack hashsize to 131072
�[36mserver_1  |�[0m E0512 23:40:17.355077       1 conntrack.go:85] failed to set conntrack hashsize to 131072: write /sys/module/nf_conntrack/parameters/hashsize: operation not supported
�[36mserver_1  |�[0m I0512 23:40:17.355132       1 conntrack.go:103] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
�[36mserver_1  |�[0m I0512 23:40:17.355255       1 conntrack.go:103] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
�[36mserver_1  |�[0m I0512 23:40:17.355392       1 config.go:315] Starting service config controller
�[36mserver_1  |�[0m I0512 23:40:17.355401       1 shared_informer.go:240] Waiting for caches to sync for service config
�[36mserver_1  |�[0m I0512 23:40:17.355419       1 config.go:224] Starting endpoint slice config controller
�[36mserver_1  |�[0m I0512 23:40:17.355427       1 shared_informer.go:240] Waiting for caches to sync for endpoint slice config
�[36mserver_1  |�[0m I0512 23:40:17.367238       1 controller.go:611] quota admission added evaluator for: addons.k3s.cattle.io
�[36mserver_1  |�[0m I0512 23:40:17.405039       1 controller.go:611] quota admission added evaluator for: deployments.apps
�[36mserver_1  |�[0m time="2021-05-12T23:40:17.451971745Z" level=info msg="Starting /v1, Kind=Pod controller"
�[36mserver_1  |�[0m time="2021-05-12T23:40:17.451970333Z" level=info msg="Starting /v1, Kind=Node controller"
�[36mserver_1  |�[0m time="2021-05-12T23:40:17.452034368Z" level=info msg="Starting batch/v1, Kind=Job controller"
�[36mserver_1  |�[0m time="2021-05-12T23:40:17.452039312Z" level=info msg="Starting /v1, Kind=Endpoints controller"
�[36mserver_1  |�[0m time="2021-05-12T23:40:17.451972878Z" level=info msg="Starting helm.cattle.io/v1, Kind=HelmChartConfig controller"
�[36mserver_1  |�[0m time="2021-05-12T23:40:17.452055018Z" level=info msg="Starting /v1, Kind=Service controller"
�[36mserver_1  |�[0m time="2021-05-12T23:40:17.452032296Z" level=info msg="Starting helm.cattle.io/v1, Kind=HelmChart controller"
�[36mserver_1  |�[0m I0512 23:40:17.455742       1 shared_informer.go:247] Caches are synced for endpoint slice config 
�[36mserver_1  |�[0m I0512 23:40:17.455781       1 shared_informer.go:247] Caches are synced for service config 
�[36mserver_1  |�[0m I0512 23:40:17.459420       1 controller.go:611] quota admission added evaluator for: helmcharts.helm.cattle.io
�[36mserver_1  |�[0m E0512 23:40:17.493966       1 proxier.go:1612] "Failed to execute iptables-restore" err="exit status 2 (ip6tables-restore v1.8.5 (nf_tables): unknown option \"--random-fully\"\nError occurred at line: 18\nTry `ip6tables-restore -h' or 'ip6tables-restore --help' for more information.\n)"
�[36mserver_1  |�[0m I0512 23:40:17.493982       1 proxier.go:859] "Sync failed" retryingTime="30s"
�[36mserver_1  |�[0m time="2021-05-12T23:40:17.588125596Z" level=info msg="Starting /v1, Kind=Secret controller"
�[36mserver_1  |�[0m time="2021-05-12T23:40:18.980108233Z" level=info msg="Stopped tunnel to 127.0.0.1:6443"
�[36mserver_1  |�[0m time="2021-05-12T23:40:18.980143418Z" level=info msg="Proxy done" err="context canceled" url="wss://127.0.0.1:6443/v1-k3s/connect"
�[36mserver_1  |�[0m time="2021-05-12T23:40:18.980145649Z" level=info msg="Connecting to proxy" url="wss://172.18.0.2:6443/v1-k3s/connect"
�[36mserver_1  |�[0m time="2021-05-12T23:40:18.980257747Z" level=info msg="error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF"
�[36mserver_1  |�[0m time="2021-05-12T23:40:18.982751179Z" level=info msg="Handling backend connection request [cc9d29cc51b1]"
�[36mserver_1  |�[0m time="2021-05-12T23:40:19.000672810Z" level=info msg="Waiting for node cc9d29cc51b1 CIDR not assigned yet"
�[36mserver_1  |�[0m W0512 23:40:19.026769       1 info.go:53] Couldn't collect info from any of the files in "/etc/machine-id,/var/lib/dbus/machine-id"
�[36mserver_1  |�[0m I0512 23:40:19.042530       1 server.go:660] "--cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /"
�[36mserver_1  |�[0m I0512 23:40:19.042703       1 container_manager_linux.go:291] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
�[36mserver_1  |�[0m I0512 23:40:19.042762       1 container_manager_linux.go:296] "Creating Container Manager object based on Node Config" nodeConfig={RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:remote CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:cgroupfs KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: ReservedSystemCPUs: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:imagefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>}]} QOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalTopologyManagerScope:container ExperimentalCPUManagerReconcilePeriod:10s ExperimentalMemoryManagerPolicy:None ExperimentalMemoryManagerReservedMemory:[] ExperimentalPodPidsLimit:-1 EnforceCPULimits:true CPUCFSQuotaPeriod:100ms ExperimentalTopologyManagerPolicy:none Rootless:false}
�[36mserver_1  |�[0m I0512 23:40:19.042784       1 topology_manager.go:120] "Creating topology manager with policy per scope" topologyPolicyName="none" topologyScopeName="container"
�[36mserver_1  |�[0m I0512 23:40:19.042795       1 container_manager_linux.go:327] "Initializing Topology Manager" policy="none" scope="container"
�[36mserver_1  |�[0m I0512 23:40:19.042803       1 container_manager_linux.go:332] "Creating device plugin manager" devicePluginEnabled=true
�[36mserver_1  |�[0m I0512 23:40:19.043009       1 kubelet.go:275] "Adding static pod path" path="/var/lib/rancher/k3s/agent/pod-manifests"
�[36mserver_1  |�[0m I0512 23:40:19.043035       1 kubelet.go:286] "Watching apiserver"
�[36mserver_1  |�[0m I0512 23:40:19.043115       1 kubelet.go:465] "Kubelet client is not nil"
�[36mserver_1  |�[0m I0512 23:40:19.043812       1 kuberuntime_manager.go:222] "Container runtime initialized" containerRuntime="containerd" version="v1.4.4-k3s1" apiVersion="v1alpha2"
�[36mserver_1  |�[0m I0512 23:40:19.044170       1 server.go:1191] "Started kubelet"
�[36mserver_1  |�[0m I0512 23:40:19.044427       1 server.go:149] "Starting to listen" address="0.0.0.0" port=10250
�[36mserver_1  |�[0m E0512 23:40:19.044708       1 cri_stats_provider.go:369] "Failed to get the info of the filesystem with mountpoint" err="unable to find data in memory cache" mountpoint="/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs"
�[36mserver_1  |�[0m E0512 23:40:19.044734       1 kubelet.go:1313] "Image garbage collection failed once. Stats initialization may not have completed yet" err="invalid capacity 0 on image filesystem"
�[36mserver_1  |�[0m I0512 23:40:19.045064       1 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
�[36mserver_1  |�[0m I0512 23:40:19.045173       1 volume_manager.go:271] "Starting Kubelet Volume Manager"
�[36mserver_1  |�[0m I0512 23:40:19.045218       1 desired_state_of_world_populator.go:141] "Desired state populator starts to run"
�[36mserver_1  |�[0m I0512 23:40:19.045468       1 server.go:405] "Adding debug handlers to kubelet server"
�[36mserver_1  |�[0m I0512 23:40:19.050354       1 cpu_manager.go:199] "Starting CPU manager" policy="none"
�[36mserver_1  |�[0m I0512 23:40:19.050373       1 cpu_manager.go:200] "Reconciling" reconcilePeriod="10s"
�[36mserver_1  |�[0m I0512 23:40:19.050391       1 state_mem.go:36] "Initialized new in-memory state store"
�[36mserver_1  |�[0m I0512 23:40:19.050523       1 state_mem.go:88] "Updated default CPUSet" cpuSet=""
�[36mserver_1  |�[0m I0512 23:40:19.050536       1 state_mem.go:96] "Updated CPUSet assignments" assignments=map[]
�[36mserver_1  |�[0m I0512 23:40:19.050541       1 policy_none.go:44] "None policy: Start"
�[36mserver_1  |�[0m I0512 23:40:19.052285       1 controller.go:611] quota admission added evaluator for: leases.coordination.k8s.io
�[36mserver_1  |�[0m I0512 23:40:19.058924       1 kubelet_network_linux.go:56] "Initialized protocol iptables rules." protocol=IPv4
�[36mserver_1  |�[0m E0512 23:40:19.069484       1 kubelet_network_linux.go:167] "Failed to ensure SNAT rule for packets marked by KUBE-MARK-MASQ chain in nat table KUBE-POSTROUTING chain" err="error checking rule: exit status 2: ip6tables v1.8.5 (nf_tables): unknown option \"--random-fully\"\nTry `ip6tables -h' or 'ip6tables --help' for more information.\n"
�[36mserver_1  |�[0m I0512 23:40:19.069503       1 kubelet_network_linux.go:64] "Failed to initialize protocol iptables rules; some functionality may be missing." protocol=IPv6
�[36mserver_1  |�[0m I0512 23:40:19.069515       1 status_manager.go:157] "Starting to sync pod status with apiserver"
�[36mserver_1  |�[0m I0512 23:40:19.069533       1 kubelet.go:1853] "Starting kubelet main sync loop"
�[36mserver_1  |�[0m E0512 23:40:19.069578       1 kubelet.go:1877] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
�[36mserver_1  |�[0m I0512 23:40:19.069891       1 request.go:668] Waited for 1.048577221s due to client-side throttling, not priority and fairness, request: GET:https://127.0.0.1:6444/apis/discovery.k8s.io/v1?timeout=32s
�[36mserver_1  |�[0m E0512 23:40:19.075499       1 node_container_manager_linux.go:57] "Failed to create cgroup" err="cannot enter cgroupv2 \"/sys/fs/cgroup/kubepods\" with domain controllers -- it is in an invalid state" cgroupName=[kubepods]
�[36mserver_1  |�[0m E0512 23:40:19.075518       1 kubelet.go:1391] "Failed to start ContainerManager" err="cannot enter cgroupv2 \"/sys/fs/cgroup/kubepods\" with domain controllers -- it is in an invalid state"
�[36mk3s_server_1 exited with code 1
�[0m

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[support] Trouble getting started with the example docker-compose.yml file #3324

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 0 comments

Select a reply

[support] Trouble getting started with the example docker-compose.yml file #3324

fubarhouse May 13, 2021

Replies: 0 comments

fubarhouse
May 13, 2021