Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

modprobe in hardened-kubernetes image is unusable due to missing shared libraries #7444

Closed
shalomjacob opened this issue Dec 20, 2024 · 4 comments
Closed

modprobe in hardened-kubernetes image is unusable due to missing shared libraries #7444

shalomjacob opened this issue Dec 20, 2024 · 4 comments
Assignees
@brandond @VestigeJ @rafaelbreno
Labels
kind/bug Something isn't working
Milestone
2025-01 Release C...

Comments

@shalomjacob
Copy link

Environmental Info:
RKE2 Version: v1.28.15+rke2r1

Node(s) CPU architecture, OS, and Version: Ubuntu 22.04
Linux ip-172-31-6-165 5.15.0-1028-aws #32-Ubuntu SMP Mon Jan 9 12:28:07 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:
1 Server node

Describe the bug:
trying to run kube-proxy in IPVS mode. However, we see the following warning in the kube-proxy logs:
time="2024-12-18T19:52:07Z" level=warning msg="Running modprobe ip_vs failed with message: modprobe: error while loading shared libraries: libzstd.so.1: cannot open shared object file: No such file or directory, error: exit status 127"

Steps To Reproduce:
Setup single node RKE2 cluster with Kubernetes v1.28.15+rke2r1. Set arguments for kube-proxy to run in ipvs.

kube-proxy-arg:
- proxy-mode=ipvs
- ipvs-scheduler=lc
- ipvs-strict-arp=true
kube-proxy-extra-mount:
- /usr/lib/:/usr/lib/

Once RKE2 starts up, kube-proxy logs have warning for missing libzstd.so.1 library.

kubectl logs kube-proxy-ip-172-31-6-165 -n kube-system
I1219 19:37:02.953843       1 node.go:141] Successfully retrieved node IP: 172.31.6.165
I1219 19:37:02.995881       1 server.go:632] "kube-proxy running in dual-stack mode" primary ipFamily="IPv4"
time="2024-12-19T19:37:02Z" level=warning msg="Running modprobe ip_vs failed with message: `modprobe: error while loading shared libraries: libzstd.so.1: cannot open shared object file: No such file or directory`, error: exit status 127"
I1219 19:37:03.003876       1 server_others.go:218] "Using ipvs Proxier"
I1219 19:37:03.003907       1 server_others.go:421] "Detect-local-mode set to ClusterCIDR, but no cluster CIDR for family" ipFamily="IPv6"
I1219 19:37:03.003913       1 server_others.go:438] "Defaulting to no-op detect-local"
I1219 19:37:03.004267       1 ipset.go:116] "Ipset name truncated" ipSetName="KUBE-6-LOAD-BALANCER-SOURCE-CIDR" truncatedName="KUBE-6-LOAD-BALANCER-SOURCE-CID"
I1219 19:37:03.004281       1 ipset.go:116] "Ipset name truncated" ipSetName="KUBE-6-NODE-PORT-LOCAL-SCTP-HASH" truncatedName="KUBE-6-NODE-PORT-LOCAL-SCTP-HAS"
I1219 19:37:03.004305       1 server.go:846] "Version info" version="v1.28.15+rke2r1"
I1219 19:37:03.004312       1 server.go:848] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
I1219 19:37:03.008236       1 config.go:188] "Starting service config controller"
I1219 19:37:03.008267       1 shared_informer.go:311] Waiting for caches to sync for service config
I1219 19:37:03.008298       1 config.go:97] "Starting endpoint slice config controller"
I1219 19:37:03.008307       1 shared_informer.go:311] Waiting for caches to sync for endpoint slice config
I1219 19:37:03.008314       1 config.go:315] "Starting node config controller"
I1219 19:37:03.008325       1 shared_informer.go:311] Waiting for caches to sync for node config
I1219 19:37:03.109390       1 shared_informer.go:318] Caches are synced for node config
I1219 19:37:03.109552       1 shared_informer.go:318] Caches are synced for endpoint slice config
I1219 19:37:03.109568       1 shared_informer.go:318] Caches are synced for service config
I1219 20:43:03.042786       1 graceful_termination.go:102] "Removed real server from graceful delete real server list" realServer="10.43.248.141:443/TCP/10.42.0.74:444"
I1219 20:43:03.043096       1 graceful_termination.go:102] "Removed real server from graceful delete real server list" realServer="10.43.248.141:443/TCP/10.42.0.75:444"
I1219 20:49:03.047448       1 graceful_termination.go:102] "Removed real server from graceful delete real server list" realServer="10.43.95.181:8080/TCP/10.42.0.109:8080"
I1219 20:51:03.049130       1 graceful_termination.go:102] "Removed real server from graceful delete real server list" realServer="10.43.181.243:8080/TCP/10.42.0.113:8080"
I1219 20:51:03.049182       1 graceful_termination.go:102] "Removed real server from graceful delete real server list" realServer="10.43.13.77:8080/TCP/10.42.0.111:8080"
I1219 20:51:03.049223       1 graceful_termination.go:102] "Removed real server from graceful delete real server list" realServer="10.43.203.157:9090/TCP/10.42.0.119:8081"
I1219 20:51:03.049276       1 graceful_termination.go:102] "Removed real server from graceful delete real server list" realServer="10.43.229.178:8080/TCP/10.42.0.114:8080"

Expected behavior:
No warning in the kube-proxy logs for missing libzstd.so.1 library

Actual behavior:
Warning message exists.

Additional context / logs:
The kube-proxy logs the warning but does not crash and stays running.
@brandond
Copy link
Member

We’re copying over modprobe without copying over any of the shared libraries it needs.
https://github.com/rancher/image-build-kubernetes/blob/v1.30.6-rke2r1-build20241023/Dockerfile#L84-L88

brandond@dev01:~$ docker run --rm -it docker.io/rancher/hardened-kubernetes:v1.30.6-rke2r1-build20241023 modprobe
modprobe: error while loading shared libraries: libzstd.so.1: cannot open shared object file: No such file or directory

We need to be more careful about copying things out of BCI, they're not statically linked.

@brandond
Copy link
Member

brandond commented Dec 20, 2024 •
edited
Loading

cc @rafaelbreno @brooksn

@brandond brandond added the kind/bug Something isn't working label Dec 20, 2024
@brandond brandond added this to the 2025-01 Release Cycle milestone Dec 20, 2024
@brandond brandond self-assigned this Dec 20, 2024
@brandond brandond changed the title kube-proxy in IPVS mode has warning: modprobe: error while loading shared libraries: libzstd.so.1 modprobe in hardened-kubernetes image is unusable due to missing shared libraries Dec 20, 2024
@brandond brandond mentioned this issue Dec 23, 2024
Merged
@VestigeJ VestigeJ modified the milestone: 2025-01 Release Cycle Jan 17, 2025
@VestigeJ
Copy link
Contributor

Sorry my finger has a stint on it and it's accidentally closed/changed a few things like milestones on accident this week.

@VestigeJ
Copy link
Contributor

##Environment Details
Reproduced using VERSION=v1.32.0+rke2r1
Validated using VERSION=v1.29.13-rc3+rke2r1 && VERSION=v1.30.9-rc3+rke2r1

Infrastructure

  • Cloud

Node(s) CPU architecture, OS, and version:

Linux 6.4.0-150600.23.17-default x86_64 GNU/Linux
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP6"

Cluster Configuration:

NAME              STATUS   ROLES                       AGE     VERSION
ip                Ready    control-plane,etcd,master   5m17s   v1.32.0+rke2r1 //reproduction version

Config.yaml:

write-kubeconfig-mode: 644
debug: true
token: YOUR_TOKEN_HERE
cni: multus,cilium
profile: cis
selinux: true
node-external-ip: 1.3.2.3
kube-proxy-arg:
- proxy-mode=ipvs
- ipvs-scheduler=lc
- ipvs-strict-arp=true
kube-proxy-extra-mount:
- /usr/lib/:/usr/lib/

YOUR_REPRODUCED_RESULTS_HERE 

$ curl https://get.rke2.io --output install-"rke2".sh
$ sudo chmod +x install-"rke2".sh
$ sudo groupadd --system etcd && sudo useradd -s /sbin/nologin --system -g etcd etcd
$ sudo modprobe ip_vs_rr
$ sudo modprobe ip_vs_wrr
$ sudo modprobe ip_vs_sh
$ sudo printf "on_oovm.panic_on_oom=0 \nvm.overcommit_memory=1 \nkernel.panic=10 \nkernel.panic_ps=1 \nkernel.panic_on_oops=1 \n" > ~/60-rke2-cis.conf
$ sudo cp 60-rke2-cis.conf /etc/sysctl.d/
$ sudo systemctl restart systemd-sysctl
$ VERSION=v1.32.0+rke2r1
$ sudo INSTALL_RKE2_VERSION=$VERSION INSTALL_RKE2_EXEC=server ./install-rke2.sh
$ sudo systemctl start rke2-server
$ export KUBECONFIG=/etc/rancher/rke2/rke2.yaml
$ watch -n 3 kubectl get pods -A
$ k logs kube-proxy-ip-1-3-2-3 -n kube-system

Results:

$ k logs kube-proxy-ip -n kube-system

E0117 21:30:58.475338       1 proxier.go:733] "Error cleaning up nftables rules" err="could not find nftables binary: exec: \"nft\": executable file not found in $PATH"
E0117 21:30:58.478123       1 proxier.go:733] "Error cleaning up nftables rules" err="could not find nftables binary: exec: \"nft\": executable file not found in $PATH"
E0117 21:30:58.481184       1 server.go:687] "Failed to retrieve node info" err="Get \"https://127.0.0.1:6443/api/v1/nodes/ip-1-3-2-3\": dial tcp 127.0.0.1:6443: connect: connection refused"
E0117 21:30:59.527292       1 server.go:687] "Failed to retrieve node info" err="Get \"https://127.0.0.1:6443/api/v1/nodes/ip-1-3-2-3\": dial tcp 127.0.0.1:6443: connect: connection refused"
E0117 21:31:01.610526       1 server.go:687] "Failed to retrieve node info" err="Get \"https://127.0.0.1:6443/api/v1/nodes/ip-1-3-2-3\": dial tcp 127.0.0.1:6443: connect: connection refused"
E0117 21:31:05.907794       1 server.go:687] "Failed to retrieve node info" err="Get \"https://127.0.0.1:6443/api/v1/nodes/ip-1-3-2-3\": dial tcp 127.0.0.1:6443: connect: connection refused"
E0117 21:31:15.503877       1 server.go:687] "Failed to retrieve node info" err="Get \"https://127.0.0.1:6443/api/v1/nodes/ip-1-3-2-3\": dial tcp 127.0.0.1:6443: connect: connection refused"
E0117 21:31:31.740567       1 server.go:687] "Failed to retrieve node info" err="nodes \"ip-1-3-2-3\" is forbidden: User \"system:kube-proxy\" cannot get resource \"nodes\" in API group \"\" at the cluster scope"
I0117 21:31:31.740607       1 server.go:667] "Can't determine this node's IP, assuming loopback; if this is incorrect, please set the --bind-address flag"
E0117 21:31:31.740649       1 server.go:234] "Kube-proxy configuration may be incomplete or incorrect" err="nodePortAddresses is unset; NodePort connections will be accepted on all local IPs. Consider using `--nodeport-addresses primary`"
I0117 21:31:31.998016       1 server.go:243] "kube-proxy running in dual-stack mode" primary ipFamily="IPv4"
time="2025-01-17T21:31:31Z" level=warning msg="Running modprobe ip_vs failed with message: `modprobe: error while loading shared libraries: libzstd.so.1: cannot open shared object file: No such file or directory`, error: exit status 127"
I0117 21:31:32.054380       1 server_linux.go:231] "Using ipvs Proxier"
I0117 21:31:32.062356       1 ipset.go:119] "Ipset name truncated" ipSetName="KUBE-6-LOAD-BALANCER-SOURCE-CIDR" truncatedName="KUBE-6-LOAD-BALANCER-SOURCE-CID"
I0117 21:31:32.068894       1 ipset.go:119] "Ipset name truncated" ipSetName="KUBE-6-NODE-PORT-LOCAL-SCTP-HASH" truncatedName="KUBE-6-NODE-PORT-LOCAL-SCTP-HAS"
I0117 21:31:32.068956       1 server.go:497] "Version info" version="v1.32.0+rke2r1"
I0117 21:31:32.068968       1 server.go:499] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
I0117 21:31:32.070802       1 config.go:199] "Starting service config controller"
I0117 21:31:32.070836       1 shared_informer.go:313] Waiting for caches to sync for service config
I0117 21:31:32.070879       1 config.go:105] "Starting endpoint slice config controller"
I0117 21:31:32.070885       1 shared_informer.go:313] Waiting for caches to sync for endpoint slice config
I0117 21:31:32.071382       1 config.go:329] "Starting node config controller"
I0117 21:31:32.071389       1 shared_informer.go:313] Waiting for caches to sync for node config
I0117 21:31:32.171237       1 shared_informer.go:320] Caches are synced for endpoint slice config
I0117 21:31:32.171281       1 shared_informer.go:320] Caches are synced for service config
I0117 21:31:32.171508       1 shared_informer.go:320] Caches are synced for node config

$ k logs kube-proxy-ip-1-3-2-3 -n kube-system

E0117 22:58:29.810611       1 proxier.go:733] "Error cleaning up nftables rules" err="could not find nftables binary: exec: \"nft\": executable file not found in $PATH"
E0117 22:58:29.810780       1 proxier.go:733] "Error cleaning up nftables rules" err="could not find nftables binary: exec: \"nft\": executable file not found in $PATH"
E0117 22:58:29.822625       1 server.go:687] "Failed to retrieve node info" err="Get \"https://127.0.0.1:6443/api/v1/nodes/ip-1-3-2-3\": dial tcp 127.0.0.1:6443: connect: connection refused"
E0117 22:58:30.922090       1 server.go:687] "Failed to retrieve node info" err="Get \"https://127.0.0.1:6443/api/v1/nodes/ip-1-3-2-3\": dial tcp 127.0.0.1:6443: connect: connection refused"
E0117 22:58:32.948909       1 server.go:687] "Failed to retrieve node info" err="Get \"https://127.0.0.1:6443/api/v1/nodes/ip-1-3-2-3\": dial tcp 127.0.0.1:6443: connect: connection refused"
E0117 22:58:38.312312       1 server.go:687] "Failed to retrieve node info" err="nodes \"ip-1-3-2-3\" is forbidden: User \"system:kube-proxy\" cannot get resource \"nodes\" in API group \"\" at the cluster scope"
I0117 22:58:46.891038       1 server.go:698] "Successfully retrieved node IP(s)" IPs=["1.3.2.3"]
E0117 22:58:46.891108       1 server.go:234] "Kube-proxy configuration may be incomplete or incorrect" err="nodePortAddresses is unset; NodePort connections will be accepted on all local IPs. Consider using `--nodeport-addresses primary`"
I0117 22:58:47.102001       1 server.go:243] "kube-proxy running in dual-stack mode" primary ipFamily="IPv4"
time="2025-01-17T22:58:47Z" level=warning msg="Running modprobe ip_vs failed with message: `modprobe: WARNING: Module ip_vs not found in directory /lib/modules/6.4.0-150600.23.17-default`, error: exit status 1"
I0117 22:58:47.121059       1 server_linux.go:231] "Using ipvs Proxier"
I0117 22:58:47.131811       1 ipset.go:119] "Ipset name truncated" ipSetName="KUBE-6-LOAD-BALANCER-SOURCE-CIDR" truncatedName="KUBE-6-LOAD-BALANCER-SOURCE-CID"
I0117 22:58:47.131839       1 ipset.go:119] "Ipset name truncated" ipSetName="KUBE-6-NODE-PORT-LOCAL-SCTP-HASH" truncatedName="KUBE-6-NODE-PORT-LOCAL-SCTP-HAS"
I0117 22:58:47.131878       1 server.go:497] "Version info" version="v1.32.1+rke2r1"
I0117 22:58:47.131888       1 server.go:499] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
I0117 22:58:47.141489       1 config.go:199] "Starting service config controller"
I0117 22:58:47.141525       1 shared_informer.go:313] Waiting for caches to sync for service config
I0117 22:58:47.141556       1 config.go:105] "Starting endpoint slice config controller"
I0117 22:58:47.141562       1 shared_informer.go:313] Waiting for caches to sync for endpoint slice config
I0117 22:58:47.142432       1 config.go:329] "Starting node config controller"
I0117 22:58:47.142442       1 shared_informer.go:313] Waiting for caches to sync for node config
I0117 22:58:47.242738       1 shared_informer.go:320] Caches are synced for node config
I0117 22:58:47.243408       1 shared_informer.go:320] Caches are synced for service config
I0117 22:58:47.243484       1 shared_informer.go:320] Caches are synced for endpoint slice config

$ k logs kube-proxy-ip -n kube-system | grep -i librar

time="2025-01-24T00:25:26Z" level=warning msg="Running modprobe ip_vs failed with message: `modprobe: error while loading shared libraries: libzstd.so.1: cannot open shared object file: No such file or directory`, error: exit status 127"

$ kgn

NAME               STATUS   ROLES                       AGE   VERSION
ip                 Ready    control-plane,etcd,master   10m   v1.29.12+rke2r1

$ k logs kube-proxy-ip -n kube-system

I0124 00:45:29.303797       1 server.go:1050] "Successfully retrieved node IP(s)" IPs=["172.31.18.133"]
I0124 00:45:29.340376       1 server.go:652] "kube-proxy running in dual-stack mode" primary ipFamily="IPv4"
time="2025-01-24T00:45:29Z" level=warning msg="Running modprobe ip_vs failed with message: `modprobe: WARNING: Module ip_vs not found in directory /lib/modules/6.4.0-150600.23.17-default`, error: exit status 1"
I0124 00:45:29.345169       1 server_others.go:236] "Using ipvs Proxier"
I0124 00:45:29.346248       1 server_others.go:512] "Detect-local-mode set to ClusterCIDR, but no cluster CIDR for family" ipFamily="IPv6"
I0124 00:45:29.346298       1 server_others.go:529] "Defaulting to no-op detect-local"
I0124 00:45:29.346649       1 ipset.go:116] "Ipset name truncated" ipSetName="KUBE-6-LOAD-BALANCER-SOURCE-CIDR" truncatedName="KUBE-6-LOAD-BALANCER-SOURCE-CID"
I0124 00:45:29.346779       1 ipset.go:116] "Ipset name truncated" ipSetName="KUBE-6-NODE-PORT-LOCAL-SCTP-HASH" truncatedName="KUBE-6-NODE-PORT-LOCAL-SCTP-HAS"
I0124 00:45:29.347027       1 server.go:865] "Version info" version="v1.29.13+rke2r1"
I0124 00:45:29.347092       1 server.go:867] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
I0124 00:45:29.348062       1 config.go:188] "Starting service config controller"
I0124 00:45:29.348247       1 shared_informer.go:311] Waiting for caches to sync for service config
I0124 00:45:29.348342       1 config.go:97] "Starting endpoint slice config controller"
I0124 00:45:29.348376       1 shared_informer.go:311] Waiting for caches to sync for endpoint slice config
I0124 00:45:29.348968       1 config.go:315] "Starting node config controller"
I0124 00:45:29.349055       1 shared_informer.go:311] Waiting for caches to sync for node config
I0124 00:45:29.448629       1 shared_informer.go:318] Caches are synced for service config
I0124 00:45:29.450015       1 shared_informer.go:318] Caches are synced for node config
I0124 00:45:29.451497       1 shared_informer.go:318] Caches are synced for endpoint slice config

$ sudo lsmod | grep -i ip_vs

ip_vs_lc               12288  5
ip_vs_sh               12288  0
ip_vs_wrr              12288  0
ip_vs_rr               12288  0
ip_vs                 237568  13 ip_vs_rr,ip_vs_sh,ip_vs_wrr,ip_vs_lc
nf_conntrack          204800  6 xt_conntrack,nf_nat,xt_nat,nf_conntrack_netlink,xt_MASQUERADE,ip_vs
nf_defrag_ipv6         24576  2 nf_conntrack,ip_vs
libcrc32c              12288  5 nf_conntrack,nf_nat,nf_tables,xfs,ip_vs

$ ls /lib/modules/6.4.0-150600.23.17-default/

kernel  modules.alias  modules.alias.bin  modules.builtin  modules.builtin.alias.bin  modules.builtin.bin  modules.builtin.modinfo  modules.dep  modules.dep.bin  modules.devname  modules.fips  modules.order  modules.softdep  modules.symbols  modules.symbols.bin

$ ls /lib/modules/6.4.0-150600.23.17-default/kernel/

arch  block  crypto  drivers  fs  lib  mm  net  sound  virt

$ k logs pod/kube-proxy-ip -n kube-system

E0124 23:23:05.254092       1 server.go:1039] "Failed to retrieve node info" err="Get \"https://127.0.0.1:6443/api/v1/nodes/ip-172-31-25-92\": dial tcp 127.0.0.1:6443: connect: connection refused"
E0124 23:23:06.331749       1 server.go:1039] "Failed to retrieve node info" err="Get \"https://127.0.0.1:6443/api/v1/nodes/ip-172-31-25-92\": dial tcp 127.0.0.1:6443: connect: connection refused"
E0124 23:23:08.537566       1 server.go:1039] "Failed to retrieve node info" err="Get \"https://127.0.0.1:6443/api/v1/nodes/ip-172-31-25-92\": dial tcp 127.0.0.1:6443: connect: connection refused"
E0124 23:23:13.322999       1 server.go:1039] "Failed to retrieve node info" err="Get \"https://127.0.0.1:6443/api/v1/nodes/ip-172-31-25-92\": dial tcp 127.0.0.1:6443: connect: connection refused"
E0124 23:23:22.582847       1 server.go:1039] "Failed to retrieve node info" err="Get \"https://127.0.0.1:6443/api/v1/nodes/ip-172-31-25-92\": dial tcp 127.0.0.1:6443: connect: connection refused"
E0124 23:23:38.706736       1 server.go:1039] "Failed to retrieve node info" err="Get \"https://127.0.0.1:6443/api/v1/nodes/ip-172-31-25-92\": dial tcp 127.0.0.1:6443: connect: connection refused"
I0124 23:23:38.706808       1 server.go:1020] "Can't determine this node's IP, assuming loopback; if this is incorrect, please set the --bind-address flag"
I0124 23:23:38.766580       1 server.go:652] "kube-proxy running in dual-stack mode" primary ipFamily="IPv4"
time="2025-01-24T23:23:38Z" level=warning msg="Running modprobe ip_vs failed with message: `modprobe: WARNING: Module ip_vs not found in directory /lib/modules/6.4.0-150600.23.17-default`, error: exit status 1"
I0124 23:23:38.922632       1 server_others.go:236] "Using ipvs Proxier"
I0124 23:23:38.933566       1 server_others.go:512] "Detect-local-mode set to ClusterCIDR, but no cluster CIDR for family" ipFamily="IPv6"
I0124 23:23:38.933587       1 server_others.go:529] "Defaulting to no-op detect-local"
I0124 23:23:38.934084       1 ipset.go:116] "Ipset name truncated" ipSetName="KUBE-6-LOAD-BALANCER-SOURCE-CIDR" truncatedName="KUBE-6-LOAD-BALANCER-SOURCE-CID"
I0124 23:23:38.934101       1 ipset.go:116] "Ipset name truncated" ipSetName="KUBE-6-NODE-PORT-LOCAL-SCTP-HASH" truncatedName="KUBE-6-NODE-PORT-LOCAL-SCTP-HAS"
I0124 23:23:38.934180       1 server.go:865] "Version info" version="v1.29.13+rke2r1"
I0124 23:23:38.934189       1 server.go:867] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
I0124 23:23:38.935290       1 config.go:188] "Starting service config controller"
I0124 23:23:38.935313       1 shared_informer.go:311] Waiting for caches to sync for service config
I0124 23:23:38.935333       1 config.go:97] "Starting endpoint slice config controller"
I0124 23:23:38.935339       1 shared_informer.go:311] Waiting for caches to sync for endpoint slice config
I0124 23:23:38.936080       1 config.go:315] "Starting node config controller"
I0124 23:23:38.936090       1 shared_informer.go:311] Waiting for caches to sync for node config
W0124 23:23:38.937394       1 reflector.go:539] k8s.io/client-go/informers/factory.go:159: failed to list *v1.Node: Get "https://127.0.0.1:6443/api/v1/nodes?fieldSelector=metadata.name%3Dip-172-31-25-92&limit=500&resourceVersion=0": dial tcp 127.0.0.1:6443: connect: connection refused
E0124 23:23:38.937562       1 reflector.go:147] k8s.io/client-go/informers/factory.go:159: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://127.0.0.1:6443/api/v1/nodes?fieldSelector=metadata.name%3Dip-172-31-25-92&limit=500&resourceVersion=0": dial tcp 127.0.0.1:6443: connect: connection refused
W0124 23:23:38.937760       1 reflector.go:539] k8s.io/client-go/informers/factory.go:159: failed to list *v1.EndpointSlice: Get "https://127.0.0.1:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&limit=500&resourceVersion=0": dial tcp 127.0.0.1:6443: connect: connection refused
E0124 23:23:38.937814       1 reflector.go:147] k8s.io/client-go/informers/factory.go:159: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: Get "https://127.0.0.1:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&limit=500&resourceVersion=0": dial tcp 127.0.0.1:6443: connect: connection refused
W0124 23:23:38.937880       1 reflector.go:539] k8s.io/client-go/informers/factory.go:159: failed to list *v1.Service: Get "https://127.0.0.1:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&limit=500&resourceVersion=0": dial tcp 127.0.0.1:6443: connect: connection refused
E0124 23:23:38.937916       1 reflector.go:147] k8s.io/client-go/informers/factory.go:159: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://127.0.0.1:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&limit=500&resourceVersion=0": dial tcp 127.0.0.1:6443: connect: connection refused
E0124 23:23:38.938321       1 event_broadcaster.go:279] "Unable to write event (may retry after sleeping)" err="Post \"https://127.0.0.1:6443/apis/events.k8s.io/v1/namespaces/default/events\": dial tcp 127.0.0.1:6443: connect: connection refused"
W0124 23:23:39.829127       1 reflector.go:539] k8s.io/client-go/informers/factory.go:159: failed to list *v1.Service: Get "https://127.0.0.1:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&limit=500&resourceVersion=0": dial tcp 127.0.0.1:6443: connect: connection refused
E0124 23:23:39.829174       1 reflector.go:147] k8s.io/client-go/informers/factory.go:159: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://127.0.0.1:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&limit=500&resourceVersion=0": dial tcp 127.0.0.1:6443: connect: connection refused
W0124 23:23:40.190818       1 reflector.go:539] k8s.io/client-go/informers/factory.go:159: failed to list *v1.Node: Get "https://127.0.0.1:6443/api/v1/nodes?fieldSelector=metadata.name%3Dip-172-31-25-92&limit=500&resourceVersion=0": dial tcp 127.0.0.1:6443: connect: connection refused
E0124 23:23:40.190878       1 reflector.go:147] k8s.io/client-go/informers/factory.go:159: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://127.0.0.1:6443/api/v1/nodes?fieldSelector=metadata.name%3Dip-172-31-25-92&limit=500&resourceVersion=0": dial tcp 127.0.0.1:6443: connect: connection refused
W0124 23:23:40.403949       1 reflector.go:539] k8s.io/client-go/informers/factory.go:159: failed to list *v1.EndpointSlice: Get "https://127.0.0.1:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&limit=500&resourceVersion=0": dial tcp 127.0.0.1:6443: connect: connection refused
E0124 23:23:40.404001       1 reflector.go:147] k8s.io/client-go/informers/factory.go:159: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: Get "https://127.0.0.1:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&limit=500&resourceVersion=0": dial tcp 127.0.0.1:6443: connect: connection refused
W0124 23:23:42.944827       1 reflector.go:539] k8s.io/client-go/informers/factory.go:159: failed to list *v1.Service: Get "https://127.0.0.1:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&limit=500&resourceVersion=0": dial tcp 127.0.0.1:6443: connect: connection refused
E0124 23:23:42.945001       1 reflector.go:147] k8s.io/client-go/informers/factory.go:159: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://127.0.0.1:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&limit=500&resourceVersion=0": dial tcp 127.0.0.1:6443: connect: connection refused
W0124 23:23:43.036648       1 reflector.go:539] k8s.io/client-go/informers/factory.go:159: failed to list *v1.Node: Get "https://127.0.0.1:6443/api/v1/nodes?fieldSelector=metadata.name%3Dip-172-31-25-92&limit=500&resourceVersion=0": dial tcp 127.0.0.1:6443: connect: connection refused
E0124 23:23:43.036691       1 reflector.go:147] k8s.io/client-go/informers/factory.go:159: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://127.0.0.1:6443/api/v1/nodes?fieldSelector=metadata.name%3Dip-172-31-25-92&limit=500&resourceVersion=0": dial tcp 127.0.0.1:6443: connect: connection refused
W0124 23:23:43.231993       1 reflector.go:539] k8s.io/client-go/informers/factory.go:159: failed to list *v1.EndpointSlice: Get "https://127.0.0.1:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&limit=500&resourceVersion=0": dial tcp 127.0.0.1:6443: connect: connection refused
E0124 23:23:43.232049       1 reflector.go:147] k8s.io/client-go/informers/factory.go:159: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: Get "https://127.0.0.1:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&limit=500&resourceVersion=0": dial tcp 127.0.0.1:6443: connect: connection refused
W0124 23:23:47.293208       1 reflector.go:539] k8s.io/client-go/informers/factory.go:159: failed to list *v1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User "system:kube-proxy" cannot list resource "endpointslices" in API group "discovery.k8s.io" at the cluster scope
E0124 23:23:47.293791       1 reflector.go:147] k8s.io/client-go/informers/factory.go:159: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User "system:kube-proxy" cannot list resource "endpointslices" in API group "discovery.k8s.io" at the cluster scope
W0124 23:23:47.618036       1 reflector.go:539] k8s.io/client-go/informers/factory.go:159: failed to list *v1.Node: nodes "ip-172-31-25-92" is forbidden: User "system:kube-proxy" cannot list resource "nodes" in API group "" at the cluster scope
E0124 23:23:47.618074       1 reflector.go:147] k8s.io/client-go/informers/factory.go:159: Failed to watch *v1.Node: failed to list *v1.Node: nodes "ip-172-31-25-92" is forbidden: User "system:kube-proxy" cannot list resource "nodes" in API group "" at the cluster scope
W0124 23:23:48.201024       1 reflector.go:539] k8s.io/client-go/informers/factory.go:159: failed to list *v1.Service: services is forbidden: User "system:kube-proxy" cannot list resource "services" in API group "" at the cluster scope
E0124 23:23:48.201091       1 reflector.go:147] k8s.io/client-go/informers/factory.go:159: Failed to watch *v1.Service: failed to list *v1.Service: services is forbidden: User "system:kube-proxy" cannot list resource "services" in API group "" at the cluster scope
I0124 23:23:54.637014       1 shared_informer.go:318] Caches are synced for node config
I0124 23:23:55.836524       1 shared_informer.go:318] Caches are synced for service config
I0124 23:23:59.836067       1 shared_informer.go:318] Caches are synced for endpoint slice config

$ k logs pod/kube-proxy-ip -n kube-system | grep -i librar

no output for the shared library as it's found and correctly mapped for the pod.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@brandond brandond

@VestigeJ VestigeJ

@rafaelbreno rafaelbreno

Labels
kind/bug Something isn't working
Projects
None yet
Milestone
2025-01 Release Cycle
Development

No branches or pull requests
4 participants
@brandond @VestigeJ @shalomjacob @rafaelbreno