Agent node unable to start container using embedded registry

[barrettc]

I'm attempting to use the embedded registry in an air-gapped environment with just a single server node and agent to start with. I've followed the air gap documentation and all images that I need are scp'ed over to the server node where I import them. The server node starts up fine, pods are created using the containers I've imported, and all looks normal on the command line with ctr image ls with all the images appearing that I imported.

When I install the agent on another machine, k3s starts up but the pods are stuck in ContainerCreating. On this second machine, ctr image ls produces no results although I'm not positive if that's the expectation with spegel:

bc@nuc2:~$ sudo ctr i ls
REF TYPE DIGEST SIZE PLATFORMS LABELS 

On the agent machine, I see these types of errors in syslog:

Mar  5 19:01:46 nuc2 k3s[64801]: E0305 19:01:46.084930   64801 kuberuntime_sandbox.go:72] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to get sandbox image \"rancher/mirrored-pause:3.6\": failed to pull image \"rancher/mirrored-pause:3.6\": failed to pull and unpack image \"docker.io/rancher/mirrored-pause:3.6\": failed to resolve reference \"docker.io/rancher/mirrored-pause:3.6\": unexpected status from HEAD request to https://127.0.0.1:6443/v2/rancher/mirrored-pause/manifests/3.6?ns=docker.io: 500 Internal Server Error" pod="longhorn-system/longhorn-manager-hq9x6"
Mar  5 19:01:46 nuc2 k3s[64801]: E0305 19:01:46.084956   64801 kuberuntime_manager.go:1172] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to get sandbox image \"rancher/mirrored-pause:3.6\": failed to pull image \"rancher/mirrored-pause:3.6\": failed to pull and unpack image \"docker.io/rancher/mirrored-pause:3.6\": failed to resolve reference \"docker.io/rancher/mirrored-pause:3.6\": unexpected status from HEAD request to https://127.0.0.1:6443/v2/rancher/mirrored-pause/manifests/3.6?ns=docker.io: 500 Internal Server Error" pod="longhorn-system/longhorn-manager-hq9x6"
Mar  5 19:01:46 nuc2 k3s[64801]: E0305 19:01:46.085011   64801 pod_workers.go:1298] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"longhorn-manager-hq9x6_longhorn-system(3ab49e67-1528-4ce0-87ce-758208ce7a97)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"longhorn-manager-hq9x6_longhorn-system(3ab49e67-1528-4ce0-87ce-758208ce7a97)\\\": rpc error: code = Unknown desc = failed to get sandbox image \\\"rancher/mirrored-pause:3.6\\\": failed to pull image \\\"rancher/mirrored-pause:3.6\\\": failed to pull and unpack image \\\"docker.io/rancher/mirrored-pause:3.6\\\": failed to resolve reference \\\"docker.io/rancher/mirrored-pause:3.6\\\": unexpected status from HEAD request to https://127.0.0.1:6443/v2/rancher/mirrored-pause/manifests/3.6?ns=docker.io: 500 Internal Server Error\"" pod="longhorn-system/longhorn-manager-hq9x6" podUID="3ab49e67-1528-4ce0-87ce-758208ce7a97"
Mar  5 19:01:46 nuc2 k3s[64801]: E0305 19:01:46.086596   64801 remote_runtime.go:193] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to get sandbox image \"rancher/mirrored-pause:3.6\": failed to pull image \"rancher/mirrored-pause:3.6\": failed to pull and unpack image \"docker.io/rancher/mirrored-pause:3.6\": failed to resolve reference \"docker.io/rancher/mirrored-pause:3.6\": unexpected status from HEAD request to https://127.0.0.1:6443/v2/rancher/mirrored-pause/manifests/3.6?ns=docker.io: 500 Internal Server Error"
Mar  5 19:01:46 nuc2 k3s[64801]: E0305 19:01:46.086645   64801 kuberuntime_sandbox.go:72] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to get sandbox image \"rancher/mirrored-pause:3.6\": failed to pull image \"rancher/mirrored-pause:3.6\": failed to pull and unpack image \"docker.io/rancher/mirrored-pause:3.6\": failed to resolve reference \"docker.io/rancher/mirrored-pause:3.6\": unexpected status from HEAD request to https://127.0.0.1:6443/v2/rancher/mirrored-pause/manifests/3.6?ns=docker.io: 500 Internal Server Error" pod="longhorn-system/engine-image-ei-acb7590c-dgmqk"
Mar  5 19:01:46 nuc2 k3s[64801]: E0305 19:01:46.086673   64801 kuberuntime_manager.go:1172] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to get sandbox image \"rancher/mirrored-pause:3.6\": failed to pull image \"rancher/mirrored-pause:3.6\": failed to pull and unpack image \"docker.io/rancher/mirrored-pause:3.6\": failed to resolve reference \"docker.io/rancher/mirrored-pause:3.6\": unexpected status from HEAD request to https://127.0.0.1:6443/v2/rancher/mirrored-pause/manifests/3.6?ns=docker.io: 500 Internal Server Error" pod="longhorn-system/engine-image-ei-acb7590c-dgmqk"
Mar  5 19:01:46 nuc2 k3s[64801]: E0305 19:01:46.086746   64801 pod_workers.go:1298] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"engine-image-ei-acb7590c-dgmqk_longhorn-system(b5df50a1-dac8-4aaa-b7ce-305c738ebee2)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"engine-image-ei-acb7590c-dgmqk_longhorn-system(b5df50a1-dac8-4aaa-b7ce-305c738ebee2)\\\": rpc error: code = Unknown desc = failed to get sandbox image \\\"rancher/mirrored-pause:3.6\\\": failed to pull image \\\"rancher/mirrored-pause:3.6\\\": failed to pull and unpack image \\\"docker.io/rancher/mirrored-pause:3.6\\\": failed to resolve reference \\\"docker.io/rancher/mirrored-pause:3.6\\\": unexpected status from HEAD request to https://127.0.0.1:6443/v2/rancher/mirrored-pause/manifests/3.6?ns=docker.io: 500 Internal Server Error\"" pod="longhorn-system/engine-image-ei-acb7590c-dgmqk" podUID="b5df50a1-dac8-4aaa-b7ce-305c738ebee2"

My registries.yaml on both server and agent machines is:

mirrors:
  docker.io:
  quay.io:
  registry.k8s.io:

I'm open to any suggestions to debug or clues as to what might be going on.

k3s version v1.29.1+k3s2 (57482a1)
go version go1.21.6

Node(s) CPU architecture, OS, and Version:
Linux nuc1 5.4.0-172-generic #190-Ubuntu SMP Fri Feb 2 23:24:22 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

[brandond]

I've followed the air gap documentation and all images that I need are scp'ed over to the server node where I import them.

Where did you put the images? Did you place the archives in the images dir so that they are automatically imported on startup, or did you manually import them? If you manually imported them, did you import them into the correct namespace?

Can you show the output of ctr -n k8s.io image ls on both nodes?

Can you confirm that the embedded registry ports (5001 and 6443) are open between both nodes?

[barrettc]

Right now our process is that an internet connected machine fetches them directly from the registries noted above -- docker.io, quay.io, registry.k8s.io and then they are scp'ed to the server node. Next k3s is installed using the airgap instructions and then they are each imported with the command ctr -n k8s.io image import <image tar>. Then /etc/rancher/k3s/registries.yaml is updated with appropriate values.

Server node (nuc1):

bc@nuc1:~$ sudo ctr -n k8s.io image ls
REF                                                                                          TYPE                                                 DIGEST                                                                  SIZE      PLATFORMS   LABELS                                                                                      
docker.io/amd64/registry:2.8.3                                                               application/vnd.oci.image.manifest.v1+json           sha256:dc772a85878064eb78894d554f02f11662616d3b29701e678a4b5e245f98a3ab 24.8 MiB  linux/amd64 io.cri-containerd.image=managed                                                             
docker.io/bats/bats:v1.4.1                                                                   application/vnd.oci.image.manifest.v1+json           sha256:75cc8a28050360f40df57f3fd9a3fe5e4276e6adde03efd83b029ef40b9c218d 51.8 MiB  linux/amd64 io.cri-containerd.image=managed                                                             
docker.io/grafana/grafana:10.2.2                                                             application/vnd.oci.image.manifest.v1+json           sha256:faac3c6d4dcebc6838d943321151be5a7e78fbb6d98fbfabd48b6c146542d181 388.1 MiB linux/amd64 io.cri-containerd.image=managed                                                             
docker.io/library/busybox:1.35.0                                                             application/vnd.oci.image.manifest.v1+json           sha256:ed38bfab764aa781df00bec90fccfe1fd4af8013c983a3eb41a75dd316209b68 4.3 MiB   linux/amd64 io.cri-containerd.image=managed                                                             
docker.io/library/hello-world:linux                                                          application/vnd.oci.image.manifest.v1+json           sha256:c55c4ab38c73b5b59d76afd01083be207b1897142fee10222e56d1fd708efb62 15.5 KiB  linux/amd64 io.cri-containerd.image=managed                                                             
docker.io/library/nginx:stable-alpine                                                        application/vnd.oci.image.manifest.v1+json           sha256:4387a2da149618aa4e18b4e455fc478fee4704888ef610a03797ee764f2270e1 40.9 MiB  linux/amd64 io.cri-containerd.image=managed                                                             
docker.io/longhornio/backing-image-manager:v1.6.0                                            application/vnd.oci.image.manifest.v1+json           sha256:9fbd965a15b635db41439b8ab0ee73d051b7bc0768b654634ba0bca932f804c4 353.7 MiB linux/amd64 io.cri-containerd.image=managed                                                             
docker.io/longhornio/csi-attacher:v4.4.2                                                     application/vnd.oci.image.manifest.v1+json           sha256:20a5ab2802398c01e4ee423ad7d19bca0909355d0f549b6cd7cf029bfce6d0b9 60.7 MiB  linux/amd64 io.cri-containerd.image=managed                                                             
docker.io/longhornio/csi-node-driver-registrar:v2.9.2                                        application/vnd.oci.image.manifest.v1+json           sha256:71ab0d25e404680a317398403b7926c8f70bacab9486d9dd833df8f3cd2ccbd8 22.1 MiB  linux/amd64 io.cri-containerd.image=managed                                                             
docker.io/longhornio/csi-provisioner:v3.6.2                                                  application/vnd.oci.image.manifest.v1+json           sha256:11c0ce63bd7d111624abcfaf3151fa8e4c4cb47c02795445579ba48f29d6a8bc 64.9 MiB  linux/amd64 io.cri-containerd.image=managed                                                             
docker.io/longhornio/csi-resizer:v1.9.2                                                      application/vnd.oci.image.manifest.v1+json           sha256:acb0d1baad094e5f6b160ec0d532a4c4125f9f909ace7d952ef4da3c9ee93a3b 61.3 MiB  linux/amd64 io.cri-containerd.image=managed                                                             
docker.io/longhornio/csi-snapshotter:v6.3.2                                                  application/vnd.oci.image.manifest.v1+json           sha256:7da7fd56d590583e341f18dd9a271bc0f4d907f867598e506c159f6fd61b9bc2 61.1 MiB  linux/amd64 io.cri-containerd.image=managed                                                             
docker.io/longhornio/livenessprobe:v2.11.0                                                   application/vnd.oci.image.manifest.v1+json           sha256:ae52ab6242b9045f9177f5ac5b3cb90dbe75f0d8ffbdcdf6e487ce8da8ce3695 19.9 MiB  linux/amd64 io.cri-containerd.image=managed                                                             
docker.io/longhornio/longhorn-engine:v1.6.0                                                  application/vnd.oci.image.manifest.v1+json           sha256:61c870b4f9dcac2d8782f0a95c594096067748e52e12d38febfd4983b5692619 334.9 MiB linux/amd64 io.cri-containerd.image=managed                                                             
docker.io/longhornio/longhorn-instance-manager:v1.6.0                                        application/vnd.oci.image.manifest.v1+json           sha256:bb38c2f25cff25b9118a9431f3c8fbc710d007c4d01480b5ae35a93fe6f5954b 704.4 MiB linux/amd64 io.cri-containerd.image=managed                                                             
docker.io/longhornio/longhorn-manager:v1.6.0                                                 application/vnd.oci.image.manifest.v1+json           sha256:95acdd40872291d14d6d314dc378786bb22a497fa3a8b067c990f0db77facd17 297.8 MiB linux/amd64 io.cri-containerd.image=managed                                                             
docker.io/longhornio/longhorn-share-manager:v1.6.0                                           application/vnd.oci.image.manifest.v1+json           sha256:1ddb60835e582d7569c46cd1c646bd287acb3d40fb17ec8e82b14b365edffa1d 195.3 MiB linux/amd64 io.cri-containerd.image=managed                                                             
docker.io/longhornio/longhorn-ui:v1.6.0                                                      application/vnd.oci.image.manifest.v1+json           sha256:8f8866e7ae4b6a7a7cf69739dfcbebc684f006f5b488fa0c4009a217887d5957 200.7 MiB linux/amd64 io.cri-containerd.image=managed                                                             
docker.io/longhornio/support-bundle-kit:v0.0.33                                              application/vnd.oci.image.manifest.v1+json           sha256:7ac0cbddf115abd5d532b9bc81832d187690474fdef4ed97408827434f4ea781 401.9 MiB linux/amd64 io.cri-containerd.image=managed                                                             
docker.io/rancher/klipper-helm:v0.8.0-build20230510                                          application/vnd.oci.image.manifest.v1+json           sha256:fbe939c6454ddb25cee74fd8e4a527b5b633861f8b2687d75d82886afb5d9ea7 250.3 MiB linux/amd64 io.cri-containerd.image=managed                                                             
docker.io/rancher/klipper-helm:v0.8.2-build20230815                                          application/vnd.oci.image.manifest.v1+json           sha256:adeaf82acc7921296fc1f26623bfb08ee02733cce42e5047cd5b5efb9231613d 244.5 MiB linux/amd64 io.cattle.k3s.pinned=pinned,io.cri-containerd.image=managed,io.cri-containerd.pinned=pinned 
docker.io/rancher/klipper-lb:v0.4.5                                                          application/vnd.oci.image.manifest.v1+json           sha256:048c5777eb18fefbc262b31a5eb2c73bb54be776057ac437a9176a2f990120e2 18.0 MiB  linux/amd64 io.cattle.k3s.pinned=pinned,io.cri-containerd.image=managed,io.cri-containerd.pinned=pinned 
docker.io/rancher/local-path-provisioner:v0.0.24                                             application/vnd.oci.image.manifest.v1+json           sha256:8f9e096d2affa2126b23dcb1adab71a876b81d4fbc00140adc3a7990b9c4b55f 38.6 MiB  linux/amd64 io.cattle.k3s.pinned=pinned,io.cri-containerd.image=managed,io.cri-containerd.pinned=pinned 
docker.io/rancher/mirrored-coredns-coredns:1.10.1                                            application/vnd.oci.image.manifest.v1+json           sha256:b8c4fe91777dc2ef921d07c1887cd713f2e5f7abbb44613e96955ce685c01eb7 51.1 MiB  linux/amd64 io.cattle.k3s.pinned=pinned,io.cri-containerd.image=managed,io.cri-containerd.pinned=pinned 
docker.io/rancher/mirrored-library-busybox:1.36.1                                            application/vnd.oci.image.manifest.v1+json           sha256:c4924233690b14ec2a47e7548cd510b6a1512894d0c1baac7a8504c6c513e55e 4.3 MiB   linux/amd64 io.cattle.k3s.pinned=pinned,io.cri-containerd.image=managed,io.cri-containerd.pinned=pinned 
docker.io/rancher/mirrored-library-traefik:2.10.5                                            application/vnd.oci.image.manifest.v1+json           sha256:1c5765c6fda7ed7a2323016a2db63457a5d965eb7547394952b2e97a15a6a743 145.7 MiB linux/amd64 io.cattle.k3s.pinned=pinned,io.cri-containerd.image=managed,io.cri-containerd.pinned=pinned 
docker.io/rancher/mirrored-library-traefik:2.9.10                                            application/vnd.oci.image.manifest.v1+json           sha256:ec8726f16d7625ff39a3bed2e10ca3ebf993d6aaff41bfe5b27789a98c24687b 132.8 MiB linux/amd64 io.cri-containerd.image=managed                                                             
docker.io/rancher/mirrored-metrics-server:v0.6.3                                             application/vnd.oci.image.manifest.v1+json           sha256:0e4540da9cc482a4bfa8f74630e75c88f55fba72d6848cd09789788d706a4155 67.0 MiB  linux/amd64 io.cattle.k3s.pinned=pinned,io.cri-containerd.image=managed,io.cri-containerd.pinned=pinned 
docker.io/rancher/mirrored-pause:3.6                                                         application/vnd.oci.image.manifest.v1+json           sha256:d727eafaa9ce4252c1711e1b2f77292e14f69a54d537d626ed9a39b186530b1f 669.8 KiB linux/amd64 io.cattle.k3s.pinned=pinned,io.cri-containerd.image=managed,io.cri-containerd.pinned=pinned 
quay.io/kiwigrid/k8s-sidecar:1.25.2                                                          application/vnd.oci.image.manifest.v1+json           sha256:37b841c1c7cff6e268cd374018ac606dc4298a826daaa3e140cfc3496be30a19 81.9 MiB  linux/amd64 io.cri-containerd.image=managed                                                             
quay.io/prometheus-operator/prometheus-config-reloader:v0.70.0                               application/vnd.oci.image.manifest.v1+json           sha256:56030512da501c03fc2627b827256aa3d53e44fbfe8f8b01c3aedd3ac52a63b8 43.4 MiB  linux/amd64 io.cri-containerd.image=managed                                                             
quay.io/prometheus-operator/prometheus-operator:v0.70.0                                      application/vnd.oci.image.manifest.v1+json           sha256:243099235d3c3ae22a58b2d01d433bced5678b5badf0d94891c0fea02c734013 54.0 MiB  linux/amd64 io.cri-containerd.image=managed                                                             
quay.io/prometheus/alertmanager:v0.26.0                                                      application/vnd.oci.image.manifest.v1+json           sha256:e0f6e92e0027d9e7a9894cb5dff8a21a8a7d111f67ca145b5a02cfd73e219323 64.8 MiB  linux/amd64 io.cri-containerd.image=managed                                                             
quay.io/prometheus/node-exporter:v1.7.0                                                      application/vnd.oci.image.manifest.v1+json           sha256:70497619a994d608182fb86c987667f925db8eb72cc5680ef42605c091f59111 22.8 MiB  linux/amd64 io.cri-containerd.image=managed                                                             
quay.io/prometheus/prometheus:v2.48.1                                                        application/vnd.oci.image.manifest.v1+json           sha256:09091a624ce276b2b4d51f6daf44eb742f117b1b10de318c769bc11d1d46d0cc 236.9 MiB linux/amd64 io.cri-containerd.image=managed                                                             
registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20221220-controller-v1.5.1-58-g787ea74b6 application/vnd.oci.image.manifest.v1+json           sha256:1dc09318a5e6c2d1235c9febaedfe500cc29c773880ef50bec2fabe9a732a00a 45.3 MiB  linux/amd64 io.cri-containerd.image=managed                                                             
registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.10.1                                application/vnd.oci.image.manifest.v1+json           sha256:f05c5f4f371e004f35c888e7510faf91633f2a47570823aeb4b0038d0fb9be7c 42.0 MiB  linux/amd64 io.cri-containerd.image=managed                                                             
sha256:057e08108889890dcd4bf6db63f2c72d24b0ddba549cafc2ea3ceea67599b990                      application/vnd.oci.image.manifest.v1+json           sha256:1dc09318a5e6c2d1235c9febaedfe500cc29c773880ef50bec2fabe9a732a00a 45.3 MiB  linux/amd64 io.cri-containerd.image=managed                                                             
sha256:06e5d59b720d562ac6f2f8bff4dc7155862e5553ffce60423b593dce71de8d39                      application/vnd.oci.image.manifest.v1+json           sha256:faac3c6d4dcebc6838d943321151be5a7e78fbb6d98fbfabd48b6c146542d181 388.1 MiB linux/amd64 io.cri-containerd.image=managed                                                             
sha256:0c00acac9c2794adfa8bb7b13ef38504300b505a043bf68dff7a00068dcc732b                      application/vnd.oci.image.manifest.v1+json           sha256:ed38bfab764aa781df00bec90fccfe1fd4af8013c983a3eb41a75dd316209b68 4.3 MiB   linux/amd64 io.cri-containerd.image=managed                                                             
sha256:118137d698bbd0de7f3a3299804730010daf5f29f4d191fd0c804425987fa46e                      application/vnd.oci.image.manifest.v1+json           sha256:7da7fd56d590583e341f18dd9a271bc0f4d907f867598e506c159f6fd61b9bc2 61.1 MiB  linux/amd64 io.cri-containerd.image=managed                                                             
sha256:17066c233afa320bfba42ee5085e5cf767b1b2739562b40950267f416e952231                      application/vnd.docker.distribution.manifest.v2+json sha256:370ba55e8e4731a6e54d9c9ca8c99b517a2c00c8733499f0088d38bb141daeff 18.0 MiB  linux/amd64 io.cri-containerd.image=managed                                                             
sha256:29c0bdda322756e873f7888a85fc496ef30cdfcc2e5be902fd12fca66a1930e8                      application/vnd.oci.image.manifest.v1+json           sha256:7ac0cbddf115abd5d532b9bc81832d187690474fdef4ed97408827434f4ea781 401.9 MiB linux/amd64 io.cri-containerd.image=managed                                                             
sha256:2c3daafa6cc7a96b7e008553c87b5dd520fd26bc4aab3bc5af29373e16c69935                      application/vnd.oci.image.manifest.v1+json           sha256:4387a2da149618aa4e18b4e455fc478fee4704888ef610a03797ee764f2270e1 40.9 MiB  linux/amd64 io.cri-containerd.image=managed                                                             
sha256:2e4eb3ba8dca0a9578e4a25221607403129f142c923b95b39967cd589f6b1167                      application/vnd.oci.image.manifest.v1+json           sha256:bb38c2f25cff25b9118a9431f3c8fbc710d007c4d01480b5ae35a93fe6f5954b 704.4 MiB linux/amd64 io.cri-containerd.image=managed                                                             
sha256:3b5c8a042d9fb8585a8ae302becdbfb1a13b3ddf02b866f15f01ab5fb502d4c6                      application/vnd.oci.image.manifest.v1+json           sha256:75cc8a28050360f40df57f3fd9a3fe5e4276e6adde03efd83b029ef40b9c218d 51.8 MiB  linux/amd64 io.cri-containerd.image=managed                                                             
sha256:3f57d9401f8d42f986df300f0c69192fc41da28ccc8d797829467780db3dd741                      application/vnd.docker.distribution.manifest.v2+json sha256:4e150f11172dadd73cda60c662893399da872cbfca82ec214f6d918f0369013d 4.3 MiB   linux/amd64 io.cri-containerd.image=managed                                                             
sha256:438c692b0cb6d50af207439fba2bae492a64b3469a263246bdeb9179b2ee9601                      application/vnd.oci.image.manifest.v1+json           sha256:71ab0d25e404680a317398403b7926c8f70bacab9486d9dd833df8f3cd2ccbd8 22.1 MiB  linux/amd64 io.cri-containerd.image=managed                                                             
sha256:494ea5379400e4f01e49f20be885b1ff011794abebdf81dbd86f606e171d796d                      application/vnd.oci.image.manifest.v1+json           sha256:ae52ab6242b9045f9177f5ac5b3cb90dbe75f0d8ffbdcdf6e487ce8da8ce3695 19.9 MiB  linux/amd64 io.cri-containerd.image=managed                                                             
sha256:5f89cb8137ccbd39377d91b9d75faf4ec4ee0a2d2a3a63635535b10c69c935fa                      application/vnd.docker.distribution.manifest.v2+json sha256:9f6b0a352533fe34763f81f014952f0595b9bd2ad531b179767c81ef77172668 244.5 MiB linux/amd64 io.cri-containerd.image=managed                                                             
sha256:6270bb605e12e581514ada5fd5b3216f727db55dc87d5889c790e4c760683fee                      application/vnd.docker.distribution.manifest.v2+json sha256:79b611631c0d19e9a975fb0a8511e5153789b4c26610d1842e9f735c57cc8b13 669.8 KiB linux/amd64 io.cri-containerd.image=managed                                                             
sha256:642f6d2ac4c18182afcf0fd795b4ad0e3c18d9438382d1bb6094ddb2ac6c49b6                      application/vnd.oci.image.manifest.v1+json           sha256:243099235d3c3ae22a58b2d01d433bced5678b5badf0d94891c0fea02c734013 54.0 MiB  linux/amd64 io.cri-containerd.image=managed                                                             
sha256:6bc8bdd63e4084be7140ecebb348f231b21416d75661d31481f51f9203e0e544                      application/vnd.oci.image.manifest.v1+json           sha256:20a5ab2802398c01e4ee423ad7d19bca0909355d0f549b6cd7cf029bfce6d0b9 60.7 MiB  linux/amd64 io.cri-containerd.image=managed                                                             
sha256:6f42df210d7faa0b6eb56f5f28bb061930c7834af1e7dd4b3586108ec6b1485b                      application/vnd.oci.image.manifest.v1+json           sha256:fbe939c6454ddb25cee74fd8e4a527b5b633861f8b2687d75d82886afb5d9ea7 250.3 MiB linux/amd64 io.cri-containerd.image=managed                                                             
sha256:72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e                      application/vnd.oci.image.manifest.v1+json           sha256:70497619a994d608182fb86c987667f925db8eb72cc5680ef42605c091f59111 22.8 MiB  linux/amd64 io.cri-containerd.image=managed                                                             
sha256:73ec9f309a82e9938106d4784ff2aa909a41f9e8041197bfc1c373be26ba0840                      application/vnd.oci.image.manifest.v1+json           sha256:56030512da501c03fc2627b827256aa3d53e44fbfe8f8b01c3aedd3ac52a63b8 43.4 MiB  linux/amd64 io.cri-containerd.image=managed                                                             
sha256:817bbe3f2e5179b5fa7dd4085c87aab1faaf9dc879e3b1d05796901a006f58ab                      application/vnd.docker.distribution.manifest.v2+json sha256:f46cf066256b7d95f1d7e428f9014f27955be2fa67ee9793f0ae9199fd354fa3 67.0 MiB  linux/amd64 io.cri-containerd.image=managed                                                             
sha256:9f27df16978d8a27fc102b72eae63c6e9cd6d4e1675c414ec79df7f280868098                      application/vnd.oci.image.manifest.v1+json           sha256:e0f6e92e0027d9e7a9894cb5dff8a21a8a7d111f67ca145b5a02cfd73e219323 64.8 MiB  linux/amd64 io.cri-containerd.image=managed                                                             
sha256:9f4c1b666bd8cfe324188a87cb12ce4f3fcdb4bb3037c59674de666ee2857e8f                      application/vnd.oci.image.manifest.v1+json           sha256:95acdd40872291d14d6d314dc378786bb22a497fa3a8b067c990f0db77facd17 297.8 MiB linux/amd64 io.cri-containerd.image=managed                                                             
sha256:a8781fe3b7a280475ac394dc1754b4c08a39cdd400e05869d0da3586ffb82f7f                      application/vnd.oci.image.manifest.v1+json           sha256:dc772a85878064eb78894d554f02f11662616d3b29701e678a4b5e245f98a3ab 24.8 MiB  linux/amd64 io.cri-containerd.image=managed                                                             
sha256:b29384aeb4b13e047448ccfd312c52b4d023abcbbaafcab174293a97821dddb0                      application/vnd.docker.distribution.manifest.v2+json sha256:0c39d541b3696f919b8a59098766b5407ee2ccd956070fa4d1148437e0974c00 38.6 MiB  linux/amd64 io.cri-containerd.image=managed                                                             
sha256:cc365cbb0397b708abc260501574769de3891d756c7a2e43c3839e9126cf9aec                      application/vnd.docker.distribution.manifest.v2+json sha256:3883ce8abfbd8d158bcc7cfa876fb2ff6b9c49221518d7c666eb4ed9022ec919 145.7 MiB linux/amd64 io.cri-containerd.image=managed                                                             
sha256:ceb991e912e7380aa4f6120f403529a2406e445adad8a1afc71ff19070672669                      application/vnd.oci.image.manifest.v1+json           sha256:37b841c1c7cff6e268cd374018ac606dc4298a826daaa3e140cfc3496be30a19 81.9 MiB  linux/amd64 io.cri-containerd.image=managed                                                             
sha256:d11bd1f0b3e575e4e32df907b4646928b4d144d1636be5e1899b1dec01efb68b                      application/vnd.oci.image.manifest.v1+json           sha256:acb0d1baad094e5f6b160ec0d532a4c4125f9f909ace7d952ef4da3c9ee93a3b 61.3 MiB  linux/amd64 io.cri-containerd.image=managed                                                             
sha256:d1e26b5f8193d713eb8f39d0ab0e3d513278f86e41644b07e189e1aef536b335                      application/vnd.oci.image.manifest.v1+json           sha256:ec8726f16d7625ff39a3bed2e10ca3ebf993d6aaff41bfe5b27789a98c24687b 132.8 MiB linux/amd64 io.cri-containerd.image=managed                                                             
sha256:d2c94e258dcb3c5ac2798d32e1249e42ef01cba4841c2234249495f87264ac5a                      application/vnd.oci.image.manifest.v1+json           sha256:c55c4ab38c73b5b59d76afd01083be207b1897142fee10222e56d1fd708efb62 15.5 KiB  linux/amd64 io.cri-containerd.image=managed                                                             
sha256:e561f36d6e02b8a81d168826bae864c0b3ee9b30fc4833348946b195899e5645                      application/vnd.oci.image.manifest.v1+json           sha256:1ddb60835e582d7569c46cd1c646bd287acb3d40fb17ec8e82b14b365edffa1d 195.3 MiB linux/amd64 io.cri-containerd.image=managed                                                             
sha256:ea7f826b15534016a9b8da3bda24ccebe3ff5066e0c351674533ddbd26fe521b                      application/vnd.oci.image.manifest.v1+json           sha256:09091a624ce276b2b4d51f6daf44eb742f117b1b10de318c769bc11d1d46d0cc 236.9 MiB linux/amd64 io.cri-containerd.image=managed                                                             
sha256:ead0a4a53df89fd173874b46093b6e62d8c72967bbf606d672c9e8c9b601a4fc                      application/vnd.docker.distribution.manifest.v2+json sha256:b64ac4c9100dd8ba255f2a31c2ad165f062e2cf087af9b6ffc68910e90de55d8 51.1 MiB  linux/amd64 io.cri-containerd.image=managed                                                             
sha256:ec9b9398017975117cb3dc8b57008130fabea764a7a476cfaf71c13ffa93e8d6                      application/vnd.oci.image.manifest.v1+json           sha256:11c0ce63bd7d111624abcfaf3151fa8e4c4cb47c02795445579ba48f29d6a8bc 64.9 MiB  linux/amd64 io.cri-containerd.image=managed                                                             
sha256:efa6d3d087772d1847c292e1ee576e17211c680fb2f12ecb4dc76d27aee902bf                      application/vnd.oci.image.manifest.v1+json           sha256:8f8866e7ae4b6a7a7cf69739dfcbebc684f006f5b488fa0c4009a217887d5957 200.7 MiB linux/amd64 io.cri-containerd.image=managed                                                             
sha256:f5f9f4739d5f434e759f6372f391527e842b878e864b71dbad5597ff45d69de0                      application/vnd.oci.image.manifest.v1+json           sha256:f05c5f4f371e004f35c888e7510faf91633f2a47570823aeb4b0038d0fb9be7c 42.0 MiB  linux/amd64 io.cri-containerd.image=managed                                                             
sha256:f772ce9ba10a47d09576b5b420a85e02b20c27b79100dcc4d40169686d402c89                      application/vnd.oci.image.manifest.v1+json           sha256:61c870b4f9dcac2d8782f0a95c594096067748e52e12d38febfd4983b5692619 334.9 MiB linux/amd64 io.cri-containerd.image=managed                                                             
sha256:fd4639eeb10a198a4126b1bf209bbea9b0abf47f39d7fb643dbd56f81f4eb02b                      application/vnd.oci.image.manifest.v1+json           sha256:9fbd965a15b635db41439b8ab0ee73d051b7bc0768b654634ba0bca932f804c4 353.7 MiB linux/amd64 io.cri-containerd.image=managed                                                             
bc@nuc1:~$ 

Agent node (nuc2):

bc@nuc2:~$ sudo ctr -n k8s.io image ls
REF TYPE DIGEST SIZE PLATFORMS LABELS 
bc@nuc2:~$ 

Ports are open - similar output on nuc1. The NUCs are vanilla Ubuntu Server 20.04 installs.

bc@nuc2:~$ sudo netstat -anp | grep 5001
tcp        0      0 192.168.10.52:5001      0.0.0.0:*               LISTEN      75146/k3s agent     
tcp        0      0 192.168.10.52:5001      192.168.10.51:5001      ESTABLISHED 75146/k3s agent     
unix  2      [ ]         DGRAM                    25001    743/udisksd          
bc@nuc2:~$ sudo netstat -anp | grep 6443
tcp        0      0 192.168.10.52:39388     192.168.10.51:6443      ESTABLISHED 75146/k3s agent     
tcp        0      0 127.0.0.1:59546         127.0.0.1:6443          ESTABLISHED 75160/containerd    
tcp        0      0 127.0.0.1:59534         127.0.0.1:6443          ESTABLISHED 75160/containerd    
tcp        0      0 127.0.0.1:35490         127.0.0.1:6443          ESTABLISHED 75160/containerd    
tcp        0      0 192.168.10.52:39534     192.168.10.51:6443      ESTABLISHED 75146/k3s agent     
tcp        0      0 127.0.0.1:59558         127.0.0.1:6443          ESTABLISHED 75160/containerd    
tcp        0      0 192.168.10.52:39252     192.168.10.51:6443      ESTABLISHED 75146/k3s agent     
tcp        0      0 192.168.10.52:39620     192.168.10.51:6443      ESTABLISHED 75146/k3s agent     
tcp        0      0 192.168.10.52:39640     192.168.10.51:6443      ESTABLISHED 75146/k3s agent     
tcp        0      0 127.0.0.1:40410         127.0.0.1:6443          ESTABLISHED 75160/containerd    
tcp        0      0 127.0.0.1:35470         127.0.0.1:6443          ESTABLISHED 75160/containerd    
tcp        0      0 192.168.10.52:39542     192.168.10.51:6443      ESTABLISHED 75146/k3s agent     
tcp        0      0 127.0.0.1:35474         127.0.0.1:6443          ESTABLISHED 75160/containerd    
tcp        0      0 127.0.0.1:59572         127.0.0.1:6443          ESTABLISHED 75160/containerd    
tcp        0      0 192.168.10.52:39390     192.168.10.51:6443      ESTABLISHED 75146/k3s agent     
tcp6       0      0 :::6443                 :::*                    LISTEN      75146/k3s agent     
tcp6       0      0 127.0.0.1:6443          127.0.0.1:35490         ESTABLISHED 75146/k3s agent     
tcp6       0      0 127.0.0.1:6443          127.0.0.1:59572         ESTABLISHED 75146/k3s agent     
tcp6       0      0 127.0.0.1:6443          127.0.0.1:40410         ESTABLISHED 75146/k3s agent     
tcp6       0      0 127.0.0.1:6443          127.0.0.1:59534         ESTABLISHED 75146/k3s agent     
tcp6       0      0 127.0.0.1:6443          127.0.0.1:59546         ESTABLISHED 75146/k3s agent     
tcp6       0      0 127.0.0.1:6443          127.0.0.1:35474         ESTABLISHED 75146/k3s agent     
tcp6       0      0 127.0.0.1:6443          127.0.0.1:59558         ESTABLISHED 75146/k3s agent     
tcp6       0      0 127.0.0.1:6443          127.0.0.1:35470         ESTABLISHED 75146/k3s agent     
bc@nuc2:~$ 

I wasn't aware I could place the images in a directory and they would be imported on startup. I could try that out if you think behavior might be different?

[brandond]

The images directory support is covered here:

• https://docs.k3s.io/installation/registry-mirror#sharing-air-gap-or-manually-loaded-images
• https://docs.k3s.io/installation/airgap#manually-deploy-images-method

As an added bonus, automatically imported images are pinned so that they are not removed by kubelet garbage collection when you run low on disk space.

Showing that the ports are listening on the nodes does not confirm that they are not being blocked by firewalls. Can you confirm that the ports are actually accessible between the two nodes?

You might also restart both nodes with --debug or debug: true; you'll get much more information from spegel in the logs. It should show you if the nodes are able to communicate with each other via the p2p mesh used by the embedded registry. If you can attach the k3s/k3s-agent logs from both nodes once they've been restarted with debug enabled, I can take a look.

[barrettc]

I ran nc on both hosts and I believe this proves the ports are open. nuc1 (server) is 192.168.10.51 and nuc2 (agent) is 192.168.10.52:

bc@nuc1:~$ nc -v 192.168.10.52 5001
Connection to 192.168.10.52 5001 port [tcp/*] succeeded!
?޳��&�P;��;.
            f�bp��t��Pj��?�t5-^C
bc@nuc1:~$ nc -v 192.168.10.52 6443
Connection to 192.168.10.52 6443 port [tcp/*] succeeded!

bc@nuc2:~$ nc -v 192.168.10.51 5001
Connection to 192.168.10.51 5001 port [tcp/*] succeeded!
�D������<�[<W�t`�ӧ+׸�y�4�^C
bc@nuc2:~$ nc -v 192.168.10.51 6443
Connection to 192.168.10.51 6443 port [tcp/*] succeeded!

I uninstalled both nodes and ran through the install process again putting the image files in /var/lib/rancher/k3s/agent/images to be imported automatically. I also added the --debug flag to both nodes. Attached are both logs - appreciate any insight!
nuc1-syslog.log
nuc2-syslog.log

[brandond]

My registries.yaml on both server and agent machines is:

Mar 5 22:38:17 nuc1 k3s[1298106]: time="2024-03-05T22:38:17Z" level=info msg="Starting distributed registry mirror at https://192.168.10.51:6443/v2 for registries [127.0.0.1:6443]"
Mar 5 22:42:01 nuc2 k3s[80507]: time="2024-03-05T22:42:01Z" level=info msg="Starting distributed registry mirror at https://192.168.10.52:6443/v2 for registries [127.0.0.1:6443 docker.io quay.io registry.k8s.io]"

Are you sure that you also populated registries.yaml on nuc1? The log output suggests that you have not.

[barrettc]

Thanks so much - turns out our automation was creating and populating the registries.yaml after starting up nuc1. I just bounced the server and agent nodes and now both can pull images correctly. Thanks for your help and patience!