Preloading Images in an Airgap Environment

[NullOranje]

I'm attempting to preload images into an airgapped RKE2 installation with the intent of bootstrapping other services like a registry. So far, I've attempted to save the image using skopeo and placing that file in the path /var/lib/rancher/rke2/

For testing purposes, I'm attempting to

skopeo copy docker://nginx:1.25.4 docker-archive:nginx_1.25.4.tar

But when RKE2 starts, I don't see anything imported like I do when I import the images from /var/lib/rancher/rke2/agent/images/rke2-images.linux-amd64.tar.gz

Mar 19 16:25:38 rke-test-01 rke2[140300]: time="2024-03-19T16:25:38-04:00" level=info msg="Importing images from /var/lib/rancher/rke2/agent/images/nginx_1.25.4.tar"
Mar 19 16:25:40 rke-test-01 rke2[140300]: time="2024-03-19T16:25:40-04:00" level=info msg="Imported images from /var/lib/rancher/rke2/agent/images/nginx_1.25.4.tar in 2
.6164437s"

And I don't see the images using crictl:

IMAGE                                                                TAG                                        IMAGE ID            SIZE
docker.io/rancher/hardened-calico                                    v3.27.0-build20240206                      ce486240f5e24       563MB
docker.io/rancher/hardened-cluster-autoscaler                        v1.8.10-build20240124                      cd00dc5289588       38.3MB
docker.io/rancher/hardened-coredns                                   v1.11.1-build20240123                      c7bf5a5dbd615       59.3MB
docker.io/rancher/hardened-dns-node-cache                            1.22.28-build20240125                      2563e24ce2d2c       64MB
docker.io/rancher/hardened-etcd                                      v3.5.9-k3s1-build20230802                  c6b7a4f2f79b2       168MB
docker.io/rancher/hardened-flannel                                   v0.24.2-build20240122                      40a249b9c4933       221MB
docker.io/rancher/hardened-k8s-metrics-server                        v0.6.3-build20231009                       9051c01da7c50       173MB
docker.io/rancher/hardened-kubernetes                                v1.27.11-rke2r1-build20240214              1cb1295bad8f5       699MB
docker.io/rancher/klipper-helm                                       v0.8.2-build20230815                       5f89cb8137ccb       256MB
docker.io/rancher/klipper-lb                                         v0.4.5                                     17066c233afa3       18.9MB
docker.io/rancher/mirrored-ingress-nginx-kube-webhook-certgen        v20230312-helm-chart-4.5.2-28-g66a760794   5a86b03a88d23       48.5MB
docker.io/rancher/mirrored-pause                                     3.6                                        6270bb605e12e       686kB
docker.io/rancher/mirrored-sig-storage-snapshot-controller           v6.2.1                                     1ef6c138bd5f2       58.4MB
docker.io/rancher/mirrored-sig-storage-snapshot-validation-webhook   v6.2.2                                     ff52c2bcf9f88       49MB
docker.io/rancher/nginx-ingress-controller                           nginx-1.9.3-hardened1                      bfdece8fa3f14       800MB
docker.io/rancher/rke2-cloud-provider                                v1.28.2-build20231016                      00bad7e3c2263       188MB
docker.io/rancher/rke2-runtime                                       v1.27.11-rke2r1                            9bcc407d2d47d       330MB

Is this something that is possible to do? If so, what is the proper way to load the images?

[brandond]

I've attempted to save the image using skopeo and placing that file in the path /var/lib/rancher/rke2/

That's not where the files go. They are supposed to be put in /var/lib/rancher/rke2/agent/images, as the logs indicate.

The archive also needs to be in the tarball format produced by docker save. Skopeo does not provide the same output format.

[NullOranje]

I've attempted to save the image using skopeo and placing that file in the path /var/lib/rancher/rke2/
That's not where the files go. They are supposed to be put in /var/lib/rancher/rke2/agent/images, as the logs indicate.

Ignore the bad copypasta.

The archive also needs to be in the tarball format produced by docker save. Skopeo does not provide the same output format.

I think this was my missing link. What's different between the tools? We don't use docker in our workflow for a number of reasons, so I'm curious if it is just some deficiency in Skopeo or some other issue that could be resolved in the future.

[brandond]

From the skopeo docs:

docker-archive:path[:docker-reference] An image is stored in a docker save-formatted file. docker-reference is only used when creating such a file, and it must not contain a digest.

I think you would need to include the :docker-reference portion of the output spec for the image to be imported properly, otherwise the tarball doesn't contain information indicating what the actual image should be named when it is loaded. That's my understanding at least.