Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow optimizing images with prioritiezed files info shared via registry #642

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

ktock
Copy link
Member

@ktock ktock commented Feb 15, 2022

This commit enables to share the "record file" (a JSON file containing a list of prioritized files) among hosts/users via container registry. The record file is packed as an OCI image called "record file image".

"Record file image" is an OCI image only contains one layer that contains the record file as a tar entry named stargz.record.json.

ctr-remote image optimize with --record-out-ref creates a record file image. ctr-remote image convert with --estargz-record-in-ref enables to create an eStargz image with applying the prioritized files information described in the specified record file image.

Generating record file image

The following command generates a record file image named registry2:5000/golang:record-1.15.3-esgz-go-version.

ctr-remote image optimize --oci --record-out-ref=registry2:5000/golang:record-1.15.3-esgz-go-version \
           --entrypoint='[ "/bin/bash", "-c" ]' --args='[ "go version" ]' \
           ghcr.io/stargz-containers/golang:1.15.3-buster-org \
           registry2:5000/golang:1.15.3-esgz-go-version

The follwoing is the manifest of the record file image registry2:5000/golang:record-1.15.3-esgz-go-version.

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:518c804adcec04f5148877af99ec6d216eb50fc47b0b1430217163b838ef7f15",
    "size": 164
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:5b37372a28490bb0737aaadb48241b72ca6e54e023b15ac481a5a3df0da69dcc",
      "size": 344
    }
  ]
}

Layer blob sha256:5b37372a28490bb0737aaadb48241b72ca6e54e023b15ac481a5a3df0da69dcc is a tar.gz blob that contains the record file as a tar entry named stargz.record.json.

# cat /tmp/img/blobs/sha256/5b37372a28490bb0737aaadb48241b72ca6e54e023b15ac481a5a3df0da69dcc | tar -zxO stargz.record.json
{"path":"bin/bash","manifestDigest":"sha256:2322be6aa1e61c36cfa34a0d355b7f088022e34c6f17016bafae6041c67a0a1b","layerIndex":0}
{"path":"bin/bash","manifestDigest":"sha256:2322be6aa1e61c36cfa34a0d355b7f088022e34c6f17016bafae6041c67a0a1b","layerIndex":0}
{"path":"lib/x86_64-linux-gnu/ld-2.28.so","manifestDigest":"sha256:2322be6aa1e61c36cfa34a0d355b7f088022e34c6f17016bafae6041c67a0a1b","layerIndex":0}
...(omit)...

Consuming record file image

ctr-remote image convert command with --estargz-record-in-ref option allowes you to use the recrod file image for optimizing a new eStargz image. The newly-created eStargz will be applied the prioritized files information described in the record file.

ctr-remote image convert --oci --estargz \
           --estargz-record-in-ref registry2:5000/golang:record-1.15.3-esgz-go-version \
           ghcr.io/stargz-containers/golang:1.15.3-buster-org \
           registry2:5000/golang:1.15.3-esgz-go-version-static

Prioritized files information described in the record file image is applied to the newly-creating eStargz in a best-effort manner.
Therefore, files contained in both of the record file and the newly-creating eStargz are considered as the prioritized files.

Copying prioritized files information from an existing eStargz

This commit also adds an ability to copy prioritized files information from an existing eStargz image to a newly-creating eStargz.
The following example creates a new eStargz image registry2:5000/golang:1.15.3-esgz-go-version-static with the same prioritized files configuration as an existing eStargz registry2:5000/golang:1.15.3-esgz-go-version.

ctr-remote image convert --oci --estargz \
           --estargz-record-copy registry2:5000/golang:1.15.3-esgz-go-version \
           ghcr.io/stargz-containers/golang:1.15.3-buster-org \
           registry2:5000/golang:1.15.3-esgz-go-version-static

Now we have 3 approaches for creating an eStargz image statically (i.e. without running the profiler).
Updated ctr-remote.md docs to describe all of these methods.

  • Copying prioritized files information from an existing eStargz image (ctr-remote image convert --estargz-record-copy)
  • Acquiering prioritized files information shared as a file (ctr-remote image convert --estargz-record-in)
  • Acquiering prioritized files information shared via registry (ctr-remote image convert --estargz-record-in-ref)

@AkihiroSuda
Copy link
Member

What's the expected usecase?
Isn't landmark data already available in eStargz images?

@ktock ktock marked this pull request as draft February 15, 2022 08:15
@ktock
Copy link
Member Author

ktock commented Feb 15, 2022

This patch intended to provide a way to share pre-defined record files to such tools that don't have container runtime functionality (e.g. crane by go-containerregistry).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants