Doc: Provide example of building image #14
Labels
documentation
Improvements or additions to documentation
Comments
What's non-reproducible? |
|
Here is my console output: ➜ repro-sources-list.sh git:(master) ✗ docker buildx build -o type=oci,dest=image.img,rewrite-timestamp=true -f Dockerfile.debian-12 --platform linux/amd64 --no-cache --build-arg SOURCE_DATE_EPOCH=1675298208 .
[+] Building 97.3s (8/8) FINISHED docker-container:multiarch2
=> [internal] load build definition from Dockerfile.debian-12 0.0s
=> => transferring dockerfile: 416B 0.0s
=> [internal] load metadata for docker.io/library/debian:bookworm-20230904-slim 1.3s
=> [auth] library/debian:pull token for registry-1.docker.io 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 43B 0.0s
=> CACHED [stage-0 1/2] FROM docker.io/library/debian:bookworm-20230904-slim@sha256:050f00e86cc4d9 0.0s
=> => resolve docker.io/library/debian:bookworm-20230904-slim@sha256:050f00e86cc4d928b21de66096126 0.0s
=> [stage-0 2/2] RUN --mount=type=cache,target=/var/cache/apt,sharing=locked --mount=type=cac 81.8s
=> exporting to oci image format 14.1s
=> => exporting layers 6.5s
=> => rewriting layers with source-date-epoch 1675298208 (2023-02-02 00:36:48 +0000 UTC) 6.9s
=> => exporting manifest sha256:d6997b9712dcae63aa13aba30a08fab9a18d7a65981d921fcb91c296eec8359e 0.0s
=> => exporting config sha256:3fae53ed7a611e1ad0f533e234cc89a7238276f8bc766bac496111a1d65da272 0.0s
=> => sending tarball 0.8s
➜ repro-sources-list.sh git:(master) ✗ docker buildx build -o type=oci,dest=image2.img,rewrite-timestamp=true -f Dockerfile.debian-12 --platform linux/amd64 --no-cache --build-arg SOURCE_DATE_EPOCH=1675298208 .
[+] Building 89.6s (7/7) FINISHED docker-container:multiarch2
=> [internal] load build definition from Dockerfile.debian-12 0.0s
=> => transferring dockerfile: 416B 0.0s
=> [internal] load metadata for docker.io/library/debian:bookworm-20230904-slim 0.5s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 43B 0.0s
=> CACHED [stage-0 1/2] FROM docker.io/library/debian:bookworm-20230904-slim@sha256:050f00e86cc4d9 0.0s
=> => resolve docker.io/library/debian:bookworm-20230904-slim@sha256:050f00e86cc4d928b21de66096126 0.0s
=> [stage-0 2/2] RUN --mount=type=cache,target=/var/cache/apt,sharing=locked --mount=type=cac 74.9s
=> exporting to oci image format 14.2s
=> => exporting layers 6.5s
=> => rewriting layers with source-date-epoch 1675298208 (2023-02-02 00:36:48 +0000 UTC) 6.8s
=> => exporting manifest sha256:f3f7ebaa5cd2498431d4c1cc275e3ecc7490a0f77743e7b02d7bf787d45faf6c 0.0s
=> => exporting config sha256:2b0a504fbecbdce0009173ef63d3db74df58800aaab8aaf44c3fd50608c0b2f1 0.0s
=> => sending tarball 0.8s
➜ repro-sources-list.sh git:(master) ✗ sha256sum image.img
2001dd290e510347e01bc04a03e2b73b842cda6256418fb44fc802284a562843 image.img
➜ repro-sources-list.sh git:(master) ✗ sha256sum image2.img
\2fa5d75d6665dad76924ca277ccf3543be5c94f071e12cc78e9ecdc27f7f1316 image2.imgManfest, config and actual image files have different hashes with two back to back runs on the same machine. Do you have an example buildx command where they come out the same? |
|
The differences are caused by:
You may just remove them with Full example: |
|
@AkihiroSuda I'm not sure if this is the place to talk about this but you seem like the most knowledgeable person on this process. With the changes made in #16 I now have a reproducible build for linux/amd64 and the same dockerfile for linux/arm64 but when I tell buildx to make both into a multi-arch image for some reason the output is different even though the two separate arches are reproducible when built alone. Where is the right place to raise this? This was my command: docker buildx build -o type=oci,dest=image.img,rewrite-timestamp=true -f Dockerfile.debian-12 --platform linux/amd64,linux/arm64 --no-cache --build-arg SOURCE_DATE_EPOCH=1675298208 .Interestingly, I'm building on two different machines and comparing the output. One is an amd64, and one is arm64. They both produce the same output (when producing a single arch image) even though obviously each time for one of them it's a non-native build. |
|
You may use https://github.com/reproducible-containers/diffoci to check the diff. It should be noted that the scope of repro build is limited to manifests, not indexes, as an index may contain references to SPDX provenances, etc. |
|
Thanks for the tool @AkihiroSuda! Output of diffoci: DebainBuild ➜ repro-sources-list.sh git:(master) ✗ ./diffoci-v0.1.4.linux-amd64 diff docker://maxedw/image1 docker://maxedw/image2
INFO[0000] Target platforms: [linux/amd64]
INFO[0000] Loading image "docker.io/maxedw/image1:latest" from "docker"
docker.io/maxedw/image1:latest saved
Importing elapsed: 0.8 s total: 0.0 B (0.0 B/s)
INFO[0001] Loading image "docker.io/maxedw/image2:latest" from "docker"
docker.io/maxedw/image2:latest saved
Importing elapsed: 0.8 s total: 0.0 B (0.0 B/s)
TYPE NAME INPUT-0 INPUT-1
Desc application/vnd.oci.image.manifest.v1+json fd5379d8878302e938698586db3869b25bbb0bca1030eb48835bd84a47187b6d 1ea38231b7eea6d91ac8bd96602b30aafd6e6c88f25b5e514f420be676a421d0
Desc application/vnd.oci.image.manifest.v1+json fd5379d8878302e938698586db3869b25bbb0bca1030eb48835bd84a47187b6d 1ea38231b7eea6d91ac8bd96602b30aafd6e6c88f25b5e514f420be676a421d0
Mani ctx:/manifest ? ?
Desc application/vnd.oci.image.config.v1+json 0611c3c137f98c0f4990d26f57a8edaa156fc4be613cfd34811356dee03c3dfe 1ba11898aa9b6d3e579bb6a5b51c82b1e2dade40fc1c834164a4ae09fda879ef
Cfg ctx:/config/config ? ?
Desc application/vnd.oci.image.layer.v1.tar cf3420c34116c7777c0660d1e99d477bb48d347fa660da216867ccbdee7fd02c f02cc727caf22e74251fcdec0164e1e6b78e87b2edbedccac4b2ca975a73af65
File blobs/sha256/155eab17d86c47443adc8cebe7fc62c847c03db8cfb1ca53aa6276564fff23ef Index 3 Index 4
File blobs/sha256/d2c7ba5b367ab427569aa8e0f6551a224ee6b1b31cc585b3625fb676355666af Index 13 Index 12
File blobs/sha256/d5a1697de149c4584e824aed15ffe82412466e700f7a1c15451b255cd8a79511 Index 14 Index 13
File blobs/sha256/ee6efcfa9510fa714668f0a2f00b256cfcaa3538c1108a42e36e32ea6faf12ec Index 15 Index 14
File blobs/sha256/1f2a5c5f21abf788427963c515a0b60359ebc0b594789f425428a6b8cf97eda9 Index 4 Index 5
Layer ctx:/layers-0/layer name "blobs/sha256/2ee6937ec697a293be718f27df65fc665d355f2edede02fa0a071b43f8089302" only appears in input 0
Layer ctx:/layers-0/layer name "blobs/sha256/47f94f0e4dda0e127756f9aa48541b15e140b923ceddcc9c8e2597e40779fd80" only appears in input 0
File blobs/sha256/0b9622d7cd405af087a352f882207fa2e54e85988bb395956840c3a13afcf21d Index 2 Index 3
Layer ctx:/layers-0/layer name "blobs/sha256/a44de156a7f46cac4ae15feb3e8e2d3127b9b6349761089b2f0f9bcf43e62bd5" only appears in input 0
Layer ctx:/layers-0/layer name "blobs/sha256/8abb5ebae0c47c283756a9de0ca2b2287a7563e935b2162a57ea7c6651107774" only appears in input 0
Layer ctx:/layers-0/layer name "blobs/sha256/94f26548633ca6e1b1af4486913c750c5628c79179b2a3cc67443a9bef6b625b" only appears in input 0
File index.json c9549cee48733bf4aa5ad635a19113f925a3b0508061af409adf017ac50d387c d3b66896f36f0645af0086ac1beeab1a648b3c79e823e7eeaafb607c5d4aa537
Layer ctx:/layers-0/layer name "blobs/sha256/33bf26bf869c7497c4ffdde1842fc76d1fc8db78cc2103395d37d59c85c61392" only appears in input 0
Layer ctx:/layers-0/layer name "blobs/sha256/5abc1189feeab99aad971fdae2a199a70e559b440a119ba3b9a928e7db746b55" only appears in input 0
Layer ctx:/layers-0/layer name "blobs/sha256/ae656186cf62c3ffff1f3ffaff1e569e92c8f79302e333fa9a4ff14e6570fd43" only appears in input 1
Layer ctx:/layers-0/layer name "blobs/sha256/d22f304ed9b9719867ebdab1f6a4fce6a8b4d14e66a241dff3a1d4b4e7d297fe" only appears in input 1
Layer ctx:/layers-0/layer name "blobs/sha256/0b8418c757238f8c0bebdf63724fa00c36d3ecb1b7eb866780364a56f982f7de" only appears in input 1
Layer ctx:/layers-0/layer name "blobs/sha256/27153e7f7ca35cfe83c9e6ed9b12b32dbcac84c613776d3ea34ef9d4fcc0ef86" only appears in input 1
Layer ctx:/layers-0/layer name "blobs/sha256/c3eebf09ec733b8f01c4d3db506b42fd66331fe157335faa34a94e3c95c436ee" only appears in input 1
Layer ctx:/layers-0/layer name "blobs/sha256/acf1548d261354d48be809a35ae2e8730a178866adbdf0b32d98d7010e3148b0" only appears in input 1
Layer ctx:/layers-0/layer name "blobs/sha256/ee87862996bc639ac76fd54dfd0778c97a44bbd2efb2184a1c3957bf198bc6be" only appears in input 1Don't fully follow what the doffoci is telling me here but shall I just assume that it's currently not possible to make reproducible multi-arch images? |
Possible, but you may have to specify registry reference strings rather than |
|
This is the result I got (repro-sources-list.sh v0.1.3, buildx v0.14.0, BuildKit v0.13.1): $ docker buildx build -o type=image,name=docker.io/akihirosuda/tmp20240424:0,push=true,rewrite-timestamp=true -f Dockerfile.debian-12 --platform linux/amd64,linux/arm64 --no-cache --build-arg SOURCE_DATE_EPOCH=1675298208 .
[...]
=> exporting to image 61.9s
=> => exporting layers 12.9s
=> => rewriting layers with source-date-epoch 1675298208 (2023-02-02 00:36:48 +0000 UTC) 10.8s
=> => exporting manifest sha256:efbda9bc79ec583cd48edd015ace1f05a5807e27477f868add652e6d64dc6f20 0.0s
=> => exporting config sha256:d5a1697de149c4584e824aed15ffe82412466e700f7a1c15451b255cd8a79511 0.0s
=> => exporting attestation manifest sha256:4e97c4b92482f5798563eb0294eddbd3d991573726261e4dfc3230c21db58fee 0.0s
=> => exporting manifest sha256:1f2a5c5f21abf788427963c515a0b60359ebc0b594789f425428a6b8cf97eda9 0.0s
=> => exporting config sha256:ee6efcfa9510fa714668f0a2f00b256cfcaa3538c1108a42e36e32ea6faf12ec 0.0s
=> => exporting attestation manifest sha256:c94dde3b8ce891dfca5ad673b253d04ebe48cc9246e3415e9a89d45fd3cfb267 0.0s
=> => exporting manifest list sha256:787688757b1caad6539976fd5a64dc50416b24505d40f01e6d034b70341f4d98 0.0s
=> => pushing layers 19.9s
=> => pushing manifest for docker.io/akihirosuda/tmp20240424:0@sha256:787688757b1caad6539976fd5a64dc50416b24505d40f01e6d034b70341f4d98 4.5s
=> [auth] akihirosuda/tmp20240424:pull,push token for registry-1.docker.io 0.0s$ docker buildx build -o type=image,name=docker.io/akihirosuda/tmp20240424:1,push=true,rewrite-timestamp=true -f Dockerfile.debian-12 --platform linux/amd64,linux/arm64 --no-cache --build-arg SOURCE_DATE_EPOCH=1675298208 .
[...]
=> exporting to image 51.4s
=> => exporting layers 12.7s
=> => rewriting layers with source-date-epoch 1675298208 (2023-02-02 00:36:48 +0000 UTC) 13.7s
=> => exporting manifest sha256:efbda9bc79ec583cd48edd015ace1f05a5807e27477f868add652e6d64dc6f20 0.0s
=> => exporting config sha256:d5a1697de149c4584e824aed15ffe82412466e700f7a1c15451b255cd8a79511 0.0s
=> => exporting attestation manifest sha256:64aa10db60f094ade29be8c949c83e012a35f140e7f8da6ea9f3beef869e73d6 0.0s
=> => exporting manifest sha256:1f2a5c5f21abf788427963c515a0b60359ebc0b594789f425428a6b8cf97eda9 0.0s
=> => exporting config sha256:ee6efcfa9510fa714668f0a2f00b256cfcaa3538c1108a42e36e32ea6faf12ec 0.0s
=> => exporting attestation manifest sha256:6c0dc5e87e4e5a0434c000d524d6f06b7ad3d5e5ae4196a99574beea6da34731 0.0s
=> => exporting manifest list sha256:71b28118565e07eece365aabaa6b40d3acdc2eb210c57c22a62addc9c4586189 0.0s
=> => pushing layers 5.3s
=> => pushing manifest for docker.io/akihirosuda/tmp20240424:1@sha256:71b28118565e07eece365aabaa6b40d3acdc2eb210c57c22a62addc9c4586189 4.9s
=> [auth] akihirosuda/tmp20240424:pull,push token for registry-1.docker.ioWhile the digest of the manifest list (aka OCI index) is not reproducible as it contains the attestation manifest (SLSA provenance), the image manifests themselves are 100% reproducible:
See https://explore.ggcr.dev/?image=docker.io%2Fakihirosuda%2Ftmp20240424%3A0 and https://explore.ggcr.dev/?image=docker.io%2Fakihirosuda%2Ftmp20240424%3A1 |
|
@AkihiroSuda I truly appreciate you taking the time to run those examples for me! |
|
The manifest list (index) can be made reproducible too by specifying $ docker buildx build -o type=image,name=docker.io/akihirosuda/tmp20240424:no-provenance-0,push=true,rewrite-timestamp=true -f Dockerfile.debian-12 --platform linux/amd64,linux/arm64 --no-cache --build-arg SOURCE_DATE_EPOCH=1675298208 --provenance=false .
[...]
=> exporting to image 46.8s
=> => exporting layers 14.1s
=> => rewriting layers with source-date-epoch 1675298208 (2023-02-02 00:36:48 +0000 UTC) 11.7s
=> => exporting manifest sha256:9959b500c85383c0c534d67d4411b98e4031db54d629f28cfef0a945d7dfdd09 0.0s
=> => exporting config sha256:d5a1697de149c4584e824aed15ffe82412466e700f7a1c15451b255cd8a79511 0.0s
=> => exporting manifest sha256:c463b35ef852663064a67fe055b59df86991403275de7da113f4eaa5cdc50e26 0.0s
=> => exporting config sha256:ee6efcfa9510fa714668f0a2f00b256cfcaa3538c1108a42e36e32ea6faf12ec 0.0s
=> => exporting manifest list sha256:13954d9f9d01070d648d2e00b15d152fcb58ecf4a93fac1cde89672c61fb0e7e 0.0s
=> => pushing layers 2.4s
=> => pushing manifest for docker.io/akihirosuda/tmp20240424:no-provenance-0@sha256:13954d9f9d01070d648d2e00b15d152fcb58ecf4a93fac1cde89672c61fb0e7e 3.9s
=> [auth] akihirosuda/tmp20240424:pull,push token for registry-1.docker.io 0.0s$ docker buildx build -o type=image,name=docker.io/akihirosuda/tmp20240424:no-provenance-1,push=true,rewrite-timestamp=true -f Dockerfile.debian-12 --platform linux/amd64,linux/arm64 --no-cache --build-arg SOURCE_DATE_EPOCH=1675298208 --provenance=false .
[...]
=> exporting to image 45.0s
=> => exporting layers 13.5s
=> => rewriting layers with source-date-epoch 1675298208 (2023-02-02 00:36:48 +0000 UTC) 11.2s
=> => exporting manifest sha256:9959b500c85383c0c534d67d4411b98e4031db54d629f28cfef0a945d7dfdd09 0.0s
=> => exporting config sha256:d5a1697de149c4584e824aed15ffe82412466e700f7a1c15451b255cd8a79511 0.0s
=> => exporting manifest sha256:c463b35ef852663064a67fe055b59df86991403275de7da113f4eaa5cdc50e26 0.0s
=> => exporting config sha256:ee6efcfa9510fa714668f0a2f00b256cfcaa3538c1108a42e36e32ea6faf12ec 0.0s
=> => exporting manifest list sha256:13954d9f9d01070d648d2e00b15d152fcb58ecf4a93fac1cde89672c61fb0e7e 0.0s
=> => pushing layers 2.5s
=> => pushing manifest for docker.io/akihirosuda/tmp20240424:no-provenance-1@sha256:13954d9f9d01070d648d2e00b15d152fcb58ecf4a93fac1cde89672c61fb0e7e 1.6s
=> [auth] akihirosuda/tmp20240424:pull,push token for registry-1.docker.io |
|
BTW |
Amazing!
Great catch |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I'm trying to build and I'm not getting reproducible images. This is the buildx command I'm using:
docker buildx build -o type=oci,dest=image.img,rewrite-timestamp=true -f Dockerfile.debian-12 --platform linux/amd64 --no-cache --build-arg SOURCE_DATE_EPOCH=1675298208 .Could there be an example buildx invocation in the README to help someone use this strategy?
The text was updated successfully, but these errors were encountered: