ctr images pull --all-platforms --local docker.io/library/hello-world:latest fails: ctr: mismatched image rootfs and manifest layers

[AkihiroSuda]

Description

ctr images pull --all-platforms --local docker.io/library/hello-world:latest fails: ctr: mismatched image rootfs and manifest layers

Steps to reproduce the issue

ctr images pull --all-platforms --local docker.io/library/hello-world:latest

Describe the results you received and expected

Received:

[...]
docker.io/library/hello-world:latest:                                             resolved       |++++++++++++++++++++++++++++++++++++++| 
index-sha256:a26bff933ddc26d5cdf7faa98b4ae1e3ec20c4985e6f87ac0973052224d24302:    done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:decbb9d66399e68ef1d5ad6e0a84057a0f9bb3c7bfd9df9f454ff3c30b60fec6: done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:e2fc4e5012d16e7fe466f5291c476431beaa1f9b90a5c2125b493ed28e2aba57: done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:579b3724a7b189f6dca599a46f16d801a43d5def185de0b7bcd5fb9d1e312c27: done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:c2d891e5c2fb4c723efb72b064be3351189f62222bd3681ce7e57f2a1527362c: done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:6901d6a88eee6e90f0baa62b020bb61c4f13194cbcd9bf568ab66e8cc3f940dd: done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:c19784034d46da48550487c5c44639f5f92d48be7b9baf4d67b5377a454d92af: done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:951bcd144ddccd1ee902dc180b435faabaaa6a8747e70cbc893f2dca16badb94: done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:1363c810cc39a563c6f315e26951d2ed9e93f3bf929fde8223633ecf81a4a430: done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:f0c95f1ebb50c9b0b3e3416fb9dd4d1d197386a076c464cceea3d1f94c321b8f: done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:2d4e459f4ecb5329407ae3e47cbc107a2fbace221354ca75960af4c047b3cb13: done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:1f11fbd1720fcae3e402fc3eecb7d57c67023d2d1e11becc99ad9c7fe97d65ca: done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:65f4b0d1802589b418bb6774d85de3d1a11d5bd971ee73cb8569504d928bb5d9: done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:50f420e8710676da03668e446f1f51097b745e3e2c9807b018e569d26d4f65f7: done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:dbbd3cf666311ad526fad9d1746177469268f32fd91b371df2ebd1c84eb22f23: done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:c4383750f165e8167731ca109f3d81f30352e16e012304cb063963a0013fa582: done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:085e87951950ac62a771af158d4d8275505088897a0e520a8a5bd582343631b0: done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:838d191bca398e46cddebc48e816da83b0389d4ed2d64f408d618521b8fd1a57: done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:18b1c92de36d42c75440c6fd6b25605cc91709d176faaccca8afe58b317bc33a: done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:20aea1c63c90d5e117db787c9fe1a8cd0ad98bedb5fd711273ffe05c084ff18a: done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:8d064a6fc27fd5e97fa8225994a1addd872396236367745bea30c92d6c032fa3: done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:70304c314d8a61ba1b36518624bb00bfff8d4b6016153792042de43f0453ca61: done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:48147407c4594e45b7c3f0be1019bb0f44d78d7f037ce63e0e3da75b256f849e: done           |++++++++++++++++++++++++++++++++++++++| 
unknown-sha256:60925ab5943cc0f44164cbbaaaf26ca0647782b5d902aa5fc83dae6c73b78628:  done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:dd37070384f387096e10612bbfd83adae987f68f33e2c9ad969305add6178655:   done           |++++++++++++++++++++++++++++++++++++++| 
unknown-sha256:fcb31a5ff5fd111c07861e3095c730ecacd563c518b292229a3468a01573e475:  done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:59cba918efaeeed104a400d75a7061d0a9006af8e612512aa6d80f28cebe635b:   done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:9d2db35bc612e2795e6d57b74c5c11b8f795caf2a44a58d9d485d0d265cd5430:    done           |++++++++++++++++++++++++++++++++++++++| 
unknown-sha256:8b338b9c4c5a42cc78090f082d81ed4010c013d8ec03311051a613571f7f988a:  done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:cab575807a42d73fc069fad670c2c048565d9c81e91bf067b9e48b7e3d3fdae2:   done           |++++++++++++++++++++++++++++++++++++++| 
unknown-sha256:f0f2bb83c74908dafb652044b180ffa2f529712e248f0e9985e93699bdf3fbae:  done           |++++++++++++++++++++++++++++++++++++++| 
unknown-sha256:dc1c1f4e6745302150e513a0482baa032a41ef4171eb1fbf28f023008259d80a:  done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:a584f38a30b625084f4ed8df476de29a444f8696deb87e0e66bed224bcf59189:   done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:c1ec31eb59444d78df06a974d155e597c894ab4cda84f08294145e845394988e:    done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:d2c94e258dcb3c5ac2798d32e1249e42ef01cba4841c2234249495f87264ac5a:   done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:f19d1e240d64f25f24ec52b9399161571a6f124d583242557b40b291b7f499d4:    done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:37fc523148ea1437220df9c50eb5c90be527d1dd07629361ec9164a4172d6579:   done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:0c578c8e40773064b768668d7fa5f103f3810770fb1b59c74966f4453bea0be9:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:a9b4234352dbe48c2ab26f66b300829ca94d2fc63738ee6d4221f9962d33cf5c:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:6e7a4d21244056d6850efb8b3f1aea49148ce8cafd45059eef5c6eec348a3769:    done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:ea9f32d288d684ef2485c6aa1c5a498f76e926d998edc011f6c3da7c38f93adf:   done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:651a906a741b1aa0a559f1574eda2de929a757951e7475cf68a2460ebdffd212:    done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:8035a99ff60b3859336dc153247778542feaa3c14dab5cf03e8c7ffe5c1feb3a:   done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:347fc929ddc17ac8da9fb3271c1110d22298e5c8a030e25d9ea1509d4db74b73:    done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:ef56fe44c1ae4bebc38df8c301c8c5e8ab561b36dae0c734d1018f6c0e1b4684:   done           |++++++++++++++++++++++++++++++++++++++| 
unknown-sha256:b755f6ea9ed4e8b3579a5329933d9eec575ba1117510f2320726222a901e21bf:  done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:cee5a2917bd17938631e187dd0d685b7478685ef373078b92fc710c36ea5a817:   done           |++++++++++++++++++++++++++++++++++++++| 
unknown-sha256:ae1a9373da65899c4799f2eeb01f1545ab9767b6ba84a3bf39763be56a0299df:  done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:e9924fb81b02dc8bf974f18221cab43b0dfc32201ed86f0d1bdac33b3494a6b8:   done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:8c6776bffe9a8194dc50562905067f170b243d56364cce3bf8c12fcd7917fe2c:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:96b0fb4a0e42365a744646b3ace65ea47e6405b7f1d228307785f5b6baaaceb9:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:755fc767289b8847bd0d0d8d75efc308c040140acf2a3426973ba9fbf022c4c0:    done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:6183bba86cfeb06cb9cbe4f8722eda62896f191d6c42390bf3202eff8d7c5434:   done           |++++++++++++++++++++++++++++++++++++++| 
unknown-sha256:b382bcf7337b5acc658c4405b2d429d02f3dcaf87af69e9ce591a444b11ce23a:  done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:ae48fa9f4e38f262092e9ff13cbd5e3af7acdd29508ea5b05992c45e3fa0ecf7:   done           |++++++++++++++++++++++++++++++++++++++| 
unknown-sha256:9cec6b6d71d9885aff5e18ea66f5c98d2b4af09146a883c7de871a12076407c9:  done           |++++++++++++++++++++++++++++++++++++++| 
unknown-sha256:e0a75ff9ee852d6b8f9bf24a8587be3660e533249b04490c747c68cbc0eb3609:  done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:bfe083c3a47454aba7b2170f4d9955a86790a207d9ca92901086f09942df54ce:   done           |++++++++++++++++++++++++++++++++++++++| 
unknown-sha256:e5a36287fc1de00b683dd6cad99c6b4325917d4f58a68753f88e794c3c70184d:  done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:a78434e319074562cf136bd314faceedf90ac6626b371049cba4bcef51216454:    done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:dcd093bd8f96a1df8802da850e1a294799eef703a4e19e4475eaa789e193d5e9:   done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:2d3d56bba6abc27107251dcf47ca42bedaeda0105fc72cccbb2d618a2a79dcf8:    done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:7884a9d2ecf1870a475eb7118d49a1852d531b7077227854c7c008d5c53078f8:   done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:2c01f93040aa75d3ea306a02d7ba2b73aec9901fdec0df7b02174a76f710b0d7:    done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:ad522fe3b32489a86816cd6a5361b492eff94375c815c6be9b69edf7a9572096:   done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:eb7f8df2799ef21d497d2eb91368427b34e11c4ae737f72252dc0007a926f073:    done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:f0c407f2ecb92dc59aad1f53fda245bfa57b69c0935f55c9e28eb9756cfb292f:   done           |++++++++++++++++++++++++++++++++++++++| 
unknown-sha256:ce386759ac9119e33d2394bcfe30cf512f4fa1882fb3b3b4062d1d93240193d8:  done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:0dab78f2b0b8d96532b67783b03d2043d5a97a517780f9563c7c15d5758469bc:   done           |++++++++++++++++++++++++++++++++++++++| 
unknown-sha256:d0ed9663345f35049e061d1ebd5284ac03d6e4ace01966c581325221dbe7f996:  done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:0ee862cccdc937d47088ae305dc23c6532525f0a2999b8d063a5e0130a0035d5:   done           |++++++++++++++++++++++++++++++++++++++| 
unknown-sha256:a6205992dc80deb04831de9bcd9e3beec18e651598204e6e3c46c265815d9aaf:  done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:478afc9190022e867bb857b1a25cc5abc7678287af6cb930562ec25be709f1b7:    done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:ee301c921b8aadc002973b2e0c3da17d701dcd994b606769a7e6eaa100b81d44:   done           |++++++++++++++++++++++++++++++++++++++| 
unknown-sha256:cece95ee2767d2d10fbe1ae5875121ae9958d04ab439bc8af5e1c7138d3f4bf5:  done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:5c92b05ea639c36a03ec5c207a81694b498e0ce407b927ff02fd490da2cc8187:   done           |++++++++++++++++++++++++++++++++++++++| 
unknown-sha256:33b66aea0cd5e5861300a3a657eef734442172ac7d5d29af958aecf3940f6aa4:  done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 19.4s                                                                    total:  215.2  (11.1 MiB/s)                                      
unpacking linux/amd64 sha256:a26bff933ddc26d5cdf7faa98b4ae1e3ec20c4985e6f87ac0973052224d24302...
unpacking unknown/unknown sha256:a26bff933ddc26d5cdf7faa98b4ae1e3ec20c4985e6f87ac0973052224d24302...
ctr: mismatched image rootfs and manifest layers

Expected: no error

What version of containerd are you using?

containerd github.com/containerd/containerd v2.0.0-rc.1 2d19e9b

Any other relevant information

No response

Show configuration if it is related to CRI plugin.

No response

[akhilerm]

/assign

[akhilerm]

@AkihiroSuda Was trying to debug this issue. What confuses me is that it works when --local flag is not used. i.e if we are using transfer service, it works fine.
What I observed is on old releases(tried with v1.6.31, v1.6.28) of containerd also, we are hitting the same issue

[dmcgowan]

@akhilerm The way ctr handles local, it will call images.Platform and then iterate over each of those to call unpack. The unknown platform should not get returned from images.Platform. Seems we are probably correctly filtering out the unknown platform when the unpacker is used, although that might not be intentional and only because no rootfs is found in unknown images.

[akhilerm]

The unknown platform should not get returned from images.Platform

Is this the correct behaviour? Platforms() should return the platform that is in the image irrespective of whether its valid or not. Shouldnt the filtering happen from the output of Platforms() rather than from within the function.

[dmcgowan]

The unknown being there is a bit of a hack to indicate "not an image artifact type". Hopefully in the future this can be cleaned up but right now we should not return it as a result for "platforms supported by the image".

[dmcgowan]

Also I noticed in platforms package we will return "unknown" on Format when OS is empty however we don't handle that as input to Parse. I wonder if we should have a special case in platforms for this to treat unknown as empty values.

[akhilerm]

After checking in the platforms repo, I am confused on how unknown is really used. Format returns unknown when only the OS is empty, without any check on the architecture. But in the above image which triggered the issue , both OS and arch are set as unknown. In fix PR, I am checking both OS and arch.