public-mirror.server (publishConfig.publicMirrorAcr.server) is now required after the change to run syft directly

[dagood]

PR #1787 adds an unconditional dependency on having a public mirror defined:

docker-tools/eng/common/templates/jobs/build-images.yml

Line 107 in 7841dec

$syftImageName = "${{ parameters.publishConfig.publicMirrorAcr.server }}/$(imageNames.syft)"

In the microsoft/go-images pipelines, we define public-mirror.server to be ''. This has seemed reasonable because there's some logic elsewhere that detects/handles it:

docker-tools/eng/common/templates/steps/init-matrix-build-publish.yml

Lines 26 to 28 in 7841dec

	if ("$(System.TeamProject)" -eq "public" -and "$(public-mirror.server)" -ne "") {
	$commonMatrixAndBuildOptions = "$commonMatrixAndBuildOptions --base-override-regex '^(?!mcr\.microsoft\.com)' --base-override-sub '$(public-mirror.server)/'"
	}

So, our build fails when trying to upgrade past it: https://dev.azure.com/dnceng/internal/_build/results?buildId=2787270&view=logs&j=fc59f0f2-c1bd-58ae-b870-833d1e8a924c&t=42b16712-47b7-5d4e-b8e4-5c4616c57a47&l=133

What's the best path forward for us?

(I'm not sure what it takes to maintain a public mirror--I'm not necessarily against it. I'm just not sure if it's necessary for our attempting-to-be-low-maintenance reuse of the .NET Docker infra. 🙂 Dev time is still much costlier than build time in our space.)

[lbussell]

Ack - I don't think it's intentional. Not using a mirror opens you up to DockerHub rate limiting.
You could use our mirror just for this usage of Syft, since we'll have it mirrored already. But I'm not sure that's going to work without additional changes. If you define parameters.publishConfig.publicMirrorAcr.server, it's probably going to trigger other conditions.

/cc @mthalman

[mthalman]

How have you been getting images from DockerHub otherwise? Do you have your own login?

[lbussell]

P.S. we can probably work around this in the short term with an expression like:

${{ iif(parameters.publishConfig.publicMirrorAcr, format('{0}/', parameters.publishConfig.publicMirrorAcr.server), '') }}

[dagood]

How have you been getting images from DockerHub otherwise? Do you have your own login?

We only use base images from MAR. (If there's another DockerHub image being used in common infra already, I guess we just haven't noticed any rate limiting yet, or perhaps we're sharing access.)

But I'm not sure that's going to work without additional changes. If you define parameters.publishConfig.publicMirrorAcr.server, it's probably going to trigger other conditions.

So far it looks like this is the only place publicMirrorAcr shows up in eng/common, and even setting public-mirror.server seems like it would only turn on base image redirect if we had any non-MAR images in our Dockerfiles, but that's just from a quick read.

But... I think that means we'd need a service connection to access your public-mirror ACR?

[dagood]

Setting up public-mirror seems to work! Up to you all whether this issue deserves any more followup/tracking. This state seems like it will be fine for us in particular.

[dagood]

I suppose I'll say though: it seems like it would be nice if this infra could work with only dependencies on MAR. But that's just a gut feeling.

[mthalman]

One option is to put syft into the Image Builder image.

[lbussell]

[Triage] If we added syft to the Image Builder image we could just include the SBOM generating direclty in the build command as well (which would simplify the pipeline).