Fix image caching logic to detect changes in intermediate stage FROM images

[Copilot]

Problem

The image caching logic was incorrectly skipping builds when a Dockerfile had intermediate stages based on images that had changed. The cache validation only checked the final stage's base image, not all FROM images used throughout the Dockerfile.

Example Scenario

Consider this multi-stage Dockerfile from dotnet-buildtools-prereqs-docker:

# Intermediate stage - crossdeps-builder changes
FROM crossdeps-builder:3.0 AS builder
RUN build-dependencies

# Final stage - crossdeps-llvm doesn't change
FROM crossdeps-llvm:3.0
COPY --from=builder /artifacts /

Before this fix: Cache hit reported ❌ (only checked crossdeps-llvm, missed the crossdeps-builder change)
After this fix: Cache miss reported ✅ (checks both crossdeps-builder and crossdeps-llvm, detects the change)

This caused builds to skip images that actually needed to be rebuilt because they copied artifacts from changed intermediate stages.

Solution

1. Track All FROM Images

Added a FromImages property to PlatformData that stores a dictionary mapping each FROM image to its digest:

[JsonProperty(NullValueHandling = NullValueHandling.Ignore)]
public Dictionary<string, string>? FromImages { get; set; }

This property is serialized to the image-info.json file and tracks all FROM images in the Dockerfile, not just the final stage.

2. Populate FROM Images During Build

Created SetPlatformDataFromImagesAsync() in BuildCommand to:

• Collect all internal FROM images (from the manifest) and external FROM images (from registries)
• Resolve their digests from either the platform data cache or image digest cache
• Store the mappings in the FromImages dictionary

3. Validate All FROM Images During Cache Check

Enhanced ImageCacheService with AreFromImageDigestsUpToDateAsync() to:

• Iterate through all current FROM images in the Dockerfile
• Compare each image's current digest with the recorded digest from FromImages
• Return a cache miss if any FROM image has changed

4. Maintain Backward Compatibility

If FromImages is null or empty (e.g., in existing image-info files), the code falls back to the legacy behavior of checking only the final stage base image. This ensures existing workflows continue to work while new builds benefit from the improved caching logic.

Changes

• PlatformData.cs: Added FromImages property
• BuildCommand.cs: Added logic to populate and copy FromImages during build
• ImageCacheService.cs: Enhanced cache validation to check all FROM images
• BuildCommandTests.cs: Added comprehensive test for intermediate stage cache invalidation

Testing

Added BuildCommand_Caching_IntermediateStageChange test that validates:

• When an intermediate stage FROM image changes, the cache is correctly invalidated
• When the final stage FROM image changes, the cache is correctly invalidated
• Images are rebuilt when any of their FROM dependencies change

Impact

This fix ensures that images with multi-stage Dockerfiles are correctly rebuilt when any of their FROM dependencies change, not just the final stage. This is particularly important for the dotnet-buildtools-prereqs-docker repository where this issue was originally observed.

Fixes scenario described in the issue where changes to crossdeps-builder Dockerfile were not triggering rebuilds of the cross image that depended on it.

---

Implementation suggested by @lbussell in issue comments.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• 11vvsblobprodcus336.vsblob.vsassets.io
  • Triggering command: dotnet build (dns block)
• 2kmvsblobprodcus39.vsblob.vsassets.io
  • Triggering command: dotnet build (dns block)
• 2zrvsblobprodcus388.vsblob.vsassets.io
  • Triggering command: dotnet build (dns block)
• 37cvsblobprodcus359.vsblob.vsassets.io
  • Triggering command: dotnet build (dns block)
• 4myvsblobprodcus32.vsblob.vsassets.io
  • Triggering command: dotnet build (dns block)
• 4zjvsblobprodcus390.vsblob.vsassets.io
  • Triggering command: dotnet build (dns block)
• 5dkvsblobprodcus355.vsblob.vsassets.io
  • Triggering command: dotnet build (dns block)
• 5rqvsblobprodcus385.vsblob.vsassets.io
  • Triggering command: dotnet build (dns block)
• 7devsblobprodcus323.vsblob.vsassets.io
  • Triggering command: dotnet build (dns block)
• 7tjvsblobprodcus341.vsblob.vsassets.io
  • Triggering command: dotnet build (dns block)
• 8xbvsblobprodcus382.vsblob.vsassets.io
  • Triggering command: dotnet build (dns block)
• ajhvsblobprodcus363.vsblob.vsassets.io
  • Triggering command: dotnet build (dns block)
• bcnvsblobprodcus378.vsblob.vsassets.io
  • Triggering command: dotnet build (dns block)
• fdpvsblobprodcus345.vsblob.vsassets.io
  • Triggering command: dotnet build (dns block)
• frdvsblobprodcus327.vsblob.vsassets.io
  • Triggering command: dotnet build (dns block)
• ibzvsblobprodcus369.vsblob.vsassets.io
  • Triggering command: dotnet build (dns block)
• imzvsblobprodcus368.vsblob.vsassets.io
  • Triggering command: dotnet build (dns block)
• jd4vsblobprodcus366.vsblob.vsassets.io
  • Triggering command: dotnet build (dns block)
• josvsblobprodcus372.vsblob.vsassets.io
  • Triggering command: dotnet build (dns block)
• jrqvsblobprodcus343.vsblob.vsassets.io
  • Triggering command: dotnet build (dns block)
• k0ivsblobprodcus356.vsblob.vsassets.io
  • Triggering command: dotnet build (dns block)
• kgfvsblobprodcus314.vsblob.vsassets.io
  • Triggering command: dotnet build (dns block)
• kh4vsblobprodcus325.vsblob.vsassets.io
  • Triggering command: dotnet build (dns block)
• kijvsblobprodcus387.vsblob.vsassets.io
  • Triggering command: dotnet build (dns block)
• l49vsblobprodcus358.vsblob.vsassets.io
  • Triggering command: dotnet build (dns block)
• m16vsblobprodcus374.vsblob.vsassets.io
  • Triggering command: dotnet build (dns block)
• p2ovsblobprodcus312.vsblob.vsassets.io
  • Triggering command: dotnet build (dns block)
• pdfvsblobprodcus380.vsblob.vsassets.io
  • Triggering command: dotnet build (dns block)
• pe4vsblobprodcus351.vsblob.vsassets.io
  • Triggering command: dotnet build (dns block)
• pkvvsblobprodcus321.vsblob.vsassets.io
  • Triggering command: dotnet build (dns block)
• s4uvsblobprodcus326.vsblob.vsassets.io
  • Triggering command: dotnet build (dns block)
• tphvsblobprodcus375.vsblob.vsassets.io
  • Triggering command: dotnet build (dns block)
• u3hvsblobprodcus371.vsblob.vsassets.io
  • Triggering command: dotnet build (dns block)
• ukkvsblobprodcus352.vsblob.vsassets.io
  • Triggering command: dotnet build (dns block)
• uy6vsblobprodcus34.vsblob.vsassets.io
  • Triggering command: dotnet build (dns block)
• vb4vsblobprodcus33.vsblob.vsassets.io
  • Triggering command: dotnet build (dns block)
• vwvvsblobprodcus334.vsblob.vsassets.io
  • Triggering command: dotnet build (dns block)
• xupvsblobprodcus332.vsblob.vsassets.io
  • Triggering command: dotnet build (dns block)
• yluvsblobprodcus367.vsblob.vsassets.io
  • Triggering command: dotnet build (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Image caching logic doesn't take into account dependencies from intermediate stages</issue_title>
<issue_description>There is a scenario where the image caching logic incorrectly skips building a Dockerfile when that Dockerfile has an intermediate stage based on an image that has a cache miss. An example of this scenario is the this build (internal link). That build was triggered from this PR: dotnet/dotnet-buildtools-prereqs-docker#1207. In that PR, a change was made only to the crossdeps-builder Dockerfile. The build only ended up publishing that crossdeps-builder image and none of the images that are dependent on it.

Here is an example Dockerfile with such a dependency: https://github.com/dotnet/dotnet-buildtools-prereqs-docker/blob/642ddfa47486760a15781758c7e8c0709c4c96c0/src/azurelinux/3.0/net9.0/cross/amd64/Dockerfile#L3. The image caching logic only accounts for the dependency of the final image. In this case, the final image is based on crossdeps-llvm, which has not changed. So the cross Dockerfile never gets built. But it needs to be built because the content it contains is copied from crossdeps-builder, which has changed.</issue_description>

Comments on the Issue (you are @copilot in this section)

@lbussell Instead of adding lots of complexity to our custom caching mechanism, maybe it's time to evaluate using [Docker's built-in caching](https://docs.docker.com/build/cache/backends/registry/). There have been lots of changes since BuildKit became the default. If we store the cache on the registry, Docker *should* cache most or all layers using the normal [cache invalidation rules](https://docs.docker.com/build/cache/invalidation/), which includes base image changes and individual Dockerfile instructions. If we use the "`max`" cache option, then it'll also store intermediate stages/layers on the registry. @mthalman > Instead of adding lots of complexity to our custom caching mechanism, maybe it's time to evaluate using [Docker's built-in caching](https://docs.docker.com/build/cache/backends/registry/).

Maybe. But that defeats the whole purpose of #1449. 🤷‍♂️

@lbussell [Triage] One way to fix this would be to include all `FROM` images in the Dockerfile in the image-info file on https://github.com/dotnet/versions. Then we could re-build when any of those images change. However, this is not a small amount of work for this one scenario that really only affects the buildtools-prereqs repo. Therefore, we should close this as not planned for now, and consider re-opening in the future if it becomes a bigger issue. @lbussell [Triage] This would still be high-cost to fix, still only applies to the buildtools prereqs repo, and still has valid workarounds. A lower-cost solution would be to write a guide/workflow on the buildtools-prereqs repo for handling these scenarios.

To validate these scenarios in PRs, it would also be good to add a no-cache build leg to that repo. Special care should be taken to avoid spending too much compute time with this.</comment_new>
<comment_new>@lbussell
This has been on the sprint column for 5 months. I am moving it back to current release. @mthalman I see this is on two boards at once. Should it be removed from the .NET Docker board?</comment_new>
<comment_new>@mthalman
Yeah, I think it's fine to remove it since it only applies to the buildtools repo.</comment_new>
<comment_new>@lbussell
If I were going to fix this today, I'd probably add a new FromImages property to PlatformData alongside what already exists.

docker-tools/src/ImageBuilder/Models/Image/PlatformData.cs

Lines 35 to 36 in 46e34e7

	[JsonProperty(NullValueHandling = NullValueHandling.Ignore)]
	public string? BaseImageDigest { get; set; }

You would need to add more logic to BuildCommand.cs to fill out the FromImages property in PlatformData. I would probably add it to SetPlatformDataBaseDigest since it already does something similar:

docker-tools/src/ImageBuilder/Commands/BuildCommand.cs

Lines 257 to 285 in 46e34e7

	private void SetPlatformDataBaseDigest(PlatformData platform, Dictionary<string, PlatformData> platformDataByTag)
	{
	string? baseImageDigest = platform.BaseImageDigest;
	if (platform.BaseImageDigest is null && platform.PlatformInfo?.FinalStageFromImage is not null)
	{
	if (!platformDataByTag.TryGetValue(platform.PlatformInfo.FinalStageFromImage, out PlatformData? basePlatformData))
	{
	throw new InvalidOperationException(
	$"Unable to find platform data for tag '{platform.PlatformInfo.FinalStageFromImage}'. " +
	"It's likely that the platforms are not ordered according to dependency.");
	}
	if (basePlatformData.Digest == null)
	{
	throw new InvalidOperationException($"Digest for platform '{basePlatformData.GetIdentifier()}' has not been calculated yet.");
	}
	baseImageDigest = basePlatformData.Digest;
	}
	if (platform.PlatformInfo?.FinalStageFromImage is not null && baseImageDigest is not null)
	{
	baseImageDigest = DockerHelper.GetDigestString(
	DockerHelper.GetRepo(_imageNameResolver.Value.GetFromImagePublicTag(platform.PlatformInfo.FinalStageFromImage)),
	DockerHelper.GetDigestSha(baseImageDigest));
	}
	platform.BaseImageDigest = baseImageDigest;
	}

Then, we would need to update ImageCacheService.cs to invalidate the cache when any of the FROM images were changed, not just the base image digest.

This would also probably get us some rudiment...

Fixes #1455

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

[lbussell]

I chatted with @mthalman to iron out a couple of scenarios. The lowest cost option is to re-build whenever any FROM stage changes. Going with that approach means that we might re-build a couple types of images more frequently than necessary:

• FROM scratch images/distroless images
• Images with lots of stages and/or different targets

We decided we're OK with both of those. We need to include a way to enable/disable re-building based on FROM images so that we can turn this off if it malfunctions/re-builds too frequently.