Cache manifests should be fetched in the background

[oehme]

Description

Adding --cache-from currently creates a significant slowdown for "up-to-date" builds, i.e. builds where I already have all the layers in my local cache. Docker waits until the cache manifest is downloaded/imported before it proceeds building.

I think instead it should fetch the manifest in the background and only block on the download if it gets to a build step that can't be served from my local cache.

[tonistiigi]

if it gets to a build step that can't be served from my local cache.

This doesn't work because there are always build steps that run without matching cache. Eg. context/Dockerfile transfer.

[oehme]

If a step doesn't use the cache, then it also doesn't need the remote cache. So my point still stands.