BuildKit produces image with duplicated empty layer causing overlayfs failure on containerd (GKE) #6543
Description
Contributing guidelines and issue reporting guide
- I've read the contributing guidelines and wholeheartedly agree. I've also read the issue reporting guide.
Well-formed report checklist
- I have found a bug that the documentation does not mention anything about my problem
- I have found a bug that there are no open or closed issues that are related to my problem
- I have provided version/information about my environment and done my best to provide a reproducer
Description of bug
Bug description
When building a very simple Dockerfile using BuildKit, the resulting image contains the same empty layer twice in the RootFS layer list.
This causes pods to fail at startup on Google Kubernetes Engine (GKE) nodes using containerd, with errors such as:
overlayfs: conflicting lowerdir path
and
Error: failed to create containerd container: failed to mount /var/lib/containerd/tmpmounts/containerd-mount1406420590: too many levels of symbolic links
The same image works locally with Docker
Not sure if this problem here is more on buildkit or on containerd or even if this is the root cause of the crash, but seeing 2 empty layers is definitely weird
Reproduction
FROM hashicorp/tfc-agent:1.28@sha256:47baf87291a4c58720e7702b0af77dc3dc7f7a4b51568f4f46dda21d2b1144ff
RUN mkdir -p /home/tfc-agent/.tfc-agent
(note that folder /home/tfc-agent/.tfc-agent already exists in base image)
docker buildx build -t broken:latest .
Observed Result
Running:
docker inspect broken:latest
Shows that the same layer digest appears twice in RootFS.Layers:
"Layers": [
...
"sha256:eab602d70e4210d68665818e5af4300da5806915c278481b50c51d54c44eba27",
"sha256:e27b0941b25a53726f96426d92a8ca41a496c749cc0a349d9e22a11cdf260c5c",
"sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef",
"sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef"
]
I also have to say that i was able to reproduce this only with hashicorp/tfc-agent:1.28, taking other images like alpine or python and creating a folder that already exists, did not create a duplicate empty layer
Version information
buildkit version: 0.27.0
containerd version: 2.0.6
gke version: 1.33.5-gke.2326000