Skip to content

Windows: bindfilter.ApplyFileBinding fails with 'Access is denied' inside HostProcess containers #6590

@doctorpangloss

Description

@doctorpangloss

Bug description

bindfilter.ApplyFileBinding (from github.com/Microsoft/go-winio/pkg/bindfilter) returns "Access is denied" for all calls made from a Kubernetes HostProcess container, even between two ordinary host directories. This prevents buildkitd from mounting containerd snapshots when running as a HostProcess container on Windows.

Reproduction

Deploy buildkitd as a HostProcess DaemonSet:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: buildkitd-windows
  namespace: buildkit
spec:
  selector:
    matchLabels:
      app: buildkitd-windows
  template:
    metadata:
      labels:
        app: buildkitd-windows
    spec:
      nodeSelector:
        kubernetes.io/os: windows
      securityContext:
        windowsOptions:
          hostProcess: true
          runAsUserName: "NT AUTHORITY\\SYSTEM"
      hostNetwork: true
      containers:
        - name: buildkitd
          image: <registry>/buildkitd:windows-0.28.0-servercore-ltsc2022
          command:
            - \$env:CONTAINER_SANDBOX_MOUNT_POINT/buildkit/buildkitd.exe
          args:
            - --addr
            - tcp://0.0.0.0:1234
            - --containerd-worker-namespace=k8s.io
            - --root
            - C:/buildkitd

buildkitd starts and finds the containerd worker:

level=info msg="found worker \"...\", labels=map[...worker.containerd.namespace:k8s.io ...worker.executor:containerd ...worker.snapshotter:windows], platforms=[windows/amd64]"
level=info msg="running server on [::]:1234"

Any build fails at the first step:

#1 [internal] load build definition from Dockerfile
#1 ERROR: failed to mount {windows-layer ...\snapshots\120494  [rw]}:
  failed to set volume mount path for layer ...\snapshots\120494:
  failed to bind target "...\\snapshots\\120494\\Files"
  to root "C:\\buildkitd\\tmp\\buildkit-mount1819228504": Access is denied.

Root cause

To isolate the failure, we compiled a standalone test using go-winio/pkg/bindfilter and ran it inside the HostProcess container:

package main

import (
	"fmt"
	"os"
	"github.com/Microsoft/go-winio/pkg/bindfilter"
)

func main() {
	os.MkdirAll(os.Args[2], 0700)
	err := bindfilter.ApplyFileBinding(os.Args[2], os.Args[1], true)
	if err != nil {
		fmt.Printf("FAILED: %v\n", err)
	} else {
		fmt.Println("OK")
		bindfilter.RemoveFileBinding(os.Args[2])
	}
}

Test 1 — bind between two regular host directories (no containerd involvement):

```
C:> test-bind.exe C:\buildkitd\src-test C:\buildkitd\dst-test
FAILED: failed to bind target "C:\buildkitd\src-test" to root "C:\buildkitd\dst-test": Access is denied.
```

Test 2 — the HCS layer APIs succeed from the same process:

```
C:> test-mount.exe C:\ProgramData\containerd...\snapshots\117263 C:\buildkitd\tmp\test
ActivateLayer... OK
PrepareLayer... OK
GetLayerMountPath... OK: C:\ProgramData\containerd...\snapshots\117263
ApplyFileBinding failed: Access is denied.
```

The process has SeBackupPrivilege and SeRestorePrivilege enabled, all directories have SYSTEM:Full Control ACLs, and bindflt.sys is running. Since the bind filter fails even between two fresh directories created by the same process, the issue is not related to containerd snapshots, ACLs, or privileges.

containerd 1.7+ elevates HostProcess containers to partial silos (hcsshim PR #1344). The bind filter driver does not support attaching filesystem filters inside silos (microsoft/Windows-Containers#268, microsoft/hcsshim#1699).

Workaround

The HostProcess container installs buildkitd as a native Windows service via --register-service. The service process runs outside the silo context, so the bind filter works.

Version information

```
buildkitd v0.28.0
containerd 1.7.29
Windows Server 2022 build 20348.4773
Kubernetes v1.34.2+k0s
```

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions