buildah image v1.34: Error: open /usr/lib/containers/storage/overlay-images/images.lock: permission denied

[zmjackson]

Description
When run with Docker (haven't tested Podman) and our security configs, buildah image v1.34 throws an error on most operations. No previous versions have this issue. Although I submitted the report for an Ubuntu system, I observed the exact same behavior on Centos. We use the Podman seccomp.json and the following AppArmor policy:

#include <tunables/global>

profile docker-buildah flags=(attach_disconnected,mediate_deleted) {
  #include <abstractions/base>

  network,
  capability,
  file,
  umount,
  # The buildah container requires mount
  mount,
  # Host (privileged) processes may send signals to container processes.
  signal (receive) peer=unconfined,
  # dockerd may send signals to container processes (for "docker kill").
  signal (receive) peer=/usr/bin/docker,
  # Container processes may send signals amongst themselves.
  signal (send,receive) peer=docker-buildah,

  deny @{PROC}/* w,   # deny write for all files directly in /proc (not in a subdir)
  # deny write to files not in /proc/<number>/** or /proc/sys/**
  deny @{PROC}/{[^1-9],[^1-9][^0-9],[^1-9s][^0-9y][^0-9s],[^1-9][^0-9][^0-9][^0-9/]*}/** w,
  deny @{PROC}/sys/[^k]** w,  # deny /proc/sys except /proc/sys/k* (effectively /proc/sys/kernel)
  deny @{PROC}/sys/kernel/{?,??,[^s][^h][^m]**} w,  # deny everything except shm* in /proc/sys/kernel/
  deny @{PROC}/sysrq-trigger rwklx,
  deny @{PROC}/kcore rwklx,

  deny /sys/[^f]*/** wklx,
  deny /sys/f[^s]*/** wklx,
  deny /sys/fs/[^c]*/** wklx,
  deny /sys/fs/c[^g]*/** wklx,
  deny /sys/fs/cg[^r]*/** wklx,
  deny /sys/firmware/** rwklx,
  deny /sys/devices/virtual/powercap/** rwklx,
  deny /sys/kernel/security/** rwklx,

  # suppress ptrace denials when using 'docker ps' or using 'ps' inside a container
  ptrace (trace,read,tracedby,readby) peer=docker-buildah,
}

Steps to reproduce the issue:

1. docker run -t --device /dev/fuse --security-opt seccomp=/usr/share/containers/seccomp.json --security-opt apparmor=docker-buildah -u build quay.io/buildah/stable:v1.34 buildah pull fedora

Describe the results you received:
buildah image v1.34 throws the following error when pulling, building, etc.:

docker run -t --device /dev/fuse  --security-opt seccomp=/usr/share/containers/seccomp.json --security-opt apparmor=docker-buildah -u build quay.io/buildah/stable:v1.34 buildah pull fedora
Error: open /usr/lib/containers/storage/overlay-images/images.lock: permission denied
WARN[0000] failed to shutdown storage: "open /usr/lib/containers/storage/overlay-images/images.lock: permission denied"

Describe the results you expected:
Buildah operations complete successfully when I use v1.33.2 or earlier:

docker run -t --device /dev/fuse  --security-opt seccomp=/usr/share/containers/seccomp.json --security-opt apparmor=docker-buildah -u build quay.io/buildah/stable:v1.33.2 buildah pull fedora
Resolved "fedora" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull registry.fedoraproject.org/fedora:latest...
Getting image source signatures
Copying blob 718a00fe3212 done   |
Copying config 368a084ba1 done   |
Writing manifest to image destination

Output of rpm -q buildah or apt list buildah:

N/A

Output of buildah version:

N/A

Output of podman version if reporting a podman build issue:

N/A

Output of cat /etc/*release:

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.3 LTS"
PRETTY_NAME="Ubuntu 22.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.3 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

Output of uname -a:

Linux ****.**************.com 5.15.0-76-generic #83-Ubuntu SMP Thu Jun 15 19:16:32 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Output of cat /etc/containers/storage.conf:

N/A

[yossicohn]

We have the same issue, seems like a regression after upgrading to v1.34.0. (upgraded from v1.32.2 where everything runs well)
We run on k8s and we get
Error: open /usr/lib/containers/storage/overlay-images/images.lock: permission denied

Any Advice?
Do we need to set an `EmptyDir' over this path?

[flouthoc]

@zmjackson Does it work if you add --privileged --security-opt seccomp=unconfined --security-opt apparmor=unconfined ? Just to make sure that issue is not with the new profiles ? if it works we can work to figure out what is causing the issue.

[zmjackson]

I just checked and I get the same error when using --privileged --security-opt seccomp=unconfined --security-opt apparmor=unconfined.

[yossicohn]

@zmjackson just changed the security context from

container:
    securityContext:
      allowPrivilegeEscalation: false
      runAsUser: 1000
      runAsGroup: 1000
      capabilities:
        drop:
          - ALL

to

securityContext:
   priviledged: true

and it worked...but as you know we don't want to run buildah with these kind of priviledges

[yossicohn]

@zmjackson what do you think about that ^^ any ideas how we can proceed?

[TomSweeneyRedHat]

@flouthoc or @rhatdan thoughts on what might be tweaked to fix this?

[pionicus]

Seems to be caused by the configuration of the container storage library.

Buildah 1.27 do not have a storage.conf in $HOME/.config/containers/storage but Buildah 1.34 does.

I did a test without the storage.conf in $HOME and it almost works excepted some warnings not present with v1.27.

I ran minikube with podman rootless for the tests. The storage is configured with the kernel overlay and not the fuse one.

[build@buildah-latestfixed containers]$ buildah info
Error: open /usr/lib/containers/storage/overlay-images/images.lock: permission denied
WARN[0000] failed to shutdown storage: "open /usr/lib/containers/storage/overlay-images/images.lock: permission denied" 

[build@buildah-latestfixed containers]$ pwd
/home/build/.config/containers

[build@buildah-latestfixed containers]$ mv storage.conf storage.conf.ori

[build@buildah-latestfixed containers]$ buildah info
{
    "host": {
        "CgroupVersion": "v2",
        "Distribution": {
            "distribution": "fedora",
            "version": "39"
        },
        "MemFree": 196026368,
        "MemTotal": 3080105984,
        "OCIRuntime": "crun",
        "SwapFree": 0,
        "SwapTotal": 0,
        "arch": "amd64",
        "cpus": 2,
        "hostname": "buildah-latestfixed",
        "kernel": "6.1.0-17-amd64",
        "os": "linux",
        "rootless": true,
        "uptime": "5h 58m 54.82s (Approximately 0.21 days)",
        "variant": ""
    },
    "store": {
        "ContainerStore": {
            "number": 0
        },
        "GraphDriverName": "overlay",
        "GraphOptions": null,
        "GraphRoot": "/home/build/.local/share/containers/storage",
        "GraphStatus": {
            "Backing Filesystem": "extfs",
            "Native Overlay Diff": "true",
            "Supports d_type": "true",
            "Supports shifting": "false",
            "Supports volatile": "true",
            "Using metacopy": "false"
        },
        "ImageStore": {
            "number": 0
        },
        "RunRoot": "/var/tmp/storage-run-1000/containers"
    }
}

sylvain@debian12:~$ minikube kubectl -- exec -it buildah-latestfixed -- bash -c 'buildah --version; buildah info; buildah run $(buildah from ubuntu) bash -c "ls -la"'
buildah version 1.34.0 (image-spec 1.1.0-rc.5, runtime-spec 1.1.0)
{
    "host": {
        "CgroupVersion": "v2",
        "Distribution": {
            "distribution": "fedora",
            "version": "39"
        },
        "MemFree": 77193216,
        "MemTotal": 3080105984,
        "OCIRuntime": "crun",
        "SwapFree": 0,
        "SwapTotal": 0,
        "arch": "amd64",
        "cpus": 2,
        "hostname": "buildah-latestfixed",
        "kernel": "6.1.0-17-amd64",
        "os": "linux",
        "rootless": true,
        "uptime": "5h 26m 15.22s (Approximately 0.21 days)",
        "variant": ""
    },
    "store": {
        "ContainerStore": {
            "number": 0
        },
        "GraphDriverName": "overlay",
        "GraphOptions": null,
        "GraphRoot": "/home/build/.local/share/containers/storage",
        "GraphStatus": {
            "Backing Filesystem": "extfs",
            "Native Overlay Diff": "true",
            "Supports d_type": "true",
            "Supports shifting": "false",
            "Supports volatile": "true",
            "Using metacopy": "false"
        },
        "ImageStore": {
            "number": 0
        },
        "RunRoot": "/var/tmp/storage-run-1000/containers"
    }
}
Resolved "ubuntu" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull docker.io/library/ubuntu:latest...
Getting image source signatures
Copying blob bccd10f490ab done   | 
Copying config ca2b0f2696 done   | 
Writing manifest to image destination
WARN[0000] Path "/run/secrets/etc-pki-entitlement" from "/etc/containers/mounts.conf" doesn't exist, skipping 
WARN[0000] Path "/run/secrets/rhsm" from "/etc/containers/mounts.conf" doesn't exist, skipping 
total 56
dr-xr-xr-x   1 root   root    4096 Mar  7 16:54 .
dr-xr-xr-x   1 root   root    4096 Mar  7 16:54 ..
lrwxrwxrwx   1 root   root       7 Feb 27 15:59 bin -> usr/bin
drwxr-xr-x   2 root   root    4096 Apr 18  2022 boot
drwxr-xr-x   5 nobody nogroup  360 Mar  7 16:52 dev
drwxr-xr-x  32 root   root    4096 Feb 27 16:03 etc
drwxr-xr-x   2 root   root    4096 Apr 18  2022 home
lrwxrwxrwx   1 root   root       7 Feb 27 15:59 lib -> usr/lib
lrwxrwxrwx   1 root   root       9 Feb 27 15:59 lib32 -> usr/lib32
lrwxrwxrwx   1 root   root       9 Feb 27 15:59 lib64 -> usr/lib64
lrwxrwxrwx   1 root   root      10 Feb 27 15:59 libx32 -> usr/libx32
drwxr-xr-x   2 root   root    4096 Feb 27 15:59 media
drwxr-xr-x   2 root   root    4096 Feb 27 15:59 mnt
drwxr-xr-x   2 root   root    4096 Feb 27 15:59 opt
dr-xr-xr-x 328 nobody nogroup    0 Mar  7 16:52 proc
drwx------   2 root   root    4096 Feb 27 16:02 root
drwxr-xr-x   1 root   root    4096 Mar  7 16:54 run
lrwxrwxrwx   1 root   root       8 Feb 27 15:59 sbin -> usr/sbin
drwxr-xr-x   2 root   root    4096 Feb 27 15:59 srv
dr-xr-xr-x  13 nobody nogroup    0 Mar  7 12:02 sys
drwxrwxrwt   2 root   root    4096 Feb 27 16:02 tmp
drwxr-xr-x  14 root   root    4096 Feb 27 15:59 usr
drwxr-xr-x  11 root   root    4096 Feb 27 16:02 var

sylvain@debian12:~$ cat buildah-latest2.yaml 
apiVersion: v1
kind: Pod
metadata:
 name: buildah-latestfixed
spec:
 containers:
   - name: buildah
     image: quay.io/buildah/stable:latest
     args:
       - sleep
       - "1000000"
     securityContext:
       runAsUser: 1000

[yossicohn]

@sylvainpelletier thanks,
is there a reason for the storage.conf to be added, is there a way to have the previous behaviour like at 1.27 ?

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[yossicohn]

@zmjackson FYI,
After adding the Buildha POD and EmpytDir mounted over /usr/lib/containers/storage/overlay-images it fixes the problem.
Though I still see error (which are then fall back) for set UID and GID, and I could not resolved with given capabilities
but it least it can work

container:
        securityContext:
          allowPrivilegeEscalation: false
          runAsUser: 1000
          runAsGroup: 1000
          capabilities:
            drop:
              - ALL
        image: "quay.io/containers/buildah:v1.35.3"
        imagePullPolicy: IfNotPresent       
        volumeMounts:
           - name: buildah-storage
              mountPath: /usr/lib/containers/storage/overlay-images
volumes:
   - name: buildah-storage
      emptyDir: {}