'operation not permitted' when running 'buildah mount' inside a buildah container

[yrro]

Description

Is it possible to use buildah mount inside a buildah container? I'm getting:

Error: mount /var/lib/containers/storage/overlay:/var/lib/containers/storage/overlay, flags: 0x1000: operation not permitted

For context, I'm within several nested rabbit holes. My goal is to see if I can get a public GitHub Actions runner to build a container image from a ubi-micro image (which does not have a package manager). The problem is that my workflow uses the ubuntu-latest GitHub actions image, which doesn't have DNF available. I'm trying to work around this by nesting containers:

The outer container will run the ubi image, Within this, I will use buildah to create an inner container from ubi-micro, mount the inner container within the outer container, and then I can use the outer container's dnf command with the --installroot= option to install stuff within the inner container.

I mention that in case there's a better way to approach this problem. 😄

BTW, so far I'm just running this on my local (Fedora) machine... I'd like to get it working there first.

Steps to reproduce the issue:

$ buildah from -v /tmp/ct1:/var/lib/containers:Z  docker.io/library/fedora:rawhide
fedora-working-container

$ buildah run fedora-working-container -- dnf -y install buildah
[...]

$ buildah run fedora-working-container -- buildah from scratch
working-container-3

$ buildah run fedora-working-container -- buildah mount working-container-3
Error: mount /var/lib/containers/storage/overlay:/var/lib/containers/storage/overlay, flags: 0x1000: operation not permitted
WARN[0000] failed to shutdown storage: "mount /var/lib/containers/storage/overlay:/var/lib/containers/storage/overlay, flags: 0x1000: operation not permitted" 
Error: while running runtime: exit status 125

Additional troubleshooting

No AVC messages are written to the audit log so this doesn't appear to be because of SELinux.

I've added --security-opt=label=disable --security-opt=seccomp=unconfined to the outer and inner buildah from commands and they don't help.

buildah mount works when run from a privileged podman container:

$ podman run --privileged -v /tmp/ct1:/var/lib/containers:Z --rm -it docker.io/library/fedora:rawhide

[root@0731245902d9 /]# dnf -y install buildah
[...]

[root@fe9f2496b491 /]# buildah from scratch
working-container-5

[root@fe9f2496b491 /]# buildah mount working-container-5
/var/lib/containers/storage/overlay/6b7b5cf263448e70917fb34107b0b8f7af2fa0156e052b27e638ed90c538e868/merged

... but there's no --privileged flag for buildah or podman build, unless I'm missing something?

Output of rpm -q buildah or apt list buildah:

The above was ran on a host with:

buildah-1.35.3-1.fc39.x86_64
podman-4.9.4-1.fc39.x86_64

Inside the rawhide container, this buildah was used:

buildah-1.35.3-1.fc41.x86_64

Output of buildah version:

On the host:

Version:         1.35.3
Go Version:      go1.21.8
Image Spec:      1.1.0
Runtime Spec:    1.1.0
CNI Spec:        1.0.0
libcni Version:  
image Version:   5.30.0
Git Commit:      
Built:           Mon Apr  1 12:11:26 2024
OS/Arch:         linux/amd64
BuildPlatform:   linux/amd64

Inside the rawhide container:

Version:         1.35.3
Go Version:      go1.22.1
Image Spec:      1.1.0
Runtime Spec:    1.1.0
CNI Spec:        1.0.0
libcni Version:  
image Version:   5.30.0
Git Commit:      
Built:           Mon Apr  1 11:06:33 2024
OS/Arch:         linux/amd64
BuildPlatform:   linux/amd64

Output of cat /etc/*release:

==> /etc/fedora-release <==
Fedora release 39 (Thirty Nine)

==> /etc/os-release <==
NAME="Fedora Linux"
VERSION="39 (Workstation Edition)"
ID=fedora
VERSION_ID=39
VERSION_CODENAME=""
PLATFORM_ID="platform:f39"
PRETTY_NAME="Fedora Linux 39 (Workstation Edition)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:39"
DEFAULT_HOSTNAME="fedora"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f39/system-administrators-guide/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=39
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=39
SUPPORT_END=2024-11-12
VARIANT="Workstation Edition"
VARIANT_ID=workstation

==> /etc/redhat-release <==
Fedora release 39 (Thirty Nine)

==> /etc/system-release <==
Fedora release 39 (Thirty Nine)

Output of uname -a:

Linux syzyzy 6.8.5-201.fc39.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Apr 11 18:25:26 UTC 2024 x86_64 GNU/Linux

Output of cat /etc/containers/storage.conf:

File does not exist - no customizations applied

[yrro]

My goal is to see if I can get a public GitHub Actions runner to build a container image from a ubi-micro image (which does not have a package manager).

Turns out sudo install dnf works just fine. So I no longer need to use buildah mount inside a buildah container; however I'll leave this issue open because I think it should work.

[yrro]

Due to differences between the configuration of RPM in the host system and within the ubi-micro container image, it actually is desirable to be able to run DNF within an outer 'builder' container, installing into a nested 'runtime' container after all.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.