/proc and /sys are owned by nobody inside rootless containers

[clarfonthey]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

When running podman in rootless mode, /proc and /sys are recognized as owned by nobody, whereas /dev is correctly recognized as owned by root.

I discovered this when running pacman -Qkk inside an Arch Linux container. It's not a super big issue (who ever checks the owners of these directories except when… checking the integrity of installed packages), but it seems like a bug, so, I figured I'd report it.

Steps to reproduce the issue:

1. Pick your favourite image and podman run -it
2. ls -lad /proc /sys

Describe the results you received:
You see that /proc and /sys are owned by nobody:nobody according to ls.

Describe the results you expected:
These directories should be mapped to a user that is recognized as root.

Additional information you deem important (e.g. issue happens only occasionally):
N/A

Output of podman version:

Client:       Podman Engine
Version:      4.3.1
API Version:  4.3.1
Go Version:   go1.19.3
Git Commit:   814b7b003cc630bf6ab188274706c383f9fb9915-dirty
Built:        Thu Nov 10 09:59:17 2022
OS/Arch:      linux/amd64

Output of podman info:

host:
  arch: amd64
  buildahVersion: 1.28.0
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: /usr/bin/conmon is owned by conmon 1:2.1.5-1
    path: /usr/bin/conmon
    version: 'conmon version 2.1.5, commit: c9f7f19eb82d5b8151fc3ba7fbbccf03fdcd0325'
  cpuUtilization:
    idlePercent: 85.45
    systemPercent: 4.49
    userPercent: 10.06
  cpus: 12
  distribution:
    distribution: arch
    version: unknown
  eventLogger: journald
  hostname: waverider
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 984
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.0.8-hardened1-1-hardened
  linkmode: dynamic
  logDriver: journald
  memFree: 4220608512
  memTotal: 33612390400
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: /usr/bin/crun is owned by crun 1.7-1
    path: /usr/bin/crun
    version: |-
      crun version 1.7
      commit: 40d996ea8a827981895ce22886a9bac367f87264
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: /usr/bin/slirp4netns is owned by slirp4netns 1.2.0-1
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 0
  swapTotal: 0
  uptime: 22h 34m 50.00s (Approximately 0.92 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries: {}
store:
  configFile: /home/ltdk/.config/containers/storage.conf
  containerStore:
    number: 41
    paused: 0
    running: 1
    stopped: 40
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/ltdk/.local/share/containers/storage
  graphRootAllocated: 241794809856
  graphRootUsed: 43639242752
  graphStatus:
    Backing Filesystem: zfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 194
  runRoot: /run/user/1000/containers
  volumePath: /home/ltdk/.local/share/containers/storage/volumes
version:
  APIVersion: 4.3.1
  Built: 1668092357
  BuiltTime: Thu Nov 10 09:59:17 2022
  GitCommit: 814b7b003cc630bf6ab188274706c383f9fb9915-dirty
  GoVersion: go1.19.3
  Os: linux
  OsArch: linux/amd64
  Version: 4.3.1

Package info (e.g. output of rpm -q podman or apt list podman or brew info podman):

Name            : podman
Version         : 4.3.1-1
Description     : Tool and library for running OCI-based containers in pods
Architecture    : x86_64
URL             : https://github.com/containers/podman
Licenses        : Apache
Groups          : None
Provides        : None
Depends On      : catatonit  conmon  containers-common  crun  iptables  libdevmapper.so=1.02-64  libgpgme.so=11-64  libseccomp.so=2-64
                  slirp4netns
Optional Deps   : apparmor: for AppArmor support [installed]
                  btrfs-progs: support btrfs backend devices [installed]
                  netavark: for a new container-network-stack implementation [installed]
                  podman-compose: for docker-compose compatibility [installed]
                  podman-docker: for Docker-compatible CLI [installed]
Required By     : podman-compose  podman-docker
Optional For    : cross  woodpecker-agent
Conflicts With  : None
Replaces        : None
Installed Size  : 66.95 MiB
Packager        : David Runge <[email protected]>
Build Date      : Thu 10 Nov 2022 09:59:17 EST
Install Date    : Sat 12 Nov 2022 11:05:55 EST
Install Reason  : Explicitly installed
Install Script  : No
Validated By    : Signature

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)
Yes

Additional environment details (AWS, VirtualBox, physical, etc.):
Just my desktop running Arch.

[rhatdan]

This is controlled by the linux kernel. Within the user namespace we are allowed to mount the following file systems
tmpfs, procfs, sysfs, bind, overlay.
The kernel does not allow chowning of procfs and sysfs, meaning that these directories retain the host UID==0. Since we don't and can't map UID==0 into the container, the kernel shows their ownership as the nobody user.

[mailinglists35]

is there a way to include this footnote in EL docs? I thought I am doing something wrong, had to download several os images to triple check, had to google "podman proc sys nobody" until I found this explanation.

[Agalin]

It seems to affect nested podman.

$ podman run --rm -ti --userns=auto:size=65536 --user podman  --device /dev/fuse --security-opt label=disable quay.io/podman/stable
[podman@6e9e23b1595f /]$ podman run --privileged alpine
Resolved "alpine" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull docker.io/library/alpine:latest...
Getting image source signatures
Copying blob ca7dd9ec2225 done  
Copying config bfe296a525 done  
Writing manifest to image destination
Storing signatures
Error: crun: mount `/proc` to `/proc`: Operation not permitted: OCI permission denied

[rhatdan]

Nested Podman is affected for other reasons, but yes you need to volume mount in /proc into your container to make this work.
The problem is the kernel will not allow a process to mount /proc over a file system with files and directories masked out.
@giuseppe Would know more.

[Agalin]

Thanks for explanation. Maybe it can be added to the rootless tutorial?

[giuseppe]

in order to mount /proc there is need a procfs is already fully visible. For that to happen, the outer podman needs to run with --security-opt unmask=/proc/*

[paulcalabro]

Is the EACCES (Permission denied) expected? I'm not using nested Podman.

[pid 480980] openat(AT_FDCWD, "/sys/fs/cgroup/cgroup.controllers", O_RDONLY) = 9
[pid 480980] statx(9, "", AT_STATX_DONT_SYNC|AT_SYMLINK_NOFOLLOW|AT_EMPTY_PATH, STATX_SIZE, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_blksize=4096, stx_attributes=0, stx_nlink=1, stx_uid=65534, stx_gid=65534, stx_mode=S_IFREG|0444, stx_ino=4, stx_size=0, stx_blocks=0, stx_attributes_mask=STATX_ATTR_AUTOMOUNT|STATX_ATTR_MOUNT_ROOT|STATX_ATTR_DAX, stx_atime={tv_sec=1713349956, tv_nsec=180999970} /* 2024-04-17T06:32:36.180999970-0400 */, stx_btime={tv_sec=0, tv_nsec=0}, stx_ctime={tv_sec=1713349956, tv_nsec=180999970} /* 2024-04-17T06:32:36.180999970-0400 */, stx_mtime={tv_sec=1713349956, tv_nsec=180999970} /* 2024-04-17T06:32:36.180999970-0400 */, stx_rdev_major=0, stx_rdev_minor=0, stx_dev_major=0, stx_dev_minor=25}) = 0
[pid 480980] read(9, "cpuset cpu io memory hugetlb pids rdma misc\n", 1023) = 44
[pid 480980] read(9, "", 979)           = 0
[pid 480980] close(9)                   = 0
[pid 480980] openat(AT_FDCWD, "/sys/fs/cgroup/cgroup.subtree_control", O_WRONLY|O_CREAT|O_TRUNC, 0700) = -1 EACCES (Permission denied)

[giuseppe]

it is owned by nobody because root on the host is not mapped inside the user namespace.

[paulcalabro]

Thanks @giuseppe, that makes sense.

The reason why I'm digging into this is we're not able to run rootless containers in one of our environments after a recent system upgrade. When we do, we encounter this error message:

podman run --rm -it alpine:3.19.1 sh
Error: crun: mount `cgroup2` to `sys/fs/cgroup`: Permission denied: OCI permission denied

Both /etc/subuid and /etc/subgid are configured for the user account.

I see the access denied message above for /sys/fs/cgroup/cgroup.subtree_control as well as this:

536444<3> 11:20:59 openat2(14</tmp/debug/overlay/73e12779318b4a7e477504f2ef2713e1067f0e1c34183863ed2b7e6ca051e1c2/merged>, "sys/fs/cgroup", {flags=O_RDONLY|O_CLOEXEC|O_PATH, resolve=RESOLVE_IN_ROOT}, 24) = 6</tmp/debug/overlay/73e12779318b4a7e477504f2ef2713e1067f0e1c34183863ed2b7e6ca051e1c2/merged/sys/fs/cgroup>

536444<3> 11:20:59 mount("cgroup2", "/proc/self/fd/6", "cgroup2", MS_NOSUID|MS_NODEV|MS_NOEXEC|MS_REC|MS_RELATIME, NULL) = -1 EACCES (Permission denied)

I've looked under a lot of rocks and I'm a little stuck. Have you seen this before? Any ideas on how to fix this?

[giuseppe]

is there anything particular about your environment?

Have you performed any customization in containers.conf?

[paulcalabro]

We're able to run rootless containers using local accounts, however, we're unable to run rootless containers using AD accounts (manually adding entries for subuids/subgids).

WRT local customizations to containers.conf, the only changes that differ from /usr/share/containers/containers.conf are:

log_driver = "journald"
userns = "auto"
default_subnet = "172.W.X.Y/Z"

[paulcalabro]

Also, here's some additional info.

podman unshare cat /proc/self/uid_map
         0     <service account UID>          1
         1     <first subordinate UID for service account>       65536