docker exec doesnt work on centos stream 9 #43960

zhouwencn · 2022-08-13T10:57:14Z

zhouwencn
Aug 13, 2022

i use m1 mac + pd build a centos stream9 vm,then i remove podman,install dokcer ,i run docker pull nginx, docker run --name nginx -dp 80:80 nginx,then i run docker exec -it nginx bash, but error happend,the content is
"OCI runtime exec failed: exec failed: unable to start container process: open /dev/pts/0: operation not permitted: unknown", i want to know that anyone meet same issues?

Answered by AkihiroSuda

Sep 5, 2022

Fixed in https://github.com/opencontainers/runc/releases/tag/v1.1.4

View full answer

thaJeztah · 2022-08-16T10:14:12Z

thaJeztah
Aug 16, 2022
Maintainer

Looks to be SELinux blocking runc 🤔;

docker run -it --rm busybox
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: open /dev/pts/0: operation not permitted: unknown.

ls -la /dev/pts/0
crw--w----. 1 root tty 136, 0 Aug 16 06:06 /dev/pts/0

ausearch -m avc --start recent

...

time->Tue Aug 16 06:11:12 2022
type=PROCTITLE msg=audit(1660644672.677:490): proctitle=72756E63002D2D726F6F74002F7661722F72756E2F646F636B65722F72756E74696D652D72756E632F6D6F6279002D2D6C6F67002F72756E2F636F6E7461696E6572642F696F2E636F6E7461696E6572642E72756E74696D652E76322E7461736B2F6D6F62792F32643961343339383566313539356238363132333135386637
type=SYSCALL msg=audit(1660644672.677:490): arch=c000003e syscall=321 success=no exit=-13 a0=d a1=c00013e040 a2=8 a3=7f7a553af3b0 items=0 ppid=13615 pid=13625 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="runc" exe="/usr/bin/runc" subj=system_u:system_r:unconfined_service_t:s0 key=(null)
type=AVC msg=audit(1660644672.677:490): avc:  denied  { prog_run } for  pid=13625 comm="runc" scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:system_r:init_t:s0 tclass=bpf permissive=0

Temporarily disabling selinux, it looks to be working;

setenforce 0
docker run -it --rm busybox
/ # exit

ausearch -m avc --start recent

...

time->Tue Aug 16 06:12:50 2022
type=PROCTITLE msg=audit(1660644770.358:506): proctitle=72756E63002D2D726F6F74002F7661722F72756E2F646F636B65722F72756E74696D652D72756E632F6D6F6279002D2D6C6F67002F72756E2F636F6E7461696E6572642F696F2E636F6E7461696E6572642E72756E74696D652E76322E7461736B2F6D6F62792F35613362303434333030316537666339316239303534353733
type=SYSCALL msg=audit(1660644770.358:506): arch=c000003e syscall=321 success=yes exit=17 a0=d a1=c00013e040 a2=8 a3=7f162404d3b0 items=0 ppid=13705 pid=13716 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="runc" exe="/usr/bin/runc" subj=system_u:system_r:unconfined_service_t:s0 key=(null)
type=AVC msg=audit(1660644770.358:506): avc:  denied  { prog_run } for  pid=13716 comm="runc" scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:system_r:init_t:s0 tclass=bpf permissive=1

docker version and info

Client: Docker Engine - Community
 Version:           20.10.17
 API version:       1.41
 Go version:        go1.17.11
 Git commit:        100c701
 Built:             Mon Jun  6 23:03:29 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.17
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.17.11
  Git commit:       a89b842
  Built:            Mon Jun  6 23:01:12 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.7
  GitCommit:        0197261a30bf81f1ee8e6a4dd2dea0ef95d67ccb
 runc:
  Version:          1.1.3
  GitCommit:        v1.1.3-0-g6724737
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0


Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.8.2-docker)
  compose: Docker Compose (Docker Inc., v2.6.0)
  scan: Docker Scan (Docker Inc., v0.17.0)

Server:
 Containers: 2
  Running: 0
  Paused: 0
  Stopped: 2
 Images: 3
 Server Version: 20.10.17
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 0197261a30bf81f1ee8e6a4dd2dea0ef95d67ccb
 runc version: v1.1.3-0-g6724737
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.14.0-70.el9.x86_64
 Operating System: CentOS Stream 9
 OSType: linux
 Architecture: x86_64
 CPUs: 1
 Total Memory: 960.5MiB
 Name: centos-s-1vcpu-1gb-ams3-01
 ID: R2MF:2W6M:5T6Z:ONWV:E2DR:Y52D:T3JM:VBLB:BIDP:MYB6:TG7J:PEXT
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

5 replies

thaJeztah Aug 16, 2022
Maintainer

Hmm, so it appears downgrading the containerd.io package from 1.6.7 to 1.6.6 (and runc to 1.1.2) fixes it as well;

yum list containerd.io --showduplicates | sort -r
Last metadata expiration check: 0:00:56 ago on Tue 16 Aug 2022 06:19:53 AM EDT.
Installed Packages
containerd.io.x86_64               1.6.7-3.1.el9               docker-ce-stable
containerd.io.x86_64               1.6.7-3.1.el9               @docker-ce-stable
containerd.io.x86_64               1.6.6-3.1.el9               docker-ce-stable
containerd.io.x86_64               1.6.4-3.1.el9               docker-ce-stable


yum install containerd.io-1.6.6-3.1.el9

Last metadata expiration check: 0:01:34 ago on Tue 16 Aug 2022 06:19:53 AM EDT.
Dependencies resolved.
============================================================================================================================================================================================================
 Package                                           Architecture                               Version                                            Repository                                            Size
============================================================================================================================================================================================================
Downgrading:
 containerd.io                                     x86_64                                     1.6.6-3.1.el9                                      docker-ce-stable                                      32 M

Transaction Summary
============================================================================================================================================================================================================
Downgrade  1 Package

Total download size: 32 M
Is this ok [y/N]: y

systemctl restart containerd docker

setenforce 1
docker run -it --rm busybox
/ # exit

ausearch -m avc --start recent

...

time->Tue Aug 16 06:22:13 2022
type=PROCTITLE msg=audit(1660645333.583:683): proctitle=72756E63002D2D726F6F74002F7661722F72756E2F646F636B65722F72756E74696D652D72756E632F6D6F6279002D2D6C6F67002F72756E2F636F6E7461696E6572642F696F2E636F6E7461696E6572642E72756E74696D652E76322E7461736B2F6D6F62792F38366566386262323163313233343234376266393632316538
type=SYSCALL msg=audit(1660645333.583:683): arch=c000003e syscall=321 success=no exit=-13 a0=d a1=c000116040 a2=8 a3=7f0ede6753b0 items=0 ppid=14390 pid=14399 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="runc" exe="/usr/bin/runc" subj=system_u:system_r:unconfined_service_t:s0 key=(null)
type=AVC msg=audit(1660645333.583:683): avc:  denied  { prog_run } for  pid=14399 comm="runc" scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:system_r:init_t:s0 tclass=bpf permissive=0

# backup the 1.1.2 version of runc
cp /usr/bin/runc /usr/bin/runc-112

# upgrade to containerd.io 1.6.7 (runc 1.1.3)
yum install containerd.io

# backup the 1.1.3 version of runc
cp /usr/bin/runc /usr/bin/runc-113

docker run -it --rm busybox
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: open /dev/pts/0: operation not permitted: unknown.
ERRO[0000] error waiting for container: context canceled

# manually downgrade runc to 1.1.2
cp /usr/bin/runc-112 /usr/bin/runc
cp: overwrite '/usr/bin/runc'? y

docker run -it --rm busybox
/ # exit

So looks to be a regression in runc 1.1.3?

thaJeztah Aug 16, 2022
Maintainer

I also tried on CentOS Stream 8, and not seeing the issue there

docker version and info

Client: Docker Engine - Community
 Version:           20.10.17
 API version:       1.41
 Go version:        go1.17.11
 Git commit:        100c701
 Built:             Mon Jun  6 23:03:11 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.17
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.17.11
  Git commit:       a89b842
  Built:            Mon Jun  6 23:01:29 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.7
  GitCommit:        0197261a30bf81f1ee8e6a4dd2dea0ef95d67ccb
 runc:
  Version:          1.1.3
  GitCommit:        v1.1.3-0-g6724737
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.8.2-docker)
  compose: Docker Compose (Docker Inc., v2.6.0)
  scan: Docker Scan (Docker Inc., v0.17.0)

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 1
 Server Version: 20.10.17
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 0197261a30bf81f1ee8e6a4dd2dea0ef95d67ccb
 runc version: v1.1.3-0-g6724737
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.18.0-277.el8.x86_64
 Operating System: CentOS Stream 8
 OSType: linux
 Architecture: x86_64
 CPUs: 1
 Total Memory: 808.8MiB
 Name: centos-s-1vcpu-1gb-ams3-01
 ID: A5QZ:QR6Z:QBRW:SYE5:2BES:3PUM:32AS:FXS2:BJGX:CIKF:FT3F:B2NS
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

But CentOS 8 stream uses an older kernel version, and I see tclass=bpf in the audit logs, so wondering if it doesn't hit that on the older kernel.

kolyshkin Aug 16, 2022
Maintainer

@thaJeztah can you test the runc fix from opencontainers/runc#3554? The static binary is available from https://github.com/opencontainers/runc/actions/runs/2836752368 (scroll down to "Artifacts")

thaJeztah Aug 16, 2022
Maintainer

Ah, yes, let me give that a try. I already destroyed that test-VM, but let me re-create

thaJeztah Aug 16, 2022
Maintainer

@kolyshkin looks to fix the issue;

docker run -it --rm busybox echo hello
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: open /dev/pts/0: operation not permitted: unknown.

Replace runc;

mv runc.amd64 $(which runc)
mv: overwrite '/usr/bin/runc'? y

systemctl restart docker

docker version

Client: Docker Engine - Community
 Version:           20.10.17
 API version:       1.41
 Go version:        go1.17.11
 Git commit:        100c701
 Built:             Mon Jun  6 23:03:29 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.17
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.17.11
  Git commit:       a89b842
  Built:            Mon Jun  6 23:01:12 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.7
  GitCommit:        0197261a30bf81f1ee8e6a4dd2dea0ef95d67ccb
 runc:
  Version:          1.1.3+dev
  GitCommit:        v1.1.3-15-g6e8d15ef
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Try again;

docker run -it --rm busybox echo hello
hello

AkihiroSuda · 2022-09-05T00:19:39Z

AkihiroSuda
Sep 5, 2022
Maintainer

Fixed in https://github.com/opencontainers/runc/releases/tag/v1.1.4

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docker exec doesnt work on centos stream 9 #43960

{{title}}

Replies: 2 comments 5 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

docker exec doesnt work on centos stream 9 #43960

zhouwencn Aug 13, 2022

Replies: 2 comments · 5 replies

thaJeztah Aug 16, 2022 Maintainer

thaJeztah Aug 16, 2022 Maintainer

thaJeztah Aug 16, 2022 Maintainer

kolyshkin Aug 16, 2022 Maintainer

thaJeztah Aug 16, 2022 Maintainer

thaJeztah Aug 16, 2022 Maintainer

AkihiroSuda Sep 5, 2022 Maintainer

zhouwencn
Aug 13, 2022

Replies: 2 comments 5 replies

thaJeztah
Aug 16, 2022
Maintainer

thaJeztah Aug 16, 2022
Maintainer

thaJeztah Aug 16, 2022
Maintainer

kolyshkin Aug 16, 2022
Maintainer

thaJeztah Aug 16, 2022
Maintainer

thaJeztah Aug 16, 2022
Maintainer

AkihiroSuda
Sep 5, 2022
Maintainer