docker kill prevents containers with unless-stopped restart policy to be started after reboot

[RomanValov]

Description

According to the doc docker kill command may be used to issue non-terminating signals to processes running inside containers. I.e. it's a regular practice for daemons to handle SIGHUP signal to reload its configuration. Another example is Java applications which dump their stack traces on SIGQUIT.

When a container is signalled with non-terminating signal with docker kill or docker-compose kill it continues to run and no indication is given that the container is a subject for termination (i.e. via docker inspect command). However if the container has restart policy set to unless-stopped it's no longer started after host system reboot (be it either soft or hard one).

Also there is no any indication in logs or in output of docker inspect why it has not started on boot. The exit code may differ depending on kind of reboot/reset happened though.

Reproduce

$ docker run --detach --restart unless-stopped nginx
a8d12aff922b44a0500c43c1aa4881184d4bb0331299805efb8fe265b5dbb998
$ docker kill --signal HUP a8d12aff922b44a0500c43c1aa4881184d4bb0331299805efb8fe265b5dbb998
a8d12aff922b44a0500c43c1aa4881184d4bb0331299805efb8fe265b5dbb998
$ docker ps -a
CONTAINER ID   IMAGE         COMMAND                  CREATED          STATUS                       PORTS     NAMES
a8d12aff922b   nginx         "/docker-entrypoint.…"   11 seconds ago   Up 10 seconds                80/tcp    nice_proskuriakova
$ reboot  # or reset the machine
...
$ docker ps -a
CONTAINER ID   IMAGE         COMMAND                  CREATED          STATUS                        PORTS     NAMES
a8d12aff922b   nginx         "/docker-entrypoint.…"   14 minutes ago   Exited (0) 13 seconds ago               nice_proskuriakova

Expected behavior

docker kill should not affect restarts of the container on reboot

docker version

Client: Docker Engine - Community
 Version:           26.1.1
 API version:       1.45
 Go version:        go1.21.9
 Git commit:        4cf5afa
 Built:             Tue Apr 30 11:47:54 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          26.1.1
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.9
  Git commit:       ac2de55
  Built:            Tue Apr 30 11:47:54 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.31
  GitCommit:        e377cd56a71523140ca6ae87e30244719194a521
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client: Docker Engine - Community
 Version:    26.1.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.14.0
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.27.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 3
  Running: 2
  Paused: 0
  Stopped: 1
 Images: 2
 Server Version: 26.1.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: e377cd56a71523140ca6ae87e30244719194a521
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.8.0-31-generic
 Operating System: Ubuntu 24.04 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 1.027GiB
 Name: flaxseed
 ID: 18420a47-b0af-44c5-9fec-a70de7269b5a
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional Info

Initially observed on docker 24.0.5 from Ubuntu distribution packages. We bumped into this in the production. It was especially annoying because issuing non-terminating kills to containers to inspect them and reboots/resets were spaced out in time and there were no any hints provided by docker about the reasons containers were failed to restart.