Memory use increases continually over time when running uvicorn[standard]

[vaughantnrc]

Description

I'm deploying a Docker container that runs uvicorn. During my testing I came across unexpected behaviour. If I pip install uvicorn[standard], then run my container, I see continually-increasing memory usage that looks a lot like a memory leak until I abort (see screenshot for memory usage over a period of about one hour). But if instead I pip install uvicorn, then my memory usage appears stable.

Reproduce

1. Clone MRE at: https://github.com/vaughantnrc/docker-uvicorn-memory-leak
2. docker build -t memory_leak:debug .
3. docker run memory_leak:debug
4. Observe memory use by e.g. docker exec -it <container_name> /bin/bash and using a tool like top

I see memory use go up about 5-10 MB/second and it never decreases.

Expected behavior

MRE should not use noticeably more memory over time.

docker version

Client:
 Cloud integration: v1.0.35+desktop.4
 Version:           24.0.6
 API version:       1.43
 Go version:        go1.20.7
 Git commit:        ed223bc
 Built:             Mon Sep  4 12:32:48 2023
 OS/Arch:           windows/amd64
 Context:           default

Server: Docker Desktop 4.23.0 (120376)
 Engine:
  Version:          24.0.6
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.7
  Git commit:       1a79695
  Built:            Mon Sep  4 12:32:16 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.22
  GitCommit:        8165feabfdfe38c65b599c4993d227328c231fca
 runc:
  Version:          1.1.8
  GitCommit:        v1.1.8-0-g82f18fe
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    24.0.6
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2-desktop.4
    Path:     C:\Program Files\Docker\cli-plugins\docker-buildx.exe
  compose: Docker Compose (Docker Inc.)
    Version:  v2.21.0-desktop.1
    Path:     C:\Program Files\Docker\cli-plugins\docker-compose.exe
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-dev.exe
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.20
    Path:     C:\Program Files\Docker\cli-plugins\docker-extension.exe
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v0.1.0-beta.7
    Path:     C:\Program Files\Docker\cli-plugins\docker-init.exe
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-sbom.exe
  scan: Docker Scan (Docker Inc.)
    Version:  v0.26.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-scan.exe
  scout: Command line tool for Docker Scout (Docker Inc.)
    Version:  0.24.1
    Path:     C:\Program Files\Docker\cli-plugins\docker-scout.exe

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 9
 Server Version: 24.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8165feabfdfe38c65b599c4993d227328c231fca
 runc version: v1.1.8-0-g82f18fe
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
 Kernel Version: 5.10.16.3-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 12.36GiB
 Name: docker-desktop
 ID: 679ac11c-4e3b-47b0-b274-732f08a17c37
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support
WARNING: daemon is not using the default seccomp profile

Diagnostics ID

33C45366-87F7-42F4-BD08-3FFB73427284/20231019214035

Additional Info

I made a MRE here and asked a colleague to help test: https://github.com/vaughantnrc/docker-uvicorn-memory-leak

These are our observations:

• We all had "memory leaks" when using the ubuntu:22.04 Docker image as a base;
• I had the "memory leak" when using the python:3.11.6 Docker image as a base (I also tried 3.11.5 and slim-bookworm variants at different points);
• Running the example outside of Docker did not produce a "memory leak" in either Window or Linux.

My version of docker is:

docker --version
Docker version 24.0.6, build ed223bc

Python packages:

pip freeze
anyio==4.0.0
click==8.1.7
exceptiongroup==1.1.3
h11==0.14.0
httptools==0.6.1
idna==3.4
python-dotenv==1.0.0
PyYAML==6.0.1
sniffio==1.3.0
typing_extensions==4.8.0
uvicorn==0.23.2
uvloop==0.18.0
watchfiles==0.21.0
websockets==11.0.3

At this point I'm not sure if the issue would be in uvicorn or Docker. So far the issue has only occurred in Docker so I'm starting the discussion here, but I'd be happy to report a parallel issue in uvicorn for further discussion... or even elsewhere if that's where the discussions lead.

Potentially-related issues, pending further investigation:

• New uvicorn[standard] version cause memory leak due to tcp ping tiangolo/uvicorn-gunicorn-docker#98
• Using UvicornWorkers in Gunicorn cause OOM on K8s encode/uvicorn#1226

[vaughantnrc]

For completeness' sake I updated Docker Desktop to the following (latest) version, and have same result with the MRE:

Client:
 Cloud integration: v1.0.35+desktop.5
 Version:           24.0.6
 API version:       1.43
 Go version:        go1.20.7
 Git commit:        ed223bc
 Built:             Mon Sep  4 12:32:48 2023
 OS/Arch:           windows/amd64
 Context:           default

Server: Docker Desktop 4.24.2 (124339)
 Engine:
  Version:          24.0.6
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.7
  Git commit:       1a79695
  Built:            Mon Sep  4 12:32:16 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.22
  GitCommit:        8165feabfdfe38c65b599c4993d227328c231fca
 runc:
  Version:          1.1.8
  GitCommit:        v1.1.8-0-g82f18fe
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

[vaughantnrc]

Just as another minor update, it appears as though pip install uvicorn alone does not cause the issue, but if I then pip install watchfiles (which is included by uvicorn[standard], then I get the issue.

[vaughantnrc]

I updated the MRE. I'm less convinced the issue lies in docker versus the specific use case. :-P But I'll keep testing and ask for this issue to be reopened if I discover another related problem.

[Lindi9999999]

i had the same problem, the problem seems to lie in the reload param when using a uvicorn instance, when handling it with gunicorn the reload flag can be left on true, i just removed it and it solved the problem