Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Uvicorn causes linux session crash with out of memory error #759

Open
GameKyuubi opened this issue Jan 9, 2023 · 3 comments
Open

Uvicorn causes linux session crash with out of memory error #759

GameKyuubi opened this issue Jan 9, 2023 · 3 comments
Labels
bug Something isn't working

Comments

@GameKyuubi
Copy link

GameKyuubi commented Jan 9, 2023

SDUI/Uvicorn eventually crashes the user's session from OOM error, even while not actively generating images. journalctl reports:

Jan 09 14:15:16 kernel: Out of memory: Killed process 569725 (uvicorn) total-vm:44099472kB, anon-rss:19645212kB, file-rss:200140kB, shmem-rss:16384kB, UID:1000 pgtables:41196kB oom_score_adj:0
Jan 09 14:15:16 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=user.slice,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/session-1.scope,task=uvicorn,pid=569725,uid=1000
Jan 09 14:15:16 kernel: [ 569725]  1000 569725 11024868  4965434 42184704     1332             0 uvicorn

To Reproduce
Run SDUI for a while, or just leave it sitting.

Expected behavior
Don't crash the user session

Desktop (please complete the following information):

  • OS: Linux 6.1.3-arch1-1 SMP PREEMPT_DYNAMIC Wed, 04 Jan 2023 16:28:15 +0000 x86_64 GNU/Linux
  • Browser: firefox
  • Version: 108.0.2 (64-bit)

Additional context
seems OOM problems are common with uvicorn:
tiangolo/uvicorn-gunicorn-docker#98
encode/uvicorn#1583
encode/uvicorn#1226

@GameKyuubi GameKyuubi added the bug Something isn't working label Jan 9, 2023
@JeLuF
Copy link
Contributor

JeLuF commented Jan 10, 2023

From the log it looks like uvicorn was killed. How did this impact the user session? Did Gnome/KDE crash due to memory shortage?

If your system runs out of memory, the kernel will start to kill processes. It should start with the largest process (uvicorn in this case), but there are additional criteria that the kernel uses to decide which process to kill. Those criteria are outside of our control.

Consider adding more swap space, or use ulimits to restrict the uvicorn process's memory consumption.

@GameKyuubi
Copy link
Author

From the log it looks like uvicorn was killed. How did this impact the user session? Did Gnome/KDE crash due to memory shortage?

I only copied the uvicorn kill but it killed everything in the session and kicked me back to tty login. Running picom on XFCE.

If your system runs out of memory, the kernel will start to kill processes. It should start with the largest process (uvicorn in this case), but there are additional criteria that the kernel uses to decide which process to kill. Those criteria are outside of our control.

I mean ok but I've got 32g ram and swap space is 8gb, not really running much otherwise and SDUI runs fine as many times as I want, it seems specifically tied to how long SDUI is running, is that not memory leak behavior?

I'll add more swap but I feel like it will still happen if it's a memory leak.

@GameKyuubi
Copy link
Author

GameKyuubi commented Jan 16, 2023

Sometimes this happens when starting up SDUI, before it's even fully loaded. This particular lockup happens after loading/unloading SDUI several times. Eventually on load it will lock up everything forcing a hard reset or if I'm lucky the session will crash pointing to Uvicorn.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants