New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Workers go into restarting/crash cycle (WORKER TIMEOUT / signal 6) #145
Comments
BTW I saw this ticket here #47 but I think its not the same issue. |
Facing the same issue. Whenever following code is executed with incorrect smtp_url, port, my worker crashes:
There is no crash if smtp_url or port is valid.
|
I'm also facing the same issue. any workarounds? |
i resolved this issue by adding worker timeout while initiating my gunicorn application.
|
Facing the same issue when running long processes on websockets and it ends up terminating the websocket connection. Any fixes? |
Facing the same issue when I use the haystack. I modifed the docker-compose.yml as following: |
Facing this issue while using docker. Working perfectly fine if run directly with None of the following suggested solutions worked:
Can someone please point me in the right direction to resolve this issue? |
Facing this same issue (on both CentOS and Ubuntu VM's), it happens during typical load and all resources are not near limits. |
Same here, anyone can suggest a good alternative? |
In my case it seemed to happen due to request timeout to external service. |
Any solution for this ? Facing the same issue when calling an endpoint which takes 1-2 min to execute. |
@atTheShikhar for me - I switched to combination Flask + uWSGI. |
Unfortunately i cannot change the SGI and framework, since most of the work are already done in my case. Just needed this one endpoint to work. |
Can you show your launching command? what parameters do you use? I assume gunicorn should run smoothly when using 1 worker on a single process. |
I am using docker so the final running command is this (the timeout part was added after reading above discussion) Btw, this runs just fine locally. Problem only happens after i deploy it on GCR. |
hmm, for me it struggles both on EC2 machine and on Fargate... but so did you try running 1 worker? just to make work, |
yup i tried with 1 worker, still no luck. |
anyone have any update? |
I've had the same issue. BUT, it only comes up when I started using max_requests setting... |
Adding some info if it helps. Gunicorn config below is run via
Server RAM: 1.9GB Thanks |
Managed to resolve this issue, sharing in case this helps. Our issue originated from making external API calls from within an async endpoint. These API calls did not support async, which introduced blocking calls to the event loop, resulting in the uvicorn worker timing out. Our reliance on FastAPI Cache decorators for these async endpoints prevented us from simply redefining these endpoints as sync ( To resolve, we made use of the from fastapi.concurrency import run_in_threadpool
@api.get('/handler')
async def handler():
...
# Slow async function
await my_async_function()
....
# Slow running sync function
await run_in_threadpool(sync_function) We released this update over 2 weeks ago and haven't seen any worker timeouts. Hopefully this helps 🙂 |
@nicholasmccrea wow, should have been hard to identify it! Great news! |
@nicholarmccrea your solution seems like a good solution, but i imagined removing the "async" from the endpoint functions allowed the requests to be handled in a different threadpool. Do you need to explicitly run it in a seperate function? I am taking the knowledge from the comment on this post : |
@mcazim98 in our case we were not able to redefine our endpoints as sync due to our reliance on FastAPI cache decorators. The FastAPI cache version we were using was Thankfully, FastAPI cache now support sync functions as of version |
Yes I was having similar issues where endpoint was requesting a response from an ML model running within the saem fastapi application. Redefining the endpoints to sync versions fixed the issue. |
I encountered a similar issue, running FastAPI backend, where endpoint seemed to randomly ignore my requests sometimes, but sometimes 'fixes itself' and works. After spending 2 weeks on this, I realized this had to do with ports. My fix was to:
|
I am struggling to know which layer is the root cause here.
My app runs fine, but then suddenly it is unable to serve requests for a while and then "fixes itself". While it's unable to serve requests my logs show:
Initially, I thought it was related to load and resource limits, but it seems to also happen during "typical load" and when resources are nowhere near their limits.
The text was updated successfully, but these errors were encountered: