Controlling and understanding parallel requests

[bodograumann]

Due to some issues with requests that take a long time, we wondered about how many requests are actually run in parallel / concurrently. We don't necessarily want to have a large limit, but we want to be able to control the maximal number of requests processed in parallel.
We are using fastapi and had 3 guincorn workers defined, but there were 7 requests processed in parallel. My guess is that this is 8 cores - 1, but I couldn't find any place where this is configured explicitely. During my research only more questions popped up.

FastAPI uses Starlette, which does parallelization with a thread-pool using AnyIO. So why does it make sense to still have gunicorn with multiple workers on top? (In this case we define our controller method without asyncio, so no concurrency is used.)

The default AnyIO threadpool has size 40, so that is not the limiting factor.

Uvicorn also has some options for workers and limit_concurrency, but I don't understand what these would do in the complex context...

Could anybody enlighten me?

[bodograumann]

I did some testing and found the following.

Most likely the observation of a maximum of 7 parallel requests handled was not correct and all requests received by the service where handled in parallel.

There is a vital difference between using def and async def for endpoints in FastAPI.

When using def, starlette (which FastAPI is based upon) uses an AnyIO threadpool to process multiple requests in parallel. The limit of parallel requests is the size of the threadpool (so 40 by default, which is difficult to change). These execution paths share the same FastAPI app instance, python modules and any global variables. The GIL prevents most serious problems, but you might still have to use locks.

When using async def there is only one thread per gunicorn worker with a single FastAPI instance. In this case multiple requests might be handled concurrently at the places where await is used. Within the non-concurrent parts of the code you don't have to worry about synchronization, but you might still want to be careful when using global state.

I didn't find a built-in way in FastAPI to limit the concurrency per worker, but a standard asyncio.Semaphore could probably be used. One open question here would be how gunicorn assigns requests to workers. When a worker is blocked, successive requests should be handled by the other workers instead. At the point the semaphore blocks this is probably too late.

Sources:

• https://stackoverflow.com/questions/70446584/how-does-fastapi-uvicorn-parallelize-requests
• https://stackoverflow.com/questions/71516140/fastapi-runs-api-calls-in-serial-instead-of-parallel-fashion
• https://stackoverflow.com/questions/48483348/how-to-limit-concurrency-with-python-asyncio
• https://anyio.readthedocs.io/en/stable/synchronization.html#capacity-limiters

[br3ndonland]

Thank you for sharing some insights on this. Concurrency can be difficult to understand and control.

There is a vital difference between using def and async def for endpoints in FastAPI.

Yes, definitely. I think it's summarized well in the post you shared:

Note that async does not mean parallel, but concurrently. Asynchronous code with async and await is many times summarised as using coroutines. Coroutines are collaborative (or cooperatively multitasked), meaning that "at any given time, a program with coroutines is running only one of its coroutines, and this running coroutine suspends its execution only when it explicitly requests to be suspended" (see here and here for more info on coroutines)...

If, however, a blocking I/O-bound or CPU-bound operation was directly executed/called inside an async def function/endpoint, it would block the main thread (and hence, the event loop). Hence, a blocking operation such as time.sleep() in an async def endpoint would block the entire server (as in the code example provided in your question). Thus, if your endpoint is not going to make any async calls, you could declare it with just def instead, which would be run in an external threadpool that would then be awaited...

This means that def or async def should be chosen based on what the function does.

• When a function contains entirely sync logic, using def instead of async def will allow Starlette to run the function in its threadpool.
• When a function contains async logic with await statements, async def must be used. But what if an async def function has multiple awaits? If not handled correctly, the function can actually return as slowly as a standard synchronous function. One strategy for handling multiple awaits is using an AnyIO task group. This is not explained well in the FastAPI or Starlette docs. Dagger's concurrency guide may be helpful (📖 Concurrency guide dagger/dagger#4766).
• When a function mixes sync and async logic, a threadpool executor or starlette.concurrency.run_in_threadpool may be helpful (Best practice when a single handler calls both async and sync functions tiangolo/fastapi#7623 (comment)).

Due to some issues with requests that take a long time, we wondered about how many requests are actually run in parallel / concurrently. We don't necessarily want to have a large limit, but we want to be able to control the maximal number of requests processed in parallel.

There is some interest in customizing the size of the Starlette threadpool (encode/starlette#1724).

I didn't find a built-in way in FastAPI to limit the concurrency per worker

Each Gunicorn worker runs a separate FastAPI app instance. If using a Uvicorn worker with Gunicorn, concurrency limits can be specified through Uvicorn. Uvicorn has a limit_concurrency config setting that may help, but Uvicorn settings cannot be customized by default when using workers because uvicorn.workers.UvicornWorker has hard-coded CONFIG_KWARGS.

The solution recommended in the Uvicorn docs is to subclass uvicorn.workers.UvicornWorker and modify the CONFIG_KWARGS. It might look like this:

from uvicorn.workers import UvicornWorker


class MyUvicornWorker(UvicornWorker):
    CONFIG_KWARGS = {"loop": "auto", "http": "auto", "limit_concurrency": 30}

Let's say this subclass is in my_package.MyUvicornWorker. When running inboard, specify the environment variable WORKER_CLASS="my_package.MyUvicornWorker". This will be passed to the Gunicorn worker_class setting. Gunicorn will run the custom worker and apply its config settings.