axum hangs with tokio::spawn, but works with #tokio::main

[vlabel]

why having a blocking handler (or long running) prevents axum from serving new clients -> when we using it with tokio::spawn()
but it works when we just await in main()

working example:

#[tokio::main]
async fn main() {
    let app = Router::new().route("/1", get(handler1)).route("/2", get(handler2));
    let s = async move {
         let listener = tokio::net::TcpListener::bind("127.0.0.1:3000")
        .await
        .unwrap();
         axum::serve(listener, app).await.unwrap();
    };
    s.await;
}

async fn handler1() -> Html<&'static str> {
    while true {  // long running handler
    }
    Html("<h1>Never return!</h1>")
}
async fn handler2() -> Html<&'static str> {
    Html("<h1>Hello, World!</h1>")
}

curl 127.0.0.1:3000/1 -> stuck
curl 127.0.0.1:3000/2 -> keeps working

broken example:

#[tokio::main]
async fn main() {
    let app = Router::new().route("/1", get(handler1)).route("/2", get(handler2));
    let s = async move {
         let listener = tokio::net::TcpListener::bind("127.0.0.1:3000")
        .await
        .unwrap();
         axum::serve(listener, app).await.unwrap();
    };
    tokio::spawn(s).await;
}

async fn handler1() -> Html<&'static str> {
    while true {  // long running handler
    }
    Html("<h1>Never return!</h1>")
}
async fn handler2() -> Html<&'static str> {
    Html("<h1>Hello, World!</h1>")
}

curl 127.0.0.1:3000/1 -> stuck
curl 127.0.0.1:3000/2 -> not connected

I know that I should use tokio::task::spawn_blocking and do not call long computation in handlers
but anyways -> blocking 1 handler should not block whole server from serving new clients

[asonix]

In tokio, the main thread is not actually one of the tokio worker threads. Your infinite handler ties up a tokio worker thread regardless of how Axum is running, but in the case that you spawn the server, it has the chance of running on the same thread as Axum and preventing progress.

[mladedav]

That said I don't understand why calling it just once would prevent axum from accepting new connections. Since tokio implements work stealing, I'd expect other workers to pick up the accepting task while the one worker is blocked.

[mladedav]

I've been digging into it a bit and there's this issue so I guess what happens here is that the task runs on the same thread that has the IO lock and so that gets blocked. This is an issue just because all the other threads are parked. I've tried sending a lot of parallel requests to handler2 and in that scenario sending one or two requests to handler1 indeed did not break the server.

So this really is a tokio issue and I'm not sure we can reasonably do something about it here. In the linked issue there is an example of how to mitigate this by spawning an additional thread but the user has to do that, I wouldn't do it from axum.

[vlabel]

Thanks @mladedav
I also see that tokio schedules tasks a bit different in 2 cases:

• when we just .await in main - it really uses different threads under the hood for handlers

• when we use spawn -> handlers scheduled to the same thread (I checke with thread id)

[vlabel]

what axum could do is to use dedicated runtime to accept connections like risingwavelabs/risingwave#16838

[vlabel]

Ahm, I see, #[tokio::main] uses block_on under the hood. it uses different scheduling logic, it actually schedule all nested async tasks on the thread pool. tokio::spawn actually tries to schedule nested tasks on the same thread.

• when we run axum::serve is tokio::main -> we just lucky that due to so scheduling we are less pron to One bad task can halt all executor progress forever tokio#4730

• when we run in tokio::spawn -> we more likely can get our thread accepting connection(and whole service) stuck due to a single slow handler

I would say, consider not using axum::serve in tokio.spawn on prod. or make sure all the handlers are async, safe and fast.