Can't see exception tracebacks in pytest with TestClient

[mikemcowie]

Hi there

Nice framework. It's been interesting to see pieces coming together over time, and it's starting to click overall for me.

When testing with pytest, if there's an unhandled exception raised that leads to a server error, I'd expect to see a traceback in pytest that helps me pinpoint the nature of the exception that was raised. But all I'm seeing is that the status code was 500, which doesn't especially help me. I wind up being better off spinning up the app with uvicorn and trying to recreate it there, but that's much slower, especially if it wound up being a cause that needed a lot of steps to recreate.

Am I missing something? Details below:

versions

pip freeze | grep litestar
litestar==2.0.0b2

pip freeze | grep pytest
pytest==7.4.0

sample

from litestar import Litestar, get
from litestar.testing import TestClient


@get("")
def raise_something() -> str:
    raise Exception("A bad thing happened")


app = Litestar(
    route_handlers=[raise_something]
)


client = TestClient(app)


def test_that_fails():
    response = client.get("")
    assert response.status_code == 200

command

pytest test_sample.py

output

latform linux -- Python 3.11.2, pytest-7.4.0, pluggy-1.2.0
rootdir: /home/mikec/Projects/project_revisions
plugins: Faker-19.1.0, asyncio-0.21.1, anyio-3.7.1
asyncio: mode=Mode.STRICT
collected 1 item                                                                                                                                                                                                                

tests/test_client_not_showing_exception_in_pytest.py F                                                                                                                                                                    [100%]

=========================================================================================================== FAILURES ============================================================================================================
________________________________________________________________________________________________________ test_that_fails ________________________________________________________________________________________________________

    def test_that_fails():
        response = client.get("")
>       assert response.status_code == 200
E       assert 500 == 200
E        +  where 500 = <Response [500 Internal Server Error]>.status_code

tests/test_client_not_showing_exception_in_pytest.py:19: AssertionError
----------------------------------------------------------------------------------------------------- Captured stderr call ------------------------------------------------------------------------------------------------------
INFO - 2023-07-19 18:29:20,352 - httpx - _client - HTTP Request: GET http://testserver.local "HTTP/1.1 500 Internal Server Error"
------------------------------------------------------------------------------------------------------- Captured log call -------------------------------------------------------------------------------------------------------
INFO     httpx:_client.py:1013 HTTP Request: GET http://testserver.local "HTTP/1.1 500 Internal Server Error"
======================================================================================================= warnings summary ========================================================================================================
tests/test_client_not_showing_exception_in_pytest.py:4
  /home/mikec/Projects/project_revisions/tests/test_client_not_showing_exception_in_pytest.py:4: LitestarWarning: Use of a synchronous callable <function raise_something at 0x7f8bc0bc4ae0> without setting sync_to_thread is discouraged since synchronous callables can block the main thread if they perform blocking operations. If the callable is guaranteed to be non-blocking, you can set sync_to_thread=False to skip this warning, or set the environmentvariable LITESTAR_WARN_IMPLICIT_SYNC_TO_THREAD=0 to disable warnings of this type entirely.
    @get("")

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
==================================================================================================== short test summary info ====================================================================================================
FAILED tests/test_client_not_showing_exception_in_pytest.py::test_that_fails - assert 500 == 200
================================================================================================= 1 failed, 1 warning in 0.31s ==================================================================================================

Expected

I would have expected the above pytest output to include a traceback that pinpoints the line of code where the issue occurred.

[Goldziher]

You can set debug=True in your test client and that will give you a traceback.

@provinzkraut - we should switch testing to use debug by default.

[provinzkraut]

We should, yes.

I think we talked about this back when that flag was added to the test clients, but I can't remember why we did not implement it.

[mikemcowie]

Thanks @Goldziher

Seems to be a correction, though - the "debug" parameter appears to be on the app, rather than on the testclient.

app = Litestar(
    route_handlers=[raise_something],
    debug=True
)

I definitely agree it should be default, as this is definitely is a surprising behaviour when other frameworks are doing this by default.

On a more general level, generally server errors aren't logging tracebacks at all unless I setup with app (or CLI command litestar run) to enable debug? And then when debug is enabled, the tracebacks are in the HTTP response body (when I'd only want them on my server logs)?

My expectation there was that the client gets a generic "Server error" message without much detail (maybe utilities/contrib components might modify it to provide a correlation ID), and the traceback comes in the server log in the standard python format if I've done nothing intentional to change the behaviour.

i.e. how can I get this response on my API:

while getting a proper traceback on my server logs, rather than just this ?

[provinzkraut]

Seems to be a correction, though - the "debug" parameter appears to be on the app, rather than on the testclient.

What @Goldziher was referring to were the create_test_client and create_async_test_client functions, which do have this parameter.

[mikemcowie]

What @Goldziher was referring to were the create_test_client and create_async_test_client functions, which do have this parameter.

Decent point, I do recall seeing that in the document but for whatever reason didn't revisit those - so what you have there is feedback from "someone who skims documentation, but mostly tries to instantiate things in ways the ways he expects them to work from fastapi/django/flask" etc, however much store you want to hold by that!

[mikemcowie]

To add a bit more, https://docs.litestar.dev/2/usage/testing.html ,doesn't cover that usage of those functions yet, it talks about their usecase only in the sense of testing parts of the application without instantiating the entire application. If the behaviour I originally expected remains non-default for some reason, definitely the doc could be worked on. (I'm not disinclined to try to step up and make a PR sometime at least for the doc).