api-performance-tests vs TechEmpower benchmarks

[gsakkis]

I did some quick runs of the api-performance-tests on a AWS instance and Starlite/Litestar generally comes at the top among the 5 contenders, often by a large margin 🚀

The TechEmpower web framework benchmarks however paint a different view. Just looking at the latest JSON serialization results, Starlite ranks almost in the middle of ~50 Python frameworks. Limiting the list to the ones covered by api-performance-tests, it's at the bottom:

Granted, these are almost a year old now but new versions aside, any insights on other reasons for such a glaring discrepancy?

[provinzkraut]

any insights on other reasons for such a glaring discrepancy?

Yes. To sum it up: The TechEmpower benchmarks are useless, because they don't impose any constraints on how the applications and their environments are set up. What you're seeing there is mostly a difference in performance of servers, not frameworks.

You actually provided a very good example of this. In the screenshot you shared, we can see that FastAPI is supposedly faster than Starlette which, if you think about it, doesn't make any sense whatsoever, seeing how FastAPI is implemented on top of Starlette, and does not alter its routing or core functionality in any way; All it does is add some logic on top of it. The only result that should be possible here is that it's slower. But the results show it to be almost twice as fast, so how is that possible?

Well in this benchmark, Starlette (And Starlite) were running on gunicorn + uvicorn workers, using a stock uvicorn, while FastAPI was running uvicorn directly, with Cython compiled dependencies (which handle the protocol layer) and uvloop.

This of course makes the comparison pretty much useless, since what's being benchmarked there are effectively different servers and setups thereof.

---

In our benchmarking suite, we use the exact same runner image for all frameworks, making for a much better comparison. Of course there are issues with this as well, since not all configurations will benefit all frameworks in a similar manner. Sanic for example seems to be slower when running uvicorn with Cython dependencies for some reason, while blacksheep's performance was almost unaffected, and therein lies the crux of framework benchmarks really; Different configurations and environments will impact each framework differently, and you can pretty much tweak the setups to give you any result you want. (There is an interesting article from Miguel Grinberg on this topic)

It is very important to keep in mind that benchmarks like these only tell you how a framework will perform under these exact conditions.

Another difference between the TechEmpower benchmarks and ours is what's actually being tested. The TechEmpower ones are very synthetic, while we try to simulate more realistic loads. There can be a stark difference between how a framework handles 10 bytes of plaintext vs 100kB, which is something TechEmpower does not test for. Our suite is quite extensive, and tries to cover a lot of basic scenarios with varying load sizes.

---

I think the takeaway from this should be that synthetic benchmarks like these are almost never a good indicator of how well an application might perform.

Personally, I think people really ought to stop referring to the TechEmpower benchmarks and frameworks should stop participating. This "chasing higher numbers" thing in that manner is quite pointless anyway and has almost no influence on real world applications.

[provinzkraut]

Would it make sense to bring up the issues and submit fixes to make apples to apples comparisons

Well.. It depends. On one hand, yes, that might make things better, but it doesn't address the fundamental issue, which is inherently unfixable.

I also think it would be a hard sell to e.g. the Sanic people, telling them "hey, how about we make your framework perform worse in these benchmarks?".

It makes sense of course that you'd want to give your benchmark the ideal environment, just like it makes sense that frameworks have specific environments in mind and are (deliberately) optimised for those.

What would need to happen for this to be a somewhat useful benchmark is to fundamentally change how the tests are run, provide a set of common environments, and then run the frameworks in those. A bit like what FastAPI does with it's 10+ different configurations, so you'd have e.g. a "gunicorn + uvicorn", "uvicorn + uvloop", "hypercorn", etc. environment, and then run each benchmark round in those.

But that's not how their testing framework was designed, and so it would require a collaborative effort of to add those things manually to all the frameworks suites, which I don't think is something that's going to happen any time soon.

[provinzkraut]

Oh, and also, I'm not sure if this is even a good idea to begin with. All it would do is give the benchmarks more apparent credibility, while they would still be fundamentally flawed.

So maybe instead of trying to fix something inherently unfixable, we should just move away from metrics like these.

[winstxnhdw]

Is there any reason why Litestar avoid benchmarks against Falcon and Blacksheep? Is Litestar intentionally avoiding comparisons against web frameworks that are faster than it?

[JacobCoffee]

If Litestar, Flask, FastAPI, Django, etc. were to be compared against Blacksheep they would look like an ant compared to a skyscraper.

Falcon I'm not sure about, @provinzkraut would have insight

[provinzkraut]

Is there any reason why Litestar avoid benchmarks against Falcon and Blacksheep? Is Litestar intentionally avoiding comparisons against web frameworks that are faster than it?

@JacobCoffee makes a good point; Yes, there's frameworks significantly faster, and also significantly slower (e.g. Django). The benchmarks on the website include a biased selection of Frameworks Litestar might reasonably be compared to in terms of feature set and target audience. It does not aim to be a good or representative general purpose performance comparison, also because I do not believe such thing exists (see my lengthier explanation above).

It's been a while since I wrote the performance test thing and knowing what I know now, I regret having included the benchmarks on our website in the first place. They were intended as a internal tool first and foremost, publishing the results was an afterthought. They are biased because of the tests and deployment methods used (even though I'd argue to a lesser degree than a lot of others out there) and don't really tell you much about how the respective frameworks behave under real world conditions.