switch default gateway server from hypercorn to twisted

[alexrashed]

Motivation

@thrau added twisted as supported gateway server with #9834. Since then this was tested by selected users, and it turned out to be a great server. In reduced complexity and improved performance.

Changes

• Switches the default for GATEWAY_SERVER from hypercorn to twisted.

Testing

• Since the gateway is the foundation for every single request served in LocalStack, this change has an insane coverage. If all our tests are 💚, we should be good to go!
• We also need to execute tests for our downstream projects, to make sure other projects don't break.

TODO

• Execute tests in downstream projects.
• Fix issues with streaming responses in our downstream usage.

[github-actions]

S3 Image Test Results (AMD64 / ARM64)

  2 files    2 suites   3m 0s ⏱️
398 tests 346 ✅  52 💤 0 ❌
796 runs  692 ✅ 104 💤 0 ❌

Results for commit 3a53be3.

♻️ This comment has been updated with latest results.

[github-actions]

LocalStack Community integration with Pro

    2 files  ±0      2 suites  ±0   1h 37m 39s ⏱️ + 6m 37s
2 922 tests ±0  2 632 ✅ ±0  290 💤 ±0  0 ❌ ±0 
2 924 runs  ±0  2 632 ✅ ±0  292 💤 ±0  0 ❌ ±0 

Results for commit 3a53be3. ± Comparison against base commit 0aebc1e.

♻️ This comment has been updated with latest results.

[alexrashed]

FYI: Changed the PR to draft and changed the milestone to 3.5, because we found some issues with downstream projects.

[thrau]

Following up on our discussion @dfangl, I found that there are at least two checks in the stack trace whether the data buffer is empty or not:

• https://github.com/twisted/twisted/blob/02a2b658cd1ade5d7f41f97d898913686313e615/src/twisted/web/http.py#L1251-L1255
• https://github.com/twisted/twisted/blob/02a2b658cd1ade5d7f41f97d898913686313e615/src/twisted/internet/abstract.py#L359-L363

The switch makes perfect sense to me. Why write anything when there's no data? While we could forego these conditions, I found no way to trigger a write buffer flush. I found this SO post on the topic, where one of the maintainers also suggest you shouldn't be talking to the reactor in this way in the first place.

So in summary I would say this is not a problem with the server, but rather in the particular downstream implementation that relies on the assumption that a buffer will be flushed when you write empty bytes to it.

My recommendation is therefore to fix the particular downstream implementation (for instance by sending a type of "stream initiated" event that can be ignored by clients) to make it more compatible with different IO frameworks.

[alexrashed]

@thrau @bentsku The issues in our downstream projects have been addressed by @dfangl (in a backwards compatible manner 🦸🏽) and there have been multiple runs validating that the fix works properly.
In my opinion this PR is ready to be merged! 🥳 🚀

[bentsku]

LGTM! Nice to see everything addressed, thanks a lot for fixing it! 🚀

Can't wait to see all the nice performance increases showed in #9834!