add: use SO_REUSEPORT on platform supporting it

[mkleczek]

DISCLAIMER:
This commit was authored entirely by a human without the assistance of LLMs.

Fixes #4694

Stacked on top of #4702 as it is not enough to start a new instance, it is also necessary not to fail in-flight requests on the old instance.

[steve-chavez]

This is failing all tests, I'd suggest to put these type of PRs as draft.

[mkleczek]

This is failing all tests, I'd suggest to put these type of PRs as draft.

Hmm... worked before latest push. Will switch to draft and fix.

[mkleczek]

This is failing all tests, I'd suggest to put these type of PRs as draft.

Hmm... worked before latest push. Will switch to draft and fix.

@steve-chavez - it looks like the issue is that on my machine PostgREST startup is fast enough so that it loads the schema cache before it accepts any requests. Here in CI the new instance fails with 503 because it didn't yet load the schema cache.

The question: is there any particular reason why we return 503, instead of simply not start listening on the socket until schema cache is loaded? The way we have it right now means we can't really support zero-downtime upgrades because once we start the new instance but before it loads the schema cache, some clients will get 503.

[steve-chavez]

The question: is there any particular reason why we return 503, instead of simply not start listening on the socket until schema cache is loaded?

We were aiming to have requests wait instead of 503, this waiting does happen during schema cache reload but not on startup; we discussed this on #4129. Would it be better to not listen on the socket? How would clients behave in this case?

[develop7]

They would get a "connection refused" error, which means nobody there and is imo more confusing that any 5xx error. UX-wise I would prefer some waiting to a presumably hard fail any day.

[mkleczek]

We were aiming to have requests wait instead of 503, this waiting does happen during schema cache reload but not on startup; we discussed this on #4129. Would it be better to not listen on the socket? How would clients behave in this case?

They would get a "connection refused" error, which means nobody there and is imo more confusing that any 5xx error. UX-wise I would prefer some waiting to a presumably hard fail any day.

I am not convinced, see below.

This is a complex topic so let's dig into it a little more. The startup sequence right now is:

1. Postgrest is not running.
2. Clients get connection refused
3. Postgrest starts listening on a socket
4. Clients get 503
5. Postgrest loaded schema cache
6. Normal traffic

The alternatives are:
a - blocking during schema cache loading

1. Postgrest is not running.
2. Clients get connection refused
3. Postgrest starts listening on a socket
4. Clients are blocked and potentially timeout getting some network error
5. Postgrest loaded schema cache
6. Normal traffic

b - listening on a socket only after schema cache loaded

1. Postgrest is not running.
2. Clients get connection refused
3. Postgrest loaded schema cache and starts listening
4. Normal traffic

So from the point of view of the clients (they don't know when Postgrest was started), we have 3 alternatives:

1. connection refused -> 503 -> normal
2. connection refused -> blocked/time out -> normal
3. connection refused -> normal

I am not sure what value clients get from the first two options comparing to the third one. Diagnostics and readiness checks should be done using admin server anyway.

In case of SO_REUSEPORT the situation is even worse if we start listening early. We have the following situation: instance 1 is running, instance 2 is started. From the point of view of clients:

1. Today: normal traffic -> some clients get 503 (both instances are listening only one is ready) -> normal traffic
2. Blocking: normal traffic -> some clients blocked/time out (both instances are listening one is blocking) -> normal traffic
3. Not listening: just normal traffic (there are no disruptions at all because all requests are handled by instance 1 until instance 2 is ready and starts listening).

So the first two options cause disruptions whereas the third one is fully zero-downtime and transparent to the clients.

My take on it would be:

1. Start admin server as early as possible.
2. Load schema cache.
3. Start listening on main socket.

This would require splitting binding from listening on the main socket (ie. we need to bind without listening first so that we can pass the socket to the admin server).

@steve-chavez @develop7 thoughts?

[mkleczek]

Dependent on resolving (or having a workaround to) yesodweb/wai#853

[steve-chavez]

So the first two options cause disruptions whereas the third one is fully zero-downtime and transparent to the clients.
My take on it would be:

Agree, sounds much better.

[steve-chavez]

Start admin server as early as possible.
Load schema cache.
Start listening on main socket.

@mkleczek Q: why is the above dependent on yesodweb/wai#853? It looks like it can be done without it?

[mkleczek]

Start admin server as early as possible.
Load schema cache.
Start listening on main socket.

@mkleczek Q: why is the above dependent on yesodweb/wai#853? It looks like it can be done without it?

@steve-chavez - changing startup sequence is independent and can be done now (will raise a PR, it should be easy thing to do).

But this PR and prerequisite #4702 needs warp to be fixed.