Concurrent request handling for event based guests?

[lrowe]

While I don't think this is a high priority for now, it would probably make sense to support concurrent request handling per VM for event based guests in non-ephemeral mode.

I imagine this might be particularly useful for minimising overhead when handling web sockets / server sent events.

[fwsGonzo]

There's a laundry list of things to support before that can work, like timed mutexes and such, in order to guarantee request timeouts. But it's not an impossible task. The current design was to separate everything into their own VMs, which share a lot of memory. That's a form of concurrency, but I guess it's not optimal for low-latency tasks. If you add:

    "concurrency": 64

You will handle requests from 64 VMs, and they can share a websocket if you direct the websocket frames (somehow) to the right request VM.

So, is the overall goal to have multi-processing with shared memory? Or just more concurrency and lower latency?

[lrowe]

I was thinking of this more as a potential solution for handling large numbers (millions) of long lived connections for web socket or server sent events. A typical use case would be some kind of pubsub, e.g. news tickers, forums, chats.

My Varnish knowledge may be out of date here, but as I understand it support is limited to piping the connection to an origin server. With TinyKVM, that origin server might increasingly run embedded within Varnish.

[fwsGonzo]

I already experimented with this (and removed it, but obviously all the code is in that commit). My issue is that Varnish can't give me the encrypted socket so I have nothing to play with. Everything is TLS these days, and I could only "steal" regular connections. Of course it's technically possible, but it would require a new feature in Varnish proper, which has to be planned, prioritized etc.

It's probably easier to just open a port yourself, and the WebSocket protocol is fairly straight-forward. I've implemented it once, and I can easily do it again. We can experiment with this if you want. But if changing port is not OK, then I'm not sure how to get that client connection.

[lrowe]

I'm not sure my mental model is correct here but I assumed when Varnish piped a connection to a backend server it had already been decrypted and it copied the bytes between the two connections, re-encrypting when the backend is https. And if the incoming connection was http2 it would be doing this on the stream level.

So I imagine it should work similarly when the backend is in TinyKVM, rather than "steal" the existing socket it would open a new socket and copy the bytes between the two sockets.

[fwsGonzo]

Yes Varnish uses a pipe there, so it's probably not any extra copying in that. I imagine the Linux kernel short-circuits the transport. We can read out the bytes, parse the WebSocket frames and present the data to the guest (1) or forward a virtual fd to the guest (2).

When I stole an fd it was from a client connection directly, which avoided some machinery. However, I think going through vcl_pipe is probably OK (if not better).

[fwsGonzo]

I've written a basic epoll loop around virtual FDs. Events on the FD gets handled in a storage VM, so that program needs storage enabled.

I'm not sure if using request VMs is a good idea, as they have a lot of expectations around what is available. There is no VSL and no VRT CTX. A vcl_pipe transition means "hands off", Varnish is no longer in the picture. ... but it works.

$ ./wrk -c1 -t1 http://127.0.0.1:8080/ -H "Upgrade: X"
Running 10s test @ http://127.0.0.1:8080/
  1 threads and 1 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    16.68us    5.42us 190.00us   94.10%
    Req/Sec    59.82k     8.06k   65.13k    90.10%
  601329 requests in 10.10s, 64.23MB read
Requests/sec:  59540.78
Transfer/sec:      6.36MB

$ ./wrk -c4 -t4 http://127.0.0.1:8080/ -H "Upgrade: X"
Running 10s test @ http://127.0.0.1:8080/
  4 threads and 4 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    41.48us   78.24us   5.70ms   99.87%
    Req/Sec    24.97k     3.56k   29.99k    70.79%
  1003883 requests in 10.10s, 107.23MB read
Requests/sec:  99395.77
Transfer/sec:     10.62MB

Program: https://github.com/varnish/tinykvm_examples/blob/main/cpp/src/hello_world.cpp

With this API it should be possible to handle WebSocket frames.

$ ./wrk -c32 -t32 http://127.0.0.1:8080/ -H "Upgrade: X"
Running 10s test @ http://127.0.0.1:8080/
  32 threads and 32 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    41.77us   15.69us   1.17ms   85.01%
    Req/Sec    23.90k     2.79k   37.16k    67.33%
  7687504 requests in 10.10s, 821.11MB read
Requests/sec: 761176.16
Transfer/sec:     81.30MB

$ ./wrk -c64 -t64 http://127.0.0.1:8080/ -H "Upgrade: X"
Running 10s test @ http://127.0.0.1:8080/
  64 threads and 64 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    64.02us   63.90us   5.94ms   95.70%
    Req/Sec    16.15k     6.31k   34.45k    78.31%
  10388720 requests in 10.10s, 1.08GB read
Requests/sec: 1028573.51
Transfer/sec:    109.86MB

I've added some extra pipelining and a dynamic number of epoll systems which call into a forked VM. No resets. I think this is quite good considering that we have to go through Varnish first and then a TCP buffering scheme in order to get the load-balancing.

[lrowe]

The callback function approach is somewhat awkward for runtimes with an event loop like Deno/V8. Would it be possible to epoll from within the VM? That seems like it would be the most flexible solution and should make it possible to use existing libraries.

I guess the alternative would be to have something like wait_for_events_paused and then check the type of event within the VM and handle it as appropriate.

The use case I'm imagining would have one or more upstream http1 connections / http2 streams from which events are copied to thousands of downstream http1 connections / http2 streams using web sockets / server sent events / long polling. So it would need to be able to poll across both upstream and downstream events together.

All fd events being handled by the single storage VM my be too limiting when handling very large numbers of connections.

No VSL presumably means no logging which would not be ideal. As I understand it pipe is necessary to avoid buffering of incoming/outgoing data but maybe there's another way to achieve that.

[fwsGonzo]

epoll from within the VM requires epoll emulation, which is complicated and I can't be bothered. So nah.
Some kind of paused API would be OK, but seeing your intention to have (thousands?) of outgoing connections, which is currently not supported, I think that's a completely different picture than the one that is currently painted. Well, unless every connection gets created from connecting to Varnish, and then you multiplex from there.

The storage VM thing is history, I just forgot to edit it out. There's forked VMs now, otherwise there wouldn't be 1M req/s. So that means it might be possible to reset in the future, but let's not get ahead of ourselves. The reason why reset is hard is because request isolation is only possible if you have a dedicated VM for each request. And a paused-type API is also possible. The way it works now is possibly the lowest latency it will ever be.

When using pipe Varnish is hands-off, so there's no context to log to anymore. No client-side, no backend-side, no nothing. Pass is possible, of course, but only one side will log and it's not the epoll systems. It also adds latency. So how to retrieve that context? And how to safely outlive it? Don't ask me. Also can't use VSL across threads unless both sides lock. If the goal is low-latency then I suggest to not involve Varnish more than necessary. You will still go through vcl_recv before the pipe, of course.

A final note is that the paused API is not really necessary. You can use fibers (or coroutines) to enter and leave the event loop. Just make sure the underlying library doesn't try to use threads (which are supported, but not at the level some libraries expect).
Something simple like context-rs will do the trick: https://github.com/zonyitoo/context-rs/blob/master/examples/basic.rs

[lrowe]

Agree it makes no sense to combine reset and concurrent request handling within a single VM.

Websockets are most useful for long lived but largely idle connections, so you want to be able to handle a lots of connections per core. However I'm struggling to find good options benchmarking websockets since wrk isn't really able to do this.

This web socket benchmark of a seemingly efficient C++ framework gets 2M connections on 8 VCPUs with 150k messages/second (one per connection each 13 seconds). https://oatpp.io/benchmark/websocket/2-million/. So 50k connections a core seems like a reasonable target to aim for, though putting Varnish in front of it would presumably double the work involved and so halve the performance. (EDIT: 2M/8 is 250k not 50k... supporting lots of connections is important for websockets.)

It's definitely possible to make the callback API approach work even with V8, it's just a little more awkward to work with and is likely not quite as efficient in the case of just handling requests in a loop.

Websocket benchmarking options I've tried so far:

• The client code for the Oat benchmark linked from the article. Got it to build by updating the Dockerfile but it doesn't seem to want to run from within docker/podman and it the install script wanted to be run as root.
• The underlying Rust websocket library used for deno has a load_test.c benchmarking utility built atop uWebSockets which seems to be the fastest C++ implementation they could find. However it is extremely barebones and struggles to add above 10k connections (though it can be manually run in parallel.) https://github.com/denoland/fastwebsockets
  • Checkout uWebSockets (https://github.com/uNetworking/uWebSockets) as a sibling to fastwebsockets and git submodule update --init --recursive
  • Run make in the uWebSockets/benchmarks directory
  • Run make in fastwebsockets/benches
  • cargo build --release --example echo_server --features="upgrade"
  • Run target/release/examples/echo_server
  • 10k connections with 20 byte payload (do not specify size argument!) benches/load_test 10000 0.0.0.0 8080 0 0
  • 10k connections with 1024 byte payload benches/load_test 10000 0.0.0.0 8080 0 0 1024
  • Will run into port limits above about 20k concurrently. Ideally one would be able to specify a range of bind addresses to avoid this.

[fwsGonzo]

That's great. I've done some thinking and I think I'm not willing to support WebSockets as part of the outer API (due to attack surface). I also want to eventually upgrade the epoll system to use io_uring if that improves performance. So, right now the sockets would be dumb pipes that feed into the guest where the magic happens. That's probably exactly what you want anyway.

I've added a pause/resume API now.

$ ./wrk -c64 -t64 http://127.0.0.1:8080/ -H "Upgrade: X"
Running 10s test @ http://127.0.0.1:8080/
  64 threads and 64 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    57.04us  153.31us  14.82ms   99.87%
    Req/Sec    18.65k     6.86k   44.17k    79.08%
  11992962 requests in 10.10s, 1.25GB read
Requests/sec: 1187452.08
Transfer/sec:    126.83MB

A cool 1.2 million req/s.

Example:

static void
on_socket_prepare(int thread)
{
	while (true) {
		struct kvm_socket_event se = {};
		wait_for_socket_event_paused(&se);
		switch (se.event) {
		case SOCKET_CONNECT:
			//Print("Socket %d connected: %s\n", se.fd, se.remote);
			break;
		case SOCKET_READ:
			//Print("Socket %d read: %zu bytes\n", se.fd, se.data_len);
			break;
		case SOCKET_WRITABLE:
			//Print("Socket %d writable\n", se.fd);
			/* Write to the socket. */
			write(se.fd, response, sizeof(response)-1);
			break;
		case SOCKET_DISCONNECT:
			//Print("Socket %d disconnected: %s\n", se.fd, se.remote);
			break;
		}
		/* Resume the socket event. */
	}
}

int main(int argc, char **argv)
{
	set_socket_prepare_for_pause(on_socket_prepare);
	...
	wait_for_requests();
}

[fwsGonzo]

I've had a think. I cannot find any truly transport-agnostic WebSocket frameworks, and even though I could just write my own the reality is that the "TCP API" has to be useful. It's probably not useful right now. So, my choices are:

1. Heavily increased latency by supporting more system calls
2. Implement WebSockets on the host-side

So, I'm leaning towards implementing WebSockets on the host-side. What do you think? The API would still remain largely the same except perhaps also events for certain WebSocket frame opcodes.

[lrowe]

It would be great to keep the TCP level abstraction if possible since that would provide flexibility for implementing Server Sent Events or long polling as well as WebSockets.

Rust's hyper (which fastwebsockets is built upon) does appear to be truly transport agnostic. It looks like I would just need to implement its asynchronous Read and Write traits for it to serve a connection. https://docs.rs/hyper/latest/hyper/server/conn/http1/struct.Builder.html#method.serve_connection

Would it be possible to provide a combined wait_for_events_paused where I could subscribe to multiple event types, ideally filling an array of events to amortize the guest/host communication overhead and offering a timeout? I need to do some more research here but I think that would provide enough functionality to build a Selector to fully integrate with the Tokio event loop.

[fwsGonzo]

I've thought about handling several events in the paused API, yes. Probably somewhere down the line. Right now I'm adding Boost Beast w/websockets as a separate type of server that lives besides the pure TCP epoll solution. I think if nothing else it will show how much performance is lost in the TCP solution compared to the "simpler" WebSocket solution.

But yeah, keeping the TCP thing and using WebSocket inside would be nice, however I really tried and I gave up. So, I can only blame the API. There is no C/C++ WebSocket solution that will work with a simple connect/read/writable/disconnect type of thing. The closest is Wslay which expects at least some epoll-like behavior, which might have been OK, but I'm scared of the 0.5us system call overhead. Creating an API that depends on multiple system calls per message is a boondoggle. And your idea, to ship multiple events at a time to the guest, is exactly what the final API should be doing.

To make the API able to handle more events, you can imagine it will look something like this:

struct SocketEvent se[16];
int count = wait_for_socket_events_paused(&se, 16);
for (int i = 0; i < count; i++) {
    ... se[i];
}

For every event, a X MB read buffer will be pre-allocated in the guest from the host and pre-page-walked. It's a secret-sauce kind of thing.

[lrowe]

This might be overly simplistic but could the io_uring syscalls be supported in the guest using KVM's io_uring passthrough feature? It seems to have been built to speed up virtio drivers but I wonder if the sys calls could just be patched through after applying appropriate restrictions.

• Article: https://www.phoronix.com/news/KVM-IO-uring-Passthrough-LF2020
• Slides: https://kvm-forum.qemu.org/2020/KVMForum_2020_io_uring_passthrough_Stefano_Garzarella.pdf
• Video: https://www.youtube.com/watch?v=MXwjMW2t0gE
• LWN article on restrictions: https://lwn.net/Articles/826053/

That would provide batched submission for socket reads and writes.

[fwsGonzo]

That looks like something that we can do down the line, for sure. Seems very fitting for this, and especially since it can be limited to "previously registered fds" on the ring.