Add --device virtio-net,type=unix{gram,stream} #63

jakecorrenti · 2025-08-11T21:23:34Z

Add a new type=unix{gram,stream} argument option to the virtio-net device cmdline. This will allow the user to take advantage of the new krun_add_net_unix{gram,stream} APIs to create an independent unix datagram or stream socket-based virtio-net device.

Additionally, this PR changes the underlying implementation of the unixSocketPath argument to now use the krun_add_net_unixgram instead of the deprecated krun_set_gvproxy_path.

This will also allow the user to enable or disable offloading, proving them possible improvements outlined in the associated issue below.

Resolves: #58
Resolves: #56
Resolves: #24

jakecorrenti · 2025-08-11T21:36:05Z

~~I'll need to either facilitate a libkrun release or modify the CI to build libkrun-efi from source to fix the build failure~~

jakecorrenti · 2025-08-12T19:13:11Z

@nirs could you outline a way for me to test this such that it covers your use-case

nirs · 2025-08-15T11:40:20Z

@nirs could you outline a way for me to test this such that it covers your use-case

To test with vment-helper we need to chnage it to use the unigram device instead of unixSocketPath. I'll try to test it today.

nirs

Let's focus on the documentation first since we need to decide on the behavior before we implement it.

I will try to test the current implementation to verify that it works with vmnet-helper.

docs/usage.md

src/virtio.rs

Future krunkit provides unixgram socket option supporting offloading argument to enable of disable offloading. This is a quick and dirty change to test the new option. To complete this change we need to remove the "auto" feature for offloading since we never want to enable it automatically. It can be nice to keep it for testing krunkit with offloading, since it does increase performance for vm-to-host traffic. Requires krunkit PR: containers/krunkit#63 And installing local libkrun-efi from main.

nirs · 2025-08-15T14:36:41Z

This can be tested with this vment-helper PR:
nirs/vmnet-helper#92

Example test

Check out vment-helper

git clone https://github.com/nirs/vmnet-helper.git
cd vmnet-helper
git checkout krunkit-no-offloading

Install vment-helper

meson setup build
meson compile -C build
meson install -C build
sudo install -m 0640 sudoers.d/vmnet-helper /etc/sudoers.d/

Create virtual environment with pyyaml

python3 -m venv .venv                                                                                                                                                                        
source .venv/bin/activate
pip install pyyaml

Start server vm for testing

% ./example server --driver krunkit
Starting vmnet-helper for 'server' with interface id 'bdc7b3e0-8594-4814-aa25-05187ad2a36e'
Creating cloud-init iso '/Users/nir/.vmnet-helper/vms/server/cidata.iso'
Starting 'krunkit' virtual machine 'server' with mac address '92:c9:52:b7:6c:08'
Virtual machine IP address: 192.168.105.2

Options:

Use --driver vfkit to compare with vfkit.
Use --vmnet-offload on to test with offloading enabled

Install iperf3 in the server vm

ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -l ubuntu 192.168.105.2 sudo apt-get update
ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -l ubuntu 192.168.105.2 sudo apt-get install iperf3

Test host-to-vm performance (on the host)

% iperf3 -c 192.168.105.2
Connecting to host 192.168.105.2, port 5201
[  5] local 192.168.105.1 port 62908 connected to 192.168.105.2 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.01   sec  1.11 GBytes  9.47 Gbits/sec                  
[  5]   1.01-2.01   sec  1.12 GBytes  9.65 Gbits/sec                  
[  5]   2.01-3.01   sec  1.14 GBytes  9.82 Gbits/sec                  
[  5]   3.01-4.01   sec  1.13 GBytes  9.75 Gbits/sec                  
[  5]   4.01-5.01   sec  1.12 GBytes  9.66 Gbits/sec                  
[  5]   5.01-6.01   sec  1.13 GBytes  9.70 Gbits/sec                  
[  5]   6.01-7.01   sec  1.13 GBytes  9.73 Gbits/sec                  
[  5]   7.01-8.00   sec  1.12 GBytes  9.61 Gbits/sec                  
[  5]   8.00-9.01   sec  1.14 GBytes  9.76 Gbits/sec                  
[  5]   9.01-10.00  sec  1.13 GBytes  9.76 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  11.3 GBytes  9.69 Gbits/sec                  sender
[  5]   0.00-10.00  sec  11.3 GBytes  9.69 Gbits/sec                  receiver

Test vm-to-host performance (on the host)

% iperf3 -c 192.168.105.2 -R
Connecting to host 192.168.105.2, port 5201
Reverse mode, remote host 192.168.105.2 is sending
[  5] local 192.168.105.1 port 62910 connected to 192.168.105.2 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.01   sec   979 MBytes  8.17 Gbits/sec                  
[  5]   1.01-2.01   sec   996 MBytes  8.35 Gbits/sec                  
[  5]   2.01-3.00   sec   987 MBytes  8.31 Gbits/sec                  
[  5]   3.00-4.00   sec   992 MBytes  8.31 Gbits/sec                  
[  5]   4.00-5.01   sec   992 MBytes  8.30 Gbits/sec                  
[  5]   5.01-6.01   sec   996 MBytes  8.36 Gbits/sec                  
[  5]   6.01-7.01   sec   990 MBytes  8.30 Gbits/sec                  
[  5]   7.01-8.00   sec   983 MBytes  8.28 Gbits/sec                  
[  5]   8.00-9.01   sec   994 MBytes  8.30 Gbits/sec                  
[  5]   9.01-10.01  sec   991 MBytes  8.32 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.01  sec  9.67 GBytes  8.30 Gbits/sec    0            sender
[  5]   0.00-10.01  sec  9.67 GBytes  8.30 Gbits/sec                  receiver

Start client vm for testing

% ./example client --driver krunkit
Starting vmnet-helper for 'client' with interface id 'db4ff019-2514-45f1-a82a-623598529d7a'
Creating cloud-init iso '/Users/nir/.vmnet-helper/vms/client/cidata.iso'
Starting 'krunkit' virtual machine 'client' with mac address 'f2:22:2a:68:d0:03'
Virtual machine IP address: 192.168.105.3

Test vm-to-vm perfomance (in the client vm)

% ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -l ubuntu 192.168.105.3 iperf3 -c 192.168.105.2
Warning: Permanently added '192.168.105.3' (ED25519) to the list of known hosts.
Connecting to host 192.168.105.2, port 5201
[  5] local 192.168.105.3 port 46106 connected to 192.168.105.2 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.04 GBytes  8.88 Gbits/sec    0   3.94 MBytes       
[  5]   1.00-2.00   sec  1.03 GBytes  8.84 Gbits/sec    0   4.14 MBytes       
[  5]   2.00-3.00   sec  1.03 GBytes  8.81 Gbits/sec    0   4.14 MBytes       
[  5]   3.00-4.00   sec  1.04 GBytes  8.93 Gbits/sec    0   4.14 MBytes       
[  5]   4.00-5.00   sec  1.03 GBytes  8.89 Gbits/sec    0   4.14 MBytes       
[  5]   5.00-6.00   sec  1.04 GBytes  8.91 Gbits/sec    0   4.14 MBytes       
[  5]   6.00-7.00   sec  1.04 GBytes  8.89 Gbits/sec    0   4.14 MBytes       
[  5]   7.00-8.00   sec  1.04 GBytes  8.90 Gbits/sec    0   4.14 MBytes       
[  5]   8.00-9.00   sec  1.03 GBytes  8.89 Gbits/sec    0   4.14 MBytes       
[  5]   9.00-10.00  sec  1.04 GBytes  8.92 Gbits/sec    0   4.14 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  10.3 GBytes  8.89 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  10.3 GBytes  8.88 Gbits/sec                  receiver

It works 👍

Newer krunkit allows disabling offloading for faster networking. Keep the Offloading option in case offloading is improved in future versions. Testing shows 6.7 times faster network performance, and 9p mount is 3 times faster. Starting cluster: % minikube start --driver krunkit --container-runtime containerd 😄 minikube v1.36.0 on Darwin 15.6 (arm64) ✨ Using the krunkit (experimental) driver based on user configuration 👍 Starting "minikube" primary control-plane node in "minikube" cluster 🔥 Creating krunkit VM (CPUs=2, Memory=6144MB, Disk=20000MB) ... 📦 Preparing Kubernetes v1.33.2 on containerd 1.7.23 ... 🔗 Configuring bridge CNI (Container Networking Interface) ... 🔎 Verifying Kubernetes components... ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5 🌟 Enabled addons: default-storageclass, storage-provisioner 🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default Testing iperf3: % kubectl apply -f iper3-server.yaml deployment.apps/iperf3 created service/iperf3 created % kubectl get deploy iperf3 NAME READY UP-TO-DATE AVAILABLE AGE iperf3 1/1 1 1 9s % kubectl get service iperf3 NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE iperf3 NodePort 10.105.127.180 <none> 5201:30201/TCP 17s % iperf3 -c $(minikube ip) -p 30201 Connecting to host 192.168.105.10, port 30201 [ 5] local 192.168.105.1 port 50630 connected to 192.168.105.10 port 30201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 1.05 GBytes 9.03 Gbits/sec [ 5] 1.00-2.01 sec 1.09 GBytes 9.29 Gbits/sec [ 5] 2.01-3.01 sec 1.06 GBytes 9.09 Gbits/sec [ 5] 3.01-4.00 sec 1.08 GBytes 9.33 Gbits/sec [ 5] 4.00-5.00 sec 1.08 GBytes 9.31 Gbits/sec [ 5] 5.00-6.00 sec 1.07 GBytes 9.23 Gbits/sec [ 5] 6.00-7.00 sec 1.09 GBytes 9.35 Gbits/sec [ 5] 7.00-8.01 sec 1.08 GBytes 9.20 Gbits/sec [ 5] 8.01-9.00 sec 1.07 GBytes 9.18 Gbits/sec [ 5] 9.00-10.01 sec 1.08 GBytes 9.28 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.01 sec 10.7 GBytes 9.23 Gbits/sec sender [ 5] 0.00-10.01 sec 10.7 GBytes 9.23 Gbits/sec receiver Testing 9p mount: % minikube mount ~/models:/mnt/models 📁 Mounting host path /Users/nir/models into VM as /mnt/models ... ▪ Mount type: 9p ▪ User ID: docker ▪ Group ID: docker ▪ Version: 9p2000.L ▪ Message Size: 262144 ▪ Options: map[] ▪ Bind Address: 192.168.105.1:50614 🚀 Userspace file server: ufs starting ✅ Successfully mounted /Users/nir/models to /mnt/models 📌 NOTE: This process must stay alive for the mount to be accessible ... $ time cat /mnt/models/DeepSeek-R1-0528-Qwen3-8B-Q4_K_M.gguf >/dev/null real 0m10.219s user 0m0.007s sys 0m0.263s Requires krunkit PR: containers/krunkit#63 And installing local libkrun-efi from main.

nirs · 2025-08-15T17:02:50Z

Also tested with minikube:
kubernetes/minikube#21341

nirs · 2025-08-15T19:33:23Z

Testing offloading=true:

Start vm

% ./example server --driver krunkit --vmnet-offload on
Starting vmnet-helper for 'server' with interface id 'bdc7b3e0-8594-4814-aa25-05187ad2a36e'
Creating cloud-init iso '/Users/nir/.vmnet-helper/vms/server/cidata.iso'
Starting 'krunkit' virtual machine 'server' with mac address '92:c9:52:b7:6c:08'
Virtual machine IP address: 192.168.105.3

iperf3 - host to vm

% iperf3 -c 192.168.105.3
Connecting to host 192.168.105.3, port 5201
[  5] local 192.168.105.1 port 52348 connected to 192.168.105.3 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   163 MBytes  1.36 Gbits/sec                  
[  5]   1.00-2.00   sec   154 MBytes  1.28 Gbits/sec                  
[  5]   2.00-3.00   sec   136 MBytes  1.14 Gbits/sec                  
[  5]   3.00-4.00   sec   147 MBytes  1.23 Gbits/sec                  
[  5]   4.00-5.00   sec   154 MBytes  1.29 Gbits/sec                  
[  5]   5.00-6.00   sec   164 MBytes  1.37 Gbits/sec                  
[  5]   6.00-7.00   sec   164 MBytes  1.38 Gbits/sec                  
[  5]   7.00-8.00   sec   164 MBytes  1.37 Gbits/sec                  
[  5]   8.00-9.00   sec   162 MBytes  1.36 Gbits/sec                  
[  5]   9.00-10.01  sec   162 MBytes  1.35 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec  1.53 GBytes  1.31 Gbits/sec                  sender
[  5]   0.00-10.03  sec  1.53 GBytes  1.31 Gbits/sec                  receiver

iperf3 - vm to host

% iperf3 -c 192.168.105.3 -R
Connecting to host 192.168.105.3, port 5201
Reverse mode, remote host 192.168.105.3 is sending
[  5] local 192.168.105.1 port 52350 connected to 192.168.105.3 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  5.45 GBytes  46.7 Gbits/sec                  
[  5]   1.00-2.00   sec  5.47 GBytes  47.2 Gbits/sec                  
[  5]   2.00-3.00   sec  5.45 GBytes  46.9 Gbits/sec                  
[  5]   3.00-4.00   sec  5.47 GBytes  46.9 Gbits/sec                  
[  5]   4.00-5.00   sec  5.48 GBytes  46.9 Gbits/sec                  
[  5]   5.00-6.00   sec  5.44 GBytes  46.9 Gbits/sec                  
[  5]   6.00-7.00   sec  5.51 GBytes  47.1 Gbits/sec                  
[  5]   7.00-8.00   sec  5.48 GBytes  47.0 Gbits/sec                  
[  5]   8.00-9.00   sec  5.41 GBytes  46.7 Gbits/sec                  
[  5]   9.00-10.00  sec  5.47 GBytes  46.8 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.01  sec  54.6 GBytes  46.9 Gbits/sec    0            sender
[  5]   0.00-10.00  sec  54.6 GBytes  46.9 Gbits/sec                  receiver

nirs · 2025-08-15T19:52:32Z

Testing legacy mode

Using vment-helper main

Command:

krunkit --memory=1024 --cpus=1 --restful-uri=none:// --device=virtio-blk,path=/Users/nir/.vmnet-helper/vms/server/disk.img --device=virtio-blk,path=/Users/nir/.vmnet-helper/vms/server/cidata.iso --device=virtio-net,unixSocketPath=/Users/nir/.vmnet-helper/vms/server/vmnet.sock,mac=92:c9:52:b7:6c:08 --device=virtio-serial,logFilePath=/Users/nir/.vmnet-helper/vms/server/serial.log --krun-log-level=3

We did not get ip address:

% ./example server --driver krunkit -v
Starting vmnet-helper for 'server' with interface id 'bdc7b3e0-8594-4814-aa25-05187ad2a36e'
Creating cloud-init iso '/Users/nir/.vmnet-helper/vms/server/cidata.iso'
Starting 'krunkit' virtual machine 'server' with mac address '92:c9:52:b7:6c:08'
Timeout looking up ip address

Looking in vment-helper log we see on packet from the vm, only from the host:

% head -100 ~/.vmnet-helper/vms/server/vmnet-helper.log
INFO  [main] running /opt/vmnet-helper/bin/vmnet-helper v0.6.0-1-gfef1065 on macOS 15.6.0
INFO  [main] enabling bulk forwarding
DEBUG [main] starting vmnet interface
INFO  [main] started vmnet interface
INFO  [main] running as uid: 501 gid: 20
DEBUG [main] created lockfile "/Users/nir/.vmnet-helper/vms/server/vmnet.sock.lock"
DEBUG [main] created socket "/Users/nir/.vmnet-helper/vms/server/vmnet.sock"
INFO  [main] waiting for client on "/Users/nir/.vmnet-helper/vms/server/vmnet.sock"
DEBUG [main] connecting to "/Users/nir/.vmnet-helper/vms/server/vmnet.sock-krun.sock"
DEBUG [main] dropping invalid packet (4 bytes)
INFO  [main] serving client "/Users/nir/.vmnet-helper/vms/server/vmnet.sock-krun.sock"
DEBUG [main] allocating 64 packets of 65550 bytes for host
DEBUG [main] allocating 64 packets of 65550 bytes for vm
DEBUG [main] enable host forwarding
INFO  [main] started host formwarding
INFO  [main] started vm forwarding
INFO  [main] waiting for termination
DEBUG [vm->host] started
DEBUG [host->vm] 1 packets available
DEBUG [host->vm] forwarded 1 packets 142 bytes 0 retries
DEBUG [host->vm] 1 packets available
DEBUG [host->vm] forwarded 1 packets 142 bytes 0 retries
DEBUG [host->vm] 1 packets available
DEBUG [host->vm] forwarded 1 packets 142 bytes 0 retries
DEBUG [host->vm] 2 packets available
DEBUG [host->vm] forwarded 2 packets 526 bytes 0 retries
DEBUG [host->vm] 1 packets available
DEBUG [host->vm] forwarded 2 packets 526 bytes 0 retries
DEBUG [host->vm] 1 packets available
DEBUG [host->vm] forwarded 2 packets 526 bytes 0 retries
DEBUG [host->vm] 1 packets available
DEBUG [host->vm] forwarded 1 packets 142 bytes 0 retries
DEBUG [host->vm] 1 packets available
DEBUG [host->vm] forwarded 2 packets 526 bytes 0 retries
DEBUG [host->vm] 1 packets available
DEBUG [host->vm] forwarded 1 packets 142 bytes 0 retries
DEBUG [host->vm] 1 packets available
DEBUG [host->vm] forwarded 1 packets 142 bytes 0 retries
DEBUG [host->vm] 1 packets available
DEBUG [host->vm] forwarded 2 packets 396 bytes 0 retries
DEBUG [host->vm] 1 packets available
DEBUG [host->vm] forwarded 2 packets 364 bytes 0 retries
DEBUG [host->vm] 1 packets available
DEBUG [host->vm] forwarded 2 packets 364 bytes 0 retries
DEBUG [host->vm] 1 packets available
DEBUG [host->vm] forwarded 2 packets 364 bytes 0 retries
DEBUG [host->vm] 1 packets available
DEBUG [host->vm] forwarded 2 packets 696 bytes 0 retries
DEBUG [host->vm] 1 packets available
DEBUG [host->vm] forwarded 2 packets 396 bytes 0 retries
DEBUG [host->vm] 1 packets available
DEBUG [host->vm] forwarded 2 packets 512 bytes 0 retries
DEBUG [host->vm] 1 packets available
DEBUG [host->vm] forwarded 2 packets 696 bytes 0 retries
DEBUG [host->vm] 1 packets available
DEBUG [host->vm] forwarded 2 packets 696 bytes 0 retries
DEBUG [host->vm] 1 packets available
DEBUG [host->vm] forwarded 2 packets 266 bytes 0 retries
DEBUG [host->vm] 1 packets available
DEBUG [host->vm] forwarded 2 packets 266 bytes 0 retries
DEBUG [host->vm] 1 packets available
DEBUG [host->vm] forwarded 2 packets 266 bytes 0 retries
DEBUG [host->vm] 1 packets available
DEBUG [host->vm] forwarded 1 packets 142 bytes 0 retries
DEBUG [host->vm] 1 packets available
DEBUG [host->vm] forwarded 1 packets 142 bytes 0 retries
DEBUG [host->vm] 1 packets available
DEBUG [host->vm] forwarded 1 packets 142 bytes 0 retries
DEBUG [host->vm] 1 packets available
DEBUG [host->vm] forwarded 1 packets 142 bytes 0 retries

nirs · 2025-08-15T20:12:42Z

Looking at the code it seems that unixSocketPath is doing the right thing - using the legacy API, but it does not work now. So it looks like krun_set_gvproxy_path() is broken in libkrun main().

But since we have the new API we can use instead of the legacy API. With the following patch on top of this PR unixSocketPath works:

diff --git a/src/virtio.rs b/src/virtio.rs
index a6fe60f..7a32604 100644
--- a/src/virtio.rs
+++ b/src/virtio.rs
@@ -12,6 +12,10 @@ use std::{
 use anyhow::{anyhow, Context, Result};
 use mac_address::MacAddress;
 
+/* Send the VFKIT magic after establishing the connection,
+as required by gvproxy in vfkit mode. */
+const VFKIT_MAGIC: u32 = 1 << 0;
+
 /* Taken from uapi/linux/virtio_net.h */
 const NET_FEATURE_CSUM: u32 = 1 << 0;
 const NET_FEATURE_GUEST_CSUM: u32 = 1 << 1;
@@ -402,9 +406,17 @@ impl KrunContextSet for NetConfig {
         match &self.socket {
             SocketConfig::UnixSocketPath(path) => {
                 let path_cstr = path_to_cstring(path)?;
-                if krun_set_gvproxy_path(id, path_cstr.as_ptr()) < 0 {
+                if krun_add_net_unixgram(
+                    id,
+                    path_cstr.as_ptr(),
+                    -1,
+                    self.mac_address.bytes().as_ptr(),
+                    NET_COMPAT_FEATURES,
+                    VFKIT_MAGIC,
+                ) < 0
+                {
                     return Err(anyhow!(format!(
-                        "unable to set gvproxy path {}",
+                        "unable to add unixgram {}",
                         path_cstr.into_string().unwrap(),
                     )));
                 }

% ./example server --driver krunkit -v
Starting vmnet-helper for 'server' with interface id 'bdc7b3e0-8594-4814-aa25-05187ad2a36e'
Creating cloud-init iso '/Users/nir/.vmnet-helper/vms/server/cidata.iso'
Starting 'krunkit' virtual machine 'server' with mac address '92:c9:52:b7:6c:08'
Virtual machine IP address: 192.168.105.3

The log looks normal now:

% tail -f ~/.vmnet-helper/vms/server/vmnet-helper.log
DEBUG [host->vm] forwarded 1 packets 342 bytes 0 retries
DEBUG [vm->host] forwarded 1 packets 332 bytes
DEBUG [host->vm] 1 packets available
DEBUG [host->vm] forwarded 1 packets 342 bytes 0 retries
DEBUG [vm->host] forwarded 1 packets 90 bytes
DEBUG [vm->host] forwarded 1 packets 90 bytes
DEBUG [vm->host] forwarded 1 packets 70 bytes
DEBUG [host->vm] 1 packets available
DEBUG [host->vm] forwarded 1 packets 142 bytes 0 retries
DEBUG [vm->host] forwarded 1 packets 90 bytes
DEBUG [vm->host] forwarded 1 packets 90 bytes
DEBUG [vm->host] forwarded 1 packets 86 bytes
DEBUG [vm->host] forwarded 1 packets 42 bytes
DEBUG [host->vm] 1 packets available
DEBUG [host->vm] forwarded 1 packets 42 bytes 0 retries
DEBUG [vm->host] forwarded 2 packets 170 bytes
DEBUG [host->vm] 1 packets available
DEBUG [host->vm] forwarded 1 packets 149 bytes 0 retries
DEBUG [host->vm] 1 packets available
DEBUG [host->vm] forwarded 1 packets 169 bytes 0 retries
DEBUG [vm->host] forwarded 1 packets 90 bytes
DEBUG [host->vm] 1 packets available
DEBUG [host->vm] forwarded 1 packets 90 bytes 0 retries
DEBUG [host->vm] 1 packets available
DEBUG [host->vm] forwarded 1 packets 142 bytes 0 retries
DEBUG [vm->host] forwarded 1 packets 90 bytes
DEBUG [vm->host] forwarded 1 packets 90 bytes

jakecorrenti · 2025-08-15T20:43:34Z

Looking at the code it seems that unixSocketPath is doing the right thing - using the legacy API, but it does not work now. So it looks like krun_set_gvproxy_path() is broken in libkrun main().

But since we have the new API we can use instead of the legacy API. With the following patch on top of this PR unixSocketPath works:

diff --git a/src/virtio.rs b/src/virtio.rs
index a6fe60f..7a32604 100644
--- a/src/virtio.rs
+++ b/src/virtio.rs
@@ -12,6 +12,10 @@ use std::{
 use anyhow::{anyhow, Context, Result};
 use mac_address::MacAddress;
 
+/* Send the VFKIT magic after establishing the connection,
+as required by gvproxy in vfkit mode. */
+const VFKIT_MAGIC: u32 = 1 << 0;
+
 /* Taken from uapi/linux/virtio_net.h */
 const NET_FEATURE_CSUM: u32 = 1 << 0;
 const NET_FEATURE_GUEST_CSUM: u32 = 1 << 1;
@@ -402,9 +406,17 @@ impl KrunContextSet for NetConfig {
         match &self.socket {
             SocketConfig::UnixSocketPath(path) => {
                 let path_cstr = path_to_cstring(path)?;
-                if krun_set_gvproxy_path(id, path_cstr.as_ptr()) < 0 {
+                if krun_add_net_unixgram(
+                    id,
+                    path_cstr.as_ptr(),
+                    -1,
+                    self.mac_address.bytes().as_ptr(),
+                    NET_COMPAT_FEATURES,
+                    VFKIT_MAGIC,
+                ) < 0
+                {
                     return Err(anyhow!(format!(
-                        "unable to set gvproxy path {}",
+                        "unable to add unixgram {}",
                         path_cstr.into_string().unwrap(),
                     )));
                 }

% ./example server --driver krunkit -v
Starting vmnet-helper for 'server' with interface id 'bdc7b3e0-8594-4814-aa25-05187ad2a36e'
Creating cloud-init iso '/Users/nir/.vmnet-helper/vms/server/cidata.iso'
Starting 'krunkit' virtual machine 'server' with mac address '92:c9:52:b7:6c:08'
Virtual machine IP address: 192.168.105.3

The log looks normal now:

% tail -f ~/.vmnet-helper/vms/server/vmnet-helper.log
DEBUG [host->vm] forwarded 1 packets 342 bytes 0 retries
DEBUG [vm->host] forwarded 1 packets 332 bytes
DEBUG [host->vm] 1 packets available
DEBUG [host->vm] forwarded 1 packets 342 bytes 0 retries
DEBUG [vm->host] forwarded 1 packets 90 bytes
DEBUG [vm->host] forwarded 1 packets 90 bytes
DEBUG [vm->host] forwarded 1 packets 70 bytes
DEBUG [host->vm] 1 packets available
DEBUG [host->vm] forwarded 1 packets 142 bytes 0 retries
DEBUG [vm->host] forwarded 1 packets 90 bytes
DEBUG [vm->host] forwarded 1 packets 90 bytes
DEBUG [vm->host] forwarded 1 packets 86 bytes
DEBUG [vm->host] forwarded 1 packets 42 bytes
DEBUG [host->vm] 1 packets available
DEBUG [host->vm] forwarded 1 packets 42 bytes 0 retries
DEBUG [vm->host] forwarded 2 packets 170 bytes
DEBUG [host->vm] 1 packets available
DEBUG [host->vm] forwarded 1 packets 149 bytes 0 retries
DEBUG [host->vm] 1 packets available
DEBUG [host->vm] forwarded 1 packets 169 bytes 0 retries
DEBUG [vm->host] forwarded 1 packets 90 bytes
DEBUG [host->vm] 1 packets available
DEBUG [host->vm] forwarded 1 packets 90 bytes 0 retries
DEBUG [host->vm] 1 packets available
DEBUG [host->vm] forwarded 1 packets 142 bytes 0 retries
DEBUG [vm->host] forwarded 1 packets 90 bytes
DEBUG [vm->host] forwarded 1 packets 90 bytes

Thanks for finding the issue and providing a patch. Do you think you could submit an issue in libkrun for tracking purposes?

src/virtio.rs

nirs · 2025-08-15T20:52:11Z

Thanks for finding the issue and providing a patch. Do you think you could submit an issue in libkrun for tracking purposes?

I can add libkrun issue, but I think we need to test this again with krunkit main to ensure that this is an issue with libkrun and not some other change in this PR. Do we have any automated tests for krunkit that can reproduce this issue?

nirs · 2025-08-15T20:58:46Z

Thanks for finding the issue and providing a patch. Do you think you could submit an issue in libkrun for tracking purposes?

I can add libkrun issue, but I think we need to test this again with krunkit main to ensure that this is an issue with libkrun and not some other change in this PR. Do we have any automated tests for krunkit that can reproduce this issue?

I tested with krunkit main (with modified search path in build.rs)

make clean
make
sudo make install

% otool -L /usr/local/bin/krunkit 
/usr/local/bin/krunkit:
	/usr/local/lib/libkrun-efi.1.dylib (compatibility version 1.0.0, current version 1.14.0)
        ...

Issue reproduced:

timeout getting ip address
no traffic from vm to host

Newer krunkit allows disabling offloading for faster networking. Keep the Offloading option in case offloading is improved in future versions. Testing shows 6.7 times faster network performance, and 9p mount is 3 times faster. Starting cluster: % minikube start --driver krunkit --container-runtime containerd 😄 minikube v1.36.0 on Darwin 15.6 (arm64) ✨ Using the krunkit (experimental) driver based on user configuration 👍 Starting "minikube" primary control-plane node in "minikube" cluster 🔥 Creating krunkit VM (CPUs=2, Memory=6144MB, Disk=20000MB) ... 📦 Preparing Kubernetes v1.33.2 on containerd 1.7.23 ... 🔗 Configuring bridge CNI (Container Networking Interface) ... 🔎 Verifying Kubernetes components... ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5 🌟 Enabled addons: default-storageclass, storage-provisioner 🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default Testing iperf3: % kubectl apply -f iper3-server.yaml deployment.apps/iperf3 created service/iperf3 created % kubectl get deploy iperf3 NAME READY UP-TO-DATE AVAILABLE AGE iperf3 1/1 1 1 9s % kubectl get service iperf3 NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE iperf3 NodePort 10.105.127.180 <none> 5201:30201/TCP 17s % iperf3 -c $(minikube ip) -p 30201 Connecting to host 192.168.105.10, port 30201 [ 5] local 192.168.105.1 port 50630 connected to 192.168.105.10 port 30201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 1.05 GBytes 9.03 Gbits/sec [ 5] 1.00-2.01 sec 1.09 GBytes 9.29 Gbits/sec [ 5] 2.01-3.01 sec 1.06 GBytes 9.09 Gbits/sec [ 5] 3.01-4.00 sec 1.08 GBytes 9.33 Gbits/sec [ 5] 4.00-5.00 sec 1.08 GBytes 9.31 Gbits/sec [ 5] 5.00-6.00 sec 1.07 GBytes 9.23 Gbits/sec [ 5] 6.00-7.00 sec 1.09 GBytes 9.35 Gbits/sec [ 5] 7.00-8.01 sec 1.08 GBytes 9.20 Gbits/sec [ 5] 8.01-9.00 sec 1.07 GBytes 9.18 Gbits/sec [ 5] 9.00-10.01 sec 1.08 GBytes 9.28 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.01 sec 10.7 GBytes 9.23 Gbits/sec sender [ 5] 0.00-10.01 sec 10.7 GBytes 9.23 Gbits/sec receiver Testing 9p mount: % minikube mount ~/models:/mnt/models 📁 Mounting host path /Users/nir/models into VM as /mnt/models ... ▪ Mount type: 9p ▪ User ID: docker ▪ Group ID: docker ▪ Version: 9p2000.L ▪ Message Size: 262144 ▪ Options: map[] ▪ Bind Address: 192.168.105.1:50614 🚀 Userspace file server: ufs starting ✅ Successfully mounted /Users/nir/models to /mnt/models 📌 NOTE: This process must stay alive for the mount to be accessible ... $ time cat /mnt/models/DeepSeek-R1-0528-Qwen3-8B-Q4_K_M.gguf >/dev/null real 0m10.219s user 0m0.007s sys 0m0.263s Requires krunkit PR: containers/krunkit#63 And installing local libkrun-efi from main.

nirs · 2025-08-24T14:20:19Z

I tested also with socket_vmmet by patching vmnet-helper example to use socket_vment:

diff --git a/vmnet/vm.py b/vmnet/vm.py
index cb2d232..5864cc7 100644
--- a/vmnet/vm.py
+++ b/vmnet/vm.py
@@ -182,7 +182,7 @@ class VM:
             f"--restful-uri=none://",
             f"--device=virtio-blk,path={self.disk['image']}",
             f"--device=virtio-blk,path={self.cidata}",
-            f"--device=virtio-net,type=unixgram,path={self.socket},mac={self.mac_address},offloading={self.vmnet_offload}",
+            f"--device=virtio-net,type=unixstream,path=/var/run/socket_vmnet,mac={self.mac_address},offloading={self.vmnet_offload}",
             f"--device=virtio-serial,logFilePath={self.serial}",
             "--krun-log-level=3",
         ]

I already have socket_vment on this host using launchd service.

Starting vm:

% ./example server --driver krunkit
Starting vmnet-helper for 'server' with interface id 'bdc7b3e0-8594-4814-aa25-05187ad2a36e'
Creating cloud-init iso '/Users/nir/.vmnet-helper/vms/server/cidata.iso'
Starting 'krunkit' virtual machine 'server' with mac address '92:c9:52:b7:6c:08'
Virtual machine IP address: 192.168.105.5

iperf3 benchmarks:

% iperf3 -c 192.168.105.5                    
Connecting to host 192.168.105.5, port 5201
[  5] local 192.168.105.1 port 62081 connected to 192.168.105.5 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   533 MBytes  4.47 Gbits/sec                  
[  5]   1.00-2.00   sec   534 MBytes  4.48 Gbits/sec                  
[  5]   2.00-3.00   sec   531 MBytes  4.45 Gbits/sec                  
[  5]   3.00-4.00   sec   528 MBytes  4.43 Gbits/sec                  
[  5]   4.00-5.00   sec   532 MBytes  4.46 Gbits/sec                  
[  5]   5.00-6.00   sec   530 MBytes  4.46 Gbits/sec                  
[  5]   6.00-7.00   sec   531 MBytes  4.45 Gbits/sec                  
[  5]   7.00-8.00   sec   534 MBytes  4.48 Gbits/sec                  
[  5]   8.00-9.01   sec   534 MBytes  4.47 Gbits/sec                  
[  5]   9.01-10.01  sec   532 MBytes  4.46 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec  5.20 GBytes  4.46 Gbits/sec                  sender
[  5]   0.00-10.01  sec  5.19 GBytes  4.46 Gbits/sec                  receiver

% iperf3 -c 192.168.105.5 -R
Connecting to host 192.168.105.5, port 5201
Reverse mode, remote host 192.168.105.5 is sending
[  5] local 192.168.105.1 port 62089 connected to 192.168.105.5 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.01   sec   305 MBytes  2.55 Gbits/sec                  
[  5]   1.01-2.00   sec   305 MBytes  2.56 Gbits/sec                  
[  5]   2.00-3.00   sec   307 MBytes  2.57 Gbits/sec                  
[  5]   3.00-4.01   sec   310 MBytes  2.60 Gbits/sec                  
[  5]   4.01-5.00   sec   307 MBytes  2.58 Gbits/sec                  
[  5]   5.00-6.01   sec   306 MBytes  2.57 Gbits/sec                  
[  5]   6.01-7.01   sec   308 MBytes  2.58 Gbits/sec                  
[  5]   7.01-8.01   sec   311 MBytes  2.61 Gbits/sec                  
[  5]   8.01-9.01   sec   306 MBytes  2.57 Gbits/sec                  
[  5]   9.01-10.01  sec   307 MBytes  2.57 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.01  sec  3.00 GBytes  2.58 Gbits/sec    0            sender
[  5]   0.00-10.01  sec  3.00 GBytes  2.58 Gbits/sec                  receiver

Newer krunkit allows disabling offloading for faster networking. Keep the Offloading option in case offloading is improved in future versions. Testing shows 6.7 times faster network performance, and 9p mount is 3 times faster. Starting cluster: % minikube start --driver krunkit --container-runtime containerd 😄 minikube v1.36.0 on Darwin 15.6 (arm64) ✨ Using the krunkit (experimental) driver based on user configuration 👍 Starting "minikube" primary control-plane node in "minikube" cluster 🔥 Creating krunkit VM (CPUs=2, Memory=6144MB, Disk=20000MB) ... 📦 Preparing Kubernetes v1.33.2 on containerd 1.7.23 ... 🔗 Configuring bridge CNI (Container Networking Interface) ... 🔎 Verifying Kubernetes components... ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5 🌟 Enabled addons: default-storageclass, storage-provisioner 🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default Testing iperf3: % kubectl apply -f iper3-server.yaml deployment.apps/iperf3 created service/iperf3 created % kubectl get deploy iperf3 NAME READY UP-TO-DATE AVAILABLE AGE iperf3 1/1 1 1 9s % kubectl get service iperf3 NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE iperf3 NodePort 10.105.127.180 <none> 5201:30201/TCP 17s % iperf3 -c $(minikube ip) -p 30201 Connecting to host 192.168.105.10, port 30201 [ 5] local 192.168.105.1 port 50630 connected to 192.168.105.10 port 30201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 1.05 GBytes 9.03 Gbits/sec [ 5] 1.00-2.01 sec 1.09 GBytes 9.29 Gbits/sec [ 5] 2.01-3.01 sec 1.06 GBytes 9.09 Gbits/sec [ 5] 3.01-4.00 sec 1.08 GBytes 9.33 Gbits/sec [ 5] 4.00-5.00 sec 1.08 GBytes 9.31 Gbits/sec [ 5] 5.00-6.00 sec 1.07 GBytes 9.23 Gbits/sec [ 5] 6.00-7.00 sec 1.09 GBytes 9.35 Gbits/sec [ 5] 7.00-8.01 sec 1.08 GBytes 9.20 Gbits/sec [ 5] 8.01-9.00 sec 1.07 GBytes 9.18 Gbits/sec [ 5] 9.00-10.01 sec 1.08 GBytes 9.28 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.01 sec 10.7 GBytes 9.23 Gbits/sec sender [ 5] 0.00-10.01 sec 10.7 GBytes 9.23 Gbits/sec receiver Testing 9p mount: % minikube mount ~/models:/mnt/models 📁 Mounting host path /Users/nir/models into VM as /mnt/models ... ▪ Mount type: 9p ▪ User ID: docker ▪ Group ID: docker ▪ Version: 9p2000.L ▪ Message Size: 262144 ▪ Options: map[] ▪ Bind Address: 192.168.105.1:50614 🚀 Userspace file server: ufs starting ✅ Successfully mounted /Users/nir/models to /mnt/models 📌 NOTE: This process must stay alive for the mount to be accessible ... $ time cat /mnt/models/DeepSeek-R1-0528-Qwen3-8B-Q4_K_M.gguf >/dev/null real 0m10.219s user 0m0.007s sys 0m0.263s Requires krunkit PR: containers/krunkit#63 And installing local libkrun-efi from main.

nirs · 2025-08-24T14:31:19Z

Also tested with the minikube krunkit PR:

Starting cluster:

% minikube start --driver krunkit
😄  minikube v1.36.0 on Darwin 15.6.1 (arm64)
✨  Using the krunkit (experimental) driver based on user configuration
👍  Starting "minikube" primary control-plane node in "minikube" cluster
💾  Downloading Kubernetes v1.33.2 preload ...
    > preloaded-images-k8s-v18-v1...:  328.19 MiB / 328.19 MiB  100.00% 25.45 M
🔥  Creating krunkit VM (CPUs=2, Memory=6144MB, Disk=20000MB) ...
🐳  Preparing Kubernetes v1.33.2 on Docker 28.3.2 ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: default-storageclass, storage-provisioner
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

Deploying iperf3 server:

% kubectl apply -f ~/tmp/vmnet-test/iper3-server.yaml 
deployment.apps/iperf3 created
service/iperf3 created

% kubectl get service iperf3   
NAME     TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
iperf3   NodePort   10.110.14.62   <none>        5201:30201/TCP   46s

iperf3 benchmarks:

% iperf3 -c $(minikube ip) -p 30201
Connecting to host 192.168.105.2, port 30201
[  5] local 192.168.105.1 port 62166 connected to 192.168.105.2 port 30201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.01   sec  1004 MBytes  8.38 Gbits/sec                  
[  5]   1.01-2.00   sec   988 MBytes  8.30 Gbits/sec                  
[  5]   2.00-3.00   sec   995 MBytes  8.37 Gbits/sec                  
[  5]   3.00-4.00   sec  1010 MBytes  8.47 Gbits/sec                  
[  5]   4.00-5.01   sec  1003 MBytes  8.38 Gbits/sec                  
[  5]   5.01-6.00   sec   988 MBytes  8.32 Gbits/sec                  
[  5]   6.00-7.00   sec  1004 MBytes  8.39 Gbits/sec                  
[  5]   7.00-8.00   sec  1002 MBytes  8.40 Gbits/sec                  
[  5]   8.00-9.00   sec  1002 MBytes  8.43 Gbits/sec                  
[  5]   9.00-10.01  sec  1015 MBytes  8.48 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec  9.78 GBytes  8.39 Gbits/sec                  sender
[  5]   0.00-10.01  sec  9.78 GBytes  8.39 Gbits/sec                  receiver

% iperf3 -c $(minikube ip) -p 30201 -R
Connecting to host 192.168.105.2, port 30201
Reverse mode, remote host 192.168.105.2 is sending
[  5] local 192.168.105.1 port 62168 connected to 192.168.105.2 port 30201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   872 MBytes  7.28 Gbits/sec                  
[  5]   1.00-2.01   sec   890 MBytes  7.46 Gbits/sec                  
[  5]   2.01-3.00   sec   870 MBytes  7.31 Gbits/sec                  
[  5]   3.00-4.00   sec   870 MBytes  7.30 Gbits/sec                  
[  5]   4.00-5.01   sec   830 MBytes  6.95 Gbits/sec                  
[  5]   5.01-6.00   sec   837 MBytes  7.03 Gbits/sec                  
[  5]   6.00-7.00   sec   891 MBytes  7.47 Gbits/sec                  
[  5]   7.00-8.00   sec   992 MBytes  8.32 Gbits/sec                  
[  5]   8.00-9.00   sec  1.02 GBytes  8.79 Gbits/sec                  
[  5]   9.00-10.00  sec  1.03 GBytes  8.83 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  8.94 GBytes  7.67 Gbits/sec  700            sender
[  5]   0.00-10.00  sec  8.94 GBytes  7.67 Gbits/sec                  receiver

nirs · 2025-08-28T16:56:27Z

@Lsp @tylerfanelli can we get another review? with libkrun release planned tomorrow I hope we can merge this and release krunkit soon.

slp · 2025-08-29T05:06:57Z

.github/workflows/unit_tests-aarch64-darwin.yaml

+        run: git clone https://www.github.com/containers/libkrun.git ~/libkrun
+
+      - name: Build libkrun source
+        run: cd ~/libkrun && make EFI=1 NET=1 BLK=1 GPU=1 && sudo make EFI=1 NET=1 BLK=1 GPU=1 install && cd ~/


I'm not entirely convinced about this. I'd say having CI use a libkrun release helps ensuring krunkit doesn't accidentally try to use a feature that isn't in a release. If testing against libkrun.git is desired, I think we should duplicate the pipeline, having one working on it and another on the brew release.

Good point, I'll update that.

nirs

Not sure that 2 workloads are needed, we can use build matrix since the only difference is how we install libkrun and how we build. This can be set using matrix options.

But the last change is not needed for this PR so to avoid delays I would split the last commit to another PR so we can merge quickly and release soon.

nirs · 2025-08-29T14:43:29Z

.github/workflows/unit_tests-source-aarch64-darwin.yaml

+        run: git clone https://www.github.com/containers/libkrun.git ~/libkrun
+
+      - name: Build libkrun source
+        run: cd ~/libkrun && make EFI=1 NET=1 BLK=1 GPU=1 && sudo make EFI=1 NET=1 BLK=1 GPU=1 install && cd ~/


A cleaner way:

- name: Build libkrun source working-directory: libkrun run: | make EFI=1 NET=1 BLK=1 GPU=1 sudo make EFI=1 NET=1 BLK=1 GPU=1 install The last cd is not needed since each step run in a separate shell.

nirs · 2025-08-29T14:43:33Z

.github/workflows/unit_tests-source-aarch64-darwin.yaml

+        uses: Homebrew/actions/setup-homebrew@master
+
+      - name: Install dependencies
+        run: brew tap slp/krunkit && brew install virglrenderer clang-format


A cleaner way is:

run: | brew tap slp/krunkit brew install virglrenderer clang-format

nirs · 2025-08-29T14:44:56Z

.github/workflows/unit_tests-source-aarch64-darwin.yaml

+    runs-on: macos-latest
+    steps:
+      - name: Code checkout
+        uses: actions/checkout@v2


current version of checkout is 5:
https://github.com/actions/checkout

nirs

Looks good

jakecorrenti · 2025-08-29T17:31:12Z

I believe this will also fix #24 on top of the others already listed

nirs · 2025-08-29T17:58:56Z

I believe this will also fix #24 on top of the others already listed

Yes, I'll try to test it with vmnet-helper.

Latest krunkit can use a file descriptor instead of unix socket. Update the krunkit command to support this. Since krunkit behaves like vfkit and qemu, we can remove the special "auto" connection. Depends on krunkit PR: containers/krunkit#63 Linked with libkrun built from git.

nirs · 2025-08-30T17:54:11Z

Tested with fd= with libkrun fix:
containers/libkrun#402 (comment)

Future krunkit provides unixgram socket option supporting offloading argument to enable of disable offloading. This is a minimal change to test this option. Depends on: - containers/krunkit#63

Latest krunkit can use a file descriptor instead of unix socket. Update the krunkit command to support this. Since krunkit behaves like vfkit and qemu, we can remove the special "auto" connection. Depends on: - containers/krunkit#63 - containers/libkrun#402

Previously we use "auto", "on", "off" to enable offloading automatically for krunkit since it was always required. Now that this is an optional features we can simplify to boolean flag. Depends on: - containers/krunkit#63 - containers/libkrun#402

nirs · 2025-08-30T18:58:42Z

@jakecorrenti if you test again with offloading=on, note that the flag in vmnet-helper was changed to --enable-offloading and it does not take any value.

nirs · 2025-08-31T09:20:51Z

@jakecorrenti see updated performance results with this PR and containers/libkrun#402.

You may want to use the updated graphs from M3 machine instead of table in the readme:

slp

LGTM.

Newer krunkit allows disabling offloading for faster networking. Keep the Offloading option in case offloading is improved in future versions. Testing shows 6.7 times faster network performance, and 9p mount is 3 times faster. Starting cluster: % minikube start --driver krunkit --container-runtime containerd 😄 minikube v1.36.0 on Darwin 15.6 (arm64) ✨ Using the krunkit (experimental) driver based on user configuration 👍 Starting "minikube" primary control-plane node in "minikube" cluster 🔥 Creating krunkit VM (CPUs=2, Memory=6144MB, Disk=20000MB) ... 📦 Preparing Kubernetes v1.33.2 on containerd 1.7.23 ... 🔗 Configuring bridge CNI (Container Networking Interface) ... 🔎 Verifying Kubernetes components... ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5 🌟 Enabled addons: default-storageclass, storage-provisioner 🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default Testing iperf3: % kubectl apply -f iper3-server.yaml deployment.apps/iperf3 created service/iperf3 created % kubectl get deploy iperf3 NAME READY UP-TO-DATE AVAILABLE AGE iperf3 1/1 1 1 9s % kubectl get service iperf3 NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE iperf3 NodePort 10.105.127.180 <none> 5201:30201/TCP 17s % iperf3 -c $(minikube ip) -p 30201 Connecting to host 192.168.105.10, port 30201 [ 5] local 192.168.105.1 port 50630 connected to 192.168.105.10 port 30201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 1.05 GBytes 9.03 Gbits/sec [ 5] 1.00-2.01 sec 1.09 GBytes 9.29 Gbits/sec [ 5] 2.01-3.01 sec 1.06 GBytes 9.09 Gbits/sec [ 5] 3.01-4.00 sec 1.08 GBytes 9.33 Gbits/sec [ 5] 4.00-5.00 sec 1.08 GBytes 9.31 Gbits/sec [ 5] 5.00-6.00 sec 1.07 GBytes 9.23 Gbits/sec [ 5] 6.00-7.00 sec 1.09 GBytes 9.35 Gbits/sec [ 5] 7.00-8.01 sec 1.08 GBytes 9.20 Gbits/sec [ 5] 8.01-9.00 sec 1.07 GBytes 9.18 Gbits/sec [ 5] 9.00-10.01 sec 1.08 GBytes 9.28 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.01 sec 10.7 GBytes 9.23 Gbits/sec sender [ 5] 0.00-10.01 sec 10.7 GBytes 9.23 Gbits/sec receiver Testing 9p mount: % minikube mount ~/models:/mnt/models 📁 Mounting host path /Users/nir/models into VM as /mnt/models ... ▪ Mount type: 9p ▪ User ID: docker ▪ Group ID: docker ▪ Version: 9p2000.L ▪ Message Size: 262144 ▪ Options: map[] ▪ Bind Address: 192.168.105.1:50614 🚀 Userspace file server: ufs starting ✅ Successfully mounted /Users/nir/models to /mnt/models 📌 NOTE: This process must stay alive for the mount to be accessible ... $ time cat /mnt/models/DeepSeek-R1-0528-Qwen3-8B-Q4_K_M.gguf >/dev/null real 0m10.219s user 0m0.007s sys 0m0.263s Requires krunkit PR: containers/krunkit#63 And installing local libkrun-efi from main.

Future krunkit provides unixgram socket option supporting offloading argument to enable of disable offloading. This is a minimal change to test this option. Depends on: - containers/krunkit#63

Latest krunkit can use a file descriptor instead of unix socket. Update the krunkit command to support this. Since krunkit behaves like vfkit and qemu, we can remove the special "auto" connection. Depends on: - containers/krunkit#63 - containers/libkrun#402

Previously we use "auto", "on", "off" to enable offloading automatically for krunkit since it was always required. Now that this is an optional features we can simplify to boolean flag. Depends on: - containers/krunkit#63 - containers/libkrun#402

jakecorrenti force-pushed the disable-virtio-net-feats branch 8 times, most recently from 3dfafba to af0ff10 Compare August 12, 2025 19:07

jakecorrenti changed the title ~~WIP: allow for virtio-net offloading with --virtio-net,unixgram~~ Add --device virtio-net,unixgram Aug 12, 2025

jakecorrenti marked this pull request as ready for review August 12, 2025 19:11

jakecorrenti requested review from tylerfanelli and slp as code owners August 12, 2025 19:11

nirs reviewed Aug 15, 2025

View reviewed changes

nirs mentioned this pull request Aug 15, 2025

Support krunkit new virtio-net unixgram options nirs/vmnet-helper#92

Merged

nirs mentioned this pull request Aug 15, 2025

krunkit: Disable offloading for faster networking kubernetes/minikube#21341

Open

nirs reviewed Aug 15, 2025

View reviewed changes

src/virtio.rs Outdated Show resolved Hide resolved

nirs mentioned this pull request Aug 21, 2025

krun_set_gvproxy_path broken containers/libkrun#392

Closed

nirs approved these changes Aug 24, 2025

View reviewed changes

nirs mentioned this pull request Aug 28, 2025

Support libkrun (mostly for Virtio-GPU Venus), maybe via krunkit lima-vm/lima#2277

Open

slp reviewed Aug 29, 2025

View reviewed changes

jakecorrenti force-pushed the disable-virtio-net-feats branch from 0d888e8 to 169d6b5 Compare August 29, 2025 13:24

nirs reviewed Aug 29, 2025

View reviewed changes

nirs mentioned this pull request Aug 29, 2025

Consider adding --device virtio-net,type=unixgram for krunkit compatibility crc-org/vfkit#352

Open

jakecorrenti force-pushed the disable-virtio-net-feats branch from 169d6b5 to 7b66479 Compare August 29, 2025 17:08

nirs approved these changes Aug 29, 2025

View reviewed changes

nirs mentioned this pull request Aug 30, 2025

unixgram: Make fd non-blocking containers/libkrun#402

Merged

slp approved these changes Sep 2, 2025

View reviewed changes

slp merged commit 971e0b1 into containers:main Sep 2, 2025
3 checks passed

Add --device virtio-net,type=unix{gram,stream} #63

Add --device virtio-net,type=unix{gram,stream} #63

Uh oh!

Conversation

jakecorrenti commented Aug 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jakecorrenti commented Aug 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jakecorrenti commented Aug 12, 2025

Uh oh!

nirs commented Aug 15, 2025

Uh oh!

nirs left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nirs commented Aug 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Example test

Check out vment-helper

Install vment-helper

Create virtual environment with pyyaml

Start server vm for testing

Install iperf3 in the server vm

Test host-to-vm performance (on the host)

Test vm-to-host performance (on the host)

Start client vm for testing

Test vm-to-vm perfomance (in the client vm)

Uh oh!

nirs commented Aug 15, 2025

Uh oh!

nirs commented Aug 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Testing offloading=true:

Start vm

iperf3 - host to vm

iperf3 - vm to host

Uh oh!

nirs commented Aug 15, 2025

Testing legacy mode

Uh oh!

nirs commented Aug 15, 2025

Uh oh!

jakecorrenti commented Aug 15, 2025

Uh oh!

Uh oh!

nirs commented Aug 15, 2025

Uh oh!

nirs commented Aug 15, 2025

Uh oh!

nirs commented Aug 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nirs commented Aug 24, 2025

Uh oh!

nirs commented Aug 28, 2025

Uh oh!

slp Aug 29, 2025

Choose a reason for hiding this comment

Uh oh!

jakecorrenti Aug 29, 2025

Choose a reason for hiding this comment

Uh oh!

nirs left a comment

Choose a reason for hiding this comment

Uh oh!

nirs Aug 29, 2025

Choose a reason for hiding this comment

Uh oh!

jakecorrenti commented Aug 11, 2025 •

edited

Loading

jakecorrenti commented Aug 11, 2025 •

edited

Loading

nirs commented Aug 15, 2025 •

edited

Loading

nirs commented Aug 15, 2025 •

edited

Loading

nirs commented Aug 24, 2025 •

edited

Loading

nirs commented Aug 30, 2025 •

edited

Loading