WIP: krunkit driver #20826

nirs · 2025-05-23T19:11:24Z

krunkit is a tool to launch configurable virtual machines using the libkrun platform, optimized for GPU accelerated virtual machines and AI workloads on Apple silicon.

It is mostly compatible with vfkit; the driver is a simplified copy of the vfkit driver.

To get accelerated gpu the arch iso was updated to longterm kernel 6.6.92 and Virtio GPU was enabled.

Limitations:

Only one machine can be created since we use the same port for krunkit
--restful-uri. This should be fix to use an unused port or use unix socket when supported[1]

[1] containers/krunkit#47

Status:

✅ The machine boots
✅ Networking with vment-helper works
✅ /dev/dri/ is available
❌ qemu does does not boot with kernel 6.6

Based on these PRs for testing:

Fixes #20803

k8s-ci-robot · 2025-05-23T19:11:33Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: nirs
Once this PR has been reviewed and has the lgtm label, please assign spowelljr for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2025-05-23T19:11:34Z

Hi @nirs. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

minikube-bot · 2025-05-23T19:12:02Z

Can one of the admins verify this patch?

The serial console name depends on the driver. We had setting for qemu that does not work for vfkit and krunkit, breaking boot from minikube iso. Fixed by using 2 console= options, one is known to work for qemu, and one for vfkit and krunkit. With this we can use the same iso image with qemu, vfkit, and krunkit. This will allow simplifying vfkit driver. Previously we had to extract the kernel and initrd and start it using the legacy --kernel, --kernel-cmdline and --initrd options. I tested this by building the iso with this fix and running with --iso-url. Example run with qemu: % minikube start -p qemu --driver qemu --container-runtime containerd \ --iso-url file://$PWD/minikube-arm64.iso 😄 [qemu] minikube v1.36.0 on Darwin 15.5 (arm64) ✨ Using the qemu2 driver based on user configuration 🌐 Automatically selected the socket_vmnet network 👍 Starting "qemu" primary control-plane node in "qemu" cluster 🔥 Creating qemu2 VM (CPUs=2, Memory=6000MB, Disk=20000MB) ... 📦 Preparing Kubernetes v1.33.1 on containerd 1.7.23 ... ▪ Generating certificates and keys ... ▪ Booting up control plane ... ▪ Configuring RBAC rules ... 🔗 Configuring bridge CNI (Container Networking Interface) ... 🔎 Verifying Kubernetes components... ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5 🌟 Enabled addons: default-storageclass, storage-provisioner 🏄 Done! kubectl is now configured to use "qemu" cluster and "default" namespace by default Example run with krunkit: % minikube start -p krunkit --driver krunkit --container-runtime containerd \ --iso-url file://$PWD/minikube-arm64.iso 😄 [krunkit] minikube v1.36.0 on Darwin 15.5 (arm64) ✨ Using the krunkit (experimental) driver based on user configuration 👍 Starting "krunkit" primary control-plane node in "krunkit" cluster 🔥 Creating krunkit VM (CPUs=2, Memory=6000MB, Disk=20000MB) ... 📦 Preparing Kubernetes v1.33.1 on containerd 1.7.23 ... ▪ Generating certificates and keys ... ▪ Booting up control plane ... ▪ Configuring RBAC rules ... 🔗 Configuring bridge CNI (Container Networking Interface) ... 🔎 Verifying Kubernetes components... ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5 🌟 Enabled addons: default-storageclass, storage-provisioner 🏄 Done! kubectl is now configured to use "krunkit" cluster and "default" namespace by default

This speeds up machine boot by 5 seconds. The timeout may be helpful for debugging boot issues but we don't have a way to access the serial console for debugging currently. Testing shows about 4-5 seconds speedup. | driver | timeout | start time | |---------|---------|------------| | vfkit | 5.0 | 24.01 | | vfkit | 0.0 | 19.90 | | qemu | 5.0 | 29.46 | | qemu | 0.0 | 24.28 | | krunkit | 5.0 | 25.14 | | krunkit | 0.0 | 20.51 | vfkit tested booting using iso instead of direct kernel boot. Direct kernel boot is little bit faster that booting from iso even with timeout=0.

nirs · 2025-05-25T13:01:51Z

@afbjorklund can you review this?

I think the issue of not having /dev/dri is using too old kernel. We are using 5.10.207 while libkrun seems to require 5.16 or later.

libkrun for macOS is using venus, which requires

The Venus driver requires supports for

VIRTGPU_PARAM_3D_FEATURES
VIRTGPU_PARAM_CAPSET_QUERY_FIX
VIRTGPU_PARAM_RESOURCE_BLOB
VIRTGPU_PARAM_HOST_VISIBLE
VIRTGPU_PARAM_CROSS_DEVICE
VIRTGPU_PARAM_CONTEXT_INIT

from the virtio-gpu kernel driver, unless vtest is used. That usually means the guest kernel should be at least 5.16

So it seems that we need to move to newer kernel. Do you know why are stuck with 5.10?

minikube/deploy/iso/minikube-iso/configs/minikube_aarch64_defconfig

Line 38 in ec2b7c1

BR2_LINUX_KERNEL_CUSTOM_VERSION_VALUE="5.10.207"

libkrun virtio-net driver enables TSO offloading and checksum offloading by default, so we must use vment-helper --enable-tso and --enable-checksum-offload with krunkit. These options do not work with vfkit.

krunkit is a tool to launch configurable virtual machines using the libkrun platform, optimized for GPU accelerated virtual machines and AI workloads on Apple silicon. It is mostly compatible with vfkit; the driver is a simplified copy of the vfkit driver. Unlike vfkit, krunkit is available only on Apple silicon. Changes compared to vfkit driver: - krunkit requires unix socket for netwroking, so we must use vment-helper. - krunkit can be controlled only via HTTP, not via unix socket. - krunkit does not support --kernel-cmdline - We must enable vmnet offloading, required for krunkit. - The code was simplified since vmnet-helper is always used - Code was cleaned up to use .ResolveStorePath() Limitations: - Only one machine can be created since we use the same port for krunkit --restful-uri. This should be fixed to use an unused port, or use a unix socket when unix socket is supported[1]. [1] containers/krunkit#47

Previously it was used only for vfkit, so we suggested to fallback to the `nat` network. This advice is not relevant to krunkit or to qemu (which can also use vmnet-helper). Change the error to recommend installing vment-helper. We need to think how we can recommend other networks for vfkit and qemu. Another solution is to create error for every driver+network combination but this seems hard to manage.

This is the same way that we test vfkit. This test is not running in the CI. Issues: - Need to install and configure vment-helper (requires root).

afbjorklund · 2025-05-25T14:07:53Z

So it seems that we need to move to newer kernel. Do you know why are stuck with 5.10?

I don't think it is stuck with anything, just that it was using the LTS versions...

It seems that a new kernel version was not included, in the minikube OS upgrade.

Kicbase/ISO: Update buildroot from 2023.02.9 to 2025.2 #20720

Buildroot supports many: https://github.com/buildroot/buildroot/blob/2025.02.x/linux/linux.hash

5.4
5.10
5.15
6.1
6.6
6.12

See also https://www.kernel.org/ ("longterm")

nirs · 2025-05-25T14:21:56Z

So it seems that we need to move to newer kernel. Do you know why are stuck with 5.10?

I don't think it is stuck with anything, just that it was using the LTS versions...

It seems that a new kernel version was not included, in the minikube OS upgrade.

Kicbase/ISO: Update buildroot from 2023.02.9 to 2025.2 #20720

Buildroot supports many: https://github.com/buildroot/buildroot/blob/2025.02.x/linux/linux.hash

5.4

5.10

5.15

6.1

6.6

6.12

See also https://www.kernel.org/ ("longterm")

So should we try to update the kernel to latest longterm version (6.12.30)?

afbjorklund · 2025-05-25T14:23:01Z

There was some talk about bumping to a newer kernel: (6.6?)

ISO upgrades to support user namespaces #18488 (comment)

But that was last year, and for the 2024.02.x OS. So maybe.

It seems that it is only needed for this AI feature, though?

nirs · 2025-05-25T17:52:11Z

It seems that it is only needed for this AI feature, though?

Yes, this is for AI on Apple silicon. I can try to get build an iso to play with it, and when we a working iso we can discuss how to proceed with the upgrade.

afbjorklund · 2025-05-25T18:07:55Z

So you could still use vfkit for all other (non-AI) usage of Kubernetes.

Or maybe even make it a single driver, if the syntax is close enough...

nirs · 2025-05-26T00:48:53Z

pkg/drivers/krunkit/krunkit.go

+}
+
+func (d *Driver) stopKrunkit() error {
+	if err := d.SetKrunkititState("Stop"); err != nil {


krunkit ignores this value, but it is documented, so we should keep it same as vfkit.
https://github.com/containers/krunkit/blob/9421c0e1daca8af0eb84d6afc67bfb55b78214c4/src/status.rs#L101

nirs · 2025-05-26T00:51:04Z

pkg/drivers/krunkit/krunkit.go

+func (d *Driver) killkrunkit() error {
+	if err := d.SetKrunkititState("HardStop"); err != nil {
+		// Typically fails with EOF due to https://github.com/crc-org/krunkit/issues/277.
+		log.Debugf("Failed to set krunkit state to 'HardStop': %s", err)


krunkit ignores the value and does not document this value, so we should not set the state here but just kill it.
https://github.com/containers/krunkit/blob/9421c0e1daca8af0eb84d6afc67bfb55b78214c4/src/status.rs#L101

nirs · 2025-05-26T00:53:32Z

pkg/drivers/krunkit/krunkit.go

+}
+
+// Make a boot2docker VM disk image.
+func (d *Driver) generateDiskImage(size int) error {


Duplicate of vfkit version, which may duplicate of qemu?

nirs · 2025-05-26T00:55:30Z

pkg/drivers/krunkit/krunkit.go

+	State string `json:"state"`
+}
+
+func (d *Driver) GetkrunkitState() (string, error) {


Same as vfkit using http client instead of unix http client.

nirs · 2025-05-26T00:55:38Z

pkg/drivers/krunkit/krunkit.go

+	return vmstate.State, nil
+}
+
+func (d *Driver) SetKrunkititState(state string) error {


Same as vfkit using http client instead of unix http client.

nirs · 2025-05-26T00:56:51Z

pkg/drivers/krunkit/krunkit.go

+	return nil
+}
+
+func WaitForTCPWithDelay(addr string, duration time.Duration) error {


Duplicate from vfkit, likely duplicated in qemu and other drivers. Can move to some util package.

nirs · 2025-05-26T00:58:28Z

pkg/drivers/krunkit/krunkit.go

+	return fmt.Errorf("hosts without a driver cannot be upgraded")
+}
+
+func (d *Driver) getKrunkitState() (state.State, error) {


same as vfkit

nirs · 2025-05-26T00:59:44Z

pkg/drivers/krunkit/krunkit.go

+	return os.OpenFile(logfile, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o600)
+}
+
+func (d *Driver) setupIP(mac string) error {


Same as vfkit (and likey qemu and other drivers).

- Update to longterm kernel 6.6.92[1] - aarch64: Enable Virtio GPU, needed for krunkit driver Generated using by running: make iso-menuconfig-aarch64 make linux-menuconfig-aarch64 make iso-menuconfig-x86_64 make linux-menuconfig-x86_64 This generated many changes in the configs, maybe they were updated mnaully previously. With this change we can boot krunkit witht the built iso: % minikube start -p krunkit --driver krunkit --container-runtime containerd --iso-url file://$PWD/minikube-arm64-vgpu.iso 😄 [krunkit] minikube v1.36.0 on Darwin 15.5 (arm64) ✨ Using the krunkit (experimental) driver based on user configuration 👍 Starting "krunkit" primary control-plane node in "krunkit" cluster 🔥 Creating krunkit VM (CPUs=2, Memory=6000MB, Disk=20000MB) ... 📦 Preparing Kubernetes v1.33.1 on containerd 1.7.23 ... ▪ Generating certificates and keys ... ▪ Booting up control plane ... ▪ Configuring RBAC rules ... 🔗 Configuring bridge CNI (Container Networking Interface) ... 🔎 Verifying Kubernetes components... ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5 🌟 Enabled addons: storage-provisioner, default-storageclass 🏄 Done! kubectl is now configured to use "krunkit" cluster and "default" namespace by default And now we have accelerated gpu: $ tree /dev/dri /dev/dri |-- by-path | |-- platform-a007000.virtio_mmio-card -> ../card0 | `-- platform-a007000.virtio_mmio-render -> ../renderD128 |-- card0 `-- renderD128 [1] https://www.kernel.org/

nirs · 2025-05-26T20:22:07Z

ok-to-build-iso

medyagh · 2025-05-26T20:24:02Z

See the logs at:
https://storage.googleapis.com/minikube-builds/logs/PRNumber/7DigitSHA/iso_build.txt

for example for this PR
https://storage.googleapis.com/minikube-builds/logs/20669/22ed44e/iso_build.txt

nirs · 2025-05-26T22:08:51Z

So you could still use vfkit for all other (non-AI) usage of Kubernetes.

Yes, the only interesting feature in krunkit is accelerated GPU, so if you don't need this I don't see a reason to use it.

Or maybe even make it a single driver, if the syntax is close enough...

We can use a single driver that will use vfkit or krunkit, maybe if you use --enable-gpu-acceleration. But calling it vfkit and using krunkit will be confusing. Another issue is networking - for vfkit we have nat (default), or vmnet-helper (optional, for multi clusters or multi-node cluster). For krunkit we must use vmnet-helper so the --network option is not used. I think having 2 separate drivers is the best way for user experience and maintenance.

We have lot of duplicate code, but this can be solved by extracting shared pieces out of the drivers. Most of the duplicate code is also duplicated from other drivers (e.g. qemu).

nirs · 2025-05-26T22:44:29Z

pkg/drivers/krunkit/krunkit.go

+
+func (d *Driver) restfulURI() string {
+	// TODO: use unused port or a unix socket when supported.
+	// https://github.com/containers/krunkit/issues/47


We can use getAvailableTCPPortFromRange() from qemu driver.

minikube-bot · 2025-05-27T02:09:17Z

Hi @nirs, we have updated your PR with the reference to newly built ISO. Pull the changes locally if you want to test with them or update your PR further.

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels May 23, 2025

k8s-ci-robot requested a review from ComradeProgrammer May 23, 2025 19:11

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label May 23, 2025

k8s-ci-robot requested a review from spowelljr May 23, 2025 19:11

k8s-ci-robot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label May 23, 2025

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 24, 2025

nirs force-pushed the krunkit branch from d5c40d6 to cb9d010 Compare May 24, 2025 21:15

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 24, 2025

nirs force-pushed the krunkit branch 3 times, most recently from 7cda3a5 to cbe0012 Compare May 25, 2025 01:25

nirs added 5 commits May 25, 2025 16:58

vmnet: Support offloading for krunkit

31f9fe3

libkrun virtio-net driver enables TSO offloading and checksum offloading by default, so we must use vment-helper --enable-tso and --enable-checksum-offload with krunkit. These options do not work with vfkit.

hack: Add krunkit integration test

ee8fffd

This is the same way that we test vfkit. This test is not running in the CI. Issues: - Need to install and configure vment-helper (requires root).

site: Add krunkit driver documentation

de02e9e

nirs force-pushed the krunkit branch from cbe0012 to de02e9e Compare May 25, 2025 13:58

nirs mentioned this pull request May 25, 2025

clean up duplicate code in multiple files #20828

Open

nirs force-pushed the krunkit branch from de02e9e to 4afebdd Compare May 25, 2025 21:33

k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels May 25, 2025

nirs force-pushed the krunkit branch from 4afebdd to 7d04f73 Compare May 25, 2025 22:20

nirs commented May 26, 2025

View reviewed changes

nirs force-pushed the krunkit branch from 7d04f73 to 970518c Compare May 26, 2025 16:19

nirs force-pushed the krunkit branch from 970518c to ba9b02e Compare May 26, 2025 17:58

medyagh mentioned this pull request May 26, 2025

Set Correct URL for Failed ISO Builds #20851

Open

nirs commented May 26, 2025

View reviewed changes

Updating ISO to v1.36.0-1748291054-20826

8aca39b

nirs mentioned this pull request May 28, 2025

Support for GPU's on Rancher-Desktop (MAC) rancher-sandbox/rancher-desktop#5561

Open

WIP: krunkit driver #20826

Are you sure you want to change the base?

WIP: krunkit driver #20826

Conversation

nirs commented May 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-ci-robot commented May 23, 2025

Uh oh!

k8s-ci-robot commented May 23, 2025

Uh oh!

minikube-bot commented May 23, 2025

Uh oh!

nirs commented May 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

afbjorklund commented May 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nirs commented May 25, 2025

Uh oh!

afbjorklund commented May 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nirs commented May 25, 2025

Uh oh!

afbjorklund commented May 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nirs May 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nirs commented May 26, 2025

Uh oh!

medyagh commented May 26, 2025

Uh oh!

nirs commented May 26, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

minikube-bot commented May 27, 2025

Uh oh!

Uh oh!

nirs commented May 23, 2025 •

edited

Loading

nirs commented May 25, 2025 •

edited

Loading

afbjorklund commented May 25, 2025 •

edited

Loading

afbjorklund commented May 25, 2025 •

edited

Loading

afbjorklund commented May 25, 2025 •

edited

Loading

nirs May 26, 2025 •

edited

Loading