applehv machine suddenly fails with "gvproxy exited unexpectedly"

[jfrantzius]

Hi,
on a Mac with M1 processor, I was able to run an applehv machine for some time. But all of a sudden, after a few days, it won't start anymore:

podman --debug machine start
INFO[0000] podman filtering at log level debug
DEBU[0000] Using Podman machine with `applehv` virtualization provider
DEBU[0000] connection refused: http://localhost:8081/vm/state
Starting machine "podman-machine-default"
DEBU[0000] connection refused: http://localhost:8081/vm/state
DEBU[0000] gvproxy binary being used: /opt/podman/qemu/bin/gvproxy
DEBU[0000] [-debug -mtu 1500 -ssh-port 65153 -listen-vfkit unixgram:///var/folders/04/bfd6wk9n385174twdm8hx_dc0000gp/T/podman/gvproxy.sock -forward-identity /Users/joerg.frantzius/.ssh/podman-machine-default -forward-sock /Users/joerg.frantzius/.local/share/containers/podman/machine/applehv/podman.sock -forward-dest /run/user/502/podman/podman.sock -forward-user core -pid-file /var/folders/04/bfd6wk9n385174twdm8hx_dc0000gp/T/podman/gvproxy.pid]
DEBU[0000] gvproxy unixgram socket "/var/folders/04/bfd6wk9n385174twdm8hx_dc0000gp/T/podman/gvproxy.sock" not found: stat /var/folders/04/bfd6wk9n385174twdm8hx_dc0000gp/T/podman/gvproxy.sock: no such file or directory
Error: gvproxy exited unexpectedly with exit code 1
DEBU[0000] Shutting down engines

This is with the Podman Desktop installer, and after uninstalling any remainders from a previous brew installation (for details see https://ibm.box.com/s/706fg5om1goxde1pr4di5bzk14w0rx25 )

Does anybody perhaps have an idea what's going wrong?

[jfrantzius]

gvproxy seems to be the one coming with the Podman Desktop installer:

DEBU[0000] gvproxy binary being used: /opt/podman/qemu/bin/gvproxy

vfkit is installed via brew (as it is missing in the Podman Desktop installer):

brew info vfkit
==> cfergeau/crc/vfkit: stable 0.5.0, HEAD
Command line hypervisor using Apple's Virtualization Framework
https://github.com/crc-org/vfkit
/opt/homebrew/Cellar/vfkit/0.5.0_2 (5 files, 15.7MB) *
  Built from source on 2024-01-11 at 18:25:55

[cfergeau]

gvproxy 0.7.2 had fixes for podman/applehv, make sure you use this version.
To set expectations straight, podman/applehv is still work in progress/under development, and hasn't been extensively tested, so this kind of issues are to be expected.

[benoitf]

@jfrantzius default is to use qemu

it looks like you changed to applehv so either:

you remove all files of podman to get back to qemu default for 4.x of podman

(you can do that from the light bulb in status bar then 'cleanup/purge data' or change inside containers.conf the engine

or you upgrade to podman v4.9.0 installer also (or wait Podman Desktop 1.7 embedding this one)
this installer will provide gvproxy 0.7.2 that fixes your issue

[benoitf]

podman desktop 1.7.0 embeds this installer

[cfergeau]

I'm stuck with the gvproxy that comes with Podman Desktop .dmg installer, and I don't even know what version that is :-/

The --version flag was added in 0.7.2, since it errors out in your case, it indicates it's a version older than 0.7.2.

[cfergeau]

If you want to test applehv, note that vfkit is not installed by the podman installer yet, so you'll need to get it from brew or from github.com/crc-org/vfkit.

[jfrantzius]

@cfergeau thx, so I guess that answers my question concerning vfkit installation in containers/podman#21064 (comment) for the moment

Using the 1.7 Podman Desktop pre-release (from installer), I get this from gvproxy now (not a version number, but as Git short hash maybe):

❯
gvproxy -version
gvproxy version 012bc90

The only chance of getting hold of vfkit then seems to be brew, so hopefully I should be fine with this one (just updated via brew, and latest release listed on https://github.com/crc-org/vfkit/releases):

❯
brew info vfkit
==> cfergeau/crc/vfkit: stable 0.5.1, HEAD

[cfergeau]

(not a version number, but as Git short hash maybe)

yup, will be fixed for the next release.

[jfrantzius]

Thanks @benoitf and @cfergeau , I'm able to run Podman with applehv and it hasn't ceased to work yet ;)

In case you're interested, I put together installation instructions on how to get it to work with docker-compose, and as far as I got with Testcontainers: https://gist.github.com/jfrantzius/0996e224cbe81d885553ece6f8abf99a

[jfrantzius]

Hi @benoitf and @cfergeau , unfortunately a restart of the host reproducably results in the above error again:

❯
podman machine start --log-level=debug
INFO[0000] podman filtering at log level debug
DEBU[0000] Using Podman machine with `applehv` virtualization provider
DEBU[0000] connection refused: http://localhost:8081/vm/state
Starting machine "podman-machine-default"
DEBU[0000] connection refused: http://localhost:8081/vm/state
DEBU[0000] gvproxy binary being used: /opt/podman/qemu/bin/gvproxy
DEBU[0000] [-debug -mtu 1500 -ssh-port 54328 -listen-vfkit unixgram:///var/folders/04/bfd6wk9n385174twdm8hx_dc0000gp/T/podman/gvproxy.sock -forward-sock /Users/joerg.frantzius/.local/share/containers/podman/machine/applehv/podman.sock -forward-dest /run/user/502/podman/podman.sock -forward-user core -forward-identity /Users/joerg.frantzius/.ssh/podman-machine-default -pid-file /var/folders/04/bfd6wk9n385174twdm8hx_dc0000gp/T/podman/gvproxy.pid]
DEBU[0000] gvproxy unixgram socket "/var/folders/04/bfd6wk9n385174twdm8hx_dc0000gp/T/podman/gvproxy.sock" not found: stat /var/folders/04/bfd6wk9n385174twdm8hx_dc0000gp/T/podman/gvproxy.sock: no such file or directory
Error: gvproxy exited unexpectedly with exit code 1
DEBU[0000] Shutting down engines

The only known workaround is to delete and recreate the machine :-/

Before restarting my laptop, I had stopped the machine. Also, I wasn't running Podman Desktop, but only used command line as described here: https://gist.github.com/jfrantzius/0996e224cbe81d885553ece6f8abf99a

[jfrantzius]

OK that's likely containers/podman#21288 , a fix for which is merged already containers/podman#21291

Will try the fix ...

[jfrantzius]

Looks like I cannot test it, because it looks like containers/podman#21291 was merged into main after 4.9.0, and no recent build of main is available in https://github.com/containers/podman/releases