shared network mode not working on Mac M1

[annemirasol]

Note
(by @AkihiroSuda )

The following commands are reported to fix the issue on some machines:

sudo /usr/libexec/ApplicationFirewall/socketfilterfw --add /usr/libexec/bootpd
sudo /usr/libexec/ApplicationFirewall/socketfilterfw --unblock /usr/libexec/bootpd

---

Description

Environment

macOS Monterey version 12.6.2
Apple M1 Pro chip
limactl version 0.14.1

What I expected

Guest IP address of VM created with limactl start --name=default template://vmnet should be accessible (i.e. can be pinged) from host.

What actually happened

No accessible IP address.

❯ limactl list
NAME       STATUS     SSH                VMTYPE    ARCH       CPUS    MEMORY    DISK      DIR
default    Running    127.0.0.1:60022    qemu      aarch64    4       4GiB      100GiB    ~/.lima/default
❯ lima
am@lima-default:/Users/am$ sudo apt install net-tools
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
net-tools is already the newest version (1.60+git20181103.0eebece-1ubuntu5).
0 upgraded, 0 newly installed, 0 to remove and 18 not upgraded.
am@lima-default:/Users/am$ ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.5.15  netmask 255.255.255.0  broadcast 192.168.5.255
        inet6 fec0::5055:55ff:fe72:7117  prefixlen 64  scopeid 0x40<site>
        inet6 fe80::5055:55ff:fe72:7117  prefixlen 64  scopeid 0x20<link>
        ether 52:55:55:72:71:17  txqueuelen 1000  (Ethernet)
        RX packets 16673  bytes 23072545 (23.0 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1307  bytes 146839 (146.8 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lima0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fdc4:fde4:aceb:14bd:5055:55ff:fee7:9ac4  prefixlen 64  scopeid 0x0<global>
        inet6 fe80::5055:55ff:fee7:9ac4  prefixlen 64  scopeid 0x20<link>
        ether 52:55:55:e7:9a:c4  txqueuelen 1000  (Ethernet)
        RX packets 38  bytes 5883 (5.8 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 65  bytes 10104 (10.1 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 174  bytes 15527 (15.5 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 174  bytes 15527 (15.5 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Some notes

• /etc/sudoers.d/lima is properly set up.
• ping 192.168.5.15 times out. ping 192.168.105.2 times out.

[AkihiroSuda]

DHCP daemon might not be working? (sudo /usr/libexec/bootpd on the host)

[annemirasol]

Great lead, thank you so much!

Running sudo /usr/libexec/bootpd, then limactl stop default && limactl delete default && limactl start --name=default template://vmnet, still gives me no IP address. (I also tried running bootpd with the Enable DHCP service flag -D.)

With the bootpd lead, I found this issue for a different software (canonical/multipass#2387), and the suggested workaround was to disable the firewall. It's not entirely ideal, but it also works for this case. I'm still trying to figure out how to make the allowlist work, but for now, it looks like the firewall needs to be off when starting the instance.

Please feel free to close this issue, as I'm not sure if anything can be done about this on lima's side.

[annemirasol]

Some details, in case helpful:

With firewall off:

~
❯ ps ax | grep bootpd
 8852 s000  S+     0:00.00 grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn --exclude-dir=.idea --exclude-dir=.tox bootpd
~
❯ limactl stop -f default && limactl delete -f default && limactl start --name=default template://vmnet
...
~
❯ ps ax | grep bootpd
 8899   ??  Ss     0:00.03 /usr/libexec/bootpd
 8943 s000  S+     0:00.00 grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn --exclude-dir=.idea --exclude-dir=.tox bootpd

With firewall on:

~ 
❯ ps ax | grep bootpd
 9055 s000  R+     0:00.00 grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn --exclude-dir=.idea --exclude-dir=.tox bootpd
~
❯ limactl stop -f default && limactl delete -f default && limactl start --name=default template://vmnet
...
~
❯ ps ax | grep bootpd
 9202 s000  S+     0:00.00 grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn --exclude-dir=.idea --exclude-dir=.tox bootpd

With firewall on, bootpd somehow gets killed.

I ran this same process on an Intel Mac, on macOS Ventura 13.1. bootpd stays alive whether or not firewall is on.

[AkihiroSuda]

Apple's firewall or a thirdparty firewall product?

The shared network works for me with Apple's firewall on Intel macOS 13

[annemirasol]

Apple's firewall or a thirdparty firewall product?

Apple's built-in firewall.

The shared network works for me with Apple's firewall on Intel macOS 13

Same here, no problems on my other machine that is an Intel macOS 13.

[pecigonzalo]

FWIW, I do believe the problem lies in bootp as I see the packets in tcpdump but no reply.

17:43:40.833473 IP 0.0.0.0.bootpc > broadcasthost.bootps: BOOTP/DHCP, Request from 52:55:55:a5:c0:bb (oui Unknown), length 300

I then tried isc-dhcp with sudo /opt/homebrew/opt/isc-dhcp/sbin/dhcpd -f bridge100 and a config like

❯ cat /opt/homebrew/etc/dhcpd.conf
# dhcpd.conf
#

default-lease-time 600;
max-lease-time 7200;

# Use this to enble / disable dynamic dns updates globally.
#ddns-update-style none;

# If this DHCP server is the official DHCP server for the local
# network, the authoritative directive should be uncommented.
authoritative;

# Use this to send dhcp log messages to a different log file (you also
# have to hack syslog.conf to complete the redirection).
#log-facility local7;


# My First Subnet
subnet 192.168.106.0 netmask 255.255.255.0 {
  range 192.168.106.10 192.168.106.200;
  option domain-name-servers 1.1.1.1, 1.0.0.1;
  option routers 192.168.106.1;
  option broadcast-address 192.168.106.255;

  # Let's let folks keep their IP's for a while
  default-lease-time 6000;
  max-lease-time 72000;
}

Which successfully gave me an IP on the interface. So while we do have the error like this one (which might be related), I believe the core issue is bootp not working, either due to firewall or something else. Unfortunately, I could not find any logs for it.

[pecigonzalo]

I ran multiple tests on my macOS (disable firewall, disable this, disable that, etc) and I believe bootp is at fault here. Even running it manually it shows nothing, no log, no anything. I suspect its just not working as a DHCP. If anyone knows how to validate it does work, that would be great.

[aelsnz]

I just tested same in my setup and agree, it looks like bootp is it fault. If I create a seperate DHCP server similar to what you did above with isc-dhcp I get the same, a DHCP address is obtained for my underlying QEMU setup.

I was getting this in Colima which seem to be related to this issue discussed here:

FATA[0071] error starting kubernetes: error running [lima kubectl cluster-info], output: "The connection to the server localhost:8080 was refused - did you specify the right host or port?", err: "exit status 1"

The issue is that one network interface does not get a DHCP Address.

I updated to latest Mac OS 13.3 and started getting this issue. Disabled ipv6, which temporary worked, but updated to 13.3.1 and back again not working. Which made me dig into it a bit more.
Spend time looking at socket_vmnet and all seem fine, but a DHCP address does not get assigned.

Steps followed:

Step 1: Start colima: colima -p abc start -k --network-address -c4 -m6 -d10
It fails again with same error

FATA[0071] error starting kubernetes: error running [lima kubectl cluster-info], output: "The connection to the server localhost:8080 was refused - did you specify the right host or port?", err: "exit status 1"

Step 2: Stop the following:

/bin/launchctl unload -w /System/Library/LaunchDaemons/bootps.plist
/bin/launchctl unload -w /System/Library/LaunchDaemons/com.apple.dhcp6d.plist

Step 3: Start isc-dhcp service for bridge100, I used "-d" flag to get more output.
Note - install with brew install isc-dhcp and create dhcp config similar to earlier mentioned in post.

sudo /opt/homebrew/opt/isc-dhcp/sbin/dhcpd -d -f bridge100

Step 4: SSH into colima host an obtain address for col0

$ colima ssh
colima:/Users/aelsnz$ sudo su -
colima:~# ifconfig col0
col0      Link encap:Ethernet  HWaddr 52:55:55:A2:EA:9D
          inet6 addr: fde2:fcb5:a448:d160:5055:55ff:fea2:ea9d/64 Scope:Global
          inet6 addr: fe80::5055:55ff:fea2:ea9d/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:17 errors:0 dropped:0 overruns:0 frame:0
          TX packets:39 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2646 (2.5 KiB)  TX bytes:9754 (9.5 KiB)

colima:~# udhcpc -i col0
udhcpc: started, v1.35.0
udhcpc: broadcasting discover
udhcpc: broadcasting select for 192.168.106.3, server 192.168.106.1
udhcpc: lease of 192.168.106.3 obtained from 192.168.106.1, lease time 5540

IP Address now gets assigned.

I can also now create new colima environments and they work out of box while the isc-dhcp service is running.

Does anyone know how we can troubleshoot or figure out why bootp is not providing DHCP to bridge100
Is ipv6 maybe an issue here - as said, seen setting ipv6 to Link-local Only does help resolve after reboot.

[aelsnz]

Additional update:

When I adjust the Mac network ipv6 settings (I am using a LAN connection - not wifi) - and set under TCP/IP the "Configure IPv6" to "Link-Local Only and reboot (in my case, had to reboot twice which I guess does not make sense) - but then it seem as if the bootpd process starts, previously I just got launchd (one process).

Not sure if anyone knows how we can debug or make sure bootpd start properly for ipv4 DHCP?

below we can see what is using dhcp port 67

sudo lsof -nP -i4UDP:67

COMMAND   PID USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
launchd     1 root   46u  IPv4 0x9b76825c118326e7      0t0  UDP *:67
launchd     1 root   48u  IPv4 0x9b76825c118326e7      0t0  UDP *:67
launchd     1 root   49u  IPv4 0x9b76825c118326e7      0t0  UDP *:67
launchd     1 root   50u  IPv4 0x9b76825c118326e7      0t0  UDP *:67
bootpd  11431 root    0u  IPv4 0x9b76825c118326e7      0t0  UDP *:67
bootpd  11431 root    1u  IPv4 0x9b76825c118326e7      0t0  UDP *:67
bootpd  11431 root    2u  IPv4 0x9b76825c118326e7      0t0  UDP *:67

I did try a few times with something like this to manually try below before the changes to ipv6, but it was as if it starts, but nothing happens. Used below command (did unload /System/Library/LaunchDaemons/bootps.plist prior to this)

sudo /usr/libexec/bootpd -D -d 

even tried:

sudo /usr/libexec/bootpd -D -d -i bridge100

I know of a few people now that has updated to latest Mac OS 13.x and seeing the issues as discussed here.

To add you can use this to see what happens in bootpd and when things work, you can see the DHCP requests, otherwise there is nothing...

sudo log stream --process bootpd --info --debug

some working messages when I start an environment that requests dhcp via socket_vmnet / bridge100

$ sudo log stream --process bootpd --info --debug
...
...
...
2023-04-15 10:07:23.040348+1200 0x22899    Info        0x0                  13018  0    bootpd: interface <private>: ip <private> mask <private>
2023-04-15 10:07:23.040371+1200 0x22899    Info        0x0                  13018  0    bootpd: interface <private>: ip <private> mask <private>
2023-04-15 10:07:23.040837+1200 0x22899    Info        0x0                  13018  0    bootpd: use_open_directory is FALSE
2023-04-15 10:07:23.041370+1200 0x22899    Info        0x0                  13018  0    bootpd: DHCP DISCOVER [<private>]: <private> <<private>>
2023-04-15 10:07:23.041400+1200 0x22899    Debug       0x0                  13018  0    bootpd: default domain name added
2023-04-15 10:07:23.041421+1200 0x22899    Debug       0x0                  13018  0    bootpd: replying to <private>
2023-04-15 10:07:23.041747+1200 0x22899    Info        0x0                  13018  0    bootpd: OFFER sent <private> <private> pktsize 300
2023-04-15 10:07:23.077748+1200 0x2289a    Info        0x0                  13018  0    bootpd: DHCP REQUEST [<private>]: <private> <<private>>
2023-04-15 10:07:23.078603+1200 0x2289a    Debug       0x0                  13018  0    bootpd: default domain name added
2023-04-15 10:07:23.078634+1200 0x2289a    Debug       0x0                  13018  0    bootpd: replying to <private>
2023-04-15 10:07:23.078864+1200 0x2289a    Info        0x0                  13018  0    bootpd: ACK sent <private> <private> pktsize 300
2023-04-15 10:07:25.353654+1200 0x2289a    Info        0x0                  13018  0    bootpd: DHCP RELEASE [<private>]: <private>
2023-04-15 10:07:28.467289+1200 0x2289a    Info        0x0                  13018  0    bootpd: DHCP DISCOVER [<private>]: <private> <<private>>
2023-04-15 10:07:28.467444+1200 0x2289a    Debug       0x0                  13018  0    bootpd: default domain name added
2023-04-15 10:07:28.467498+1200 0x2289a    Debug       0x0                  13018  0    bootpd: replying to <private>
2023-04-15 10:07:28.470553+1200 0x2289a    Info        0x0                  13018  0    bootpd: OFFER sent <private> <private> pktsize 300
2023-04-15 10:07:28.568087+1200 0x2289a    Info        0x0                  13018  0    bootpd: DHCP REQUEST [<private>]: <private> <<private>>
2023-04-15 10:07:28.579613+1200 0x2289a    Debug       0x0                  13018  0    bootpd: default domain name added
2023-04-15 10:07:28.579838+1200 0x2289a    Debug       0x0                  13018  0    bootpd: replying to <private>
2023-04-15 10:07:28.580281+1200 0x2289a    Info        0x0                  13018  0    bootpd: ACK sent <private> <private> pktsize 300

[AravindGopala]

@aelsnz thank you for the debug commands, did you find any permanent resolution for this, your steps did help, I did set the IPV6 to link locally and it seems to work after a reboot.

[aelsnz]

@AravindGopala unfortunately the issue seem that bootpd does not provide the DHCP address.
This might also relate to - canonical/multipass#2387

As work-around in colima (latest 0.5.5) you can now pass in an environment variable COLIMA_IP and set a fixed IP in the 192.168.106.0/24 subnet - (ideally use above 200) - it is at least an option to get past this if the change on ipv6 does not work for you - I know for some this does not work, and the only option then is to use this environment variable to get a fixed IP. Hope this helps.

colima start -c 1 -d 10 -m 2 --network-address --env COLIMA_IP=192.168.106.201

INFO[0000] starting colima
INFO[0000] runtime: docker
INFO[0000] preparing network ...                         context=vm
INFO[0001] creating and starting ...                     context=vm
INFO[0032] provisioning ...                              context=docker
INFO[0032] starting ...                                  context=docker
INFO[0038] done

$ colima list
PROFILE    STATUS     ARCH       CPUS    MEMORY    DISK     RUNTIME    ADDRESS
default    Running    aarch64    1       2GiB      10GiB    docker     192.168.106.201

[AravindGopala]

@aelsnz Finally this worked for me, I have to run the below command every time I boot once, looks like bootp is being blocked by firewall.

sudo /usr/libexec/ApplicationFirewall/socketfilterfw --unblock /usr/libexec/bootpd