Proper (libvirt) network setup to work with vagrant-libvirt

[bolausson]

Hello all,

I am struggling to start my VMs because of various network issues. Could someone explain how to setup the libvirt network the correct way to make it work with vagrant-libvirt?

My network looks like this
(the physical interphase enp0s31f6 is bridged with br0 which then is used in my VMs with a bridge public network)

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
2: dummy0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
3: enp0s31f6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP mode DEFAULT group default qlen 1000
4: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN mode DEFAULT group default qlen 1000
5: ip_vti0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN mode DEFAULT group default qlen 1000
6: ip6_vti0@NONE: <NOARP> mtu 1428 qdisc noop state DOWN mode DEFAULT group default qlen 1000
7: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN mode DEFAULT group default qlen 1000
8: ip6tnl0@NONE: <NOARP> mtu 1452 qdisc noop state DOWN mode DEFAULT group default qlen 1000
9: br0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
10: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
11: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UNKNOWN mode DEFAULT group default qlen 1000
12: vnet1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UNKNOWN mode DEFAULT group default qlen 1000

vnet0 and vnet1 are from two VMs I did setup with virt-manager - they are using br0 for bridged public network:

<interface type="bridge">
  <mac address="52:54:00:34:49:70"/>
  <source network="bridged-network" portid="28d39071-29b0-4b24-bb47-2f59ac3b9577" bridge="br0"/>
  <target dev="vnet0"/>
  <model type="virtio"/>
  <alias name="net0"/>
  <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
</interface>

The libvirt network looks like this (and is working perfectly fine with the two VMs)

$ virsh net-list
 Name              State    Autostart   Persistent
----------------------------------------------------
 bridged-network   active   yes         yes
 default           active   yes         yes

$ virsh net-dumpxml bridged-network 
<network connections='2'>
  <name>bridged-network</name>
  <uuid>574c50b6-b406-45eb-8eef-b7ec3d3cc69e</uuid>
  <forward mode='bridge'/>
  <bridge name='br0'/>
</network>

$ virsh net-dumpxml default 
<network>
  <name>default</name>
  <uuid>bd52c54a-ed00-4ac2-af0c-16eeef39afdd</uuid>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr0' stp='on' delay='0'/>
  <mac address='52:54:00:13:76:30'/>
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.122.2' end='192.168.122.254'/>
    </dhcp>
  </ip>
</network>

With the above config, I get the following error when starting a VM with vagrant-libvirt:

$ cat vagrantfile 
#!/usr/bin/env ruby
#
ENV['VAGRANT_DEFAULT_PROVIDER'] = 'libvirt'
Vagrant.configure("2") do |config|
	config.vm.provider "libvirt" do |libvirt|
		libvirt.clock_offset = "localtime"
	end

	config.vm.define "testvm1" do |node|
		node.vm.box = "client_libvirt"
		node.vm.hostname = "testvm1"

		node.ssh.username = "root"
		node.ssh.password = "SomeStrongPassword"
		
		#node.vm.network :public_network, :dev => "virbr0", :mode => "bridge", :type => "bridge"

		node.vm.provider "libvirt" do |vbox|
			vbox.default_prefix = "libvirt_"
			vbox.memory = "1024"
			vbox.cpus = "2"
			vbox.video_vram = "128"
			vbox.keymap = "de-de"
		end
	end
end

$ vagrant up --provider=libvirt
Bringing machine 'testvm1' up with 'libvirt' provider...
==> testvm1: Creating image (snapshot of base box volume).
==> testvm1: Creating domain with the following settings...
==> testvm1:  -- Name:              libvirt_testvm1
==> testvm1:  -- Description:       Source: /home/mythtv/vagranttest/Vagrantfile
==> testvm1:  -- Domain type:       kvm
==> testvm1:  -- Cpus:              2
==> testvm1:  -- Feature:           acpi
==> testvm1:  -- Feature:           apic
==> testvm1:  -- Feature:           pae
==> testvm1:  -- Clock offset:      localtime
==> testvm1:  -- Memory:            1024M
==> testvm1:  -- Management MAC:    
==> testvm1:  -- Loader:            
==> testvm1:  -- Nvram:             
==> testvm1:  -- Base box:          client_libvirt
==> testvm1:  -- Storage pool:      default
==> testvm1:  -- Image():     /home/mythtv/libvirt_storage/libvirt_testvm1.img, 40G
==> testvm1:  -- Disk driver opts:  cache='default'
==> testvm1:  -- Kernel:            
==> testvm1:  -- Initrd:            
==> testvm1:  -- Graphics Type:     vnc
==> testvm1:  -- Graphics Port:     -1
==> testvm1:  -- Graphics IP:       127.0.0.1
==> testvm1:  -- Graphics Password: Not defined
==> testvm1:  -- Video Type:        cirrus
==> testvm1:  -- Video VRAM:        128
==> testvm1:  -- Video 3D accel:    false
==> testvm1:  -- Sound Type:	
==> testvm1:  -- Keymap:            de-de
==> testvm1:  -- TPM Backend:       passthrough
==> testvm1:  -- TPM Path:          
==> testvm1:  -- INPUT:             type=mouse, bus=ps2
==> testvm1: Creating shared folders metadata...
Error while activating network: Call to virNetworkCreate failed: internal error: Failed to apply firewall rules /sbin/iptables -w --table filter --insert LIBVIRT_INP --in-interface virbr1 --protocol tcp --destination-port 67 --jump ACCEPT: iptables: No chain/target/match by that name.

This seems to create a network named "vagrant-libvirt" which I can't start manually as well:

$ virsh net-dumpxml --network vagrant-libvirt 
<network ipv6='yes'>
  <name>vagrant-libvirt</name>
  <uuid>63607064-8f14-4c60-8bc3-d34b8bf9dc47</uuid>
  <forward mode='nat'/>
  <bridge name='virbr1' stp='on' delay='0'/>
  <mac address='52:54:00:cf:a8:4c'/>
  <ip address='192.168.121.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.121.1' end='192.168.121.254'/>
    </dhcp>
  </ip>
</network>

$ virsh net-start --network vagrant-libvirt 
error: Failed to start network vagrant-libvirt
error: internal error: Failed to apply firewall rules /sbin/iptables -w --table filter --insert LIBVIRT_INP --in-interface virbr1 --protocol tcp --destination-port 67 --jump ACCEPT: iptables: No chain/target/match by that name.

When I add a macvtab network, I get the following issue:

$ virsh net-define libvirt-macvtap-def.xml 
Network macvtap-net defined from libvirt-macvtap-def.xml

$ virsh net-dumpxml --network macvtap-net 
<network>
  <name>macvtap-net</name>
  <uuid>1fbd860d-ad78-44c7-951a-996bfb96a5b4</uuid>
  <forward dev='enp0s31f6' mode='bridge'>
    <interface dev='enp0s31f6'/>
  </forward>
</network>

$ virsh net-start --network macvtap-net 
Network macvtap-net started

$ virsh net-list
 Name              State    Autostart   Persistent
----------------------------------------------------
 bridged-network   active   yes         yes
 default           active   yes         yes
 macvtap-net       active   no          yes

$ vagrant up
Bringing machine 'testvm1' up with 'libvirt' provider...
==> testvm1: Creating image (snapshot of base box volume).
==> testvm1: Creating domain with the following settings...
==> testvm1:  -- Name:              libvirt_testvm1
==> testvm1:  -- Description:       Source: /home/mythtv/vagranttest/Vagrantfile
==> testvm1:  -- Domain type:       kvm
==> testvm1:  -- Cpus:              2
==> testvm1:  -- Feature:           acpi
==> testvm1:  -- Feature:           apic
==> testvm1:  -- Feature:           pae
==> testvm1:  -- Clock offset:      localtime
==> testvm1:  -- Memory:            1024M
==> testvm1:  -- Management MAC:    
==> testvm1:  -- Loader:            
==> testvm1:  -- Nvram:             
==> testvm1:  -- Base box:          client_libvirt
==> testvm1:  -- Storage pool:      default
==> testvm1:  -- Image():     /home/mythtv/libvirt_storage/libvirt_testvm1.img, 40G
==> testvm1:  -- Disk driver opts:  cache='default'
==> testvm1:  -- Kernel:            
==> testvm1:  -- Initrd:            
==> testvm1:  -- Graphics Type:     vnc
==> testvm1:  -- Graphics Port:     -1
==> testvm1:  -- Graphics IP:       127.0.0.1
==> testvm1:  -- Graphics Password: Not defined
==> testvm1:  -- Video Type:        cirrus
==> testvm1:  -- Video VRAM:        128
==> testvm1:  -- Video 3D accel:    false
==> testvm1:  -- Sound Type:	
==> testvm1:  -- Keymap:            de-de
==> testvm1:  -- TPM Backend:       passthrough
==> testvm1:  -- TPM Path:          
==> testvm1:  -- INPUT:             type=mouse, bus=ps2
==> testvm1: Creating shared folders metadata...
Traceback (most recent call last):
	59: from /usr/lib64/ruby/gems/2.7.0/gems/vagrant-2.2.18/lib/vagrant/batch_action.rb:86:in `block (2 levels) in run'
	58: from /usr/lib64/ruby/gems/2.7.0/gems/vagrant-2.2.18/lib/vagrant/machine.rb:201:in `action'
	57: from /usr/lib64/ruby/gems/2.7.0/gems/vagrant-2.2.18/lib/vagrant/machine.rb:201:in `call'
	56: from /usr/lib64/ruby/gems/2.7.0/gems/vagrant-2.2.18/lib/vagrant/environment.rb:614:in `lock'
	55: from /usr/lib64/ruby/gems/2.7.0/gems/vagrant-2.2.18/lib/vagrant/machine.rb:215:in `block in action'
	54: from /usr/lib64/ruby/gems/2.7.0/gems/vagrant-2.2.18/lib/vagrant/machine.rb:246:in `action_raw'
	53: from /usr/lib64/ruby/gems/2.7.0/gems/vagrant-2.2.18/lib/vagrant/action/runner.rb:89:in `run'
	52: from /usr/lib64/ruby/gems/2.7.0/gems/vagrant-2.2.18/lib/vagrant/util/busy.rb:19:in `busy'
	51: from /usr/lib64/ruby/gems/2.7.0/gems/vagrant-2.2.18/lib/vagrant/action/runner.rb:89:in `block in run'
	50: from /usr/lib64/ruby/gems/2.7.0/gems/vagrant-2.2.18/lib/vagrant/action/builder.rb:149:in `call'
	49: from /usr/lib64/ruby/gems/2.7.0/gems/vagrant-2.2.18/lib/vagrant/action/warden.rb:48:in `call'
	48: from /usr/lib64/ruby/gems/2.7.0/gems/vagrant-2.2.18/lib/vagrant/action/builtin/config_validate.rb:25:in `call'
	47: from /usr/lib64/ruby/gems/2.7.0/gems/vagrant-2.2.18/lib/vagrant/action/warden.rb:48:in `call'
	46: from /usr/lib64/ruby/gems/2.7.0/gems/vagrant-2.2.18/lib/vagrant/action/builtin/box_check_outdated.rb:36:in `call'
	45: from /usr/lib64/ruby/gems/2.7.0/gems/vagrant-2.2.18/lib/vagrant/action/warden.rb:48:in `call'
	44: from /usr/lib64/ruby/gems/2.7.0/gems/vagrant-2.2.18/lib/vagrant/action/builtin/call.rb:53:in `call'
	43: from /usr/lib64/ruby/gems/2.7.0/gems/vagrant-2.2.18/lib/vagrant/action/runner.rb:89:in `run'
	42: from /usr/lib64/ruby/gems/2.7.0/gems/vagrant-2.2.18/lib/vagrant/util/busy.rb:19:in `busy'
	41: from /usr/lib64/ruby/gems/2.7.0/gems/vagrant-2.2.18/lib/vagrant/action/runner.rb:89:in `block in run'
	40: from /usr/lib64/ruby/gems/2.7.0/gems/vagrant-2.2.18/lib/vagrant/action/builder.rb:149:in `call'
	39: from /usr/lib64/ruby/gems/2.7.0/gems/vagrant-2.2.18/lib/vagrant/action/warden.rb:48:in `call'
	38: from /usr/lib64/ruby/gems/2.7.0/gems/vagrant-2.2.18/lib/vagrant/action/warden.rb:127:in `block in finalize_action'
	37: from /usr/lib64/ruby/gems/2.7.0/gems/vagrant-2.2.18/lib/vagrant/action/warden.rb:48:in `call'
	36: from /home/mythtv/.vagrant.d/gems/2.7.5/gems/vagrant-libvirt-0.7.0/lib/vagrant-libvirt/action/set_name_of_domain.rb:34:in `call'
	35: from /usr/lib64/ruby/gems/2.7.0/gems/vagrant-2.2.18/lib/vagrant/action/warden.rb:48:in `call'
	34: from /home/mythtv/.vagrant.d/gems/2.7.5/gems/vagrant-libvirt-0.7.0/lib/vagrant-libvirt/action/handle_storage_pool.rb:63:in `call'
	33: from /usr/lib64/ruby/gems/2.7.0/gems/vagrant-2.2.18/lib/vagrant/action/warden.rb:48:in `call'
	32: from /usr/lib64/ruby/gems/2.7.0/gems/vagrant-2.2.18/lib/vagrant/action/builtin/handle_box.rb:56:in `call'
	31: from /usr/lib64/ruby/gems/2.7.0/gems/vagrant-2.2.18/lib/vagrant/action/warden.rb:48:in `call'
	30: from /home/mythtv/.vagrant.d/gems/2.7.5/gems/vagrant-libvirt-0.7.0/lib/vagrant-libvirt/action/handle_box_image.rb:123:in `call'
	29: from /usr/lib64/ruby/gems/2.7.0/gems/vagrant-2.2.18/lib/vagrant/action/warden.rb:48:in `call'
	28: from /home/mythtv/.vagrant.d/gems/2.7.5/gems/vagrant-libvirt-0.7.0/lib/vagrant-libvirt/action/create_domain_volume.rb:94:in `call'
	27: from /usr/lib64/ruby/gems/2.7.0/gems/vagrant-2.2.18/lib/vagrant/action/warden.rb:48:in `call'
	26: from /home/mythtv/.vagrant.d/gems/2.7.5/gems/vagrant-libvirt-0.7.0/lib/vagrant-libvirt/action/create_domain.rb:443:in `call'
	25: from /usr/lib64/ruby/gems/2.7.0/gems/vagrant-2.2.18/lib/vagrant/action/warden.rb:48:in `call'
	24: from /usr/lib64/ruby/gems/2.7.0/gems/vagrant-2.2.18/lib/vagrant/action/builtin/provision.rb:80:in `call'
	23: from /usr/lib64/ruby/gems/2.7.0/gems/vagrant-2.2.18/lib/vagrant/action/warden.rb:48:in `call'
	22: from /home/mythtv/.vagrant.d/gems/2.7.5/gems/vagrant-libvirt-0.7.0/lib/vagrant-libvirt/action/prepare_nfs_valid_ids.rb:14:in `call'
	21: from /usr/lib64/ruby/gems/2.7.0/gems/vagrant-2.2.18/lib/vagrant/action/warden.rb:48:in `call'
	20: from /usr/lib64/ruby/gems/2.7.0/gems/vagrant-2.2.18/plugins/synced_folders/nfs/action_cleanup.rb:25:in `call'
	19: from /usr/lib64/ruby/gems/2.7.0/gems/vagrant-2.2.18/lib/vagrant/action/warden.rb:48:in `call'
	18: from /usr/lib64/ruby/gems/2.7.0/gems/vagrant-2.2.18/lib/vagrant/action/builtin/synced_folder_cleanup.rb:28:in `call'
	17: from /usr/lib64/ruby/gems/2.7.0/gems/vagrant-2.2.18/lib/vagrant/action/warden.rb:48:in `call'
	16: from /usr/lib64/ruby/gems/2.7.0/gems/vagrant-2.2.18/lib/vagrant/action/builtin/delayed.rb:19:in `call'
	15: from /usr/lib64/ruby/gems/2.7.0/gems/vagrant-2.2.18/lib/vagrant/action/warden.rb:48:in `call'
	14: from /usr/lib64/ruby/gems/2.7.0/gems/vagrant-2.2.18/lib/vagrant/action/builtin/synced_folders.rb:87:in `call'
	13: from /usr/lib64/ruby/gems/2.7.0/gems/vagrant-2.2.18/lib/vagrant/action/warden.rb:48:in `call'
	12: from /home/mythtv/.vagrant.d/gems/2.7.5/gems/vagrant-libvirt-0.7.0/lib/vagrant-libvirt/action/prepare_nfs_settings.rb:21:in `call'
	11: from /usr/lib64/ruby/gems/2.7.0/gems/vagrant-2.2.18/lib/vagrant/action/warden.rb:48:in `call'
	10: from /home/mythtv/.vagrant.d/gems/2.7.5/gems/vagrant-libvirt-0.7.0/lib/vagrant-libvirt/action/share_folders.rb:22:in `call'
	 9: from /usr/lib64/ruby/gems/2.7.0/gems/vagrant-2.2.18/lib/vagrant/action/warden.rb:48:in `call'
	 8: from /home/mythtv/.vagrant.d/gems/2.7.5/gems/vagrant-libvirt-0.7.0/lib/vagrant-libvirt/action/create_networks.rb:40:in `call'
	 7: from /home/mythtv/.vagrant.d/gems/2.7.5/gems/vagrant-libvirt-0.7.0/lib/vagrant-libvirt/action/create_networks.rb:40:in `synchronize'
	 6: from /home/mythtv/.vagrant.d/gems/2.7.5/gems/vagrant-libvirt-0.7.0/lib/vagrant-libvirt/action/create_networks.rb:43:in `block in call'
	 5: from /home/mythtv/.vagrant.d/gems/2.7.5/gems/vagrant-libvirt-0.7.0/lib/vagrant-libvirt/action/create_networks.rb:43:in `each'
	 4: from /home/mythtv/.vagrant.d/gems/2.7.5/gems/vagrant-libvirt-0.7.0/lib/vagrant-libvirt/action/create_networks.rb:55:in `block (2 levels) in call'
	 3: from /home/mythtv/.vagrant.d/gems/2.7.5/gems/vagrant-libvirt-0.7.0/lib/vagrant-libvirt/util/network_util.rb:148:in `libvirt_networks'
	 2: from /home/mythtv/.vagrant.d/gems/2.7.5/gems/vagrant-libvirt-0.7.0/lib/vagrant-libvirt/util/network_util.rb:148:in `each'
	 1: from /home/mythtv/.vagrant.d/gems/2.7.5/gems/vagrant-libvirt-0.7.0/lib/vagrant-libvirt/util/network_util.rb:179:in `block in libvirt_networks'
/home/mythtv/.vagrant.d/gems/2.7.5/gems/vagrant-libvirt-0.7.0/lib/vagrant-libvirt/util/network_util.rb:179:in `bridge_name': Call to virNetworkGetBridgeName failed: internal error: network 'macvtap-net' does not have a bridge name. (Libvirt::Error)

I can use the macvtab network if I start VMs via virt-manager and if I add a macvlan device to my host, I can even get host <-> guest communication to work.

Could anyone help to sort this out - I am completely stuck and out of ideas.

Cheers and thanks for any suggestions,
Bjoern

[electrofelix]

The error message

error: Failed to start network vagrant-libvirt
error: internal error: Failed to apply firewall rules /sbin/iptables -w --table filter --insert LIBVIRT_INP --in-interface virbr1 --protocol tcp --destination-port 67 --jump ACCEPT: iptables: No chain/target/match by that name.

Suggests an issue between libvirt and iptables, something around the internal setup is incorrect.

https://bugzilla.redhat.com/show_bug.cgi?id=1683174 hints that this can be down to the kernel and potentially what modules are enabled.

I can't see why that would be the case here as the default network is running and it appears to be using similar settings for the most part.

Probably worth debugging why you can't start the network definition for vagrant-libvirt with your distro support first though. Once you can start that via virsh net-start, the machine bring up should work just fine.

I'm not sure that we support handling macvtap networks correctly yet, the network code is a bit knarly and definitely lacking tests so I can't be sure either way.

[bolausson]

Hej and thanks for the quick reply!

LIBVIRT_INP chain was missing.
I had to restart libvirtd and then I could run "vagrant up" successfully.

The culprit is:
libvirtd starts and crates the chain, after that shorewall starts and clears everything.

I was able to reproduce this.

• Stop shorewall, restart libvirtd, vagrant up --> success
• Stop shorewall, restart libvirtd, start shorewall, vagrant up --> failed

And the init startup sequence is indeed to first start libvirtd and some time after that shorewall is started - stupid problem which luckily can be resolved fairly easy.

I don't understand one comment from you though:

I'm not sure that we support handling macvtap networks correctly yet, the network code is a bit knarly and definitely lacking tests so I can't be sure either way.

if I look at the documentation, it says (https://github.com/vagrant-libvirt/vagrant-libvirt#networks):

Public Network interfaces are currently implemented using the macvtap driver. The macvtap driver is only available with the Linux Kernel version >= 2.6.24. See the following Libvirt documentation for the details of the macvtap usage.

Which suggest that macvtap/macvlan is the way to go, no?

Cheers,
Bjoern

[electrofelix]

LIBVIRT_INP chain was missing. I had to restart libvirtd and then I could run "vagrant up" successfully.

The culprit is: libvirtd starts and crates the chain, after that shorewall starts and clears everything.

I was able to reproduce this.

* Stop shorewall, restart libvirtd, vagrant up --> success

* Stop shorewall, restart libvirtd, start shorewall, vagrant up --> failed

And the init startup sequence is indeed to first start libvirtd and some time after that shorewall is started - stupid problem which luckily can be resolved fairly easy.

That's good to know, even if a little frustrating to have bumped into it.

I don't understand one comment from you though:

I'm not sure that we support handling macvtap networks correctly yet, the network code is a bit knarly and definitely lacking tests so I can't be sure either way.

if I look at the documentation, it says (https://github.com/vagrant-libvirt/vagrant-libvirt#networks):

Public Network interfaces are currently implemented using the macvtap driver. The macvtap driver is only available with the Linux Kernel version >= 2.6.24. See the following Libvirt documentation for the details of the macvtap usage.

Which suggest that macvtap/macvlan is the way to go, no?

The issue you commented on originally before opening the discussion (#599) has the error:
Call to virNetworkGetBridgeName failed: internal error: network 'macvtap-net' does not have a bridge name. (Libvirt::Error)

Based on previous comments suggest that while particular definitions of macvtap network definitions will work, the existing code doesn't support all definitions that are supported by libvirt, which can result in unexpected results depending on how the libvirt network was created.

[bolausson]

Thanks a lot for the explanation!