Reflect NICs defined in the VirtualMachine resource to the corresponding VirtualMachineNetworkConfig resource

[starbops]

IMPORTANT: Please do not create a Pull Request without creating an issue first.

Problem:

The Managed DHCP add-on cannot catch any changes in the virtual machine's MAC address updates. The only way to let the add-on serve IP addresses for the updated MAC addresses is to remove the corresponding VirtualMachineNetworkConfig resource and the controller will automatically create a new one with updated MAC addresses.

Solution:

Introduce a new condition in VirtualMachineNetworkConfig—InSynced—which represents the synchronization between the VirtualMachine and VirtualMachineNetworkConfig resources, especially for the MAC addresses. The vm-controller will flag the condition and reflect the MAC addresses to the vmnetcfg resource. The vmnetcfg-controller will catch the changes, try to remove any stale records, and then update the IPPool resource. After that, InSynced will become True. This unblocks the regular allocation procedure, so the new MAC addresses (if any) will get new IP addresses from the IPPool resource.

Related Issue:

harvester/harvester#7046

Test plan:

1. Prepare a Harvester cluster of version v1.4.1
2. Install the harvester-vm-dhcp-controller chart (this PR includes CRD changes and will later update to the harvester/charts repo)

git clone https://github.com/starbops/harvester-vm-dhcp-controller.git -b enh-7046
cd harvester-vm-dhcp-controller
cat <<EOF | tee -a values.yaml
image:
  repository: starbops/harvester-vm-dhcp-controller
  tag: enh-7046-head
agent:
  image:
    repository: starbops/harvester-vm-dhcp-agent
    tag: enh-7046-head
webhook:
  image:
    repository: starbops/harvester-vm-dhcp-webhook
    tag: enh-7046-head
EOF
helm upgrade --install harvester-vm-dhcp-controller ./chart \
--namespace=harvester-system \
--create-namespace \
-f values.yaml

3. Create two virtual machine networks (NADs)
4. Create two IPPools for the two networks
5. Create a virtual machine attaching to the first network
6. Log in to the virtual machine and check the IP address. It should match the record in VirtualMachineNetworkConfig (vmnetcfg) and the first IPPool. Jot down the IP address for later comparison.
7. Update the virtual machine; Create a new NIC and attach it to the second network. Restart the virtual machine.
8. Observe the vmnetcfg; it should reflect the new MAC address, and a new IP address should be acquired from the second pool. Also, check the second IPPool, which should have the same updates. However, the first IP/MAC address should remain the same.
9. Log in to the virtual machine and check the IP addresses. They should match the records in vmnetcfg and the IPPools. Jot down the IP address for later comparison.
10. Update the second NIC's MAC address of the virtual machine. Restart the virtual machine.
11. Observe the vmnetcfg; it should reflect the updated MAC address, and a new IP address might be acquired from the pool (depends on the remaining IP addresses available in the pool). Also, check the IPPool, which should have the same updates. It might obtain the same IP address as before; it could happen, and it's totally normal. Keep an eye on the first IP IP/MAC address, they should remain the same as before.
12. Log in to the virtual machine and check if the IP addresses match the record in vmnetcfg and the IPPools.
13. Remove the first NIC of the virtual machine and reboot the virtual machine.
14. Observe the vmnetcfg; it should contain only the second IP/MAC address. Also, check the first IPPool, the IP address should be given back to the pool.
15. Log in to the virtual machine and check if the IP address matches the record in vmnetcfg and the IPPool.