Think of NICs with VF support having an L2 bridge behind each port, but before the PF and VF interfaces on that NIC.
port 0 -> L2 bridge -> | PF
| VF 0
| VF 1
| VF n
port 1 -> L2 bridge -> | PF
| VF 0
| VF 1
| VF n
(edit: found a better diagram from DPDK : https://doc.dpdk.org/guides/_images/perf_benchmark.png)
When VF drivers start, they request a MAC address from the PF driver, which adds it to the bridge, so that VF starts receiving traffic to that destination MAC address. From the point of view of devices behind the associated port, the PF/VF interfaces are behind an L2 switch (effectively, that is the case).
The bridge will forward all broadcast traffic, so all PF/VF interfaces behind a port will see broadcast traffic and subscribed multicast traffic.
Like a real bridge, any PF/VF will only see traffic received for it’s MAC address, so that leads into addressing question 1:
To forward traffic between ports and to send ARPs out on both ports you need a bridge of either two PF’s or two VF’s (in a VM), doesn’t make a difference, BUT the IP config will only work in the host or the VM.
You then need another bridge in every VM that also needs to send ARPs out on both ports, but disable forwarding broadcast packets (every bridge will duplicate them), i.e. with ebtables:
ebtables -A FORWARD -i eth0 -o eth1 -d ff:ff:ff:ff:ff:ff -j DROP
ebtables -A FORWARD -i eth1 -o eth0 -d ff:ff:ff:ff:ff:ff -j DROP
# gets more complicated with multicast ...
Question 2: When you enable bonding, the switch will send packets on either port regardless of the destination MAC (except with balance-xor with L2 hash policy), according to the balance policy (unless you have some very clever SDN switch…), so as soon as bonding is enabled, you also need to handle it in any VMs using VFs (and pass a VF for every port in the aggregation group).
When you start the bond driver in a VM, it’ll set the MAC address of the bond interface and both the VF’s to the same address, which you can see in
ip link output in the host.
To make it as simple as possible, I’d use passive LACP or manual balancing like balance-xor with L3+L4 hash policy on the PF’s and the same config inside VM’s using VF’s.
Another option, is to ignore all that, create a bridge on the host, and add VM’s as tap devices to the bridge, using virtio-net devices - in my experience the difference between SR-IOV and virtio-net is quite small, but do benchmark