Help me understand bridging/bonding with SR-IOV

Hello people,
through some luck at work, i was able to get 3 dual port Intel X710 NICs with RJ45 Ports. As I don’t have a 10G Switch just yet, p2p connection it is:
Backup <--> Server <--> Desktop

The server is connected to both other host and I created a bridge for the two Ports, so all the devices can reach each other no problem.
Now I wanted to play with SR-IOV as these cards support it.
Question 1:

  • How do I do bridging on the Host (Server, which is where my VMs are)? Do I use the physical interfaces ? When I do this and pass a VF to a VM, only the the other Host connected to this physical port is reachable. Do I have to pass through two VF to the VM and create a Bridge there to?
    Question 2:
  • How would I go about bonding. Assume, I will have a 10G Switch at some point and want to make a bond of these two interfaces be it only for the Host. Use a VF? And if I’d also want link reundancy in the VM i’d have to passthrough one VF from each port of the NIC to the VM and create an bond there?

I hope this is clear, otherwise feel free to let me clarify.
Cheers!

Anybody?

you kinda left out a lot of detail…

Like what OS?

I cant help you with SR-IOV

Think of NICs with VF support having an L2 bridge behind each port, but before the PF and VF interfaces on that NIC.

port 0 -> L2 bridge -> | PF
                       | VF 0
                       | VF 1
                       | VF n

port 1 -> L2 bridge -> | PF
                       | VF 0
                       | VF 1
                       | VF n

(edit: found a better diagram from DPDK : https://doc.dpdk.org/guides/_images/perf_benchmark.png)

When VF drivers start, they request a MAC address from the PF driver, which adds it to the bridge, so that VF starts receiving traffic to that destination MAC address. From the point of view of devices behind the associated port, the PF/VF interfaces are behind an L2 switch (effectively, that is the case).

The bridge will forward all broadcast traffic, so all PF/VF interfaces behind a port will see broadcast traffic and subscribed multicast traffic.

Like a real bridge, any PF/VF will only see traffic received for it’s MAC address, so that leads into addressing question 1:

To forward traffic between ports and to send ARPs out on both ports you need a bridge of either two PF’s or two VF’s (in a VM), doesn’t make a difference, BUT the IP config will only work in the host or the VM.

You then need another bridge in every VM that also needs to send ARPs out on both ports, but disable forwarding broadcast packets (every bridge will duplicate them), i.e. with ebtables:

ebtables -A FORWARD -i eth0 -o eth1 -d ff:ff:ff:ff:ff:ff -j DROP
ebtables -A FORWARD -i eth1 -o eth0 -d ff:ff:ff:ff:ff:ff -j DROP
# gets more complicated with multicast ...

Question 2: When you enable bonding, the switch will send packets on either port regardless of the destination MAC (except with balance-xor with L2 hash policy), according to the balance policy (unless you have some very clever SDN switch…), so as soon as bonding is enabled, you also need to handle it in any VMs using VFs (and pass a VF for every port in the aggregation group).

When you start the bond driver in a VM, it’ll set the MAC address of the bond interface and both the VF’s to the same address, which you can see in ip link output in the host.

To make it as simple as possible, I’d use passive LACP or manual balancing like balance-xor with L3+L4 hash policy on the PF’s and the same config inside VM’s using VF’s.

Another option, is to ignore all that, create a bridge on the host, and add VM’s as tap devices to the bridge, using virtio-net devices - in my experience the difference between SR-IOV and virtio-net is quite small, but do benchmark :slight_smile:

1 Like