SR-IOV VF NIC (VM) can't talk to the host

Operating Systems & Open Source Virtualization
1

I have a network configuration on Ubuntu that can be reduced to this:

network:
  version: 2

  ethernets:
    intel-ocp-1:
      match:
        macaddress: redacted
      set-name: intel-ocp-1
      optional: true

  bridges:
    lan:
      interfaces:
        - intel-ocp-1
      addresses:
        - 192.168.1.99/24
      routes:
        - to: default
          via: 192.168.1.1
          on-link: true
      nameservers:
        addresses:
          - 192.168.1.1
      parameters:
        stp: true
      optional: true

intel-ocp-1 is an Intel 82599ES NIC where I have created virtual functions and passed through to the VM.

The issue is that VM (192.168.1.1) can’t talk to the host (192.168.1.99) even though to the best of my understanding it should be able to.

Desktop (192.168.1.2) connected to intel-ocp-1 SFP+ port can talk both to the host and to the VM just fine, it is just VM and host can’t talk to each other, which is confusing.

Is there some kind of configuration I might be missing here?

2

I don’t know about Intel, but Chelsio NICs don’t expose VF to PF (and vice versa) by default:

By default, the VFs (in VM) can not communicate with PFs (on Host). To enable this
communication, set ethtool private flag port_tx_vm_wr for PF interface (on Host).

[root@host~]# ethtool --set-priv-flags ethX port_tx_vm_wr on

Perhaps you need to find the right flags for your controller.

2 Likes
3

From my experience, whenever you include vf or pf of the interface into a bridge, it will break.
All vfs and the pf of the same interface are bridged internally in 82599ES. They can talk with each other out of the box.
When you tell linux to create a bridge out of them, the internal bridge is disabled implicitily. That creates all kinds of weird behaviors.
That’s just based on my observation. I might be wrong here.

4

So after some digging using keywords mentioned above (thanks!) I found in multiple places:

That this should work since kernel 3.8.5:

bridge fdb add 26:03:0b:7e:68:01 dev intel-ocp-1

Where 26:03:0b:7e:68:01 is MAC address of the virtual function that VM uses and intel-ocp-1 corresponding physical interface.

However this doesn’t work, it even prevents my desktop from communicating with VM, which is confusing.

I tried to assigning MAC address to virtual function on the host before booting VM (though I suspect it is done by libvirt during VM startup anyway and not necessary) as well as disabling spoof checking for virtual function (which should not be necessary either since MAC address in VM was not modified).

Someone wrote they abandoned Intel NIC and switched to NIC from Broadcom where it worked fine.

I am at dead end again it seems.

1 Like