Edit: Resolved the issue by going back to Intel version 25 of the drivers, instead of 26/27 that is currently available on the website. I figured since my cards are a decade old, maybe something changed in the driver. Sure enough, older driver installed, card initialized just fine!
Built a new to me virtualization system to mess around with, based on a Gigabyte X99-UD4P board, Xeon 2683v4 cpu, 64gb ram and Intel x520-da1 network card.
I have setup the network card to create 14 virtual functions, using SR-IOV.
When I passthrough a VF to a Linux guest, it works great. Guest is able to talk on my 10g network as if it were directly attached. Assign a VF to a Windows guest, and it says the cable is disconnected.
In the kernel log, I get a ton of messages from ixgbe module Unhandled message 00000010, over and over again while the Windows guest is running and has the Intel driver loaded. No such message comes up for Linux guests.
Similar card, x520-da2 in my other virtualization host, a Ryzen system, works as expected with Linux and Windows guests. Both hosts are running Fedora with the latest kernel and libvirt versions.
1 Like
This is the XML I type in, and libvirt expands it, shown in the second example. I am using SR-IOV pooling, so a virtual function is picked at boot, rather than statically assigned. However, I have also tried the old fashioned method of a static assigned PCI device.
Here my sr-iov pool is just named passthrough
<interface type='network'>
<mac address='52:54:00:f9:c2:bf'/>
<source network='passthrough'>
</interface>
expands automatically to this at boot (while machine is running)
<interface type='hostdev' managed='yes'>
<mac address='52:54:00:f1:22:b8'/>
<driver name='vfio'/>
<source>
<address type='pci' domain='0x0000' bus='0x02' slot='0x10' function='0x0'/>
</source>
<model type='virtio'/>
<alias name='hostdev0'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</interface>
full xml here
<domain type='kvm'>
<name>win10</name>
<uuid>0820156e-41a8-4be0-98f9-3298498fc3ae</uuid>
<metadata>
<libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
<libosinfo:os id="http://microsoft.com/win/10"/>
</libosinfo:libosinfo>
</metadata>
<memory unit='KiB'>8388608</memory>
<currentMemory unit='KiB'>8388608</currentMemory>
<vcpu placement='static'>4</vcpu>
<os>
<type arch='x86_64' machine='pc-q35-6.1'>hvm</type>
<loader readonly='yes' type='pflash'>/usr/share/edk2/ovmf/OVMF_CODE.fd</loader>
<nvram>/var/lib/libvirt/qemu/nvram/win10_VARS.fd</nvram>
<bootmenu enable='no'/>
</os>
<features>
<acpi/>
<apic/>
<hyperv>
<relaxed state='on'/>
<vapic state='on'/>
<spinlocks state='on' retries='8191'/>
</hyperv>
<vmport state='off'/>
</features>
<cpu mode='host-model' check='partial'/>
<clock offset='localtime'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
<timer name='hypervclock' present='yes'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled='no'/>
<suspend-to-disk enabled='no'/>
</pm>
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='none' io='native'/>
<source dev='/dev/zvol/ssd/win10'/>
<target dev='sda' bus='sata'/>
<boot order='1'/>
<address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>
<disk type='file' device='cdrom'>
<driver name='qemu' type='raw'/>
<source file='/mnt/storage/iso/Windows21H1.iso'/>
<target dev='sdb' bus='sata'/>
<readonly/>
<address type='drive' controller='0' bus='0' target='0' unit='1'/>
</disk>
<disk type='file' device='cdrom'>
<driver name='qemu' type='raw'/>
<source file='/usr/share/virtio-win/virtio-win-0.1.171.iso'/>
<target dev='sdc' bus='sata'/>
<readonly/>
<address type='drive' controller='0' bus='0' target='0' unit='2'/>
</disk>
<controller type='usb' index='0' model='qemu-xhci' ports='15'>
<address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
</controller>
<controller type='sata' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
</controller>
<controller type='pci' index='0' model='pcie-root'/>
<controller type='pci' index='1' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='1' port='0x10'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
</controller>
<controller type='pci' index='2' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='2' port='0x11'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
</controller>
<controller type='pci' index='3' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='3' port='0x12'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
</controller>
<controller type='pci' index='4' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='4' port='0x13'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
</controller>
<controller type='pci' index='5' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='5' port='0x14'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
</controller>
<controller type='virtio-serial' index='0'>
<address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
</controller>
<interface type='network'>
<mac address='52:54:00:f1:22:b8'/>
<source network='passthrough'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</interface>
<serial type='pty'>
<target type='isa-serial' port='0'>
<model name='isa-serial'/>
</target>
</serial>
<console type='pty'>
<target type='serial' port='0'/>
</console>
<channel type='spicevmc'>
<target type='virtio' name='com.redhat.spice.0'/>
<address type='virtio-serial' controller='0' bus='0' port='1'/>
</channel>
<input type='tablet' bus='usb'>
<address type='usb' bus='0' port='1'/>
</input>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='spice' autoport='yes' listen='0.0.0.0'>
<listen type='address' address='0.0.0.0'/>
<image compression='off'/>
</graphics>
<sound model='ich9'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1b' function='0x0'/>
</sound>
<audio id='1' type='spice'/>
<video>
<model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
</video>
<redirdev bus='usb' type='spicevmc'>
<address type='usb' bus='0' port='2'/>
</redirdev>
<redirdev bus='usb' type='spicevmc'>
<address type='usb' bus='0' port='3'/>
</redirdev>
<memballoon model='virtio'>
<address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
</memballoon>
</devices>
</domain>
here is the XML for the sr-iov pool
# virsh net-dumpxml passthrough
<network connections='3'>
<name>passthrough</name>
<uuid>e741f587-21b2-47a7-9529-2de4e255cbdb</uuid>
<forward mode='hostdev' managed='yes'>
<pf dev='ens6'/>
<address type='pci' domain='0x0000' bus='0x02' slot='0x10' function='0x0'/>
<address type='pci' domain='0x0000' bus='0x02' slot='0x10' function='0x2'/>
<address type='pci' domain='0x0000' bus='0x02' slot='0x10' function='0x4'/>
<address type='pci' domain='0x0000' bus='0x02' slot='0x10' function='0x6'/>
<address type='pci' domain='0x0000' bus='0x02' slot='0x11' function='0x0'/>
<address type='pci' domain='0x0000' bus='0x02' slot='0x11' function='0x2'/>
<address type='pci' domain='0x0000' bus='0x02' slot='0x11' function='0x4'/>
<address type='pci' domain='0x0000' bus='0x02' slot='0x11' function='0x6'/>
<address type='pci' domain='0x0000' bus='0x02' slot='0x12' function='0x0'/>
<address type='pci' domain='0x0000' bus='0x02' slot='0x12' function='0x2'/>
<address type='pci' domain='0x0000' bus='0x02' slot='0x12' function='0x4'/>
<address type='pci' domain='0x0000' bus='0x02' slot='0x12' function='0x6'/>
<address type='pci' domain='0x0000' bus='0x02' slot='0x13' function='0x0'/>
<address type='pci' domain='0x0000' bus='0x02' slot='0x13' function='0x2'/>
</forward>
</network>
i am also using an intel x500 series SR_IOV on a windows VM. but i dont do any of that. i just pass through the virtual function to the VM like its any other PCIe device to pass through. and it works flawlessly. i dont specify any network block in my VM, just the hostdev to passthrough the virtual function.
Aye, the setup works fine on my x399 Ryzen system. Just having trouble with the x99 Xeon system, was wondering if anyone had encountered vfio quirks.
Here’s info on a sr-iov pool. Saves a lot of frustration for remembering what VF is assigned to what machine on a busy system (I have over a dozen active VMs on the Ryzen, each using their own 10g network card passthrough)https://wiki.libvirt.org/page/Networking#Assignment_from_a_pool_of_SRIOV_VFs_in_a_libvirt_.3Cnetwork.3E_definition
1 Like
virsh wont let you define a VM that uses a device already passed through to another VM.
my setup is far simpler
ACTION=="add", SUBSYSTEM=="net", ENV{ID_NET_DRIVER}=="ixgbe", ATTR{device/sriov_numvfs}="4"
and in my VM’s XML
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x0a' slot='0x10' function='0x6'/>
</source>
<address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</hostdev>
this makes my VM use the 6th virtual function.
I can confirm the OP’s update that this is a driver issue. I updated the Intel driver in my Windows VM (to v26.7) and got “network cable unplugged” and these errors in the host logs. Downgrading to v26.4 (neither .5 nor .6 were listed) resolved.
1 Like
Altkey
June 29, 2022, 12:05am
8
Just wanted to confirm that this solution worked for me on a Windows 2022 VM with a NIC that RHEL 9.0 sees as Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (Ethernet Server Adapter X520-1)
Had to uninstall the newer drivers then BAM , worked.
1 Like