VFIO in 2019 – Fedora Workstation (General Guide though) [Branch, Draft]

Operating Systems & Open Source Virtualization
,
61

Hi all,

sorry for the late replay, I only get to do this stuff on weekends right now.

here is CPU-Z and NVIDIA side-by-side. I see the same thing as @chase9 with CPU-Z–nothing.

gpu-passthrough-with-looking-glass-2019-11-02_15-24-09
gpu-passthrough-with-looking-glass-2019-11-02_15-24-09.png1499×753 218 KB

Here is my VM config. I did not write it myself. I installed windows 10 and then tweaked the file for Code 43 and that was about it. Hence, this may not be the best setup–the first one I was able to get working really, so input is welcome.

@BansheeHero, I don’t know much about huge pages but those notes would be great.

I think that I will try to remove my LSI card and see what happens here.

> user0]# cat /etc/libvirt/qemu/win10.xml 
> <!--
> WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
> OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
>   virsh edit win10
> or other application using the libvirt API.
> -->
> 
> <domain type='kvm'>
>   <name>win10</name>
>   <uuid>c27c0a73-6ac5-4ee5-9dc0-cc8b1a913475</uuid>
>   <metadata>
>     <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
>       <libosinfo:os id="http://microsoft.com/win/10"/>
>     </libosinfo:libosinfo>
>   </metadata>
>   <memory unit='KiB'>8388608</memory>
>   <currentMemory unit='KiB'>8388608</currentMemory>
>   <vcpu placement='static'>8</vcpu>
>   <os>
>     <type arch='x86_64' machine='pc-q35-3.1'>hvm</type>
>     <loader readonly='yes' type='pflash'>/usr/share/edk2/ovmf/OVMF_CODE.fd</loader>
>     <nvram>/var/lib/libvirt/qemu/nvram/win10_VARS.fd</nvram>
>     <boot dev='hd'/>
>   </os>
>   <features>
>     <acpi/>
>     <apic/>
>     <hyperv>
>       <relaxed state='on'/>
>       <vapic state='on'/>
>       <spinlocks state='on' retries='8191'/>
>       <vendor_id state='on' value='1234567890ab'/>
>     </hyperv>
>     <kvm>
>       <hidden state='on'/>
>     </kvm>
>     <vmport state='off'/>
>   </features>
>   <cpu mode='host-model' check='partial'>
>     <model fallback='allow'/>
>     <topology sockets='1' cores='4' threads='2'/>
>   </cpu>
>   <clock offset='localtime'>
>     <timer name='rtc' tickpolicy='catchup'/>
>     <timer name='pit' tickpolicy='delay'/>
>     <timer name='hpet' present='no'/>
>     <timer name='hypervclock' present='yes'/>
>   </clock>
>   <on_poweroff>destroy</on_poweroff>
>   <on_reboot>restart</on_reboot>
>   <on_crash>destroy</on_crash>
>   <pm>
>     <suspend-to-mem enabled='no'/>
>     <suspend-to-disk enabled='no'/>
>   </pm>
>   <devices>
>     <emulator>/usr/bin/qemu-system-x86_64</emulator>
>     <disk type='block' device='disk'>
>       <driver name='qemu' type='raw' cache='none' io='native'/>
>       <source dev='/dev/sdb'/>
>       <target dev='vda' bus='virtio'/>
>       <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
>     </disk>
>     <disk type='file' device='cdrom'>
>       <driver name='qemu' type='raw'/>
>       <source file='/var/lib/libvirt/images/en_windows_10_enterprise_2016_ltsb_x64_dvd_9059483.iso'/>
>       <target dev='sdb' bus='sata'/>
>       <readonly/>
>       <address type='drive' controller='0' bus='0' target='0' unit='1'/>
>     </disk>
>     <disk type='file' device='cdrom'>
>       <driver name='qemu' type='raw'/>
>       <source file='/var/lib/libvirt/images/virtio-win-0.1.141.iso'/>
>       <target dev='sdc' bus='sata'/>
>       <readonly/>
>       <address type='drive' controller='0' bus='0' target='0' unit='2'/>
>     </disk>
>     <controller type='usb' index='0' model='qemu-xhci' ports='15'>
>       <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
>     </controller>
>     <controller type='sata' index='0'>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
>     </controller>
>     <controller type='pci' index='0' model='pcie-root'/>
>     <controller type='pci' index='1' model='pcie-root-port'>
>       <model name='pcie-root-port'/>
>       <target chassis='1' port='0x8'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
>     </controller>
>     <controller type='pci' index='2' model='pcie-root-port'>
>       <model name='pcie-root-port'/>
>       <target chassis='2' port='0x9'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
>     </controller>
>     <controller type='pci' index='3' model='pcie-root-port'>
>       <model name='pcie-root-port'/>
>       <target chassis='3' port='0xa'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
>     </controller>
>     <controller type='pci' index='4' model='pcie-root-port'>
>       <model name='pcie-root-port'/>
>       <target chassis='4' port='0xb'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/>
>     </controller>
>     <controller type='pci' index='5' model='pcie-root-port'>
>       <model name='pcie-root-port'/>
>       <target chassis='5' port='0xc'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/>
>     </controller>
>     <controller type='pci' index='6' model='pcie-root-port'>
>       <model name='pcie-root-port'/>
>       <target chassis='6' port='0xd'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/>
>     </controller>
>     <controller type='pci' index='7' model='pcie-root-port'>
>       <model name='pcie-root-port'/>
>       <target chassis='7' port='0xe'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x6'/>
>     </controller>
>     <controller type='pci' index='8' model='pcie-root-port'>
>       <model name='pcie-root-port'/>
>       <target chassis='8' port='0xf'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x7'/>
>     </controller>
>     <controller type='pci' index='9' model='pcie-to-pci-bridge'>
>       <model name='pcie-pci-bridge'/>
>       <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
>     </controller>
>     <interface type='network'>
>       <mac address='52:54:00:f1:b9:45'/>
>       <source network='default'/>
>       <model type='e1000e'/>
>       <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
>     </interface>
>     <serial type='pty'>
>       <target type='isa-serial' port='0'>
>         <model name='isa-serial'/>
>       </target>
>     </serial>
>     <console type='pty'>
>       <target type='serial' port='0'/>
>     </console>
>     <input type='mouse' bus='ps2'/>
>     <input type='keyboard' bus='ps2'/>
>     <hostdev mode='subsystem' type='usb' managed='yes'>
>       <source>
>         <vendor id='0x046d'/>
>         <product id='0xc31c'/>
>       </source>
>       <address type='usb' bus='0' port='1'/>
>     </hostdev>
>     <hostdev mode='subsystem' type='usb' managed='yes'>
>       <source>
>         <vendor id='0x046d'/>
>         <product id='0xc043'/>
>       </source>
>       <address type='usb' bus='0' port='2'/>
>     </hostdev>
>     <hostdev mode='subsystem' type='pci' managed='yes'>
>       <source>
>         <address domain='0x0000' bus='0x0f' slot='0x00' function='0x0'/>
>       </source>
>       <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
>     </hostdev>
>     <hostdev mode='subsystem' type='pci' managed='yes'>
>       <source>
>         <address domain='0x0000' bus='0x0f' slot='0x00' function='0x1'/>
>       </source>
>       <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
>     </hostdev>
>     <redirdev bus='usb' type='spicevmc'>
>       <address type='usb' bus='0' port='3'/>
>     </redirdev>
>     <redirdev bus='usb' type='spicevmc'>
>       <address type='usb' bus='0' port='4'/>
>     </redirdev>
>     <memballoon model='virtio'>
>       <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
>     </memballoon>
>     <shmem name='looking-glass'>
>       <model type='ivshmem-plain'/>
>       <size unit='M'>64</size>
>       <address type='pci' domain='0x0000' bus='0x09' slot='0x01' function='0x0'/>
>     </shmem>
>   </devices>
> </domain>

win10.txt (7.0 KB)

62

So, removing the LSI card did not do anything in as far as seeing the x1 in the windows vm.

image
image.png1207×738 157 KB

63

I just noticed that I was using the wrong -Z. GPU-Z shows what I would expect: PCI Gen 3 x8; nVIDIA still shows x1. Not sure what to make of this. One of these must be wrong.

@SgtAwesomesauce are you running Threadripper and using ZFS for this? I am curious because my next step would be to get the ZFS setup.

image
image.png900×518 87.5 KB

64

yes

No. ZFS is storage only, not VMs on my system.

Nvidia tends to be the one that’s wrong.

If you want to be 100% sure, you could try this tool:

https://forums.evga.com/PCIE-bandwidth-test-cuda-m1972266.aspx

Or, you could just see if you’re getting poor performance from your GPU.

Yeah, without hugepages, you’re going to get bad performance or stuttering in games.

65

Thank you @SgtAwesomesauce the bandwidth seems alright. I guess nVIDIA is misreporting things when in guest. I tried Mass Effect 2 and that seems to be OK. I am surprised that I was even able to record with ShadowPlay. Had some frame drops to 54 FPS when recording but this is running on GTX 760 with 4GB of RAM and on an old Intel 250 GB SSD so probably fine …

Now to figure out this hugepages and ZFS.

image

2 Likes
66

Any one got this working on Fedora 31… i have tried every options and nothing.

67

I’ve been running successfully on Fedora 31. Could you elaborate more on what’s happening?

68

i am also having troubble on fedora 31. i have gone thru multiple guides but i cannot seem to get the drivers to load for the card it always just loads the radeon drivers instead. but i am very new to linux so it might be something im doing wrong from lack of base linux skill.

69

I reinstalled it with F31 and it worked mostly the same. Let us know what seems to be the problem.

71

try adding rd.driver.pre=vfio-pci to grub cmdline

sudo vi /etc/sysconfig/grub
1 Like
72

Thanks, using Fedora 31, this was key to getting VFIO-PCI to grab my GPU.

73

Thanx but still no luck. ive edited the file and put it in that line, but in the guide it says to reinstall kernel after editin that file and then to run a command that reads: grub2-editenv list. what does this command do? when i run it i get the output: [love@localhost ~]$ sudo grub2-editenv list
[sudo] password for love:
saved_entry=b7aa99183f38428a92e0dd21b864fa36-5.3.16-300.fc31.x86_64
menu_auto_hide=1
boot_success=0
kernelopts=root=/dev/mapper/fedora_localhost–live-root ro resume=/dev/mapper/fedora_localhost–live-swap rd.lvm.lv=fedora_localhost-live/root rd.lvm.lv=fedora_localhost-live/swap rhgb quiet
boot_indeterminate=0

74

The grub2-editenv list command lists out the active grub boot parameters I believe. However, I don’t know if it is reliable as I did not use it on my set. Reason being I believe that my grub/EFI/boot is borked due to re-installing Fedora on the same set of md/lvm device. So no changes I make to grub config ever takes effect except when I update the kernel. That said, there was I believe some bug about grub2 config on Fedora, which is why the instructions says to reinstall kernel because otherwise grub2-mkconfig should had suffice.

So for me, I simply did a dnf update to get the latest kernel. You might want to try downgrading your kernel, make the changes to boot parameters then upgrading the kernel.

Additional note is that I did not use the pci override script method. Since I had two different GPU, I went with the device ID approach.

75

Would it be possible for you to include instructions for Fedora Silverblue in the guide? Thanks!

76

Thanx for your swift reply!
Ah okay, seems like its not registering the changes i made then.
im sorry for being a complete noob, but how would i go about downgrading the kernel? and to what version? do i have earlier versions localy or do i get them from the repo?

77

If you haven’t updated your system recently, try making your changes then do: dnf upgrade kernel
Otherwise try dnf reinstall kernel as suggested by the guide. If neither of these work then dnf downgrade kernel as last resort because downgrading may cause other things to break or get downgraded due to dependencies.

78

It seems to be like he was trying to go for the simplest way of regenerating your GRUB config, which is why it says to reinstall your kernel. Here’s how I do it and haven’t had a problem yet:

  1. Edit /etc/default/grub
  2. Regenerate grub with grub2-mkconfig -o /etc/grub2-efi.cfg
79

thanx so much for your help guys, really appreciate it. allright so now it regenerated and now the stuff seems to be there when i run grub2-editenv list, so that is some progress.
But it still loads amd drivers when i reboot, so verry stubborn =(. I might have to give up and dual boot atleast for a while.

80

Never hurts to take a break and come back when you’re feeling fresh :slight_smile:

If I could give you a tip, I’d say triple check all of your config is correct by running through everything again. I was banging my head against the wall until I realized I had confused some dashes and periods.

81

Yeah i absolutely will, i wish i knew more about how to check that everything is loading correctly and in the right order.
Oh well i think ill learn more and more now that i have finally moved to linux on the desktop =D.
Thank you for the help! means the world with a friendly community when youre a beginner!