VFIO in 2019 – Fedora Workstation (General Guide though) [Branch, Draft]

I’m a bit confused about why you talk about 000:41 and use the 41 space in your script when it appears your card is entirely in 42?

1 Like

Reading that Redhat paper on huge pages makes my head hurt. Why in 2019 would anyone need to manually provision and then mount and initialize memory pages for a virtual machine?

Did I miss something?

That is the original article and he has 2 RTX cards. Script output shows the card he wants to use for Linux. The other one will be used for VFIO.

I am still working on that part in Fedora, I prefer to use the old way and struggle implementing his.

1 Like

There is performance gain for some systems. I do want to include both ways in the article. Personally I was ok waiting a bit for kernel to sort the pages instead of manually doing it.

If I follow this guide using f30 should I be good to go? have you tried yet?

I believe the guide is incomplete, but no I have not. I got busy with my pc build.

If I attempt this in an empty drive and maybe f**k it up, can I just boot back into my regular os drive?

well its just a vm so it shouldnt affect your boot drive. maybe I dont get what you’re asking.

I would also be installing linux first, on said empty drive. So I guess id be dual booting, with linux on my second drive and that is what i want to run the VM on to test.

Sorry for my yahoo questions level of question

I thought you had a spare empty disk to run the VM on. @SgtAwesomesauce would be better to ask. I don’t mess with vfio

ok sry I am ret4rd. You though right. Problem is, I do not know of a term describing a computer that has two different drives with two different OS, but that is what i intend on running the VM on (the drive that I will have fedora installed onto).

just really bored at work. :funeral_urn:

AFAIK that is best practice to have your guest on a separate drive from the host.

So I can have the VM run my currently installed win10. That’s pretty cool. I figured I would have to wipe my current windows and all my games and shit to do this (I have no important data on my gaming comp) Thanks for bearing with me man. :drooling_face:

Not exactly. You will likely have to set up windows again but you would want to run it on its own drive.

I have so much to learn.

1 Like

Would I store my games on the windows drive?

Yes you would be playing the games from a windows drive. The idea of this approach is to have at least 2 drives (can be 2 parts of a large drive). Windows installation can survive both dual boot and virtual machine, while linux would be separate and untouched.

If you are primarly a windows user you might want to start with VirtualBox solution and use linux under windows. Linux does not require VFIO to perform for your day to day work. Then as you start doing most things in linux you can think about switching to Linux as primary system.

Heh I am already very deep my friend. It is too late for all of that, but I have already been using Linux on my laptop for about 8 months now and messed with mint and fedora. I got the VM working, but am at a blocker of figuring out how to setup the VM on a drive different than the one the host is on.

I followed this guide for a recent “3 gamers 1 PC” build on Fedora 30 and noticed a few things I had to change to make things work:

The entire /etc/initramfs directory is missing in F30 as all of the initramfs stuff is done with dracut, so I followed Wendell’s 2017 guide on what to do for that step. The update-initramfs command is also nonexistent as it too was replaced by dracut.

For RHEL 7 / CentOS 7 users I noticed that OVMF support is kind of broken as the package has not been updated yet in the EPEL to support the Q35 chipset.

One thing I don’t understand about the default CPU topology for new VMs is why everything goes to separate sockets even when I tell it to copy the actual CPU topology. I haven’t seem performance degradation when manually setting the specs from 4 sockets, 1 core per socket, 1 thread per core to 1 socket, 2 cores per socket, 2 threads per core.

1 Like

A couple days ago I finally managed to get this working. It was a lot of trouble, so I’m willing to help out people and try to save them some trouble! Here’s my setup:

Host Specs
[chase@chase-linux build]$ neofetch  
OS: Fedora 30 (Thirty) x86_64                                   
Kernel: 5.1.18-300.agesa.fc30.x86_64                            
Uptime: 2 days, 20 hours, 19 mins                               
Packages: 2447 (rpm), 15 (flatpak)                              
Shell: bash 5.0.7                                               
Resolution: 2560x1440, 1920x1080                                
DE: KDE                                                         
WM: KWin                                                        
Theme: Breeze Dark [KDE], Breeze-Dark [GTK2/3]                  
Icons: breeze-dark [KDE], breeze-dark [GTK2/3]                  
Terminal: konsole                                               
CPU: AMD Ryzen 7 2700X (16) @ 3.700GHz                          
GPU: AMD ATI Radeon RX 470/480/570/570X/580/580X/590            
GPU: NVIDIA GeForce RTX 2060 Rev. A                             
Memory: 12841MiB / 16027MiB
  • Notice how I had to patch the kernel with the agesa patch. I believe this is due to my BIOS but should this also be affecting you, here’s a nice guide on compiling a new kernel with the patch (on Fedora, of course): https://passthroughpo.st/agesa_fix_fedora/

What I did instead is put this script into /etc/dracut.conf.d/local.conf

add_drivers+="vfio vfio_iommu_type1 vfio_pci vfio_virqfd"
install_items="/usr/sbin/vfio-pci-override.sh /usr/bin/find /usr/bin/dirname"

add_drivers will add the vfio drivers we need into the initramfs, and install_items will add our vfio override script to initramfs. I also include the find and dirname binaries, should I decide to make my override script dynamic later.

Here’s the contents of /usr/sbin/vfio-pci-override.sh, which is where the magic actually happens:

#!/bin/sh
PREREQS=""
DEVS="0000:0a:00.0 0000:0a:00.1 0000:0a:00.2 0000:0a:00.3"
for DEV in $DEVS; do
        echo "vfio-pci" > /sys/bus/pci/devices/$DEV/driver_override
done

modprobe -i vfio-pci

Line by line, this script has my Nvidia GPU’s components saved as $DEVS, then loops through the components, instructing the kernel to essentially blacklist them from the host OS. Finally, the last line loads the vfio-pci module into the kernel.

With these files in place, I ran sudo dracut -fv to build a new initramfs and apply the above configuration. The -f stands for --force which just tells dracut to override any existing config, and the -v stands for --verbose so that dracut will output what it’s doing.

Lastly, here’s my VM config, which can be accessed with sudo virsh edit <VMNAME>. A few notes:

Config Dump
<domain type='kvm' id='3'>
  <name>win10</name>
  <uuid>305e53f1-49be-4d1d-82ae-c630f88af39f</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://microsoft.com/win/10"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <vcpu placement='static'>8</vcpu>
  <iothreads>1</iothreads>
  <cputune>
    <vcpupin vcpu='0' cpuset='1'/>
    <vcpupin vcpu='1' cpuset='9'/>
    <vcpupin vcpu='2' cpuset='2'/>
    <vcpupin vcpu='3' cpuset='10'/>
    <vcpupin vcpu='4' cpuset='3'/>
    <vcpupin vcpu='5' cpuset='11'/>
    <vcpupin vcpu='6' cpuset='4'/>
    <vcpupin vcpu='7' cpuset='12'/>
    <emulatorpin cpuset='0-1'/>
    <iothreadpin iothread='1' cpuset='0-1'/>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-q35-3.1'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/win10_VARS.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='whatever'/>
    </hyperv>
    <kvm>
      <hidden state='on'/>
    </kvm>
    <vmport state='off'/>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='4' threads='2'/>
    <feature policy='require' name='topoext'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='yes'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/home/chase/win10.qcow2'/>
      <backingStore/>
      <target dev='sdc' bus='scsi'/>
      <boot order='1'/>
      <alias name='scsi0-0-0-2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='2'/>
    </disk>
    <controller type='usb' index='0' model='qemu-xhci' ports='15'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x12'/>
      <alias name='pci.3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-to-pci-bridge'>
      <model name='pcie-pci-bridge'/>
      <alias name='pci.4'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0x13'/>
      <alias name='pci.5'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0x14'/>
      <alias name='pci.6'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    </controller>
    <controller type='pci' index='7' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='7' port='0x15'/>
      <alias name='pci.7'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>
    </controller>
    <controller type='pci' index='8' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='8' port='0x16'/>
      <alias name='pci.8'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x6'/>
    </controller>
    <controller type='pci' index='9' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='9' port='0x17'/>
      <alias name='pci.9'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x7'/>
    </controller>
    <controller type='scsi' index='0' model='virtio-scsi'>
      <alias name='scsi0'/>
      <address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:56:f1:62'/>
      <source network='default' bridge='virbr0'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/2'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/2'>
      <source path='/dev/pts/2'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-3-win10/org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'>
      <alias name='input0'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input1'/>
    </input>
    <graphics type='spice' port='5900' autoport='yes' listen='127.0.0.1'>
      <listen type='address' address='127.0.0.1'/>
      <image compression='off'/>
      <gl enable='no' rendernode='/dev/dri/by-path/pci-0000:09:00.0-render'/>
    </graphics>
    <video>
      <model type='virtio' heads='1' primary='yes'>
        <acceleration accel3d='yes'/>
      </model>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0a' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x01' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0a' slot='0x00' function='0x1'/>
      </source>
      <alias name='hostdev1'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x02' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0a' slot='0x00' function='0x2'/>
      </source>
      <alias name='hostdev2'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x03' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0a' slot='0x00' function='0x3'/>
      </source>
      <alias name='hostdev3'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x04' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x13fe'/>
        <product id='0x6300'/>
        <address bus='2' device='8'/>
      </source>
      <alias name='hostdev6'/>
      <address type='usb' bus='0' port='5'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x046d'/>
        <product id='0xc332'/>
        <address bus='5' device='70'/>
      </source>
      <alias name='hostdev7'/>
      <address type='usb' bus='0' port='1'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x1532'/>
        <product id='0x010d'/>
        <address bus='5' device='69'/>
      </source>
      <alias name='hostdev8'/>
      <address type='usb' bus='0' port='4'/>
    </hostdev>
    <redirdev bus='usb' type='spicevmc'>
      <alias name='redir0'/>
      <address type='usb' bus='0' port='2'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <alias name='redir1'/>
      <address type='usb' bus='0' port='3'/>
    </redirdev>
    <memballoon model='virtio'>
      <stats period='5'/>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </memballoon>
    <shmem name='looking-glass'>
      <model type='ivshmem-plain'/>
      <size unit='M'>32</size>
      <alias name='shmem0'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x05' function='0x0'/>
    </shmem>
  </devices>
  <seclabel type='dynamic' model='selinux' relabel='yes'>
    <label>system_u:system_r:svirt_t:s0:c53,c527</label>
    <imagelabel>system_u:object_r:svirt_image_t:s0:c53,c527</imagelabel>
  </seclabel>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+107:+107</label>
    <imagelabel>+107:+107</imagelabel>
  </seclabel>
</domain>
4 Likes