Return to Level1Techs.com

Ryzen Virtualization Success (GTX 1080ti passthrough with Windows 10 client)


#1

I spent a LOT of time on this. As in, all day, every day for 7 days straight. But it worked!

I now have a Fedora host with a Windows 10 virtual machine running smoothly and reliably on a Ryzen processor with Nvidia Geforce 1080ti passthrough. With this setup I can boot natively into Windows and utilize the 1080ti graphics card, or launch it within Fedora as a virtual machine... And still utilize the full power of the 1080ti graphics card!

Hardware:

  • Ryzen 7 1800x
  • Asrock X370 Gaming K4 (Fatal1ty AM4) Motherboard
  • Samsung 960 EVO M.2 SSD 500GB - For Fedora host.
  • Sandisk Ultra 2 240GB SSD - For Windows 10 client.
  • Nvidia Geforce GT710 1GB Graphics Card for Host - In primary PCI slot.
  • EVGA Black Edition SC Nvidia Geforce GTX 1080TI Graphics Card for Client Passthrough - In secondary PCI slot.
  • Corsair Vengeance 32GB 3000MHz DDR4 Memory
  • Second Keyboard and Second Mouse
  • Two HDMI monitors

Software

  • Fedora 26 Beta for Host (Update 2017-07-12: Fedora 26 is now out, all still working well after upgrade) - Virtualization tools (@Virtualization) / Nvidia driver 375.66 for GT710 Host graphics card (not the passthrough one!)
  • Windows 10 N Pro iso - Latest Nvidia Driver for 1080ti installed by booting natively into the Sandisk SSD after Windows installation as per Nvidia website.

Instructions - Parts to change are in bold:

Part A: Install Fedora 26 in UEFI mode

Step 1. Use the Fedora Media Writer tool to write the Fedora 26 image to a bootable USB drive.
Step 2. Boot your computer while hitting F11 or whatever your motherboard uses to launch the boot menu if it's different.
Step 3. Make sure you boot the USB drive in EFI mode - the option will usually be there. You should come to a Fedora menu with a BLACK background, not a blue one, otherwise this will not work.
Step 4. Install Fedora.
Step 5. Open terminal and run:

sudo dnf update

Step 6: Reboot, then open terminal and run:

sudo dnf install @virtualization

Part B: Bind your pass-through GPU to the VFIO-PCI driver:

Step 1. Commands to edit the Grub bootloader to turn on IOMMU and load the VFIO-PCI driver early enough to grab your card:

sudo -i
vi /etc/default/grub

Edit this file:

GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="rd.lvm.lv=fedora/root rd.lvm.lv=fedora/swap iommu=1 amd_iommu=on rd.driver.pre=vfio-pci"
GRUB_DISABLE_RECOVERY="true"

Step 2. Now you want to take note of the vendor & device ID pairs for the graphics card you are passing through to the virtual machine:

lspci -nn

Then look for your graphics card and it's sound device and write down the two sets of codes e.g. [10de:1b06] for each of those two devices.

Add this to configure the graphics card and its associated audio device - change the two pairs of codes to those of your graphics card and attached sound device - then save:
First create/open the file to edit:

vi /etc/modprobe.d/vfio.conf

Then add the following line:

options vfio-pci ids=10de:1b06,10de:10ef

Step 3. Now you want to edit the linux initial ramdisk to add the VFIO drivers with the following commands:

vi /etc/dracut.conf.d/vfio.conf

Then add the following line:

add_drivers+="vfio vfio_iommu_type1 vfio_pci"

Step 4. Edit kvm.conf:

vi /etc/modprobe.d/kvm.conf

There should be a pound (#) sign at the start of both lines to comment them out as these technologies are a bit unstable at the moment:

options kvm_intel nested=1
options kvm_amd nested=1

Step 5. Create a file in modprobe.d called blacklist.conf in order to blacklist the nouveau driver that will automatically grab your passthrough graphics card and your host graphics card if they're Nvidia, in turn preventing the vfio-pci driver from binding to your passthrough gpu:

vi /etc/modprobe.d/blacklist.conf

Add this to the file and save:

blacklist nouveau
blacklist lbm-nouveau
options nouveau modeset=0
alias nouveau off
alias lbm-nouveau off

Step 6. Update the initial ramdisk so vfio-pci driver loads up early:

sudo dracut -f --kver `uname -r`

Step 7. Update the grub bootloader:

sudo grub2-mkconfig > /etc/grub2-efi.cfg

Once you've executed everything without errors, reboot your system.

Step 8. Verify that vfio-pci has bound to your passthrough graphics card. You should see that the current driver for your graphics card says vfio-pci:

lspci -nnv

Part C: Install your host (not the passthrough) graphics card if Nvidia:

Follow the full guide to manually install it at https://www.if-not-true-then-false.com/2015/fedora-nvidia-guide/

Part D: Fire up Virtual Machine Manager

Step 1. Create a new virtual machine using the GUI.
Step 2. Choose the option to use local install media (iso) and choose your Windows 10 iso file - I downloaded mine to my Fedora installation's Downloads folder.
Step 3. Follow the wizard, select how much RAM to give the VM, choose the size of your virtual disk (I chose 30GB as this is only a temporary disk, we'll delete it later). Just before finalizing the install, select the "configure before install" checkbox.
Step 4. Now, in the overview tab, change the firmware to UEFI, remove Graphics and Video devices and, after plugging in a second USB keyboard and mouse into your computer, go to Add Hardware, go to USB controllers, and add the second USB keyboard and second USB mouse you have plugged in - these will be used to control the second HDMI monitor that will be running your Windows VM. Then go to PCI controllers and add your passthrough graphics card and it's associated audio device.
Step 5. Click install to launch the Windows installation. At this point, with your host monitor plugged into the host graphics card and your client monitor plugged into the passthrough graphics card, you should see a TIanocore logo appear on the client monitor. Hit any key when you're prompted to boot from CD (the iso you selected earlier) and install Windows to the virtual drive. If for any reason you end up at a shell, type exit and it should take you to a page where you can select boot manager and fire up the virtual DVD-rom drive.
Step 6. Once that's done shut down Windows, the virtual machine should be powered down in Virtual Machine Manager, then back in Fedora open a terminal and type:

sudo -i
cd /etc/libvirt/qemu
virsh define win10.xml
virsh edit win10

Step 7. Your file should look similar to the following. The important step is changing the virtual hard drive to your extra blank physical SATA drive onto which a fresh install of Windows will go (i.e. my Sandisk Ultra 2 drive), as well as the very first line, and the last 5 lines of the document below, which ensures that Nvidia cards don't throw up an error 43 in your VM:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/2.0'>
  <name>win10</name>
  <uuid>b461a6d9-cac1-419d-950e-c9710e93d0eb</uuid>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <vcpu placement='static' current='1'>8</vcpu>
  <iothreads>4</iothreads>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='1'/>
    <vcpupin vcpu='2' cpuset='2'/>
    <vcpupin vcpu='3' cpuset='3'/>
    <vcpupin vcpu='4' cpuset='4'/>
    <vcpupin vcpu='5' cpuset='5'/>
    <vcpupin vcpu='6' cpuset='6'/>
    <vcpupin vcpu='7' cpuset='7'/>
    <iothreadpin iothread='1' cpuset='0-1'/>
    <iothreadpin iothread='2' cpuset='2-3'/>
    <iothreadpin iothread='3' cpuset='4-5'/>
    <iothreadpin iothread='4' cpuset='6-7'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.9'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/edk2/ovmf/OVMF_CODE.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/win10_VARS.fd</nvram>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>
    <kvm>
      <hidden state='on'/>
    </kvm>
    <vmport state='off'/>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='4' threads='2'/>
    <feature policy='disable' name='smep'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='yes'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-kvm</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw'/>
      <source dev='/dev/sda'/>
      <target dev='vdb' bus='sata'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/home/msamy/Win10_1703_N_English_x64.iso'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:02:51:4d'/>
      <source network='default'/>
      <model type='rtl8139'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x0d' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x0d' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x1a2c'/>
        <product id='0x2124'/>
      </source>
      <address type='usb' bus='0' port='1'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x17ef'/>
        <product id='0x608b'/>
      </source>
      <address type='usb' bus='0' port='4'/>
    </hostdev>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='2'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='3'/>
    </redirdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </memballoon>
  </devices>
  <qemu:commandline>
    <qemu:arg value='-cpu'/>
    <qemu:arg value='host,hv_time,kvm=off,hv_vendor_id=null'/>
  </qemu:commandline>
</domain>

Step 8. Once you've edited that file hit ESC to go out of Insert mode, then press Shift+Z+Z to save and exit. Type:

virsh define win10.xml

once more and you're ready to move on.

Step 9. Go back to Virtual Machine Manager and in your virtual machine conguration GUI, go to the Boot tab and make sure the DVD/CD rom drive is the boot device. With your blank SATA drive connected, start the VM again, and you'll either get a "press any key" prompt, which should launch Windows installer, or a shell, where again, you'd type exit and then manually choose the virtual DVD-rom drive to boot from. In the Windows installer, go to custom install, choose your SATA drive and install Windows.

Step 10. Shut down that Windows installation and ensure the virtual machine goes into powered down state in Virtual Machine Manager on your Fedora host. Then shut down Fedora and press F11 or your equvalent to go to the host's boot menu. Choose the SATA drive you just installed Windows on and boot from that. Install your Nvidia drivers. Once successful, reboot into Fedora, start your VM up through Virtual Machine Manager, and your VM should now be running Windows 10 with up to date passthrough GPU drivers.

Disclaimer: I am a novice myself, so can't guarantee the reliability of the information I've provided on this web page. I don't accept any responsibility or liability for any damage or injury or any other event that occurs as a result of following my guide above. Use at your own risk, and to your own peril if anything goes wrong!


GPU for Fedora 26 (VM Rig)
Fedora apply npt patch
2700X or 1900X for VMs, PCi'e gaming passtrough - Unable to decide
Should I get a gtx 1080 or Vega 64
#2

well done, nice work, good job. I have been meaning to "clean up" and finalize my tutorial but yours is awesome.

Also on my to do list is "VFIO Sound: Troubleshooting common issues and fixing your VM's sound especially in games."


#3

Thanks! After all the guides I read and the hours of experimenting and raging at things not working, I thought it would be worthwhile documenting my steps before I forget them, as well as giving back to this great community. Ah yes, my HDMI audio sounds like something out of the exorcist so that's my first challenge.

Next step after sorting the sound: Hot swappable remoting, with text message powered power switch, into Ubuntu (for Blender, scientific/anatomy volume-rendering applications, Inkscape and web development) and Windows (Autodesk Suite, Unity and Unreal development) VMs from a nice portable 4K thin client for my 3D medical game development work.


#4

there was a reply on the story thread that has a hot key you can bind to toggle usb keyboards between host and vm. that is the coolest thing ever, tbh. kudos to that guy.

I fixed the hdmi audio on my other box, but I need to look at what I did.


#5

I think the

sudo dracut -f --kver uname -r

should be

sudo dracut -f --kver uname -r

And not normal " ' " either but shift + key next to backspace (on swedish keyboard). Slanted sign.

I have one question. Does the GPU you passthrough have to be in 2nd slot? Every time I blacklist my Radeon, I get blank screen and my Nvidia in 2nd slot displays nothing. Can't boot in, have to start all over again.


#6

Sorry about that. You are correct. Fixed the formatting.

I wouldn't blacklist your Radeon if that's your host GPU. You want it to work.
If you're passing your Nvidia GPU through to the client you want to blacklist Nouveau so it doesn't bind the passthrough GPU and the vfio-pci placeholder driver binds instead.
Once you have your correct Radeon drivers running on your host device in the primary PCIe slot and the Nvidia in the secondary slot with vfio-pci bound to it and not Nouveau, you're in a good place.


#7

No, I want it the other way, pass on my Radeon. That's why I blacklist it.
Nvidia card says it uses nouveau-drivers when I run lspci -nnk. But output stays blank.


#8

So what I would do is put the Radeon in the secondary PCIe slot and the Nvidia in the primary PCIe slot, connect the cards to different monitors then boot the host (Fedora). As the Nvidia is in the primary PCIe slot it should boot up into Fedora (your host) on the monitor connected to the primary Nvidia card. On doing lspci -nnv you should see the Radeon card using the vfio-pci driver and then it's ready to be passed through to the VM.


#10

Are you able to pass this 1080ti back to the Linux system without rebooting? To play Linux games for example. Also are you using the CPU and system memory virtually so you can pass that back to Linux when your not using Windows VM?


#11

Praise


#12

How is yours vm performance, i have fps drops in games, but not in native windows env. (my sys: 1700X, GTX 1080)


#13

Game performance has been pretty low for me too, Overwatch has been running between 40 and 90 FPS and FFXIV is at a constant 20 fps. I'm running a 1700 GTX 1070 and a m.2 passthrough to the VM.


#14

same here, less than 40 fps in Overwatch, although I get around 120fps when running it native
System: R7 1700 & GTX 1080 ti

there seems to be a known issue with NPT:
https://community.amd.com/thread/215931

turning NPT off boosts the GPU to the expected performance, but the system becomes almost unusable for me outside of games with stutters and hickups (even with pinned cores)

also relevant:


#15

Hi, thank you for your tutorial, a little question, use UEFI is a must in order to make this work?


#16

Thi There,
I was wondering whether having a graphics card configured as the windows host,- is it still possible to use the same card as a normal HDMI output or do I have to use a second high end graphics card if I would like to use something like a UWQHD/100Hz Widescreen?


#17

I just like to say thanks for the guide i was able to follow this guide pretty well and i am now able to run windows in the box it belongs thanks to this