I hope not! My 1070 won’t quite fit in the bottom slot without some… creativity.
iirc bottom slot is routed through the chipset, and is less likely to be isolated, so no.
Alright, a couple things here. B350 is a pain in the ass to make working, if your IOMMU groups are okay, that’s going to be good, but your host GPU MUST be the boot GPU. Which one does the UEFI show up on? That’s the one that you need to use for the host. You can switch the slots to switch which GPU is for host, but that’s going to be your main problem.
RE: your guest GPU not binding to vfio-pci, I’m not too sure. I use AMD for host and nvidia for guest because it simplifies things significantly.
The host gpu doesn’t have to be the boot gpu. I’m booting my MSI UEFI with the 1050Ti, but then reloading my gfx’s uefi rom in the virtual machine XML file. I’ve also configured x server to run on my second slot x4 ati 4850.
I had trouble isolating my second x4 pcie slot which is why I’m running the guest gpu in the primary slot.
Try to add video=efifb:off to the kernel boot parameters, and don’t forget to reload the initial ramdisk so those kernel modules can load on the next boot.
Some help from anyone with any idea on this would be incredibly appreciated…
System:
TAICHI x399
Ryzen Threadripper 1950X
GPU 1: Quadro K5200 in Slot 2 (Won’t fit in 1)
GPU 2: GeForce 9300 GE in Slot 4
I have re-installed Fedora 27 a few times and started over… With the same results… I also have Windows Installed in a 10 TB Sata drive which will be my VM.
I have NOT created the VM, I have been going through the steps to passthrough the QUADRO K5200 GPU:
sudo dnf update
sudo dnf install @virtualization
sudo dnf install nano
sudo nano /etc/default/grub
added
iommu=1 amd_iommu=on rd.driver.pre=vfio-pci
to GRUB_CMDLINE_LINUX
sudo nano /etc/modprobe.d/kvm.conf
checked if kvm_amd nested=1 and kvm_intel nested=1 are not activated
sudo nano /etc/modprobe.d/vfio.conf
set the ids of the GPU and audio to passthrough (GPU 1)
options vfio-pci ids=10de:103c,10de:0e1a
updated/created a vfio.conf in /etc/dracut.conf.d with
add_drivers+=“vfio vfio_iommu_type1 vfio_pci” andracut -f --kver
uname -rkveruname -r
then
sudo grub2-mkconfig -o /etc/grub2-efi.cfg
absolutely NO ERRORS
rebooted
I have both GPU’s connected to monitors and both are Black after reboot… Actually the one connected to the QUADRO has a white small line in the top left corner. I set up VNC and SSH beforehand and am able to use both with no problems (as well as cloning the drive before making these changes again lol )
My GPU’s
IOMMU Group 12 08:00.0 VGA compatible controller [0300]: NVIDIA Corporation G98 [GeForce 9300 GE] [10de:06e0] (rev a1)
IOMMU Group 24 43:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK110GL [Quadro K5200] [10de:103c] (rev a1)
IOMMU Group 24 43:00.1 Audio device [0403]: NVIDIA Corporation GK110 HDMI Audio [10de:0e1a] (rev a1)
lspci -vnn shows
08:00.0 VGA compatible controller [0300]: NVIDIA Corporation G98 [GeForce 9300 GE] [10de:06e0] (rev a1) (prog-if 00 [VGA controller])
Subsystem: NVIDIA Corporation Device [10de:060c]
Flags: bus master, fast devsel, latency 0, IRQ 63, NUMA node 0
Memory at ee000000 (32-bit, non-prefetchable) [size=16M]
Memory at d0000000 (64-bit, prefetchable) [size=256M]
Memory at ec000000 (64-bit, non-prefetchable) [size=32M]
I/O ports at 7000 [size=128]
Expansion ROM at ef000000 [disabled] [size=128K]
Capabilities:
Kernel driver in use: nouveau
Kernel modules: nouveau
43:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK110GL [Quadro K5200] [10de:103c] (rev a1) (prog-if 00 [VGA controller])
Subsystem: NVIDIA Corporation Device [10de:1095]
Flags: bus master, fast devsel, latency 0, IRQ 11
Memory at cc000000 (32-bit, non-prefetchable) [size=16M]
Memory at b0000000 (64-bit, prefetchable) [size=256M]
Memory at c0000000 (64-bit, prefetchable) [size=32M]
I/O ports at f000 [size=128]
Expansion ROM at cd000000 [disabled] [size=512K]
Capabilities:
Kernel driver in use: vfio-pci
Kernel modules: nouveau
43:00.1 Audio device [0403]: NVIDIA Corporation GK110 HDMI Audio [10de:0e1a] (rev a1)
Subsystem: NVIDIA Corporation Device [10de:1095]
Flags: bus master, fast devsel, latency 0, IRQ 10
Memory at cd080000 (32-bit, non-prefetchable) [size=16K]
Capabilities:
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel
lsmod | grep -i vfio shows
vfio_pci 45056 0
irqbypass 16384 2 kvm,vfio_pci
vfio_virqfd 16384 1 vfio_pci
vfio_iommu_type1 24576 0
vfio 28672 2 vfio_iommu_type1,vfio_pci
vfio-pci exists…
filename: /lib/modules/4.14.13-300.fc27.x86_64/kernel/drivers/vfio/pci/vfio-pci.ko.xz
@bsodmike has a lot of experience with the Virtualization side of Threadripper. He may be able to lend some insight.
2 Likes
That would be incredibly awesome… Thanks for the info 
No luck.
/etc/default/grub:
...
GRUB_CMDLINE_LINUX="rd.luks.uuid=luks-a2af3e95-64f5-4716-b7da-94d7cb837daf rd.luks.uuid=luks-a8e10744-5d7c-4e8e-84e0-e54d911899cf rhgb quiet iommu=1 amd_iommu=on rd.driver.pre=vfio-pci video=efifb:off"
...
$ sudo dracut -f --kver `uname -r`
$ sudo grub2-mkconfig | sudo tee /etc/grub2-efi.cfg
$ sudo reboot
Both GPUs still work in Linux. No change in lspci.
$ dmesg | grep -i efifb
[ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-4.14.13-300.fc27.x86_64 root=UUID=7e7e0710-a359-41e3-bac0-ef751ff658a8 ro rd.luks.uuid=luks-a2af3e95-64f5-4716-b7da-94d7cb837daf rd.luks.uuid=luks-a8e10744-5d7c-4e8e-84e0-e54d911899cf rhgb quiet iommu=1 amd_iommu=on rd.driver.pre=vfio-pci video=efifb:off
[ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-4.14.13-300.fc27.x86_64 root=UUID=7e7e0710-a359-41e3-bac0-ef751ff658a8 ro rd.luks.uuid=luks-a2af3e95-64f5-4716-b7da-94d7cb837daf rd.luks.uuid=luks-a8e10744-5d7c-4e8e-84e0-e54d911899cf rhgb quiet iommu=1 amd_iommu=on rd.driver.pre=vfio-pci video=efifb:off
[ 0.078213] pci 0000:0b:00.0: BAR 3: assigned to efifb
$ dmesg | grep -i vfio
[ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-4.14.13-300.fc27.x86_64 root=UUID=7e7e0710-a359-41e3-bac0-ef751ff658a8 ro rd.luks.uuid=luks-a2af3e95-64f5-4716-b7da-94d7cb837daf rd.luks.uuid=luks-a8e10744-5d7c-4e8e-84e0-e54d911899cf rhgb quiet iommu=1 amd_iommu=on rd.driver.pre=vfio-pci video=efifb:off
[ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-4.14.13-300.fc27.x86_64 root=UUID=7e7e0710-a359-41e3-bac0-ef751ff658a8 ro rd.luks.uuid=luks-a2af3e95-64f5-4716-b7da-94d7cb837daf rd.luks.uuid=luks-a8e10744-5d7c-4e8e-84e0-e54d911899cf rhgb quiet iommu=1 amd_iommu=on rd.driver.pre=vfio-pci video=efifb:off
$ lsmod | grep -i vfio
Could you share your XML and X config?
Thanks all for the help!
Hi all,
first of all thanks @wendell for the tutorial. I tried fiddling this together for a while on my own, but since I’m kind of a scrubb on linux, I haven’t been able to completly get it working … “yet”.
As for the problem: I want to pass my nvme controller through to windows, to boot from it. The controller as a pcie device is recognized and I even can select it as a boot device. But within 3 seconds of booting the vm just freezes. I had the xml file first created by the gui, then edited it to use sata, like in the guide.
My setup is Fedora 27 with Kernel 4.14, other than that I have followed the procedere mentioned by the guide. For the hardware I’m using an Asus Prime x370 mainboard with a ryzen 7 1800k.
Have a nice day and greetings from Germany
you need the latest ovmf – I had this problem myself and a kind soul from the stream pointed out it is a fantastically ancient version. on the test box I just searched for and downloaded a more recent version of ovmf. At the moment I dont have a link handy but perhaps a kind soul will add the link here before I get back to my other computer.
1 Like
Don’t use Q35. Had the same issue trying to use Q35 chipset. The FX chipset is the only viable option now.
1 Like
Thanks guys, that did it for me <3
I’ll be playing around with this for a bit now and let you know how performance is.
On unrelated note, did you ever got linux to completly freeze ?
Firefox seems to crash the os whenever I resize the browser sideways. Interestingly enough music, like from youtube is still playing, but the system is completly unresponsive else.
Let me know if you know this, else I post it if I can find the error myself.
Thanks again and have a nice day.
regards Warui
Yup, I’ve gotten the whole system to freeze. When my Fresco Logic controller is improperly uninitialized, trying the restart the VM will cause the entire host OS to lock up.
1 Like
Thanks for replying this fast again
This time I meant within Fedora though. I did some more testing, as well as deactivated hardware acceleration, but if I resize the window using the mouse, the system still freezes completely.
But I wil have so to some more tinkering tomorrow, since it’s already late for me.
Have a nice day and a good evening, whenever that is <3
regards Warui
That could be the Meltdown patches. This is not Fedora specific as it happened on my 6 core using a KPTI kernel on Ubuntu too. You may want to roll back to the pre-Meltdown kernels.
1 Like
Ok, i tried another gpu and i seems to be working, for now at least, but thanks I keep an eye out for that <3
regards Warui
just gonna leave this here for people thinking of building a passthrough gaming specific rig:
This guide assumes 2 monitors, right?
Edit: Fuck it i’m buying. I’ll report results when i have them available.
Anyone got this working on a Gigabyte AB350-Gaming 3?