a lot of people on /r/VFIO are having more success with arch, not sure how much that helps.
Never had a problem doing the same on intel, even with mobos with broken IOMMU.
Guys, I know it’s all VFIO in here, but if you want to try things, try doing it with pci-stub??
- Add to Kernel Line in /etc/default/grub (instead of vfio)
amd_iommu=on pci-stub.ids=1002:6899,1002:aa50
- sudo grub2-mkconfig -o /boot/grub2/grub.cfg
Reboot
- To Check if pci-stub Worked
dmesg | grep pci-stub
- Try it
which part is not working? is the stub driver bound to your 1070?
I have a fedora 26 host and several VM including fedora 25 that I upgraded to 26. It work fine up until I think upgrade to kernel 4.12.5. Now the VM wont boot or install fresh fedora 26 with just KVM / QEMU no passthrough. So there is a problem that needs to be fixed for either the host or guest with fedora 26.
Maybe some else can confirm this as Im not a expert but I have rolled back the guest too kernel 4.11-* via a snapshot and it works but will update and lock up on boot.
It always has nouveau bound to the 1070 and amdgpu bound to the RX 550. I would have thought the *.pre in the grub would have nixed all that.
Right after boot if you do an lsmod does it show vfio?
Finally got it working myself:
It seems that Linux guests working only using the Chipset i440FX (at least for me), but as far as I read just Q35 supports PCIE passthrough and i440FX only PCI passthrough, so I am still a little bit confused.
BTW: If you want to use synergy to share keyboard and mouse, you have to use Xorg desktop instead of wayland
The last thing that is not working properly for me is the mentioned “Saving VM state with GPU passthrough” and of course performance could be better.
2 Likes
Thanks for the tip. Your suggestion for the grub changes worked just fine and the stub is grabbing the GPU as intended.
Still not sure why the vfio is such a PITA for me, though.
1 Like
Yeah, I don’t know, what are the advantages of vfio vs stub?
Mainly vfio let’s you set power state to low power. Supposedly you can also disable the root hub to let many cards be reinitialized but idk about that.
Maybe you can do low power with stub now too… Not sure really
1 Like
So how’s the performance with your setup? If I ever decide to pull the trigger, I would get something like your setup. is nvidia behaving in the vm?
I don’t know, but I have seen that my 5850 can reboot with the vm if I’m on windows 10 without issues, but it will freeze the system with windows 7. That’s with vfio. I don’t recall with pci-stub.
Has there been any answers from AMD regarding the NPT issue, and the fact that Threadripper is “broken” with regards to PCIe passthrough ( reddit link )?
[root@localhost ~]# dracut -f --kver `uname -r`
/etc/dracut.conf.d/vfio.conf: line 1: vfio_iommu_type1: command not found`
Thoughts? getting the same error as Chop91 in journalctl. And still seeing the nvida drivers:
lspci -vnn
21:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM206 [GeForce GTX 960] [10de:1401] (rev a1) (prog-if 00 [VGA controller])
Subsystem: eVga.com. Corp. Device [3842:3967]
Flags: bus master, fast devsel, latency 0, IRQ 66
Memory at f4000000 (32-bit, non-prefetchable) [size=16M]
Memory at e0000000 (64-bit, prefetchable) [size=256M]
Memory at f0000000 (64-bit, prefetchable) [size=32M]
I/O ports at d000 [size=128]
[virtual] Expansion ROM at f5000000 [disabled] [size=512K]
Capabilities: [60] Power Management version 3
Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [78] Express Legacy Endpoint, MSI 00
Capabilities: [100] Virtual Channel
Capabilities: [258] L1 PM Substates
Capabilities: [128] Power Budgeting <?>
Capabilities: [420] Advanced Error Reporting
Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
Kernel driver in use: nvidia
Kernel modules: nouveau, nvidia_drm, nvidia
system:
Ryzen 1700
ASUS B350-Plus – bios 0806
gtx 1030 (primary slot)
gtx 960 (secondary, attempting to vfio)
kernel:
4.12.9-300.fc26.x86_64
lsinitrd should show you what modules are installed in the initial ramdisk. What distro and kernel ? Try find /var |grep vfio and let us know the output of those commands ?
fedora26
kernel 4.12.9-300.fc26.x86_64
[root@localhost ~]# lsinitrd | grep vfio
-rw-r–r-- 1 root root 41 Aug 11 08:19 etc/modprobe.d/vfio.conf
[root@localhost ~]# find /var |grep vfio
[root@localhost ~]#
I can grep on something else in lsinitrd if it helps. its a huge output
I’m basically just following your guide; brand new fedora install and all. no prior experience with vfio / gpu passthrough
Try find / grep vfio kernel is slightly newer than the guide
We are looking for the name(s) of the vfio modules to tell dracut to include in the initial ramdisk
[root@localhost ~]# find / | grep vfio
Summary
/etc/modprobe.d/vfio.conf
/etc/dracut.conf.d/vfio.conf
find: ‘/run/user/1000/gvfs’: Permission denied
/usr/lib/modules/4.12.9-300.fc26.x86_64/kernel/drivers/vfio
/usr/lib/modules/4.12.9-300.fc26.x86_64/kernel/drivers/vfio/vfio.ko.xz
/usr/lib/modules/4.12.9-300.fc26.x86_64/kernel/drivers/vfio/mdev
/usr/lib/modules/4.12.9-300.fc26.x86_64/kernel/drivers/vfio/mdev/vfio_mdev.ko.xz
/usr/lib/modules/4.12.9-300.fc26.x86_64/kernel/drivers/vfio/mdev/mdev.ko.xz
/usr/lib/modules/4.12.9-300.fc26.x86_64/kernel/drivers/vfio/pci
/usr/lib/modules/4.12.9-300.fc26.x86_64/kernel/drivers/vfio/pci/vfio-pci.ko.xz
/usr/lib/modules/4.12.9-300.fc26.x86_64/kernel/drivers/vfio/vfio_virqfd.ko.xz
/usr/lib/modules/4.12.9-300.fc26.x86_64/kernel/drivers/vfio/vfio_iommu_type1.ko.xz
/usr/lib/modules/4.11.8-300.fc26.x86_64/kernel/drivers/vfio
/usr/lib/modules/4.11.8-300.fc26.x86_64/kernel/drivers/vfio/vfio.ko.xz
/usr/lib/modules/4.11.8-300.fc26.x86_64/kernel/drivers/vfio/mdev
/usr/lib/modules/4.11.8-300.fc26.x86_64/kernel/drivers/vfio/mdev/vfio_mdev.ko.xz
/usr/lib/modules/4.11.8-300.fc26.x86_64/kernel/drivers/vfio/mdev/mdev.ko.xz
/usr/lib/modules/4.11.8-300.fc26.x86_64/kernel/drivers/vfio/pci
/usr/lib/modules/4.11.8-300.fc26.x86_64/kernel/drivers/vfio/pci/vfio-pci.ko.xz
/usr/lib/modules/4.11.8-300.fc26.x86_64/kernel/drivers/vfio/vfio_virqfd.ko.xz
/usr/lib/modules/4.11.8-300.fc26.x86_64/kernel/drivers/vfio/vfio_iommu_type1.ko.xz
/usr/src/kernels/4.12.9-300.fc26.x86_64/samples/vfio-mdev
/usr/src/kernels/4.12.9-300.fc26.x86_64/samples/vfio-mdev/Makefile
/usr/src/kernels/4.12.9-300.fc26.x86_64/drivers/vfio
/usr/src/kernels/4.12.9-300.fc26.x86_64/drivers/vfio/platform
/usr/src/kernels/4.12.9-300.fc26.x86_64/drivers/vfio/platform/Makefile
/usr/src/kernels/4.12.9-300.fc26.x86_64/drivers/vfio/platform/Kconfig
/usr/src/kernels/4.12.9-300.fc26.x86_64/drivers/vfio/platform/reset
/usr/src/kernels/4.12.9-300.fc26.x86_64/drivers/vfio/platform/reset/Makefile
/usr/src/kernels/4.12.9-300.fc26.x86_64/drivers/vfio/platform/reset/Kconfig
/usr/src/kernels/4.12.9-300.fc26.x86_64/drivers/vfio/Makefile
/usr/src/kernels/4.12.9-300.fc26.x86_64/drivers/vfio/Kconfig
/usr/src/kernels/4.12.9-300.fc26.x86_64/drivers/vfio/mdev
/usr/src/kernels/4.12.9-300.fc26.x86_64/drivers/vfio/mdev/Makefile
/usr/src/kernels/4.12.9-300.fc26.x86_64/drivers/vfio/mdev/Kconfig
/usr/src/kernels/4.12.9-300.fc26.x86_64/drivers/vfio/pci
/usr/src/kernels/4.12.9-300.fc26.x86_64/drivers/vfio/pci/Makefile
/usr/src/kernels/4.12.9-300.fc26.x86_64/drivers/vfio/pci/Kconfig
/usr/src/kernels/4.12.9-300.fc26.x86_64/include/uapi/linux/vfio.h
/usr/src/kernels/4.12.9-300.fc26.x86_64/include/uapi/linux/vfio_ccw.h
/usr/src/kernels/4.12.9-300.fc26.x86_64/include/config/vfio.h
/usr/src/kernels/4.12.9-300.fc26.x86_64/include/config/vfio
/usr/src/kernels/4.12.9-300.fc26.x86_64/include/config/vfio/pci.h
/usr/src/kernels/4.12.9-300.fc26.x86_64/include/config/vfio/virqfd.h
/usr/src/kernels/4.12.9-300.fc26.x86_64/include/config/vfio/mdev.h
/usr/src/kernels/4.12.9-300.fc26.x86_64/include/config/vfio/mdev
/usr/src/kernels/4.12.9-300.fc26.x86_64/include/config/vfio/mdev/device.h
/usr/src/kernels/4.12.9-300.fc26.x86_64/include/config/vfio/iommu
/usr/src/kernels/4.12.9-300.fc26.x86_64/include/config/vfio/iommu/type1.h
/usr/src/kernels/4.12.9-300.fc26.x86_64/include/config/vfio/pci
/usr/src/kernels/4.12.9-300.fc26.x86_64/include/config/vfio/pci/vga.h
/usr/src/kernels/4.12.9-300.fc26.x86_64/include/config/vfio/pci/intx.h
/usr/src/kernels/4.12.9-300.fc26.x86_64/include/config/vfio/pci/mmap.h
/usr/src/kernels/4.12.9-300.fc26.x86_64/include/config/vfio/pci/igd.h
/usr/src/kernels/4.12.9-300.fc26.x86_64/include/config/kvm/vfio.h
/usr/src/kernels/4.12.9-300.fc26.x86_64/include/linux/vfio.h
/usr/include/linux/vfio.h
/usr/include/linux/vfio_ccw.h
/dev/vfio
/dev/vfio/vfio
True, about the kernel. I can downgrade if that will be easier.
There it is
/usr/lib/modules/4.12.9-300.fc26.x86_64/kernel/drivers/vfio/vfio_iommu_type1.ko.xz
Try dracut --add-drivers “vfio_iommu_type1 vfio vfio-pci” ? Then the dracut -f command. Maybe a typo in your vfio.conf ?
cant be a typo, used copy paste 
[root@localhost ~]# dracut --add-drivers “vfio_iommu_type1 vfio vfio-pci”
dracut: Cannot find module directory /lib/modules/vfio-pci”/
dracut: and --no-kernel was not specified
EDIT: editing my foot as we speak. Typed out the "add_drivers… " and it worked fine. must be a copy /paste error with vim??
updating grub2 now. fingers crossed
Ls -l /lib/modules ?
The quotation marks sometimes get messed up on copy paste.
usr/lib/modules/4.12.9-300.fc26.x86_64/kernel/drivers/vfio/pci/vfio-pci.ko.xz
There is vfio-pci so … maybe the /lib/modules is the wrong symlink or something? Weird it says it can’t find it