I’ve pretty much set everything up, in the middle of writing out a bash script for QEMU trying to work out dedicated nvme for passthrough inside it atm, lots of reading haha.
Do you know which IOMMU is the top NVME?
IOMMU Group 15 03:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43d0] (rev 01)
IOMMU Group 15 03:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43c8] (rev 01)
IOMMU Group 15 03:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43c6] (rev 01)
IOMMU Group 15 20:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 02)
IOMMU Group 22 31:00.2 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
Got the GPU Pasing over, along with keyboard on kvm working fine. just having issues with qemu bash script and the nvme drive boot with a preinstalled windows 10 on it. I dont think im using the right controller for it.
Whats the easiest way to differentiate them?
–Edit found how to do that…
IOMMU Group 14 01:00.0 Non-Volatile memory controller [0108]: Micron/Crucial Technology Device [c0a9:2263] (rev 03)
joykiller@battlestation:~$ ls -l /sys/block/nvme0n1
lrwxrwxrwx 1 root root 0 Feb 18 00:19 /sys/block/nvme0n1 -> …/devices/pci0000:00/0000:00:01.1/0000:01:00.0/nvme/nvme0/nvme0n1
– Think on the right track, However hit a wall. Googling for answers now.
Getting this error:
QEMU 2.11.1 monitor - type ‘help’ for more information
(qemu) qemu-system-x86_64: AMD CPU doesn’t support hyperthreading. Please configure -smp options properly.
qemu-system-x86_64: -device vfio-pci,host=01:00.0: vfio error: 0000:01:00.0: failed to add PCI capability 0x11[0x50]@0xb0: table & pba overlap, or they don’t fit in BARs, or don’t align
So update, There is a bug with some NVME Controllers.
https://bugzilla.kernel.org/show_bug.cgi?id=202055#c29
However, the above patch should fix it for anyone using the same NVME as I, which is a cheap Crucial P1 CT500P1SSD8. Just keeping this as a log/diary if others have same issues as I.
Or easier way to just buy a Samsung NVME lol. - Which is probably the method im going to take as this controller isnt that great supposedly. Going to pick up a Samsung instead of saving 10$
gpu passthrough
Hi,
I am using an ASRock, X470 Taichi Ultimate and no matter what I try, I am not able to pass-through my EVGA NVIDIA Geforce GTX 760 to a windows VM. Everytime I try, I either get some sort of 127 error or the machine starts but I get no video signal anywhere and I know that the cards work. What am I missing?
Build
Motherboard:
ASRock, X470 Taichi Ultimate
BIOS Version: SMBIOS 3.2.1 present
Processor:
AMD Ryzen 7 2700X Eight-Core Processor
Disk:
Linux: Samsung Electronics NVMe SSD Controller SM961/PM961
Windows(7/10) VMSamsung SSD 860
GPUs:
Linux: PNY nVidia Quadro M2000
For VM Pass-through: EVGA nVidia GeForce GTX 760
Linux:
Fedora 30
Kernel: 5.2.9-200.fc30.x86_6
vfio
[root@localhost ~]# dmesg | grep vfio
[ 0.000000] Command line: BOOT_IMAGE=(hd1,msdos2)/vmlinuz-5.2.9-200.fc30.x86_64 root=UUID=87faaa8b-9fad-488a-b180-5d6c516d19c9 ro resume=UUID=5799d52a-c9b5-4325-b228-a650ceda9066 rhgb iommu=1 amd_iommu=on rd.driver.blacklist=nouveau rd.driver.pre=vfio-pci
[ 0.000000] Kernel command line: BOOT_IMAGE=(hd1,msdos2)/vmlinuz-5.2.9-200.fc30.x86_64 root=UUID=87faaa8b-9fad-488a-b180-5d6c516d19c9 ro resume=UUID=5799d52a-c9b5-4325-b228-a650ceda9066 rhgb iommu=1 amd_iommu=on rd.driver.blacklist=nouveau rd.driver.pre=vfio-pci
[ 3.668937] vfio-pci 0000:0f:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
[ 3.682022] vfio_pci: add [10de:1187[ffffffff:ffffffff]] class 0x000000/00000000
[ 3.695089] vfio_pci: add [10de:0e0a[ffffffff:ffffffff]] class 0x000000/00000000
[ 4.855427] vfio-pci 0000:0f:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
nouveau is blacklisted
user0@localhost modprobe.d]$ cat blacklist.conf
blacklist nouveau
options nouveau modeset=0
vfio.conf exists
[user0@localhost modprobe.d]$ cat vfio.conf
options vfio-pci ids=10de:1187,10de:0e0a
ran dracut and grub2-mkconfig
the kernel command line arguments are set
[user0@localhost modprobe.d] cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="(sed ‘s, release .*$,g’ /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT=“console”
GRUB_CMDLINE_LINUX=“resume=UUID=5799d52a-c9b5-4325-b228-a650ceda9066 rhgb iommu=1 amd_iommu=on rd.driver.blacklist=nouveau rd.driver.pre=vfio-pci”
GRUB_DISABLE_RECOVERY=“true”
GRUB_ENABLE_BLSCFG=true
[root@localhost proc]# cat cmdline
BOOT_IMAGE=(hd1,msdos2)/vmlinuz-5.2.9-200.fc30.x86_64 root=UUID=87faaa8b-9fad-488a-b180-5d6c516d19c9 ro resume=UUID=5799d52a-c9b5-4325-b228-a650ceda9066 rhgb iommu=1 amd_iommu=on rd.driver.blacklist=nouveau rd.driver.pre=vfio-pci
iommu groups look right
[user0@localhost ~]$ ./iommu.sh | grep “IOMMU Group 1[6,7]”
IOMMU Group 16 0e:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM206GL [Quadro M2000] [10de:1430] (rev a1)
IOMMU Group 16 0e:00.1 Audio device [0403]: NVIDIA Corporation GM206 High Definition Audio Controller [10de:0fba] (rev a1)
IOMMU Group 17 0f:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104 [GeForce GTX 760] [10de:1187] (rev a1)
IOMMU Group 17 0f:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1)
looks like passthrough works
[root@localhost ~]# lspci -knn | grep -e NVIDIA -A 4
0e:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM206GL [Quadro M2000] [10de:1430] (rev a1)
Subsystem: NVIDIA Corporation Device [10de:1190]
Kernel driver in use: nvidia
Kernel modules: nouveau, nvidia_drm, nvidia
0e:00.1 Audio device [0403]: NVIDIA Corporation GM206 High Definition Audio Controller [10de:0fba] (rev a1)
Subsystem: NVIDIA Corporation Device [10de:1190]
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
0f:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104 [GeForce GTX 760] [10de:1187] (rev a1)
Subsystem: eVga.com. Corp. Device [3842:2766]
Kernel driver in use: vfio-pci
Kernel modules: nouveau, nvidia_drm, nvidia
0f:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1)
Subsystem: eVga.com. Corp. Device [3842:2766]
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel
10:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function [1022:145a
Unfortunately I have no real experience with VFIO (Only VMware ESXi). Can you attempt to passthrough the Nvidia GPU into a Linux VM and see if it gives any errors as well? Another thing I’d do is update the BIOS to the latest (P3.20 as of now) and make sure to explicitly enable IOMMU in the BIOS (Auto has issues)
Yes, can you give that a shot?
Gotcha. Thanks. I will try.
Also, I just don’t want to start creating duplicate questions everywhere. Do you know if I can also post this question elsewhere in the forum?
So, I got BIOS is 3.2 and a Linux VM.
The results are the same with a Linux VM. Linux install, and I can use VirtIO for the SSD but everything stops working when I change the SPICE gpu for NVIDIA.
I also start experiencing some kind of lock after trying the NVIDIA GPU and shutting-down the VM–Error 127. I must be missing something.
Error starting domain: internal error: Unknown PCI header type ‘127’
Traceback (most recent call last):
File “/usr/share/virt-manager/virtManager/asyncjob.py”, line 75, in cb_wrapper
callback(asyncjob, *args, **kwargs)
File “/usr/share/virt-manager/virtManager/asyncjob.py”, line 111, in tmpcb
callback(*args, **kwargs)
File “/usr/share/virt-manager/virtManager/libvirtobject.py”, line 66, in newfn
ret = fn(self, *args, **kwargs)
File “/usr/share/virt-manager/virtManager/domain.py”, line 1420, in startup
self._backend.create()
File “/usr/lib64/python3.7/site-packages/libvirt.py”, line 1089, in create
if ret == -1: raise libvirtError (‘virDomainCreate() failed’, dom=self)
libvirt.libvirtError: internal error: Unknown PCI header type ‘127’
So doing some searches, it may be the Nvidia card suffers the same thing the RX400/500 series suffers from the reset bug.
@2bitmarksman, thanks very much. Sorry for the late reply. Just went through this. This is nice info but I can never actually get to get Windows to show me my NVIDIA card because I am not able to get the VM to boot when I remove the default gpu and replace it with the NVIDIA.
Ah alright
I found a bunch of things here though: https://forum.level1techs.com/c/software/vfio
May need to compile the kernel with a patch…
Thanks for all the help:)
1 Like