Hey there,
I’m currently trying to set up a(nother, this is my third one on this machine) GPU passthrough for my RX 480, but I am running into a problem with vfio-pci, because no matter what I try, my GPU output turns on after every reboot, leading me to believe that it’s not actually isolated.
I’ll tell you what I did so far:
-
Enabeled VT-d and some other virtualization thing in my UEFI and told it to use the iGPU by default
-
Edited /etc/default/grub and added iommu=1 intel_iommu=on iommu=pt rd.driver.pre=vfio-pci to the GRUB_CMDLINE_LINUX line
(I decided to add iommu=1and iommu=pt just to make absolutely sure that iommu is working, because I wasn’t sure if those are just specific to AMD since I found conflicting information on what to put there)
-
Edited /etc/modprobe.d/vfio.conf and added the line options vfio-pci ids=1002:67df,1002:aaf0 , the IDs corresponding to the IDs of my RX480 and the sound device
-
Ran dracut --force, worked without any errors
-
Generated a new grub.cfg by running grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg (Thats where the F27 documentation says the grub.cfg for UEFI systems lies), completed without errors
Reboot the system, the monitor attached to my RX480 still turns on as Fedora loads.
Where did I fail? Any suggestions to find the issue?
Can you post full system specs?
Are you trying to pass through your primary GPU?
Also, post your IOMMU groups.
#!/bin/bash
for d in /sys/kernel/iommu_groups/*/devices/*; do
n=${d#*/iommu_groups/*}; n=${n%%/*}
printf 'IOMMU Group %s ' "$n"
lspci -nns "${d##*/}"
done
Have you tried making vfio-pci a dep of amdgpu?
/etc/modprobe.d/amdgpu.conf
softdep amdgpu pre: vfio vfio_pci
IOMMU Group 0 00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor Host Bridge/DRAM Registers [8086:191f] (rev 07)
IOMMU Group 10 00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (2) I219-V [8086:15b8] (rev 31)
IOMMU Group 11 03:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller [1b21:1242]
IOMMU Group 1 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 07)
IOMMU Group 1 01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/580] [1002:67df] (rev c7)
IOMMU Group 1 01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 580] [1002:aaf0]
IOMMU Group 2 00:02.0 VGA compatible controller [0300]: Intel Corporation HD Graphics 530 [8086:1912] (rev 06)
IOMMU Group 3 00:14.0 USB controller [0c03]: Intel Corporation Sunrise Point-H USB 3.0 xHCI Controller [8086:a12f] (rev 31)
IOMMU Group 4 00:16.0 Communication controller [0780]: Intel Corporation Sunrise Point-H CSME HECI #1 [8086:a13a] (rev 31)
IOMMU Group 5 00:17.0 SATA controller [0106]: Intel Corporation Sunrise Point-H SATA controller [AHCI mode] [8086:a102] (rev 31)
IOMMU Group 6 00:1b.0 PCI bridge [0604]: Intel Corporation Sunrise Point-H PCI Root Port #17 [8086:a167] (rev f1)
IOMMU Group 7 00:1c.0 PCI bridge [0604]: Intel Corporation Sunrise Point-H PCI Express Root Port #1 [8086:a110] (rev f1)
IOMMU Group 8 00:1d.0 PCI bridge [0604]: Intel Corporation Sunrise Point-H PCI Express Root Port #9 [8086:a118] (rev f1)
IOMMU Group 9 00:1f.0 ISA bridge [0601]: Intel Corporation Sunrise Point-H LPC Controller [8086:a145] (rev 31)
IOMMU Group 9 00:1f.2 Memory controller [0580]: Intel Corporation Sunrise Point-H PMC [8086:a121] (rev 31)
IOMMU Group 9 00:1f.3 Audio device [0403]: Intel Corporation Sunrise Point-H HD Audio [8086:a170] (rev 31)
IOMMU Group 9 00:1f.4 SMBus [0c05]: Intel Corporation Sunrise Point-H SMBus [8086:a123] (rev 31)
Here are my IOMMU groups, should work just fine looking at that
Relevant specs are:
i7 6700k stock
16gb Gskill Ripjaws V
ASUS Maximus VIII Ranger
XFX RX 480 Black Edition
Got one monitor attached to IGPU, one to my RX 480
Fedora 27 installed on a SATA SSD, let me know if you need anything else 
Haven’t tried that yet, I’ll give it a try, run dracut again and reboot, i’ll get back to you^^
Wait, is it vfio_pci or vfio-pci?
I might have messed that up while adding it to the grub.cfg
Tried the softdep thing, still boots up 
Alright. Let’s see here:
IOMMU groups look good.
So that’s the odd thing. I’ve seen both and different config areas seem to respond to different configuration. I’m not really sure of the reason for that, but vfio_pci is what I use for modprobe, vfio-pci is what I use for grub command line.
Drop iommu=1 from your commandline. You only need pt or 1.
Try using /etc/grub2-efi.cfg as your target for grub2-mkconfig. That’s a symlink to the proper grub config file.
Another thing you could try is blacklist amdgpu in modprobe.d
See if any of this helps. I’m still trying to figure out exactly what the problem is.
Thanks a ton man, will try
1 Like
sudo grub2-mkconfig -o /etc/grub2-efi.cfg returns
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.15.14-300.fc27.x86_64
Found initrd image: /boot/initramfs-4.15.14-300.fc27.x86_64.img
Found linux image: /boot/vmlinuz-4.13.9-300.fc27.x86_64
Found initrd image: /boot/initramfs-4.13.9-300.fc27.x86_64.img
Found linux image: /boot/vmlinuz-0-rescue-f41323c99fb746ebabe9e8b60a4fd81a
Found initrd image: /boot/initramfs-0-rescue-f41323c99fb746ebabe9e8b60a4fd81a.img
device-mapper: remove ioctl on osprober-linux-sdb1 failed: Device or resource busy
Command failed
done
sdb 8:16 0 2.7T 0 disk
└─sdb1 8:17 0 2.7T 0 part
└─osprober-linux-sdb1 253:3 0 2.7T 1 dm
Not sure what’s up with that, it’s my 3TB WD Red
Also tells me already mounted/mountpoint busy when I try to open the lower one, very weird
Try running, as root, the following:
dmsetup remove osprober-linux-sdb1
$ sudo dmsetup remove osprober-linux-sdb1
[sudo] password for uniqueactive:
device-mapper: remove ioctl on osprober-linux-sdb1 failed: Device or resource busy
Command failed
Nvm, unmounted again, executed without error this time
Command failed
That’s odd.
So, you’re probably good on the grub2-mkconfig front. A reboot will fix the stale device map.
Okay so after I unmounted and tried again it executed and I was able to grub2-mkconfig to the right place, I’ll see if I get a different result this time around on reboot
1 Like
Alright, still both monitors lighting up with Fedora^^
How would I go about blacklisting amdgpu?
in /etc/modprobe.d/amdgpu.conf, put the following:
blacklist amdgpu
Then you need to dracut -f --kver $(uname -r) again.
Just to verify, your packages are up to date, right?
1 Like
Everything up to date, checked everything again, regenerated everything again, i’ll reboot and see
1 Like
Blacklisting amdgpu did the trick, seems to be working now, thank you a ton for your help!
Also, the /dev/sdb shenanigans are still happening, how would I go about fixing that for good? 
Sorry for abusing your help^^
Should I just try to set up some proper mounting points for my additional drives?
EDIT: Seems like it’s fine after setting up my mount points, thank you!
1 Like
Glad to hear it.
I’m not really sure. You could try passing the --force flag to the command I mentioned above (dmsetup remove --force osprober-linux-sdb1) and see if that nets you any results.
That’s probably the best option if it’s not a removable drive.
Don’t worry, I keep coming back for more!
All seriousness though, I’m quite experienced with passthrough and whatnot, so I do enjoy sharing my knowledge with people.
1 Like
