I’ve tried multiple guides on Fedora 26, 29 and Ubuntu 18.10. I might say I have a special setup: I have two identical RX 580 that I want to passthrough to 2 separate VMs, one GPU for each VM. I also have a GT 710 for my host video out. Regardless of how I configure the files specified in each guide, my host will still grab my 580s, and when I try to start a VM with a GPU my whole system will freeze. That’s the farthest I’ve gotten. I would appreciate any help and any pointers to some resources I can use to solve my dilemma
Here’s a full Spec:
Ryzen 5 1600
MSI x370 SLI-Plus
16GB RAM
2x Gigabyte Gaming RX 580 8GB
1x MSI GT 710 2GB
Can you show us the output of lspci -nnk, what your IOMMU groups are, and config files you have changed for passthrough setup?
1 Like
lspci -nnk
// MSI GT 710 2GB
24:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208B [GeForce GT 710] [10de:128b] (rev a1)
Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:8c93]
Kernel driver in use: nouveau
Kernel modules: nouveau
24:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)
Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:8c93]
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
// Gigabyte Gaming RX 580 8GB
29:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] [1002:67df] (rev e7)
Subsystem: Gigabyte Technology Co., Ltd Device [1458:22fc]
Kernel driver in use: amdgpu
Kernel modules: amdgpu
29:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 580] [1002:aaf0]
Subsystem: Gigabyte Technology Co., Ltd Device [1458:aaf0]
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
// Gigabyte Gaming RX 580 8GB
2a:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] [1002:67df] (rev e7)
Subsystem: Gigabyte Technology Co., Ltd Device [1458:22fc]
Kernel driver in use: amdgpu
Kernel modules: amdgpu
2a:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 580] [1002:aaf0]
Subsystem: Gigabyte Technology Co., Ltd Device [1458:aaf0]
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
IOMMU Groups
// MSI GT 710 2GB
IOMMU Group 13 24:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208B [GeForce GT 710] [10de:128b] (rev a1)
IOMMU Group 13 24:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)
// Gigabyte Gaming RX 580 8GB
IOMMU Group 14 29:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] [1002:67df] (rev e7)
IOMMU Group 14 29:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 580] [1002:aaf0]
// Gigabyte Gaming RX 580 8GB
IOMMU Group 15 2a:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] [1002:67df] (rev e7)
IOMMU Group 15 2a:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 580] [1002:aaf0]
Changed Files
// Original /etc/default/grub
GRUB_CMDLINE_LINUX=“resume=UUID=6fd44923-960c-4e2a-ad61-7d1f0088bb40 rhgb quiet”
// Modified /etc/default/grub
GRUB_CMDLINE_LINUX=“resume=UUID=6fd44923-960c-4e2a-ad61-7d1f0088bb40 rhgb quiet iommu=1 amd_iommu=on rd.driver.pre=vfio-pci”
// Created /etc/modprobe.d/vfio.conf
// 1002:67df -> Gigabyte Gaming RX580 8GB
// 1002:aaf0 -> GPU Audio Device
options vfio-pci ids=1002:67df,1002:aaf0
After all this, I ran
sudo grub2-mkconfig -o /etc/grub2-efi.cfg
uname -r
sudo dracut -f --kver 4.19.10-300.fc29.x86_64
I did try this, but this wasn’t in Wendell’s Guide and it didn’t seem to help
// Created /etc/modprobe.d/amdgpu.conf
softdep amdgpu pre: vfio-pci
This is all for Fedora 29. I also have 3 monitors total, one for each card.
Thanks. Everything looks more or less ok so far.
Did you reboot after updating your initramfs?
Also, what GPU is getting the motherboard post output? One of the 580s or the 710?
1 Like
I reinstalled Fedora 29 and started over with everything mentioned above (including amdgpu.conf), and now when I boot into Fedora, it’s hangs on
started dracut cmdline hook
VFIO - User Level meta-driver version: 0.3
vio-pci 0000:29:00.0 vgaarb: changed VGA decodes: olddecodes=io+member,decodes=io+member:owns=io+mem
I did reboot after dracut command, and my top slot RX 580 gets motherboard post output.
I tried booting with monitors unplugged from my RX 580s, but same result.
There’s one of your problem. Try moving the gt710 to top slot.
So both my RX 580s are dual slot, and my GT 710 is single slot. The only way all the cards fit in my case is if both RX 580s are in the top two slots, and the GT 710 is the the last slot. But I’ll try that configuration to see if that solves the issue. I’d much rather have my current configuration, but I suppose some things can’t be helped.
Some motherboards have an option to set the boot GPU. That is the ideal solution.
Ah. :/. Figured that would come up as soon as I posted. That.
I’m gonna agree with @TheCakeIsNaOH
I’ll look into it, thanks!
So switching the placement of the GPUS helps. Thank you!
1 Like
An update:
I was able to boot Fedora 29 without moving around my GPUs. Here’s what I did
// Original /etc/default/grub
GRUB_CMDLINE_LINUX=“resume=UUID=6fd44923-960c-4e2a-ad61-7d1f0088bb40 rhgb quiet iommu=1 amd_iommu=on rd.driver.pre=vfio-pci”
// Modified /etc/default/grub
GRUB_CMDLINE_LINUX=“resume=UUID=6fd44923-960c-4e2a-ad61-7d1f0088bb40 rhgb quiet iommu=1 amd_iommu=on rd.driver.pre=vfio-pci video=efifb:off”
// Created /etc/X11/xorg.conf.d/driver-nouveau.conf
Section “Device”
Identifier “gk208b”
Driver “nouveau”
EndSection
This allows me to boot with my host GT 710 in the bottom slot, whilst booting with my RX 580 in the top slot
2 Likes
Ah, glad you’ve got it working!
1 Like