Help - Fedora 28 vs. Arch documentation VFIO with same GPUs

Operating Systems & Open Source Linux
, ,
1

So, background:

X470 taichi
R7-2700x
2x Vega 64

i’m trying to to apply this ARCH guide to Fedora 28, as i have multiple GPUs of the same type (Vega) - and this seems to be the only guide i’ve found thus far that deals with that situation (via the override script).

My Vegas are in different IOMMU groups:

IOMMU Group 16 30:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Vega 10 XT [Radeon RX Vega 64] [1002:687f] (rev c1)
IOMMU Group 17 30:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:aaf8]

IOMMU Group 20 33:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Vega 10 XT [Radeon RX Vega 64] [1002:687f] (rev c1)
IOMMU Group 21 33:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:aaf8]

My issue is when i get to part 5 - configuring vfio-pci-override

I’ve got the script, made it executable (using my secondary GPU’s device IDs), but Fedora evidently doesn’t use initramfs and I really don’t know how to get the vfio module loaded using the override script.

I tried to load it via a script in /etc/modprobe.d as per the instructions but it does nothing.

I’ve regenerated the boot environment using:
dracut -f --kver uname -r
grub2-mkconfig > /etc/grub2-efi.cfg

But i really don’t know what I’m doing from here.

It feels like i’m pretty close and just need to get the vfio module loaded before AMDGPU (?) but not sure where to go from here… any ideas?

:slight_smile:

2

Fedora uses dracut for its initramfs. what you need to do is create a file in /etc/dracut.conf.d called vfio.conf with the lines below:

add_drivers+="vfio vfio_iommu_type1 vfio_pci vfio_virqfd"
install_items+="/sbin/vfio-pci-override.sh /usr/bin/find /usr/bin/dirname"

Then regenerate dracut with dracut -f --kver uname -r and run grub2-mkconfig again. This should load your script in the initramfs and attach to your GPU.

2 Likes
3

Awesome, thanks. I was aware it used dracut instead but no idea how it worked.

If this all works I’ll try and get a Fedora same GPU tutorial written from this.

Cheers

1 Like
4

Excellent!

Success!

Well. So far. I haven’t done the windows VM side of things yet, but i can see via the GPU tach activity lights (and the fan speed) that the second card is no longer under the control of AMDGPU. :slight_smile:

Also:
33:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Vega 10 XT [Radeon RX Vega 64] [1002:687f] (rev c1)
Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:6b76]
Kernel driver in use: vfio-pci
Kernel modules: amdgpu
33:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:aaf8]
Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:aaf8]
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel

:slight_smile:

Thanks so much.

1 Like
5

Glad you got it working, happy to help :smiley:

6

So, i did a brief test of trying to create a new VM, allocated the GPU to it, and trying to install windows (i know, more stuff required).

I did manage to get UEFI output on my second card. :slight_smile:

Guessing at this point i’ll need to pass through a dedicated mouse and keyboard which i don’t currently have, but things are looking up.

I’m not sure if i’ll do looking glass, as i can probably just use steam link, or steam remote play for what i want to do… :slight_smile:

edit:
alternatively: could i install the VM using the regular spice driver, set up RDP, then remove the virtual video card, RDP in, install AMD drivers via RDP, etc? My main use for this i think will be gaming via either steam link or via steam game streaming to the host over the network (if it works).

7

So for some reason i did an update and now VFIO is no longer a thing and the kernel/AMDGPU has reclaimed my second card.

Looking into it… weird…

8

So, at some point… definitely vfio has gone away and is no longer bound to my GPU. amdgpu has taken control of it, all i have done is install updates. grub entry still looks as i left it.

my vfio script is in the initrd (lsinitrd shows it present) but for whatever reason the override isn’t working. i’m not sure what else to check…