Vfio-pci refuses to hook on to GPU

I followed the pinned guide exactly, and my RX580 refuses to be picked up by vfio-pci. I even tried to blacklist the amdgpu driver and it didn’t work.

The only difference in the guide is that I’m using systemd-boot, and used part of the guide from https://github.com/rmayobre/PopOs-VFIO-Tutorial/wiki/VFIO-Steps

Any help would be appreciated.

Have you seen this at the bottom of the pinned thread?

It seems to me that your initram isn’t getting updated.

I tried doing that, and I have come across something that I didn’t notice before that might be relevant.

Whenever I run update-initramfs -u, this shows up in the output

/etc/initramfs-tools/scripts/init-top/bind_vfio.sh: 4: 0000:09:00.1”: not found

When I run lspci -vnn, though, 9:00.1 is the audio controller for the 580 I want to passthrough.

Without seeing all of your scripts and output logs of stuff you run it is difficult to say. I for 1 know that one version of pop os didn’t copy kernel module to boot efi when doing update-initrams. Are you getting in output log the fact that kernel stub copied generated kernel image? Post information.

I will try and provide as much information as necessary

Here is my bind_vfio.sh file
#!/bin/sh

PREREQS=""
DEVS=“0000:09:00.0 0000:09:00.1”
for DEV in $DEVS;
do echo “vfio-pci” > /sys/bus/pci/devices/$DEV/driver_override
done

I ran these commands:

  • sudo kernelstub -a 'amd_iommu=on'
  • sudo kernelstub -a 'iommu=pt'
  • sudo kernelstub -a 'iommu=1'

And this is the update-initramfs -u output
update-initramfs: Generating /boot/initrd.img-5.4.0-7629-generic
cryptsetup: WARNING: Resume target cryptswap uses a key file
W: Possible missing firmware /lib/firmware/amdgpu/navi12_vcn.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/arcturus_vcn.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/navi12_smc.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/arcturus_smc.bin for module amdgpu
/etc/initramfs-tools/scripts/init-top/bind_vfio.sh: 4: 0000:09:00.1”: not found
kernelstub.Config : INFO Looking for configuration…
kernelstub : INFO System information:

OS:..................Pop!_OS 20.04
Root partition:....../dev/sdc3
Root FS UUID:........09225331-5019-444d-ba12-1d6b933efce9
ESP Path:............/boot/efi
ESP Partition:......./dev/sdc1
ESP Partition #:.....1
NVRAM entry #:.......-1
Boot Variable #:.....0000
Kernel Boot Options:.quiet loglevel=0 systemd.show_status=false splash amd_iommu=on iommu=pt iommu=1
Kernel Image Path:.../boot/vmlinuz-5.4.0-7629-generic
Initrd Image Path:.../boot/initrd.img-5.4.0-7629-generic
Force-overwrite:.....False

kernelstub.Installer : INFO Copying Kernel into ESP
kernelstub.Installer : INFO Copying initrd.img into ESP
kernelstub.Installer : INFO Setting up loader.conf configuration
kernelstub.Installer : INFO Making entry file for Pop!_OS
kernelstub.Installer : INFO Backing up old kernel
kernelstub.Installer : INFO Making entry file for Pop!_OS

Let me know if there is anymore information you need. Thanks!

Why are you manually binding?

you should specify your vfio_pci device ids in /etc/modprobe.d/vfio.conf, and use softdep amdgpu pre: vfio vfio-pci to make sure that vfio loads before amdgpu

Short version of the binding guide:

  1. lspci -knn, get your device ids, for everything.
  2. throw those ids into this line in /etc/modprobe.d/vfio.conf: options vfio-pci ids=[ids_here_comma_separated]
  3. preload vfio-pci: softdep amdgpu pre: vfio vfio-pci. Repeat this with nouveau and nvidia as well.
  4. make sure vfio is in your initramfs, put vfio, vfio_iommu_type1, vfio_virqfd, vfio_pci into /etc/initramfs-tools/modules
  5. rebuild initramfs. update-initramfs
  6. ???
  7. Profit

Unfortunately, it’s still not working. My GPU is still bound to amdgpu.

This is how I formatted vfio.conf:
options vfio-pci ids=1002:67df,1462:aaf0

softdep amdgpu pre: vfio vfio-pci

And this is what modules looks like:
vfio_pci
vfio
vfio_iommu_type1
vfio_virqfd

You rebooted, right?

Assuming you did, you might need to add vfio-pci.ids=1002:67df,1462:aaf0 to your kernel command line in /etc/default/grub and rebuild your grub config.

On my Ubuntu 19.10 system, my passthrough setup is working with the exact config I mentioned in my above post, so I’m not sure why yours isn’t.

If you continue to have issues, can you confirm that vfio is built, by running sudo modprobe vfio

If it returns with no output or error, it exists.

I use systemd-boot, so I can’t really do that lol.

Do you think switching to GRUB would potentially fix the issue?

Oh, hmmm.

I would figure out how to edit the kernel command line in systemd boot, and add that in there then…

The important thing is to pass the flag.

Sorry, I am not familiar with systemd-boot.

I highly doubt bootloader is the issue.

Finally, after 2 days of work, vfio-pci is finally binding to my GPU.

Thank you both for your help!

I added the device ids to systemd-boot by doing sudo kernelstub -a vfio-pci.ids=[ids here]

1 Like

You’re very welcome. I’m glad you got it!