My /etc/modprobe.d/amdgpu.conf file looks like this:
softdep amdgpu pre: vfio vfio_pci
My /etc/modprobe.d/vfio_pci.conf file looks like this:
options vfio_pci ids=1002:67df, 1002:aaf0
I also read on the Arch Wiki that I need to make a script to identify which GPU to use. So I have a file in /usr/bin/vfio-pci-override.sh:
#!/bin/sh
for i in /sys/bus/pci/devices/*/boot_vga; do
if [ $(cat "$i") -eq 0 ]; then
GPU="${i%/boot_vga}"
AUDIO="$(echo "$GPU" | sed -e "s/0$/1/")"
echo "vfio-pci" > "$GPU/driver_override"
if [ -d "$AUDIO" ]; then
echo "vfio-pci" > "$AUDIO/driver_override"
fi
fi
done
modprobe -i vfio-pci
Questions are as follows:
How do I make this scipt work with initramfs on Ubuntu?
Do I need to remove the manual assignment further up in /etc/initramfs-tools/modules and /etc/modules and /etc/modprobe.d/vfio_pci.conf files?
With identical GPUs it becomes an issue, but I think you already know that. I came across this page when I was trying to do the same thing. I never got it to work, but maybe you will have better luck.
Well, the thing is, I don’t even think I should have those there because both cards have the same ID.
So, even if this worked, it’d probably mess up by binding the VFIO driver to both cards.
I guess what I’m confused about is where to call the script selecting the non-boot card. Somewhere in the /etc/initramfs-tools/modules file maybe?
Just a thought. Have you tried using the other script in the archwiki to specify the gpu to be passed through instead of relying on the script that attempts to stub out all but the boot-vga gpu? Maybe that would work. As far as whether or not you need to remove the other modifications upstream in your modules configs, amdgpu.conf and vfio_pci.conf… Like I said before, I never actually got this to work so I’m unsure, however, it seems like having those commands in those configs will ultimately bind the vfio_pci driver to both of those cards.
Another thought is to try this in reverse. For instance, you could allow the system to initially stub out both GPUs with the VFIO driver and then use a downstream script to force the boot GPU to use the correct AMDGPU driver. This is all theoretical but I’d be super interested to find out if this works.
I have not, because I’m not really sure where/how to call the script on Ubuntu. Arch uses mkinitcpio, and the guide on the wiki says to call a hook there, but I’ve no idea how to do that with initramfs instead.
This thought did occur to me, because I think one of the issues I’m having is that the amdgpu driver set is loading to quickly and it’s super difficult to get the vfio to load. Although I’d have thought that the softep line would prevent that.
So that works, and allowed me to verify that it is indeed 0000:09:00.0 that is the second, non-display card. However, I can’t go back to the desktop (At least using Ctrl+Alt+F1) after binding it to vfio-pci. But lspci -nnv shows it has indeed bound it to vfio-pci.
Just not sure how to get from this to having it bound at startup/having my desktop work. Do I need to restart X?
Second GPU (trying the other makes the entire screen go black, so I’m pretty sure I have the right one). And no, I don’t have anything plugged into the 2nd card.
i cant remember, but the link is there; we should somehow pinned or something. This is big. It seems a lot of People are having Trouble VFIOing similar gpus, and this worked for me. So i think ist a pretty big deal.
The Thing is, it doesnt have any Dual GPU or Similar GPU tags in the title, so anyone havingthe dual/similar gpu Problem, might not find it.
Also, ist written with another Linux OS in mind, so that doesnt help either.
