Return to Level1Techs.com

Identical GPU Passthrough - Ubuntu

#1

So I have two RX 580s and I want to pass one through to a VM. I’ve been having an absolute nightmare getting this to work with Ubuntu.

I followed the instructions here:

My IOMMU is set properly.

My /etc/initramfs-tools/moduleslooks like this:

softdep amdgpu pre: vfio vfio_pci

vfio
vfio_iommu_type1
vfio_virqfd
options vfio_pci ids=1002:67df, 1002:aaf0
vfio_pci ids=1002:67df, 1002:aaf0
vfio-pci
amdgpu

My /etc/modules file looks like this:

8812au
vfio
vfio_iommu_type1
vfio_pci ids=1002:67df, 1002:aaf0

My /etc/modprobe.d/amdgpu.conf file looks like this:

softdep amdgpu pre: vfio vfio_pci

My /etc/modprobe.d/vfio_pci.conf file looks like this:

options vfio_pci ids=1002:67df, 1002:aaf0

I also read on the Arch Wiki that I need to make a script to identify which GPU to use. So I have a file in /usr/bin/vfio-pci-override.sh:

#!/bin/sh

for i in /sys/bus/pci/devices/*/boot_vga; do
	if [ $(cat "$i") -eq 0 ]; then
		GPU="${i%/boot_vga}"
		AUDIO="$(echo "$GPU" | sed -e "s/0$/1/")"
		echo "vfio-pci" > "$GPU/driver_override"
		if [ -d "$AUDIO" ]; then
			echo "vfio-pci" > "$AUDIO/driver_override"
		fi
	fi
done

modprobe -i vfio-pci

Questions are as follows:

  1. How do I make this scipt work with initramfs on Ubuntu?
  2. Do I need to remove the manual assignment further up in /etc/initramfs-tools/modules and /etc/modules and /etc/modprobe.d/vfio_pci.conf files?

Thanks for the help.

0 Likes
#2

With identical GPUs it becomes an issue, but I think you already know that. I came across this page when I was trying to do the same thing. I never got it to work, but maybe you will have better luck.

0 Likes
#3

Should ids= be two strings separated by a space, or one string without a space?

Original src 17.04 link has no space.

0 Likes
#4

Well, the thing is, I don’t even think I should have those there because both cards have the same ID.
So, even if this worked, it’d probably mess up by binding the VFIO driver to both cards.

I guess what I’m confused about is where to call the script selecting the non-boot card. Somewhere in the /etc/initramfs-tools/modules file maybe?

0 Likes
#5

Just a thought. Have you tried using the other script in the archwiki to specify the gpu to be passed through instead of relying on the script that attempts to stub out all but the boot-vga gpu? Maybe that would work. As far as whether or not you need to remove the other modifications upstream in your modules configs, amdgpu.conf and vfio_pci.conf… Like I said before, I never actually got this to work so I’m unsure, however, it seems like having those commands in those configs will ultimately bind the vfio_pci driver to both of those cards.

Another thought is to try this in reverse. For instance, you could allow the system to initially stub out both GPUs with the VFIO driver and then use a downstream script to force the boot GPU to use the correct AMDGPU driver. This is all theoretical but I’d be super interested to find out if this works.

0 Likes
#6

I have not, because I’m not really sure where/how to call the script on Ubuntu. Arch uses mkinitcpio, and the guide on the wiki says to call a hook there, but I’ve no idea how to do that with initramfs instead.

This thought did occur to me, because I think one of the issues I’m having is that the amdgpu driver set is loading to quickly and it’s super difficult to get the vfio to load. Although I’d have thought that the softep line would prevent that.

0 Likes
#7

Have you tried unbinding post boot? Then rebind to VFIO. I have heard mixed reports about this, but it might work.

echo "device id" > /sys/bus/pci/devices/"device id"/driver/unbind

modprobe vfio-pci

echo "device id" > /sys/bus/pci/drivers/vfio-pci/bind

Edit-
Also, you probably will need to do this with X not running.

0 Likes
#8

So that works, and allowed me to verify that it is indeed 0000:09:00.0 that is the second, non-display card. However, I can’t go back to the desktop (At least using Ctrl+Alt+F1) after binding it to vfio-pci. But lspci -nnv shows it has indeed bound it to vfio-pci.

Just not sure how to get from this to having it bound at startup/having my desktop work. Do I need to restart X?

0 Likes
#9

Yes.

How to restart X depends on your setup. Often you can just restart your display manager(GDM for gnome KDM for KDE) and X will start.

0 Likes
#10

Hmm, unfortunately no dice there. Using sddm, after I bound the card to vfio-pci driver, and run sudo service sddm restart, everything just hangs.

Also, stopping sddm before doing this kicks me out of the terminal and puts me at the boot screen for some reason.

0 Likes
#11

Are you binding the boot GPU or the second GPU? And do you have anything plugged into the card you are binding to vfio?

0 Likes
#12

Second GPU (trying the other makes the entire screen go black, so I’m pretty sure I have the right one). And no, I don’t have anything plugged into the 2nd card.

0 Likes
#13

So the entire process you are using is?-

  1. switch to a non-desktop tty
  2. login to tty
  3. un-bind the gpu
  4. modprobe vfio
  5. rebind to vfio
  6. try to restart sddm
  7. (doesn’t work because system hangs at #5) switch back to desktop tty

Try stopping the sddm service after step 2.

1 Like
#14

Ok, trying that just gives me a blinking cursor:

IMG_20190214_123746
IMG_20190214_123746.jpg4032×3024 2.67 MB

Edit: but going back to desktop worked and vfio-pci is still bound!!

1 Like
#15

You should be able to automate it fairly easily. Make it a script and stick it in crontab with @reboot.

0 Likes
#16

Oh my god, isnt there a guide on the whole internet that can properly show us how to bind to vfio one of two identical gpus in UBUNTU ?

1 Like
#17

Shit i found it, and it worked, i managed to give one of two identical GPUs under Ubuntu to VFIO
Here is the guide

1 Like
#18

what terms were you searching for that you had trouble finding it?

0 Likes
#19

i cant remember, but the link is there; we should somehow pinned or something. This is big. It seems a lot of People are having Trouble VFIOing similar gpus, and this worked for me. So i think ist a pretty big deal.

The Thing is, it doesnt have any Dual GPU or Similar GPU tags in the title, so anyone havingthe dual/similar gpu Problem, might not find it.

Also, ist written with another Linux OS in mind, so that doesnt help either.

0 Likes