Ubuntu 18.04 – VFIO PCIe Passthrough

gcardinal · April 29, 2018, 3:00pm

I’m running Threadripper 1920x with Vega 56 as host and Pro WX 7100 as guest GPU.

Trying this on 18.04 LTS

I did try to follow guide on forum: Ubuntu 17.04 – VFIO PCIe Passthrough

Installed 4.16 with ukuu
in /etc/default/grub changed line to
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=1 amd_iommu=on"
Run ls-iommu.sh

IOMMU Group 16 0a:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon Pro WX 7100] [1002:67c4]
IOMMU Group 16 0a:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 580] [1002:aaf0]

Change files accordingly
4.1 /etc/initramfs-tools/modules

softdep amdgpu pre: vfio vfio_pci
vfio
vfio_iommu_type1
vfio_virqfd
options vfio_pci ids=1002:67c4,1002:aaf0
vfio_pci ids=1002:67c4,1002:aaf0
vfio_pci
amdgpu

4.2 /etc/modules

vfio
vfio_iommu_type1
vfio_pci ids=1002:67c4,1002:aaf0

4.3 /etc/modprobe.d/amdgpu.conf

softdep amdgpu pre: vfio vfio_pci

4.4 /etc/modprobe.d/vfio_pci.conf

options vfio_pci ids=1002:67c4,1002:aaf0

Run lspci -nnv |less

0a:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon Pro WX 7100] [1002:67c4] (prog-if 00 [VGA controller])
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon Pro WX 7100] [1002:0b0d]
        Flags: bus master, fast devsel, latency 0, IRQ 123, NUMA node 0
        Memory at d0000000 (64-bit, prefetchable) [size=256M]
        Memory at e0000000 (64-bit, prefetchable) [size=2M]
        I/O ports at 7000 [size=256]
        Memory at efd00000 (32-bit, non-prefetchable) [size=256K]
        Expansion ROM at efd40000 [disabled] [size=128K]
        Capabilities: <access denied>
        Kernel driver in use: amdgpu
        Kernel modules: amdgpu

0a:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 580] [1002:aaf0]
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 580] [1002:aaf0]
        Flags: bus master, fast devsel, latency 0, IRQ 143, NUMA node 0
        Memory at efd60000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: <access denied>
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel

But still stuck with:

        Kernel driver in use: amdgpu
        Kernel modules: amdgpu

Being a super noob, I’m very lost at this stage. Help if hugely appreciated thx!

SgtAwesomesauce · April 29, 2018, 9:25pm

Try adding vfio_pci.ids=1002:67c4,1002:aaf0 to the kernel command line.

You did rebuild both grub and initramfs, correct?

gcardinal · May 3, 2018, 7:51am

thx for reply, I did change /etc/default/grub to:
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=1 amd_iommu=on vfio_pci.ids=1002:67c4,1002:aaf0"
and run:
sudo update-grub and sudo dracut -f for the initramfs

Unfortunately same result.

gcardinal · May 3, 2018, 8:02am

d0h It helps to run the correct stuff, after running sudo update-initramfs -u it shows up correctly:

        Kernel driver in use: vfio-pci
        Kernel modules: amdgpu

Thank you so much for the help!

hondaman · May 4, 2018, 3:34am

I got bit by this same hiccup too following the same guide. Glad you fixed it.

SgtAwesomesauce · May 4, 2018, 9:49pm

Glad to help.

For those in the future, troubleshooting steps for kernel drivers not loading properly are the following:

check modprobe.d and make sure your configuration (including device ids, pcie paths and whatnot) is how you want it.
check your distros initramfs/initcpio config tool to make sure the proper modules are being included.
check to make sure you’ve updated your initramfs/initcpio and bootloader config.
ask the forums.

Usually, 1 2 and 3 can solve your problem a lot faster than 4!

BillFleming · June 11, 2018, 12:54pm

I am having issues getting past the first reboot stage of vfio setup where you try to grab the GPU from the driver.

Using the following: Kubuntu 18.04 with kernel 4.17 with open source amdgpu running
Asrock X370 Pro Gaming with 1600x (bios 4.64 with agesa iommu updates and iommu turned on)
2x16 GB RAM
Vega 64 in slot 2 (16x PCI-E slot from CPU)
RX 460 in slot 5 (8x slot from CPU)

File setup:
4.1 /etc/initramfs-tools/modules

softdep amdgpu pre: vfio vfio_pci
vfio
vfio_iommu_type1
vfio_virqfd
options vfio_pci ids=1002:687f,1002:aaf8
vfio_pci ids=1002:687f,1002:aaf8
vfio_pci
amdgpu

4.2 /etc/modules

vfio
vfio_iommu_type1
vfio_pci ids=1002:687f,1002:aaf8

4.3 /etc/modprobe.d/amdgpu.conf

softdep amdgpu pre: vfio vfio_pci

4.4 /etc/modprobe.d/vfio_pci.conf

options vfio_pci ids=1002:687f,1002:aaf8

kernel param settings:

quiet splash modprobe.blacklist=amdgpu vfio_pci.ids=1002:687f,1002:aaf8 iommu=pt amd_iommu=on

I have tried with/without the blacklist, various combinations of other settings but when I reboot I always get a black screen.
It seems to get farther in the boot process with the blacklist off though.

Should I physically switch the GPUs with each other? Is there anything else that I am missing?

bill@Bill-X370:~$ sudo dmesg | grep AMD-Vi
[sudo] password for bill: 
[    0.743141] AMD-Vi: IOMMU performance counters supported
[    0.745359] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40
[    0.745360] AMD-Vi: Extended features (0xf77ef22294ada):
[    0.745362] AMD-Vi: Interrupt remapping enabled
[    0.745362] AMD-Vi: virtual APIC enabled
[    0.745452] AMD-Vi: Lazy IO/TLB flushing enabled
bill@Bill-X370:~$ ./ls-iommu.sh 
IOMMU Group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 10 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 11 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
IOMMU Group 12 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 59)
IOMMU Group 12 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 13 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0 [1022:1460]
IOMMU Group 13 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1 [1022:1461]
IOMMU Group 13 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2 [1022:1462]
IOMMU Group 13 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3 [1022:1463]
IOMMU Group 13 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4 [1022:1464]
IOMMU Group 13 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5 [1022:1465]
IOMMU Group 13 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric Device 18h Function 6 [1022:1466]
IOMMU Group 13 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7 [1022:1467]
IOMMU Group 14 01:00.0 Non-Volatile memory controller [0108]: Toshiba America Info Systems Device [1179:0115] (rev 01)
IOMMU Group 15 03:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b9] (rev 02)
IOMMU Group 15 03:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b5] (rev 02)
IOMMU Group 15 03:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b0] (rev 02)
IOMMU Group 15 1d:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU Group 15 1d:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU Group 15 1d:03.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU Group 15 1d:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU Group 15 1e:00.0 Ethernet controller [0200]: Aquantia Corp. AQC108 NBase-T/IEEE 802.3bz Ethernet Controller [AQtion] [1d6a:d108] (rev 02)
IOMMU Group 15 20:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 02)
IOMMU Group 15 21:00.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1184]
IOMMU Group 15 26:01.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1184]
IOMMU Group 15 26:03.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1184]
IOMMU Group 15 26:05.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1184]
IOMMU Group 15 26:07.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1184]
IOMMU Group 15 27:00.0 Network controller [0280]: Intel Corporation Device [8086:24fb] (rev 10)
IOMMU Group 15 2a:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
IOMMU Group 16 2d:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1470] (rev c1)
IOMMU Group 17 2e:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1471]                                                            
IOMMU Group 18 2f:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Vega [Radeon RX Vega] [1002:687f] (rev c1)                 
IOMMU Group 19 2f:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:aaf8]                                                      
IOMMU Group 1 00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]                        
IOMMU Group 20 30:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Baffin [Radeon RX 460] [1002:67ef] (rev e5)                
IOMMU Group 20 30:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:aae0]                                                      
IOMMU Group 21 31:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:145a]                                         
IOMMU Group 22 31:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor [1022:1456]
IOMMU Group 23 31:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) USB 3.0 Host Controller [1022:145c]           
IOMMU Group 24 32:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:1455]                                         
IOMMU Group 25 32:00.2 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)                     
IOMMU Group 26 32:00.3 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) HD Audio Controller [1022:1457]                 
IOMMU Group 2 00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]                        
IOMMU Group 3 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge [1022:1452]                
IOMMU Group 4 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge [1022:1452]                
IOMMU Group 5 00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]                        
IOMMU Group 6 00:03.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]                        
IOMMU Group 7 00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge [1022:1452]                
IOMMU Group 8 00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge [1022:1452]                
IOMMU Group 9 00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]

BillFleming · June 18, 2018, 2:54am

To anyone reading this I got past this issue just by swapping the GPUs and making sure it booted through the BIOS on the RX460 in the “main” PCIe slot.

2d:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Baffin [Radeon RX 460] [1002:67ef] (rev e5) (prog-if 00 [VGA controller])
        Subsystem: Tul Corporation / PowerColor Baffin [Radeon RX 460/560D / Pro 450/455/460/560] [148c:2385]
        Flags: bus master, fast devsel, latency 0, IRQ 80
        Memory at e0000000 (64-bit, prefetchable) [size=256M]
        Memory at f0000000 (64-bit, prefetchable) [size=2M]
        I/O ports at f000 [size=256]
        Memory at fd600000 (32-bit, non-prefetchable) [size=256K]
        Expansion ROM at 000c0000 [disabled] [size=128K]
        Capabilities: <access denied>
        Kernel driver in use: amdgpu
        Kernel modules: amdgpu

2d:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:aae0]
        Subsystem: Tul Corporation / PowerColor Device [148c:aae0]
        Flags: bus master, fast devsel, latency 0, IRQ 91
        Memory at fd660000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: <access denied>
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel

30:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Vega [Radeon RX Vega] [1002:687f] (rev c1) (prog-if 00 [VGA controller])
        Subsystem: ASUSTeK Computer Inc. Vega 10 XT [Radeon RX Vega 64] [1043:04c4]
        Flags: fast devsel, IRQ 5
        Memory at c0000000 (64-bit, prefetchable) [disabled] [size=256M]
        Memory at d0000000 (64-bit, prefetchable) [disabled] [size=2M]
        I/O ports at e000 [disabled] [size=256]
        Memory at fd300000 (32-bit, non-prefetchable) [disabled] [size=512K]
        Expansion ROM at fd380000 [disabled] [size=128K]
        Capabilities: <access denied>
        Kernel driver in use: vfio-pci
        Kernel modules: amdgpu

30:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:aaf8]
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:aaf8]
        Flags: fast devsel, IRQ 11
        Memory at fd3a0000 (32-bit, non-prefetchable) [disabled] [size=16K]
        Capabilities: <access denied>
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel

SgtAwesomesauce · June 18, 2018, 4:15pm

That’s the correct solution to your problem. Sorry I didn’t notice your first post last week, otherwise I’d have responded then.

flid · September 30, 2018, 12:05pm

Hello @SgtAwesomesauce
I’ve spent a week trying to make my setup work, no luck. I have a similar setup to what @BillFleming has above:

ROG CROSSHAIR VI HERO motherboard with the latest BIOS
HP AMD Radeon HD 8490 DP (1GB) PCIe x16 in a top slot
ASUS Vega 64 OC in a second videocard slot (8x)
CPU: Ryzen 2700x 
Tried with both default 4.15 and latest 4.18.11 kernel

All the same configs for grub and initramfs. It all looks good:

anton@anton-vhost:~$ lspci -knn | grep 0a:00 -A3
    0a:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Caicos XTX [Radeon HD 8490 / R5 235X OEM] [1002:6771]
            Subsystem: Hewlett-Packard Company Caicos XTX [Radeon HD 8490 / R5 235X OEM] [103c:90b8]
            Kernel driver in use: radeon
            Kernel modules: radeon
    0a:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Caicos HDMI Audio [Radeon HD 6450 / 7450/8450/8490 OEM / R5 230/235/235X OEM] [1002:a...
            Subsystem: Hewlett-Packard Company Caicos HDMI Audio [Radeon HD 6450 / 7450/8450/8490 OEM / R5 230/235/235X OEM] [103c:aa98]
            Kernel driver in use: snd_hda_intel
            Kernel modules: snd_hda_intel
    anton@anton-vhost:~$ lspci -knn | grep 0d:00 -A3
    0d:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Vega [Radeon RX Vega] [1002:687f] (rev c1)
            Subsystem: ASUSTeK Computer Inc. Vega 10 XT [Radeon RX Vega 64] [1043:04c4]
            Kernel driver in use: vfio-pci
            Kernel modules: amdgpu
    0d:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:aaf8]
            Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:aaf8]
            Kernel driver in use: vfio-pci
            Kernel modules: snd_hda_intel

Ubuntu works in UEFI mode.
IOMMU groups are different for Vega64 and its audio.
I start the VM with:

cp /usr/share/OVMF/OVMF_VARS.fd /tmp/my_vars.fd
qemu-system-x86_64 \
        -enable-kvm \
        -name $vmname,process=$vmname \
        -machine type=pc,accel=kvm \
        -cpu host \
        -smp 4 \
        -m 8G \
        -balloon none \
        -rtc clock=host,base=localtime \
        -vga none \
        -nographic \
        -serial none \
        -parallel none \
        -device vfio-pci,host=0d:00.0,multifunction=on \
        -device vfio-pci,host=0d:00.1 \
        -drive if=pflash,format=raw,readonly,file=/usr/share/OVMF/OVMF_CODE.fd \
        -drive if=pflash,format=raw,file=/tmp/my_vars.fd \
        -boot order=dc \
        -drive id=disk0,if=virtio,cache=none,format=raw,file=/var/win.img \
        -drive file=/var/win10.iso,index=1,media=cdrom \
        -drive file=/var/virtio-win.iso,index=2,media=cdrom

aaaaand nothing happens. Well, the monitor connected to Vega64 has a signal initially, showing a black screen, but when I start a VM - it looses a singal completely. Qemu starts, info pci shows a device connected. I also checked dmesg for IOMMU and VFIO things - it all seams to work.
I tried switching the cards around - no luck.
I tried to run a VM with no passthrou and with VGA - it boots properly, showing Windows “press any key”.

I’ll either find a way, or go crazy soon. Please help me stay sane. Any additional information required? Anything looks wrong?
Thank you!

BillFleming · October 1, 2018, 12:54pm

If you have only 1 video source on your VM (either VEGA with 1 monitor) or QXL and you don’t see the BIOS logo appear, then you did something wrong on your VM setup. In general you want to boot up with QXL first to install OS and make sure that basic features are working, then add GPU.

Then you should try having both QXL video (with spice add-ons) and VEGA selected into the VM until you actually install AMD drivers.

If your VEGA gets initialized during boot by mobo, turn of CSM and move slots, or pass through the VBIOS using a working kernel. (see the thread I made)
Currently I am using still using 4.19 rc1 as at least some older kernels the VBIOS passthrough+CSM off didn’t work on them. Also note that I tried the RC4 kernel and my VM stopped working completely, but I haven’t investigated the cause yet. So there be bugs in some of them.

SgtAwesomesauce · October 1, 2018, 10:03pm

So @BillFleming is correct on trying to boot with both emulated video and the gpu. That would be valuable troubleshooting information.

It’s also worth noting that there are lots of awful ovmf packages floating around there that flat out don’t work and can exhibit problems either initializing or booting with passthrough. If you want to be 100% sure they work, compile your own from source.

Bytales · July 1, 2019, 8:01am

How is this suppose to work ? I have 2 similar GPUs, AMD Vega Frontier Edition Watercooled., they both have the same PCI ID. How can i make sure i use the GPU with the sinked Output ?

“softdep amdgpu pre: vfio vfio_pci
vfio
vfio_iommu_type1
vfio_virqfd
options vfio_pci ids=1002:67c4,1002:aaf0
vfio_pci ids=1002:67c4,1002:aaf0
vfio_pci
amdgpu”

COGlory · July 1, 2019, 2:13pm

If they have the same PCI ID then you’re going to have to jump through some hoops. I never was able to make it work on Ubuntu, but here is what worked for me on Manjaro:

https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#Using_identical_guest_and_host_GPUs

I had to use that script and regenerate the initramfs and boot to a custom kernel to get it to work, but Ubuntu doesn’t use mkinitcpio so I couldn’t figure out how to do it on Ubuntu.

Bytales · July 3, 2019, 8:27am

I managed to VFIO one of two identical GPUs in Ubuntu, there is a guide on this Forums, that i used and worked, ist something new, update in 2019.
Here is the link to the guide:

VFIO in 2019 -- Pop!_OS How-To (General Guide though) [DRAFT] Wikis & How-to Guides

The Ultimate (Same GPU) VFIO Passthrough Guide for 2019 (now notes on getting samey graphics cards working) What are we doing? What is VFIO/Passthrough? If you are an absolute newbie – all the hub-ubb is about being able to run Linux as your primary OS, with a Windows virtual machine running under Linux for everything else. The key aspect here, though, is that you are running a computer with two graphics cards and full control over one graphics card is given to the Windows virtual machine. That means you can run games and GPU-accelerated apps (like the Adobe creative suite) just like they were running on a full computer, but in a virtual machine. Windows in a VM gives you great flexibility for backups, snapshots and versioning. It gives you better control over your privacy and what the operating system might be doing and it means that Windows only runs on your computer when you have to. I’ve been running VFIO for about 10 years and I have found that I rarely even need Windows for much of anything in my personal computation needs these days (even for gaming). Still, there is some usefulness in having the ‘security blanket’ of being able to get to your old Windows install exactly as it was before you switched operating systems without having to reboot. The System (76) For this guide, we are using Pop!_OS from System76 as our Linux distro of choice. I got to borrow Eric Raymond’s new computer, a monster 32-core Threadripper 2990WX system and, for this guide, it is equipped with dual Nvidia RTX 2080s from Gigabyte (Eric ultimately gets a WX7100 graphics card). Our Hardware This is the System76 Thelio Major r1 It has: AMD Threadripper 2990WX (2x) Gigabyte RTX 2080 GPUs (4x) 16gb Dual Rank Kingston HyperX 2933 DDR4 (16-18-18-36 1T @ 2933) Samsung SM97x 2tb NVMe Intel 750 1.2tb NVMe (guest OS) Arctic Cooling fans With Lutris/DXVK and Steam on Linux – I like the idea of having the same GPU horsepower for both guest and host. This system? It has two Gigabyte RTX…

It doesnt have “similar gpu” tags in the title, so ist maybe hard to find.
I must mention that i run latest kernel 5.1.15 (with zfs 0.8.1) so perhaps thats a factor as well, i dont know

wUFr · July 8, 2019, 7:12pm

used this guide with RX570 and 580, which have same IDs, works fine, u should probably remove all the PCI IDs from configs first, before atempting this one.

Also in my case, it assigned vfio-pci driver to the first (top) gpu, probably because it is in lower IOMMU group (group 8, vs group 43 for lower gpu), so keep that in mind

jurand · March 1, 2021, 9:11am

Hello, maybe a bit off-topic, but is there a way/tutorial for passing through the custom PIC (not GPU/VGA etc) device to the QEMU? I have a custom PCI device and would like to have it inside the QEMU to make driver testing easier.