Installing AMD 6800xt drivers in VM with vfio passthrough leads to host crash/reboot

Software & Operating Systems VFIO
, ,
1

I’m wondering if anyone else is running into similar issues when trying to pass through one of the new AMD cards to a win10 guest.

Hardware:

  • CPU: 5900x
  • Mobo: X570 Aorus master
  • Memory: 64gb
  • GPU1: Sapphire 6800xt nitro+ (for pass-through to win10)
  • GPU2: Vega64 (for host Ubuntu 20.10 running Kernel 5.10)

OS: Ubuntu 20.10
Kernel: 5.10
AMD driver: amdgpu-pro-20.45-1164792-ubuntu-20.04.tar.xz

The VM can see the graphics card as a 6800xt but shows a code43 error on it, re-installing the drivers causes the host system to reboot making it a bit challenging to nab any output of whats crashing exactly. When booting the NVMe drive directly the drive works properly in Windows.

lspci -nnv

0d:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:73bf] (rev c1) (prog-if 00 [VGA controller])
	Subsystem: Sapphire Technology Limited Device [1da2:e438]
	Flags: fast devsel, IRQ 43, IOMMU group 30
	Memory at 7400000000 (64-bit, prefetchable) [size=16G]
	Memory at 7800000000 (64-bit, prefetchable) [size=256M]
	I/O ports at f000 [size=256]
	Memory at fca00000 (32-bit, non-prefetchable) [size=1M]
	Expansion ROM at fcb00000 [disabled] [size=128K]
	Capabilities: [48] Vendor Specific Information: Len=08 <?>
	Capabilities: [50] Power Management version 3
	Capabilities: [64] Express Legacy Endpoint, MSI 00
	Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [150] Advanced Error Reporting
	Capabilities: [200] Physical Resizable BAR
	Capabilities: [240] Power Budgeting <?>
	Capabilities: [270] Secondary PCI Express
	Capabilities: [2a0] Access Control Services
	Capabilities: [2d0] Process Address Space ID (PASID)
	Capabilities: [320] Latency Tolerance Reporting
	Capabilities: [410] Physical Layer 16.0 GT/s <?>
	Capabilities: [440] Lane Margining at the Receiver <?>
	Kernel driver in use: vfio-pci
	Kernel modules: amdgpu

0d:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:ab28]
	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:ab28]
	Flags: fast devsel, IRQ 51, IOMMU group 31
	Memory at fcb20000 (32-bit, non-prefetchable) [size=16K]
	Capabilities: [48] Vendor Specific Information: Len=08 <?>
	Capabilities: [50] Power Management version 3
	Capabilities: [64] Express Legacy Endpoint, MSI 00
	Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [150] Advanced Error Reporting
	Capabilities: [2a0] Access Control Services
	Kernel driver in use: vfio-pci
	Kernel modules: snd_hda_intel

kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used

sudo dmesg |grep AMD-Vi
[ 2.402816] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[ 2.404178] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[ 2.404178] pci 0000:00:00.2: AMD-Vi: Extended features (0x58f77ef22294ade):
[ 2.404181] AMD-Vi: Interrupt remapping enabled
[ 2.404181] AMD-Vi: Virtual APIC enabled
[ 2.404182] AMD-Vi: X2APIC enabled
[ 2.404305] AMD-Vi: Lazy IO/TLB flushing enabled
[ 4.938394] AMD-Vi: AMD IOMMUv2 driver by Joerg Roedel [email protected]

grub config
GRUB_CMDLINE_LINUX_DEFAULT=“quiet iommu=on amd_iommu=on kvm.ignore_msrs=1 iommu=pt video=efifb:off vfio-pci.ids=1002:73bf,1002:ab28 kvm_a
md.npt=1 kvm_amd.avic=1”

Any ideas?

2

You could probably add ls-iommu output and VM xml.
Crystal ball is kinda hazy otherwise, and requires lot of guesswork :wink:

3

Here is the output from the command you listed above, I tried to add the vm xml, but it was complaining about not being able to post links.

IOMMU Group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 10 00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 11 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 12 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 13 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61)
IOMMU Group 13 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 14 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 0 [1022:1440]
IOMMU Group 14 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 1 [1022:1441]
IOMMU Group 14 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 2 [1022:1442]
IOMMU Group 14 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 3 [1022:1443]
IOMMU Group 14 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 4 [1022:1444]
IOMMU Group 14 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 5 [1022:1445]
IOMMU Group 14 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 6 [1022:1446]
IOMMU Group 14 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 7 [1022:1447]
IOMMU Group 15 01:00.0 Non-Volatile memory controller [0108]: ADATA Technology Co., Ltd. XPG SX8200 Pro PCIe Gen3x4 M.2 2280 Solid State Drive [1cc1:8201] (rev 03)
IOMMU Group 16 02:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse Switch Upstream [1022:57ad]
IOMMU Group 17 03:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
IOMMU Group 18 03:03.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
IOMMU Group 19 03:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
IOMMU Group 1 00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 20 03:05.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
IOMMU Group 21 03:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]
IOMMU Group 21 08:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
IOMMU Group 21 08:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU Group 21 08:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU Group 22 03:09.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]
IOMMU Group 22 09:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 23 03:0a.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]
IOMMU Group 23 0a:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 24 04:00.0 Non-Volatile memory controller [0108]: ADATA Technology Co., Ltd. XPG SX8200 Pro PCIe Gen3x4 M.2 2280 Solid State Drive [1cc1:8201] (rev 03)
IOMMU Group 25 05:00.0 Network controller [0280]: Intel Corporation Wi-Fi 6 AX200 [8086:2723] (rev 1a)
IOMMU Group 26 06:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
IOMMU Group 27 07:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller [10ec:8125] (rev 05)
IOMMU Group 28 0b:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch [1002:1478] (rev c1)
IOMMU Group 29 0c:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch [1002:1479]
IOMMU Group 2 00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 30 0d:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:73bf] (rev c1)
IOMMU Group 31 0d:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:ab28]
IOMMU Group 32 0e:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1470] (rev c1)
IOMMU Group 33 0f:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1471]
IOMMU Group 34 10:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Vega 10 XL/XT [Radeon RX Vega 56/64] [1002:687f] (rev c1)
IOMMU Group 35 10:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Vega 10 HDMI Audio [Radeon Vega 56/64] [1002:aaf8]
IOMMU Group 36 11:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
IOMMU Group 37 12:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
IOMMU Group 38 12:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP [1022:1486]
IOMMU Group 39 12:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU Group 3 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 40 12:00.4 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller [1022:1487]
IOMMU Group 4 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 5 00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 6 00:03.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 7 00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 8 00:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 9 00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
4

Here’s a link to the win10.xml win10.xml

5

Add in <features>

<kvm>
      <hidden state="on"/>
</kvm>

Also I would try to remove NVME pcie bus and post it as block device if its still crashing after hiding HV.

Also this:
rendernode='/dev/dri/by-path/pci-0000:10:00.0-render'
Have caused me troubles in the past.

6

I’ve tried with the hidden state=“on” as well, I had actually just finished removing it before posting this cause I had read that AMD cards sometimes don’t need it to be hidden.

I have have a second VM that I tried with just a block device and a fresh instead instead of the passed through drive since I thought as well that it may be conflicting. But the same issue persists.

What would you suggest as an alternative to this, that points to my Vega running the host OS.

7

Nothing really, at least for now. When troubleshooting remove as much variables as you can.
Once you get it stable, then you can tweak.

I thought it too, but Wendell said it should be there. Alas don’t have new 6xxx to test it myself, so I will rely on his word for now.

Edit: Taking into consideration that you have the same mobo and trying to pass NVME as pcie, I would say you have similar problem.

So to reiterate, I would try to pass GPU only, and nothing else. So disk as block storage or even file. Once that is working I would try to add more, one by one. Otherwise you never know what is causing troubles.

8

Cool I’ll take a look at the other post, and try removing everything but the gpu.

9

@misiektw reset my BIOS to the defaults, and applied all the IOMMU options to the same as I had before, now it’s working with no changes to the OS or VM configs.

3 Likes
10

Glad to hear it worked out :slight_smile:

1 Like
11

Glad to see it was solved!
I have the same mobo and planning to get the same CPU/GPU for VM setup, so thankfully, it works :slight_smile:

12

Hi @robbbot!

Glad that you got that working. Would you mind to share the XML of your VM which you got working? And what do you mean by “reset my BIOS to the defaults”? Q35 and BIOS? Q35 and OVMF?

I’m trying to passthrough my 6900 XT and I’m always getting a code 43. I tried everything I was able to find (including this thread :wink: ). I completely ran out of ideas what could be wrong.

My VM XML

(I tried also Q35 chipset + OVMF, and i440fx + OVMF, none of them worked)

Kernel parameters:

GRUB_CMDLINE_LINUX="amd_iommu=force_isolation iommu=pt rd.driver.pre=vfio-pci"

(I also tried amd_iommu=on)

I tried three versions of Radeon drivers:

  • 21.2.3 (22.2.2021)
  • 20.12.2 (12/18/2020)
  • 20.12.1 (12/8/2020)

The problem is clearly related to Windows and its drivers. I can passthrough the same card to a Linux VM and it works perfectly. Here goes Unigine Heaven running on it.

IMG_20210227_125953
IMG_20210227_1259534000×3000 5.53 MB

The XML of the Linux VM, just in case.

IOMMU groups seperation looks good to me.

My hardware:

  • AMD Ryzen 5950x
  • ASUS ROG Strix x570-F
  • 64 GB RAM DDR4 3600
  • XFX Radeon 6900 XT
  • XFX Radeon 6800 XT

Yes, I want to use 6800 XT on host and passthrough 6900 XT. Crazy setup, but as I mentioned, it works for a Linux VM, so I think it shouldn’t be a problem itself. :slight_smile:

I would appreciate any help or any ideas.

14

Hi there, The XML I’m using is linked above in the post. And the BIOS settings aren’t the virt-manager bios settings, but rather my actual motherboards bios settings. I reset the bios to default configuration, then re-applied the IOMMU / svm options. Basically I had all the right “linux” / “bootloader” settings in order, but my motherboard settings were screwy despite being set properly.

The actual bios settings I needed to have set to get it working can be found here: Dev Oops - Odds and Ends

1 Like
15

May also want to include the results or something like lspci -nnv |grep ATI -C 5to make sure that the vfio-pci driver is assigned properly to the card.

image
image1702×301 55.6 KB

16

Thanks for quick reply.

i didn’t solve the issue yet, but your hint about BIOS settings made me realize that the IOMMU option was on Auto instead of Enabled. SVM was already enabled, CSM already disabled.

My BIOS has no option of setting a “boot GPU”, but i keep my “host” card (6800 XT) in the 1st PCIe slot, the card I want to passthrough (6900 XT) in the 2nd slot.

However, that didn’t fix the problem (even when I enabled IOMMU after restoring default settings), but I will check if there are more related options which could help.

I’ve seen a lot of AMD CBS options being on Auto and the IOMMU setting was in that section. I’ll try enabling them.

This is the fragment of my lspci -nnv output, with the devices I want to passthrough:

0e:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 [Radeon RX 6800/6800 XT / 6900 XT] [1002:73bf] (rev c0) (prog-if 00 [VGA controller])
	Subsystem: XFX Limited Device [1eae:6901]
	Flags: bus master, fast devsel, latency 0, IRQ 150, IOMMU group 30
	Memory at 7000000000 (64-bit, prefetchable) [size=16G]
	Memory at 7400000000 (64-bit, prefetchable) [size=256M]
	I/O ports at d000 [size=256]
	Memory at fc600000 (32-bit, non-prefetchable) [size=1M]
	Expansion ROM at fc700000 [disabled] [size=128K]
	Capabilities: [48] Vendor Specific Information: Len=08 <?>
	Capabilities: [50] Power Management version 3
	Capabilities: [64] Express Legacy Endpoint, MSI 00
	Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
	Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [150] Advanced Error Reporting
	Capabilities: [200] Physical Resizable BAR
	Capabilities: [240] Power Budgeting <?>
	Capabilities: [270] Secondary PCI Express
	Capabilities: [2a0] Access Control Services
	Capabilities: [2d0] Process Address Space ID (PASID)
	Capabilities: [320] Latency Tolerance Reporting
	Capabilities: [410] Physical Layer 16.0 GT/s <?>
	Capabilities: [440] Lane Margining at the Receiver <?>
	Kernel driver in use: vfio-pci
	Kernel modules: amdgpu

0e:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:ab28]
	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:ab28]
	Flags: bus master, fast devsel, latency 0, IRQ 147, IOMMU group 31
	Memory at fc724000 (32-bit, non-prefetchable) [size=16K]
	Capabilities: [48] Vendor Specific Information: Len=08 <?>
	Capabilities: [50] Power Management version 3
	Capabilities: [64] Express Legacy Endpoint, MSI 00
	Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [150] Advanced Error Reporting
	Capabilities: [2a0] Access Control Services
	Kernel driver in use: vfio-pci
	Kernel modules: snd_hda_intel

0e:00.2 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:73a6] (prog-if 30 [XHCI])
	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:73a6]
	Flags: bus master, fast devsel, latency 0, IRQ 148, IOMMU group 32
	Memory at fc500000 (64-bit, non-prefetchable) [size=1M]
	Capabilities: [48] Vendor Specific Information: Len=08 <?>
	Capabilities: [50] Power Management version 3
	Capabilities: [64] Express Endpoint, MSI 00
	Capabilities: [a0] MSI: Enable- Count=1/8 Maskable- 64bit+
	Capabilities: [c0] MSI-X: Enable+ Count=8 Masked-
	Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [150] Advanced Error Reporting
	Capabilities: [2a0] Access Control Services
	Kernel driver in use: vfio-pci
	Kernel modules: xhci_pci

0e:00.3 Serial bus controller [0c80]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:73a4]
	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:0408]
	Flags: bus master, fast devsel, latency 0, IRQ 151, IOMMU group 33
	Memory at fc720000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [48] Vendor Specific Information: Len=08 <?>
	Capabilities: [50] Power Management version 3
	Capabilities: [64] Express Endpoint, MSI 00
	Capabilities: [a0] MSI: Enable+ Count=1/2 Maskable- 64bit+
	Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [150] Advanced Error Reporting
	Capabilities: [2a0] Access Control Services
	Kernel driver in use: vfio-pci

Looks fine, vfio-pci is a driver in use.

For comparison, there is the “host” card:

0b:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 [Radeon RX 6800/6800 XT / 6900 XT] [1002:73bf] (rev c1) (prog-if 00 [VGA controller])
	Subsystem: XFX Limited Device [1eae:6701]
	Flags: bus master, fast devsel, latency 0, IRQ 137, IOMMU group 26
	Memory at 7800000000 (64-bit, prefetchable) [size=16G]
	Memory at 7c00000000 (64-bit, prefetchable) [size=256M]
	I/O ports at e000 [size=256]
	Memory at fcc00000 (32-bit, non-prefetchable) [size=1M]
	Expansion ROM at fcd00000 [disabled] [size=128K]
	Capabilities: [48] Vendor Specific Information: Len=08 <?>
	Capabilities: [50] Power Management version 3
	Capabilities: [64] Express Legacy Endpoint, MSI 00
	Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
	Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [150] Advanced Error Reporting
	Capabilities: [200] Physical Resizable BAR
	Capabilities: [240] Power Budgeting <?>
	Capabilities: [270] Secondary PCI Express
	Capabilities: [2a0] Access Control Services
	Capabilities: [2d0] Process Address Space ID (PASID)
	Capabilities: [320] Latency Tolerance Reporting
	Capabilities: [410] Physical Layer 16.0 GT/s <?>
	Capabilities: [440] Lane Margining at the Receiver <?>
	Kernel driver in use: amdgpu
	Kernel modules: amdgpu

0b:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:ab28]
	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:ab28]
	Flags: bus master, fast devsel, latency 0, IRQ 144, IOMMU group 27
	Memory at fcd20000 (32-bit, non-prefetchable) [size=16K]
	Capabilities: [48] Vendor Specific Information: Len=08 <?>
	Capabilities: [50] Power Management version 3
	Capabilities: [64] Express Legacy Endpoint, MSI 00
	Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
	Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [150] Advanced Error Reporting
	Capabilities: [2a0] Access Control Services
	Kernel driver in use: snd_hda_intel
	Kernel modules: snd_hda_intel

The “host” card (6800 XT) doesn’t have the “USB controller” and the “bus controller”, because it has no USB-C port, and the 6900 XT card has it.

17

Seems you’re in a similar situation as I was when I made this post, your configs for the most part look good. But you likely have some wonky motherboard BIOS setting that’s not toggled to the right state. Lemme know if you find out what it is.

1 Like
18

I fixed the issue.

The problem was about the SRIOV option in my host BIOS being enabled. Disabling it made Radeon drivers inside Windows VM working. Which is quite odd and I don’t really understand why that would be an issue. But that was the only non-default option which I enabled manually after reset, apart from those from the article that @robbbot linked (IOMMU, SVM, CSM). And I have a network card with SRIOV support which I was hoping to use for the Windows VM.

Thank you @robbbot for help!

2 Likes
19

The VirtIO network drivers work super well, try those out if you haven’t for your NIC.

1 Like
20

I had the similar issue with 5700 XT passthrough. I guess when you enable SR-IOV you will switch the system to what amd calls infinity fabric. SR-IOV is to split pci-e cards virtual functionality many vms without loosing performance. It is used for mainly ethernet card performance. So that you make many vms with high performance vitio ethernet tapped to single physical card.

However when you

lspci -tvv

Edit:
Found an example in web:

           +-02.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
           +-03.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
           +-03.1-[2a-2c]----00.0-[2b-2c]----00.0-[2c]--+-00.0  Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 [Radeon RX 5600 OEM/5600 XT / 5700/5700 XT]
           |                                            \-00.1  Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 HDMI Audio

you see a cool structure where there is downstream and upstream ports connected to buses which connects to cards gpu and audio

If you emulate these downstream and upstream ports and see same structure in a linux VM you are done.

I dont have the card with me however I have some xml not sure if it is the correct one. I am pasting one here as a hint :

<controller type='pci' index='0' model='pcie-root'/>
<controller type='pci' index='1' model='dmi-to-pci-bridge'>
  <model name='i82801b11-bridge'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
</controller>
<controller type='pci' index='2' model='pci-bridge'>
  <model name='pci-bridge'/>
  <target chassisNr='2'/>
  <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</controller>
<controller type='pci' index='3' model='pcie-root-port'>
  <model name='pcie-root-port'/>
  <target chassis='3' port='0x8'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
</controller>
<controller type='pci' index='4' model='pcie-root-port'>
  <model name='ioh3420'/>
  <target chassis='4' port='0x9'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0' multifunction='on'/>
</controller>
<controller type='pci' index='5' model='pcie-switch-upstream-port'>
  <model name='x3130-upstream'/>
  <target chassis='5' port='0x10'/>
  <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
</controller>
<controller type='pci' index='6' model='pcie-switch-downstream-port'>
  <model name='xio3130-downstream'/>
  <target chassis='6' port='0x11'/>
  <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
</controller>
<controller type='pci' index='7' model='pcie-root-port'>
  <model name='pcie-root-port'/>
  <target chassis='7' port='0x8'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
</controller>
<controller type='pci' index='8' model='pcie-root-port'>
  <model name='pcie-root-port'/>
  <target chassis='8' port='0x9'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
</controller>

<hostdev mode='subsystem' type='pci' managed='yes'>
  <source>
    <address domain='0x0000' bus='0x0b' slot='0x00' function='0x0'/>
  </source>
  <rom file='/usr/share/vgabios/5700XT.rom'/>
  <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0' multifunction='on'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
  <source>
    <address domain='0x0000' bus='0x0b' slot='0x00' function='0x1'/>
  </source>
  <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x1'/>
</hostdev>
1 Like
21

I was going to glom on to this but instead posted a new post where I’m having similar issues.