Return to Level1Techs.com

VFIO/Passthrough Issues - Threadripper Pro & Asus WRX80E-Sage

I am having trouble with IOMMU passthrough on proxmox using a Threadripper Pro 3975X on an Asus WRX80E-SAGE motherboard.

When I try to pass an Nvidia Quadro P400 to a virtual machine, the virtual machine sits and hangs, and the graphics card puts itself in a state where I cannot reset it. In my research online, it looks similar to the AMD Vega reset bugs.

I think that this issue is likely due to a bios setting in the motherboard, and there is a plethora of different settings related to PCIE that I am not thoroughly versed in. I have performed IOMMU passthrough on other systems before, but not encountered this issue.

First, I’ll print configuration files and lspci before booting the vm. Then, I’ll print the error messages I get from lspci and from dmesg.

Software Versions

BIOS: Version 0405, 2021/04/19 (Newest)
BMC: Version 1.11.0, 2021/05/06 (Newest)

[email protected]:~# uname -a
   Linux pve-01 5.4.119-1-pve #1 SMP PVE 5.4.119-1 (Tue, 01 Jun 2021 15:32:00 +0200) x86_64 GNU/Linux
[email protected]:~# pveversion
   pve-manager/6.4-8/185e14db (running kernel: 5.4.119-1-pve)

Bios Settings

# I've included ALL sub-menu settings from these sections below:
Advanced - CPU Configuration
Advanced - PCI Subsystem Settings
Advanced - AMD CBS - NBIO Common options
Images Hidden Here

Advanced - CPU Configuration - SVM Mode = Enabled
Advanced - PCI Subsystem Settings - Above 4G Decoding = Enabled
Advanced - PCI Subsystem Settings - Re-Size BAR Support = Auto
Advanced - PCI Subsystem Settings - SR-IOV Support = Enabled
Advanced - PCI Subsystem Settings - BME DMA Mitigation = Enabled 
Advanced - PCI Subsystem Settings - Hot-Plug Support = Enabled
Advanced - PCI Subsystem Settings - OnBoard Device - PCI Latency Timer = 32 PCI Bus Clocks
Advanced - PCI Subsystem Settings - OnBoard Device - PCI-X Latency Timer = 32 PCI Bus Clocks
Advanced - PCI Subsystem Settings - OnBoard Device - VGA Palette Snoop = Off
Advanced - PCI Subsystem Settings - OnBoard Device - PERR# Generation = Off
Advanced - PCI Subsystem Settings - OnBoard Device - SERR# Generation = Off
Advanced - PCI Subsystem Settings - OnBoard Device - Disable Above 4G Decoding = Off
Advanced - PCI Subsystem Settings - OnBoard Device - Disable Hot-Plug = Off
Advanced - PCI Subsystem Settings - OnBoard Device - Disable PCIe Init = Off
Advanced - PCI Subsystem Settings - OnBoard Device - PCI Express Gen 1 Settings - Relaxed Ordering = On
Advanced - PCI Subsystem Settings - OnBoard Device - PCI Express Gen 1 Settings - Extended Tag = Off
Advanced - PCI Subsystem Settings - OnBoard Device - PCI Express Gen 1 Settings - No Snoop = On
Advanced - PCI Subsystem Settings - OnBoard Device - PCI Express Gen 1 Settings - Maximum Payload = Auto
Advanced - PCI Subsystem Settings - OnBoard Device - PCI Express Gen 1 Settings - Maximum Read Request = Auto
Advanced - PCI Subsystem Settings - OnBoard Device - PCI Express Gen 1 Settings - ASPM Support = Disabled
Advanced - PCI Subsystem Settings - OnBoard Device - PCI Express Gen 1 Settings - Extended Synch = Off
Advanced - PCI Subsystem Settings - OnBoard Device - PCI Express Gen 1 Settings - Clock Power Management = Off
Advanced - PCI Subsystem Settings - OnBoard Device - PCI Express Gen 1 Settings - Link Training Retry = Disabled
Advanced - PCI Subsystem Settings - OnBoard Device - PCI Express Gen 1 Settings - Link Training Timeout (uS) = 1000
Advanced - PCI Subsystem Settings - OnBoard Device - PCI Express Gen 1 Settings - Disable Empty Links = Off
Advanced - PCI Subsystem Settings - OnBoard Device - PCI Hot-Plug Settings - PCI Busses Padding = Disabled
Advanced - PCI Subsystem Settings - OnBoard Device - PCI Hot-Plug Settings - I/O Resources Padding = Disabled
Advanced - PCI Subsystem Settings - OnBoard Device - PCI Hot-Plug Settings - MIMO 32 bit Resources Padding = Disabled
Advanced - PCI Subsystem Settings - OnBoard Device - PCI Hot-Plug Settings - PFMIMO 32 bit Resources Padding = Disabled
Advanced - PCI Subsystem Settings - OnBoard Device - PCI Hot-Plug Settings - PFMIMO 64 bit Resources Padding = Disabled
Advanced - AMD CBS - NBIO Common options - IOMMU = Auto
Advanced - AMD CBS - NBIO Common options - ACS Enable = Auto
Advanced - AMD CBS - NBIO Common options - PCIe Ten Bit Tag Support = Auto
Advanced - AMD CBS - NBIO Common options - Early Link Speed = Auto
Advanced - AMD CBS - NBIO Common options - Preferred IO = Auto
Advanced - AMD CBS - NBIO Common options - CV test = Auto
Advanced - AMD CBS - NBIO Common options - Loopback Mode = Auto
Advanced - AMD CBS - NBIO Common options - SMU Common Options - cTDP Control = Auto
Advanced - AMD CBS - NBIO Common options - SMU Common Options - EfficiencyModeEn = Auto
Advanced - AMD CBS - NBIO Common options - SMU Common Options - Package Power Limit Control = Auto
Advanced - AMD CBS - NBIO Common options - SMU Common Options - APBDIS = Auto
Advanced - AMD CBS - NBIO Common options - SMU Common Options - DF Cstates = Auto
Advanced - AMD CBS - NBIO Common options - SMU Common Options - CPPC = Auto
Advanced - AMD CBS - NBIO Common options - SMU Common Options - CPPC Preferred Cores = Auto
Advanced - AMD CBS - NBIO Common options - SMU Common Options - NBIO LCLK DPM - NBIO DPM Control = Auto

Before VM Boot

Modules:

[email protected]:~# egrep "options|blacklist" /etc/modprobe.d/*
  /etc/modprobe.d/blacklist.conf:blacklist radeon
  /etc/modprobe.d/blacklist.conf:blacklist nouveau
  /etc/modprobe.d/blacklist.conf:blacklist nvidia
  /etc/modprobe.d/kvm.conf:options kvm ignore_msrs=1
   # Tried with and without `options kvm ignore_msrs=1`
  /etc/modprobe.d/pve-blacklist.conf:blacklist nvidiafb
  /etc/modprobe.d/vfio_iommu_type1.conf:options vfio_iommu_type1 allow_unsafe_interrupts=1
   # Tried with and without `options vfio_iommu_type1 allow_unsafe_interrupts=1`
  /etc/modprobe.d/vfio-pci.conf:options vfio-pci ids=10de:1cb3,10de:0fb9 disable_vga=1 disable_idle_d3=1
   # Tried with and without `disable_idle_d3=1`

Boot:

#Using systemd-boot instead of grub. Needed for ZFS Raid 1 Boot Disk(s)
[email protected]:~# cat /etc/kernel/cmdline 
   root=ZFS=rpool/ROOT/pve-1 boot=zfs amd_iommu=on iommu=pt
   # Tried with and without `rd.driver.pre=vfio-pci`
[email protected]:~# update-initramfs -u
[email protected]:~# pve-efiboot-tool refresh
[email protected]:~# reboot

Virtual Machine:

[email protected]:~# cat /etc/pve/qemu-server/101.conf 
    bios: ovmf
    boot: order=scsi0;ide2;net0
    cores: 64
    cpu: host
    efidisk0: local-zfs:vm-101-disk-1,size=1M
    hostpci0: 0000:41:00.0,pcie=1,x-vga=1
    hostpci1: 0000:41:00.1
    ide2: local:iso/manjaro-xfce-21.0.7-210614-linux510.iso,media=cdrom
    machine: q35
    memory: 2048
    name: workstation
    net0: virtio=6E:D8:D5:EF:27:98,bridge=vmbr0,firewall=1
    numa: 0
    ostype: l26
    scsi0: local-zfs:vm-101-disk-0,size=32G
    scsihw: virtio-scsi-pci
    smbios1: uuid=9ef0b069-4125-4a78-bca2-d1185c2fb4a2
    sockets: 1
    vga: none
    vmgenid: e6429e56-559f-4384-8a93-8fc51c3f6b9f

PCIE Device:

[email protected]:~# lspci -s 41:00 -v
    41:00.0 VGA compatible controller: NVIDIA Corporation GP107GL [Quadro P400] (rev a1) (prog-if 00 [VGA controller])
    	Subsystem: Dell GP107GL [Quadro P400]
    	Physical Slot: 6
    	Flags: fast devsel, IRQ 255
    	Memory at ce000000 (32-bit, non-prefetchable) [size=16M]
    	Memory at 18030000000 (64-bit, prefetchable) [size=256M]
    	Memory at 18040000000 (64-bit, prefetchable) [size=32M]
    	I/O ports at 9000 [size=128]
    	Expansion ROM at cf000000 [disabled] [size=512K]
    	Capabilities: [60] Power Management version 3
    	Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
    	Capabilities: [78] Express Legacy Endpoint, MSI 00
    	Capabilities: [100] Virtual Channel
    	Capabilities: [250] Latency Tolerance Reporting
    	Capabilities: [128] Power Budgeting <?>
    	Capabilities: [420] Advanced Error Reporting
    	Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
    	Capabilities: [900] #19
    	Kernel driver in use: vfio-pci
    	Kernel modules: nvidiafb, nouveau

  41:00.1 Audio device: NVIDIA Corporation GP107GL High Definition Audio Controller (rev a1)
    	Subsystem: Dell GP107GL High Definition Audio Controller
    	Physical Slot: 6
    	Flags: fast devsel, IRQ 255
    	Memory at cf080000 (32-bit, non-prefetchable) [disabled] [size=16K]
    	Capabilities: [60] Power Management version 3
    	Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
    	Capabilities: [78] Express Endpoint, MSI 00
    	Capabilities: [100] Advanced Error Reporting
    	Kernel driver in use: vfio-pci
    	Kernel modules: snd_hda_intel

Additional Dmesg Information:

[email protected]:~# dmesg | grep "41:00"
    [    1.754274] pci 0000:41:00.0: [10de:1cb3] type 00 class 0x030000
    [    1.754274] pci 0000:41:00.0: reg 0x10: [mem 0xce000000-0xceffffff]
    [    1.754283] pci 0000:41:00.0: reg 0x14: [mem 0x18030000000-0x1803fffffff 64bit pref]
    [    1.754297] pci 0000:41:00.0: reg 0x1c: [mem 0x18040000000-0x18041ffffff 64bit pref]
    [    1.754306] pci 0000:41:00.0: reg 0x24: [io  0x9000-0x907f]
    [    1.754315] pci 0000:41:00.0: reg 0x30: [mem 0xcf000000-0xcf07ffff pref]
    [    1.754458] pci 0000:41:00.0: 32.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s x16 link at 0000:40:01.1 (capable of 126.016 Gb/s with 8 GT/s x16 link)
    [    1.754490] pci 0000:41:00.1: [10de:0fb9] type 00 class 0x040300
    [    1.754508] pci 0000:41:00.1: reg 0x10: [mem 0xcf080000-0xcf083fff]
    [    1.781620] pci 0000:41:00.0: vgaarb: VGA device added: decodes=io+mem,owns=none,locks=none
    [    1.781620] pci 0000:41:00.0: vgaarb: bridge control possible
    [    1.811278] pci 0000:41:00.1: D0 power state depends on 0000:41:00.0
    [    2.258616] pci 0000:41:00.0: Adding to iommu group 21
    [    2.258653] pci 0000:41:00.1: Adding to iommu group 21
    [    7.847822] vfio-pci 0000:41:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none

After VM Boot

PCIE Device:

[email protected]:~# lspci -s 41:00 -v
    41:00.0 VGA compatible controller: NVIDIA Corporation GP107GL [Quadro P400] (rev ff) (prog-if ff)
      	!!! Unknown header type 7f
      	Kernel driver in use: vfio-pci
      	Kernel modules: nvidiafb, nouveau

    41:00.1 Audio device: NVIDIA Corporation GP107GL High Definition Audio Controller (rev ff) (prog-if ff)
      	!!! Unknown header type 7f
      	Kernel driver in use: vfio-pci
      	Kernel modules: snd_hda_intel

Dmesg:

[email protected]:~# dmesg
    ...
    [ 3062.262871] vfio-pci 0000:41:00.0: vfio_ecap_init: hiding ecap [email protected]
    [ 3063.702406] vfio-pci 0000:41:00.0: vfio_bar_restore: reset recovery - restoring BARs
    [ 3063.733764] vfio-pci 0000:41:00.0: vfio_bar_restore: reset recovery - restoring BARs
    [ 3063.759983] vfio-pci 0000:41:00.0: vfio_bar_restore: reset recovery - restoring BARs
    [ 3063.789152] vfio-pci 0000:41:00.0: vfio_bar_restore: reset recovery - restoring BARs
    [ 3063.800319] vfio-pci 0000:41:00.0: Invalid PCI ROM header signature: expecting 0xaa55, got 0xffff
    [ 3063.810980] vfio-pci 0000:41:00.0: vfio_bar_restore: reset recovery - restoring BARs
    [ 3063.825283] vfio-pci 0000:41:00.0: vfio_bar_restore: reset recovery - restoring BARs
    [ 3063.829676] vfio-pci 0000:41:00.0: vfio_bar_restore: reset recovery - restoring BARs
    [ 3063.837817] vfio-pci 0000:41:00.0: vfio_bar_restore: reset recovery - restoring BARs
    [ 3063.853392] vfio-pci 0000:41:00.0: vfio_bar_restore: reset recovery - restoring BARs
    [ 3063.894470] vfio-pci 0000:41:00.0: vfio_bar_restore: reset recovery - restoring BARs
    [ 3063.916454] vfio-pci 0000:41:00.0: vfio_bar_restore: reset recovery - restoring BARs
    [ 3063.925763] vfio-pci 0000:41:00.0: vfio_bar_restore: reset recovery - restoring BARs
    [ 3063.945511] vfio-pci 0000:41:00.0: vfio_bar_restore: reset recovery - restoring BARs
    [ 3082.748642] kvm [43517]: vcpu0, guest rIP: 0xffffffff8106b364 ignored wrmsr: 0xda0 data 0x0
    [ 3082.790655] kvm [43517]: vcpu0, guest rIP: 0xffffffff8106b364 ignored wrmsr: 0xda0 data 0x0
    [ 3082.851592] kvm [43517]: vcpu1, guest rIP: 0xffffffff8106b364 ignored wrmsr: 0xda0 data 0x0
    [ 3082.938058] kvm [43517]: vcpu2, guest rIP: 0xffffffff8106b364 ignored wrmsr: 0xda0 data 0x0
    [ 3083.021439] kvm [43517]: vcpu3, guest rIP: 0xffffffff8106b364 ignored wrmsr: 0xda0 data 0x0
    [ 3083.108127] kvm [43517]: vcpu4, guest rIP: 0xffffffff8106b364 ignored wrmsr: 0xda0 data 0x0
    [ 3083.191588] kvm [43517]: vcpu5, guest rIP: 0xffffffff8106b364 ignored wrmsr: 0xda0 data 0x0
    [ 3083.278314] kvm [43517]: vcpu6, guest rIP: 0xffffffff8106b364 ignored wrmsr: 0xda0 data 0x0
    [ 3083.365051] kvm [43517]: vcpu7, guest rIP: 0xffffffff8106b364 ignored wrmsr: 0xda0 data 0x0
    [ 3083.448406] kvm [43517]: vcpu8, guest rIP: 0xffffffff8106b364 ignored wrmsr: 0xda0 data 0x0
    [ 3087.788173] kvm_set_msr_common: 51 callbacks suppressed
    [ 3087.788174] kvm [43517]: vcpu60, guest rIP: 0xffffffff8106b364 ignored wrmsr: 0xda0 data 0x0
    [ 3087.871836] kvm [43517]: vcpu61, guest rIP: 0xffffffff8106b364 ignored wrmsr: 0xda0 data 0x0
    [ 3087.955876] kvm [43517]: vcpu62, guest rIP: 0xffffffff8106b364 ignored wrmsr: 0xda0 data 0x0
    [ 3088.040166] kvm [43517]: vcpu63, guest rIP: 0xffffffff8106b364 ignored wrmsr: 0xda0 data 0x0
    [ 3088.215955] vfio-pci 0000:41:00.0: vfio_bar_restore: reset recovery - restoring BARs
    [ 3088.219107] vfio-pci 0000:41:00.0: vfio_bar_restore: reset recovery - restoring BARs
    [ 3100.100816] kvm [43517]: vcpu6, guest rIP: 0xffffffff8106b284 ignored rdmsr: 0xc001029b
    [ 3100.103434] kvm [43517]: vcpu6, guest rIP: 0xffffffff8106b284 ignored rdmsr: 0xc0010299
    [ 3101.622908] kvm [43517]: vcpu49, guest rIP: 0xffffffff8106b2c4 ignored rdmsr: 0xda0
    [ 3101.624318] kvm [43517]: vcpu49, guest rIP: 0xffffffff8106b284 ignored rdmsr: 0x3a
    [ 3101.624689] kvm [43517]: vcpu49, guest rIP: 0xffffffff8106b284 ignored rdmsr: 0xd90
    [ 3101.624943] kvm [43517]: vcpu49, guest rIP: 0xffffffff8106b284 ignored rdmsr: 0x570
    [ 3101.625181] kvm [43517]: vcpu49, guest rIP: 0xffffffff8106b284 ignored rdmsr: 0x571
    [ 3101.625401] kvm [43517]: vcpu49, guest rIP: 0xffffffff8106b284 ignored rdmsr: 0x572
    [ 3101.625628] kvm [43517]: vcpu49, guest rIP: 0xffffffff8106b284 ignored rdmsr: 0x560
    [ 3101.625885] kvm [43517]: vcpu49, guest rIP: 0xffffffff8106b284 ignored rdmsr: 0x561

D3 Error:

Before adding disable_idle_d3=1 to /etc/modprobe.d/vfio-pci.conf, proxmox would give the following errors:

kvm: vfio: Unable to power on device, stuck in D3
kvm: vfio: Unable to power on device, stuck in D3

However, after adding disable_idle_d3=1 the proxmox message goes away, and proxmox does not report any issue. The VM ‘starts’ but doesn’t have any display output. This had little to no effect on dmesg or lspci output.

PCIE Gen 1 Corellation

What I found from the additional dmesg output is that the card at 16x is using Gen 1 PCIe,

    [    1.754458] pci 0000:41:00.0: 32.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s x16 link at 0000:40:01.1 (capable of 126.016 Gb/s with 8 GT/s x16 link)

It is listed as capable of 16x Gen 3, but, is presumably operating at PCIE Gen 1 at this moment.

This would make the bios settings that specifically modify PCIE Gen 1 an interesting place to dig.

Shutdown the VM and Retry

When the graphics card is reporting “Unknown Header Type 7f” in lspci, I have tried to restart the VM in proxmox, and get the following DMESG output. In order to test again, I need to reboot the host.

[   78.473349] vfio-pci 0000:41:00.0: timed out waiting for pending transaction; performing function level reset anyway
[   79.721260] vfio-pci 0000:41:00.0: not ready 1023ms after FLR; waiting
[   80.777226] vfio-pci 0000:41:00.0: not ready 2047ms after FLR; waiting
[   82.985161] vfio-pci 0000:41:00.0: not ready 4095ms after FLR; waiting
[   87.337063] vfio-pci 0000:41:00.0: not ready 8191ms after FLR; waiting
[   95.784989] vfio-pci 0000:41:00.0: not ready 16383ms after FLR; waiting
[  112.936730] vfio-pci 0000:41:00.0: not ready 32767ms after FLR; waiting
[  147.752952] vfio-pci 0000:41:00.0: not ready 65535ms after FLR; giving up

Further Discussion

At this point, I’m somewhat stumped. I’ve looked around on the internet trying to find issues where people report these errors:

“Invalid PCI ROM header signature: expecting”
“!!! Unknown header type 7f”
“Unable to power on device, stuck in D3”

And I have found less than I would have hoped. I expect this is not an issue with my proxmox configuration, but rather with my BIOS settings.

Does anyone have any ideas?

Edit 1: Fixed Images
Edit 2: Added Additional Dmesg Information
Edit 3: Added PCIE Gen 1 Corellation
Edit 4: Added Shutdown the VM and Retry

I have simultaneously plugged in a second Quadro P400 graphics card in another PCIE slot and it works fine for passthrough. It looks like perhaps the proxmox OS is grabbing the first card, despite the default graphics adapter in the bios being set to the BMC VGA header.

If I find a way to pass the first card through properly, then I will post it here.

1 Like

Well, I tested 4 NVIDIA Quadro P400 cards, and it turns out that only one of them runs into this issue. So, that tells me it’s a card issue. Not necessarily bad, but likely some kind of firmware issue on the card. It just happened to be bad luck of the draw that it was the first card I tried.

At this point, I’m content to retire the odd card and live with the 3 that are working.

1 Like

Aaaand, I spoke too soon.

Turns out, any card in the top PCIE slot has issues with this passthrough. Still unsure as to why, but, I have worked around the issue by simply not using that slot for passthrough.

2 Likes

Thank you mate !
I was starting to go nuts as well. Thank god I had a leftover graphic card.
But yeah can’t make the 1 slot work (GC)
As soon as I plugged the GC as a 2nd GC, work perfectly.

1 Like

I have performed passthrough on several of the other PCIe slots with zero problems. I’m going to eventually probably need that top slot for passthrough, but at this time I still don’t need it.

Also, in the last few days (June 22, and July 1st respectively) the BIOS and Firmware received updates. (BIOS: 0504, Firmware: 1.13) I have not yet tested these to see if they have any affect on this top-slot passthrough behavior. I will be testing this in the near future.

BiOS Version 0504
2021/06/22 7.42 MBytes

"1. Updated AGESA CastlePeakWS PI 1.0.0.6. for LR-DIMM support.
2. AMD DOCP DRAM support, Please also fully update FW version to 1.13 simultaneously.
3. Improve system stability.

Firmware Version 1.13.0
2021/07/01 30.5 MBytes

BMC FW V1.13.0

I would suggest updating to the latest stable bios first.
then try again.

Finnaly my issue had nothing to do with yours…
I don’t get it.

Looks like I cannot run passthrough with x-vga=1 in the VM conf.

And I don’t have the same hardware but, removing x-vga=1 and letting vga option to default work!

hostpci0: 0000:01:00,pcie=1
vga: default

And I pass everything. I don’t split the audio from the video.

Like this log in RDP install the driver reboot and that’s it.

I have 2 VM I hve no idea why I can’t make it work with x-vga on.

We have have/had the same issue with the first pcie slot on that board. An Radeon Pro WX 5100 did passthrough to linux host just fine in the first slot but not to Windows. In the second it worked directly and after that it seems to work in the first slot also. Strange behaviour.

1 Like

Well, I’ve updated the Bios and BMC firmware with no luck. I’m still in the same boat.

One thing of note. I can get the same error in a different slot, if I bifurcate the slot, then attempt to passthrough the graphics card. So, a 16x card that is touching 4x4x4x4x pins causes the same type of issue.

This leads me to think that perhaps the top slot isn’t precicesly operating at 16x despite what the BIOS says? Okay, now I have to go try this and see if I can get a card to operate at 16x speeds in the top slot on the host operating system.

1 Like

Well, it looks like that is not the issue.

Nvtop shows PCIE Gen [email protected] on my 2080 TI.

I wonder if resetting the PCIE bridge – or whatever happens when passthrough is triggered temporarily – causes a temporary bifurcation? No, it’s probably more complicated than that.

@wendell You seem to be watching this thread, do you have one of these boards and the ability to test passthrough in the top slot? I found another thread on the unraid forums with the same issue.

Okkkkay, so, I may not have completely solved the problem, but, I do have new suspicions.

I think, that there is some connection between the 3x M.2 slots on the motherboard and some of the PCIE slots. The manual does not indicate a mapping between the M.2 slots and any specific PCIe slot, but, I had trouble getting devices to appear in the top slot when having M2_2 and M2_3 populated. More thorough investigation is required, but, my suspicions are peaked.

I have been unable to find a block diagram of this motherboard, but really wish I had one right now.

I’ve run into issues with using the m2 and the u2 slots together, even though the manual says those shouldnt have been in contention.

from the manual

• The M.2_2 slot shares bandwidth with U.2_1. When M.2_2 is populated, U.2_1 will
be disabled.
• The M.2_3 slot shares bandwidth with U.2_2. When M.2_3 is populated, U.2_2 will
be disabled.

so only 2_1 is not in contention

1 Like

I have not attempted to use the M.2 and U.2 slots at the same time. I think that these M.2/U.2 slots are somehow co-mingled with the PCIE slots. Hence why I want a block diagram.

ID Mode 4x 4x 4x 4x
PCIE 1 4x4x4x4x Corsair MP600 [500GB] Corsair MP600 [500GB] Empty Empty
PCIE 2 4x4x4x4x Teamgroup Z440 [1TB] Teamgroup Z440 [1TB] Teamgroup Z440 [1TB] Teamgroup Z440 [1TB]
PCIE 3 4x4x4x4x Fresco FL1100 USB3 Empty Empty Empty
PCIE 4 16x Nvidia GTX 1080 Ti
PCIE 5 16x Nvidia Quadro P400
PCIE 6 16x Nvidia Quadro P400
PCIE 7 16x Nvidia GTX 2080 Ti
M.2_1 4x Corsair MP600 [500GB]
M.2_2/U.2_1 4x Empty
M.2_3/U.2_2 4x Empty

All of my passthrough testing of PCIE 1 was done when M.2_1/2/3 were all fully populated with Corsair MP600s. I have since moved 2 of them to PCIE 1. This freed up me to use U.2_1 and U.2_2 for other purposes. However, when I tried to use U.2_1/2 I could not boot. I reseated cables and tried again and again. I still could not boot. I then installed a 4x4x4x4x to SFF-8643 (connector used for U.2) card into PCIE 3 and was able to use the device I had been trying to install in U.2_1. I have not tried the U.2_1 or U.2_2 since, but I’ll be looking to get an Intel P5800x in the next month or two and I will probably try again then.

PCIE Bus IDs:

ID Mode 4x 4x 4x 4x
PCIE 1 4x4x4x4x 41 42 Empty Empty
PCIE 2 4x4x4x4x 01 02 03 04
PCIE 3 4x4x4x4x 2f Empty Empty Empty
PCIE 4 16x 62
PCIE 5 16x 61
PCIE 6 16x 43
PCIE 7 16x 05
M.2_1 4x 2e
M.2_2/U.2_1 4x Empty
M.2_3/U.2_2 4x Empty

IOMMU Group IDs:

ID Mode 4x 4x 4x 4x
PCIE 1 4x4x4x4x 27 28 Empty Empty
PCIE 2 4x4x4x4x 84 85 86 87
PCIE 3 4x4x4x4x 62 Empty Empty Empty
PCIE 4 16x 12
PCIE 5 16x 11
PCIE 6 16x 29
PCIE 7 16x 88
M.2_1 4x 61
M.2_2/U.2_1 4x Empty
M.2_3/U.2_2 4x Empty

More Detailed PCIE Layouts:

[email protected]:~# lspci -tv
-+-[0000:60]-+-00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex
 |           +-00.2  Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU
 |           +-01.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-01.1-[61]--+-00.0  NVIDIA Corporation GP107GL [Quadro P400]
 |           |            \-00.1  NVIDIA Corporation GP107GL High Definition Audio Controller
 |           +-02.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-03.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-03.1-[62]--+-00.0  NVIDIA Corporation GP102 [GeForce GTX 1080 Ti]
 |           |            \-00.1  NVIDIA Corporation GP102 HDMI Audio Controller
 |           +-04.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-05.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-07.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-07.1-[63]----00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
 |           +-08.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           \-08.1-[64]----00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
 +-[0000:40]-+-00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex
 |           +-00.2  Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU
 |           +-01.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-01.1-[41]----00.0  Phison Electronics Corporation Device 5016
 |           +-01.2-[42]----00.0  Phison Electronics Corporation Device 5016
 |           +-02.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-03.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-03.1-[43]--+-00.0  NVIDIA Corporation GP107GL [Quadro P400]
 |           |            \-00.1  NVIDIA Corporation GP107GL High Definition Audio Controller
 |           +-04.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-05.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-07.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-07.1-[44]----00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
 |           +-08.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           \-08.1-[45]----00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
 +-[0000:20]-+-00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex
 |           +-00.2  Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU
 |           +-01.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-01.1-[21-2d]----00.0-[22-2d]--+-01.0-[23]----00.0  ASMedia Technology Inc. Device 3242
 |           |                               +-02.0-[24-25]--+-00.0  Intel Corporation Ethernet Controller 10G X550T
 |           |                               |               \-00.1  Intel Corporation Ethernet Controller 10G X550T
 |           |                               +-03.0-[26]----00.0  Intel Corporation Device 2723
 |           |                               +-04.0-[27]----00.0  ASMedia Technology Inc. ASM1061 SATA IDE Controller
 |           |                               +-05.0-[28]----00.0  ASMedia Technology Inc. ASM1061 SATA IDE Controller
 |           |                               +-06.0-[29-2a]----00.0-[2a]----00.0  ASPEED Technology, Inc. ASPEED Graphics Family
 |           |                               +-08.0-[2b]--+-00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
 |           |                               |            +-00.1  Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
 |           |                               |            \-00.3  Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
 |           |                               +-09.0-[2c]----00.0  Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode]
 |           |                               \-0a.0-[2d]----00.0  Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode]
 |           +-01.2-[2e]----00.0  Phison Electronics Corporation Device 5016
 |           +-02.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-03.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-03.1-[2f]----00.0  Fresco Logic FL1100 USB 3.0 Host Controller
 |           +-04.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-05.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-07.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-07.1-[30]----00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
 |           +-08.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           \-08.1-[31]--+-00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
 |                        +-00.1  Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP
 |                        +-00.3  Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller
 |                        \-00.4  Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller
 \-[0000:00]-+-00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex
             +-00.2  Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU
             +-01.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
             +-01.1-[01]----00.0  Phison Electronics Corporation Device 5016
             +-01.2-[02]----00.0  Phison Electronics Corporation Device 5016
             +-01.3-[03]----00.0  Phison Electronics Corporation Device 5016
             +-01.4-[04]----00.0  Phison Electronics Corporation Device 5016
             +-02.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
             +-03.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
             +-03.1-[05]--+-00.0  NVIDIA Corporation TU102 [GeForce RTX 2080 Ti Rev. A]
             |            +-00.1  NVIDIA Corporation TU102 High Definition Audio Controller
             |            +-00.2  NVIDIA Corporation TU102 USB 3.1 Controller
             |            \-00.3  NVIDIA Corporation TU102 UCSI Controller
             +-04.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
             +-05.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
             +-07.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
             +-07.1-[06]----00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
             +-08.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
             +-08.1-[07]--+-00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
             |            \-00.3  Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller
             +-14.0  Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller
             +-14.3  Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge
             +-18.0  Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 0
             +-18.1  Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 1
             +-18.2  Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 2
             +-18.3  Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 3
             +-18.4  Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 4
             +-18.5  Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 5
             +-18.6  Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 6
             \-18.7  Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 7

And finally a snippet from lstopo

[email protected]:~# lstopo -v 
...
     Bridge Host->PCI L#0 (P#0 buses=0000:[00-07])
        Bridge PCI->PCI (P#17 busid=0000:00:01.1 id=1022:1483 class=0604(PCI_B) link=7.88GB/s buses=0000:[01-01] PCIVendor="Advanced Micro Devices, Inc. [AMD]") "Advanced Micro Devices, Inc. [AMD]"
          PCI 1987:5016 (P#4096 busid=0000:01:00.0 class=0108(NVMExp) link=7.88GB/s)
            Block(Disk) L#0 (LinuxDeviceID=259:4 Model="T-FORCE TM8FP7001T" Revision=EGFM13.0 SerialNumber=TPBF2012280040201489 Type=Disk) "nvme3n1"
        Bridge PCI->PCI (P#18 busid=0000:00:01.2 id=1022:1483 class=0604(PCI_B) link=7.88GB/s buses=0000:[02-02] PCIVendor="Advanced Micro Devices, Inc. [AMD]") "Advanced Micro Devices, Inc. [AMD]"
          PCI 1987:5016 (P#8192 busid=0000:02:00.0 class=0108(NVMExp) link=7.88GB/s)
            Block(Disk) L#1 (LinuxDeviceID=259:5 Model="T-FORCE TM8FP7001T" Revision=EGFM13.0 SerialNumber=TPBF2101080050300570 Type=Disk) "nvme4n1"
        Bridge PCI->PCI (P#19 busid=0000:00:01.3 id=1022:1483 class=0604(PCI_B) link=7.88GB/s buses=0000:[03-03] PCIVendor="Advanced Micro Devices, Inc. [AMD]") "Advanced Micro Devices, Inc. [AMD]"
          PCI 1987:5016 (P#12288 busid=0000:03:00.0 class=0108(NVMExp) link=7.88GB/s)
            Block(Disk) L#2 (LinuxDeviceID=259:3 Model="T-FORCE TM8FP7001T" Revision=EGFM13.0 SerialNumber=TPBF2012280060200142 Type=Disk) "nvme5n1"
        Bridge PCI->PCI (P#20 busid=0000:00:01.4 id=1022:1483 class=0604(PCI_B) link=7.88GB/s buses=0000:[04-04] PCIVendor="Advanced Micro Devices, Inc. [AMD]") "Advanced Micro Devices, Inc. [AMD]"
          PCI 1987:5016 (P#16384 busid=0000:04:00.0 class=0108(NVMExp) link=7.88GB/s)
            Block(Disk) L#3 (LinuxDeviceID=259:6 Model="T-FORCE TM8FP7001T" Revision=EGFM13.0 SerialNumber=TPBF2101080050300167 Type=Disk) "nvme6n1"
        Bridge PCI->PCI (P#49 busid=0000:00:03.1 id=1022:1483 class=0604(PCI_B) link=4.00GB/s buses=0000:[05-05] PCIVendor="Advanced Micro Devices, Inc. [AMD]") "Advanced Micro Devices, Inc. [AMD]"
          PCI 10de:1e07 (P#20480 busid=0000:05:00.0 class=0300(VGA) link=4.00GB/s PCIVendor="NVIDIA Corporation") "NVIDIA Corporation"
      Bridge Host->PCI L#6 (P#1 buses=0000:[20-31])
        Bridge PCI->PCI (P#131089 busid=0000:20:01.1 id=1022:1483 class=0604(PCI_B) link=15.75GB/s buses=0000:[21-2d] PCIVendor="Advanced Micro Devices, Inc. [AMD]") "Advanced Micro Devices, Inc. [AMD]"
          Bridge PCI->PCI (P#135168 busid=0000:21:00.0 id=1022:57ad class=0604(PCI_B) link=15.75GB/s buses=0000:[22-2d] PCIVendor="Advanced Micro Devices, Inc. [AMD]") "Advanced Micro Devices, Inc. [AMD]"
            Bridge PCI->PCI (P#139296 busid=0000:22:02.0 id=1022:57a3 class=0604(PCI_B) link=3.94GB/s buses=0000:[24-25] PCIVendor="Advanced Micro Devices, Inc. [AMD]") "Advanced Micro Devices, Inc. [AMD]"
              PCI 8086:1563 (P#147456 busid=0000:24:00.0 class=0200(Ether) link=3.94GB/s PCIVendor="Intel Corporation" PCIDevice="Ethernet Controller 10G X550T") "Intel Corporation Ethernet Controller 10G X550T"
                Network L#4 (Address=fc:34:97:a5:cc:7c) "enp36s0f0"
              PCI 8086:1563 (P#147457 busid=0000:24:00.1 class=0200(Ether) link=3.94GB/s PCIVendor="Intel Corporation" PCIDevice="Ethernet Controller 10G X550T") "Intel Corporation Ethernet Controller 10G X550T"
                Network L#5 (Address=fc:34:97:a5:cc:7d) "enp36s0f1"
            Bridge PCI->PCI (P#139312 busid=0000:22:03.0 id=1022:57a3 class=0604(PCI_B) link=0.50GB/s buses=0000:[26-26] PCIVendor="Advanced Micro Devices, Inc. [AMD]") "Advanced Micro Devices, Inc. [AMD]"
              PCI 8086:2723 (P#155648 busid=0000:26:00.0 class=0280(Net) link=0.50GB/s PCIVendor="Intel Corporation") "Intel Corporation"
                Network L#6 (Address=44:af:28:79:65:1c) "wlp38s0"
            Bridge PCI->PCI (P#139328 busid=0000:22:04.0 id=1022:57a3 class=0604(PCI_B) link=0.50GB/s buses=0000:[27-27] PCIVendor="Advanced Micro Devices, Inc. [AMD]") "Advanced Micro Devices, Inc. [AMD]"
              PCI 1b21:0611 (P#159744 busid=0000:27:00.0 class=0101(IDE) link=0.50GB/s PCIVendor="ASMedia Technology Inc." PCIDevice="ASM1061 SATA IDE Controller") "ASMedia Technology Inc. ASM1061 SATA IDE Controller"
            Bridge PCI->PCI (P#139344 busid=0000:22:05.0 id=1022:57a3 class=0604(PCI_B) link=0.50GB/s buses=0000:[28-28] PCIVendor="Advanced Micro Devices, Inc. [AMD]") "Advanced Micro Devices, Inc. [AMD]"
              PCI 1b21:0611 (P#163840 busid=0000:28:00.0 class=0101(IDE) link=0.50GB/s PCIVendor="ASMedia Technology Inc." PCIDevice="ASM1061 SATA IDE Controller") "ASMedia Technology Inc. ASM1061 SATA IDE Controller"
            Bridge PCI->PCI (P#139360 busid=0000:22:06.0 id=1022:57a3 class=0604(PCI_B) link=0.50GB/s buses=0000:[29-2a] PCIVendor="Advanced Micro Devices, Inc. [AMD]") "Advanced Micro Devices, Inc. [AMD]"
              Bridge PCI->PCI (P#167936 busid=0000:29:00.0 id=1a03:1150 class=0604(PCI_B) link=0.50GB/s buses=0000:[2a-2a] PCIVendor="ASPEED Technology, Inc." PCIDevice="AST1150 PCI-to-PCI Bridge") "ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge"
                PCI 1a03:2000 (P#172032 busid=0000:2a:00.0 class=0300(VGA) PCIVendor="ASPEED Technology, Inc." PCIDevice="ASPEED Graphics Family") "ASPEED Technology, Inc. ASPEED Graphics Family"
                  GPU L#7 "controlD64"
                  GPU L#8 "card0"
            Bridge PCI->PCI (P#139408 busid=0000:22:09.0 id=1022:57a4 class=0604(PCI_B) link=31.51GB/s buses=0000:[2c-2c] PCIVendor="Advanced Micro Devices, Inc. [AMD]") "Advanced Micro Devices, Inc. [AMD]"
              PCI 1022:7901 (P#180224 busid=0000:2c:00.0 class=0106(SATA) link=31.51GB/s PCIVendor="Advanced Micro Devices, Inc. [AMD]" PCIDevice="FCH SATA Controller [AHCI mode]") "Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode]"
            Bridge PCI->PCI (P#139424 busid=0000:22:0a.0 id=1022:57a4 class=0604(PCI_B) link=31.51GB/s buses=0000:[2d-2d] PCIVendor="Advanced Micro Devices, Inc. [AMD]") "Advanced Micro Devices, Inc. [AMD]"
              PCI 1022:7901 (P#184320 busid=0000:2d:00.0 class=0106(SATA) link=31.51GB/s PCIVendor="Advanced Micro Devices, Inc. [AMD]" PCIDevice="FCH SATA Controller [AHCI mode]") "Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode]"
        Bridge PCI->PCI (P#131090 busid=0000:20:01.2 id=1022:1483 class=0604(PCI_B) link=7.88GB/s buses=0000:[2e-2e] PCIVendor="Advanced Micro Devices, Inc. [AMD]") "Advanced Micro Devices, Inc. [AMD]"
          PCI 1987:5016 (P#188416 busid=0000:2e:00.0 class=0108(NVMExp) link=7.88GB/s)
            Block(Disk) L#9 (LinuxDeviceID=259:2 Model="Force MP600" Revision=EGFM11.3 SerialNumber=20118231000128541383 Type=Disk) "nvme2n1"
      Bridge Host->PCI L#18 (P#2 buses=0000:[40-45])
        Bridge PCI->PCI (P#262161 busid=0000:40:01.1 id=1022:1483 class=0604(PCI_B) link=7.88GB/s buses=0000:[41-41] PCIVendor="Advanced Micro Devices, Inc. [AMD]") "Advanced Micro Devices, Inc. [AMD]"
          PCI 1987:5016 (P#266240 busid=0000:41:00.0 class=0108(NVMExp) link=7.88GB/s PCISlot=0)
            Block(Disk) L#10 (LinuxDeviceID=259:1 Model="Force MP600" Revision=EGFM11.3 SerialNumber=20118231000128541112 Type=Disk) "nvme0n1"
        Bridge PCI->PCI (P#262162 busid=0000:40:01.2 id=1022:1483 class=0604(PCI_B) link=7.88GB/s buses=0000:[42-42] PCIVendor="Advanced Micro Devices, Inc. [AMD]") "Advanced Micro Devices, Inc. [AMD]"
          PCI 1987:5016 (P#270336 busid=0000:42:00.0 class=0108(NVMExp) link=7.88GB/s)
            Block(Disk) L#11 (LinuxDeviceID=259:0 Model="Force MP600" Revision=EGFM11.3 SerialNumber=20118231000128541385 Type=Disk) "nvme1n1"
        Bridge PCI->PCI (P#262193 busid=0000:40:03.1 id=1022:1483 class=0604(PCI_B) link=4.00GB/s buses=0000:[43-43] PCIVendor="Advanced Micro Devices, Inc. [AMD]") "Advanced Micro Devices, Inc. [AMD]"
          PCI 10de:1cb3 (P#274432 busid=0000:43:00.0 class=0300(VGA) link=4.00GB/s PCIVendor="NVIDIA Corporation") "NVIDIA Corporation"
      Bridge Host->PCI L#22 (P#3 buses=0000:[60-64])
        Bridge PCI->PCI (P#393233 busid=0000:60:01.1 id=1022:1483 class=0604(PCI_B) link=15.75GB/s buses=0000:[61-61] PCIVendor="Advanced Micro Devices, Inc. [AMD]") "Advanced Micro Devices, Inc. [AMD]"
          PCI 10de:1cb3 (P#397312 busid=0000:61:00.0 class=0300(VGA) link=15.75GB/s PCIVendor="NVIDIA Corporation") "NVIDIA Corporation"
        Bridge PCI->PCI (P#393265 busid=0000:60:03.1 id=1022:1483 class=0604(PCI_B) link=4.00GB/s buses=0000:[62-62] PCIVendor="Advanced Micro Devices, Inc. [AMD]") "Advanced Micro Devices, Inc. [AMD]"
          PCI 10de:1b06 (P#401408 busid=0000:62:00.0 class=0300(VGA) link=4.00GB/s PCIVendor="NVIDIA Corporation") "NVIDIA Corporation"
...

So, anyone had any luck getting the topmost slot to passthrough?

I have a similar setup and problem and it did get solved by moving the GPU one slot down, but that puts it flush against another card and it starves of air and throttles a lot.

The only M.2 slot used is M.2_1.

I just noticed: in your screenshots of the bios you have the setting on automatic instead of on.

Automatic only halve activates iommu settings.