X399 pass through more than one GPU?

Software & Operating Systems Linux
1

System is a new TR2 build; Designare Ex x399, 64gb Corsair 4x16, Ryzen TR2 2950x, bios F11

GPUs x16 top slot GeForce GT310 for host, x8 slot under it AMD Firepro 59xx, bottom x16 slot Asus 1080 Strix

OS; multiple, tried Fedora, Debian, esxi 6.7, unraid 666

Only GPU I can pass through is the one in the bottom x16 slot. The other cards don’t even show up in iommu groups. It’s not specific to the 1080, any of the cards in that slot are able to pass through.

I get a vfio-pci error -22 for the card in the top x8 slot if I try to force it to bind to that driver. No idea what error -22 means.

2

this should work. bios configured to boot off the bottom slot?

output of lspci -vvvvnn ?

3

Switched the boot slot from 1 to 4.

Here’s the lspci:

https://pastebin.com/VcWWA3cx

No change, still only able to pass-through the card in slot 4. Also able to pass through a pcie wifi card and a nvme drive (they are attached to vfio-pci successfully)

showing my kernel boot link, and the vfio-pci error -22

dmesg |grep vfio

[ 0.000000] Command line: BOOT_IMAGE=/bzimage vfio-pci.ids=10de:1b80,10de:10f0,1002:6707,1002:aa80,15b7:5002,8086:24fd pcie_acs_override=downstream,multifunction initrd=/bzroot,/bzroot-gui
[ 0.000000] Kernel command line: BOOT_IMAGE=/bzimage vfio-pci.ids=10de:1b80,10de:10f0,1002:6707,1002:aa80,15b7:5002,8086:24fd pcie_acs_override=downstream,multifunction initrd=/bzroot,/bzroot-gui
[ 12.175871] vfio-pci 0000:0a:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=io+mem
[ 12.188064] vfio_pci: add [10de:1b80[ffff:ffff]] class 0x000000/00000000
[ 12.200289] vfio_pci: add [10de:10f0[ffff:ffff]] class 0x000000/00000000
[ 12.208250] vfio-pci: probe of 0000:41:00.0 failed with error -22
[ 12.216157] vfio_pci: add [1002:6707[ffff:ffff]] class 0x000000/00000000
[ 12.224024] vfio-pci: probe of 0000:41:00.0 failed with error -22
[ 12.232057] vfio-pci: probe of 0000:41:00.1 failed with error -22
[ 12.239960] vfio_pci: add [1002:aa80[ffff:ffff]] class 0x000000/00000000
[ 12.259269] vfio-pci: probe of 0000:41:00.0 failed with error -22
[ 12.267215] vfio-pci: probe of 0000:41:00.1 failed with error -22
[ 12.275010] vfio_pci: add [15b7:5002[ffff:ffff]] class 0x000000/00000000
[ 12.295284] vfio-pci: probe of 0000:41:00.0 failed with error -22
[ 12.303230] vfio-pci: probe of 0000:41:00.1 failed with error -22
[ 12.311048] vfio_pci: add [8086:24fd[ffff:ffff]] class 0x000000/00000000

41:00 is the FirePro, failed to bind to vfio-pci
15b7:5002 is an nvme
8086:24fd is a wifi card
10de:1b80 10f0 is the GTX 1080 and its sound interface

same command from another boot where I tried to force the gt310 to vfio

root@Tower:~# dmesg |grep vfio
[ 0.000000] Command line: BOOT_IMAGE=/bzimage vfio-pci.ids=10de:0a63,10de:0be3,10de:1b80,10de:10f0,1002:6707,1002:aa80,15b7:5002,8086:24fd pcie_acs_override=downstream,multifunction initrd=/bzroot
[ 0.000000] Kernel command line: BOOT_IMAGE=/bzimage vfio-pci.ids=10de:0a63,10de:0be3,10de:1b80,10de:10f0,1002:6707,1002:aa80,15b7:5002,8086:24fd pcie_acs_override=downstream,multifunction initrd=/bzroot
[ 7.640943] vfio-pci: probe of 0000:42:00.0 failed with error -22
[ 7.641226] vfio_pci: add [10de:0a63[ffff:ffff]] class 0x000000/00000000
[ 7.641505] vfio-pci: probe of 0000:42:00.0 failed with error -22
[ 7.641834] vfio-pci: probe of 0000:42:00.1 failed with error -22
[ 7.642113] vfio_pci: add [10de:0be3[ffff:ffff]] class 0x000000/00000000
[ 7.642416] vfio-pci 0000:0a:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=io+mem
[ 7.654216] vfio-pci: probe of 0000:42:00.0 failed with error -22
[ 7.654463] vfio-pci: probe of 0000:42:00.1 failed with error -22
[ 7.654640] vfio_pci: add [10de:1b80[ffff:ffff]] class 0x000000/00000000
[ 7.666150] vfio-pci: probe of 0000:42:00.0 failed with error -22
[ 7.674072] vfio-pci: probe of 0000:42:00.1 failed with error -22
[ 7.681859] vfio_pci: add [10de:10f0[ffff:ffff]] class 0x000000/00000000
[ 7.689664] vfio-pci: probe of 0000:41:00.0 failed with error -22
[ 7.697457] vfio-pci: probe of 0000:42:00.0 failed with error -22
[ 7.705294] vfio-pci: probe of 0000:42:00.1 failed with error -22
[ 7.713088] vfio_pci: add [1002:6707[ffff:ffff]] class 0x000000/00000000
[ 7.720922] vfio-pci: probe of 0000:41:00.0 failed with error -22
[ 7.728830] vfio-pci: probe of 0000:41:00.1 failed with error -22
[ 7.736658] vfio-pci: probe of 0000:42:00.0 failed with error -22
[ 7.744503] vfio-pci: probe of 0000:42:00.1 failed with error -22
[ 7.752284] vfio_pci: add [1002:aa80[ffff:ffff]] class 0x000000/00000000
[ 7.772114] vfio-pci: probe of 0000:41:00.0 failed with error -22
[ 7.780100] vfio-pci: probe of 0000:41:00.1 failed with error -22
[ 7.787892] vfio-pci: probe of 0000:42:00.0 failed with error -22
[ 7.795775] vfio-pci: probe of 0000:42:00.1 failed with error -22
[ 7.803572] vfio_pci: add [15b7:5002[ffff:ffff]] class 0x000000/00000000
[ 7.823195] vfio-pci: probe of 0000:41:00.0 failed with error -22
[ 7.831145] vfio-pci: probe of 0000:41:00.1 failed with error -22
[ 7.838947] vfio-pci: probe of 0000:42:00.0 failed with error -22
[ 7.846866] vfio-pci: probe of 0000:42:00.1 failed with error -22
[ 7.854697] vfio_pci: add [8086:24fd[ffff:ffff]] class 0x000000/00000000

long story short, devices in slot 1 and 2 are unable to bind to vfio, with error -22

4

Any ideas?