Well, it looks like that is not the issue.
Nvtop shows PCIE Gen 3@16x on my 2080 TI.
I wonder if resetting the PCIE bridge – or whatever happens when passthrough is triggered temporarily – causes a temporary bifurcation? No, it’s probably more complicated than that.
@wendell You seem to be watching this thread, do you have one of these boards and the ability to test passthrough in the top slot? I found another thread on the unraid forums with the same issue.
Okkkkay, so, I may not have completely solved the problem, but, I do have new suspicions.
I think, that there is some connection between the 3x M.2 slots on the motherboard and some of the PCIE slots. The manual does not indicate a mapping between the M.2 slots and any specific PCIe slot, but, I had trouble getting devices to appear in the top slot when having M2_2 and M2_3 populated. More thorough investigation is required, but, my suspicions are peaked.
I have been unable to find a block diagram of this motherboard, but really wish I had one right now.
I’ve run into issues with using the m2 and the u2 slots together, even though the manual says those shouldnt have been in contention.
from the manual
• The M.2_2 slot shares bandwidth with U.2_1. When M.2_2 is populated, U.2_1 will
be disabled.
• The M.2_3 slot shares bandwidth with U.2_2. When M.2_3 is populated, U.2_2 will
be disabled.
so only 2_1 is not in contention
I have not attempted to use the M.2 and U.2 slots at the same time. I think that these M.2/U.2 slots are somehow co-mingled with the PCIE slots. Hence why I want a block diagram.
| ID | Mode | 4x | 4x | 4x | 4x |
|---|---|---|---|---|---|
| PCIE 1 | 4x4x4x4x | Corsair MP600 [500GB] | Corsair MP600 [500GB] | Empty | Empty |
| PCIE 2 | 4x4x4x4x | Teamgroup Z440 [1TB] | Teamgroup Z440 [1TB] | Teamgroup Z440 [1TB] | Teamgroup Z440 [1TB] |
| PCIE 3 | 4x4x4x4x | Fresco FL1100 USB3 | Empty | Empty | Empty |
| PCIE 4 | 16x | Nvidia GTX 1080 Ti | |||
| PCIE 5 | 16x | Nvidia Quadro P400 | |||
| PCIE 6 | 16x | Nvidia Quadro P400 | |||
| PCIE 7 | 16x | Nvidia GTX 2080 Ti | |||
| M.2_1 | 4x | Corsair MP600 [500GB] | |||
| M.2_2/U.2_1 | 4x | Empty | |||
| M.2_3/U.2_2 | 4x | Empty |
All of my passthrough testing of PCIE 1 was done when M.2_1/2/3 were all fully populated with Corsair MP600s. I have since moved 2 of them to PCIE 1. This freed up me to use U.2_1 and U.2_2 for other purposes. However, when I tried to use U.2_1/2 I could not boot. I reseated cables and tried again and again. I still could not boot. I then installed a 4x4x4x4x to SFF-8643 (connector used for U.2) card into PCIE 3 and was able to use the device I had been trying to install in U.2_1. I have not tried the U.2_1 or U.2_2 since, but I’ll be looking to get an Intel P5800x in the next month or two and I will probably try again then.
PCIE Bus IDs:
| ID | Mode | 4x | 4x | 4x | 4x |
|---|---|---|---|---|---|
| PCIE 1 | 4x4x4x4x | 41 | 42 | Empty | Empty |
| PCIE 2 | 4x4x4x4x | 01 | 02 | 03 | 04 |
| PCIE 3 | 4x4x4x4x | 2f | Empty | Empty | Empty |
| PCIE 4 | 16x | 62 | |||
| PCIE 5 | 16x | 61 | |||
| PCIE 6 | 16x | 43 | |||
| PCIE 7 | 16x | 05 | |||
| M.2_1 | 4x | 2e | |||
| M.2_2/U.2_1 | 4x | Empty | |||
| M.2_3/U.2_2 | 4x | Empty |
IOMMU Group IDs:
| ID | Mode | 4x | 4x | 4x | 4x |
|---|---|---|---|---|---|
| PCIE 1 | 4x4x4x4x | 27 | 28 | Empty | Empty |
| PCIE 2 | 4x4x4x4x | 84 | 85 | 86 | 87 |
| PCIE 3 | 4x4x4x4x | 62 | Empty | Empty | Empty |
| PCIE 4 | 16x | 12 | |||
| PCIE 5 | 16x | 11 | |||
| PCIE 6 | 16x | 29 | |||
| PCIE 7 | 16x | 88 | |||
| M.2_1 | 4x | 61 | |||
| M.2_2/U.2_1 | 4x | Empty | |||
| M.2_3/U.2_2 | 4x | Empty |
More Detailed PCIE Layouts:
root@pve-01:~# lspci -tv
-+-[0000:60]-+-00.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex
| +-00.2 Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU
| +-01.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
| +-01.1-[61]--+-00.0 NVIDIA Corporation GP107GL [Quadro P400]
| | \-00.1 NVIDIA Corporation GP107GL High Definition Audio Controller
| +-02.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
| +-03.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
| +-03.1-[62]--+-00.0 NVIDIA Corporation GP102 [GeForce GTX 1080 Ti]
| | \-00.1 NVIDIA Corporation GP102 HDMI Audio Controller
| +-04.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
| +-05.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
| +-07.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
| +-07.1-[63]----00.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
| +-08.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
| \-08.1-[64]----00.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
+-[0000:40]-+-00.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex
| +-00.2 Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU
| +-01.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
| +-01.1-[41]----00.0 Phison Electronics Corporation Device 5016
| +-01.2-[42]----00.0 Phison Electronics Corporation Device 5016
| +-02.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
| +-03.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
| +-03.1-[43]--+-00.0 NVIDIA Corporation GP107GL [Quadro P400]
| | \-00.1 NVIDIA Corporation GP107GL High Definition Audio Controller
| +-04.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
| +-05.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
| +-07.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
| +-07.1-[44]----00.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
| +-08.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
| \-08.1-[45]----00.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
+-[0000:20]-+-00.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex
| +-00.2 Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU
| +-01.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
| +-01.1-[21-2d]----00.0-[22-2d]--+-01.0-[23]----00.0 ASMedia Technology Inc. Device 3242
| | +-02.0-[24-25]--+-00.0 Intel Corporation Ethernet Controller 10G X550T
| | | \-00.1 Intel Corporation Ethernet Controller 10G X550T
| | +-03.0-[26]----00.0 Intel Corporation Device 2723
| | +-04.0-[27]----00.0 ASMedia Technology Inc. ASM1061 SATA IDE Controller
| | +-05.0-[28]----00.0 ASMedia Technology Inc. ASM1061 SATA IDE Controller
| | +-06.0-[29-2a]----00.0-[2a]----00.0 ASPEED Technology, Inc. ASPEED Graphics Family
| | +-08.0-[2b]--+-00.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
| | | +-00.1 Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
| | | \-00.3 Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
| | +-09.0-[2c]----00.0 Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode]
| | \-0a.0-[2d]----00.0 Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode]
| +-01.2-[2e]----00.0 Phison Electronics Corporation Device 5016
| +-02.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
| +-03.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
| +-03.1-[2f]----00.0 Fresco Logic FL1100 USB 3.0 Host Controller
| +-04.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
| +-05.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
| +-07.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
| +-07.1-[30]----00.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
| +-08.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
| \-08.1-[31]--+-00.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
| +-00.1 Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP
| +-00.3 Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller
| \-00.4 Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller
\-[0000:00]-+-00.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex
+-00.2 Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU
+-01.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
+-01.1-[01]----00.0 Phison Electronics Corporation Device 5016
+-01.2-[02]----00.0 Phison Electronics Corporation Device 5016
+-01.3-[03]----00.0 Phison Electronics Corporation Device 5016
+-01.4-[04]----00.0 Phison Electronics Corporation Device 5016
+-02.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
+-03.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
+-03.1-[05]--+-00.0 NVIDIA Corporation TU102 [GeForce RTX 2080 Ti Rev. A]
| +-00.1 NVIDIA Corporation TU102 High Definition Audio Controller
| +-00.2 NVIDIA Corporation TU102 USB 3.1 Controller
| \-00.3 NVIDIA Corporation TU102 UCSI Controller
+-04.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
+-05.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
+-07.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
+-07.1-[06]----00.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
+-08.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
+-08.1-[07]--+-00.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
| \-00.3 Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller
+-14.0 Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller
+-14.3 Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge
+-18.0 Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 0
+-18.1 Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 1
+-18.2 Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 2
+-18.3 Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 3
+-18.4 Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 4
+-18.5 Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 5
+-18.6 Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 6
\-18.7 Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 7
And finally a snippet from lstopo
root@pve-01:~# lstopo -v
...
Bridge Host->PCI L#0 (P#0 buses=0000:[00-07])
Bridge PCI->PCI (P#17 busid=0000:00:01.1 id=1022:1483 class=0604(PCI_B) link=7.88GB/s buses=0000:[01-01] PCIVendor="Advanced Micro Devices, Inc. [AMD]") "Advanced Micro Devices, Inc. [AMD]"
PCI 1987:5016 (P#4096 busid=0000:01:00.0 class=0108(NVMExp) link=7.88GB/s)
Block(Disk) L#0 (LinuxDeviceID=259:4 Model="T-FORCE TM8FP7001T" Revision=EGFM13.0 SerialNumber=TPBF2012280040201489 Type=Disk) "nvme3n1"
Bridge PCI->PCI (P#18 busid=0000:00:01.2 id=1022:1483 class=0604(PCI_B) link=7.88GB/s buses=0000:[02-02] PCIVendor="Advanced Micro Devices, Inc. [AMD]") "Advanced Micro Devices, Inc. [AMD]"
PCI 1987:5016 (P#8192 busid=0000:02:00.0 class=0108(NVMExp) link=7.88GB/s)
Block(Disk) L#1 (LinuxDeviceID=259:5 Model="T-FORCE TM8FP7001T" Revision=EGFM13.0 SerialNumber=TPBF2101080050300570 Type=Disk) "nvme4n1"
Bridge PCI->PCI (P#19 busid=0000:00:01.3 id=1022:1483 class=0604(PCI_B) link=7.88GB/s buses=0000:[03-03] PCIVendor="Advanced Micro Devices, Inc. [AMD]") "Advanced Micro Devices, Inc. [AMD]"
PCI 1987:5016 (P#12288 busid=0000:03:00.0 class=0108(NVMExp) link=7.88GB/s)
Block(Disk) L#2 (LinuxDeviceID=259:3 Model="T-FORCE TM8FP7001T" Revision=EGFM13.0 SerialNumber=TPBF2012280060200142 Type=Disk) "nvme5n1"
Bridge PCI->PCI (P#20 busid=0000:00:01.4 id=1022:1483 class=0604(PCI_B) link=7.88GB/s buses=0000:[04-04] PCIVendor="Advanced Micro Devices, Inc. [AMD]") "Advanced Micro Devices, Inc. [AMD]"
PCI 1987:5016 (P#16384 busid=0000:04:00.0 class=0108(NVMExp) link=7.88GB/s)
Block(Disk) L#3 (LinuxDeviceID=259:6 Model="T-FORCE TM8FP7001T" Revision=EGFM13.0 SerialNumber=TPBF2101080050300167 Type=Disk) "nvme6n1"
Bridge PCI->PCI (P#49 busid=0000:00:03.1 id=1022:1483 class=0604(PCI_B) link=4.00GB/s buses=0000:[05-05] PCIVendor="Advanced Micro Devices, Inc. [AMD]") "Advanced Micro Devices, Inc. [AMD]"
PCI 10de:1e07 (P#20480 busid=0000:05:00.0 class=0300(VGA) link=4.00GB/s PCIVendor="NVIDIA Corporation") "NVIDIA Corporation"
Bridge Host->PCI L#6 (P#1 buses=0000:[20-31])
Bridge PCI->PCI (P#131089 busid=0000:20:01.1 id=1022:1483 class=0604(PCI_B) link=15.75GB/s buses=0000:[21-2d] PCIVendor="Advanced Micro Devices, Inc. [AMD]") "Advanced Micro Devices, Inc. [AMD]"
Bridge PCI->PCI (P#135168 busid=0000:21:00.0 id=1022:57ad class=0604(PCI_B) link=15.75GB/s buses=0000:[22-2d] PCIVendor="Advanced Micro Devices, Inc. [AMD]") "Advanced Micro Devices, Inc. [AMD]"
Bridge PCI->PCI (P#139296 busid=0000:22:02.0 id=1022:57a3 class=0604(PCI_B) link=3.94GB/s buses=0000:[24-25] PCIVendor="Advanced Micro Devices, Inc. [AMD]") "Advanced Micro Devices, Inc. [AMD]"
PCI 8086:1563 (P#147456 busid=0000:24:00.0 class=0200(Ether) link=3.94GB/s PCIVendor="Intel Corporation" PCIDevice="Ethernet Controller 10G X550T") "Intel Corporation Ethernet Controller 10G X550T"
Network L#4 (Address=fc:34:97:a5:cc:7c) "enp36s0f0"
PCI 8086:1563 (P#147457 busid=0000:24:00.1 class=0200(Ether) link=3.94GB/s PCIVendor="Intel Corporation" PCIDevice="Ethernet Controller 10G X550T") "Intel Corporation Ethernet Controller 10G X550T"
Network L#5 (Address=fc:34:97:a5:cc:7d) "enp36s0f1"
Bridge PCI->PCI (P#139312 busid=0000:22:03.0 id=1022:57a3 class=0604(PCI_B) link=0.50GB/s buses=0000:[26-26] PCIVendor="Advanced Micro Devices, Inc. [AMD]") "Advanced Micro Devices, Inc. [AMD]"
PCI 8086:2723 (P#155648 busid=0000:26:00.0 class=0280(Net) link=0.50GB/s PCIVendor="Intel Corporation") "Intel Corporation"
Network L#6 (Address=44:af:28:79:65:1c) "wlp38s0"
Bridge PCI->PCI (P#139328 busid=0000:22:04.0 id=1022:57a3 class=0604(PCI_B) link=0.50GB/s buses=0000:[27-27] PCIVendor="Advanced Micro Devices, Inc. [AMD]") "Advanced Micro Devices, Inc. [AMD]"
PCI 1b21:0611 (P#159744 busid=0000:27:00.0 class=0101(IDE) link=0.50GB/s PCIVendor="ASMedia Technology Inc." PCIDevice="ASM1061 SATA IDE Controller") "ASMedia Technology Inc. ASM1061 SATA IDE Controller"
Bridge PCI->PCI (P#139344 busid=0000:22:05.0 id=1022:57a3 class=0604(PCI_B) link=0.50GB/s buses=0000:[28-28] PCIVendor="Advanced Micro Devices, Inc. [AMD]") "Advanced Micro Devices, Inc. [AMD]"
PCI 1b21:0611 (P#163840 busid=0000:28:00.0 class=0101(IDE) link=0.50GB/s PCIVendor="ASMedia Technology Inc." PCIDevice="ASM1061 SATA IDE Controller") "ASMedia Technology Inc. ASM1061 SATA IDE Controller"
Bridge PCI->PCI (P#139360 busid=0000:22:06.0 id=1022:57a3 class=0604(PCI_B) link=0.50GB/s buses=0000:[29-2a] PCIVendor="Advanced Micro Devices, Inc. [AMD]") "Advanced Micro Devices, Inc. [AMD]"
Bridge PCI->PCI (P#167936 busid=0000:29:00.0 id=1a03:1150 class=0604(PCI_B) link=0.50GB/s buses=0000:[2a-2a] PCIVendor="ASPEED Technology, Inc." PCIDevice="AST1150 PCI-to-PCI Bridge") "ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge"
PCI 1a03:2000 (P#172032 busid=0000:2a:00.0 class=0300(VGA) PCIVendor="ASPEED Technology, Inc." PCIDevice="ASPEED Graphics Family") "ASPEED Technology, Inc. ASPEED Graphics Family"
GPU L#7 "controlD64"
GPU L#8 "card0"
Bridge PCI->PCI (P#139408 busid=0000:22:09.0 id=1022:57a4 class=0604(PCI_B) link=31.51GB/s buses=0000:[2c-2c] PCIVendor="Advanced Micro Devices, Inc. [AMD]") "Advanced Micro Devices, Inc. [AMD]"
PCI 1022:7901 (P#180224 busid=0000:2c:00.0 class=0106(SATA) link=31.51GB/s PCIVendor="Advanced Micro Devices, Inc. [AMD]" PCIDevice="FCH SATA Controller [AHCI mode]") "Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode]"
Bridge PCI->PCI (P#139424 busid=0000:22:0a.0 id=1022:57a4 class=0604(PCI_B) link=31.51GB/s buses=0000:[2d-2d] PCIVendor="Advanced Micro Devices, Inc. [AMD]") "Advanced Micro Devices, Inc. [AMD]"
PCI 1022:7901 (P#184320 busid=0000:2d:00.0 class=0106(SATA) link=31.51GB/s PCIVendor="Advanced Micro Devices, Inc. [AMD]" PCIDevice="FCH SATA Controller [AHCI mode]") "Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode]"
Bridge PCI->PCI (P#131090 busid=0000:20:01.2 id=1022:1483 class=0604(PCI_B) link=7.88GB/s buses=0000:[2e-2e] PCIVendor="Advanced Micro Devices, Inc. [AMD]") "Advanced Micro Devices, Inc. [AMD]"
PCI 1987:5016 (P#188416 busid=0000:2e:00.0 class=0108(NVMExp) link=7.88GB/s)
Block(Disk) L#9 (LinuxDeviceID=259:2 Model="Force MP600" Revision=EGFM11.3 SerialNumber=20118231000128541383 Type=Disk) "nvme2n1"
Bridge Host->PCI L#18 (P#2 buses=0000:[40-45])
Bridge PCI->PCI (P#262161 busid=0000:40:01.1 id=1022:1483 class=0604(PCI_B) link=7.88GB/s buses=0000:[41-41] PCIVendor="Advanced Micro Devices, Inc. [AMD]") "Advanced Micro Devices, Inc. [AMD]"
PCI 1987:5016 (P#266240 busid=0000:41:00.0 class=0108(NVMExp) link=7.88GB/s PCISlot=0)
Block(Disk) L#10 (LinuxDeviceID=259:1 Model="Force MP600" Revision=EGFM11.3 SerialNumber=20118231000128541112 Type=Disk) "nvme0n1"
Bridge PCI->PCI (P#262162 busid=0000:40:01.2 id=1022:1483 class=0604(PCI_B) link=7.88GB/s buses=0000:[42-42] PCIVendor="Advanced Micro Devices, Inc. [AMD]") "Advanced Micro Devices, Inc. [AMD]"
PCI 1987:5016 (P#270336 busid=0000:42:00.0 class=0108(NVMExp) link=7.88GB/s)
Block(Disk) L#11 (LinuxDeviceID=259:0 Model="Force MP600" Revision=EGFM11.3 SerialNumber=20118231000128541385 Type=Disk) "nvme1n1"
Bridge PCI->PCI (P#262193 busid=0000:40:03.1 id=1022:1483 class=0604(PCI_B) link=4.00GB/s buses=0000:[43-43] PCIVendor="Advanced Micro Devices, Inc. [AMD]") "Advanced Micro Devices, Inc. [AMD]"
PCI 10de:1cb3 (P#274432 busid=0000:43:00.0 class=0300(VGA) link=4.00GB/s PCIVendor="NVIDIA Corporation") "NVIDIA Corporation"
Bridge Host->PCI L#22 (P#3 buses=0000:[60-64])
Bridge PCI->PCI (P#393233 busid=0000:60:01.1 id=1022:1483 class=0604(PCI_B) link=15.75GB/s buses=0000:[61-61] PCIVendor="Advanced Micro Devices, Inc. [AMD]") "Advanced Micro Devices, Inc. [AMD]"
PCI 10de:1cb3 (P#397312 busid=0000:61:00.0 class=0300(VGA) link=15.75GB/s PCIVendor="NVIDIA Corporation") "NVIDIA Corporation"
Bridge PCI->PCI (P#393265 busid=0000:60:03.1 id=1022:1483 class=0604(PCI_B) link=4.00GB/s buses=0000:[62-62] PCIVendor="Advanced Micro Devices, Inc. [AMD]") "Advanced Micro Devices, Inc. [AMD]"
PCI 10de:1b06 (P#401408 busid=0000:62:00.0 class=0300(VGA) link=4.00GB/s PCIVendor="NVIDIA Corporation") "NVIDIA Corporation"
...So, anyone had any luck getting the topmost slot to passthrough?
I have a similar setup and problem and it did get solved by moving the GPU one slot down, but that puts it flush against another card and it starves of air and throttles a lot.
The only M.2 slot used is M.2_1.
I just noticed: in your screenshots of the bios you have the setting on automatic instead of on.
Automatic only halve activates iommu settings.
@DotBowder - Did you ever make any progress on this issue. I am just building a VERY similar setup right now and I am having the same issue.
@TheVault I went in and enabled IOMMU directly, with no change. I’ve been in and out the the bios so many times and tried so many different configurations.
@SuperDodge I seem to be able to pass 4X cards through using the top slot but ONLY when in 4x4x4x4x mode, not in 16x mode. So for instance, one can pass through a 1x or 4x USB card in the top slot in bifurcation mode, but not a 16x graphics card in 16x mode (Also, don’t try a 16x card in 4x mode, It is not happy about that).
What I finally resorted to was using my storage NVME 4x4x4x4x card in the top slot so I never have to worry about doing passthrough on that specific slot.
Other recommendations:
I would recommend manually setting the link speed on each one of the slots you intend to use. Manually set PCIE 3.0 when you don’t need PCIE 4.0 and so on. (That was the source of a lot of my dmesg pcie errors that muddy the waters more.)
If you want to do a USB 3 passthrough, but don’t want to take up a full PCIE slot, I picked up one of these off of ebay.
It’s a USB controller on an M.2 form factor. It requires some guest OS loading of drivers: Rensas USB 3.0 cards VFIO Passthrough possible fix through firmware upgrade (Read the recent posts) but it works really well to convert the top M.2 slot into a easy pass-through USB controller.
I have had the worst headaches trying to pass-through the USB controllers on the mobo. It works, but, then after a VM reboot, it sometimes doesn’t. If you’ve found a way to make that easier, please post a link.
Finally, I made a video on programatically pinning virtual machine to specific CPU cores which you may find helpful. Virtual Machine CPU Pinning using Proxmox - YouTube
Another painpoint I’ve run into with passing through hardware is changing PCIE locations.
When you pass a card through, you’re pass it’s Bus:Slot.Function value, but as you add devices, that can change. The values should only ever change if you enable/disable add/remove pcie devices, but this is something I do reasonably regularly.
If the system boots up and tries to pass through some other random PCIe device, you can run into lots of baaaaad issues. So, whenever doing hardware changes, I switch my VMs to not start on boot so that I can manually verify and re-assign the PCIe devices after the hw change.
Also, just don’t use the SATA controller(s).
Every time I try to add SATA devices directly to the SATA controllers (I tried again this morning) ZFS starts seeing data corruption when accessing the drives at full speed.
You’re better off setting the PCIE U.2 ports to SATA mode in the BIOS and using one of these: Amazon.com : SFF8643 to sata
ugggg @DotBowder you are seeing SATA errors too? I recently upgraded to the latest BIOS and I have tried the SATA in both SATA RAID mode as well as AHCI. After about 1TB of data written, I get errors in dmesg and ubuntu appears to kick out the drive (/dev/sda goes away). the most common error I see is “write beyond end of disk.” Have tried with a Samsung QVO 8TB but also recently saw same errors with an Exos 18TB.
Is this along the lines of what you’re seeing with ZFS? dumb question but do you have good cooling on your SATA drives? My case does not have fans there and I saw SMART reporting temps of 55C or so, which seemed a bit high.
I’d love to hear if you have any more info on your experience with the built-in SATA because I’m almost ready to RMA the hard drive.
Indeed, I see a similar behavior to you. After about 1TB of data written... The drives appear totally fine for like 5 seconds, then speeds drop down to the 10MBps, then 5MBps, then 1MBps, then plateau at nothing. ZFS logs indicated that the drives were not responding or were slow to respond. Then my ZFS pool kicks offline (after much waiting for the drives to respond).
It does seem to be agitated and initiated by intensive writing, but it doesn’t matter what drive I put into it. It happens with SanDisk SSD Plus 1TB, ADATA SU635 256GB, and Crutial BX500 1TB, Seagate Exos 14TB drives, and Seagate 8TB drives.
My best guess is that the SATA controller has a firmware bug, or the linux drivers associated with them are bugged. I also tried passing through drives to a windows machine via the qemu scsi adapter, and they do continue to work for long periods of time, but, they are no where near as performant as they should be and I have no real idea if they’re kicking out bad data or not (haven’t tested).
The way that I have worked around this issue is to use the U.2/MINISAS style connectors in SATA mode (must change in bios). When plugging in drives to these plugs, things work without any problems. The sata ports are still borked when I use the U.2 ports – What is REALLLY weird about this is:
See below an example of 8 drives on U.2 and 4 drives in SATA ports.
/dev/sda - U.2/MINISAS
/dev/sdb - U.2/MINISAS
/dev/sdc - U.2/MINISAS
/dev/sdd - U.2/MINISAS
/dev/sdi - U.2/MINISAS
/dev/sdj - U.2/MINISAS
/dev/sdk - U.2/MINISAS
/dev/sdl - U.2/MINISAS
/dev/sde - SATA
/dev/sdf - SATA
/dev/sdg - SATA
/dev/sdh - SATA
Even if I turn off the SATA controllers in the bios, a new sata controller sprouts up next to the U.2 one to show the SATA devices. The SATA devices still don’t perform properly though, only the U.2 ones.
root@pve-01:~# ls -hal /dev/disk/by-id | grep -v “part” | grep -e “sd[a-z]”
lrwxrwxrwx 1 root root 9 Feb 5 12:23 ata-CT1000BX500SSD1_2125E5AD9163 → …/…/sde
lrwxrwxrwx 1 root root 9 Feb 5 12:23 ata-CT1000BX500SSD1_2125E5AD929B → …/…/sdf
lrwxrwxrwx 1 root root 9 Feb 5 12:23 ata-SanDisk_SSD_PLUS_1000GB_202774801540 → …/…/sdg
lrwxrwxrwx 1 root root 9 Feb 5 12:23 ata-SanDisk_SSD_PLUS_1000GB_203805A0017F → …/…/sdh
lrwxrwxrwx 1 root root 9 Feb 5 11:00 ata-ST14000NM001G-2KJ103_ZL29YDP2 → …/…/sdi
lrwxrwxrwx 1 root root 9 Feb 5 11:00 ata-ST14000NM001G-2KJ103_ZTM070J3 → …/…/sdj
lrwxrwxrwx 1 root root 9 Feb 5 11:00 ata-ST14000NM001G-2KJ103_ZTM0EHTE → …/…/sdc
lrwxrwxrwx 1 root root 9 Feb 5 11:00 ata-ST14000NM001G-2KJ103_ZTM0ER63 → …/…/sdb
lrwxrwxrwx 1 root root 9 Feb 5 11:00 ata-ST14000NM001G-2KJ103_ZTM0EYW2 → …/…/sda
lrwxrwxrwx 1 root root 9 Feb 5 11:00 ata-ST14000NM001G-2KJ103_ZTM0FDJV → …/…/sdd
lrwxrwxrwx 1 root root 9 Feb 5 11:00 ata-ST8000VN0022-2EL112_ZA1J4CAV → …/…/sdl
lrwxrwxrwx 1 root root 9 Feb 5 11:00 ata-ST8000VN004-2M2101_WKD18FVF → …/…/sdkroot@pve-01:~# ls -hal /dev/disk/by-path/ | grep -v -e “part” -e “.0 ->” | grep -e “sd[a-z]”
lrwxrwxrwx 1 root root 9 Feb 5 11:00 pci-0000:2b:00.0-ata-1 → …/…/sda # U.2 Port
lrwxrwxrwx 1 root root 9 Feb 5 11:00 pci-0000:2b:00.0-ata-2 → …/…/sdb # U.2 Port
lrwxrwxrwx 1 root root 9 Feb 5 11:00 pci-0000:2b:00.0-ata-3 → …/…/sdc # U.2 Port
lrwxrwxrwx 1 root root 9 Feb 5 11:00 pci-0000:2b:00.0-ata-4 → …/…/sdd # U.2 Port
lrwxrwxrwx 1 root root 9 Feb 5 12:23 pci-0000:2c:00.0-ata-1 → …/…/sde # SATA Port
lrwxrwxrwx 1 root root 9 Feb 5 12:23 pci-0000:2c:00.0-ata-2 → …/…/sdf # SATA Port
lrwxrwxrwx 1 root root 9 Feb 5 12:23 pci-0000:2c:00.0-ata-5 → …/…/sdg # SATA Port
lrwxrwxrwx 1 root root 9 Feb 5 12:23 pci-0000:2c:00.0-ata-6 → …/…/sdh # SATA Port
lrwxrwxrwx 1 root root 9 Feb 5 11:00 pci-0000:31:00.0-ata-5 → …/…/sdi # U.2 Port
lrwxrwxrwx 1 root root 9 Feb 5 11:00 pci-0000:31:00.0-ata-6 → …/…/sdj # U.2 Port
lrwxrwxrwx 1 root root 9 Feb 5 11:00 pci-0000:31:00.0-ata-7 → …/…/sdk # U.2 Port
lrwxrwxrwx 1 root root 9 Feb 5 11:00 pci-0000:31:00.0-ata-8 → …/…/sdl # U.2 Portroot@pve-01:~# lspci | grep SATA
26:00.0 IDE interface: ASMedia Technology Inc. ASM1061 SATA IDE Controller (rev 02)
27:00.0 IDE interface: ASMedia Technology Inc. ASM1061 SATA IDE Controller (rev 02)
2b:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
2c:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
31:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
In this example, I have the SSDs plugged into the SATA ports, but I can’t trust them to work properly, so I don’t have them in use.
Anyway, that’s my headache of an experience that I’ve had with SATA devices on this motherboard.
@DotBowder wow! thank you so so much for the details here!
edit: @DotBowder I contacted ASUS support and they’re gonna let me send in a request for their lab to reproduce. thank you again! if you don’t mind I’ll note your linux version in affected softwares.
One unrelated question: i have wanted to try GPU passthru into Windows on this machine. I’ve only tried virtualbox and vmware (free version). is there a free solution that works, or have you used something more fancy?
Some more details on the SATA woes:
So what I’ve seen in dmesg is Ubuntu turning the SATA speed link from 6g to 3g then 1.5g then finally punting the drive. That might help explain the degradation you see with ZFS. That said, if I initially connect a drive and don’t do many writes at first, I’m able to get full 250MB/s or more for about 1TB before “write beyond end of drive.” If I physically unplug and re-plug the device then (hot-swap), I usually get an immediate 6g->3g->1.5g degradation, as if ubuntu wants to just kick out the drive. After reboot, ubuntu will accept the drive again but I still get errors.
Yes I saw your link about the adapters, thank you! In my case I need those M.2 slots though so I’m trying to figure out a similar mitigation.
Your note about the Windows passthru is really interesting, thank you! I myself have seen issues only with Ubuntu, but both vanilla ext4 on the disk as well as mdadm raid across two devices. If the passthru works as you say then perhaps it is indeed a kernel issue. My theory about the 1TB limit that I’m seeing is that there’s some sort of AHCI feature for large drives that’s getting borked.
I am going to try contacting ASUS. Thank you again for all these details! They also screwed up nvme RAID in the new BIOS-- new BIOS does not remember PCIe bifurcation settings (for Asus Hyper M.2 card) after a power disconnect. So reboots it’ll remember, but if you have a full unplug, you have to manually set up again in BIOS.
Here’s an output from inxi.
root@pve-01:~# inxi -F
System: Host: pve-01 Kernel: 5.13.19-3-pve x86_64 bits: 64 Console: tty 6 Distro: Debian GNU/Linux 11 (bullseye)
Machine: Type: Desktop System: ASUS product: N/A v: N/A serial: N/A
Mobo: ASUSTeK model: Pro WS WRX80E-SAGE SE WIFI v: Rev 1.xx serial: 210483465700035
UEFI: American Megatrends v: 0701 date: 09/15/2021
CPU: Info: 32-Core (4-Die) model: AMD Ryzen Threadripper PRO 3975WX s bits: 64 type: MT MCP MCM
L2 cache: 16 MiB
Speed: 4267 MHz min/max: 2200/3500 MHz Core speeds (MHz): 1: 4267 2: 4250 3: 3413 4: 4186 5: 4231 6: 4116
7: 3996 8: 4245 9: 3593 10: 3632 11: 3840 12: 3598 13: 4218 14: 3592 15: 3594 16: 3580 17: 3590 18: 4266
19: 4111 20: 4117 21: 4126 22: 4253 23: 4251 24: 4251 25: 4214 26: 4248 27: 4244 28: 4256 29: 4247
30: 4247 31: 4243 32: 4216 33: 4223 34: 4177 35: 4239 36: 3384 37: 3400 38: 3975 39: 4230 40: 4170
41: 4235 42: 3964 43: 4220 44: 3588 45: 3592 46: 4243 47: 3592 48: 3592 49: 3595 50: 3592 51: 3585
52: 3588 53: 4216 54: 4217 55: 4218 56: 4223 57: 4215 58: 4228 59: 4352 60: 4221 61: 4216 62: 4217
63: 4108 64: 3593
Graphics: Device-1: NVIDIA TU116 [GeForce GTX 1650 SUPER] driver: vfio-pci v: 0.2
Device-2: ASPEED Graphics Family driver: ast v: kernel
Device-3: NVIDIA TU102 [GeForce RTX 2080 Ti Rev. A] driver: vfio-pci v: 0.2
Device-4: NVIDIA GP102 [GeForce GTX 1080 Ti] driver: vfio-pci v: 0.2
Device-5: NVIDIA GP102 [GeForce GTX 1080 Ti] driver: vfio-pci v: 0.2
Device-6: NVIDIA GP102 [GeForce GTX 1080 Ti] driver: vfio-pci v: 0.2
Display: server: No display server data found. Headless machine? tty: 120x40
Message: Advanced graphics data unavailable in console for root.
Audio: Device-1: NVIDIA TU116 High Definition Audio driver: vfio-pci
Device-2: NVIDIA TU102 High Definition Audio driver: vfio-pci
Device-3: Advanced Micro Devices [AMD] Starship/Matisse HD Audio driver: N/A
Device-4: NVIDIA GP102 HDMI Audio driver: vfio-pci
Device-5: NVIDIA GP102 HDMI Audio driver: vfio-pci
Device-6: NVIDIA GP102 HDMI Audio driver: vfio-pci
Sound Server: ALSA v: k5.13.19-3-pve
Network: Device-1: Intel Ethernet 10G X550T driver: ixgbe
IF: enp36s0f0 state: up speed: 10000 Mbps duplex: full mac: fc:34:97:a5:cc:7c
Device-2: Intel Ethernet 10G X550T driver: ixgbe
IF: enp36s0f1 state: down mac: fc:34:97:a5:cc:7d
IF-ID-1: fwbr100i0 state: up speed: 10000 Mbps duplex: unknown mac: 96:36:96:d9:2d:dc
IF-ID-2: fwbr100i1 state: up speed: 10000 Mbps duplex: unknown mac: da:a2:a9:ea:dc:37
IF-ID-3: fwbr100i2 state: up speed: 10000 Mbps duplex: unknown mac: 1a:3f:58:43:61:68
IF-ID-4: fwbr100i3 state: up speed: 10000 Mbps duplex: unknown mac: 9a:44:54:94:31:1d
IF-ID-5: fwbr100i4 state: up speed: 10000 Mbps duplex: unknown mac: 2e:0b:16:ac:9f:43
IF-ID-6: fwbr100i5 state: up speed: 10000 Mbps duplex: unknown mac: ea:1d:e3:50:cd:30
IF-ID-7: fwbr101i0 state: up speed: 10000 Mbps duplex: unknown mac: f6:72:7c:60:ce:93
IF-ID-8: fwbr101i1 state: up speed: 10000 Mbps duplex: unknown mac: e2:6e:09:31:a2:04
IF-ID-9: fwbr101i2 state: up speed: 10000 Mbps duplex: unknown mac: aa:1f:97:77:49:e6
IF-ID-10: fwbr101i3 state: up speed: 10000 Mbps duplex: unknown mac: 0e:35:0d:a1:4b:75
IF-ID-11: fwbr101i4 state: up speed: 10000 Mbps duplex: unknown mac: 86:5b:3c:ec:65:4c
IF-ID-12: fwbr101i5 state: up speed: 10000 Mbps duplex: unknown mac: f2:cc:01:b0:41:33
IF-ID-13: fwbr104i0 state: up speed: 10000 Mbps duplex: unknown mac: 62:20:9c:5e:59:10
IF-ID-14: fwbr301i0 state: up speed: 10000 Mbps duplex: unknown mac: 66:e0:b5:a4:d9:f3
IF-ID-15: fwbr301i1 state: up speed: 10000 Mbps duplex: unknown mac: 66:9a:08:9a:db:8a
IF-ID-16: fwbr301i2 state: up speed: 10000 Mbps duplex: unknown mac: 02:08:33:ca:4b:13
IF-ID-17: fwbr301i3 state: up speed: 10000 Mbps duplex: unknown mac: 9a:2b:86:ec:80:c0
IF-ID-18: fwbr301i4 state: up speed: 10000 Mbps duplex: unknown mac: 7a:1d:0c:f8:00:8f
IF-ID-19: fwbr301i5 state: up speed: 10000 Mbps duplex: unknown mac: 7e:92:c8:57:af:d9
IF-ID-20: fwbr304i0 state: up speed: 10000 Mbps duplex: unknown mac: 3e:72:0b:92:75:6c
IF-ID-21: fwbr304i1 state: up speed: 10000 Mbps duplex: unknown mac: fa:a6:74:7e:ab:b7
IF-ID-22: fwbr304i2 state: up speed: 10000 Mbps duplex: unknown mac: ea:8d:37:6e:5c:4f
IF-ID-23: fwbr304i3 state: up speed: 10000 Mbps duplex: unknown mac: 82:2d:45:ec:10:23
IF-ID-24: fwbr304i4 state: up speed: 10000 Mbps duplex: unknown mac: ee:51:7c:ac:b7:6a
IF-ID-25: fwbr304i5 state: up speed: 10000 Mbps duplex: unknown mac: 1e:88:68:41:ca:80
IF-ID-26: fwbr501i0 state: up speed: 10000 Mbps duplex: unknown mac: 76:43:01:87:54:6c
IF-ID-27: fwln100i0 state: up speed: 10000 Mbps duplex: full mac: 66:9e:a0:7c:56:6d
IF-ID-28: fwln100i1 state: up speed: 10000 Mbps duplex: full mac: ee:04:32:c8:1a:ba
IF-ID-29: fwln100i2 state: up speed: 10000 Mbps duplex: full mac: 02:89:06:98:93:a3
IF-ID-30: fwln100i3 state: up speed: 10000 Mbps duplex: full mac: da:36:ae:bf:78:73
IF-ID-31: fwln100i4 state: up speed: 10000 Mbps duplex: full mac: 4a:cc:95:8c:de:c4
IF-ID-32: fwln100i5 state: up speed: 10000 Mbps duplex: full mac: ea:be:17:a6:f2:7d
IF-ID-33: fwln101i0 state: up speed: 10000 Mbps duplex: full mac: b2:ec:dd:da:83:37
IF-ID-34: fwln101i1 state: up speed: 10000 Mbps duplex: full mac: a2:80:a6:5f:98:d1
IF-ID-35: fwln101i2 state: up speed: 10000 Mbps duplex: full mac: 62:c0:0b:b8:88:9a
IF-ID-36: fwln101i3 state: up speed: 10000 Mbps duplex: full mac: 96:8f:25:d1:0b:2f
IF-ID-37: fwln101i4 state: up speed: 10000 Mbps duplex: full mac: 66:23:01:e4:ca:d7
IF-ID-38: fwln101i5 state: up speed: 10000 Mbps duplex: full mac: 8e:3a:a6:e0:9e:8b
IF-ID-39: fwln104i0 state: up speed: 10000 Mbps duplex: full mac: 3a:20:a2:ec:05:3a
IF-ID-40: fwln301i0 state: up speed: 10000 Mbps duplex: full mac: 1a:d5:82:f7:32:f7
IF-ID-41: fwln301i1 state: up speed: 10000 Mbps duplex: full mac: 7a:ca:c4:65:c4:2f
IF-ID-42: fwln301i2 state: up speed: 10000 Mbps duplex: full mac: be:69:96:11:be:19
IF-ID-43: fwln301i3 state: up speed: 10000 Mbps duplex: full mac: 9a:b6:1d:7c:d1:b7
IF-ID-44: fwln301i4 state: up speed: 10000 Mbps duplex: full mac: 2a:dc:66:59:79:0f
IF-ID-45: fwln301i5 state: up speed: 10000 Mbps duplex: full mac: da:8c:35:83:d8:be
IF-ID-46: fwln304i0 state: up speed: 10000 Mbps duplex: full mac: f2:7d:5b:44:86:f9
IF-ID-47: fwln304i1 state: up speed: 10000 Mbps duplex: full mac: b6:a0:75:f3:78:91
IF-ID-48: fwln304i2 state: up speed: 10000 Mbps duplex: full mac: 32:ce:74:a5:a1:b2
IF-ID-49: fwln304i3 state: up speed: 10000 Mbps duplex: full mac: aa:b4:12:e4:8b:7a
IF-ID-50: fwln304i4 state: up speed: 10000 Mbps duplex: full mac: ce:21:88:e3:eb:b6
IF-ID-51: fwln304i5 state: up speed: 10000 Mbps duplex: full mac: f6:83:c3:ef:53:cc
IF-ID-52: fwln501i0 state: up speed: 10000 Mbps duplex: full mac: ba:4b:34:39:37:43
IF-ID-53: fwpr100p0 state: up speed: 10000 Mbps duplex: full mac: c2:cd:cd:6a:2a:cf
IF-ID-54: fwpr100p1 state: up speed: 10000 Mbps duplex: full mac: 7e:28:66:2d:d5:c5
IF-ID-55: fwpr100p2 state: up speed: 10000 Mbps duplex: full mac: 36:43:13:0a:37:d6
IF-ID-56: fwpr100p3 state: up speed: 10000 Mbps duplex: full mac: ca:87:7e:c0:0f:62
IF-ID-57: fwpr100p4 state: up speed: 10000 Mbps duplex: full mac: 22:29:74:c5:b4:15
IF-ID-58: fwpr100p5 state: up speed: 10000 Mbps duplex: full mac: 22:6d:c8:cb:84:ff
IF-ID-59: fwpr101p0 state: up speed: 10000 Mbps duplex: full mac: ce:61:a2:e0:c8:ef
IF-ID-60: fwpr101p1 state: up speed: 10000 Mbps duplex: full mac: 92:0d:08:d6:12:9a
IF-ID-61: fwpr101p2 state: up speed: 10000 Mbps duplex: full mac: d6:c3:62:ae:95:bc
IF-ID-62: fwpr101p3 state: up speed: 10000 Mbps duplex: full mac: 2a:a3:e4:07:71:3e
IF-ID-63: fwpr101p4 state: up speed: 10000 Mbps duplex: full mac: 2a:42:68:e5:dd:29
IF-ID-64: fwpr101p5 state: up speed: 10000 Mbps duplex: full mac: 5a:b5:a8:e7:50:e1
IF-ID-65: fwpr104p0 state: up speed: 10000 Mbps duplex: full mac: ba:23:6d:e5:d8:c7
IF-ID-66: fwpr301p0 state: up speed: 10000 Mbps duplex: full mac: 9a:72:b0:0f:83:22
IF-ID-67: fwpr301p1 state: up speed: 10000 Mbps duplex: full mac: 82:f3:b2:bc:82:f5
IF-ID-68: fwpr301p2 state: up speed: 10000 Mbps duplex: full mac: b2:ec:a4:69:ac:26
IF-ID-69: fwpr301p3 state: up speed: 10000 Mbps duplex: full mac: 0a:23:5a:c7:40:76
IF-ID-70: fwpr301p4 state: up speed: 10000 Mbps duplex: full mac: 12:5b:fa:14:41:52
IF-ID-71: fwpr301p5 state: up speed: 10000 Mbps duplex: full mac: 7a:61:94:8d:8a:ed
IF-ID-72: fwpr304p0 state: up speed: 10000 Mbps duplex: full mac: ce:e8:d3:38:f5:01
IF-ID-73: fwpr304p1 state: up speed: 10000 Mbps duplex: full mac: da:f2:cc:b9:ea:83
IF-ID-74: fwpr304p2 state: up speed: 10000 Mbps duplex: full mac: 6e:86:4f:ac:4f:70
IF-ID-75: fwpr304p3 state: up speed: 10000 Mbps duplex: full mac: ea:9f:32:17:45:c6
IF-ID-76: fwpr304p4 state: up speed: 10000 Mbps duplex: full mac: c2:e8:48:de:e6:5a
IF-ID-77: fwpr304p5 state: up speed: 10000 Mbps duplex: full mac: ce:c9:bd:08:03:9d
IF-ID-78: fwpr501p0 state: up speed: 10000 Mbps duplex: full mac: 7a:a0:72:42:b3:98
IF-ID-79: tap100i0 state: unknown speed: 10 Mbps duplex: full mac: 9a:90:3b:38:fd:5e
IF-ID-80: tap100i1 state: unknown speed: 10 Mbps duplex: full mac: 56:01:db:c3:39:2c
IF-ID-81: tap100i2 state: unknown speed: 10 Mbps duplex: full mac: 72:14:e2:dd:be:74
IF-ID-82: tap100i3 state: unknown speed: 10 Mbps duplex: full mac: e6:41:57:71:ad:d9
IF-ID-83: tap100i4 state: unknown speed: 10 Mbps duplex: full mac: f2:cb:c3:bb:a9:c0
IF-ID-84: tap100i5 state: unknown speed: 10 Mbps duplex: full mac: be:e8:22:99:25:06
IF-ID-85: tap101i0 state: unknown speed: 10 Mbps duplex: full mac: 06:ab:c1:7a:9b:d8
IF-ID-86: tap101i1 state: unknown speed: 10 Mbps duplex: full mac: 92:80:d0:6a:e0:90
IF-ID-87: tap101i2 state: unknown speed: 10 Mbps duplex: full mac: 62:c2:7b:b1:ae:cf
IF-ID-88: tap101i3 state: unknown speed: 10 Mbps duplex: full mac: 56:84:e1:37:a2:08
IF-ID-89: tap101i4 state: unknown speed: 10 Mbps duplex: full mac: 52:7e:9a:76:78:d1
IF-ID-90: tap101i5 state: unknown speed: 10 Mbps duplex: full mac: 5e:b9:95:ce:7f:5e
IF-ID-91: tap104i0 state: unknown speed: 10 Mbps duplex: full mac: 96:5d:8c:cd:d3:f5
IF-ID-92: tap301i0 state: unknown speed: 10 Mbps duplex: full mac: 56:ab:1c:98:72:48
IF-ID-93: tap301i1 state: unknown speed: 10 Mbps duplex: full mac: fa:dc:3e:2e:8f:9d
IF-ID-94: tap301i2 state: unknown speed: 10 Mbps duplex: full mac: aa:5e:dc:a5:b1:ed
IF-ID-95: tap301i3 state: unknown speed: 10 Mbps duplex: full mac: 72:a0:26:c4:1f:ca
IF-ID-96: tap301i4 state: unknown speed: 10 Mbps duplex: full mac: de:99:3f:7c:4d:64
IF-ID-97: tap301i5 state: unknown speed: 10 Mbps duplex: full mac: 1e:cc:0c:a4:41:6a
IF-ID-98: tap304i0 state: unknown speed: 10 Mbps duplex: full mac: ea:40:99:47:91:b9
IF-ID-99: tap304i1 state: unknown speed: 10 Mbps duplex: full mac: 72:4c:dd:8d:ae:a2
IF-ID-100: tap304i2 state: unknown speed: 10 Mbps duplex: full mac: 46:99:de:4d:77:57
IF-ID-101: tap304i3 state: unknown speed: 10 Mbps duplex: full mac: 16:b4:12:f1:41:62
IF-ID-102: tap304i4 state: unknown speed: 10 Mbps duplex: full mac: a2:62:3e:78:13:13
IF-ID-103: tap304i5 state: unknown speed: 10 Mbps duplex: full mac: be:66:ef:aa:ba:72
IF-ID-104: tap501i0 state: unknown speed: 10 Mbps duplex: full mac: 9e:d3:de:9b:98:23
IF-ID-105: vmbr0 state: down mac: 66:ca:24:33:b1:da
IF-ID-106: vmbr1 state: up speed: 10000 Mbps duplex: unknown mac: 56:44:56:2d:4c:f0
IF-ID-107: vmbr2 state: down mac: 26:bc:27:cb:02:33
IF-ID-108: ztyxau3ooa state: unknown speed: 10 Mbps duplex: full mac: 06:a1:6a:f0:0c:c0
Bluetooth: Device-1: Intel AX200 Bluetooth type: USB driver: usbfs
Message: Required tool hciconfig not installed. Check --recommends
RAID: Device-1: backup type: zfs status: ONLINE level: mirror size: 7.27 TiB free: 1.8 TiB
Components: Online: N/A
Device-2: cache type: zfs status: - level: no-raid size: N/A free: N/A
Components: Online: N/A
Device-3: rpool type: zfs status: ONLINE level: no-raid size: 222 GiB free: 156 GiB
Components: Online: 1: nvme3n1p3
Device-4: tank type: zfs status: ONLINE size: 38.2 TiB free: 25.6 TiB
Array-1: mirror status: ONLINE size: 12.7 TiB free: 8.69 TiB
Components: Online: N/A
Array-2: mirror status: ONLINE size: 12.7 TiB free: 8.76 TiB
Components: Online: N/A
Array-3: mirror status: ONLINE size: 12.7 TiB free: 8.15 TiB
Components: Online: N/A
Drives: Local Storage: total: raw: 100.23 TiB usable: 145.6 TiB used: 3.2 TiB (2.2%)
ID-1: /dev/nvme0n1 vendor: Corsair model: Force MP600 size: 465.76 GiB
ID-2: /dev/nvme1n1 vendor: Corsair model: Force MP600 size: 465.76 GiB
ID-3: /dev/nvme2n1 vendor: Corsair model: Force MP510 size: 894.25 GiB
ID-4: /dev/nvme3n1 vendor: Corsair model: Force MP510 size: 223.57 GiB
ID-5: /dev/nvme4n1 model: T-FORCE TM8FP7001T size: 931.51 GiB
ID-6: /dev/nvme5n1 model: T-FORCE TM8FP7001T size: 931.51 GiB
ID-7: /dev/nvme6n1 model: T-FORCE TM8FP7001T size: 931.51 GiB
ID-8: /dev/nvme7n1 model: T-FORCE TM8FP7001T size: 931.51 GiB
ID-9: /dev/sda vendor: Seagate model: ST14000NM001G-2KJ103 size: 12.73 TiB
ID-10: /dev/sdb vendor: Seagate model: ST14000NM001G-2KJ103 size: 12.73 TiB
ID-11: /dev/sdc vendor: Seagate model: ST14000NM001G-2KJ103 size: 12.73 TiB
ID-12: /dev/sdd vendor: Seagate model: ST14000NM001G-2KJ103 size: 12.73 TiB
ID-13: /dev/sde vendor: Crucial model: CT1000BX500SSD1 size: 931.51 GiB
ID-14: /dev/sdf vendor: Crucial model: CT1000BX500SSD1 size: 931.51 GiB
ID-15: /dev/sdg vendor: SanDisk model: SSD PLUS 1000GB size: 931.52 GiB
ID-16: /dev/sdh vendor: SanDisk model: SSD PLUS 1000GB size: 931.52 GiB
ID-17: /dev/sdi vendor: Seagate model: ST14000NM001G-2KJ103 size: 12.73 TiB
ID-18: /dev/sdj vendor: Seagate model: ST14000NM001G-2KJ103 size: 12.73 TiB
ID-19: /dev/sdk vendor: Seagate model: ST8000VN004-2M2101 size: 7.28 TiB
ID-20: /dev/sdl vendor: Seagate model: ST8000VN0022-2EL112 size: 7.28 TiB
Partition: ID-1: / size: 209.35 GiB used: 60.07 GiB (28.7%) fs: zfs raid: rpool/ROOT/pve-1
Swap: Alert: No Swap data was found.
Sensors: System Temperatures: cpu: 51.6 C mobo: 35.0 C
Fan Speeds (RPM): fan-1: 0 fan-2: 0 fan-3: 0 fan-4: 0 fan-5: 0 fan-6: 0 fan-7: 0
Info: Processes: 1321 Uptime: 4d 3h 05m Memory: 251.51 GiB used: 164.77 GiB (65.5%) Init: systemd runlevel: 5
Shell: Bash inxi: 3.3.01
I am using PVE Proxmox Virtual Environment as a hypervisor which uses QEMU and KVM for virtualization. The hypervisor itself is not designed to provide a desktop experience, so, I pass through video cards and USB devices/controllers to Linux and Windows virtual machines.
Proxmox is designed as a dedicated hypervisor – so it’s build specifically to host virtual machines. However, Ubuntu is not designed from the ground up as a hypervisor. If you are running Ubuntu server, you may be better in switching to Proxmox for managing virtual machines. If you use this machine as a desktop and are using Ubuntu Desktop, you would likely need to install and use libvirt & virt-manager to manage virtual machines in the QEMU/KVM stack. (GUI friendly)
This guide here looks like it has some of the major relevant steps to managing virtual machines with QEMU/KVM on a debian based desktop distro Pop!_OS and performing gpu passthrough. https://github.com/bryansteiner/gpu-passthrough-tutorial This guide would likely have most of the important steps you would have to follow to get it working in Ubuntu Desktop.
If you can purcahse another Asus Hyper M.2 Card and have another 16x slot available, you can move the 2x M.2 cards that share the PCIE lanes with the U.2 connectors. (Also, I haven’t tested, but it may be possible that in SATA mode the U.2 connectors do not use the PCIE lanes)
Thank you again @DotBowder !! I’ve filed my info and your details to Asus and I hope they get out a fix. This should be a really compelling case to them because between us there are I think 8 distinct SATA drives tested as well as two different Linux installs. IDK why Windows might work but they can’t just say this is a system-specific issue.
And thanks for the info on PVE. I’ve never heard of it before now! I’ve used QEMU and mostly Docker, but never something like this. I mainly want to try Photoshop and some other Windows stuff and it’s great to know this stuff will work on hardware comparable to what I have.
Finally for the disks, I already have 2 Hyper M.2 cards and a total of 10 NVME devices, and the mobo manual claims you can only do 12 nvme devices (tho I’m not sure if that’s a limit of 12 for “hardware RAID” thru the BIOS settings or 12 no matter what). Another problem is I’m really tight on physical space due to GPUs and other stuff in the case.
One last tidbit: I’ve found that these QNAP dual 2.5 inch adapters appear to work with our Asus mobo (modulo Asus SATA errors): https://www.amazon.com/QNAP-Dual-SATA-Adapter-Converter/dp/B07RLM8W68 The adapter requires the mobo support SATA bifurcation… something my Mac won’t do but the Asus board does. They’re really handy cuz you can double your 2.5 inch drive density (e.g Samsung QVO SSDs) and gain some IOPS even tho the SATA 6g is shared.
@DotBowder Well I’ve gone through a couple loops with Asus support and they want me to RMA the board 
As far as I can tell, their lab did not actually do anything like, say, just try to use linux with a SATA disk 
and they are trying to just cover their butts and say it’s a one-off hardware problem. I can’t RMA the board because I use it for work… I’ll report here if Asus can find a way to be more productive. In the meantime, I guess your SATA adapter is the most viable fix.
Oh my, how weird things have gotten today. @oarphoarph
So, I was fiddling around with my server and managed to make my way to the proxmox forum. They advertised the 5.15 kernel is available for those who wish to test it.
Thinking, “I can always revert back to an older kernel” I went ahead and installed the 5.15 kernel. As you can see from my prior posts, my kernel version was 5.13.19-3.
Well, after updating to kernel 5.15, bam, SATA errors appeared on my U.2 connected drives! For kicks and giggles, I swapped all of the drives back to the SATA ports instead of the U.2 ports. Lo and behold, there are no SATA errors.
I did not touch the hardware when I upgraded from kernel 5.13 to 5.15. So, I have high confidence that there is some kind of interaction with the controller, but I haven’t found anything in the kernel patch notes that points to a fix. So 
I have no idea what the difference is.
I would suggest you see what kernel version your Ubuntu installation is on, and see if you can switch to a different kernel and see if there are any differences. As of right now, I’m on 5.15 (not yet supported by proxmox) and my SATA ports work. (And seemingly my U.2 ports as SATA ports do not work.)
@DotBowder Thanks for the info! Well I’m glad your SATA drives work!
I also have interesting news… I had to re-wire my machine a bit and I noticed that the Asus board actually has two different SATA controllers. Previously I had all my SATA drives plugged into the ports farthest from the CPU (the ones with E in the names in the pic). Do you know if these are the ports you were using? These apparently belong to a secondary ASmedia controller according to the manual. This time, I plugged the cables into the ports closest to the CPU, which the manual says are part of the AMD controller. Thus far I at least am able to copy about 2TB of data, which I have not been able to do before. Will keep testing tho.
Also, Asus sent me an unclear message that suggested to me there might be some conflict between Hyper M.2 cards / NMVE and the SATA stuff. Maybe the secondary ASMedia SATA controller is junked if you use lots of NVME. I think we are both using Hyper M.2 / lots of NVME ?
I know I’m necroing a super old thread - but did you get your sata issues resolved?
I had a proxmox machine die, I was about to move my virtualized truenas server over, but saw some of the warnings you guys mentioned.