Epyc on Asrock ROMED6U-2L2T - Proxmox Build

jtredux · July 14, 2021, 4:17pm

Interesting build - what made you choose the ROMED6 rather than the 8-DIMM D8 variant?

Also which slim SAS cables and Slim-SAS <–> PCIe slot adaptors are you using? I presume you’ve mounted them more permanently than in that picture!

MadMatt · July 17, 2021, 7:46pm

Hi, the main reasons were:
1 - I already had the micro ATX cases
2 - The Romed6 has two more network interfaces, I thought they would come in handy for passtrough
3- I thought 3x slim PCI ports would give me more flexibility than two additional 16x slots
4- I am not planning on running a CPU that would effectively make use of the additional 2x dimms, if my understanding of the mapping between chillers and access to memory is correct

The slim cables:
https://www.newegg.com/p/2VX-05TA-00002?item=9SIAEGBCYP0889

The PCI extenders:
https://www.newegg.com/p/2BN-00BX-000R7?item=9SIAEGBD4N6295

They are still mounted in that precarious position, I am still unsure whether to put everything into a single core v21 case or to keep the two stacked ones. Thermals are not bad as it is, waiting for temps to go down to see how much the fans kick in when not in the warmest part of the year…

MadMatt · August 13, 2021, 9:19am

A small update,

Everything working, performance more than adequate for my workloads.
The only issues I have had so far were:

The OSX VM bootloader would crash unless a monitor signal was available at boot time. I think the boot loader cannot handle negotiatig a resolution with my particular monitor (a Philips BDM4350) when said monitor is in power saving mode, and can’t handle the monitor being connected but completely turned off. I started looking into modifying the OVMF bios to assume a resolution of 3840x2160, but I wasn’t sure the effort would be wort it, so I cheated and bought a dummy DP plug, so that at boot time at least one display port is connected and active. That fixed it and avoided me having to be phisically in front of the pc to turn on the monitor at the appropriate time when booting the OSX VM
The OSX VM causing Latency spikes on the Windows VM when gaming/streaming audio: I have solved this by shuffling around the PCI cards in order for them to be in separate physical groups/slots, if that makes sense. Originally the OSX NVME was in the same Root Complex as the Windows GPU
The Windows GPU (XFX 5700XT 8GB) was mostly working, but if it crashed for any reason while gaming, it could not be reset other than with a full power down of the Proxmox host (not even a reset would suffice). This was really annoying, I tried multiple possible workarounds/fixes, but then just gave up and exchanged it with a 3070ti. The behaviour is the same (some games still crash, sometimes) but a shutdown and restart of the VM fixes it, the only quirk I still have is that when this happens the reboot takes anywhere from 1 to 2 minutes for the bios screen to show up, and I have to power off and on any usb device attached to the passed through USB pci card
The 3070ti properly supports PCIe Active State Power Management, the 5700XT could not, and enabling it would cause endless pci bus resets: this means that when the WIndows GPU is not in use it can be actually set to a lower power state, saving some 20-30W when idling (finger in the air measurement)

This is my current layout for the pci bus, In the end I had to make use of all three slim line pci extension boards to be able to move stuff in a way there were no sharing between passed through pci cards and different VMs

root@pve:~# lspci -tv
-+-[0000:c0]-+-00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex
 |           +-00.2  Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU
 |           +-01.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-01.1-[c1]--+-00.0  NVIDIA Corporation Device 2482
 |           |            \-00.1  NVIDIA Corporation Device 228b
 |           +-02.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-03.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-03.2-[c2]----00.0  Micron/Crucial Technology Device 540a
 |           +-04.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-05.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-07.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-07.1-[c3]--+-00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
 |           |            \-00.2  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA
 |           +-08.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           \-08.1-[c4]--+-00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
 |                        \-00.2  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA
 +-[0000:80]-+-00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex
 |           +-00.2  Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU
 |           +-01.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-01.1-[81]----00.0  Fresco Logic FL1100 USB 3.0 Host Controller
 |           +-01.3-[82-83]--+-00.0  Intel Corporation Ethernet Controller X710 for 10GBASE-T
 |           |               \-00.1  Intel Corporation Ethernet Controller X710 for 10GBASE-T
 |           +-01.4-[84]----00.0  Intel Corporation I210 Gigabit Network Connection
 |           +-01.5-[85]----00.0  Intel Corporation I210 Gigabit Network Connection
 |           +-01.6-[86-87]----00.0-[87]----00.0  ASPEED Technology, Inc. ASPEED Graphics Family
 |           +-02.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-03.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-04.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-05.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-07.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-07.1-[88]--+-00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
 |           |            \-00.2  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA
 |           +-08.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-08.1-[89]--+-00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
 |           |            \-00.2  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA
 |           +-08.2-[8a]----00.0  Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode]
 |           \-08.3-[8b]----00.0  Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode]
 +-[0000:40]-+-00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex
 |           +-00.2  Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU
 |           +-01.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-01.2-[41]----00.0  Fresco Logic FL1100 USB 3.0 Host Controller
 |           +-02.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-03.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-03.1-[42]--+-00.0  Advanced Micro Devices, Inc. [AMD/ATI] Baffin [Polaris11]
 |           |            \-00.1  Advanced Micro Devices, Inc. [AMD/ATI] Baffin HDMI/DP Audio [Radeon RX 550 640SP / RX 560/560X]
 |           +-04.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-05.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-07.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-07.1-[43]--+-00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
 |           |            \-00.2  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA
 |           +-08.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-08.1-[44]--+-00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
 |           |            +-00.1  Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP
 |           |            +-00.2  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA
 |           |            \-00.3  Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller
 |           +-08.2-[45]----00.0  Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode]
 |           \-08.3-[46]----00.0  Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode]
 \-[0000:00]-+-00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex
             +-00.2  Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU
             +-01.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
             +-01.1-[01]----00.0  Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981
             +-01.2-[02]----00.0  Phison Electronics Corporation E12 NVMe Controller
             +-02.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
             +-03.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
             +-03.2-[03]----00.0  Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961
             +-04.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
             +-05.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
             +-07.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
             +-07.1-[04]--+-00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
             |            \-00.2  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA
             +-08.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
             +-08.1-[05]--+-00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
             |            +-00.2  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA
             |            \-00.3  Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller
             +-14.0  Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller
             +-14.3  Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge
             +-18.0  Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 0
             +-18.1  Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 1
             +-18.2  Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 2
             +-18.3  Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 3
             +-18.4  Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 4
             +-18.5  Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 5
             +-18.6  Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 6
             \-18.7  Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 7

I have also played around a little with airflow, mostly because of my case not really being suited for this type of motherboard.
Room temperature today is on the warm side (26C), and these are ths temps registered with my main OSX VM running :

21 | MB_A Temp      | Temperature       | 35.00      | C     | 'OK'
22 | MB_B Temp      | Temperature       | 37.00      | C     | 'OK'
23 | Card Side Temp | Temperature       | 44.00      | C     | 'OK'
24 | X710 Temp      | Temperature       | 42.00      | C     | 'OK'
25 | CPU Temp       | Temperature       | 49.00      | C     | 'OK'
27 | TR1 Temp       | Temperature       | N/A        | C     | N/A
28 | TEMP M_2_1     | Temperature       | N/A        | C     | N/A
29 | TEMP M_2_2     | Temperature       | N/A        | C     | N/A
30 | DDR4_A Temp    | Temperature       | 42.00      | C     | 'OK'
31 | DDR4_C Temp    | Temperature       | 43.00      | C     | 'OK'
32 | DDR4_D Temp    | Temperature       | 44.00      | C     | 'OK'
33 | DDR4_E Temp    | Temperature       | 44.00      | C     | 'OK'
34 | DDR4_G Temp    | Temperature       | 45.00      | C     | 'OK'
35 | DDR4_H Temp    | Temperature       | 43.00      | C     | 'OK'

I am still shielding VM processes from OS processes, and have settled for the following config:
Cores 0-19 and 24-44 are shielded from OS processes by running the following at reboot:

# m h  dom mon dow   command
@reboot /usr/local/bin/shield-set.sh

/usr/local/bin/shield-set.sh

#!/bin/bash
cset shield --kthread on  --cpu 0-19,24-44
cset proc --move --fromset=root --toset=system --threads --kthread --force
r

This is my current kernel command line used when booting:

GRUB_CMDLINE_LINUX_DEFAULT="video=vesafb:off video=efifb:off amd_iommu=on iommu=pt default_hugepagesz=1G hugepagesz=1G hugepages=64 spec_store_bypass_disable=off nopti mitigations=off rd.driver.pre=vfio_pci apparmor=1 security=apparmor udev.log_priority=3 nohz_full=0,20,24-44 rcu_nocbs=0,20,24-44"

amd_iommu=on iommu=pt : enables AMD iommu groups
default_hugepagesz=1G hugepagesz=1G hugepages=64: allocates 64GB out of 128GB of ram to be used by huge pages, this boosts performance of latency sensitive VMs (I run audio interfaces on both the OSX and WIndows VM)
spec_store_bypass_disable=off nopti mitigations=off: diables mitigations for the cpu vulnerabilities, in exchange of some more perfromance
rd.driver.pre=vfio_pci: this should cause the vfio kernel module to.be loaded early in the boot process, improving stability/reliability in handling GPU passthrough
nohz_full=0,19,24-44 rcu_nocbs=0,19,24-44": complements the settings in the shielding script, reserves these cores from normal OS operating processes

I am still working on improving the streamdeck handling of monitoring system states and starting/stopping VMs, so far no version of the code is deemed from the to be in a publishing state

lae · August 31, 2021, 2:16am

I think there’s a typo in your kernel parameters? 0,20 instead of 0-19.

I’ve been kinda curious about doing a similar build myself but it’ll probably have to wait until I get a job again.

MadMatt · August 31, 2021, 12:24pm

Thanks mate, there indeed was a typo, I am shielding cores 0-19, 24-43 (40 cores total shielded, 8 cores total unshielded)

It’s definitely not been a cheap build, mainly due to components shortage. Here in Italy Threadripper parts were even less available, and I liked the romed6u pci express expansion capabilities

Now off with the next project: bringing the network to 10gbit and running some VMs off of Freenas

MadMatt · December 17, 2021, 4:38pm

Update …
Exchanged the 5700xt with a 3070ti, added two 4tb Samsung qlc SSD in raid1/zfs, exchanged one zoom audio interface with a Roland ua-25 and today I also swapped the Frankenstein cases with a proper one, had some major grounding issues with the old one (at least that’s what I told the wife… not sure if the issue is because of the case but…) And finally found a safe/stable position for the slimsas extensions

In hindsight, could have gone with a new case from the beginning and a full ATX Mobo, but that ship has sailed with the associated budget…

[Processing: IMG_20211217_155626.jpg…](And

oegat · May 2, 2022, 11:29am

This is a super interesting build!

I’m interested in your SlimSAS->USB solution, as I am planning to try the new 8dimm-version of the same(ish) motherboard, and I expect to be short on exactly one USB-controller for passthrough. The three x16 slots will all be accounted for by other stuff - the drawback of going small.

What I need is two (or more) IOMMU-separable USB-controllers, so that one can be given to a VM and the other can stay in the host. Just to be sure that the problem is what I think: can you confirm that you could not find two IOMMU-separable USB ports on the ROMED6U motherboard, that also map to different ports (i.e. so that one becomes available for passthrough)?

Because I need only two. It is a bit confusing, as there seems to be two IOMMU-separable controllers on the EPYC IO die (your 05:00.3 and 44:00.3 if I read your lspci -tv right). I have the corresponding two controllers on my Supermicro H12SSL-base EPYC system, and they end up in different IOMMU groups, but both seem to map to the same couple of physical ports.

(H12SSL also has an onboard ASMedia controller that can be used for passthrough, but there is afaik nothing like that on the ROMED*U boards).

MadMatt · May 2, 2022, 12:17pm

Hi,
I can confirm the two controllers on the ROMEd6 board are separate, both in the PCI hierarchy and in different IOMMU groups, without any special patching/config.
The reason I went with the PCI USB cards over the SLimSas extension is that the ASMEdia are not seen by OSX and have worse latency for audio in Windows …
I also tried a Sonnet Allegro Pro USB 3.1 PCIe Card, that has four USB controllers on board (and costs a fortune) but I couldn’t even have it working on the host, far less in the VMs when passed through … looks like a firmware (of the card) issue but I just went with the dedicated cards as they work.
In theory, if you can find a board that supports it, you could use one slimsas connection to two PciEx4 slots, and would hence need only one cable but at the time the ones I have used were the only boards I could find that accept a slimsa input …

root@pve:~# lspci -tv
-+-[0000:c0]-+-00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex
 +-[0000:80]-+-00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex
 |           +-00.2  Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU
 |           +-01.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-01.1-[81]----00.0  Fresco Logic FL1100 USB 3.0 Host Controller
 +-[0000:40]-+-00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex
 |           +-00.2  Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU
 |           +-01.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-01.2-[41]----00.0  Fresco Logic FL1100 USB 3.0 Host Controller
 |           |            \-00.2  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA
 |           +-08.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-08.1-[44]--+-00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
 |           |            +-00.1  Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP
 |           |            +-00.2  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA
 |           |            \-00.3  Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller
 \-[0000:00]-+-00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex
             +-08.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
             +-08.1-[04]--+-00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
             |            +-00.2  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA
             |            \-00.3  Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller

Iommu list:

root@pve:~# ./iommulist.sh
IOMMU Group 0 c0:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 10 c0:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 11 c1:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA104 [GeForce RTX 3070 Ti] [10de:2482] (rev a1)
IOMMU Group 11 c1:00.1 Audio device [0403]: NVIDIA Corporation GA104 High Definition Audio Controller [10de:228b] (rev a1)
IOMMU Group 12 c3:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
IOMMU Group 13 c3:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA [1022:1498]
IOMMU Group 14 c4:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
IOMMU Group 15 c4:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA [1022:1498]
IOMMU Group 16 80:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 17 80:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 18 80:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 19 80:01.4 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 1 c0:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 20 80:01.5 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 21 80:01.6 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 22 80:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 23 80:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 24 80:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 25 80:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 26 80:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 27 80:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 28 80:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 29 80:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 2 c0:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 30 80:08.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 31 80:08.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 32 81:00.0 USB controller [0c03]: Fresco Logic FL1100 USB 3.0 Host Controller [1b73:1100] (rev 10)
IOMMU Group 33 82:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller X710 for 10GBASE-T [8086:15ff] (rev 02)
IOMMU Group 34 82:00.1 Ethernet controller [0200]: Intel Corporation Ethernet Controller X710 for 10GBASE-T [8086:15ff] (rev 02)
IOMMU Group 35 84:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)
IOMMU Group 36 85:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)
IOMMU Group 37 86:00.0 PCI bridge [0604]: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge [1a03:1150] (rev 04)
IOMMU Group 37 87:00.0 VGA compatible controller [0300]: ASPEED Technology, Inc. ASPEED Graphics Family [1a03:2000] (rev 41)
IOMMU Group 38 88:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
IOMMU Group 39 88:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA [1022:1498]
IOMMU Group 3 c0:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 40 89:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
IOMMU Group 41 89:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA [1022:1498]
IOMMU Group 42 8a:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 43 8b:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 44 40:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 45 40:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 46 40:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 47 40:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 48 40:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 49 40:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 4 c0:03.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 50 40:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 51 40:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 52 40:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 53 40:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 54 40:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 55 40:08.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 56 40:08.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 57 41:00.0 USB controller [0c03]: Fresco Logic FL1100 USB 3.0 Host Controller [1b73:1100] (rev 10)
IOMMU Group 58 42:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Baffin [Radeon Pro WX 4130/4150] [1002:67e8] (rev 80)
IOMMU Group 58 42:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Baffin HDMI/DP Audio [Radeon RX 550 640SP / RX 560/560X] [1002:aae0]
IOMMU Group 59 43:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
IOMMU Group 5 c0:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 60 43:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA [1022:1498]
IOMMU Group 61 44:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
IOMMU Group 62 44:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP [1022:1486]
IOMMU Group 63 44:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA [1022:1498]
IOMMU Group 64 44:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller [1022:148c]
IOMMU Group 65 45:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 66 46:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 67 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 68 00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 69 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 6 c0:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 70 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 71 00:03.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 72 00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 73 00:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 74 00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 75 00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 76 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 77 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 78 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61)
IOMMU Group 78 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 79 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 0 [1022:1490]
IOMMU Group 79 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 1 [1022:1491]
IOMMU Group 79 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 2 [1022:1492]
IOMMU Group 79 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 3 [1022:1493]
IOMMU Group 79 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 4 [1022:1494]
IOMMU Group 79 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 5 [1022:1495]
IOMMU Group 79 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 6 [1022:1496]
IOMMU Group 79 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 7 [1022:1497]
IOMMU Group 7 c0:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 80 01:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 [144d:a808]
IOMMU Group 81 02:00.0 Non-Volatile memory controller [0108]: Sandisk Corp WD Black NVMe SSD [15b7:5001]
IOMMU Group 82 03:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
IOMMU Group 83 03:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA [1022:1498]
IOMMU Group 84 04:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
IOMMU Group 85 04:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA [1022:1498]
IOMMU Group 86 04:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller [1022:148c]
IOMMU Group 8 c0:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 9 c0:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]

MadMatt · May 2, 2022, 12:19pm

The Asrock Board has two USB ports in the back, and two routed to the internal USB3 connector, for which you need a separate cable (I connected them to my case USB embedded ports)

oegat · May 2, 2022, 4:26pm

Thanks, the “Starship” USB controllers (I assume they are on-die) are separate on my SM board too. I can send them separately to VMs, but only one of them seems to be associated with a specific port. If I assign the second one to a Windows VM it gets detected, but none of the physical USB ports seems to belong to it. Perhaps the ports are managed by the first one, and the second one only gets to handle devices through the first somehow.

Now I’m confused, as I don’t see any ASMedia USB chip in your IOMMU list or in the block diagram for your mainboard. Are you referring to an add-in card?

My H12SSL has an ASMedia chip that controls 4 physical USB ports (2 back + 2 by header), in addition to the Starship controllers associated with the two remaining physical ports (at the back).

This is a difference between the ASrock board and my H12SSL, I have only two of the back ports associated with the on-die USB controllers. The remaining ports go with the ASMedia controller.

I guess my question then becomes: can you connect to the on-die (“Starship”) controllers independently, by simply plugging things in the different USB ports? I can’t, but maybe you can as you have more ports.

For reference, here is how it looks on H12SSL:

$ lspci -nn | grep -i usb
04:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller [1022:148c]
41:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller [1b21:1142]
42:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller [1b21:1142]
45:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller [1022:148c]

Block diagram

“Starship” controllers on IO die
External ASMedia USB controller chip

There is also the “CPU USB2” orange line, which seems to go between CPU and BMC and is not connected to any external port afaik.

lspci -tv

-+-[0000:c0]-+-00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex
 |           +-00.2  Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU
 |           +-01.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-01.1-[c1]--+-00.0  NVIDIA Corporation Device 24b0
 |           |            \-00.1  NVIDIA Corporation Device 228b
 |           +-02.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-03.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-04.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-05.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-07.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-07.1-[c2]--+-00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
 |           |            \-00.2  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA
 |           +-08.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           \-08.1-[c3]--+-00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
 |                        \-00.2  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA
 +-[0000:80]-+-00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex
 |           +-00.2  Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU
 |           +-01.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-01.1-[81]----00.0  Sandisk Corp Device 5011
 |           +-01.2-[82]----00.0  Sandisk Corp Device 5011
 |           +-01.3-[83]----00.0  Sandisk Corp Device 5011
 |           +-01.4-[84]----00.0  Sandisk Corp Device 5011
 |           +-02.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-03.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-04.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-05.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-07.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-07.1-[85]--+-00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
 |           |            \-00.2  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA
 |           +-08.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-08.1-[86]--+-00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
 |           |            \-00.2  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA
 |           +-08.2-[87]----00.0  Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode]
 |           \-08.3-[88]----00.0  Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode]
 +-[0000:40]-+-00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex
 |           +-00.2  Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU
 |           +-01.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-02.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-03.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-03.3-[41]----00.0  ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller
 |           +-03.5-[42]----00.0  ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller
 |           +-03.6-[43]--+-00.0  Broadcom Inc. and subsidiaries NetXtreme BCM5720 2-port Gigabit Ethernet PCIe
 |           |            \-00.1  Broadcom Inc. and subsidiaries NetXtreme BCM5720 2-port Gigabit Ethernet PCIe
 |           +-04.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-05.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-07.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-07.1-[44]--+-00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
 |           |            \-00.2  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA
 |           +-08.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
 |           +-08.1-[45]--+-00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
 |           |            +-00.1  Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP
 |           |            +-00.2  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA
 |           |            \-00.3  Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller
 |           +-08.2-[46]----00.0  Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode]
 |           \-08.3-[47]----00.0  Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode]
 \-[0000:00]-+-00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex
             +-00.2  Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU
             +-01.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
             +-01.1-[01]--+-00.0  Advanced Micro Devices, Inc. [AMD/ATI] Lexa PRO [Radeon 540/540X/550/550X / RX 540X/550/550X]
             |            \-00.1  Advanced Micro Devices, Inc. [AMD/ATI] Baffin HDMI/DP Audio [Radeon RX 550 640SP / RX 560/560X]
             +-02.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
             +-03.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
             +-03.4-[02]----00.0  Phison Electronics Corporation Device 5018
             +-04.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
             +-05.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
             +-07.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
             +-07.1-[03]--+-00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
             |            \-00.2  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA
             +-08.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
             +-08.1-[04]--+-00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
             |            +-00.2  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA
             |            \-00.3  Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller
             +-14.0  Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller
             +-14.3  Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge
             +-18.0  Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 0
             +-18.1  Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 1
             +-18.2  Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 2
             +-18.3  Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 3
             +-18.4  Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 4
             +-18.5  Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 5
             +-18.6  Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 6
             \-18.7  Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 7

MadMatt · May 2, 2022, 5:22pm

s/Asmedia/Starship/

sorry
not enough coffee today to completely engage the brain

oegat · May 2, 2022, 6:11pm

Ok I’m with you, thanks!

And of course I’m not expecting you to test things for me, I’m merely curious whether you already tried and/or succeeded to use the two starship usb controllers with different physical ports.

MadMatt · May 2, 2022, 8:00pm

I can pass through either one of them to a VM, but I just realized the internal one is not connected to my case after all, I moved these connection to one of the FL1100 internal ports to have easy access to them in OSX … so I am not able to make sure they work as expected.
I can successfully boot two VMs with a Starship device passed through, proxmox correctly unbinds the xhci_hcd module and binds vfio. When the VMs are shut down I can re-bind the controllers to the proper driver and they will revert control to the host …

oegat · May 3, 2022, 3:55pm

I see! I could do all these things too, I just couldn’t get the second (higher PCI address) Starship controller to get through to any of my physical USB ports. Which made me suspect it is not independent from the other on my SM mobo, at least not with my 7252 CPU (which is admittedly a down-sized ROME part, though I don’t think the IO dice differ between the chips).

One possibility is that both controllers are functional, but only one controls the physical ports, but can hand off connected devices to the other controller if needed. Just speculation.

Anyway, I’m likely to get the ASRock mobo in a week or so, then I will soon test whether both starships have associated ports. If they don’t, and I stick to my plan to make this system my passthrough-workstation, then I’d try something similar to your cable solution.

Another question - when you replaced the TT cases, did you look for a new mATX case first, without finding a good alternative for your combination of hardware? Or did you decide from start to go with ATX?

MadMatt · May 3, 2022, 5:04pm

Not really, the epyc processor generates a lot of heat, add in a 3070 and an another amd gpu, plus the X710 chipsets and I felt the need to go with a bigger case with more space in it.
If I had to do it again, and not considering the pci slot options, I’d go with a full ATX solution …

oegat · May 4, 2022, 5:03pm

I see your point. I’m sort of considering the opposite direction, I have a ATX-based EPYC system, but considering to move it to mATX while at the same time going from a 120w to a 200w CPU. Maybe it’s not a good idea at all. The rest of my power budget is slightly lower than yours I think - 1 A4000 (140w) and 1 amd gpro 4300 (50w) + some nvme:s. I would at least need to select my casing carefully.

If I end up needing a cable-attached USB controller, I’m considering a solution based on something like these: Delock Products 62725 Delock 2.5″ Converter U.2 SFF-8639 > M.2 NVMe Key M - 7 mm + Delock Products 62843 Delock Converter M.2 Key B+M male > 2 x USB 3.0 Pin Header + SFF-8654->SFF8639 cable. Then at least I won’t need another card slot.

Droz · October 1, 2022, 11:33pm

Hi @MadMatt

Really curious as to how you got the GPU’s to pas through to a VM. I have the same board with Epyc 7232P CPU.

I have been having a hell of a time to get GPU and built in SATA controllers to pass through in ESXi and Proxmox.

After messing with things for 2 months and taking a break from it. I found this article an article on Reddit about “iommu not working with amd sata controllers” with the exact same CPU and board.

Seems like using the SATA controllers will not work currently. But since you have GPU pass through working. Was wondering if you have that documented somewhere?

Thanks

MadMatt · October 2, 2022, 4:33pm

Hi, no particular issues with the GPUs, followed the proxmox-vfio docs at:
https://pve.proxmox.com/wiki/Pci_passthrough
and had no issues with either AMD or Nvidia based cards
I am not passing through, and have never tried to, Sata controllers.
I used to pass through to my Osx VM and Windows VM dedicated NVME drives, off a 2 slot Nvme adapter, using bifurcation, but moved to passing through normal virtual disks so that I could manage/protect them with ZFS snapshots

If the reddit post you are talking about is this one:

then a fix has been posted, will need to recompile the kernel though …

Droz · October 4, 2022, 5:49am

OK thanks I’ll take a look at the proxmox docs

Yes that’s the one. I couldn’t post the link, new account on here.

I’m not new to linux, but have been out of the daily usage and working in command line for a decade or so. Doing a kernel recompile is not something I’m looking forward to. I may enlist some help from a friend local to me.

Thanks

Droz · October 5, 2022, 7:45am

OK so update for me…

This issue has been bugging me a lot. I’ve been planning this build since last year now and started to piece things together earlier this year.

I started poking around at IOMMU pass through videos on youtube. Since IOMMU is present on Ryzen chips as well. One video, the guy was talking about enabling AMD CBS in BIOS/UEFI to allow pass through.

I poked around in the Asrock board BIOS/UEFI and sure enough. There’s an option for it and it was disabled.

I killed off Win2k10 again, dumped ESXi on and I’m now able to pass though the SATA controllers

HAPPY DAY!

Going to be conducting testing the next few days for performance, etc.

Truenas scale on ESXi with two cores and 16GB ram seemed ok. It was passing traffic at full GigE speed (limited by my NAS its getting files from). CPU seemed VERY busy though and the mgmt dashboard wouldn’t load properly. Gave it four more cores and 6GB more of ram and its happier now.