ASUS Pro WS X570-ACE IOMMU Groups

Hi All…

I’m playing around with an ASUS Pro WS X570-ACE mainboard and thought maybe someone else is interested how the IOMMU groups look like:

IOMMU Group 0:
 	00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 1:
 	00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 2:
 	00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 3:
 	00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 4:
 	00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 5:
 	00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 6:
 	00:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 7:
 	00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
 	00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
 	0b:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
IOMMU Group 8:
 	00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
 	00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
 	00:08.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
 	00:08.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
 	0c:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
 	0c:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP [1022:1486]
 	0c:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
 	0c:00.4 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller [1022:1487]
 	0d:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
 	0e:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 9:
 	00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61)
 	00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 10:
 	00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 0 [1022:1440]
 	00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 1 [1022:1441]
 	00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 2 [1022:1442]
 	00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 3 [1022:1443]
 	00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 4 [1022:1444]
 	00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 5 [1022:1445]
 	00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 6 [1022:1446]
 	00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 7 [1022:1447]
IOMMU Group 11:
 	01:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57ad]
IOMMU Group 12:
 	02:03.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a3]
IOMMU Group 13:
 	02:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a3]
IOMMU Group 14:
 	02:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a4]
 	05:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
 	05:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
 	05:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU Group 15:
 	02:09.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a4]
 	06:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 16:
 	02:0a.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a4]
 	07:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 17:
 	03:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
IOMMU Group 18:
 	04:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. Device [10ec:816e] (rev 1a)
 	04:00.1 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 1a)
 	04:00.2 Serial controller [0700]: Realtek Semiconductor Co., Ltd. Device [10ec:816a] (rev 1a)
 	04:00.4 USB controller [0c03]: Realtek Semiconductor Co., Ltd. Device [10ec:816d] (rev 1a)
 	04:00.7 IPMI SMIC interface [0c07]: Realtek Semiconductor Co., Ltd. Device [10ec:816c] (rev 1a)
IOMMU Group 19:
 	08:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:14a0] (rev c1)
IOMMU Group 20:
 	09:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:14a1]
IOMMU Group 21:
 	0a:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Vega 20 [Radeon VII] [1002:66af] (rev c1)
IOMMU Group 22:
 	0a:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Vega 20 HDMI Audio [Radeon VII] [1002:ab20]

Cheers!

Gorgonbert

1 Like

Thank you! Would you mind checking, if & how well hardware monitoring (temp sensors, fan RPM, …) and fan control work? In theory the SuperIO chip should have great in-kernel support, but …

What are your system specs & cooling? How’s your boost, what about CPU, system, chipset temps, how bad’s the fan? Any gripes about the BIOS? Anything that doesn’t work or needed tweaks?

Have you played around with the OOB management? Does it even work in a Windows-only environment?

So many questions, I’m sorry. It’s just that I’m still on the fence between the Asus Ace and another Gigabyte Master …

I haven‘t spent a lot of time with it yet.

So far I have installed proxmox and planning to use it for homelab, nas and such. I also want to try to get a gpu passthrough windows vm going (saw wendell‘s videos on youtube).

I don‘t know much about linux (yet) so to answer some of your questions I guess I could run some commands/scripts if you give them to me. That way I can learn something too :wink:

I work in Enterprise IT but I haven’t built a complete PC in about 20 years. So far I haven‘t had any problems with the board. I’m very pleased.

You can run “sensors” in a terminal to see what’s recognised OOTB. Most probably you’ll need additional modules, run “sensors-detect” [as root / with sudo] and follow the prompts. It will spit out a list of driver modules and IIRC offer to load them for you. Now try “sensors” again, should have more output. The proper way to make sure these modules load on every boot depends on your distro.
Further reading: https://wiki.archlinux.org/index.php/Lm_sensors. The Arch Wiki is generally excellent, even if you don’t use Arch.

AFAICT the board should use the nct6775 driver (but confirmation would be nice). If so, and if all options are actually supported, that’d be excellent news.
Driver documentation showing what it should be able to do: https://www.kernel.org/doc/html/latest/hwmon/nct6775.html.

How loud is the chipset fan ? did someone try 128GB of ECC Ram in this board ?

All I can give you is my subjective impression: With a 3900X under a Noctua NH-D15 in a Fractal Design Define R5, 4x NF-A14 PWM, all on BIOS fan profile “silent”, Radeon VII, the system isn’t silent at idle, but it’s very very quiet. Under the desk it’s quiet enough I haven’t bothered to manually tweak the cpu & case fan curves, never mind the VII’s, yet. When I listen, it’s the VII’s >800 RPM fans are audible, but there’s nothing high-pitched that’d I’d associate with a small fan. On the table, case open and right next to me, I can hear it, but it isn’t unpleasant.
There’s no BIOS fan control for the chipset, but the other fans have separate hysteresis for raising & lowering speed, which works really well to deal with the CPU’s temperature spikes.

My Gigabyte X570 Aorus Master by comparison (3700X, otherwise ~identical) has a really annoying chipset fan. That one’s supposed to be semi-passive, but somehow it’s never below threshold.

It does, but inserting the module fails:
nct6775: Found NCT6798D or compatible chip at 0x2e:0x290
ACPI Warning: SystemIO range 0x0000000000000295-0x0000000000000296 conflicts with OpRegion 0x0000000000000290-0x0000000000000299 (\AMW0.SHWM) (20181213/utaddress-213)
ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver

The kernel parameter “acpi_enforce_resources=lax” would probably put paid to that, but I really don’t like the implications.

Thank god for L1Techs forums and considerate people who share important info… Considering this board myself.

If you have Proxmox running can you see if the USB controller in Group 14 passes through okay?

And what the heck is
IOMMU Group 18:
04:00.4 USB controller [0c03]: Realtek Semiconductor Co., Ltd. Device [10ec:816d] (rev 1a)

i have never seen a Realtek USB port before… can you confirm which port is actually connected to that port on the mobo please?

EDIT: https://images.anandtech.com/doci/14657/ASUS%20Pro%20WS%20X570-Ace%20Block%20Diagram.JPG seems to indicate its linked to the Flash and SPI ?

Basically wondering how many passable USB controllers are onboard in case I wanted each VM to have its own controller.

1 Like

Hate to necro this, but I’ve been looking at this 'board. Did you find out what you were asking?

1 Like

Since this is bumped & necroed already and I do have this very board I might as well jump on the train.

As of now (February 2020) there is still no PCH Fan Control in the BIOS. The PCH fan drove me absolutely insane. It’s an annoying, high pitched fan whine and you will definitely be able to hear it.
I removed the PCH heatsink, replaced the original thermal pad with regular thermal paste, put it back on, disconnected the fan and mounted a noctua 40mm on top. This sacrifices full length PCI-E card compatibility in the 2nd PCI-E (the top x8 one) slot.

sensors and fan control - except PCH fan - seem to work with linux 5.4.8 at least, but I let the BIOS do that job.

That realtek USB controller is part of that IPMI like feature which is completely useless on anything other than Windows. All devices that are related to this feature are in their own IOMMU group but they cannot be passed through to a VM - I got an error message, I don’t remember what it said exactly, but I didn’t bother.

All in all, I would definitely recommend this board for VFIO purposes, but you absolutely, positively need to replace the chipset fan and potentially sacrifice 1 full length GPU if you are sensitive to noise. IMHO this is fine, as this board as 3 x8 slots anyway. The fact that the bottom one is connected by only 4 lanes to the chipset doesn’t seem to be problematic, my RX 480 in the bottom slot suffers from virtually zero performance penalty.

1 Like

Well, I seem rather insensitive to fan noise… my Cooler Master HAF XB Evo case is using its original (cheap, old model) Cooler Master fans and they’re pretty noisy. A silent build is hard to do with any chipset fan, much less one that isn’t adjustable, I can see where that’s a problem. I would do Noctua on my case fans, but that costs money… :slight_smile:

By the way, is that x4 pcie3 or x4 pcie4? If the latter, it’s the same as x8 pcie3 and is going to not be a problem in any way with any GPU except slightly for a 2080ti/rtx Titan.

1 Like

The chipset is connected to the CPU using 4 PCI Express 4.0 lanes. Keep in mind that other devices, such as SSDs, Ethernet, USB etc. may communicate over the same bottleneck.

The best the bottom PCI Express slot can do is PCI Express 3.0 x8, but I suppose it can potentially be (slightly) worse depending on what you do. YMMV.

1 Like

Could a user with an ASUS Pro WS X570-Ace be so kind an try to boot (free) ESXi 6.7 (U3)?

I’m having PSOD issues when booting it (UEFI and CSM/Legacy), with the 6.7 U3 release image as well as with an installed version from a different computer that has the latest updated version on it.

The BIOS is the latest 1302 and as far as I know there should not be any catastrophic issues with ESXi and Ryzen 3000 except for the broken RDRAND with early AGESA versions.

@aBav.Normie-Pleb did you figure it out?

My Quadro P2200 came yesterday so I spent all day assembling my system and trying to get ESXi running. I’ve also got a WS X570-ACE paired with a Ryzen 3900x and 32GB of ECC memory. It’s replacing an older ThinkServer.

I struggled all day today trying to work around the PSOD. Finally, I came upon the idea to disable the Realtek ethernet port. Once that was done, I was finally able to boot up the ESXi install.

Unfortunately, I really do want to get both ethernet ports working, especially since the Realtek is the one that handles the remote access. Hopefully together we can figure that out.

But first, I need to figure out how to get my RAID card passed through the PCI passthrough.

@capnick helped me in my own thread mentioning the Realtek NIC being the poodle’s core.

As for resolving this, unfortunately I have no idea at this point.

I have a little hope that maybe the soon to be released vSphere/ESXi 7.0 is working better with AMD platforms.

The P2200 should work well in this board, tested Passthrough in every one of the three large PCIe slots with a P2000 and 3700X and 6.7 U3.

Nice: You can also use it for Passthrough if it’s the only GPU in the system, ESXi doesn’t need a GPU after it’s completely booted and remains controlable via the web interface.

Thanks for the info. I was able to complete my system migration. It’s been so long since I set the old machine up, that I forgot I had to enable passthrough on the card through ESXi first.

Now that I’m up and running, I’ll try passing the GPU through. And then hopefully we can figure something out about the Realtek NIC.

Does anybody know how to update the Realtek firmware on the Pro WS X570-ACE (ACCE_Firmware_Update_0111)?

I’ve installed a proper Realtek driver but I get an error message that says I am to install Realtek software prior to updating the firmware?

I have bought DDR4 128GB from supermicro store and it booted and runs without any issues at stock settings of 2666MHz out of the box.

You need to install a RealTek driver from the mobo support page first. The firmware updater will be able to start after that.

I am passing through Radeon RX580 right without any problem on ESXi 7.0 as well. The interesting part is that ESXi cannot use(no readily available drivers) Realtek NIC; HOWEVER, I can and do passthrough it to the Guest Windows 10 OS and it is recognized there normally.