ASRock Rack ROMED8U-2T first impressions
So for a week or so I’ve been running my finally acquired 7443p on the newish ROMED8U-2T board from ASRock Rack, where D8 apparently stands for 8 DIMM slots(channels) and U stands for micro-ATX. The board is a fairly recent addition from ASRock Rack, I did not know about it until about a month ago. Its nice at it is the first SP3 board to feature a full 8 channel DIMM configuration in a sub-ATX form factor.
I’m in the process of evaluating said combination of mainboard+CPU for becoming my daily driver work/play/labstation for the next 6-10 years. The alternative is that I go back to my tried-and-tested Supermicro H12SSL-I, and make the ASRock board a server together with my now-leftover 7252 CPU. I have not decided which combo to stick with, but here are my first impressions.
The mainboard
The ROMED8U-2T (D8U for short) is essentially a new version of ASRock Rack’s older ROMED6U-2L2T board (D6U), but with 2 DIMM slots added and a PCIe slot removed. There are some other IO differences, most notably that the D8U has only one M.2 slot (D6U has 2) and no 1GbE ports (D6U has 2). Both include two 10GbE ports from Intel X710-AT2. Here is a side by side comparison from product pictures:
A visible difference (in addition to what’s mentioned) is that the LAN heatsink is smaller on the newer D8U (left).
My evaluation setup
It turns out that the board can manage exactly the hardware I currently need for my main rig, and not much more (without using the SlimSAS connectors). Here is how it looks in my now ridiculously oversized inverted beQuiet Silent Base 802:
The little red light is a fan speed warning, I have not yet told IPMI to get used to my comparably low speed fans (and IPMI is right to complain as we shall see below). There is such a light beside every one of the 6 fan headers. Expansion cards do sit straight, the fisheye effect is due to the phone camera lens.
An obvious consequence of this layout is that I can make no space between PCIe cards for airflow, as I did with my H12SSL, and I am limited to one several-slot wide PCIe card. Currently I happen to have all single-slot cards, but that might not be the case for the machine’s entire lifetime.
Rest of system (for reference)
- 7443p
- Arctic Freezer 4U-SP3 (front fan missing due to DOA, RMA in progress, back fan works fine)
- 1x Gigabyte 7000s 1Tb NVMe boot drive (hidden underneath expansion cards)
- 1x SATA SSD on the back side (testing only, to be removed)
- Corsair AX850 PSU
- PCIe from top (as shown) to bottom:
- Nvidia A4000 (VM GPU & CUDA stuff)
- Sapphire GPRO 4300 (Linux host GPU)
- Asus Hyper M2 X16 Gen4 with 4x WD SN850 1Tb (as /home + VM backing)
It runs Linux (Kubuntu 20.04, 5.13.0-41-generic), the system was installed with the H12SSL board a year ago, then moved.
The mainboard (cont’d)
PCIe and SATA connections
In addition to the three x16 slots, there is a M.2 x4 slot, and two SlimSAS connectors that each can be configured as either x4x4 or x8.
There is also two regular SATA ports, 2 miniSAS HD connectors (4x SATA each). One cable is included. All route directly to the CPU, as expected on EPYC server boards.
USB connections
The board has no USB controller, it relies entirely on those on the IO die of the EPYC CPU.
# lspci | grep -i usb
09:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller
47:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller
As mentioned in a previous post, I had problem linking both these to physical ports on my H12SSL, however this was no problem on the D8U. They are arranged so that the lower-address one connects to the two backside USB3 ports, and the higher-address one connects to the internal header, as one would expect. Speaking of which:
ASRock likes cramping stuff onto their boards, D8U one is no difference. This position of this header implies limitations for some uATX cases.
The USB layout is relevant for my usage, as I typically have one Windows VM that needs a solid USB connection to keyboard, mouse, audio etc. I prefer to pass an USB controller through to the VM with VFIO. With this arrangement I would need this header, in order to get access to the second on-die USB controller.
BMC interface
I kind of liked the BMC interface, it has more options and seems more capable than the corresponding interface on my H12SSL. But it also has shortcomings, for example the “System Inventory” listing is always empty:
I believe @KeithMyers has a thread on another forum about this problem, with another ASRock board.
BIOS settings I
I have the latest BIOS according to dmesg
:
[ 0.000000] DMI: To Be Filled By O.E.M. To Be Filled By O.E.M./ROMED8U-2T, BIOS P3.20 08/17/2021
However, there is a discrepancy between the manual and the actual BIOS setup. The manual (pp. 63-64) claims that I should be able to turn on/off OPROM for each PCIe slot separately, under Boot->CSM. But I find no such option there, only a global one for all slots:
This has the drawback that I cannot select which GPU to boot first, which is mildly annoying. For now I had to put the A4000 in the first adressable slot (SLOT4) as I wanted to access the SATA port, so the A4000 is initialized first, but I want the GPRO4300 as my main display output. All this can be worked around, and I will be able to re-arrange the cards as the SATA connection is temporary, but it would have been more convenient to select the card to boot from in BIOS.
It took me a while to get the 7443p to run on full power. The key was to set the Determinism Slider
to power
in BIOS, which I learned from a previous post by @KeithMyers.
There was also a “Workload Tuning” screen:
I did not try changing these settings yet, but my guess is that these simply function as shorthands for different combinations of other settings, in accordance with the tables in the EPYC 7003 Workload Tuning Guide.
Thermals and sensors
The board has most of the sensors one would expect, with one exception: there is no reference to any VRM temperature sensor. Here are sensor readings at idle (ambient ~20c):
# ipmitool sensor
VOLT_3VSB | 3.340 | Volts | ok | 2.880 | 3.060 | na | na | 3.740 | 3.900
VOLT_5VSB | 5.010 | Volts | ok | 4.260 | 4.500 | na | na | 5.490 | 5.760
VOLT_CPU_VCORE | 0.730 | Volts | ok | 0.520 | 0.550 | na | na | 1.350 | 1.400
VOLT_CPU_SOC | 0.810 | Volts | ok | 0.590 | 0.630 | na | na | 1.210 | 1.260
VOLT_VCC_ABCD | 1.210 | Volts | ok | 1.020 | 1.080 | na | na | 1.320 | 1.380
VOLT_VCC_EFGH | 1.210 | Volts | ok | 1.020 | 1.080 | na | na | 1.320 | 1.380
VOLT_VPP_ABCD | 2.560 | Volts | ok | 2.120 | 2.240 | na | na | 2.760 | 2.880
VOLT_VPP_EFGH | 2.560 | Volts | ok | 2.120 | 2.240 | na | na | 2.760 | 2.880
VOLT_1.8VSB | 1.810 | Volts | ok | 1.530 | 1.620 | na | na | 1.980 | 2.070
VOLT_1.8V | 1.800 | Volts | ok | 1.530 | 1.620 | na | na | 1.980 | 2.070
VOLT_LAN_1.8V | 1.800 | Volts | ok | 1.530 | 1.620 | na | na | 1.980 | 2.070
VOLT_BAT | 3.040 | Volts | ok | 2.000 | 2.700 | na | na | 3.400 | 3.560
VOLT_3V | 3.340 | Volts | ok | 2.880 | 3.060 | na | na | 3.740 | 3.900
VOLT_5V | 5.070 | Volts | ok | 4.260 | 4.500 | na | na | 5.490 | 5.760
VOLT_12V | 12.200 | Volts | ok | 10.200 | 10.800 | na | na | 13.200 | 13.800
LAN_1V | 0.990 | Volts | ok | 0.850 | 0.900 | na | na | 1.100 | 1.150
VOLT_PSU1_VIN | na | Volts | na | na | na | na | na | na | na
VOLT_PSU2_VIN | na | Volts | na | na | na | na | na | na | na
CUR_PSU1_IOUT | na | Amps | na | na | na | na | na | na | na
CUR_PSU2_IOUT | na | Amps | na | na | na | na | na | na | na
Temp_MB_A | 33.000 | degrees C | ok | na | na | na | 68.000 | na | na
Temp_MB_B | 36.000 | degrees C | ok | na | na | na | 55.000 | na | na
Temp_Card Side_A | 40.000 | degrees C | ok | na | na | na | 93.000 | 94.000 | na
Temp_Card Side_B | 37.000 | degrees C | ok | na | na | na | 93.000 | 94.000 | na
Temp_CPU | 35.000 | degrees C | ok | na | na | na | 93.000 | 94.000 | na
Temp_X710 | 39.000 | degrees C | ok | na | na | na | 93.000 | 94.000 | na
TEMP_DDR4_A1 | 40.000 | degrees C | ok | na | na | na | 84.000 | na | na
TEMP_DDR4_B1 | 42.000 | degrees C | ok | na | na | na | 84.000 | na | na
TEMP_DDR4_C1 | 42.000 | degrees C | ok | na | na | na | 84.000 | na | na
TEMP_DDR4_D1 | 43.000 | degrees C | ok | na | na | na | 84.000 | na | na
TEMP_DDR4_E1 | 37.000 | degrees C | ok | na | na | na | 84.000 | na | na
TEMP_DDR4_F1 | 37.000 | degrees C | ok | na | na | na | 84.000 | na | na
TEMP_DDR4_G1 | 37.000 | degrees C | ok | na | na | na | 84.000 | na | na
TEMP_DDR4_H1 | 36.000 | degrees C | ok | na | na | na | 84.000 | na | na
TEMP M_2_1 | na | degrees C | na | na | na | na | na | na | na
TEMP_PSU1 | na | degrees C | na | na | na | na | na | na | na
TEMP_PSU2 | na | degrees C | na | na | na | na | na | na | na
FAN1_1 | na | RPM | na | na | na | 500.000 | na | na | na
FAN2_1 | 900.000 | RPM | ok | na | na | 500.000 | na | na | na
FAN3_1 | 600.000 | RPM | ok | na | na | 500.000 | na | na | na
FAN4 | 600.000 | RPM | ok | na | na | 500.000 | na | na | na
FAN5 | na | RPM | na | na | na | 500.000 | na | na | na
FAN6 | 300.000 | RPM | nc | na | na | 500.000 | na | na | na
PWR_PSU1_PIN | na | Watts | na | na | na | na | na | na | na
PWR_PSU2_PIN | na | Watts | na | na | na | na | na | na | na
PWR_PSU1_POUT | na | Watts | na | na | na | na | na | na | na
PWR_PSU2_POUT | na | Watts | na | na | na | na | na | na | na
PWR_CPU | 53.000 | Watts | ok | na | na | na | na | na | na
CPU_THERMTRIP | 0x0 | discrete | 0x0080| na | na | na | na | na | na
STS_PSU1_Status | 0x0 | discrete | 0x0080| na | na | na | na | na | na
STS_PSU2_Status | 0x0 | discrete | 0x0080| na | na | na | na | na | na
STS_PSU1_AC_LOST | na | discrete | na | na | na | na | na | na | na
STS_PSU2_AC_LOST | na | discrete | na | na | na | na | na | na | na
ChassisIntr | 0x0 | discrete | 0x0080| na | na | na | na | na | na
CPU_PROCHOT | 0x0 | discrete | 0x0080| na | na | na | na | na | na
FAN1_2 | na | RPM | na | na | na | 500.000 | na | na | na
FAN2_2 | na | RPM | na | na | na | 500.000 | na | na | na
FAN3_2 | na | RPM | na | na | na | 500.000 | na | na | na
STS_CPU_MCE | 0x0 | discrete | 0x0080| na | na | na | na | na | na
Temp_TR1 | na | degrees C | na | na | na | na | 65.000 | na | na
The lack of VRM sensor quite relevant for my use case, as we shall see. Temps at full load (stress -c 48
= all threads looping on sqrt(something)
):
# ipmitool sensor | grep -i temp
Temp_MB_A | 32.000 | degrees C | ok | na | na | na | 68.000 | na | na
Temp_MB_B | 40.000 | degrees C | ok | na | na | na | 55.000 | na | na
Temp_Card Side_A | 38.000 | degrees C | ok | na | na | na | 93.000 | 94.000 | na
Temp_Card Side_B | 37.000 | degrees C | ok | na | na | na | 93.000 | 94.000 | na
Temp_CPU | 66.000 | degrees C | ok | na | na | na | 93.000 | 94.000 | na
Temp_X710 | 41.000 | degrees C | ok | na | na | na | 93.000 | 94.000 | na
TEMP_DDR4_A1 | 44.000 | degrees C | ok | na | na | na | 84.000 | na | na
TEMP_DDR4_B1 | 44.000 | degrees C | ok | na | na | na | 84.000 | na | na
TEMP_DDR4_C1 | 44.000 | degrees C | ok | na | na | na | 84.000 | na | na
TEMP_DDR4_D1 | 44.000 | degrees C | ok | na | na | na | 84.000 | na | na
TEMP_DDR4_E1 | 41.000 | degrees C | ok | na | na | na | 84.000 | na | na
TEMP_DDR4_F1 | 40.000 | degrees C | ok | na | na | na | 84.000 | na | na
TEMP_DDR4_G1 | 40.000 | degrees C | ok | na | na | na | 84.000 | na | na
TEMP_DDR4_H1 | 38.000 | degrees C | ok | na | na | na | 84.000 | na | na
TEMP M_2_1 | na | degrees C | na | na | na | na | na | na | na
TEMP_PSU1 | na | degrees C | na | na | na | na | na | na | na
TEMP_PSU2 | na | degrees C | na | na | na | na | na | na | na
Temp_TR1 | na | degrees C | na | na | na | na | 65.000 | na | na
CPU is 66c, other sensors are lower, however my IR camera (inspired by @Nefastor) reveals what the onboard sensors don’t:
The 78.2c spot is near the VRM heatsink.
To be clear, I don’t consider the high temp on the VRMs to be ASRock Rack’s fault in any way.
It’s a server component formeost, so ASRock Rack engineers likely simply assume that the case will have a high-airflow layout and that system fan speed scales with CPU temp, and they are perfectly right to do so. The picture, if anything, makes clear that my beQuiet case won’t cut it for a CPU with this power (it would likely not cut it with my H12SSL board either - I only tested that one so far with a much weaker CPU).
Anyhow, for my kind of use case VRMs are a weak spot, and because of this it would have been great to be able to monitor the VRMs automatically. My Supermicro board has two TEMP_VRM*
(don’t remember the exact name) sensors.
This all said, the VRM cooling is actually less beefy on this board compared to the ATX counterparts from ASRock Rack and Supermicro. The heatsink is smaller, and I count fewer phases:
ROMED8U-2T (left), and ATX counterpart ROMED8-2T (right)
But this difference won’t make up for bad airflow, and I will look for another case either way
Bios Settings II: Overclocking!
I was by far most surprised by finding these settings in the BIOS setup:
(in Advanced->CPU Common Options->Performance)
I don’t know if any other SP3 board has anything like this?
CPU “overclock”
Then choosing “Custom Core Pstates” leads to a disclaimer followed by a setting to change the clock frequency of Pstate0. Apart from that, I could not find any other OC settings related to the CPU.
I did not try to change this (or even to save any BIOS settings after accepting the disclaimer to see the options). My guess is that the actual overclocking potential enabled by this single item is quite limited. NB that Pstate0 != Pstate0 Boost, I interpret this as what is changed is the Pstate0 non-boost setting, which is 2850MHz on the 7443p.
I haven’t tested and I wont experiment with a chip this expensive, but I suspect that this setting won’t enable any core to go above the max boost limit (4.0GHz for the 7443p).
Memory overclock and timings
There was a similar disclaimer screen under AMD CBS->DDR4 Common Options->DRAM Timing Configuration, then followed by this menu:
(cont’d)
This screen looks indeed more useful!
Examples:
- Memory Clock Speed allows settings in the range between 400-1800MHz (i.e. max. DDR4-3600).
- Tcl (CAS Latency) can be set using hex values up to 0x21.
I did not check the rest in detail. However I like the idea with overclockable ECC RAM, especially the ability to manipulate timings.
Summary
Those were my impressions and overview. It is clearly not “fair” as a review as I judge the board based on my own kind of non-standard use case (although the first page of the manual says both “Workstation” and “Server”). But there were things I found interesting and different from my H12SSL board, and since this is my only ASRock Rack board, I don’t know what is specific for this board or common with ASRR’s other SP3 offerings.
I really like several aspects of this board, mostly it being micro-ATX. Manipulating DDR4 timings is intriguing, and something I might try at some point. There were also aspects where I think my H12SSL board suits my workstation usage better, for example the ability to choose which PCIe slot to initialize video from, and the ability to monitor VRM temps.
The VRM thermal issues is not something I attribute to the board though, I believe it’s a consequence of the new 200w CPU and the case (beQuiet Silent Base 802). So I intend to replace the case for something more airflow-oriented.