Cables motherboard specific? So the “SlimSAS” connectors on our Supermicro boards, and the “Slimline SAS” connector on ASRock boards could be different interfaces?
Navigating these interfaces is so confusing…
It’s a complete shitshow is what it is.
If the cables are intended for generic boards, they probably are okay…
But there are two issues offhand I can think of:
- Even if the physical connector is the same, there can be different pinouts, which aren’t compatible. For example cables meant for connecting to or from broadcom stuff is terrible in regards to this. I don’t know enough about all the possible pinouts and who uses them to safely say all motherboards are one way or another.
- Signal integrity varies. A cable that works on one board may not be able to work on another because of minor signal strength differences. It’s less of an issue with PCIe 3.0, but with PCIe 4.0 the need for having redrivers on both ends of the cable becomes much more necessary. For short cables that are really meant for PCIe 4.0, you should probably be okay.
Ultimately you’ll just have to plug it and see if it works and that no transmission errors are being vomited all over your logs. I would personally be cautiously optimistic, but prepared for possible disappointment.
2 Likes
Heavily considering the 7443P + Tyan Tomcat HX S8030 board, now just need to find memory as i wont be able to use the current memory from my Thinkstation thinking of getting 8x Kingston KSM32RD4/32HDR (32gb 3200 ECC Reg) to not have any memory issues like i have had in the past using 128gb.
now need to find a cooling solution, was planning on lapping and putting a waterblock on the EPYC CPU + GPUs as it would make for a quiet and cool system i can sit on the floor next to me and get work done, as my Current p620 screams underheavy GPU workloads due to the NVIDIA A100 in there with 4x 90mm fans strapped to the thing, so when i bring the card over to the new machine would also waterblock that.
I have that Tyan board too - I couldn’t find any U.2 cables for it when I bought it a few months back, the situation may have changed though.
Yes, unfortunatley it appears that same connector != same pinout! I’m pretty sure my SM cables won’t work in my Tyan MB.
I don’t think U.2 is really meant as a ‘consumer facing’ product, if you look at dedicated U.2 backplanes in off-the-shelf servers, many of them seem to have very bespoke sets of PCIe cards and tailored length cables to hook 24 drives up to the front panel.
1 Like
@Log @jtredux Right. Physically compatible connections with different pinout should really be illegal somehow… 
I was rather hoping for a way to connect generic PCIe x4 or x8 devices via these cables. I was considering making my main EPYC rig micro-ATX, but then it seems I’m short on one USB controller.
So the ideal solution should be a USB controller directly connectable by ASRock’s “Slimline SAS” cable, or second, an x4 or x8 PCIe riser connectable by the same cable (to which I can then connect a USB card). PCIe gen3 or even gen2 should suffice, so signal integrity is perhaps less of an issue.
For now, it seems my best bet is something like (Slimline SAS->U.2 cable)->(U.2->M.2 enclosure)->(M.2-connected USB controller).
I’m pretty sure someone on here had something like this:
Though how the other end of the slimsas cable needs to be wired is beyond me.
1 Like
I completely forgot, but I have an 8i slimsas → 2x U.2 cable from supermicro that works at PCIe 4.0 speeds and connects to my Tyan S8030 board just fine. It’s the CBL-SAST-0953. It makes zero mention of PCIe 4.0, so I can’t guarantee anything but “it works for me”.
I believe wendell made a casual observation that silver colored cables (as opposed to blue) seem to work better with PCIe 4, and this one also happens to be silver. I had gone through a bunch of cables before this one finally did it.
Here’s a 400gb optane p5800x and my windows testing/benchmarking drive thrown together with absolutely zero optimations or considerations for NUMA or other stuff.
2 Likes
It isn’t necessary to lap an Epyc cpu. The IHS is large enough to provide plenty of thermal transfer area as long as the thermal cooling solution is using a full coverage coldplate.
1 Like
That would be me 
I am using two of these, connected to my ASRock romed6, using these:
And using two USB fl1000 cards into the extended slots
These are pcie3 cables though …
2 Likes
Wonderful, you have solved more or less my exact problem! It’s actually clear in your other thread (linking it for future backtracking), I just didn’t check back on it properly until now.
I’ll put my specific follow up questions in that thread
1 Like
Here’s a follow-up on this question. So the reason for my worry was the high weight of the Arctic Freezer 4U-SP3, 1288g, together with the fact that the SP3 socket has a smaller vertical distance (assuming a tower-case) between the cooler mounting screws compared to other sockets. This implies an unusually high torque applied to the mainboard PCB in the vicinity of the CPU backplate.
With help from my retailer (Nextron Norway) I reached out to Supermicro, ASRock (as I now own one SP3 mainboard from each), and Arctic. Here are the results.
Supermicro,much as I expected, said:
ASRock had more to say, perhaps as they don’t make their own coolers:
And today I finally got a reply from Arctic:
Overall I think it confirms the intuitions by @jtredux and myself in the discussion above, but at least ASRock’s response suggests some caution.
Anyhow, I mounted the cooler and I see no tendency of the PCB to bend in a static config. Also, I did come up with a way to mitigate the worry to some extent:
IKEA “PATRULL” door-stopper
In my current temporary setup it doesn’t flex much, possibly it relieves the equivalent of 100grams or so. I could put some more padding to increase the force, in case I make this config permanent.
(you might wonder where the front fan of the cooler went - as far as I could tell it was DOA, so I removed it - I will follow this up. Also the mainboard here is the newly arrived ASRock ROMED8U micro-ATX, I will return to it in a later post!)
4 Likes
Does anyone know the PPT (Package Power Tracking) values of the Epyc Milan chips?
Cannot find that information anywhere.
From what I know it is the maximum power that can be delivered to the socket.
For the Ryzen 5950X for example, the TDP is 105W and the PPT is 142W.
It depends on your motherboard/server’s VRM & cooling abilities, but 200-240W seems pretty common, 280W seems to be the high-end.
Depends on whether the cpu allows the user to set the max PPT and cTDP in the BIOS.
For example my 7402P has a default PPT of 180W. But is allowed to be set for 200W in the BIOS. Which is where I run mine at full 3.35Ghz all-core loading with the Power Determinism slider.
But as another example, my 7443P has a default PPT and cTDP of 200W and can only be lowered in the BIOS. Can be lowered to 150W if the user wants to lower the power used.
Depends also on the motherboard and BIOS features exposed.
ASRock Rack ROMED8U-2T first impressions
So for a week or so I’ve been running my finally acquired 7443p on the newish ROMED8U-2T board from ASRock Rack, where D8 apparently stands for 8 DIMM slots(channels) and U stands for micro-ATX. The board is a fairly recent addition from ASRock Rack, I did not know about it until about a month ago. Its nice at it is the first SP3 board to feature a full 8 channel DIMM configuration in a sub-ATX form factor.
I’m in the process of evaluating said combination of mainboard+CPU for becoming my daily driver work/play/labstation for the next 6-10 years. The alternative is that I go back to my tried-and-tested Supermicro H12SSL-I, and make the ASRock board a server together with my now-leftover 7252 CPU. I have not decided which combo to stick with, but here are my first impressions.
The mainboard
The ROMED8U-2T (D8U for short) is essentially a new version of ASRock Rack’s older ROMED6U-2L2T board (D6U), but with 2 DIMM slots added and a PCIe slot removed. There are some other IO differences, most notably that the D8U has only one M.2 slot (D6U has 2) and no 1GbE ports (D6U has 2). Both include two 10GbE ports from Intel X710-AT2. Here is a side by side comparison from product pictures:
A visible difference (in addition to what’s mentioned) is that the LAN heatsink is smaller on the newer D8U (left).
My evaluation setup
It turns out that the board can manage exactly the hardware I currently need for my main rig, and not much more (without using the SlimSAS connectors). Here is how it looks in my now ridiculously oversized inverted beQuiet Silent Base 802:
The little red light is a fan speed warning, I have not yet told IPMI to get used to my comparably low speed fans (and IPMI is right to complain as we shall see below). There is such a light beside every one of the 6 fan headers. Expansion cards do sit straight, the fisheye effect is due to the phone camera lens.
An obvious consequence of this layout is that I can make no space between PCIe cards for airflow, as I did with my H12SSL, and I am limited to one several-slot wide PCIe card. Currently I happen to have all single-slot cards, but that might not be the case for the machine’s entire lifetime.
Rest of system (for reference)
- 7443p
- Arctic Freezer 4U-SP3 (front fan missing due to DOA, RMA in progress, back fan works fine)
- 1x Gigabyte 7000s 1Tb NVMe boot drive (hidden underneath expansion cards)
- 1x SATA SSD on the back side (testing only, to be removed)
- Corsair AX850 PSU
- PCIe from top (as shown) to bottom:
- Nvidia A4000 (VM GPU & CUDA stuff)
- Sapphire GPRO 4300 (Linux host GPU)
- Asus Hyper M2 X16 Gen4 with 4x WD SN850 1Tb (as /home + VM backing)
It runs Linux (Kubuntu 20.04, 5.13.0-41-generic), the system was installed with the H12SSL board a year ago, then moved.
The mainboard (cont’d)
PCIe and SATA connections
In addition to the three x16 slots, there is a M.2 x4 slot, and two SlimSAS connectors that each can be configured as either x4x4 or x8.
There is also two regular SATA ports, 2 miniSAS HD connectors (4x SATA each). One cable is included. All route directly to the CPU, as expected on EPYC server boards.
USB connections
The board has no USB controller, it relies entirely on those on the IO die of the EPYC CPU.
# lspci | grep -i usb
09:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller
47:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller
As mentioned in a previous post, I had problem linking both these to physical ports on my H12SSL, however this was no problem on the D8U. They are arranged so that the lower-address one connects to the two backside USB3 ports, and the higher-address one connects to the internal header, as one would expect. Speaking of which:
ASRock likes cramping stuff onto their boards, D8U one is no difference. This position of this header implies limitations for some uATX cases.
The USB layout is relevant for my usage, as I typically have one Windows VM that needs a solid USB connection to keyboard, mouse, audio etc. I prefer to pass an USB controller through to the VM with VFIO. With this arrangement I would need this header, in order to get access to the second on-die USB controller.
BMC interface
I kind of liked the BMC interface, it has more options and seems more capable than the corresponding interface on my H12SSL. But it also has shortcomings, for example the “System Inventory” listing is always empty:
I believe @KeithMyers has a thread on another forum about this problem, with another ASRock board.
BIOS settings I
I have the latest BIOS according to dmesg:
[ 0.000000] DMI: To Be Filled By O.E.M. To Be Filled By O.E.M./ROMED8U-2T, BIOS P3.20 08/17/2021
However, there is a discrepancy between the manual and the actual BIOS setup. The manual (pp. 63-64) claims that I should be able to turn on/off OPROM for each PCIe slot separately, under Boot->CSM. But I find no such option there, only a global one for all slots:
This has the drawback that I cannot select which GPU to boot first, which is mildly annoying. For now I had to put the A4000 in the first adressable slot (SLOT4) as I wanted to access the SATA port, so the A4000 is initialized first, but I want the GPRO4300 as my main display output. All this can be worked around, and I will be able to re-arrange the cards as the SATA connection is temporary, but it would have been more convenient to select the card to boot from in BIOS.
It took me a while to get the 7443p to run on full power. The key was to set the Determinism Slider to power in BIOS, which I learned from a previous post by @KeithMyers.
There was also a “Workload Tuning” screen:
I did not try changing these settings yet, but my guess is that these simply function as shorthands for different combinations of other settings, in accordance with the tables in the EPYC 7003 Workload Tuning Guide.
Thermals and sensors
The board has most of the sensors one would expect, with one exception: there is no reference to any VRM temperature sensor. Here are sensor readings at idle (ambient ~20c):
# ipmitool sensor
VOLT_3VSB | 3.340 | Volts | ok | 2.880 | 3.060 | na | na | 3.740 | 3.900
VOLT_5VSB | 5.010 | Volts | ok | 4.260 | 4.500 | na | na | 5.490 | 5.760
VOLT_CPU_VCORE | 0.730 | Volts | ok | 0.520 | 0.550 | na | na | 1.350 | 1.400
VOLT_CPU_SOC | 0.810 | Volts | ok | 0.590 | 0.630 | na | na | 1.210 | 1.260
VOLT_VCC_ABCD | 1.210 | Volts | ok | 1.020 | 1.080 | na | na | 1.320 | 1.380
VOLT_VCC_EFGH | 1.210 | Volts | ok | 1.020 | 1.080 | na | na | 1.320 | 1.380
VOLT_VPP_ABCD | 2.560 | Volts | ok | 2.120 | 2.240 | na | na | 2.760 | 2.880
VOLT_VPP_EFGH | 2.560 | Volts | ok | 2.120 | 2.240 | na | na | 2.760 | 2.880
VOLT_1.8VSB | 1.810 | Volts | ok | 1.530 | 1.620 | na | na | 1.980 | 2.070
VOLT_1.8V | 1.800 | Volts | ok | 1.530 | 1.620 | na | na | 1.980 | 2.070
VOLT_LAN_1.8V | 1.800 | Volts | ok | 1.530 | 1.620 | na | na | 1.980 | 2.070
VOLT_BAT | 3.040 | Volts | ok | 2.000 | 2.700 | na | na | 3.400 | 3.560
VOLT_3V | 3.340 | Volts | ok | 2.880 | 3.060 | na | na | 3.740 | 3.900
VOLT_5V | 5.070 | Volts | ok | 4.260 | 4.500 | na | na | 5.490 | 5.760
VOLT_12V | 12.200 | Volts | ok | 10.200 | 10.800 | na | na | 13.200 | 13.800
LAN_1V | 0.990 | Volts | ok | 0.850 | 0.900 | na | na | 1.100 | 1.150
VOLT_PSU1_VIN | na | Volts | na | na | na | na | na | na | na
VOLT_PSU2_VIN | na | Volts | na | na | na | na | na | na | na
CUR_PSU1_IOUT | na | Amps | na | na | na | na | na | na | na
CUR_PSU2_IOUT | na | Amps | na | na | na | na | na | na | na
Temp_MB_A | 33.000 | degrees C | ok | na | na | na | 68.000 | na | na
Temp_MB_B | 36.000 | degrees C | ok | na | na | na | 55.000 | na | na
Temp_Card Side_A | 40.000 | degrees C | ok | na | na | na | 93.000 | 94.000 | na
Temp_Card Side_B | 37.000 | degrees C | ok | na | na | na | 93.000 | 94.000 | na
Temp_CPU | 35.000 | degrees C | ok | na | na | na | 93.000 | 94.000 | na
Temp_X710 | 39.000 | degrees C | ok | na | na | na | 93.000 | 94.000 | na
TEMP_DDR4_A1 | 40.000 | degrees C | ok | na | na | na | 84.000 | na | na
TEMP_DDR4_B1 | 42.000 | degrees C | ok | na | na | na | 84.000 | na | na
TEMP_DDR4_C1 | 42.000 | degrees C | ok | na | na | na | 84.000 | na | na
TEMP_DDR4_D1 | 43.000 | degrees C | ok | na | na | na | 84.000 | na | na
TEMP_DDR4_E1 | 37.000 | degrees C | ok | na | na | na | 84.000 | na | na
TEMP_DDR4_F1 | 37.000 | degrees C | ok | na | na | na | 84.000 | na | na
TEMP_DDR4_G1 | 37.000 | degrees C | ok | na | na | na | 84.000 | na | na
TEMP_DDR4_H1 | 36.000 | degrees C | ok | na | na | na | 84.000 | na | na
TEMP M_2_1 | na | degrees C | na | na | na | na | na | na | na
TEMP_PSU1 | na | degrees C | na | na | na | na | na | na | na
TEMP_PSU2 | na | degrees C | na | na | na | na | na | na | na
FAN1_1 | na | RPM | na | na | na | 500.000 | na | na | na
FAN2_1 | 900.000 | RPM | ok | na | na | 500.000 | na | na | na
FAN3_1 | 600.000 | RPM | ok | na | na | 500.000 | na | na | na
FAN4 | 600.000 | RPM | ok | na | na | 500.000 | na | na | na
FAN5 | na | RPM | na | na | na | 500.000 | na | na | na
FAN6 | 300.000 | RPM | nc | na | na | 500.000 | na | na | na
PWR_PSU1_PIN | na | Watts | na | na | na | na | na | na | na
PWR_PSU2_PIN | na | Watts | na | na | na | na | na | na | na
PWR_PSU1_POUT | na | Watts | na | na | na | na | na | na | na
PWR_PSU2_POUT | na | Watts | na | na | na | na | na | na | na
PWR_CPU | 53.000 | Watts | ok | na | na | na | na | na | na
CPU_THERMTRIP | 0x0 | discrete | 0x0080| na | na | na | na | na | na
STS_PSU1_Status | 0x0 | discrete | 0x0080| na | na | na | na | na | na
STS_PSU2_Status | 0x0 | discrete | 0x0080| na | na | na | na | na | na
STS_PSU1_AC_LOST | na | discrete | na | na | na | na | na | na | na
STS_PSU2_AC_LOST | na | discrete | na | na | na | na | na | na | na
ChassisIntr | 0x0 | discrete | 0x0080| na | na | na | na | na | na
CPU_PROCHOT | 0x0 | discrete | 0x0080| na | na | na | na | na | na
FAN1_2 | na | RPM | na | na | na | 500.000 | na | na | na
FAN2_2 | na | RPM | na | na | na | 500.000 | na | na | na
FAN3_2 | na | RPM | na | na | na | 500.000 | na | na | na
STS_CPU_MCE | 0x0 | discrete | 0x0080| na | na | na | na | na | na
Temp_TR1 | na | degrees C | na | na | na | na | 65.000 | na | na
The lack of VRM sensor quite relevant for my use case, as we shall see. Temps at full load (stress -c 48 = all threads looping on sqrt(something)):
# ipmitool sensor | grep -i temp
Temp_MB_A | 32.000 | degrees C | ok | na | na | na | 68.000 | na | na
Temp_MB_B | 40.000 | degrees C | ok | na | na | na | 55.000 | na | na
Temp_Card Side_A | 38.000 | degrees C | ok | na | na | na | 93.000 | 94.000 | na
Temp_Card Side_B | 37.000 | degrees C | ok | na | na | na | 93.000 | 94.000 | na
Temp_CPU | 66.000 | degrees C | ok | na | na | na | 93.000 | 94.000 | na
Temp_X710 | 41.000 | degrees C | ok | na | na | na | 93.000 | 94.000 | na
TEMP_DDR4_A1 | 44.000 | degrees C | ok | na | na | na | 84.000 | na | na
TEMP_DDR4_B1 | 44.000 | degrees C | ok | na | na | na | 84.000 | na | na
TEMP_DDR4_C1 | 44.000 | degrees C | ok | na | na | na | 84.000 | na | na
TEMP_DDR4_D1 | 44.000 | degrees C | ok | na | na | na | 84.000 | na | na
TEMP_DDR4_E1 | 41.000 | degrees C | ok | na | na | na | 84.000 | na | na
TEMP_DDR4_F1 | 40.000 | degrees C | ok | na | na | na | 84.000 | na | na
TEMP_DDR4_G1 | 40.000 | degrees C | ok | na | na | na | 84.000 | na | na
TEMP_DDR4_H1 | 38.000 | degrees C | ok | na | na | na | 84.000 | na | na
TEMP M_2_1 | na | degrees C | na | na | na | na | na | na | na
TEMP_PSU1 | na | degrees C | na | na | na | na | na | na | na
TEMP_PSU2 | na | degrees C | na | na | na | na | na | na | na
Temp_TR1 | na | degrees C | na | na | na | na | 65.000 | na | na
CPU is 66c, other sensors are lower, however my IR camera (inspired by @Nefastor) reveals what the onboard sensors don’t:
The 78.2c spot is near the VRM heatsink.
To be clear, I don’t consider the high temp on the VRMs to be ASRock Rack’s fault in any way.
It’s a server component formeost, so ASRock Rack engineers likely simply assume that the case will have a high-airflow layout and that system fan speed scales with CPU temp, and they are perfectly right to do so. The picture, if anything, makes clear that my beQuiet case won’t cut it for a CPU with this power (it would likely not cut it with my H12SSL board either - I only tested that one so far with a much weaker CPU).
Anyhow, for my kind of use case VRMs are a weak spot, and because of this it would have been great to be able to monitor the VRMs automatically. My Supermicro board has two TEMP_VRM* (don’t remember the exact name) sensors.
This all said, the VRM cooling is actually less beefy on this board compared to the ATX counterparts from ASRock Rack and Supermicro. The heatsink is smaller, and I count fewer phases:
ROMED8U-2T (left), and ATX counterpart ROMED8-2T (right)
But this difference won’t make up for bad airflow, and I will look for another case either way
Bios Settings II: Overclocking!
I was by far most surprised by finding these settings in the BIOS setup:
(in Advanced->CPU Common Options->Performance)
I don’t know if any other SP3 board has anything like this?
CPU “overclock”
Then choosing “Custom Core Pstates” leads to a disclaimer followed by a setting to change the clock frequency of Pstate0. Apart from that, I could not find any other OC settings related to the CPU.
I did not try to change this (or even to save any BIOS settings after accepting the disclaimer to see the options). My guess is that the actual overclocking potential enabled by this single item is quite limited. NB that Pstate0 != Pstate0 Boost, I interpret this as what is changed is the Pstate0 non-boost setting, which is 2850MHz on the 7443p.
I haven’t tested and I wont experiment with a chip this expensive, but I suspect that this setting won’t enable any core to go above the max boost limit (4.0GHz for the 7443p).
Memory overclock and timings
There was a similar disclaimer screen under AMD CBS->DDR4 Common Options->DRAM Timing Configuration, then followed by this menu:
(cont’d)
This screen looks indeed more useful!
Examples:
- Memory Clock Speed allows settings in the range between 400-1800MHz (i.e. max. DDR4-3600).
- Tcl (CAS Latency) can be set using hex values up to 0x21.
I did not check the rest in detail. However I like the idea with overclockable ECC RAM, especially the ability to manipulate timings.
Summary
Those were my impressions and overview. It is clearly not “fair” as a review as I judge the board based on my own kind of non-standard use case (although the first page of the manual says both “Workstation” and “Server”). But there were things I found interesting and different from my H12SSL board, and since this is my only ASRock Rack board, I don’t know what is specific for this board or common with ASRR’s other SP3 offerings.
I really like several aspects of this board, mostly it being micro-ATX. Manipulating DDR4 timings is intriguing, and something I might try at some point. There were also aspects where I think my H12SSL board suits my workstation usage better, for example the ability to choose which PCIe slot to initialize video from, and the ability to monitor VRM temps.
The VRM thermal issues is not something I attribute to the board though, I believe it’s a consequence of the new 200w CPU and the case (beQuiet Silent Base 802). So I intend to replace the case for something more airflow-oriented.
7 Likes
@oegat thanks for the detailed write-up!
I found the same annoyance when using multiple GPUs on my ROMED8-2T, but how did you find out the “first adressable slot”? lspci/lstopo?
Actually I’m not sure about the terminology, I meant simply the slot with lowest number on the board layout schematic (PCIE4):
Although now when I’m checking, it seems they get enumerated in opposite order:
# lspci | grep -i vga
45:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 41)
81:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Lexa PRO [Radeon 540/540X/550/550X / RX 540X/550/550X] (rev c7)
c1:00.0 VGA compatible controller: NVIDIA Corporation Device 24b0 (rev a1)
It could be that it is really the other way around, that the Radeon (on PCIE5) gets initialized first. Actually both the Radeon and the A4000 produce output until grub takes over, but the grub menu and the first kernel messages (before the GPU driver takes over) are shown on the Radeon only. So yeah, it seems its the Radeon that gets priority.
1 Like
Some ASUS epyc boards, like the RS520 series and probably RS720 series also support changing memory timings as well. You can’t change voltage (not that the board is built to handle that), but you can still get about an extra 5% performance from your sticks which is nice.
One thing I didn’t check, is if increasing the ram frequency would also potentially increase FCLK. Not too hopeful, but it’s something to try.
1 Like
Look for or ask Asrock Rack for an updated BIOS or beta BIOS to fix the missing inventory.
I have found low profile 6" USB header extenders to solve the issue of the large molded USB cables from cases not fitting. Allows you to move the male-female connection off the motherboard and to the side or under the motherboard or hidden elsewhere.
I buy them from moddiy.com. I have two more ordered but in transit for the past month because of the Chinese lockdown.
I’ve used them on just about every motherboard I own.
2 Likes