Hey all,
Im here with a SexxySensei update.
Just reached out to Asrock who provided me with a beta bios 1.23.
It fixed two things for me.
The FF issue still remains for now, looking to see if related to unraid specifically.
Thanks,
Hi guys,
I’m trying to reach the full speed on all NIC with this board (I have the 2L2T variant) and ESXi 7.0. But it seems it can’t go over 1Gbit/s even when connected to a 10Gbit/s port on my switch.
I tried with another switch at 2.5Gbit/s and the issue remains, I cant reach the full speed, it throttles at around 600-700Mbit/s which is a lot less than what I was expecting.
I try the switch, cable and everything with another computer and speeds are fine.
Did anyone have this issue and could you resolve it? I’m using the standard driver provided by VMware ( ixgben)
What BIOS and BMC version are you running? I seem to recall this was an issue with an earlier BMC version and covered earlier in this thread.
Hi all,
New to L1T, long time lurker 1st time poster, I just want to share my recent experience with X570d4u MB.
I am sure you all know, X570d4u is one of its kind, only properly equipped server board for Ryzen not Epyc. Asus has a X570 workstation mb, but it doesn’t have a proper out-of-band mgmt solution like IPMI, too bad, it’s much cheaper than X570d4u and it looks very well made.
Anyway, my current server is Supermicro X11SCH-LN4F + Xeon E-2136(6c 12T) + 32GB DDR4 2666 ECC Crucial, running multiple VMs on RHEL KVM, with 2 X 10TB WD Red Pro for virtualized TrueNAS(with Plex Jailed), rock solid. I LOVE Supermicro, I have a X10SBA mITX mb running pfSense since 2014, still strong after 8 years, and gets almost full 1gb speed from my fiber internet.
A couple weeks ago, I bought X570d4u+5900x+32GB 2666 ECC for my 2nd server running Proxmox, it’s ok the overall experience, a bit faster but I notice that idle power consumption is high and strange.
My Supermicro server idle power wattage is around 38W, with all the VMs(~6) running, it was ~ 30W ish with just RHEL installed, and it stays at 38W after all the VMs are installed and running.
The Ryzen rig however, stays at 54W with just Proxmox installed and no VM (7nm doesn’t really help), it goes up to 76W after I installed 2VMs, one pihole and one TrueNAS, and it becomes the new idle power consumption, weird!! I changed the power settings in the BIOS and set the power governors to ondemand on Proxmox, no change at all.
I know I shouldn’t keep nagging you about the idle power, I think I might switch back to RHEL from Proxmox and see how it behaves. LXC on Proxmox is fun, and docker on RHEL sucks, but now that I have 12c24T, I might stick to the old time rock & roll KVM lol.
over and out!
Hi all, also first time poster here! I wanted to say thanks for all the info in this thread! After following here, a little while ago i decided to go for this x570d4u-2l2t for my three node Ceph/Kubernetes cluster, maybe useful for other people to read that following hardware combination works really well:
In bios I configured ECO modus, the two SATA SSD’s are used for boot and are configured in software raid (not in bios, but using mdadm). The Bifurcation card is in the top x16 slot and the Mellanox 25GBe card in the bottom slot, this means the top one functions at x8. If I remember right, in bios i configured as x4x4x8. This way there are three NVMEs attached directly to CPU, these are used for Ceph while the fourth (sharing bandwidth with other devices through chipset) is used for temporary storage.
I remember that right after installation I had a couple of crashes, the chipset was really hot but hardware was not built into the case yet so no cooling going on and having read experiences here I blamed that. After having everything assembled with fans and all I never had a hang or crash anymore, things are stable.
For anyone out there running Proxmox (or, I suppose, any Debian-powered OS), a question:
Can you recommend any software for controlling the fan RPMs that lets me get the speed lower than 20 percent of full speed on some of the case fans?
I’ve built my server into a Supermicro CSE-835, and all my drives are low-powered SSDs
I’ve got active cooling on both the CPU and VRM; they idle in the low 30s and low 40s respectively. RAM is in the mid-to-low 30s at idle as well. The VRM is the hottest thing in the system, from what I can tell, and it’s got about 40 C of thermal headroom before I need to start worrying about it.
Problem: The overpowered Supermicro front intake fans are loud, even at their current 20 percent setting. I suspect these fans–which are meant to cool passively cooled Xeons, etc.–are entirely overkill for my setup at 20 percent of their max RPM. I’d like to test them at 15 percent or even 10 percent, but the IMPI’s manual fan control interface won’t let me go lower than 20 percent.
The person I bought this rig from was using Unraid, and set the fan speeds using a add-on Unraid module. I don’t see anything like that for Proxmox, though fancontrol (which I have never used) is available in the repo.
The seller told me he used the Unraid module to set the fans to their lowest speed, which achieved an acceptable low volume for working near the server in an office room and kept everything cool enough–and he was using spinning HDDs, which would have run hotter than my setup.
Is fancontrol the way to go? If it is, is there a good tutorial somewhere?
Or should I be looking at something else?
EDIT (2022 04 23 @ 2040 US Central Time): I’ve just realized that I should probably be using ipmitool for this, but up-thread it looks like that’s not working for some people.
Has anyone actually gotten it to work?
(I hate messing with the fans. I still don’t understand what the difference is between open loop and closed loop tables. I’m using the open loop for the CPU and have everything else on manual, but I have no idea if that’s actually the way to go or not w/r/t open vs. closed.)
Hello there! If you were successful with this BIOS could you share the file here? Asrock support has been unresponsive to me and I am running into the same exact issues.
I’m looking forward to this one. I’m experiencing both these issues. System Inventory doesn’t work, and I can’t access the BIOS at all without going through the virtual remote display thing.
I need to reach out to them about the power management. The board can’t see my dual PSUs at all. They’re defaulting to both being active, with an uneven load split between them, and an alarm goes off if I try to run it with one PSU only.
New Beta BIOS for X570D4U-2L2T:
There’s a new “Beta Zone” tab in the downloads section for this board.
https://www.asrockrack.com/general/productdetail.asp?Model=X570D4U-2L2T#Download
Not sure if it’s there for the D4U, but I’d suspect so, since it’s the 1.5x beta BIOS for Ryzen 5800X3D support. @Marco1 , this beta BIOS may or may not address your issues. I haven’t found release notes for it yet.
No BMC update.
The latest release of the BIOS is 1.40. I’m optimistic that the released 1.5x version will also include updates to the System Inventory/BIOS access from within the IPMI.
I still need to ask them about why the board can’t talk to my dual PSU’s Power management controller thing.
Unrelated: I’ve also asked support how much the maximum sustained (24/7) wattage load is on a single fan header. I’m running almost all my fans in manual mode, so I’m a bit concerned about that. I’ll report back when I hear something.
I contacted ASRock Rack about this issue but it was ignored.
Is there anyone who can solve this problem?
This has to do with whether you’re booting up in Legacy BIOS mode or UEFI mode.
For example, some of the older RAID cards expect legacy BIOS mode, and the keyboard shortcut won’t work when you’re in UEFI mode.
Proxmox needs me to be in UEFI mode, so I can’t directly access the RAID setup utility.
Options:
I know there are some people in this thread that upgraded their CPU from a 3000 series to a 5000 series.
I’ve got a Ryzen 9 5900X on my desk, ready to swap in to replace my Ryzen 7 3700X, without experiencing any sort of catastrophic explosion. 
What’s the recommended way to do this to avoid any issues and make sure the server turns back on when I’m done?
I’ve never swapped a processor on a set-up system before.
Proxmox is already installed, if that matters.
EDIT: Well, that all seems to have worked. I forgot to reset the BIOS first, so the first time around it didn’t see half my RAM, but after a couple of BIOS resets it’s back to normal. It also took my minor RAM overclock (DDR 3200) fine–I still need to run a memory test to make sure all is well.
Something changed with my LSI HBA cards, though. Both of them are still recognized by Proxmox, which still sees all the drives plugged into them, but only ONE of them is listed when the system boots (the part that lists all the detected HBAs and drives and prompts you to enter setup.
Aside from that, they seem to still work as they did before, but I don’t like not understanding what happened.
Re: Precision Boost Overdrive:
For power consumption reasons, I’m considering disabling this. I went and found the settings page in the BIOS, but I don’t understand what it’s showing me. It’s not disabled, or set to Auto, but rather Advanced.
It looks like this:
I haven’t set any of this, and those settings are there after two Reset to Defaults commands.
Does anyone else’s look like this? Is it some sort of optimized Asrock Rack settings thing?
Can’t say anything about swapping, just don’t try to hot-plug the CPU 
. I don’t think there will be a problem. But check BIOS in any case.I never trusted BIOS “auto detect”. 5900x runs fine with both board and proxmox.
I sometimes see wierd SATA BIOS detection too, but in practise, everything’s running just fine.
Inever touched the overclocking stuff other than 1600MHz FCLK+memory and ECO-Mode. I have to maintain my sub-100W power target 
Thanks, @Exard3k . Reassurance is always appreciated; I have no idea what I’m doing.
Running Prime95 on a Windows boot disk now.
CPU holding steady at 69 C - 70 C in the IPMI sensors window, even with the tower cooler installed and blowing air left to right instead of front to back. (I’ve got the proper mounts on order to fix this, but.)
In Windows, Hwinfo isn’t detecting the RAM sticks properly. Proxmox knows there’s 128 GB. Windows 10 knows there’s 128 GB. Hwinfo knows there’s 128GB.
But Hwinfo doesn’t actually show any memory sticks in the dropdown menu where you can go to look at detailed info.
The BIOS, OTOH, does.
My 3700X was rock solid as far as things being detected properly, so I’m a bit annoyed because I don’t know if this is an actual config problem or some sort of harmless bug in the BIOS or Hwinfo.
EDIT: Okay. Now I’m concerned. Memtest86+ only saw 64 GB of RAM, even though the BIOS sees it all, and so does Windows and Manjaro (Arch) Linux.
Help?
EDIT 2: So…I’m not in EFI mode. What.
EDIT 3: Okay, I switched to another ISO to run Memtest86, that supports the UEFI version of the test software.
With the Ryzen 9 5900X installed, attempting to run Memtest86+:
The Ryzen 3700X would run the test for 24 hours at 3200MT/s.
tl;dr things have been less stable and more weird since installing the Ryzen 5900X. I’m kind of hating it right now.
EDIT 4: Grabbed the paid version of Memtest86.
@Exard3k , where do I go to adjust the timings themselves? You mentioned a while ago that you were using the JEDEC standard ones to good effect. Also, do you see the same errors collecting SPD data from yours if you run Memtest86+? (Free or paid are both the same on this…)
The CL22 timings are the intended timings for this RAM according to the spec sheet. If the BIOS selected anything else, that’s bonkers.
Wait, wait, wait. Is that 100w for the whole system, or just the CPU? 
What are you doing currently to limit power while maintaining performance? I remember discussions in this thread several months ago about the 5900X specifically, but it’s been long enough I’m curious what you ended up going with long term.
So far, all I’ve done is set ECO mode.
I’m really not super comfortable trying to tweak anything else without someone else’s input. I don’t understand a lot of this yet…
Idle/low load power for the whole system. I get ~130W at the wall with everything running, from 10GbE and HDDs to CPU doing heavy lifting. But most of the time, system runs at 80-90W. Right now I’m using a GPU in the x16 slot because I need a gaming-capable VM until I can get myself a new daily driver (probably AM5), which increases power draw by quite a bit (140-250W). But that’s just for a couple of months.
ECO mode. Limits PPT (package power tracking) to 86W maximum instead of 142W. Really is free real estate unless you need that last 10-15% of single-thread performance or all-core scientific compute.
And I’m running my memory on the default 2667 MT/s. I don’t see improvements using 3200 MT/s on my usage as my VMs and servers don’t benefit from higher bandwidth.
I get mce memory errors while using 3200, but these have a pattern and fixed duration between them. All are 1-bit correctable ones. This seems to be a BIOS/firmware thing more than CPU or memory as I’ve seen other people experiencing this as well. May be exclusive to Vermeer chips. I just stick to 2667 as everything is just fine and also uses a bit less power.
Cool. I’ll have to check idle power when I’m sitting in (almost completely unconfigured) Proxmox–that’s as close to idle as I know how to get right now. Oddly enough, the CPU runs hotter in the BIOS than it does when idling in Proxmox.
I’ve got it in ECO Mode now, which I’m coming to understand is mostly just a preset for Precision Boost Overclock (I love how all these things tie together and depend on each other and none of this is explained in the manual.)
Where do you go in the BIOS to see the actual timings (22-22-22)? I want to make sure mine is set up right and didn’t auto set to something strange.
Have you considered locking it to its base clock? Or undervolting? The wizards of Ryzen 5000 power usage seem to swear by undervolting, but it is confusing and scary.
I’m in hour 12 of a Memtest86+ run (pass 4), and right now it’s drawing … 239.7 watts (2 amps). That seems … high, but I don’t know how much load it’s actually putting the CPU under–it’s using 12 of the 24 available threads. Temp is stable at 55 C, though, so at least there’s that.
EDIT:
It just occured to me to pull all the front hot swap drives (except the pair of VM store SSDs–I need that to be considered part of base power consumption).
(Yes, this is hotter than the Ryzen 7 3700X was, but I think that’s unavoidable. I also need to fix the orientation of the Noctua cooler I have in there so it’s not at a right angle to case airflow. That might help a bit.)
Do you have access to Memtest86+? If so, would you mind checking to see if you can read the SPD data? It should be available if you ask for information on the RAM modules. Right now, it can’t read mine.
I’m on a business trip and only have access to laptop right now. Proxmox and IPMI are inaccessible from WAN. But a full Memtest86 run was planned anyway.
Only thing I’ve done is changing the CPU governor in Proxmox and I tested powersave which sets clocks to 2200MHz, but I always end up with conservative on all my systems
AMD Overclocking → DRAM timings and frequency → timings become visible after you change auto to enabled along with frequency option. I tested a bit, but didn’t really do much, so I stuck with auto. Haven’ touched the BIOS in 2 month or so and that was only to do changes necessary after I installed the GPU.
Thanks. Hope your trip goes well. Looking forward to your test results. I really appreciate you taking the time to run and share them.
I’m almost 15 hours into a memtest86 run, on pass 4, test 12, and just saw my first ECC error with it clocked to 2933 Mhz.
I’m paranoid I somehow bent a pin on the CPU, but that doesn’t make any sense. It should be failing more consistently if I damaged something.
I don’t really understand why the 5900 is so much less stable than the 3700X w/r/t memory speeds, but once the test is done, I’m going to run it again with the default DRAM timings and frequency. If that doesn’t work, I’m giving up and going back to the 3700X.
That is why we use ECC. Errors happen and we pay for things to be automatically detected and corrected. If it isn’t periodically and persistent I see no problem, quite the opposite.
Probably some deviation on the firmware between Matisse/Renoir and Vermeer architecture.
