I have so far not seen a SP3 mainboard that accepts unbuffered RAM, so I don’t see how an Epyc could practically post without ECC. (buffered non-ECC ram does not exist afaik). Although, TR Pro does support UDIMM, so probably the memory controller in the Epyc could in principle handle it, since the chips are supposed to be of very similar design. Either way I prefer ECC. Ability to overclock memory would have been interesting (since I would actually know when things start failing), but not important for me.
Actually for me the mainboards are the breaking-point on the positive side. SP3 boards are generally cheaper to match the lack of workstation goodies, which I can add on demand instead. I need IOMMU separation, although I believe the existing TR pro boards have good separation also for the chipset stuff, as long as things are connected directly to the CPU’s pcie lanes I can be sure. So one less thing to worry about.
I have an old 4U 36bay supermicro chassis that I took out the screamers, and put in a double row of 3 ARCTIC P12’s.
This gets me good general airflow, is decent enough for my disks and SFP+ NIC, but I still always put a dedicated fan on my HBA’s. The old SAS2 models can get by with a dinky 40mm, but anything newer needs serious airflow and gets a dedicated blower.
As long as you are willing to address the needs of things directly, you can make things perfectly sane noise-wise.
Server boards are designed around high airflow for VRM cooling
Server boards do not supply audio or as many USB channels
The VRM cooling was not something that I had worried too much about, but indeed, with a TDP of 280 W and assuming a DC-DC converter efficiency of 90 %, you are going to be dissipating roughly 30 W of power in your VRM stages which is probably a lot for the small heatsinks on the Asrock board without some airflow. Fortunately even a small amount of local airflow can carry a large amount of heat away, so I might be able to get around this with either enough case fans or maybe hacking some sort of local fan to also blow on the RAM.
For cooling, I was planning on going with water cooling, primarily to avoid the loud GPU fans at full bore. This both removes some weight from the GPU and slims it down to a single slot. I am looking at a Fractal Define 7 XL with two 3x140 mm radiators (probably overkill, but I would like to be able to scale up to two high-power GPUs in the future).
That Asus board is certainly beautiful and I would not mind using it if it supported Epyc Milan CPUs. @wendell, have you done any testing with PCIe-passthrough on the Asus board? This would be my big concern: not supporting some of these more “server” features since they are not really needed for VFX workloads.
Beyond that, the big problem that I have with TR-pro is the lack of a 24-core option. I could spend a bit more than the 7443P, but 16 cores feels small and I cannot afford to spend twice as much for the 3975WX.
As with @jtredux, I really would prefer if they would just give me an OCP slot so I can add my own NIC. My local high-speed network is primarily fiber-based.
I certainly intend to ! I’ve been running a 6-core chip and 64 GB of RAM for the last 7 years so I’ve been limited in that department. Typically I’ve used VM’s to keep a Linux on hand for those engineering apps that don’t run on Windows (such as the early days Altera SoC YOCTO toolchain). But I plan on going a lot further with the 24 cores on a 7443P.
Regarding the Asus TR Pro board, be careful : because of its 7-slot layout, all the connectors on the “bottom” edge of the board are angled and will become inaccessible in a 19’’ chassis. This disqualified the board automatically for me. I tend to buy my workstation hardware with the expectation that it’ll end-up in a server rack several years down the line. Even in a normal ATX case, you need some room below the board to be able to use the Asus. The Define 7 XL has that room. It was my first choice until I found the Enthoo Pro 2.
I’ve just received a 7282 this morning. I’m about to finally put that workstation together. Expect some photos and commentary soon.
The build is proceeding smoothly. I’m taking my time. I’ve just installed the processor and the RAM and I’m playing with the IPMI while Windows 10 installs to an M.2 SSD. I can already report a few interesting things.
I’m running off a UPS. According to its display, at this point the whole system uses only 45 to 65 W depending on activity. I took this picture with the FLIR :
The only fan in the machine at this point is the one on the CPU heatsink, and it’s only running at 700 RPM making no noise a human ear can perceive. The VRM is barely above room temperature. The hottest part on the board is the X550, obviously, at around 47 °C. The RAM sticks are all under 40 °C. And in the time it took me to write this, Windows 10 installed from A to Z. Cool.
Regarding the heatsink, it appears my measurements were correct : the 15 mm fan is a nice fit.
That being said, the steel clips were meant for a 25 mm fan and are not as tight as I’d like. It’s acceptable, though. And it will make it easy to remove the fan when I add the final four DIMM’s.
So far so good. Next, I’ll install some fans at the front of the case and try running a GeForce 970 off a PCIe extension.
I installed the GTX 970 (can’t go straight to the 3090 'cuz, well, it’s what I’m using right now). Initially, the motherboard wouldn’t boot : it would try to, get to the Asrock splashscreen and then cycle. I got into the BIOS and found the option to select between IPMI and “external” for the “VGA”. It’s set to “auto” by default. I changed it to “external” and then specified which PCIe slot the 970 is in. It then booted without problem.
The motherboard comes without drivers CD-ROM, and Asrock’s website only provides drivers for Windows Server (and Linux, of course). I just downloaded Intel’s X550-T2 drivers, put them on a stick and installed them manually, then the system found the internet and I let Windows Update do its thing. Within minutes I got my desktop to 4K resolution.
I also found out that the GTX 970 can operate as a sound card and output through HDMI. So I got sound through my monitor’s speakers. Interesting.
With the added GPU and still no additional fans, Windows 10 idling in 4K or doing updates, the system draws at most 100 W from the wall. The fan on the PSU doesn’t even turn on yet. It’s eerily silent. I’m starting to wonder if that VRM will actually get hot at some point.
About that : I forgot to mention that the VRM heatsink is screwed to the board and it appears it can be replaced. The heatsink on the X550 is mounted with spring-loaded screws and some thermal grease. It does not make contact with the AST2500. That heatsink could also be replaced. Looks like this board is a modder’s dream. Did I mention I have a CNC milling machine and I cut aluminum alloy for fun and profit ?
Back to the BIOS : I’ve verified you can set each PCIe slot to operate as four x4 slots, meaning it’s theoretically possible to install 28 NVMe drives on 7 quad-M.2 expansion cards. That’s crazy
I’m satisfied with my hardware choices, it looks like everything will shake down as I plan. Tomorrow I’m doing the BIG move and installing that RTX 3090 and PCIe 4.0 SSD. It’s gonna be fun.
Not yet. This was just a verification that the mobo / CPU / RAM all worked well together. You always want to make sure before spending time tidying up the wiring, installing expensive peripherals and setting up all the software. Nothing worse than to do all that, have a nice-looking rig, only to find out it doesn’t boot
I have to say, case manufacturers have really done a lot of progress since I last bought a big tower case. I’m loving the cable management options on that Phanteks. And the connector placement on the Asrock board makes my life really easy. Check it out :
And the cables are the exact length to do this. I’ve been really lucky on this one, this wasn’t planned at all. Unless Asrock, Corsair and Phanteks conspired to help me
I’ve already installed an Nvidia 12-pin 12 V connector for the 3090 FE. It’s getting late where I live, though, so that transplant will have to wait until tomorrow.
I’ve also got to install the Sabrent Rocket NVMe. There’s a conundrum : I bought the kit with the purpose-built heatsink but it’s clear that its height will interfere with the PCIe slots if I were to install long (-enough) cards. I wonder if that heatsink is really necessary or if the airflow inside the chassis would be enough to keep it from throttling. I bought that drive based on Wendell’s video about it, I guess I should watch it again.
Regarding the benchmarks, I will definitely do some but keep in mind this is not the CPU and RAM combo I plan on running as a workstation. Those parts will end-up in a server. The real goal is that Milan 7443P and eight channels of RAM
I’m done for tonight. Stayed up longer than I should have but you know how it is when we get shiny new toys… I’m just posting about an issue I’m hoping someone can help me with.
I’ve installed and updated Windows 10 Pro, I’ve also installed the AMD EPYC drivers (which I had to get from Supermicro’s website, because Asrock didn’t see fit to provide them). I was about to turn off the PC and noticed there’s no option to hibernate. That is something I can’t live without.
I did a quick Google and tried forcing the option on. Here’s what happened :
Yes, I’m pretty sure the whole system will heat up as soon as I do actual work with it. This being my first time with EPYC I’m just getting a feel for it, raising the stakes little by little to see if it’s going to be viable for my purposes without destroying anything. The FLIR is a great help, there.
It’s also told me that the Sabrent Rocket really needs good cooling. Here’s how hot it got after just the Windows update :
This is naked with no air flow. I’m adding fans at the front of the case next, but if this isn’t enough to keep the SSD cool I will need to use its heatsink, which will eventually interfere with PCIe cards. I’m starting to think I should use a PCIe M.2 carrier card instead of the motherboard sockets. Or the OCuLink connectors.
The X550 is also worrying me. When I first unboxed the motherboard I noticed its heatsink was really loose, as if the springs on the screws were too weak. The infrared shows that it does do its job, though. But I’m worried that it’s not going to see a lot of airflow with all the obstacles in front of it. I know this NIC runs hot, but what would you say is a safe operating temperature for it ?
Fun fact : last time I used an AMD processor was the original Opteron. I guess when it comes to AMD I only like their server products
Really getting stumped by that hibernation stuff… it’s starting to look like I will have to use a ThreadRipper Pro after all… and this motherboard will end-up in my file server.
Still, a couple of interesting things to report :
I’ve tried updating to the motherboard to beta Milan BIOS and firmware. The BIOS update worked, but the BMC update doesn’t seem to have completed. The BMC now reports the same BIOS and firmware versions as before I attempted to update it. However the BIOS has more settings and is clearly the latest version. The machine boots Windows same as before. I assume it’s now Milan-compatible.
There is absolutely no fan control on this board. The options are absent both from the BIOS and the BMC, in all the versions I’ve tried. That’s all the more surprising that the board’s manual describes such fan control options. The fans are all running at their lowest speed (700 RPM on the CPU, 500 RPM on the three 140 mm chassis fans). Nevertheless, this has lowered temperature on both the SSD and the X550 by a few degrees.
I haven’t had an Asrock board in a long time but I remember them being good products. Has something changed ? It’s as if they wrote a user manual for a BIOS that doesn’t exist. Did they copy-paste it from another board’s manual ? This is weird.
It’s probably a stupid quiestion, but did you retry the BMC update that failed? I would imagine that there are some dependencies between the BIOS and BMC versions even if they don’t have to match exactlty.
It’s not a stupid question, and I haven’t. The reddit link you posted is interesting. They mention a BMC firmware version that’s not on the Asrock website.
Regarding my hibernation issue, it also appears ACPI is pretty much missing entirely, as no power state is supported
This is a bit ridiculous. Hundreds of BIOS settings, a BIOS that’s over 30 megabytes, and they couldn’t add something this basic ?
Your issues with the ROMED8-2T got me thinking about Supermicro again - the H12SSL-NT does look interesting. While I don’t need the 10-baseT, it seems to be a £30 option, so a bit of a no-brainer IMO.
I guess I’ve not given much thought to the whole hibernation thing - but that is a rather handy thing in a workstation, even if I don’t plan to run Windows on bare-metal.
I’m looking at the Supermicro’s manual. I went for the Asrock first because of its sexay 7 x16 slots but the issue with fan control and especially the inability to suspend are deal-breakers for me. I might still keep it for my new server, but the H12SSL seems like a winner for a workstation, if it really supports hibernation.
Its block-diagram shows some interesting choices in PCIe lane assignment, and honestly I can live with it.
I don’t know how good the Broadcom 10 G NIC is compared to the X550, but it’s probably serviceable and if not, I have a spare X540 PCI card laying around. I just might give this board a try. It’s available on Amazon where I live, but there’s a one-week delay.
! I’ve been running a 6-core chip and 64 GB of RAM for the last 7 years so I’ve been limited in that department. Typically I’ve used VM’s to keep a Linux on hand for those engineering apps that don’t run on Windows (such as the early days Altera SoC YOCTO toolchain). But I plan on going a lot further with the 24 cores on a 7443P.
