TLDR; WRX90 + 2xA100 & RX7900XTX + at least 256GB RAM + enterprise SSDs
Hi everyone,
I recently got permission to build a workstation for work, and could use some tips and recommendations on the planned build, as I have never built a true WS myself before.
Why:
I’m a computational astrophysicist and AI researcher. I code/run/debug/analyze hydrodynamical simulations half of my time and train AI models for these simulations and also other kinds of data such as medical images for the rest. Long story short, I need (prioritized in that order) multiple GPUs, tons of fast RAM, tons of storage (preferably fast and low latency), and as many cores as fit in my provided budget.
Components:
I cashed in some favors and managed to get an A100 GPU for free (perhaps a second one coming in this summer) and can install a RX 7900XTX (for 2x4K displays) from an “old” system where we experimented with AMD GPUs. I also already have a 1600W PSU. I could also get my hands on either Kioxia CM7, CD8, or Micron 9400 Max, when we order the next round of storage for our servers. Unfortunately I have no experience with server-grade SSDs in workstations (tips specifically on this would be appreciated, especially connectors); from what I understood, the MCIO or SlimSAS connectors of the AsRock WRX90 could be used for those… (I had bad experiences with Asus mainboards on my personal builds, so I’m biased towards AsRock).
MB: AsRock WRX90 WS Evo
CPU: TR Pro 7975WX (or TR Pro 7985WX)
GPU: AMD RX 7900XTX, Nvidia A100 (x2)
RAM: Kingston Fury 8x32GB 6000 (KF560R32RBK8-256; need at least 256GB)
I plan to exclusively run Arch linux on btrfs (in raid 0 if I get more than one SSD).
I know the AsRock WRX90 board isn’t even out yet (in most places), but would still appreciate any thoughts, tips, and recommendations on the plans. Are there better suited components, any design mistakes, or things I should rethink?
I recently learned that parameterized sweep simulation data can be used to train a DNN to predict for unsolved parameters which is neat. Also polynomial chaos expansions can be used instead and actually give an uncertainty measurement on the prediction.
It might be prudent to hold off on pulling the trigger on WRX90 because of the amount of PCIe/RAM related issues people are experiencing. To me it seems like the BIOS is currently half-baked on the platform, which is perhaps why Asrock still hasn’t released their motherboard.
If you’re time crunched, consider an 8-channel W790 platform (which unfortunately Asrock doesn’t make).
As for the SSDs, all three you mentioned are fine choices.
With that many PCIe 5.0 SSDs, you’ll probably want to skip using the MCIO connectors all together since the board doesn’t offer enough of them; The use of PCIe to U.2 carrier cards is probably your best bet (these motherboards should offer the x4x4x4x4 bifurcation option needed to run the U.2 carrier cards).
The MCIO and SlimSAS cables are really finicky with the high PCIe speeds we’re running now, if they can be avoided they should be. some of the manufactures are starting to integrate powered signal redrivers into the cables themselves because of how unreliable they are becoming at high speeds (PCIe 5.0).
It is neat The Flatiron institute is a big proponent for such methods, however I remain sceptical about such inferences as it is often unclear what assumptions went into them. Often, it is difficult to properly assess uncertainties in cosmological studies, especially when NNs are involved, which is why I’m exploring various XAI techniques for exactly these kinds of problems…
I was thinking the same… however, time is no issue (as long as I spend the budgeted money this year). For now, I’m waiting to see how the AsRock board will compare against the Asus and its myriad of problems, but I will most likely try to make the WRX90 work in any case. I’m still hoping that AsRock’s release will be smoother.
On storage, I changed my opinion after some research of my own. I need at least 60ish TB of reliable data drives, for which I’m planning to use 2-4 PCIe4-speed SSDs (costs will decide what kind exactly). A single T700 4TB will be the boot drive. As my institute just put in a bulk order for Kioxia CM-7s, I’ll probably get those for “experimentation”. I’ll look into MCIO stability (out of curiosity), but as I will have two free PCIe slot anyways, I will most likely end up using those U.2 carrier cards.
This build will most likely take an entire year. I will wait for good deals to maximize what I can get with the available budget (which I actually don’t know precisely yet). I can post updates here, if there is interest. Once it is operational, I will experiment with hardware configurations and perform some benchmarks .
The Flatiron institute is a big proponent for such methods, however I remain sceptical about such inferences as it is often unclear what assumptions went into them. Often, it is difficult to properly assess uncertainties in cosmological studies, especially when NNs are involved, which is why I’m exploring various XAI techniques for exactly these kinds of problems…