Threadripper Build

Hardware Build a PC
1

I’ve build dozens of PCs before, but its been several years since the last one and I plan going bigger then I have ever before on this (long overdue) workstation build, so I’m looking for a sanity check. I’ve been living and working off a mbp and haven’t had a proper workstation for years so I really want to build something nice.

The build is going to be centered around Threadripper (leaning towards 2990), and for kicks I’m going to cram it into a Cerberus X case (already purchased).

The primary use for this computer will be software development, and the main OS will be Linux (or possibly BSD). There may be some data pipeline processing research/prototyping done as well as varied other R&D projects.
Gaming will happen on this rig as well but is a secondary concern, I’m usually content with indie games and older games although I wouldn’t mind the option to play an A list game on occasion.

This workstation will be overkill by design, but I don’t want to fall into any traps of spending lots of money on anything that won’t actually provide any noticeably improved experience at all… so please call it out if you think that’s what I might be doing at any point.

I’m not interested posting benchmarks, and as far as overclocking is concerned, I might see if I can get a modest/slight overclock via bios tweaking without effecting stability, but its not the goal.

I want to start with ram since this seems to be the hot topic for threadripper, and I’ve litterally never been afraid of picking the wrong ram before but now I am. Take this kit, seems to be well priced for the clock speed on newegg right now: https://gskill.com/en/product/f4-3333c16q2-128gtzb. I’m not sure if I should be thrown off by the fact that it’s “for intel x99”. I don’t know if it’s b-die, I don’t know if I should care. I’ve read several threads and still I’m not sure if this is a big deal or if the whole internet is overthinking this and I’m not going to notice the difference as long as it works.

The next choice is MB and it seems like there are good options from ASUS, Gigabyte and MSI there

Cooling will require an AIO and the question comes down to whether a 280 will fit at the same time as an EATX motherboard in the ceberus X, which I can’t find a definitive answer too so I may just need to try it. 240 should fit for sure.

GPU will be ATI, thinking something like this https://www.newegg.com/sapphire-radeon-rx-vega-64-100410nt-sr/p/N82E16814202321?Item=N82E16814202321&Description=vega64&cm_re=vega64--14-202-321--Product , but the Radeon vii isn’t out of the question

1 Like
2

Welcome and good luck on your project :slight_smile:

I’m not sure if you are fishing for input, but i think people generally start with a shoping list and talk abit about their intended usecase to open up for input.

1 Like
3

Get everything but the cooling build it then order the cooler. Slim fans fit in the front.

I did a custom loop in my Cerberus and not fun fitting stuff it was a bit of a cram

Plan on moving to Cerberus X for next case probably as no love for mATX sadly

What psu mounting option did you go for?

4

If you want to go for a 2990WX build,
then i would highly recommend to buy a highend 2nd gen X399 board aswell.
Asus Zenith Extreme Alpha or Msi X399 MEG Creation.
Those 2 boards have the proper vrm implementation to run a 2990WX well.

But of course the question still would be, would you really need a 32 core cpu?
That is kinda depending on the work you do, and which software you use for it.

Also make sure your particular case fits E-ATX formfactor motherboards.
Otherwise a 2990WX is kinda out of the equation in a limited space.
Because ATX boards are mostly all gen1 X399 boards.
And those don’t really have a great enough vrm to run that 2990WX 250W tdp powerhouse.
You could eventually run a 2990WX without any overclocking on a X399 gen1 board with 8 vcore phases driving 60A IR3555 powerstages like the Asrock X399 Taichi.
However then you are going to need a decent cool airflow around the vrm area.
Otherwise the powerstage temperature in the vrm will likely heat up too quickly under load,
and at a certain powerstage temperature the system will crash.

5

So yeah long story short.

2990WX build decent X399 board required which are E-ATX.
So you need to be sure your case supports that formfactor.

  • Asus X399 Zenith Extrema Alpha
  • Msi X399 MEG Creation.
6

Is that really true? obviously you want the best VRM possible for that many cores, but are the x399 boards really that bad? What maybe a bit more heat and less efficiency maybe lower OC, does that really matter.

So doesnt really need Gen 2 boards IMO, not that they arent better, just not a major concern. He should have little issue on pretty much any x399 board.

Crucial ECC 2666 (CT16G4WFD8266) Is what I am running 4 sticks of at 2933 easily Honestly this would probably bet your best bet since you are doing workstation type of tasks

7

Nope not all X399 gen1 boards are that bad.
However in a vrm solution with like 8 powerstages 60A powerstages grouped together,
their efficiency will of course drop because of heat production.

I have seen a couple of tests with a 2990WX on several X399 boards.
And the Asrock X399 professional gaming for example which is an 8 phase with IR3555’s.
And that vrm running a stress test went upto 115C and failed eventually.
The Asus X399 Zenith Extreme gen1 did ran the 2990WX without issues stayin below 100C.
This board technically has the same vrm components as the Asrock board, except for the actual heatsink.
But with this board you can also get an active vrm cooling which is kinda needed,
or decent airflow around vrm area of course.

8

On test bench or just no air flow

On my x399m Taichi i have air flow over my vrm and its just a 1950x as vrm heatsinks these days are more like capacitors then anything.

9

Yeah the thing is that all X399 gen1 boards are mainly designed with the 180W tdp chips in mind, so upto like a 2950X 16core cpu.
The 2990WX behemoth wasn’t even out at the time those boards were launched so to speak.
The 2970WX and 2990WX are pretty power hungry in that regards.

10

Wonder if Zen2 will come to x399 boards or if they will release TR4v2 (prob hackable to put new chips on it,but idk about 64 core on x399 gen 1 even with power savings)

11

Well there will likely some new TR cpu’s,
my guess is that those will likely be backwards compatible with current x399 boards.
But i would’n be surprised if we also might get a new chipset series of boards like X499 or so maybe?
With upgraded vrm solutions.
Atleast whenn the new TR cpu’s will also get pci-e 4.0 that is.
Or Threadripper might stick to X399 with just a new revision of boards idk.
Very little info known about this atm.

1 Like
12

@mutation666

But its pretty likely that we are going to see that 16 phase infineon pwm,
used more on boards maybe likely next gen TR boards aswell.
Because that would be an ideal solution for a Threadripper vrm,
driving 16 phases straight from the pwm to 70A Infineon smart powerstages,
or IR3555’s 60A stages.

13

Sorry for the false start. I was hoping to solicit advise on the RAM selection first but didn’t make that clear.

I get excited thinking about building a 32core, 128GB top of the line monster, but as it stands now I have a lot of other projects chomping at the checking account, and I think the practical side of me is winning out. I know that I would find occasion to occupy all those cores and it would be sweet when I did, but 99% of the time 16 is going to be more than I need, same goes for the Ram. For the data processing research, its better, if slightly less fun, to launch some temporary cloud instances anyways. So I’m downgrading the requirements to a 2950x and 64GB ram. I think this also lowers the stakes a lot and it feels like the parts are probably a bit easier to match (VRMs, thermal load, ram timings, etc).

I’ll put together a parts list and post it in a little bit.

14

  • Case

  • CPU

  • Motherboard

    • MSI X399 MEG Creation
    • Thoughts : the Zenith Extreme Alpha costs way to much, this one seems like a heck of a nice board, I have a better upgrade path if gen 3 is compatible or if I try to snag a 2990wx later on, but the MEG might be overkill? not sure here.

  • Ram

    • Crucial 16GB DDR4 SDRAM ECC CT16G4WFD8266
    • Reasoning: suggested earlier in thread, and cheaper than the non ECC “b-die” ram I had been looking at.
    • Question: Will I notice a difference while using the computer between this and faster, non-ecc ram with those swanky heat spreaders?

  • Primary Storage

    • NVME 1TB XPG ADATA
    • Reasoning: Saw a review saying it was just as good as the Samsung?
    • Use: Primary OS, home dir, workspace, scratch

  • Secondary Storage

    • 2TB Sata SSD (ADATA)
    • Use: Steam, downloads, test data sets, windows boot for games and/or VM (FreeNas holds media, archives, and additional space)

  • PSU

    • Seasonic 80+Ti 1000W
    • reasoning: this just seems to be what everyone gets, 1000w gives me room to grow.

  • CPU thermal

    • Enermax Litech TR4 II 240 [ELC-LTTRTO240-TBP]((https://www.amazon.com/s?k=ELC-LTTRTO240-TBP&ref=nb_sb_noss)
    • AIO is necessary for this case. Want 280mm, but there’s a good chance it won’t work with eATX in this case (no hard confirmation), 240 should be good for 2950x right? Ideally I’d get both and take one back, if that where possible.

  • GPU

    • Gigabyte Vega 64 GV-RXVEGA64GAMING
    • Linux compatibility is important for me so I am going AMD. This card seems to be a good price point. If this card is not advisable, I wouldn’t go much more 400 without just wanting to jump up to a Radeon VII.

  • Fans

15

Primary Use: linux software development workstation

(In case you didn’t read the wall-o-text original post…)

development means front end & back end, polyglot languages, databases for testing and evaluation, lots of docker, R&D of all sorts, data processing

Gaming? I play games a few hours a week, and mostly play indie games off steam, but would like the option to play an A list title on occasion.

16

im doing a TR build too.

im going to be using pop_os, my use case is a bit like your but more towards devops (some polyglot development and a lot of kubernetes docker use) i also do some gaming, my plan is to use VFIO to passtrough a dedicated GPU to a windows machine with 4/6 cores and 16gb of ram.

1 Like
17

what are your lessons learned with k8s? I’m just getting into k8s on bare metal. One project uses k8s in azure which has worked surprisngly well. I’m tinkering with local/DIY hosting because the production k8s cluster in azure costs a lot and another project that has its own devops team gets a much better deal out of a DIY k8s solution.

I am thinking of setting up a mini-cluster of two 32c threadrippers for quasi-permanent experimentation but not sure yet. Limited to 256gb on each machine is kinda meh though.

1 Like
18

Epyc 32 core zen 2 out yet?

19

i was one of the ones crazy enough to deploy a prod kubernetes cluster to bare metal very early on pre 1.0 release. it was a nightmare due to the lack of documentation and the complexities of kube in metal back them.

configuring flannel and keep it running without failing was a challenge specially because we had the master in AWS (HA setup) and all the kubelets were bare metal.

those days were very tough on the team as it took us a while to make it work and keep it reliable but it was way faster to work with kube than say swarm or mesos/marathon or and nomad was to green back them.

now days most of my cluster are in GCP or AWS (i have zero experience with Azure) GCP is very awesome and depending on your workflow if you have stateless jobs you run in kubernetes running a cluster with preemptible instances is super cheap, especially when paired with autoscaling.

for baremetal i used to have a very complex ansible deployment to manage everything but now days i do everything with k8s-tew

https://darxkies.github.io/k8s-tew/_build/html/index.html

it does HA, deploys calico, manages ingress with nginx and also certmanager, uses Ceph for storage, manages backup also uses metalLB comunications are encrypted by default.

if you already know kubernetes by heart and dont need to actually learn all the individual components (kubelet, api-* controllers-* etc.) this is one of the best ways i found so far to manage metal.

my biggest metal deployment was like 100 nodes, and in cloud i barely get that big in a single cluster as i use multiple for HA and disaster avoidance.

i would be more than happy to help out more if you need more info or jump in on a call some day, i do mentoring every one in a while.

2 Likes
20

I’m the k8s subject matter “expert” at work, I was the main dev-ops guy for a long time. I’ve been running it in AWS via Kops and Ansible since maybe 1.5. Kops is pretty painless to get started. There have been some hiccups with specific things with Kops but I don’t have much to compare it too. EKS wasn’t out yet or I probably would have considered it. I’ve never run k8s on bare metal.

One thing I am interested in is leveraging Kubernetes for development workflows, currently developers work with local docker-compose setups depending on what they are doing, and it is getting increasingly unwieldy.

3 Likes