Workstation recommendation

Hardware Hub Build a PC
,
1

Hi,

I’ve outgrown my current workstation (amd 5950x, 64gb of ram, rtx 3090) and am looking to upgrade to something more capable. At this point in time, the 32 threads are just enough but i’m mainly hit by ram capacity, storage limitations and pcie lanes.

What i am looking for is:

  • atleast 1 10gbit nic but i would really like 1 or 2 sfp28 or sfp+ ports.
  • enough pci lanes to create a zfspool with 4 m.2 nvme or something similar
  • support for 2 gpus, mainly for ML but also for vfio
  • atleast 32 threads with current single thread performance

I’ve been chechecking out some ebay listings on epyc and supermicro cpu/mobo combos but am kind of in the middle as they’re shipped from china and do still cost about 1300$ (with 128gb ddr4 in an 8chan config). The epyc 7502p has 64 threads but it’s as good as always slower as the more efficient 7950x 32 thread cpu.

Are there any x670e motherboards or which would support my kind of “requirements” if i would skip on the network cards and go with a 10gbit built-in rj45 port? And can i get away by using the internal 7950x it’s igpu and pass through my rtx 3090 with looking glass? that would also free up some the pcie lane requirements :smile:

I’ve heard that the memory controller on the 7 series ryzen processors is kind of finicky whenever you use all 4 ram slots and expect it to hit 6000mhz so i will most probably op then for a 96GB 2dimm 6000mhz kit if that exists :stuck_out_tongue: and upgrade to 192GB and settle with slower speeds whenever i outgrow it.

So TL;DR, should I go for a brand spanking new system (if it can handle the storage needs) or go the slightly neckbeard way and buy some epyc in china :sweat_smile:

2

If EPYC is too slow, consider Threadripper (Pro). AMD will launch their latest TR platform hopefully in the not to distant future (timeslot: your guess is as good as mine :stuck_out_tongue: ) based on the latest gen EPYC CPU’s. If you don’t want to wait that long, consider the EPYC 7H12 which has a base clock of 2.6 and a boost of 3.3 GHz.

As for costs: I recently had a great deal for an EPYC Naples base system (mainboard, CPU, RAM and cooler) that cost me less the 600€ via Aliexpress. Of course, YMMV! Here’s what I got:

Mainboard, CPU and 4x Samsung 32GB ECC RDIMM RAM from this seller:
https://www.aliexpress.com/item/1005005010935691.html

2U cooler with fan:
https://www.aliexpress.com/item/1005004556694860.html

Not included in the above price:

Single 10Gb SFP+ NIC:
https://www.aliexpress.com/item/1005005541605742.html

Dual 10Gb SFP+ NIC:
https://www.aliexpress.com/item/1005004706712159.html
I ordered both brackets, on arrival it was obvious they forgot the high bracket, so I messaged them and they send the high brackets (I ordered a pair of their cards) very swiftly at no extra charge and no fuss (like I had on another seller that send me the wrong product and refused to rectify), so that’s worth a recommendation for me.

As for your NVMe suggestion: I have this card, running in a bifurcated 16x slot, with 4 NVMe drives in RAID6. Man that’s fast (over my HDD based RAID 6) :+1:
https://www.aliexpress.com/item/1005005186941984.html

Downside of course, is extended delivery times: it’s not next-day delivery I’m afraid :sloth:

3

support for 2 gpus, mainly for ML but also for vfio

Wendell rolls with Threadripper for VFIO and lanes.

4

Threadripper Pro, is the fence straddler, of serious IPC, along with them PCIe niceties
Epyc can have [some] aggressive specs, but is more dialed back… +VCache drop-in upgrade?

…Of course we still await, for some announcement / release news

5

If you can deal with slightly lower per-thread performance, the Epyc 7443P for ~1200 from China seems like an absolute banger of a deal. 15ish% slower on single threaded and 25ish% faster multithreaded vs your 5950X plus that sweet sweet memory bandwidth and 128 lanes of Gen 4 PCIE goodness. I bought a Zen 2 Epyc from tugm4470 on ebay, and he’s the real deal. Mention Servethehome and he’ll toss in free Fedex shipping (from China to Chicago in under a week) for me. There’s a thread in the deals section over in the STH forums of folks who have successfully bought used EPYCs from China-based sellers on ebay.

If you need the single-threaded performance PLUS PCIE lanes (and you need the PCIE lanes if you want 2x GPUs plus 4x NVME) PLUS memory, then it’s really just threadripper pro or the new W2XXX Intel cpus, both of which will cost a fair bit more than the 7443P.

6

It looks like you are more interested in AMD (TR Pro or Epyc), but this is what I recently invested in for a workstation.

I have multiple machines, with my previous top system being an AMD R9-5950X, ASUS ROG X570, 128GB DDR4, RTX-3080, Samsung 980 Pro.
I needed lots more memory for the work that I am doing with 3D software and Unreal Engine 5.
The new workstation system is:

Intel Xeon W7-2495X 24C 48T, Noctua NH-U14S-4677, ASUS Pro WS W790 ACE, 512GB DDR5 ECC RDIMM, ASUS ROG STRIX RTX-3090, WD SN850X 4TB plus additional drives, Corsair 5000D Airflow White, Corsair HX1200 Platinum, etc.

I can get 4.8GHz all-core and a CPU-Z score of ~800 single-thread and ~24000 multi-thread.
The system just flies on the workloads that I give it.
My plans are to upgrade to 2TB of memory once the 256GB RDIMMs start to show up in Canada.

I looked at TR Pro systems before I went Intel, and the TR Pro came in at typically $20,000 CAD while this Intel system was only $13,000 CAD.
TR Pro is crazy prices right now in Canada, for what is essentially last gen specs (DDR4 PCIe4 etc).
The box itself was about $10,000 CAD. The price above includes monitor, kbd and mouse, speakers, UPS, etc.

The ASUS ACE can manage up to 2TB of memory, and easily handle 2x PCIe x16 GPUs.
The ACE doesn’t have WiFi or Thunderbolt and the NVMe are PCIe Gen 4 is the only real limits.
The W2400 processor has 64 PCIe 5 lanes, DDR5 ECC support, Quad memory channels, is unlocked, etc.

I’m not a fanboi for either team, I have both AMD and Intel systems here, I just go for what I feel is the best deal at the time for my needs.
This W2400 system just wastes my 5950X system for performance, and even in multi-core it beats the i9-13900K by 50% faster.

2 Likes
7

Is it possible to split your workload across more than one machine?

1 Like
8

I currently expanded my machine by offloading some of the work to old laptops and intel nucs which are configured in a proxmox cluster. The main constraint there is my network setup and my truenas system. It’s becoming too much of a burden to maintain and schedule everything in the right places :sweat_smile: and i want to simplify my setup. Going full 10gbit or even fiber will only enlarge the overall bill i think. I can only offload some minimal docker containers and backend apis.

I’ve been looking at the Intel setups as well and they are strangely enough becoming more and more the cost optimised option. I do am afraid of all the performance penalties they’re getting with the mitigations for some of the CVE’s of the past year.

I fear i’m waiting for the next release :stuck_out_tongue: I hate this game of “lets wait as the next big shiny thing is just around the corner”

9

I do wonder how much i’m going to be impacted if i would switch to a x8-x8 setup for the gpu’s (pcie 5) and if i would have enough pcie lanes for the 4 nvme’s. Am i correct if i say that the 7000series (amd) has 24pcie gen5 lanes which can all be used (without including the 4 pcie lanes to the chipset)?

10

This you can test yourself by simply enforcing an x8 split. If you are running X570 and occupying the two main x16 slots, chances are you are already running x8. In my experience, expect single digit perf degradation, max 5%, probably 1-2% if any.

Depends on the motherboard. There are both 790 motherboards and X670E motherboards with x8+x8 + 5+ NVMe drives. Putting a gen 4 drive in RAID0 is kinda pointless though as they are already pushing data faster than most of your components can handle, better with a ZPool if redundancy (aka uptime) is important for you.

Yes, Zen 4 has 28 5.0 lanes in total and four are bound to chipset.

However, do not forget about the chipset lanes, which will vary but usually amount to something like 12-20 gen 4 or even gen 3 lanes. In theory this is more than enough lanes.

Also… I have no idea why we are not already at PCIe lanes only on the chipsets. SATA can go bleep itself on modern high end workstations, replace those with one more m.2 instead.

11

Yeah the nvme setup is not about speed (although it kind of is as i’m doing it with nvme) but i want to get an openzfs z1 going and offload anything that pushes me over the 6tb pool to my nas. It’s mainly going to be used as scratch storage for backup verification and integrity tests (schrödingers backups), feeding data to the gpus and holding about 350gb of git projects.

12

Then why not just go dual 8TB m.2 in mirror mode? If you already got 4090 money spending 2x$800 on something like the Corsair MP600 PRO NH 8TB should not break your bank, sure 8TB is not the best price/perf and costs $400 more than 4x fast 4TB storage devices like the $270 WD Black SN850X 4TB, but we are also talking a 3% increase of total costs if not less, sooo… :man_shrugging:

1 Like
13

I was stuck in that cycle as well on my new workstation. Do I buy now or wait for the next Threadripper?
I priced out the current TR Pro 5000 series in Canada, and Intel W2400 easily beat them in price and performance, and I needed the workstation now instead of six+ months from now, so I just went Intel.
The big issue here in Canada as well is that it takes four to six months after release before we see the workstation components in the stores, so that would have pushed me into at least early to mid 2024.
Also, if I didn’t need the 512GB+ of memory, then I would have simply gone with a newer consumer platform system like i9-13900K or R9-7950X.

14

Would love that to be the case here. For a single low end threadripper cpu and board (no RAM or anything else) I can build 1.5x am5 systems where I live.

Also, scaling out to N Systems will get further than a single expensive threadripper box.

Caveats as above re additional complexity, workload distribution etc of course.

15

Threadripper Pro is crazy here in Canada. I guess we pay a premium for location.
5955WX 16-Core is going for $1500-$2000, 5965WX 24-Core is going for $3999, and the prices just get crazier the higher the core count.
64-Core is $9999. Plus 15% taxes where I live.
These are just processor without the motherboard. WRX80 motherboards here are $1400-$1900.
The prices seem to be going up with inflation over the past year.

1 Like
16

Yup not too far off the pricing here. I guess those who need it will pay but it’s a pretty steep ask. Way more expensive than the old intel HEDT platforms.

How things have changed since threadripper 1000 series.

17

I recently priced out the Intel W2400/W3400 vs Threadripper Pro for a new workstation.
The TRPro system was $20,000 and the Intel system was $13,000.
So I went with Intel. I’m an equal platform user, I just go with the best deal. I have both Intel and AMD systems.
The 24-Core Intel outperforms the 24-Core TRPro in multi-thread, and is on par with the 32-Core TRPro in multi-thread.
Single-thread the Intel easily beats the TRPro. I’m not a power user though, I just got it for the 512GB of memory.
I’m not 100% happy with the ASUS Pro WS W790 motherboard, but that is another story, it’s “just ok”.

1 Like
18

id get a 32core genoa qs for like 800$ on ebay, and asrock/supermicro atx motherboard and call it a day.
gonna be hard to beat that value.
Actually planning to do this myself. already use a server board for my current workstation, i dont miss the lack of soundcard(use a usb dac anyways) and the lack of usb ports can be solved with a pcie card or usb hub.

19

I am documenting my genoa build here:
Amd sp5 genoa motherboards from the usa now available on newegg - Hardware / Motherboards - Level1Techs Forums

I just posted the geekbench results from within windows 11. I have:
supermicro sp5 motherboard eatx with dual 10g
96GB ddr5-4800, 9 slots still available
rtx 4070 (only covers 2 slots)
dual 118gb optane + 1TBhdd,

Lost somewhere in the house that is intended for this computer is 64GB ddr5, a tpm module and 512gb ssd. Though I have been working with stable diffusion and collecting models. I may need more storage.

2 Likes
20

Just be aware from what I hear a lot(?) of OEM EPYC processors are vendor fused. Or were with prior gen?

If it came out of a dell or whatever it maybe won’t work in a third party board.

Or so I heard.

ref:
https://www.reddit.com/r/Amd/comments/ipz2xt/amd_epyc_cpus_getting_locked_to_computer_vendor/