Designing a four-GPU Threadripper Pro System

Hardware Build a PC
, ,
1

Howdy, I’m designing a workstation for general GPU compute and would love feedback and answers to a few tricky questions from the folks here.

My primary constraints are that I want a 3975x Threaripper Pro and support for up to 4x GPUs ~300W TDP w/ 2x 8 pin PCIE connectors each (e.g., NVIDIA A100).

I’m trying to structure everything else (case, cooling, mobo, PSU, etc.) around this goal.

CPU:

3975x 32-core Threaripper Pro

GPUs:

Support for up to 4x GPUs ~300W TDP w/ 2x 8 pin PCIE connectors each (e.g., NVIDIA A100).

Cooling

Custom water cooling loop for CPU + 4x GPUs from EKWB w/ 2x 420mm EK PE radiators + dual d5 pump. I will be using the EK Quick Disconnect system as I will be swapping out GPUs frequently.

Storage

Intel Optane P5800X 800GB boot drive + 2TB Firecuda 530 NVME.

PSU

Obviously need 1600W+ PSU (though going to 2000W on a 220V is on the table).

The choice of PSU is complicated by the choice of motherboard. See below.

Case

Leaning towards Define 7 XL, but depends on choice of Mobo + PSU (i.e., if I need two PSUs) to ensure everything will fit.

I’m a little nervous about fitting the 2x 420mm radiators + 4 GPUs + an EATX motherboard.

Motherboard

Here’s where things get complicated when taken in consideration with the available PSUs.

Asus PRO WS WRX80E-SAGE SE WIFI

  • Explicitly states it supports 4x GPUs.
  • Requires 2x 8pin ATX + 1x 8pin PCIE + 2x 6pin PCIE connectors

asus_mobo
asus_mobo874×1083 178 KB

This means that I need 5x 8pin power connectors for the motherboard alone. The GPUs would need another 8x 8pin connectors, meaning I would need 13 total 8pin power connections. As far as I know, there aren’t any PSUs with this many connectors, and I’d need two. Corsair AX1600i has 10 and Super Flower Leadex Platinum has 11.

As I’ll describe below, the ASUS board is the only WRX80 board that has these two 6pin connectors. Does anyone know why ASUS has these two additional 6pin PCIE power connectors?

Gigabyte WRX80-SU8-IPMI

  • Requires 3x 8pin ATX connectors

From a connector stand point, this motherboard would work with the Super Flower Leadex.

The Gigabyte documentation doesn’t mention multi-GPU support. And I’ve seen several forum posts about issues with getting 4 GPUs to work (something in the UEFI settings).

This mobo doesn’t seem to have stock anywhere either.

Supermicro M12SWA-TF

  • Requires 3x 8pin ATX connectors

Not a lot of good information out there about this board. I’ve been eagerly waiting for @wendell to review.

The thermal design seems lackluster (e.g., VRM heatsink) and dependent on a very high airflow chassis (i.e., depends on living in a Supermicro chassis).

Summary

So my main conundrum is the choice of Mobo + PSU + Case.

The ASUS seems like it would require 2x PSUs if those 2x 6 pin connectors are required. This would also mean a case big enough to hold two PSUs.

If I go with the Gigabyte or Supermicro, I can get the Super Flower Leadex and everything should fit in the Define 7 XL. But I have reservations about the 4x GPU support on the Gigabyte, and the overall lack of reviews on the Supermicro as well as the seemingly lackluster thermals.

I’d appreciate any feedback or thoughts, especially about the motherboard/PSU/case choices.

1 Like
2

the GPUs can consume 75w through the slot. They’re necessary.

This is what it was meant for. You’ll have a great time.

The GB and Supermicro arent going to magically make this thing use less power and there’s not 150w+ of heating coils on the asus board… for 4+ gpus it is the only choice imho. The GB board might be ok with 4 gpus with aux power. The asus board would kinda tolerate more than 4. Asus might be thinking of DPU cards or other high-wattage cards other than GPUs as bar space and bootstrapping start to become a concern.

If you really are going for the A100s they don’t use quite as much power through the slot. I would recommend them for that reason alone.

My newegg order from March was only finally cancelled outright a few weeks ago. Is there another source that has them readily?

Anyway, AX1600i will work:

image
image1024×768 71.6 KB

SO there are 10 c+2/4+4 for the cpu. Think of it as 8+2. 2 for the cpu.

That leaves 8 wires each of which have two 6+2 pcie power connectors. Or if you get the EPS12v A100 (that is how the v100 is) it is just one EPS12v per gpu. So each GPU is “fine” with EPS12v (400w) or “fine” with two 6+2 connections (from different cables, 6+2 is absolute max 300w, 150w “design” wattage).

So that means you can use the little 2x PCIE to 1x EPS pigtails the A100 comes with right? Nope! That many connectors and the crappiness of the pig tail mean that, ideally, you mix runs with the pig tail. Each card is fed by two cables back to the PSU. So don’t do that. Just get the direct EPS to EPS cable. Each connector adds resistance and resistance is cumulative. It might work but not a best practice and not worth the headache.

So you have 4 “home runs” of EPS12v, same as cpu connection, and now you have used 6 of your 10 4-pin on the AX1600i.

You will use 2 more for the motherboard, and I don’t think you really need both E and F, they would be fine off a single cable with “only” 4 GPUs attached.

So in terms of cabling, you have some spare. In terms of wattage, however, you have very little spare.

My experience with the A100 is that it is ~250w nominal, 300w peak, so 1600i with “just” 4 gpus and the 350w cpu+board is probably fine.

figure 50w-75w for cooling, fans, etc. The nvme isn’t going to use much. Peak 30w for both probably.

image

Yes, you’ll have to order some “extra” cpu cables. But trust me, this is the best option.

4 Likes
3

Thanks for your help @wendell!

Yeah, I was very suspicious of the lack of these connectors on the Gigabyte and Supermicro boards. I can’t imagine ASUS would just add them for fun. So ASUS board it is. It’s also the only one that has a native U.2 port for the Optane drive.

As you mentioned in your tweet from a while back, Newegg is shipping the Threadripper Pro + M12SWA-TF bundles and I probably would have got one of those if I ultimately decided to get the Supermicro board. Otherwise, I haven’t seen stock anywhere else.

Wow, I completely overlooked the fact that all 10 connectors on the AX1600i can support EPS12v connectors. I’m surprised Corsair doesn’t document/mention this more explicitly given other 1600W PSUs I’ve found only have 2x EPS12v connectors with the rest being 6+2 PCIE.

So you’re absolutely right for cards like the A100 or A6000, I can just use 1-to-1 EPS12v cables.

So to summarize:

  • 2x EPS12v connectors to mobo for CPU (A&B)
  • 1x EPS12v connector to single PCIE 6+2 on mobo (C)
  • 4x EPS12v connectors to GPUs
  • 1x EPS12v connector to dual PCIE 6+2 connector on mobo (E&F)

That means I’d only be using 8 out of 10 of the 8pin connectors on the AX1600i?

How about for something like a 3090 with the 12 pin connector that usually requires two dedicated PCIE 6+2 connectors? Would single EPS to two 6+2 PCIE connectors work “fine”?

Obviously with 1600W, I wouldn’t be able to do 4x 3090s, but in my line of work its normal to mix and match GPUs.

Yeah, I’m definitely skating on the edge. Luckily, I’m not usually pushing the CPU + all four GPUs to the max at once, so I feel like I should be okay.

What about the case? Do you think I’ll be able to fit 4x GPUs + 2x 420mm radiators + 2x pumps + reservoir + 2CPU/4GPU EK QuickDisconnect manifold in the Define 7 XL? I won’t have any HDDs or optical drives to worry about.

1 Like
4

3090 can blow the remaining power budget if it is 3x a100 + 3090. probably doable, but cutting it close. you I would not do a single EPS cable since it is 400w limit and 3090 peak draw is ~450w. it is assuming it can get 225w per 6+2 connector, with beyond 300w melting the connector, mind you, so yeah. razor’s edge.

pcie4 u.2 cables are flaky right now. I wouldn’t be surprised if you have to use a pcie adapter card rather than a u.2 cable.

I think that is right.

the define xl can do what you want in terms of cooling. I have two v100s and a 2080ti and a PBO threadripper and 4 u.2 drives and 1600w rosewill (dont recommend) psu running ~1300 watts all the time and dual 360s were fine. They the thiccccccc bitspower though. one 420 + 1 360 would be fine, Im sure, but if the manual says you can do 2x 420mm rads, you for sure can.

I have the same big manifold in my build now, and it is just peachy for this kind of thing.

4 Likes
5

I have been running a quite similar system for a few months. The specifications are as follows:

  • CPU: 3995WX 64-core Threadripper Pro (same TDP as the 3975x)
  • GPUs: 4x 3090 (ASUS TURBO-RTX3090-24G) with a 350w TDP
  • Cooling: A Coolermaster Tripple Radiator AIO (not sure which one).
  • Storage: 2x 2TB Samsung 980 PRO
  • PSU: 2x 1200W Silverstone SST-ST1200-PTS
  • Case: Lian Li PC-O11
  • Motherboard: Asus PRO WS WRX80E-SAGE SE WIFI

My use case for this machine is machine learning and scientific computing workloads where it is not uncommon for the 4 GPUs to be at 100% load.
So far the Asus motherboard has worked like a charm with the 4 GPUs. Especially the BMC is nice to do administrative tasks remotely.
The Lian Li case fits all the hardware comfortably. The triple rad is mounted at the top of the case and it looks like there is still room for a dual rad in the front of the case.

I could not get my hands on a power supply with more than 1200W so I went for a dual PSU setup. I distributed the hardware over the PSUs as follows:
psu

Maybe knowing that this is working like a charm for me gives you some piece of mind regarding the motherboard. If you are interested in the Lian Li case I could make some pictures so you can get an idea of how much space there is left in the case.

Also keep in mind that according to NVIDIA’s spec sheets the 3090s have a 350W TDP which is 50W less than the 400W TDP of the A100s you are looking at. So our builds are not 100% comparable.

I hope this information was somewhat useful to you.

2 Likes
6

First thing I would do is get rid of any mention of TLC flash memory and go straight for the 970 Pro 1TB. TLC NAND with an SLC cache will not perform as well for consistent sustained write. Sustained write on the 970 Pro 1TB is MUCH better than any Gen4 NVMe right now. Keep your Optane as that’s really good, but you may want to use MLC NVMes.

1 Like
7

It does indeed! Thanks for sharing your experience.

What do you plug this beast into? You’re nearly at enough wattage that you could trip the breaker on a standard 15A US breaker by itself, let alone if anything else is on the circuit.

Luckily the TDP on the PCIE variant of the 80GB A100 is only 300W and 250W on the 40GB. You’re probably thinking of the SXM4 A100. So I should be safe from tripping any breakers (for now).

It’s a funny state of affairs when you’re seriously considering pulling a dedicated 220V circuit just for a workstation :slight_smile:

8

I am from the EU and I plug the system in a 16A 230V breaker, so plenty of power there. There is even another system with 4 2080 TIs on the same breaker and it has not given me trouble so far (fingers crossed :zipper_mouth_face:).

2 Likes
9

Thanks @Jinior for posting the specifications of your system. The information has been very helpful.

I intend to build one with Asus PRO WS WRX80E-SAGE SE WIFI and Lian-Li O11 Dynamic XL.

Would I require the Lian-Li O11DXL-2 Motherboard Extension Panel to fit the motherboard. https://www.amazon.co.uk/Lian-Li-O11DXL-2-Motherboard-Extension-Dynamic/dp/B0823N89DG

10

Hi @KinuthiaGichuhi - yes, you would need that Extension Panel. Sadly, Lian-Li could not supply this when I was building my system, so I made my own.

PS: I have the case set up horizontal, with 4 old Titan GPUs and an EVGA 2000W PSU. Best feature is that the PSU is out of the way (“under” the motherboard) leaving lots of space on all sides of the motherboard. Also, case is designed to take up to 3 PSUs.

1 Like
11

Thanks so much @nnunn for that confirmation.

Kindly just one more clarification.

After putting the Extension Panel would it possible to still fit a 360mm radiator?

12

Thanks @jhemstad for the elaborate discussion and summary on powering Asus PRO WS WRX80E-SAGE SE WIFI for 4-GPU support. Apologies for my basic question, newbie in PC builds.

I still cannot figure out about how to connect the socket C. PCIE_8PIN_PWR despite googling. Seems would require an extension/converter cable. If so which one would you recommend?

Your advise will be highly appreciated.

13

Hi KinuthiaGichuhi,
That connection (C) in the motherboard manual [“PCIE_8PIN_PWR”] just takes a standard 8pin GPU cable.

“PCIE_8PIN_PWR” means PCI-Express 8-pin power.

Note: It should clip on easily, just like plugging in a graphics card.

1 Like
14

Thanks @nnunn for your quick response and and advise. Highly appreciated.

1 Like
15

Did @OP build this monster system? I’d be interested to see some benchmarks!

1 Like
16

This topic was automatically closed 273 days after the last reply. New replies are no longer allowed.