HPC build

Recently there’s a flood of QS EPYC 9654s available on ebay:
ebay 9654 QS

I’m thinking of building an HPC workstation. I already have a Fractal Torrent case, nvme drives and a 1200W PSU.

Maybe start with 12x 32GB DDR5 RDIMM.

Would need to pick up a MB as well and thinking about Gigabyte MZ33-AR1 for Turin compatibility when the 9655’s hits ebay in 18 months.

My main use case is virtual Win 10 machines running under Ubuntu 24.04. I have a Looking Glass setup with a RTX4090 and a 7950X3D that works just fine, the only problem being I can’t remote into my individual Win10 machines this way. I also just need more cores.

Does this sound reasonable or should I stay away from the QS versions? I can’t justify paying USD10K for a CPU.

1 Like

Will work exactly how you want
just add a TPM for the Windows compatibility

1 Like

Will do.

I went ahead and ordered the MZ33-AR1, EPYC 9654 QS and 12x 32GB 4800MHz DDR5 ECC. I requested a quote for everything from an ebay seller with good reputation.

I’ll read up on the cooling and make sure to get a TPM as well.
@JayVenturi Did you find a more reliable source for coolers?

Shit
forgot to mention the PCIe and RAM clearance is tight on that board

The SuperMicro H13SSL clear GPU’s better

sorry about that

Ah. Good catch. I can still cancel the order. I want to put an RTX5090 in there so should I just get the H13SSL then?

Absolutely
The Gigabyte board is great for compute nodes
Huge RAM, super fast networking, and or PCIe interface cards to GPU node.

The SuperMicro is the way to go for Full size GPU acceleration on the board.
You’ll lose alot of slots with the card overhang, but PCIe 5.0 risers are not advised.

On the gigabyte board, you cannot have a full length PCIe card without interference with the RAM DIMMs. EPYC requires 1 DIMM per memory channel for optimal performance, but cards will actually interfere with the latches of empty slots.

you can still use that gigabyte motherboard, but you’ll need a PCIe Retimer Card + Cables + Risers.

something like this

https://www.aliexpress.com/item/1005006042593063.html

But that is only 4.0 Speed, and x8, plus you need a place to put the GPUs

or get a kit like this

Which is 5.0 but significantly more expensive

But like TryTwiceMedia said here

if you want a no faff approach, the Supermicro motherboard is pretty good, for up to 3 5090s ( assuming they can all get fresh air lmao)

The Gigabyte one is good if you want 24 Dimms and 12/14 NVMES straight away
IMO it is the superior motherboard, but it needs alot of adapters to configure it the way you want it (multiple GPUs), but with a bigger/custom enclousure they will be well cooled
e.g. something like this

Hopefully that helps,

Personally im still deciding between going
W880 and/or ZEN5 or Turin via H13SSL or MZ33-AR0
Im leaning more towards W880 for ST(especially because do i really need more than 256GB RAM :thinking:, and its more of an AIO Home server build than a HPC build) but Genoa / Turin is especially the 9115 CPU looks really good.

I think it just ultimately comes down to RAM + CPU cost, DDR5 RDIMMS are expensive atm, but that will lower with time

1 Like

Cool. Thanks for the detailed response!

I’m only looking at a single GPU for now.

The MZ33-AR0 and H133SL don’t mention Turin compatibility but you’re saying they will support those?

Seller got back with recommendation to go with MZ33-CP1:

For a single GPU I could have it in the third pcie slot, that would clear the RAM.

What is that huge cutout for though?
Edit: That would be an OCP slot which I gleaned from reading the manual. Is there a way to mount nvme drives there?

1 Like

OCP Mezzanine cards, mostly networking cards.

1 Like

That’s a nice motherboard, it looks like it uses pretty much all of the 128 5.0 lanes genoa/Turin has

I think it depends on the revision of the motherboard you get, the supermicro one, you need to get the latest revision
I think it’s the same with the Gigabyte Rev 3.0 iirc

1 Like

Cooling suggestion:

I think I’m getting ready to order!

only tricky part is finding an OCP compatible case

If not too late you may consider a gigabyte MZ3-LM0, and leave you with options down the road.

I made mine fit in a medium-small of the shelf normal case

J

2 Likes

I have this:

Woudn’t the I/O shield fit?

That’s what I was wondering/thinking
I haven’t found pics of the IO shield
but all it would do mechanically is retain the card in 1 axis
The OCP card itself is supported by the mounting rails on the board in the other 2 axis

yeah i thought the motherboards I/O shield would just have a cutout for the OCP, so you wouldn’t need a special case

Wow. Definitely will consider this. It’s not too late at all.

No problem leaving one of the sockets unpopulated?

To elaborate, I’ve built and deployed half a dozen EPYC 9004’s in the past year. Never seriously considered that board but have used ASRock Rack, Gigabyte, and Supermicro.
My preference is the reverse of alphabetical, but it’s all application dependent.

1 Like

based on the manual
Screenshot 2025-01-10 at 14.58.17

there should be a cutout in the IO shield

similar to this MZ32-AR0
Screenshot 2025-01-10 at 15.00.21