I’m upgrading my existing P620 threadripper servers (the one with a kludgy 3d printed SAS expansion JBOD for Truenas, and the second AI server I crammed a pair of A100s into) and looking to combine them into a single EPYC 9555 based platform. The problem is, I can’t seem to find a motherboard that has sufficient slots and DIMMs.
For a case, I was going to go with the HL15 Beast (since it has the clearance for the fan duct for the A100s). But I can’t seem to find any motherboard that will support 12 DIMMS at 5600+, support for 2xA100 (2 slots each, need to be next to each other for NVLink) for AI, a single A4000 or T4 for Plex, and a total of 24 SATA/SAS ports (could be combo of motherboard ports and an x8 slot for an HBA. So basically need 6 slots.
Where are all the good Turin boards? I just need something like the Asus Sage WRX90E, but for Epyc, not TR Pro. All the EPYC boards I’m finding seem kinda crippled by comparison.
I could drop down to 8 DIMM channels vs 12, but not ideal.
If I read this correctly, this supermicro board has 8 SATA ports, and 2x MCIO8 (which could be used for 16 SAS/SATA drives? so total of 24? If so, I wouldn’t need an HBA and this could work..?
Unfortunately that’s a dual socket board, and I just need single socket. While you can populate only one socket in those boards, pretty sure you lose a number of slots?
They should have direct access to their own sata controller or IOMMU group. There should be an option within the bios to chose sata for that MCIO header. However those ports probably aren’t meant to be bootable. You will want your primary boot partition to run off of the onboard sata ports or nvme of your choosing. I don’t own that board so i can’t really say for sure how it works.
If you’ve got room in your case, there’s some cabled PCIe breakout boards you can stick a couple HBAs in, they aren’t too expensive and if you’re using older PCIe 2.0/3.0 cards it’ll be fine - I have four P4800x Optanes hooked up to a dual SlimSAS to quad PCIe breakout board from C-Payne and it’s been working pretty well so far.
If the H14SSL is a bit too pricey, there’s a rev 2.0 version of the H13SSL which also supports Turin - although you’re limited to 400W max and there’s apparently a few percent performance difference, but they do work (posting this from a 9575F in a H13SSL).
The H13/H14 is probably your best bet for a single CPU board IF you want all 12 memory channels, the three full x16 PCIe slots are all spaced for dual slot cards, and if you’re mad enough, you could have an additional eight HBAs each with a dedicated x4 PCIe connection if you fully breakout all the MCIOs and m.2 connectors… (and there’s a -NT version with dual 10Gbe in addition to everything else)
Otherwise the Asrock GENOAD8X-2T has 8 PCIe slots (7x16 and 1x8) so that’s your three GPUs and a HBA direct on the board, catch is, only 8 memory channels instead of 12. Never personally used one though, so only going off the marketing material.
You are correct, but there are very few single Turin or Genoa motherboards, and the footprint of the dual is the same as the footprint of a single in this most comparisons. Turin and Genoa (depending on which processor you end up with) can be up to 500W on each CPU socket.
I just built a dual Turin / Epyc 9755 for a friend of mine, that was all of 500w x2 just for the CPUs.