I’m new to the forum but wanted to ask about some items I saw popping up on eBay. I’m seeing several vendors listing GH200 modules and baseboards (Grace Hopper “Superchips” from NVIDIA) for prices much cheaper than conventional PCIe H200 variants.
I was curious if anyone has looked into making a custom HPC machine from these sorts of components, and if so: how? The base development boards seem to have SXM5 sockets which mate with the GH200 SXM5 modules, but beyond that the configuration of the machines is unclear (how does one install an operating system, or even power the thing).
I’m not sure if posting eBay links is allowed or not, so I’ll err on the side of caution, but you can find these listings by simply searching for “NVIDIA GH200”.
If anyone has insight on this, I’d love to learn more!
There is a company that builds workstations from these: GPTshop.ai. I have no experience with their systems, and am not affiliated with them in anyway - but maybe you could figure out what sorts of components you would need to retrofit a GH200 module into a workstation by looking at pictures of their systems.
Cool, thanks for the info. It looks like they’re using some kind of motherboard that is different than the baseboards I’m seeing on eBay. I wonder where one might source something like that.
I’ve been down this exact rabbit hole for a few weeks and haven’t bought
yet. What I’ve worked out, in case it saves you the same digging.
They aren’t SXM5. The Grace Hopper module mates through a pair of long
high-density mezzanine connectors, not an SXM socket, so an SXM5 carrier
won’t take one and vice versa. Worth checking against listing photos before
buying anything on that assumption.
Power is 54V, not 12V. NVIDIA’s MGX firmware docs split the superchip
SKUs by rail — 0215 is the 54V branch, 0206 is 12V — so the SKU in the
listing photo tells you which one you’re looking at. David Ng’s build used
a 3000W 230V-to-48V supply, and programmable TDP across CPU, GPU and memory
together lands somewhere between 1000W and 2000W.
The thing that actually blocks you is the BMC. These carriers have no
BMC and no coin cell, and on the ones I’ve seen photos of, no slot for a
management card either. That looks architectural rather than stripped —
OCP’s DC-SCM spec splits the CPU/memory complex from a separate module
carrying the BMC and root of trust, and this is that split. Which leaves an
open question I can’t answer from photos: whether the host UEFI lives on
the carrier or on the management card that didn’t ship with it. If it’s on
the card, the board is scrap until you find one, and I haven’t seen a
single one sold separately.
Cooling is a machining job. Ng went from eight Sunon dual-fan modules
to four Arctic Liquid Freezer III 420 AIOs on CNC-milled copper adapter
plates, because nothing off the shelf mates to a superchip cold plate.
On GPTshop — agreed, their board is a different animal, and I’d assume it’s
an NVIDIA MGX-family mainboard bought as a complete unit rather than
anything you can source in ones.
Three things I’d genuinely like answers to, if anyone here has one on a
bench:
Has anyone actually powered one of these outside its original chassis?
Ng’s write-up is the only one I can find, and a sample size of one is
thin ground for spending four figures.
Ng mentions a “special power board” his system needed, with roughly a
30-pin connector doing the sequencing. Does anyone know what that board
is, or what the connector is? It’s the part I can’t identify at all, and
nothing in the listings I’ve seen includes one.
There’s a small finned heatsink near the MCIO cluster on these carriers.
I’d guess it’s the HPM FPGA and that it’s an Intel Cyclone 10 GX, on the
strength of that part turning up on Supermicro’s G1SMH — but that’s
inference, not a part number. Anyone confirm?
If I do buy and it comes up, I’ll write up the whole bring-up.