Dual PSU safety on ROMED8-2T with 4x 3090

Looking for guidance before powering on a new build. My last board died during a dual PSU install and I want to make sure I’m not repeating the mistake.

Previous build (dead):

Threadripper 3960X / Gigabyte TRX40 AORUS Master running 3x RTX 3090 on a single EVGA P2 1200W — worked great for weeks. Added a 4th 3090 with an Add2PSU relay (Corsair AX1600i primary + P2 1200W secondary), GPUs split across both PSUs via 8-pin PCIe power (2x GPUs + motherboard powered by AX1600i, 2x GPU powered by P2 1200w). GPUs connected via unpowered x16 riser cables on an open-air rack.

Board immediately wouldn’t power on — no standby LED, completely dead. Tried both PSUs individually with no GPUs attached, nothing. Both PSUs test fine (12.0V and 12.3V on 12V rail respectively). Never determined root cause.

New build:

Component Spec
Board ASRock Rack ROMED8-2T
CPU AMD EPYC 7532 (200W)
GPU (current) 4x RTX 3090 — 2x ASUS Strix (390W), 1x Zotac (350W), 1x PNY (390W)
GPU (planned) Expanding to 6-7x 3090 (board has 3x PCIe 4.0 x16 + 3x PCIe 4.0 x8)
PSU AX1600i (12.0V) + EVGA P2 1200W (12.3V) via Add2PSU
Risers Unpowered x16 PCIe 4.0 cables
Chassis Open-air mining rack
Workload vLLM inference (pipeline parallelism), not training

The concern:

With unpowered x16 risers, the GPU’s 8-pin 12V and PCIe slot 12V are connected internally on the GPU PCB. When GPUs are split across two PSUs, the 0.3V difference creates a current path through the GPU and motherboard traces. Mining rigs avoid this with powered x1-to-x16 risers that break the slot power path, but those kill bandwidth.

As I add more GPUs, the dual PSU split becomes unavoidable — and I’d eventually need a third PSU for 6-7 cards. Right now I could set power limits to fit in the budget of just the AX1600i, but this doesn’t solve the issue long-term.

Questions:

  1. Is the 12V back-feed through the GPU PCB an actual damage risk, or is the current too small to matter with quality PSUs?

  2. Anyone running a similar multi-GPU workstation/inference setup with dual PSUs and GPUs in full x16 slots?

  3. Better approach for powering 6-7x 3090 on this board?

Thank you for any help and experience!

1 Like

Option 1: Convert all PCI slots to MCIO, check my posts for pictures on how to build large 10+ GPU MCIO based rigs. Your PCIE slot + cable distance will dictate if you need redrivers/retimers

I have used your exact board with a chinese PCIE switch board to attach 4x 3090 with their own dedicated power supply separate from the main chassis (video on STH forums).

Option 2: Use a power distribution board (again, i have posts with pics) that delivers up to 1000 amps of 12v power across a dozen+ of 12vhpwr cables to your GPUs and motherboard.

Good luck! And dont forget to enable P2P DMA between your consumer cards for tensor parallel bandwidth performance boost!

Edit: Old cabling reference diagram for earlier rig design for funsies:

3 Likes

That’s an insane setup, amazing build on your other post too!

Do you have specific product links/recommendations for the power distribution board? I think at most I plan to scale up to 7 gpus - that’s the most my board supports without bifurcation and running all cards at full x16 speeds.

My biggest concern is that I fried something or will fry something again by drawing 75w from the pcie slot on one 12v rail and drawing additional power from the cards ports from a different 12v rail. Sounds like the power distribution board would be the answer

I have actually cooked a pair of 3090s with power flowing the wrong way through them when the system was “shut off” rofl not a motherboard but same rules apply I suppose. You might have lucked out losing only a mobo in todays market. This was back in the early days of my experimenting with a workstation xeon board and external $600 3090’s on risers I got off the junk section of ebay. It was a setback, but fairly minor. One was repaired (cooked a vram chip) the other was toast.

If you have time, money, and space: I would recommend one of the customizable chinese consumer GPU cases that will hold 8-10 cards. Thats the route I ultimately went after spending more than a year custom engineering my own aluminum monstrousity.

Minimum quantity is 1, price starts around 650. Comes with the power distribution board and cables, but not the CRPS power supplies (those are 20 bucks a pop for 1200w on ebay and you can buy 6 of them for the price of one normal PSU) The chassis I got is for my specific propriatary motherboard (20x MCIO 8i connections) for 8 GPUs with 6 NVME x4 and a 200gig mellanox nic at x8.

The shipping (sea freight from china) is very affordable, takes about a month, delivered to your door with no fuss any taxes/tariffs prepaid. These guys are pro’s they sell to big businesses and know how to get stuff done. I trust their engineering expertise way more than the local CDW or WWT rep.

Also budget for replacing the fans with less insane ones, they come with like 8000 rpm leaf blowers. You can drop in the arctic server replacements with a little jiggling.

If you cant wait, dont have the space, there is alot of back and forth in the PCIE mega-thread with pictures and links to MCIO risers, cables, and adapter boards. I post and stan the ones I use that work fine at 50-75cm without any AER problems.

2 Likes

There’s an error in your cabling diagram: the bottom Add2PSU Joiner 24pin ATX input is shown coming from a PSU3 8pin CPU/PCIE block when it should be coming from PSU3 “ATX 24p.”

I would imagine there are many errors in my diagrams, they were mostly created for half planning half documentation in visio that changed over time with lots of copy paste mistakes and probably went through a hundred revisions over 16ish months: