Killed a PSU with my 6x3090 Threadripper build - fluke or systemic?

From April of this year until a week ago - for approximately 6 months - I was running the following system:

Asus Pro WS WRX90E-SAGE SE
AMD Threadripper Pro 7965 WX
256GB (32x8) DDR5 6000 (EXPO)
2x EVGA 3090 FTW3 Ultra
2x Gigabyte 3090
Storage adapter card and 6x 16TB Seagate EXOS SATA Drives
Corsair HX1500i (Black label, pre-ATX 3.0 certification)
Corsair HX1000i (Blue label)

I had the HX1500i connected as the primary PSU and the HX1000i connected as secondary PSU as described in the motherboard’s manual. The motherboard has two 8-pin EPS12V connectors and two 8-pin PCIe connectors for the CPU, with adapters to turn the PCIe connectors into EPS connectors for use with dual PSUs. It also has two 8-pin PCIe connectors by the ATX connector for helping provide additional slot power in multi-GPU configurations. To support the dual-PSU configuration, the motherboard also includes an ATX Y-cable that feeds all of the conductors from the primary PSU and jumps the required conductors for synchronously starting the second PSU.

The PSUs each had the following connected:

HX1500i:
ATX connector
2x EPS12V CPU (one converted from PCIe on the motherboard end)
2x 8-pin PCIe motherboard connectors
2x Gigabyte 3090 (2x8-pin PCIe each)

HX1000i:
2x EPS12V CPU (one converted from PCIe on the motherboard end)
2x EVGA 3090 FTW3 Ultra (3x 8-pin PCIe power connectors served by two cables)
6x Seagate EXOS SATA Drives
4x 120mm fan, 1x140mm fan

The two things I know are questionable are running unequal PSUs and not using three cables to serve the three connectors on the three-connector 3090s. But I figured that if two PSU connectors was enough for Corsair’s 600W 12VHPWR cable, this should suffice - and I could always power limit the GPUs if I encountered a problem. But I never did encounter a problem, not until I ‘upgraded’ the system.

Last week, I added a 3090 FE, a 3090 Ti FE, and replaced the HX1000i with an HX1500i (ATX 3.0 certified).

The new configuration:

HX1500i (old):
ATX connector
2x EPS12V CPU (one converted from PCIe on the motherboard end)
1x 8-pin PCIe motherboard connector
2x Gigabyte 3090 (2x8-pin PCIe each)
3090 FE (2x8-pin PCIe)

HX1500i (new):
2x EPS12V CPU (one converted from PCIe on the motherboard end)
1x 8-pin PCIe motherboard connector
2x EVGA 3090 FTW3 Ultra (3x 8-pin PCIe power connectors served by two cables)
1x 3090 Ti FE (used included 2x PSU to 1x 12VHPWR cable)
6x Seagate EXOS SATA Drives
5x 120mm fan, 1x140mm fan

Basically, the 3090 Ti FE was added to the secondary PSU, the 3090 FE added to the primary PSU, and one of the motherboard PCIe power connectors was moved to the secondary PSU due to a shortage of connectors on the PSU end. But at least the PSUs were now matched…mostly.

Each of the 3090s was power limited to 300W. I left the 3090 Ti on its default limit of 450W.

This setup seemed to be working well enough at first. I performed some LLM quantization in parallel, each job having a GPU assigned to it, and they all completed successfully. However, when I was later using the machine for inference, it suddenly died on me. After some troubleshooting, which included disconnecting all the GPUs and switching the motherboard to its single-PSU configuration, I determined that the HX1500i (old) was dead. The motherboard lights up when the PSU is switched on, but when you try to start the machine, the LED on the back of the PSU briefly flashes red and nothing happens on the motherboard. HX1500i (new) is currently running the system with no GPUs attached.

My theory is that it was able to handle the quantization well because it’s a more-or-less constant load. But inference is characterized by having all the GPUs work on generating a response all at once, then falling back to idle as the user reads and types his own response, and then suddenly needing full power all at once again once the user hits send - which is exactly when the system died.

On that note, it just occurred to me that for most of the 6 months I had been doing inference without row-splitting/tensor parallelism. Prior to that, whole layers were stored on each GPU, meaning that GPU0 would spin up to service the request, then it would progress through the layers on GPU1, and so on. Whereas with tensor parallelism, each GPU receives part of each layer, and each GPU can be doing work in parallel. Trying to think back and looking at the commit history of the frontend I’m using, it looks like that change would have been pulled in a few weeks ago - but before the ‘upgrade’. So it’s possible my setup may already have been a ticking time bomb with the change in usage profile.

I’m running both PSUs on the same 120v 20A circuit, so I would have expected the breaker to trip if the PSUs actually tried to pull their full 3000W. Failing that, I would have expected over-current protection on one of the PSUs to kick in. Not permanent death.

What’s my best way forward, at this point? I don’t want to just toss in a replacement ATX 3.0-compliant HX1500i if my setup is going to kill it, too. On the other hand, the better support for power excursions could be exactly hat I need. This could have been a bad unit, but I don’t know how to be sure. No PSU calculator I’ve found has my motherboard or CPU or supports more than 2 GPUs or 2 PSUs. I’m open to the idea of connecting a third PSU, if necessary. I was also already thinking of changing to a 240V circuit to supply the PSUs.

1 Like

There are several people reporting on localllama about their 6x3090 setups for LLM inference, many of them on AMD builds. I have never heard of such an issue, so I would think it’s been an unlucky outlier and the next PSU shouldn’t cause issues.

1 Like

So

As long there was a a gap on the board electric and the video card electric with no cross pollination you should be ok as long as common grounds did not mix.

So 24 pins, 8 pins for board belong to one psu. You should have ben fine.

on 240…

you could go silverstone hela R 2050w: this psu can go on 120-240v and on 240v should be fine, -or 2x of the 2050w on the 110-120v and get 1500-1650w out of each one

Seasonic 2200w, this psu ONLY works on 240v.

Maybe that was my sin. The diagram in the manual says to connect the motherboard’s additional slot power PCIe connectors to PSU1, but I connected one to PSU2 because the alternative would have been splitting a graphics card across two PSUs, which gave me a worse feeling.

I’m leaning heavily towards adding a third PSU at this point. This way I will be able to service all three connectors on the EVGA 3090s with their own cables, and won’t have to connect the two slot power connectors to two different PSUs.

Edit to add: I’m pretty sure every PSU I’ve mentioned in this post supports 240v. My plan was just to purchase the appropriate replacement cables to support a 240V receptacle.

Yeah, that seems needed to me as well. This doesn’t look to me like an overcurrent or overpower thing either and I agree the PSU1-PSU2 cross in the mobo PCIes is pretty sus.

This topic was automatically closed 273 days after the last reply. New replies are no longer allowed.