I had some 3070’s laying around and thought I’ll throw them all onto some beefy machine to have some mild AI fun, but… 14 GPUs connected results in infinite postcode loop.
Removing any (tried different ones to be sure) of the 7 PCIE 3.0 x8x8 raisers and everything starts up fine with remaining 12 GPUs. No hardware errors for days.
Setup is
ASUS Pro WS WRX80E-SAGE SE WIFI (bios updated to latest)
AMD Ryzen Threadripper PRO 3975WX 32-Core
256GB DDR4 3200
3 installed nvme drives filling all onboard M.2 slots.
12…14 nvidia 3070s from various manufacturers.
Is 12 GPUs somehow a limit for this board? All these 14 GPUs were running on single Ryzen 5 3500X 6-Core with x1 raisers on X570 board “just fine” …in their previous life.
16x7 slot + 4x 3 nvme + 4 for chipset/io is exactly 128 pice lanes available. Am I miscounting something? Is there something specific I should disable/modify on bios to help things along? Manual doesn’t mention any warnings or special conditions for any slots, like some boards would borrow x8 lanes from one of the x16 slots for extra nvme sometimes.
I would not like to give up on the nvme drives, this sacrifice is pretty much only obvious thing I haven’t tried yet. Not sure if I would be happy or sad, if it actually worked this way.
Otherwise I disabled every extra device in BIOS I could, extra USB controllers, sata controllers, wifi, bluetooth, sound, even the onboard BMC. Dual 10G NIC’s are still on for obvious reasons.
TL;DR; Has anybody gotten this board running with 2 bifurcated GPUs in each available pcie slot? Any hints on what I have to trade in for the last 2 to start working?
Thanks for making it that far, whichever way,
Tom
Update: 13 GPUs also boots. Does not matter which slots are used.
Sorry, do not know about this motherboard specifically… But, I noticed, as a general trend, disabling something in BIOS does not work well for PCIE connected devices; the device has to be physically (mechanically or electrically) severed from the bus somehow.
I’m somewhat sure it’s some sort of UEFI resource accounting conflict not physical connection issue, just disappointed to see it on a board that’s pretty much bragging with all the available resources.
Only shared resources manual mentions are used between M.2 and U.2 slots. No mention of the 7 x16 slots sharing any settings with anything.
Some people here are going with 5 x4x4x4x4 bifurcated nvme boards on the same motherboard, so pure number of pcie devices seems hardly an issue. Then again GPUs are composite video/audio devices
Well, famously GPUs start drawing juice, when you actually give them something to do. just posting they likely don’t take more than 20…50W. I know for sure, because ‘in their previous life’ most of them were behind 1500W cheapo cheftec PSU, but got their power caps lowered accordingly before starting the payload. And it was all fine.
But since you asked, now I have 2000W evga gold (CPU + 7 gpu’s) + cheapos 1500 W Cheftec (5 gpus ) + 1000W MSI ( 2 gpus ) PSUs, I initially prepped for 14x 270W but most of the cards are actually capped at 220W by default, only 3 come with better chunks of metal on them and are capped at 270W.
I believe this is plenty of headroom for everything that’s going on. And there’s plenty going on. Would not feel proud about posting the pics
I am in a similar boat as you. Using TRPRO 5995WX and Asus WRX80e Sage II. I want to connect 8 4090s and bought this PCIE bifurcation card JMT-Expansion-PCIe-Bifurcation. I slotted that in the last PCIE Slot (7th slot) but PC won’t boot (q code 94 which is PCIE BUS Enumeration). I removed the adapter, went to bios and adjusted bifurcation setting to x8 x8 for that slot but still stuck on q code 94. I haven’t tried any other PCIE Slot yet but the fact that you are able to do only 12 makes me believe that the last slot maybe cannot handle bifurcation? Other possibility is that my adapter is messed up. Could you please link your riser adapter so maybe I can order and try that out?
Update: Switched the adapter to the second PCIE slot lo and behold, bifurcation works. I suspect the last slot is not capable of bifurcation. Maybe a bios update is needed to fix this. So my hunch is correct. The reason why 12 GPUs work for you and not 14 is cause the last slot cannot accept bifurcation for some reason that only Asus knows.
Update on that situation: I’m able to do 13!
And it doesn’t matter which half or which bifurcated slot I use, so it’s definitely nothing about physical connections but yes, all about the accounting.
I also made a video about POST code loop, but didn’t get a chance to transcribe it yet.
that is verifiably wrong assumption, I can do 12 no matter what slots I use. Actually slots 7,6,5 are all behind pcie redrivers, so they should all behave the same.
My 13 config in fact is currently using only half of the slot 1 and slot 7 works just fine.
I have bunch of these pcie 3.0 raisers riser.maxcloudon.com/en/
But from other sources I hear these are one of the few that seem to work for pcie 4.0, have not tried myself. Key point here is, that these have active redrivers, which seems to be only way extending pcie 4.0 signal reliably. c-payne.com/collections/pcie-gen4-redrivers
like mentioned the slots 7,6,5 already have redriver chips on the motherboard and there’s bunch of advanced settings I haven’t looked into about amplification and such, so it might be possible to just boost the signal using these settings and go with passive adapters, but again, this seems like a task for lab tech, not home enthusiast, since “looks like working” and “confirmed signal quality” are very different categories for me.
Hey, I am unable to use Slot 6 & 7. Haven’t tried 5 yet but I’m not too knowledgeable in this matter. Anything separate do I need to configure in order to make bifurcation work ok those slots?
Not sure, might be your bifurcation raiser is somehow bitchy about the redriver chips. My raisers are pretty dumb ones and just worked without any special effort beyond just selecting the pcie slot options.
I’m in the same boat, attempting to extend the GPU count beyond 7 on this board. Any updates on resolving the issues above? Any other suggestions on which bifurcators to use, active vs passive?
Will appreciate any input. Anything is better than buying and experimenting one by one, and failing…
