Asus ProArt X870-E - bifurcation troubleshooting

Hi! Wasn’t sure if I should post in the general bifurcation thread, but I thought for a specific problem creating a separate thread might be ok.

I have been on a bit of local AI craze, and ended up with 4x R9700. (3 ASRock, 1 XFX)
The motherboard I am using is Asus ProArt X870-E (BIOS 2503 / AGESA 1.3.0.1d), CPU is Ryzen 9950X.

Unfortunately my current platform is AM5 based, so getting even 4x4 CPU-connected gen5 lanes is a bit of a hassle. I already had a 3 GPU setup, with one GPU connected via M.2_2 slot, using ADT-Link adapter, and that has worked fine.

Using ADT-LINK R34 PCIe→4xM.2 card and a couple of K43 adapters, the setup I have been testing now looks like this:

Ryzen root complex
│
├─ 00:01.1 ── PCIEX16(G5)_1
│              LnkCap: x8
│              actual downstream link: x4
│                │ │
│                │ └─ R34 port 1 → R9700
│                │                 BDF 03:00.0
│                └─ R34 port 2 → R9700 (NOT DETECTED)
├─ 00:01.2 ── M.2_1
│                 └─ Samsung 9100 PRO
│
├─ 00:01.3 ── PCIEX16(G5)_2
│              LnkCap: x4
│                 └─ R9700
│                    BDF 07:00.0
│
└─ 00:01.4 ── M.2_2
               LnkCap: x4
                  └─ K43 → R9700
                           BDF 0a:00.0

RESULT:
    3 × R9700 detected

DESIRED PEG CONFIGURATION:
x4 + x4  |  x4  |  x4
   ↑         ↑      ↑
PCIEX16_1   SLOT2   M.2_2

But with AUTO mode the second card on the R34A does not get detected.
Conversely, when I change the SLOT1 settings to “RAID”, only the GPU on the first R34A slot is detected, and I lose the other 2. This seems like the RAID mode does not really respect the fact that I have devices on the other slots and makes the first slot x4x4x4x4.

AUTO:
00:01.1 -> PCIEX16(G5)_1, LnkCap x8
00:01.3 -> PCIEX16(G5)_2, LnkCap x4
00:01.4 -> M.2_2,         LnkCap x4

RAID:
00:01.1 -> LnkCap x4
00:01.3 -> disappears
00:01.4 -> disappears

This confused me a bit, as according to the Asus docs the following situations should be supported (albeit more specifically in case of SSD adapter bifurcation):

Official ProArt X870E-CREATOR WIFI manual:

                       Situation 1     Situation 2     Situation 3
PCIEX16(G5)_1          4 × x4          2 × x4          2 × x4
PCIEX16(G5)_2          0               2 × x4          "2 (x4)"*
PCIEX16(G4)            1 × x4          1 × x4          1 × x4

* table wording in ASUS manual as published

(Maybe using M.2_2 is complicating thing here?)

I tried different combinations of settings but I could not get all of the GPUs working. The obfuscated naming of the options in the BIOS does not help either.

I was wondering if anyone here would have some insight or tips on what I could look into. The fact that even in RAID mode which “consumes” all of the lanes I could still see only one of the GPUs is a bit concerning. EDIT: In RAID mode I do see the two GPUs now, must have been some cabling issue.
Maybe this motherboard simply does not support such bifurcation? I guess I could try relocating one of the adapters to another slot just to see, perhaps the physical allocation of lanes in this scenario (x8 → x4x4) would be split differently (not slot 1 and 2, but eg. slot 1 and 3), alas I have not had much experience with this topic so I might be talking nonsense.

A potential workaround I might test is removing the SSD from slot M.2_1 and connecting the GPU there, although if possible I would rather avoid that.
I also know that a lot of people here have had much better luck with using switches, but with my current setup I would consider that a last resort due to extra cost and overall “ergonomics” of the setup, though I am aware that even if I made this topology work I would still have the problem of quite limited bandwidth between the cards.

Regardless, I would really appreciate your input and responses.
I have been at it for hours and it is almost 3am now so I forgive me the sloppiness if I forgot to include some relevant info. I am more than happy to provide more diagnostics and test out your ideas.

1 Like

Looking into the manual the PCIe 5.0 slot 1 can only do x16 or x8 with no other options listed for “normal” use. It looks like you would not be able to split that x8 up without a PCIe switch of some kind, the x4x4x4x4 mode seems to only be for NVME card x16 slot Bifurcation, looks like a limitation of CPU/ motherboard switching?

Why can’t you do:

PCIe x16 5.0 Slot 1 R9700 (x8)

PCIe x16 5.0 Slot 2 R9700 (x4 because it shares with M.2_2)

PCIe from NVME adapter M.2_2 R9700 (5.0 x4 because it shares bandwidth from x16 5.0 slot 2)

PCIe from NVME adapter M.2_1 R9700 (5.0 x4, I assume your current SSD is in this slot now)

Either M.2_3 or M.2_4 or SATA for your system drive (4.0 x4 or SATA speeds)

That would keep all GPUs connected through the CPU at 5.0 speeds assuming the adapters are capable, the only immediate major problem I can see is PCIe power limitations from the motherboard for the GPUs.

Otherwise the 4.0 x16 (x4 speed) slot on the bottom of the MB is also an option for an R9700, though it goes through the chipset, unsure if that really matters or not.

Looking into the manual the PCIe 5.0 slot 1 can only do x16 or x8 with no other options

I thought it might work based on the following figure from the manual:

Which explicitly lists x4 + x4 config on the _1 slot. It is SSD specific though. I was hopeful it would not make much difference..

Why can’t you do:

Yes, I considered that in the OP, it should work technically, it just would not be the “perfect” layout I was hoping for.

Otherwise the 4.0 x16 (x4 speed) slot on the bottom of the MB is also an option

I tried it before, unfortunately speed aside the P2P connection did not work well.

Unfortunately it looks like the board can only do x8x4x4 despite the manual and a nice trawl through here: [Motherboard] Compatibility of PCIE bifurcation between Hyper M.2 series Cards and Add-On Graphic Cards

I did find 2 very useful diagrams that explain the PCIE layout for the board though:


and

Thanks for the diagrams, I will chew on them a bit, I am at work at the moment.

One correction to my OP is that in the RAID mode 2 of the R34A adapter connected GPUs are properly enumerating, so bifurcation works and adapter is OK, but I am getting x4x4x4x4 instead of x4x4 + x4 + x4 layout I wanted based on the table from the previous post.

One more update, as I have observed something interesting.
@roiter - it seems that something alongside the split I was interested in might be technically possible, but there are specific implementation blocks.

TL;DR:
Through some reverse engineering and experimentation, I was able to establish that the PCIEX16_1 CAN be split into two logical (x4+x4 channels), while supporting devices on PCIEX16_2 and M.2_2, provided that the device in PCIEX16_2 does not populate all of the pins on the said slot. Reverse engineering pointed to specific gpios which are checked during initialization. I checked it experimentally by replacing the GPU in PCIEX16_2 with an x1 WIFI adapter. Most notably it seems like the state of gpio 27 might participate in determining the bifurcation mode.

So close, but so far away haha.

As mentioned above, I had Sol 6.1 running on this for most of the day, it did some interesting analysis of the BIOS firmware. I will post some of the findings below, in case someone would find it relevant for their use-case.

===============================

ProArt-X870E-CREATOR-WIFI-ASUS-2503.CAP was extracted with UEFIExtract, its ASUS/AMD forms decoded with IFRExtractor-RS, and relevant PEI executables traced using objdump and Ghidra. This identified a Granite Ridge board-policy module that reads ASUS Setup directly, combines the slot settings with GPIO detection inputs, and modifies temporary PCIe engine descriptors, clock policy and GPIO40.

Comparing saved variables, PCI trees and AMD-driver GPIO dumps confirmed that the same RAID settings can produce different connector availability depending on the installed card. Replacing the directly inserted slot2 R9700 with an AX210 produced (x4+x4)|x1|x4 across slot1 / slot2 / M.2_2, using three GPUs plus Wi-Fi. All four PEG roots advertised maximum width x4.

Item Value / identifier Significance
Principal PEIM B2DDC6BE-D2EE-4EF1-992C-2CD695115DAF Direct Setup consumer; constructs Granite Ridge lane policy
Slot1 setting Setup[0xE4], question 0x384 0 Auto, 2 RAID, 4 GPU+M.2
Slot2 setting Setup[0x16E], question 0x386 0 Auto, 1 RAID
Hidden PBS lane setting AMD_PBS_SETUP[0x18] Remained 0 with byte-identical PBS snapshots across all tested topologies; not a live topology readout
GPIO27 GPU: high; AX210: low Selects between the relevant RAID policy branches
GPIO90 Low in all captures Selects the x8/x4/x4 Auto branch
GPIO40 GPU RAID: output low; AX210 RAID: input Mode-dependent board control; physical mux/net identity unresolved
Internal mode5 E4=2, 16E=1, GPIO27 low Four x4 engines; GPIO40 input; strongly corroborated by successful capture
Internal mode7 E4=2, GPIO27 high Four x4 descriptors; GPIO40 output low; only two PEG roots enumerated in GPU captures
CPM $A26 Lane ranges 0–3, 4–7, 8–11, 12–15 Explicit four-way split handed to AMD initialization
Successful roots 00:01.1/.3/.4/.5 Two R34 GPUs, slot2 AX210, M.2_2 GPU; actual widths x4/x4/x1/x4

Offsets exclude the four-byte efivarfs attribute prefix. Internal mode numbers are selector results, not Setup enum values. Mode attribution is supported by post-boot captures rather than a direct PEI execution trace.

===============================

I think this snippet summarizes the logic:

// raw GPIO levels; reconstructed from assembly
if (Setup[0xe4] == 2) {
    if (GetGPIO(27) != 0)
        mode = 7;
    else
        mode = (Setup[0x16e] == 1) ? 5 : 4;
}

So long story short, there is a possibility that installing the card in slot _2 via something that would reduce the physically connected pins, maybe PCIe x4 → M.2 → PCIe x16 adapter, would make this whole thing work. I am (softly) considering tweaking the BIOS to see if this mode selection could be altered for my use-case.

Having said that, at this point it this is more of a curiosity for me, as having confirmed that the R43 should technically work with 4 cards, this would make much more sense in terms of physical layout and eventual mounting.

Since I have already sunk some time and effort into it, if anyone has some things they are curious about I should be able to provide/extract more info from the BIOS analysis, or try some configuration out.

I would be happy to hear your thoughts on that.

Since I am already multiposting I will share that I did finally get all 4 GPUS to finally show up.

Apparently the trick was to replace the ASRock card in the PCIEX16_2 with the XFX one. Based on the analysis in the previous post, seems like there might be some vendor specific behavior with particular GPIO states? I wonder if that has anything to do with the specific compatibility with the “HYPER M.2 X16” ASUS cards, claimed by ASUS.

 Device 0 [AMD Ryzen 9 9950X 16-Core Processor] Integrated GPU RX: N/A TX: N/A
 GPU 600MHz  MEM 3000MHz  TEMP  43°C   CPU-FAN   POW   0 W
 GPU[|||||||||||                           28%] MEM[|                        0.975Gi/47.994Gi]

 Device 1 [AMD Radeon AI PRO R9700]             PCIe GEN 5@ 4x RX: N/A TX: N/A
 GPU 0MHz    MEM 96MHz    TEMP  27°C  FAN  19%   POW   1 / 300 W
 GPU[                               0%(eff 0%)] MEM[                         0.056Gi/31.859Gi]

 Device 2 [AMD Radeon AI PRO R9700]             PCIe GEN 5@ 4x RX: N/A TX: N/A
 GPU 0MHz    MEM 96MHz    TEMP  31°C  FAN  14%   POW   1 / 300 W
 GPU[                               0%(eff 0%)] MEM[                         0.056Gi/31.859Gi]

 Device 3 [AMD Radeon AI PRO R9700]             PCIe GEN 5@ 4x RX: N/A TX: N/A
 GPU 0MHz    MEM 96MHz    TEMP  28°C  FAN  19%   POW   1 / 300 W
 GPU[                               0%(eff 0%)] MEM[                         0.056Gi/31.859Gi]

 Device 4 [AMD Radeon AI PRO R9700]             PCIe GEN 5@ 4x RX: N/A TX: N/A
 GPU 0MHz    MEM 96MHz    TEMP  28°C  FAN  19%   POW   1 / 300 W
 GPU[                               0%(eff 0%)] MEM[                         0.056Gi/31.859Gi]

Perhaps someone who actually knows how PCIe works under the hood would be able to explain this behavior.
What remains to be seen is the actual stability of this setup during use, and across boots. Somehow I wouldn’t be surprised if there were some timing-related transient states at play.

1 Like

That’s awesome news. Fingers crossed on the stability front :crossed_fingers:

I’d focus on the PCIe lane allocation first, especially the interaction between PCIEX16_1 and M.2_2. The fact that AUTO gives you x4 on the first R34 port but doesn’t enumerate the second device suggests the platform may not be exposing the x4x4 bifurcation you need through that adapter. I’d try the adapter in another CPU-connected slot and test without the M.2_2 device/SSD temporarily to see how the BIOS reallocates the lanes.

Thank you so much for your post. I am planning to use the same board for 4x 5060 ti and could not find reassuring info online till now.

My plan is:

x8 on pcie1

x4 on pcie2

x4 on M.2_1

x4 on M.2_2

Storage to be connected at m.2_3 or 4.

I am aiming for all gpus to have direct connection to cpu. Please let me know if you have tested this or if I should anticipate any surprises :slight_smile:

Have not tried that particular configuration but I talked to a person with the same mobo who was using M.2_1, PCIE1, PCIE2 for GPU.
Meanwhile my original config used M.2_2, PCIE1, PCIE2. M.2_1 was used by storage.
I dont see a clear reason why this would not work. There is always a chance that some weirdness could happen with the Auto settings (as made apparent by my experience above, the BIOS does some runtime checks in certain cases to determine the mode), but I think that is more of a normal usecase, so most likely you would be ok I think.

That being said, I will try to move all of my GPUs on the PCIE1, because it would allow me a better physical layout to stack all of my cards vertically (||||) within the case, I just need to figure out how to do the mount. Maybe something 3D printed could work, of course with proper filament to handle some heat.

Your setup technically gains a bit more connectivity on one of the cards being x8 compared to my approach, but I think P2P communication, which is I think most relevant for me, would still be limited be other cards being only x4, so I opted for physically “easier” layout.

Kapton tape, cover up the addition pins on the card’s PCI connector. That way you skip an unnecessary adapter.

In terms of breaking out the PCIe slot to 4 GPUs, I’d start looking at cards like these: Buy PCIe x16 Gen 5 ReDriver to MCIO 74P Dual Port Add and cable up the GPUs. Assuming you’ve got the 4x4 bifurcation worked out, it’s the simplest solution.

Thanks. Really appreciate the information and the quick response.

I am planning 3 horizontal (2 ~ 2.5 slots cards) + one vertical (the case has a dedicated vertical mount bracket in the front side away from the pcie case slots) big card one a massive case (o11 evo xl) so as the plan is a closed case I want to reduce additional cables as much as I can.