Quad R9700's on AM4 Shenanigans

High-Performance Computing
, , , ,
83

If you get really stuck, Deepseek v4.1 and opencode really helped me out with computer janitoring :smiley:

84

are you actually getting pcie4 x16 bandwidth for each of the 4 R9700 behind PLX88096 or is it pcie4 x8 bandwidth

i’m thinking of adding a PLX88048 as I only want to split one pcie4 x16 into two pcie4 x16/x16. I wonder if it will work on a trx40 motherboard where 2 of my R9700 are getting fulll pcie4 x16 each and two will be behind the motherboard’s third pcie4 x16 slot , split through a PLX88048/PLX88096 into two pcie4 x16/x16 , 1 for each number 3 and number 4 R9700

If anyone has done this , please let do share if you are able to get bar and p2p working this way

85

Unfortunately, switch board is unable to negotiate Gen4 speeds with my (asus & sapphire) R9700, and ‘tech support’s thrown white flag and said to return it.
Trying to accomplish return&refund with Alibaba, that despite supplier agreement, doesn’t really work.

86

For those contemplating getting this board, it wasn’t able to negotiate Gen4 speeds with my R9700 GPUs.
It might be just specific board issue or board design is not good enough to handle Gen4 x16 with AMD GPUs. Vendor says, they test it with Nivida GPUs.
So this is failed experiment in this case, but still a data point for those who are considering this switch board.

87

thanks for info, stepping up to QFN and upgrading to 4 cards will have to wait though.

88

theoretically its best to have all cards behind the switch so that they can talk to eachother at full x16 speed in P2P without crossing into the root complex, however thats also the only part i haven’t been able to make work. i have full 30-ish gbyte/sec unidirectionally but since it still goes through the root complex due to some pcie flags i havent quite figured out yet(likely due to qemu) effectively its still capped a 32GB/s for the combined cluster+a latency hit.

In practice though, for inference it doesn’t seem to matter that much

1 Like
89

right , so despite having the switch , all your 4 cards connected via plx88096 go through root complex , but does rocm detected p2p okay ?

i would love to see your better bench results on radiance 1.7.0 and qfn

90

Yeah p2p works DMA works just still stuck at 32GB/s due to the root complex round trip, so far still can’t seem to fix it, luckily its not really a bottleneck for inference

1 Like