AM4 motherboard with quad R9700's behind plx88096 ROMBAR Woes

High-Performance Computing
, , , ,
1

EDIT:

So i managed to get my system working with all 4 GPU’s behind the pcie switch, however i still can’t get the damn P2P to work nice inside my VM’s, i’m suspecting it has to do with PCIE atomics not working properly for guest vm’s or something of that nature, can anyone more experienced in this than me chip-in?

So i recently got my PLX88096 chip in the mail, and have used it to install 4 R9700 AI’s onto the system, however it seems my poor prosumer grade AM4 X570-D4U-2L2T/BCM motherboard has had enough of my shenanigans and just doesnt have the BAR space to deal with it

My onboard 10Gbe also stopped working, but i suspect this is just a side effect of the lack of BAR space

I’ve added the Dmesg logs and my kernel opts

Please obi-wan janitobi’s, you guys are my only hope:

Kernel Args:

GRUB_CMDLINE_LINUX_DEFAULT=“quiet amd_iommu=on iommu=pt pcie_aspm=off amdgpu.ras_enable=0 pci=realloc=on amdgpu.runpm=0 amdgpu.gpu_recovery=1 pcie_acs_override=downstream,multifunction, default_hugepagesz=1G hugepagesz=1G hugepages=108 transparent_hugepage=always”

DMESG: See attachment

dmesg.txt (46.6 KB)

For your viewing pleasure(or distaste, depending on tolerance):

image
image1599×899 265 KB

2

It WORKS!!!, it was Legacy CSM support being enabled, i have a new problem now though, even though everything is now properly showing inside the VM P2P is still disabled in AMD-SMI topology:

ACCESS TABLE:
0000:01:00.0 0000:02:00.0 0000:03:00.0 0000:04:00.0
0000:01:00.0 ENABLED DISABLED DISABLED DISABLED
0000:02:00.0 DISABLED ENABLED DISABLED DISABLED
0000:03:00.0 DISABLED DISABLED ENABLED DISABLED
0000:04:00.0 DISABLED DISABLED DISABLED ENABLED
WEIGHT TABLE:

this is inside the VM that’s being passed through all these devices

3

So i managed to get my system working with all 4 GPU’s behind the pcie switch passed through to my VM, however i still can’t get the damn P2P to work nice inside my VM’s, i’m suspecting it has to do with PCIE atomics not working properly for guest vm’s or something of that nature, can anyone more experienced in this than me chip-in?