So, last week I dropped another 1,4 Yuks on a second R9700 Pro (Gigabyte rev1.1) to become adjunct to my XFS R9700 Pro.
Spent a day or so just getting everything installed, and mostly fighting fan profiles (figured most of that out with the help of Peter Sulyok’s smfc).
What a strange experience to actually get two R9700’s in a x8/x8 config, only to realize that the software just wasn’t there just yet.
I thought, well, yeah: I could just flip a few kernel args, and I could activate P2P. Unfortunately I found that this was not possible.
Along the way, I found a blog post by Fang-Pen:
SHe was fighting the same P2P battle, and was getting the same N/A’s with rocm-bandwidth-test as I.
As Phang-Pen wrote:
Enabling P2P PCIe Communication in the Linux Kernel
I read the Linux kernel config file and learned that the P2P PCIe feature is relatively new. To enable it, you need to activate DMABUF_MOVE_NOTIFY option while compiling the kernel.
With NixOS, you can configure it like this:
boot.kernelPatches = [
{
name = "enable-hsa-amd-p2p";
patch = null;
extraStructuredConfig = with lib.kernel; {
DMABUF_MOVE_NOTIFY = yes;
HSA_AMD_P2P = yes;
};
}
];
Then rebuild your NixOS system and reboot. After that, running the ROCm PCIe communication benchmark showed data exchanges between the GPUs.
So, I was like “I can do this!” ha! I’m using Fedora right now, and building a kernel from scratch became . . . cough . . . tedious.
Yet, I finally got it compiled, today, after switching HSA_AMD_P2P = Y (I could find no mention of DMABUF_MOVE_NOTIFY in the 7.1.0 kernel src I used).
splifingate@Fedora:~$ rocm-bandwidth-test
RocmBandwidthTest Version: 2.6.0
Launch Command is: rocm-bandwidth-test (rocm_bandwidth -a + rocm_bandwidth -A)
Device: 0, AMD EPYC 4565P 16-Core Processor
Device: 1, AMD Radeon AI PRO R9700, GPU-83f5be58d4413991, 03:0.0
Device: 2, AMD Radeon AI PRO R9700, GPU-423b3075e139d306, 06:0.0
Device: 3, AMD EPYC 4565P 16-Core Processor, GPU-XX, 12:0.0
Inter-Device Access
D/D 0 1 2 3
0 1 1 1 1
1 1 1 1 1
2 1 1 1 1
3 1 1 1 1
Inter-Device Numa Distance
D/D 0 1 2 3
0 0 20 20 20
1 20 0 40 40
2 20 40 0 40
3 20 40 40 0
Unidirectional copy peak bandwidth GB/s
D/D 0 1 2 3
0 N/A 28.711 28.697 33.187
1 28.104 522.494 28.105 28.105
2 28.165 28.169 366.429 28.168
3 32.840 15.848 15.857 14.399
Bidirectional copy peak bandwidth GB/s
D/D 0 1 2 3
0 N/A 48.684 50.645 32.934
1 48.684 N/A 55.491 31.082
2 50.645 55.491 N/A 31.106
3 32.934 31.082 31.106 N/A
splifingate@Fedora:~$ amd-smi topology
ACCESS TABLE:
0000:03:00.0 0000:06:00.0 0000:12:00.0
0000:03:00.0 ENABLED ENABLED ENABLED
0000:06:00.0 ENABLED ENABLED ENABLED
0000:12:00.0 ENABLED ENABLED ENABLED
WEIGHT TABLE:
0000:03:00.0 0000:06:00.0 0000:12:00.0
0000:03:00.0 0 40 40
0000:06:00.0 40 0 40
0000:12:00.0 40 40 0
HOPS TABLE:
0000:03:00.0 0000:06:00.0 0000:12:00.0
0000:03:00.0 0 2 2
0000:06:00.0 2 0 2
0000:12:00.0 2 2 0
LINK TYPE TABLE:
0000:03:00.0 0000:06:00.0 0000:12:00.0
0000:03:00.0 SELF PCIE PCIE
0000:06:00.0 PCIE SELF PCIE
0000:12:00.0 PCIE PCIE SELF
NUMA BW TABLE:
0000:03:00.0 0000:06:00.0 0000:12:00.0
0000:03:00.0 N/A N/A N/A
0000:06:00.0 N/A N/A N/A
0000:12:00.0 N/A N/A N/A
CACHE COHERANCY TABLE:
0000:03:00.0 0000:06:00.0 0000:12:00.0
0000:03:00.0 SELF NC NC
0000:06:00.0 NC SELF NC
0000:12:00.0 NC NC SELF
ATOMICS TABLE:
0000:03:00.0 0000:06:00.0 0000:12:00.0
0000:03:00.0 SELF 64,32 64,32
0000:06:00.0 64,32 SELF 64,32
0000:12:00.0 64,32 64,32 SELF
DMA TABLE:
0000:03:00.0 0000:06:00.0 0000:12:00.0
0000:03:00.0 SELF T T
0000:06:00.0 T SELF T
0000:12:00.0 T T SELF
BI-DIRECTIONAL TABLE:
0000:03:00.0 0000:06:00.0 0000:12:00.0
0000:03:00.0 SELF F F
0000:06:00.0 F SELF F
0000:12:00.0 F F SELF
03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 [Radeon AI PRO R9700] (rev c0) (prog-if 00 [VGA controller])
Subsystem: Gigabyte Technology Co., Ltd Device 242f
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 119
IOMMU group: 16
Region 0: Memory at ff000000000 (64-bit, prefetchable) [size=32G]
Region 2: Memory at ff800000000 (64-bit, prefetchable) [size=256M]
Region 4: I/O ports at f000 [size=256]
Region 5: Memory at f6400000 (32-bit, non-prefetchable) [size=512K]
Expansion ROM at f6480000 [disabled] [size=128K]
Capabilities: [48] Vendor Specific Information: Len=08 <?>
Capabilities: [50] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [64] Express (v2) Legacy Endpoint, IntMsgNum 0
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ TEE-IO-
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed 32GT/s, Width x16, ASPM L1, Exit Latency L1 <1us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM L1 Enabled; RCB 64 bytes, LnkDisable- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- FltModeDis-
LnkSta: Speed 32GT/s, Width x16
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
10BitTagComp+ 10BitTagReq+ OBFF Not Supported, ExtFmt+ EETLPPrefix+, MaxEETLPPrefixes 1
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS-
AtomicOpsCap: 32bit+ 64bit+ 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
AtomicOpsCtl: ReqEn+
IDOReq- IDOCompl- LTR+ EmergencyPowerReductionReq-
10BitTagReq+ OBFF Disabled, EETLPPrefixBlk-
LnkCap2: Supported Link Speeds: 2.5-32GT/s, Crosslink- Retimer+ 2Retimers+ DRS-
LnkCtl2: Target Link Speed: 32GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported, FltMode-
Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
Address: 00000000fee00000 Data: 0000
Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
Capabilities: [150 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
ECRC- UnsupReq- ACSViol- UncorrIntErr- BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
ECRC- UnsupReq- ACSViol- UncorrIntErr- BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+
ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ CorrIntErr- HeaderOF-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ CorrIntErr- HeaderOF-
AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 00000000 00000000 00000000 00000000
Capabilities: [200 v1] Physical Resizable BAR
BAR 0: current size: 32GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB
BAR 2: current size: 256MB, supported: 2MB 4MB 8MB 16MB 32MB 64MB 128MB 256MB
Capabilities: [240 v1] Power Budgeting <?>
Capabilities: [270 v1] Secondary PCI Express
LnkCtl3: LnkEquIntrruptEn- PerformEqu-
LaneErrStat: 0
Capabilities: [2a0 v1] Access Control Services
ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
Capabilities: [2d0 v1] Process Address Space ID (PASID)
PASIDCap: Exec+ Priv+, Max PASID Width: 10
PASIDCtl: Enable+ Exec- Priv-
Capabilities: [320 v1] Latency Tolerance Reporting
Max snoop latency: 1048576ns
Max no snoop latency: 1048576ns
Capabilities: [410 v1] Physical Layer 16.0 GT/s
Phy16Sta: EquComplete- EquPhase1- EquPhase2- EquPhase3- LinkEquRequest-
Capabilities: [450 v1] Lane Margining at the Receiver
PortCap: Uses Driver-
PortSta: MargReady+ MargSoftReady-
Capabilities: [500 v1] Physical Layer 32.0 GT/s
Phy32Cap: EqualizationBypass+ NoEqualizationNeeded-
ModTsMode0+ ModTsMode1- ModTsMode2-
Phy32Ctl: EqualizationBypassDis- NoEqualizationNeededDis-
Modified TS Usage Mode: PCI Express
Phy32Sta: EquComplete+ EquPhase1+ EquPhase2+ EquPhase3+ LinkEquRequest-
Received Enhanced Link Behavior Control: Equalization bypass to highest rate support
ModTsRecv- TxPrecodeOn- TxPrecodeReq- NoEqualizationNeededRecv-
Kernel driver in use: amdgpu
Kernel modules: amdgpu
06:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 [Radeon AI PRO R9700] (rev c0) (prog-if 00 [VGA controller])
Subsystem: XFX Limited Device 9801
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 120
IOMMU group: 20
Region 0: Memory at fe000000000 (64-bit, prefetchable) [size=32G]
Region 2: Memory at fe800000000 (64-bit, prefetchable) [size=256M]
Region 4: I/O ports at e000 [size=256]
Region 5: Memory at f6200000 (32-bit, non-prefetchable) [size=512K]
Expansion ROM at f6280000 [disabled] [size=128K]
Capabilities: [48] Vendor Specific Information: Len=08 <?>
Capabilities: [50] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [64] Express (v2) Legacy Endpoint, IntMsgNum 0
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ TEE-IO-
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed 32GT/s, Width x16, ASPM L1, Exit Latency L1 <1us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM L1 Enabled; RCB 64 bytes, LnkDisable- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- FltModeDis-
LnkSta: Speed 32GT/s, Width x16
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
10BitTagComp+ 10BitTagReq+ OBFF Not Supported, ExtFmt+ EETLPPrefix+, MaxEETLPPrefixes 1
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS-
AtomicOpsCap: 32bit+ 64bit+ 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
AtomicOpsCtl: ReqEn+
IDOReq- IDOCompl- LTR+ EmergencyPowerReductionReq-
10BitTagReq+ OBFF Disabled, EETLPPrefixBlk-
LnkCap2: Supported Link Speeds: 2.5-32GT/s, Crosslink- Retimer+ 2Retimers+ DRS-
LnkCtl2: Target Link Speed: 32GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported, FltMode-
Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
Address: 00000000fee00000 Data: 0000
Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
Capabilities: [150 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
ECRC- UnsupReq- ACSViol- UncorrIntErr- BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
ECRC- UnsupReq- ACSViol- UncorrIntErr- BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+
ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ CorrIntErr- HeaderOF-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ CorrIntErr- HeaderOF-
AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 00000000 00000000 00000000 00000000
Capabilities: [200 v1] Physical Resizable BAR
BAR 0: current size: 32GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB
BAR 2: current size: 256MB, supported: 2MB 4MB 8MB 16MB 32MB 64MB 128MB 256MB
Capabilities: [240 v1] Power Budgeting <?>
Capabilities: [270 v1] Secondary PCI Express
LnkCtl3: LnkEquIntrruptEn- PerformEqu-
LaneErrStat: 0
Capabilities: [2a0 v1] Access Control Services
ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
Capabilities: [2d0 v1] Process Address Space ID (PASID)
PASIDCap: Exec+ Priv+, Max PASID Width: 10
PASIDCtl: Enable+ Exec- Priv-
Capabilities: [320 v1] Latency Tolerance Reporting
Max snoop latency: 1048576ns
Max no snoop latency: 1048576ns
Capabilities: [410 v1] Physical Layer 16.0 GT/s
Phy16Sta: EquComplete- EquPhase1- EquPhase2- EquPhase3- LinkEquRequest-
Capabilities: [450 v1] Lane Margining at the Receiver
PortCap: Uses Driver-
PortSta: MargReady+ MargSoftReady-
Capabilities: [500 v1] Physical Layer 32.0 GT/s
Phy32Cap: EqualizationBypass+ NoEqualizationNeeded-
ModTsMode0+ ModTsMode1- ModTsMode2-
Phy32Ctl: EqualizationBypassDis- NoEqualizationNeededDis-
Modified TS Usage Mode: PCI Express
Phy32Sta: EquComplete+ EquPhase1+ EquPhase2+ EquPhase3+ LinkEquRequest-
Received Enhanced Link Behavior Control: Equalization bypass to highest rate support
ModTsRecv- TxPrecodeOn- TxPrecodeReq- NoEqualizationNeededRecv-
Kernel driver in use: amdgpu
Kernel modules: amdgpu
Fresh compile, and fresh boot into the 7.1 kernel on Fedora 44: everything seems to be working ok. It’s been about twelve years since I compiled a kern from scratch, and I didn’t get things just how I like 'em.
Haven’t tried any LLM stuff yet…