If you haven’t seen it, I have a whole thread on this topic.
Granted, I’m using a PCIe switch card to sidestep the p2p bugs entirely. But if you are looking for a good central place to find the configs you need, this is a good place to start:
### The core problem
RCCL hits a reproducible `hipIpcGetMemHandle failed: invalid argument`
crash at src/transport/p2p.cc:256 when attempting P2P on RDNA 4.
Documented in ROCm/rccl-tests #162 and ROCm #6148. No merged RCCL fix
as of May 2026. This is a software bug in RCCL's IPC handle export —
it happens regardless of platform (consumer or workstation).
The common workaround is NCCL_P2P_DISABLE=1, which forces shared-memory
fallback. Works, but you lose real P2P bandwidth.
The only hardware workaround that preserves true P2P: a Gen 4 PCIe
switch (e.g., PEX88096) between the GPUs. The switch changes the
topology enough that RCCL's IPC path doesn't crash.
### Known-good BIOS settings (tested on TRX40, should apply to WRX80)
- Above 4G Decoding: Enabled
- Resizable BAR: Enabled
- CSM: Disabled
- APBDIS: 1
- DF C-states: Disabled
- SOC: P0
- 10-Bit Tag: Enabled
- NBIO: 600MHz
### Kernel boot params
iommu=pt amdgpu.runpm=0 pci=realloc amdgpu.cwsr_enable=0
- iommu=pt — required for GPU passthrough/P2P
- amdgpu.runpm=0 — without this, GPUs fail to initialize
- pci=realloc — enables full 32GB ReBAR per card
- amdgpu.cwsr_enable=0 — stability fix for gfx1201
### ACS (Access Control Services) — critical and counterintuitive
ACS on PCIe switch downstream ports MUST be cleared for P2P to work
at full bandwidth. This was the single largest improvement in my
testing: 5.41 GB/s → 19.60 GB/s (3.6x jump).
Needs a persistent systemd unit to clear ACS on every boot — the
BIOS re-enables it each time. Without clearing ACS, P2P technically
works but at a fraction of the bandwidth.
### Bandwidth results (dual R9700 through PEX88096)
- RCCL: 19.60 GB/s (70% of theoretical ceiling)
- Raw P2P: 28.07 GB/s
- Three stacked fixes got here: switch (crash fix) → pci=realloc
(ReBAR) → ACS clear (3.6x bandwidth jump)
### What does NOT work on gfx1201
- hipDeviceCanAccessPeer returns 0 (RCCL uses HSA dmabuf imports instead)
- hipMemcpyPeer hangs
- NCCL_DMABUF_ENABLE=1 — no measurable effect
- NCCL_NET_GDR_LEVEL=LOC — no measurable effect
- GPU Direct RDMA is not wired to RCCL P2P on gfx1201 despite
rocminfo claiming DMAbuf Support: YES
### Container requirements (vLLM)
- --shm-size=16G (or /dev/shm will choke)
- NCCL_IB_DISABLE=1 (skip InfiniBand probe)
- NCCL_SOCKET_IFNAME=<your real NIC> (docker bridges misroute)
- disable_custom_all_reduce=True in vLLM