vllmmultigpulog.txt (38.8 KB)
I have been enjoying my single r9700 so much that recently i have added a second card to my server, and for ollama things have been mostly great (no speedup for a single user, but running gemma4-31B at full context is definitely a nice improvement, though i was hoping it would improve the tk/s for a single user if i’m honest),
VLLM however has proven more….problematic, whilst the official vllm/vllm-openai-rocm container works great when tensor-parallelism is set to 1 when i try to increase it to enable the second gpu it just won’t work despite my best efforts (i have attached the logs)
The logs seem to point to P2P/NCCL issues, So i have tried setting NCCL_P2P_DISABLE=1 to force use of the, trying and failing to make P2P work on the motherboard (its an old x570 am4 asrockrack with dual pcie 4.0 x8/x8 for the cards), but no matter what i do it always seems to fail with: [FATAL ERROR]: HIP failure: ‘the operation cannot be performed in the present state’
honestly i don’t mind if all the communication happens via the shared memory, and skip P2P entirely for now, i just want to see vllm utilize more than one graphics card.
I noticed this thread Does the LRLink PCIe Bridge Fix RCCL's Broken P2P on Dual R9700? is sort of similar but none of the options there seemed to help me much and its more focused on making the P2P work for training, whereas i just want to do inference.
My system:
-5950X
-64GB DDR4-3200
-dual R9700 connected in a pcie4.0 8/8 split directly to the CPU
-Asrockrack x570D4U-2L2T/BCM
specific software configuration:
Proxmox with an ubuntu 24.04 guest with both gpu’s passed through, amd SMI from guest:
±-----------------------------------------------------------------------------+
| AMD-SMI 26.2.2+671d39a71e amdgpu version: 6.16.13 ROCm version: 7.2.2 |
| VBIOS version: 00153259 |
| Platform: Linux Guest (Passthrough) |
|-------------------------------------±---------------------------------------|
| BDF GPU-Name | Mem-Uti Temp UEC Power-Usage |
| GPU HIP-ID OAM-ID Partition-Mode | GFX-Uti Fan Mem-Usage |
|=====================================+========================================|
| 0000:01:00.0 …Radeon AI PRO R9700 | 0 % 32 °C 0 15/300 W |
| 0 0 N/A N/A | 39 % 24.71 78/32624 MB |
|-------------------------------------±---------------------------------------|
| 0000:02:00.0 …Radeon AI PRO R9700 | 0 % 28 °C 0 3/300 W |
| 1 1 N/A N/A | 0 % 24.71 110/32624 MB |
±------------------------------------±---------------------------------------+
±-----------------------------------------------------------------------------+
| Processes: |
| GPU PID Process Name GTT_MEM VRAM_MEM MEM_USAGE CU % |
|==============================================================================|
| No running processes found |
±-----------------------------------------------------------------------------+
topo:
rocmbuntu@rocmbuntu:~$ amd-smi topology
ACCESS TABLE:
0000:01:00.0 0000:02:00.0
0000:01:00.0 ENABLED DISABLED
0000:02:00.0 DISABLED ENABLED
commands i have tried so far:
-docker run --rm --group-add=video --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --shm-size=16g --ulimit memlock=-1:-1 --device /dev/kfd --device /dev/dri -v ~/.cache/huggingface:/root/.cache/huggingface -e NCCL_DEBUG=INFO -e NCCL_P2P_DISABLE=1 -e HIP_VISIBLE_DEVICES=0,1 -e HF_TOKEN=redacted -p 8000:8000 --ipc=host vllm/vllm-openai-rocm:latest Qwen/Qwen3-0.6B --max-model-len 18000 --tensor-parallel-size 2
-docker run --rm --group-add=video --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --device /dev/kfd --device /dev/dri -v ~/.cache/huggingface:/root/.cache/huggingface -e NCCL_P2P_DISABLE=1 -e HF_TOKEN=redacted -p 8000:8000 --ipc=host vllm/vllm-openai-rocm:latest Qwen/Qwen3-0.6B --max-model-len 18000 --tensor-parallel-size 2
-many, many variations of the above
As mentioned single-gpu works great the problems don’t start until the parallelism is introduced, Trying to debug the logs with an LLM and following its advice has so far proved fruitless.
please help me obi wan janitobi you are my only hope



