No I meant those containers, running on R9700 haha ![]()
Wendell, searching Reddit today yielded RDNA4 Specific vLLM 22 Docker Image / Ecosystem dev work, a working setup. The back-n-forth you see there shows that this is a slow first launch, but this has MTP as well. You need to dig in!
Tested and working TP=4 (4 x R9700), MTP. KV cache needs to be FP8 in my testing, otherwise tool calls go haywire with MTP.
Podman view, replace USER with your username and note I am doing the HF snapshot reference, not the model name per-se.
services:
vllm-qwen36-27b-mxfp416-mtp:
image: tcclaviger/vllm22:latest
container_name: vllmq27b
restart: "no"
network_mode: host
ipc: host
cap_add:
- SYS_PTRACE
security_opt:
- seccomp=unconfined
group_add:
- video
shm_size: 32gb
devices:
- /dev/kfd
- /dev/dri
volumes:
- /home/USER/.cache/huggingface/hub/models--tcclaviger--Qwen3.6-27B-MXFP416-MTP/snapshots/b4488cbd2ecc91ff2121665576984102d14e98ae:/models/model:ro
- /home/USER/.cache/tunableop:/tunableop
- /home/USER/.cache/triton:/root/.triton/cache
- /home/USER/.cache/vllm-profiles:/profiles
environment:
- HIP_VISIBLE_DEVICES=0,1,2,3
- OMP_NUM_THREADS=4
- PYTORCH_TUNABLEOP_ENABLED=1
- PYTORCH_TUNABLEOP_TUNING=0
- PYTORCH_TUNABLEOP_RECORD_UNTUNED=0
- VLLM_ROCM_USE_AITER=0
- NCCL_PROTO=Simple
- TRITON_CACHE_DIR=/root/.triton/cache
- PYTORCH_TUNABLEOP_FILENAME=/tunableop/tunableop_results_full%d.csv
- PYTORCH_TUNABLEOP_UNTUNED_FILENAME=/tunableop/tunableop_untuned%%d.csv
- HSA_ENABLE_INTERRUPT=1
- HSA_ENABLE_MWAITX=1
- GPU_MAX_HW_QUEUES=1
command: >
/models/model
--tensor-parallel-size 4
--enable-prefix-caching
--gdn-side-cache-entries 20
--enable-auto-tool-choice
--max-num-seqs 32
--max-num-batched-tokens 8192
--enable-chunked-prefill
--gpu-memory-utilization 0.85
--host 0.0.0.0
--port 8000
--served-model-name Qwen3.6-27B-MXFP416-MTP
--max-model-len 262144
--kv-cache-dtype fp8
--enforce-eager
--speculative-config '{"method": "mtp", "num_speculative_tokens": 3}'
--compilation-config '{"cudagraph_capture_sizes": [1, 2, 3, 4, 5, 6, 7, 8, 16, 24, 32], "max_cudagraph_capture_size": 32}'
--reasoning-parser qwen3
--tool-call-parser qwen3_xml
--limit-mm-per-prompt.video 0
--limit-mm-per-prompt.image 20
--mm-processor-cache-gb 1
--trust-remote-code
healthcheck:
test: ["CMD-SHELL", "curl -f http://localhost:8000/health || exit 1"]
interval: 30s
timeout: 10s
retries: 3
start_period: 180s
Bash
#!/bin/bash
# Qwen3.6-27B MXFP416-MTP - Podman Launch
set -e
# Configuration
CONTAINER_NAME="vllmq27b"
IMAGE="tcclaviger/vllm22:latest"
SNAPSHOT_DIR="$HOME/.cache/huggingface/hub/models--tcclaviger--Qwen3.6-27B-MXFP416-MTP/snapshots/b4488cbd2ecc91ff2121665576984102d14e98ae"
PORT=8000
TP_SIZE=4
GPU_DEVICES="0,1,2,3"
# Cache directories
TRITON_CACHE="$HOME/.cache/triton"
TUNABLEOP_DIR="$HOME/.cache/tunableop"
PROFILES_DIR="$HOME/.cache/vllm-profiles"
mkdir -p "$TRITON_CACHE" "$TUNABLEOP_DIR" "$PROFILES_DIR"
echo "=== Qwen3.6-27B MXFP416-MTP ==="
echo "Container: $CONTAINER_NAME"
echo "Port: $PORT"
echo "TP: $TP_SIZE | GPUs: $GPU_DEVICES"
echo ""
# Stop existing container
podman stop $CONTAINER_NAME 2>/dev/null || true
podman rm $CONTAINER_NAME 2>/dev/null || true
# Launch
podman run -d \
--name $CONTAINER_NAME \
--network host \
--ipc host \
--cap-add SYS_PTRACE \
--security-opt seccomp=unconfined \
--group-add video \
--device /dev/kfd \
--device /dev/dri \
-v "$TRITON_CACHE:/root/.triton/cache" \
-v "$TUNABLEOP_DIR:/tunableop" \
-v "$PROFILES_DIR:/profiles" \
-v "$SNAPSHOT_DIR:/models/model:ro" \
-e HIP_VISIBLE_DEVICES=$GPU_DEVICES \
-e OMP_NUM_THREADS=4 \
-e PYTORCH_TUNABLEOP_ENABLED=1 \
-e PYTORCH_TUNABLEOP_TUNING=0 \
-e PYTORCH_TUNABLEOP_RECORD_UNTUNED=0 \
-e VLLM_ROCM_USE_AITER=0 \
-e NCCL_PROTO=Simple \
-e TRITON_CACHE_DIR=/root/.triton/cache \
-e PYTORCH_TUNABLEOP_FILENAME=/tunableop/tunableop_results_full%d.csv \
-e PYTORCH_TUNABLEOP_UNTUNED_FILENAME=/tunableop/tunableop_untuned%%d.csv \
-e HSA_ENABLE_INTERRUPT=1 \
-e HSA_ENABLE_MWAITX=1 \
-e GPU_MAX_HW_QUEUES=1 \
$IMAGE \
/models/model \
--tensor-parallel-size $TP_SIZE \
--enable-prefix-caching \
--gdn-side-cache-entries 20 \
--enable-auto-tool-choice \
--max-num-seqs 32 \
--max-num-batched-tokens 8192 \
--enable-chunked-prefill \
--gpu-memory-utilization 0.85 \
--host 0.0.0.0 \
--port $PORT \
--served-model-name Qwen3.6-27B-MXFP416-MTP \
--max-model-len 262144 \
--kv-cache-dtype fp8 \
--enforce-eager \
--speculative-config '{"method": "mtp", "num_speculative_tokens": 3}' \
--compilation-config '{"cudagraph_capture_sizes": [1, 2, 3, 4, 5, 6, 7, 8, 16, 24, 32], "max_cudagraph_capture_size": 32}' \
--reasoning-parser qwen3 \
--tool-call-parser qwen3_xml \
--limit-mm-per-prompt.video 0 \
--limit-mm-per-prompt.image 20 \
--mm-processor-cache-gb 1 \
--trust-remote-code
echo ""
echo "✅ Started: $CONTAINER_NAME"
echo "Health: http://127.0.0.1:$PORT/health"
echo "Logs: podman logs -f $CONTAINER_NAME"
But is it awesome though??
Awesome enough. 4xR9700 getting numbers in the range of 2xDGX Spark on same LLM, different variation of the same quant. Also tested with Qwen 3.5 122B A10B and similar scaling. Lots left on the table, as nvtop shows that the GPUs aren’t saturated.
Thanks for sharing! My finger is hovering over the trigger on the 4x 9700 vs. 4x B70 vs. 6x 9070xt (or 6x 9700 / B70) decision at the moment.
It is a hard place to be. I chose to try and find a suitable modern replacement for my older 3090 setup. So far, it is underwhelming, but promising. Although I have a ton of Intel cards, the only recent is a B50 that I grabbed for a specific use-case.
Best of luck in your choice.
When you say underwhelming, were you expecting more performance-wise, or were you more so noting the software / full stack friction?
To get more out of the stack, I have some ideas about splitting the agent loop so certain stages, tasks, subloops, etc, are structured (into the programmatic loop, not at agent-behest) to be dispatched to different models so the larger model on the quad 9700 node can be kept more free to decrease the pressure on VRAM capacity. The subloop would be pruned after the outcomes are validated and final, resolving back to the main context branch for the larger model to continue with a less bloated context window. Rather than over-reliance on summaries to maintain context fidelity. Not a fan of compaction (etc) at all.
Relevance to this 9700 thread, is that; Done right, a secondary outcome of this design can be that these subloops thereby generate structured training data matched to specific tasks or patterns. This provides input for a regular pipeline for iterative fine-tuning, potentially getting more out of the same stack and models. The hypothesis to test is whether (or to what extent) this closes the gap between a local rig and setups costing much more more… or thousands a month in API tokens.
Re: Intel/Alternatives; I considered TensTorrent for a bit there. Love the philosophy and the design (Risc-V heck yeah!) But the trajectory / development / community maturity doesn’t stack up for deploying into an operational environment yet, at least not in my case. Similar story from what I can work out for Intel’s stack right at this moment. And Nvidia is out purely given the likelihood of future drm-style shenanigans.
General questions I wanted to throw out there: If anyone has tried more than 4 cards, I’m really keen to hear whether 6x scales as well as 4 or 8, or if it hits the same diminishing returns curve spread across each additional pair.
Also, has anyone run R9700s on a WRX80 / Threadripper Pro platform (pcie 4.0 x16)? Most results I’ve seen are on TRX50. I’m looking at mobo: Gigabyte MC62-G40 with cpu: 5975WX, but still finalising approach. The assumption is that the slower pcie isn’t an unacceptably limiting factor and balances with lower costs and therefore more ram/cores/gpus.
ROCm layers, vLLM capabilities, all have limitations and bugs. Many bugs with no clear indication of being taken seriously. Performance wise, the GPUs are not able to surpass 3090s doing the same task, most of the challenges are in keeping the cards fed and evenly busy. My hope was that these cards would deliver a modern, lower power, more recent design with native FP8. The hardware is technically there, the software detracts.
I’ve spent near zero time with the training tooling at this point, since I have not gotten the setup running well any inference until the very custom setup I shared above. I also note that Unsloth only has their core tooling for AMD, just like Apple. I have never actually used any AMD tooling beyond messing with ROCm. Without Donato’s prior work, even that would have been a bit much.
Right now 4 x R9700 can’t keep up with 2 x DGX Spark, probably on-par or slower with a single DGX, which is sad if you know how the DGX Spark is limited in memory bandwidth and how networking is technically just 100Gbps with solid ROCE, not PCIE. I use the comparison simply as a price, they are close-ish in my region right now.
My testbed is based on Epyc Milan, 7663, so socket SP3, and I needed to hardcode Gen4 PCIE instead of leaving it in Auto, to make the Gen5 PCIE switch chip onboard the R9700 behave as expected. Since I wanted 4 x R9700, no other motherboard I have would work, either in capacity or pcie slot spacing. I also didn’t want to build another bifurcated coin rig style frankenstein with multiple active 8x8 bifurcation cards or 4x4x4x4 OCUlink style setup. And just say no to passive PCIE cables.
Ah I see. Thanks for unpacking. If not relying on Nvidia’s ecosystem takes effort I’m up for it. However, if the effort is mostly in just trying to get the cards to talk to each other and all else being equal, to get models to run with that layer, I’m wondering if that effort is better (for my use) spend somewhere like TensTorrent. TT requires it’s own investment of effort, but at least has the right approach with meshing the cards via 800g ethernet. The baseline tokens per second, per dollar rates are significantly higher also. The issue many will have, though it’s also kind of a key selling point at the same time, is that that it’s a whole other ecosystem which needs models optimised for it. That being said when you dig into that ecosystem, its open source all the way down, they’re encouraging full control down to bare metal, and are providing everything they can to engage the community on it, it seems. Hrm… More research to do.
Thanks for confirming testbed also.
I’m continuing research on a modular 4x TensTorrent Blackhole (32gb) p150(a/b) 4ru module, which can be scaled up (ideally), and will ultimately compare this to the quad AMD 9700 option which I nearly pulled the trigger on yesterday before I learned more about the nuance delta between AMD and TT’s software layer friction. I’ll report back here in case anyone finds it interesting as I’ve lurked here for a while and gained much from people’s contributions. Mods: If the comparison is not ideal for this thread I can break it off into a separate topic.
Welp.. I made a call. Went with 4x 9700s.
what models will you be running? i mean mostly in size
edited: I misread the above post as a reply to my post above it, not to Wendell’s op for some reason… *backs out of room slowly*
Looking forward to the video. It’s nice to see more real-world guides for multi-GPU setups, especially since there isn’t much info out there yet.