your experience or a writeup of how you got here would be hugely valuable for me, and I can feature in a video, if you like
Well, I can certainly have a go when I’ve got a bit of time - fair warning, a significant portion of my experience thus far has been “spending an uncomfortable amount of time looking for interesting PRs on the GGML GitHub repo” 
If you can tell me the kind of details you’d be particularly interested in, I’ll know where to focus it.
1 Like
Just in case anyone’s wondering…the new MTP PR I was talking about earlier on llama.cpp is looking good, and I suspect it’s going to be merged soon. On Qwen 3.6 27B (dense), it bring prefill down to about 600-700t/s, but decode hits ~55-60t/s on a pair of R9700s.
Basically, prefill halves and decode doubles. They’re going to work on the prefill optimisation once it’s merged. The cost is ~3GB VRAM for the MTP heads.
3 Likes
So far this has done the best for me:
./llama-cli --mmap -hf mradermacher/gemma-4-31B-it-heretic-ara-custom-i1-GGUF:Q4_K_M --n-gpu-layers 999 -sm tensor -cram 256 -ub 2048
Llama-Bench doesnt work with tensor for me unfortunately but on a warmed up system i get a very solid 27.5 tk/s with a power cap of 260 on both cards, it also scales quite well at larger contexts, this is under rocm
I’ll try vulkan tommorow, but i started at 24-ish with a sharp drop off to 19ish at 100k tokens, with this so far i start at 27-28 and havent seen a drop under 24.5 at 100k+
edit:
Surprisingly having 2 parallel threads of gemma 31b still nets a TG of 23.4 for both users, not bad scaling tbh, I think this is about ready for deploying to my family
Using llama-server in model-dir mode does have a bit of a deal-breaker for me in my homelab,
Even after going idle (–sleep-idle-seconds 10) it never fully releases the gpu’s and burns up around 180 watts doing nothing, this never goes away after the first model is loaded, has anyone else encountered this?
Please don’t judge me 
procmon.py.txt (3.8 KB)
python3 procmon.py --trigger “server is entering sleeping state” ./llama-server --host 0.0.0.0 --models-dir ~/.cache/huggingface/hub/ -ngl 999 -c 262000 -np 1 -fa 1 --cache-ram 256 --ctx-checkpoints 6 --no-mmap -sm tensor --sleep-idle-seconds 10
I will report myself to IT janitor jail now
Edit: Image updated and pushed to docker hub, the side cache value is number of GDN cache entries to be available, keep it reasonable and memory use is minimal, 10-20 is usually plenty.
https://hub.docker.com/repository/docker/tcclaviger/vllm-rocm-mxfp4-nvfp4/general
I run 4x9700s and get vastly higher speeds than llama can deliver on the system, no need to ever load llama on multigpu rdna4 (outside running huge models in sub 4 bit quants). If you’re doing more than chatting, the prefill performance of llama on RDNA4 is untenable imho.
For single requests (numbers at PCIE 5.0 4x per card, new 4 way 8x setup via 9500-4i being installed)
Qwen3.6 27B @ ~130 TPS decode / 2800 prefill
Qwen3.6 35B @ ~240 TPS / 8000 prefill
Gemma 4 31B @ ~ 100 TPS / 2000 prefill
Concurrency simply scale up.
35B does something like ~110 TPS per chat @ 16 concurrent.
27B does something like ~70 TPS per chat @ 8 concurrent.
I don’t run Gemma enough to remember concurrency speeds on it, but I expect low since it’s a generally slow model.
My docker image has many additions, based on vLLM 18.1 dev branch but …fixed, lol. I would wholeheartedly caution against use with high concurrency, I’ve tested concurrency 16-32 pretty thoroughly but beyond that just the occasional max memory pressure test, it passes but there are def edge cases I am missing.
None of the added kernels need flags on rdna4 to work, no quantization config yada yada. NVFP4, MXFP4, FP8 Block 128, all routed in non-distruptivr ways to hit the right RDNA4 kernels.
- Prefix caching GDN that uses an O1 lookup side buffer for prefix hits on linear attention snapshots, preserves the speed advantage GDN is supposed to have (vLLM 20 does not, it’s slow).
- AllGather fix so TP > 1 has non-blocking concurrent allgather whilst the next token is being processed, this alone was ~20% speed bump.
- Triton MXFP4 proper implementation aka fast
- Triton FP8 linear and MoE kernel implementations with tuned GEMM sizes for RDNA4
- NVFP4 dequant linear and MoE kernels that roughly match MXFP4 kernel speeds
- Prefix pinning, so you can fix a prefix (system prompt or whatever you use frequently) to never be purged
- Built on rocm 7.2.1 libraries from scratch then has the 7.2 fix inserted that caused blocking when loading multiple vLLM instances (and bug)
- DPC and PPC working but not done (I don’t recommend enabling yet, whilst the do double the speeds at high context use they roughly 1/2 it at low, turning the speed curve into a roughly straight line, but not fully stable yet)
- Back ported vLLM 19 models to work, adding gemma4
- More…I can’t remember everything, that’s what git is for.
Short version is, run it in docker so you get rocm 7.2.1 advantages but 7.2.1 has a few bugs so …you need to replace one compiled file with 7.2 or 7.1.1 version or you get GPU blocking, container already has it.
Setup docker compose with your options and go.
–NOTE–
RDNA4, using GEMM tuned FP8 kernels does NOT respect power limits set in the driver, the ramp rate is uncontrolled. You will see 500+ watts per cards as spikey behavior. Setting lower power limits does nothing, to control it set a negative core speed offset.
The 4 cards pull hard enough to OCP trip my EVGA T2 1600 if I leave them at full clock speed, you are warned, if you have a questionable PSU (~1000w or so) and try to run this with FP8 models on TP2 kiss the PSU goodbye.
1 Like
Qwen3.6 27B FP8 Block 128 Single User Scaling (-350 core, 2710 VRAM ECC disabled, 260w limit) :
Qwen3.6 35B A3B NVFP4 same settings on GPU:
35B Concurrency:
27B Concurrency:
That’s some pretty wild variance in decode you’ve got there - are you running MTP or some other speculative decoding strategy?
Yeah it’s MTP hit rates coupled with a RDNA 4 TP4 quirk. During normal use its not really observable.
There is some weird condition on RDNA4 at the short prompts, like the 20 token region I’ve seen recurring drops regardless of model or quantization type. Adjusting GEMM configs can move it around but not fully eliminate it. Occurs even without MTP.
At this point my intuition suggests its a driver power control thing but can’t say for sure. Possibly some quirk of the L3, who knows at this point.
My testimonial is hallelujah and thank you @digitalscream
I spent a long time trying to get speculative decoding working in my coding workflow with the mod strategy you suggested in another thread, but I found it kept corrupting file paths with a vengeance using Qwen 3 Coder. For now Im running without until I get a realistic base line.
I also was victim to this: Reddit - Please wait for verification
I will add, man MTP in llama has really breathed life into my Strix Halo setup, Qwen3.6 35B with MTP in llama is actually fast enough to use a sub agent box from the main rig now, perform entity extraction for graph rag entities/relationships, etc.
It’s pretty transformative, something like 80tps now which I think is pretty solid for the bandwidth handicap the Halo setups have.
Regarding tool calling… llama seems to just have an issue with it, haven’t really debugged, just added a proxy that heals errant tool calls that are missing the closing } (seems to be the most common failure mode), or other simple errors before passing to tool execution.
Tried Q8 and Q6 of 3.6 35B, both had occasional tool call failures for no apparent reason so just seemed easier to heal the formats than try to debug llama lol.
Minimax 2.7 Q3xs…never misses a tool call, so, something going on with the Qwen 3.x series in llama
PS: Vllm qwen3.x I’ve never seen emit a malformed tool call, FP16, FP8, or NVFP4, tools are flawless.
May want to take a look at the update I just posted. May help get your setup more stable: Does the LRLink PCIe Bridge Fix RCCL's Broken P2P on Dual R9700? (Update: Yes it does!) - #20 by TrailFeatures
Turns out it’s purely MTP or MTP associated, with MTP off its quite stable:
Forgot to hook up the FP8 configs lol, reasonable gains from adding 5 json files to a folder lol:
EDIT: Updated container with retuned Qwen 35 and 27 FP8 configs for both TP4 and TP2. Nice uplift from 37 to 43, helped stabilize variance and picked up about 10% prefill speeds for both models vs old values. They’re baked into the image now.
2 Likes
Yeah…if you can live with the reduced prefill performance, the llama.cpp MTP patch (merged a couple of hours ago) really is astonishing, although the merged version is noticeably slower in MoE setups than the original PR. With the original, I was hitting 160t/s decode on dual R9700s, whereas now it rarely gets over 135-140t/s (based on a 110t/s baseline).
Still pretty good on 27B, though - from 30 → 55t/s, which near the lower bound of what I’d consider acceptable performance as my main agent model, but the prefill renders it unusable for that purpose. Hopefully they can sort it out with follow-up PRs.
The real question, though, is whether the guy who was building the DFlash spec PR is going to come back. The performance gains on that one are nuts, albeit somewhat context-specific - up to 700% increase in decodes speed.
I hope! I don’t really use my system for light chatty work much, its generally long horizon agentic work, so prefill really is more valuable than output speed.
I quantized Qwen3.5 122B in IQ4_NL with mtp last night to test it out, lifts it from 26 or so baseline ono the strix halo system to mid 40s, nice, but the prefill is just astonishingly bad, even after apply some tricks to get more perf out of it.
I am not running the merged version, just grafted the PR version into my llama directory. Going to try some rocm vs vulkan across a few models and see if the MTP work better with one or the other (so far vulkan’s always won for the models I run on that box, but…).
Id get into it to help but mostly tied up developing an quantization extension of NVFP4 to make it more accurate, a dynamic quantization MSE aware process that massively reduces the error rates during reconstruction. Trying to get back to as close as possible to FP8 error rates in ~5.5BPW size.
You might find that just using a normal draft model (eg Qwen 3.5 0.8B) might be a decent compromise - on my R9700s, I got about 80% of the decode performance increase with much less of a cut in prefill performance (1500t/s baseline, 740t/s MTP, 1050t/s draft).
Heh…found the limit of my power supply today, testing the Lemonade build with MTP and tensor split. Tripped the RM850 with both GPUs at 330W, which tensor split actually uses, unlike Vulkan + layer split.
Running them at 300W, though, Qwen 3.6 27B + MTP goes from 50t/s decode with layer split to 73t/s with tensor split. Unfortunately, prefill tanks even harder down to ~650t/s, so…good for conversations, not so much for agentic work.
Interestingly, putting the question of tensor split aside for a moment, just using Qwen 3.6 0.8B as a normal draft model results in better prefill (~1000t/s) and a decent decode uplift relative to baseline (30t/s → 52t/s), but it’s very much content-dependent; 52t/s writing code, 35t/s writing text. MTP doesn’t really have that problem.
2 Likes
OK, a PR was merged earlier which takes Vulkan prefill from 750t/s on the 27B model and R9700 to ~950-1000t/s. Looks like MTP decode performance is ~47t/s for text and 61t/s for code.
1 Like