Does MTP models work with multiple R9700 GPUs, when using ROCm tensor mode?

I’m using 2x R9700 GPUs and layer and row split mode is working with both Vulkan and ROCm, ROCm tensor mode also works when I don’t use MTP.

If I try to use tensor mode on Qwen3.6-27b with MTP enabled my system will crash, and seems to be a hardware level crash, I need to unplug the power to turn off the system.

I’m using the ASUS ProArt X870E with x8 + x8 Gen 5 PCIe lanes for the GPUs. I’m using ROCm 7.1, llama.cpp is compiled with the libraries from my system.

Except for tensor + MTP, I don’t have any issues with stability.

Is it possible to get MTP working with tensor split mode?

Following this one because I’m looking at doing a VERY similar setup for work.

1 Like

I was setting up an MTP model today and noticed a comment that said to “keep --np 1 until MTP on multi-slot stabilizes” on llama-server.

Surprised me too to read that.

We are on the bleeding edge so maybe give it a bit more time.

2 Likes

Yes, they do - I’m running the exact same system you are (well, older generation CPU and motherboard, but x8/x8 with dual R9700s).

I had the exact same problem, and it was the power supply. I had an RM850x, which should have been enough even with the +30W power limit I had on the cards, but it just tripped it. Upgraded to an HX1000, and the problem went away.

The problem is that tensor split + MTP is the only mode which fully uses both GPUs simultaneously at their full power.

Test by reducing the power limit to 250W (EDIT: or even lower to 210W, if that doesn’t work) with LACT.

1 Like

The specs say 300 TBP, but testing shows that 400W is the actual maximum draw for the card. That left only 50W for the rest of the system. Not a lot at all.

1 Like

Although I am on Strix Halo, maybe this could prove to be a good hint for your case.
I have not tried ROCm 7.1, let alone the latest mainline 7.2.4 for MTP, but the latest ROCm I am currently using is the ROCm TP nightly 260529 (7.14) using Torch 2.12.
This is from the new “Multiarch” channel, instead of the older “per-family” channel (I previously used the 260411 release as the latest version from the per-family channel before switching).

I am on Linux though (not sure what you are using, presuming Windows due to ROCm 7.1 perhaps?), and MTP works well on vLLM, using Qwen3.6-27b.
I Had some issues getting later vLLM versions to work on earlier versions of ROCm.

1 Like

I’m using the same 850W power supply, and you are right about the 850W being the problem. Claude said it could be because the cards can have short spikes at much higher values then the rated TDP, which could be a problem when using two cards.

Guess I’ll be buying a new power supply.

Aye…the good news is that, assuming the rest of the system isn’t drawing any more than mine (3 x NVMe, 4 x SATA SSDs, 5700X), you should be able to run the cards at a good overclock with 1000W. I’ve got mine running at 330W with a -60mV undervolt, with the VRAM at 2750MHz vs 2516MHz stock. In fact, one of the cards can do 2800MHz, but the other one isn’t stable at that speed.

Really makes a difference with Qwen 3.6 27B + MTP, I had it running consistently over 70t/s at one point. Unfortunately, the PP just isn’t there on that model for my agentic usage (especially with MTP), so my daily is still the 35B MoE with 125-130t/s TG at Q6_K_XL without MTP.

Completely unrelated to the hardware there are a whole bunch of model specific MTP bugs in software. VLLM right now has a nasty one where speculative decoding on qwen3 or gemma4 PLUS constrained output (JSON only response for tool calling) can result in invalid json that gets truncated and failed tool calls. This is just one of many many recent examples: [Bugfix] Fix Gemma4 MTP streaming multi-tool calls by whytem · Pull Request #42006 · vllm-project/vllm · GitHub but with thousands of pending PRs across dozens of different libraries and inference engines, its worth noting that you MUST test your specific MTP implementation on your specific model and use case to determine if there are any regressions that impact output quality.

I don’t think there’s anything model-specific about MTP with llama.cpp, but there are two big issues with MTP itself - namely that PP gets an absolute kicking (~30-40% drop), and that it prevents parallel scaling.

These things tend not to matter in a chat context, but they basically prevent it from being useful for agentic work.

MTP is screaming fast for agent work, DS4 at tp4 even in single stream output of 200T/s is wildly fast, you easily get a 2x-3x performance boost over non-MTP. The downside is some processing power is spend running the speculative decode, so IF your token acceptance rate is low (less than 60% lets say) you are often not getting much of a benefit. The draft decode accuracy is again heavily dependent on your individual workload vs what the model was trained for.

No, it isn’t - because decode is the least important part of agentic workloads.

For coding workloads, a perfect-case model is Qwen 3.6 27B. That’s ~35-40t/s normally on my machine, 70-75t/s with MTP. However, unless you’re vibe-coding a one-shot, the model isn’t generating anywhere near as many tokens as it’s reading (context is usually at least one order of magnitude larger than generated output, if not two), so the 30-40% prompt processing loss is far more significant than the 100% generation increase.

With regard to parallel scaling, I’m not talking about the number of GPUs; I’m talking about the number of parallel requests and the way it scales (ie the sub-agent use case). Without MTP, and a notional baseline single-request response of 100t/s to make the numbers easy, llama.cpp will scale to ~250-300t/s throughput at 4 parallel requests, with each client seeing 60-75t/s. With MTP, it remains at 100t/s, so each client only sees 25t/s.

How many sub agents are you spawning up? If you have 4-8 guys running around with delegated tasks you need really good context management and high token output or your jobs take five ever. prompt processing speed is likewise super important because reading large files, parsing logs, or whatever they need to read through is likewise going to constrain TTFT.

You should performance test your workload against VLLM and SGlang for comparison to see what you might be missing out on, especially the vastly better context management, continuous batching and paged attention.

Not really the point, but…I have a number of use cases which make vLLM a non-option, not least the fact that it’s significantly slower than llama.cpp running under Vulkan on R9700s for everything up to the crossover point 6 concurrent requests on my hardware. That’s the problem…for single requests (which are also part of my workflow), vLLM + MTP under ROCm is only roughly the same speed as llama.cpp under Vulkan without MTP.

Things might be different in the CUDA world, but not for AMD (specifically RDNA4).

Then you have the flexibility of llama.cpp with regard to experimentation, which vLLM just can’t compete with, alongside the sheer pain of getting vLLM working with new models on non-CUDA devices.

Basically, it’s just not suitable for my uses.

In any case, this thread is about llama.cpp, hence my original comment.

1 Like

Looks like MTP works with multi-GPU …

Nobody’s questioning that - OP’s already found the problem :slight_smile: This thread’s about llama.cpp, though, not vLLM.

1 Like