On the May 13th Lemonade getting 380-850 t/s and accurate coding results. Ran for 12 hrs unattended and was stable, wrote a bunch of tsx that compiled and ran clean.
I gave llama.cpp a try again last night for giggles on the main system, and its slower on every level unfortunately, decode, prefill, prefix hit loading, etc. Roughly 1/5 prefill and 1/2 decode with much worse endurance across long context, plus it has the kv management overhead reducing efficiency of bandwidth available.
Upcoming exciting news for VLLM users…
In process of validating my new dynamic quantization work, it quantizes, serves, runs. I have run logprob benches, perplexity benches, and throughput benches against it and found a clear and beneficial gain vs pure NVFP4. Recovers ~70% of the loss between FP8 and NVFP4 at the cost of 1bpw of extra space usage compared to NVFP4, or, framed another way is essentially indistinguishable from FP8 output quality during actual inference for only 68% of the space and can be configured to regain ~93% of the loss whilst still saving space vs pure FP8 Block 128 quantization.
Conceptually similar to the many flavors of GGUF out there, which will allow precise vram space targeting for vllm. Will be incorporating a user argument to input a desired memory space target, model folder, and desired context length, it will generate the accuracy settings to maximize quality for available space and provide the quantized model ideal for your footprint available.
Relevant for dual card users as it will allow you to gain ~FP8 accuracy tailored to your GPUs VRAM footprint and simplify the complexity of getting vllm to behave.
Just gave it a go, can confirm im getting 47 tk/s on the nose, wow the performance improvement of MTP is really a gamechanger for how quickly and responsive chats now feel
Too bad Gemma doesn’t have a model with baked-in MTP and llama-server doesn’t support seperate assistant draft models yet, i prefer gemma 31b over qwen 27b for writing/drafting email but the the performance of qwen 27b has now roughly doubled making gemma 31b hard to justify
edit:
it’s all very fresh though, so probably will be included as an option soonish
so it specifically doesnt support gemma4 yet then?
edit (Guess i’ll just have to sit tight for a bit, good to know about the option though i was trying it with –-mtp-head):
Loading model… -0.14.204.289 E llama_model_load: error loading model: unknown model architecture: ‘gemma4_assistant’
0.14.204.306 E llama_model_load_from_file_impl: failed to load model
0.14.204.308 E srv load_model: failed to load draft model, ‘/mnt/steam/models/gemma-4-31B-it-assistant.Q4_K_M.gguf’
I’ve just tried the latest llama.cpp build with the MTP fixes from ggerganov - it’s great. I’m working at the moment, so I’ve only had time to try the Qwen 3.6 35B Q8_0 (my daily-driver model), and…it’s actually pretty good. Prefill is down to ~3200t/s from ~5200t/s, which is still totally acceptable, but decode is consistently up from 105t/s to 130-140t/s in a mix of text and code generation.
Even more awesomely, you can combine speculative decoding strategies, so I’ve got MTP generating model drafts as well as the ngram-mod self-speculation generating drafts from the KV cache. In my usual test (“Show me a combination of two novel algorithms in Ruby” → “Now make it an API using Sinatra. Code only, no commentary”), it’s hitting over 200t/s decode.
In terms of overall feel, it’s a definite improvement in agentic use cases.
Mirostat was slowing this down when I brought my settings over from qwen 3 coder. But now I’m getting 275-550 prefill and 20 t/s generation. When I first switched models to Qwen3.6 was getting about 12 t/s prior to changing this up.
Slower but much higher quality output for agentic coding. As I’ve been trying to tune for speed and quality I’m finding I feel like theres a constant trade-off and you gotta figure out what’s right for you. I’d rather leave some performance on the table in exchange for consistent accurate results.
I’ll have to compare now with a vllm build if lemonades providing it easy I hit instant road blocks with ROCm a month ago trying it.
The only reason that’s in there was because I saw it in here I’ll try without it: llama.cpp - Qwen
I’ve just been generally taking the approach that splitting gives more room for larger context. Are you suggesting change the tensor split to 1, 0? or fully just not even leveraging it (how I’m understanding)
Interesting, I thought I had seen better performance setting this higher, my cpu is fine, I only really see 3 active threads at any given time and only 1 per gpu is doing substantial work.
So I guess my question then is how do I best utilize the other gpu in this scenario just running a smaller model in parallel there?
Well, in my experience, uneven splits have always performed lower than even splits, and even splits have always performed lower than a single GPU (for decode, at least).
I would suggest getting rid of the split entirely, and specifying --device ROCm0 instead. You can’t use tensor split effectively because you have mismatched GPUs, so that would - to me - be the best way to do it.
The only caveat, to my knowledge, is that you might need to quantise the KV cache to q8_0. That’s no real sacrifice, though, and you would only notice the (marginal) difference in quality if your model was Q8 or BF16.
So…yeah, my recommendation would be to put the whole model on the R9700, including MTP layers, whack the context up to the max and quantise it to q8_0. That’s basically how I run it, and I’ve had up to 65t/s out of it in ROCm with MTP.
As for using the other GPU…you could put a smaller model on there with lower context and use it for watchdog work or similar (ie sense-checking the work of the primary model in an agent), or maybe run Qwen3.5-9B without MTP for parallel subagents (since MTP prevents that on the main model).
are you using kv cache quantisation for your agentic work btw? so far i haven’t messed with the ctv and ctk yet beyond a quick test to see the reduction in vram usage, is there any noticeable degradation for agentic work to not use the standard f16 there?
i’ve been running this as my daily driver for the past week:
I’ve run it both ways - when I was running a separate completion model, I used q8_0 on both just to avoid hitting the limit of my VRAM. Now that I’m just running one model, I don’t bother (the Q8_0 GGUF plus MTP and full context comes to about 46.5GB, which still leaves me a bit of room to play with other models should I need it). I haven’t noticed any quality difference, although - and I haven’t tested this exhaustively - prefill seems about 6% faster on average, and MTP decode might be about 2% faster; treat this as entirely anecdotal.
thanks!
