Benchmark Results (while leaving plenty of room for VMs and the like)
Note: These are the DENSE models. The A35B models are quite fast but this is more to show the benefits of ROCmFP4 on dense models where the effect is more pronounced.
I think I still recommend Q8 for agentic dev work. However, for everything else, this works great. And for some kinds of agentic dev work this can also work fine.
If you have AMD hardware, it is always worth checking out Lemonade.ai
Hardware: AMD Ryzen AI MAX+ 395 (Strix Halo) with integrated Radeon 8060S Graphics (gfx1151) — 16 cores / 32 threads, 60 GiB unified system memory. This is an APU, so the GPU shares system RAM via GTT (no dedicated VRAM).
| Component | Size |
|---|---|
| ROCmFP4 model weights | 13.79 GiB |
| KV cache (trunk, 64 layers, 256k ctx, Q4_0) | ~16 GiB |
| KV cache (MTP head, 1 layer, 256k ctx, Q4_0) | ~0.25 GiB |
| Total with MTP at 256k | ~30 GiB |
| System RAM | 60 GiB |
| Room remaining for VMs | ~30 GiB |
Benchmark Results — AMD Ryzen AI MAX+ 395 (Strix Halo) / Radeon 8060S (gfx1151)
System: 16C/32T, 60 GiB unified memory. GPU shares system RAM via GTT (no dedicated VRAM).
Environment: All rows use -fa 1 -ctk q4_0 -ctv q4_0. ROCmFP4 fork rows also use HSA_OVERRIDE_GFX_VERSION=11.5.1 and GGML_HIP_ENABLE_UNIFIED_MEMORY=1.
Benchmark method: Rows 1–9 use llama-bench -p 512 -n 256. Row 10 uses llama-server with real prompts.
Memory at 256k context:
| Config | Model Weights | KV Cache | Total GPU | Free for VMs |
|---|---|---|---|---|
| ROCmFP4 27B + MTP | 13.79 GiB | ~16 GiB | ~30 GiB | ~30 GiB |
| Q4_K_M 27B | 16.74 GiB | ~16 GiB | ~33 GiB | ~27 GiB |
| Q8_0 27B | 27.11 GiB | ~16 GiB | ~43 GiB | ~17 GiB |
| Q4_K_M 35B MoE | 20.60 GiB | ~8 GiB | ~29 GiB | ~31 GiB |
| Q8_0 35B MoE | 35.00 GiB | ~8 GiB | ~43 GiB | ~17 GiB |
Results
| # | Build | Model | Quant | Size | KV Offload | MTP | PP t/s | TG t/s |
|---|---|---|---|---|---|---|---|---|
| 1 | Stock b9542 | Qwen3.6-27B | Q8_0 | 27.11 GiB | GPU | no | 288 | 7.8 |
| 2 | Stock b9542 | Qwen3.6-27B | Q8_0 | 27.11 GiB | CPU | no | 259 | 6.0 |
| 3 | Stock b9542 | Qwen3.6-27B | Q4_K_M | 16.74 GiB | GPU | no | 312 | 11.8 |
| 4 | Lemonade ROCm b1290 | Qwen3.6-27B | Q4_K_M | 16.74 GiB | GPU | no | 310 | 11.7 |
| 5 | ROCmFP4 fork b9219 | Qwopus3.6 Chadrock | ROCmFP4 | 13.79 GiB | GPU | no | 332 | 14.0 |
| 6 | ROCmFP4 fork b9219 | Qwopus3.6 Chadrock | ROCmFP4 | 13.79 GiB | GPU | yes  |
— | 23–36 |
| 7 | Stock b9542 | Qwen3.6-35B-A3B | Q8_0 | ~40.00 GiB | GPU | no | 337¹ | 30¹ |
| 8 | Stock b9542 | Qwen3.6-35B-A3B | Q4_K_M | 20.60 GiB | GPU | no | 802 | 50.6 |
| 9 | ROCmFP4 fork b9219 | Qwen3.6-35B-A3B | Q4_K_M | 20.60 GiB | GPU | no | 859 | 49.9 |
¹ Q8_0 35B MoE: model had some quirks, sometimes 600s+ loadtime on this hardware because llama.cpp tries to load into ram before vram even with the option set to skip that. TG ~30 t/s based on Q4_K_M scaling (50.6 × 20.6/35 ≈ 30).
Key takeaways:
- ROCmFP4 27B + MTP (Row 6) is the best dense option: 23–36 t/s depending on prompt, with only 13.79 GiB for weights.
- Q4_K_M 35B MoE (Rows 8–9) dominates on throughput: 50+ t/s at 20.6 GiB, with ~3B active params per token. This is the fastest configuration by a wide margin.
- Q8_0 27B (Row 1) is the slowest dense option at 7.8 t/s, but offers highest quality at 27 GiB.
- Lemonade ROCm (Row 4) is identical to stock — no special AMD tuning in the binary.