I need an AI sanity check (9700 Pro x2)

[Strawberry]

Hello all! I have some curious questions about RDNA, my system, and software, and when you put them all together, it sounds a lot like a sanity check, so that’s what I am requesting.

Some background. AI is cool, I’ve dabbled a little and want to do something a little more serious. I obtained a threadripper system, and not wanting to spend 5 figures each on the GPUs to match, I bought two 9700 Pro cards. (I’m still not sure if that was a mistake or not.) To make a long story short, this has been the most frustrating experience I have had in a very long time (And I deal with Oracle software for a living).

My process looks something like this. Find the easiest/simplest thing that can possibly work, use it as a jumping off point, and build on that. Start with Hello World.

So far, the only thing that works AT ALL is LM-Studio, installed via the beta .deb distribution, on Linux. The best I can get is 70 tokens/sec with it, that’s with the latest Mesa on Ubuntu 26.04. Every other combination is far lower, if it even works at all. (GPU utilization gets to 33% at best.)

I have tried: using Windows, using Linux, with X, with Wayland, bare metal, in a VM, various containers using Docker/LXC, and every possible permutation of the above.

These GPUs are intended to sit in the corner and serve an OpenAI compatible endpoint with the most reasonable model I can fit on them at full context, so other services like hermes, turnstone, cursor, cline etc can all consume it. It makes sense to use whatever software happens to be the fastest, and I really wanted Tensor Parallelism. I mean, it’s half the reason to go with multiple GPUs. And for that you need vLLM.

I have never once gotten vLLM to do anything. I have spent a week trying. There is no playbook in the “AMD Playbooks” for the 9700 Pro either. There’s basically nothing you can’t get if you just installed Lemonade and didn’t bother looking in the playbooks. (Lemonade is also very slow on every platform.)

If I were to go off my personal experience alone, everyone claiming to run real AI workloads on non-nvidia consumer hardware, is being paid to do so, and it’s all just a giant marketing lie.

So here is the sanity check. Either everyone is lying, or I am missing something, and I’d bet on my own ignorance any day. Is there a start to finish guide anywhere that doesn’t assume any prior knowledge that I can use to “hello world” vLLM? Some kind of fail-safe configuration I can build on? Some examples I can dig through to maybe shore up my knowledge? I’m about ready to just give up on it, and it takes a lot for me to get to that point.

[k3ninho]

You’ve not mentioned the AMD ROCm drivers at all. How was your experience installing them?

The Strix Halo Wiki and Kyuzo’s Strix Halo Toolboxes are general enough for ROCm use that I’ve used them with my Phoenix APU as well as older Ryzen 5600’s. (I can’t find the source but) I’ve also got the AMD ROCm Nightlies from https://rocm.nightlies.amd.com/deb/ at the most-recent date in my apt sources on Debian so that it’ll install from there. Debian Wiki has bare steps for llama.cpp at the bottom of https://wiki.debian.org/ROCm.

[vitabis]

I recently posted about my level up experience with NVIDIA, focussing more on the build options.

@Strawberry

GPU utilization gets to 33% at best

With the RTX 3090’s I spotted similar utilization when CUDA graphs were not enabled in vLLM.

I see ROCm has its custom flavor of vLLM, maybe AMD has (to develop) their own special sauce to achieve the same?

[digitalscream]

OK, as always, a few questions…

• What models are you trying to run?
• What do you want to do with those models? Agentic stuff/code/just chat/all of the above/etc?
• How many parallel requests do you plan on actually using (ie not just for benchmarks)?
• What’s your reasoning in trying to get vLLM running?
• Have you tried llama.cpp?

Tensor parallelism isn’t the be-all-and-end-all. In fact, there are a number of cases where it’s not even the best option.

EDIT: OK, I’ve got a few minutes here, so why not…

• vLLM is not the best option for anything except when you’re handling 8-16+ concurrent requests. It’s also the most painful to set up, so…
• …llama.cpp is your best bet. Download the binaries for Ubuntu from the GitHub releases.
• If you’re running dense models (eg Qwen 3.6 27B), then ROCm + tensor split will be faster by about 25%.
• If you’re running MoE models (eg Qwen 3.6 35B), then Vulkan + layer split will be faster by about 50%. Don’t forget to set RADV_DEBUG=nocompute when launching the server.
• If you’re not running parallel requests at all, then use MTP for a significant boost in TG, although you’ll pay a penalty in PP.
• If you have the power budget available, use LACT to overclock the VRAM to ~2750MHz, and up the power limit to 330W.

To give you an idea, on my dual R9700 system…

• Qwen 3.6 27B
  • llama.cpp + ROCm + tensor split = 1800t/s PP, 40t/s TG
  • llama.cpp + ROCm + tensor split + MTP = 1200t/s PP, 75-80t/s TG
  • llama.cpp + Vulkan + layer split = 1600t/s PP, 30t/s TG
  • llama.cpp + Vulkan + layer split + MTP = 1100t/s PP, 45-55t/s TG
• Qwen 3.6 35B A3B
  • llama.cpp + ROCm + tensor split = 3100t/s PP, 70t/s TG
  • llama.cpp + ROCm + layer split = 3700t/s PP, 70t/s TG
  • llama.cpp + Vulkan + layer split = 5400t/s PP, 120t/s TG
  • llama.cpp + Vulkan + layer split + MTP = 2900t/s PP, 140-180t/s TG

Note that this is using the Lemonade ROCm distribution of llama.cpp - I haven’t got ROCm installed on the system, because it seems to override the amdgpu driver and causes performance problems with Vulkan (and my main models are all MoE, so Vulkan is my priority).

[Strawberry]

None of the instructions I have seen online cover this as a step. My understanding is that the required software is all bundled in the various container distributions, you allow the container to access the raw hardware, and you don’t install anything on the host. To answer the question more directly; I never did this.

[Strawberry]

Answers:
I’d like to run Qwen3.6 or future models in that size range.
Agentic stuff, coding stuff. I’d like agents to do maintenance, and coding assistants to help with coding. Or just do everything, if that’s in the realm of possibility. (I know it’s close)
I’d like to run many operations in parallel if it’s at all possible. Usually 1 or 2, but up to 8, depending on needs.
My reasoning is (and I might be wrong) that using tensor parallelism, you can fully utilize the hardware because you don’t need the cards to constantly back and forth, you front load the memory division.
I have not tried llama.cpp outside of LM-Studio because from my understanding it’s the same thing under the hood, just without an easy gui to configure it. (I may also be wrong here)

VERY interesting. I’ve been trying to run Qwen3.6-35B-A3B at Q6/Q8 and Qwen3.6-27B at Q6/Q8 to start with. I have run it on Nvidia/Intel before so it’s my “hello world” test here. LM-Studio uses llama.cpp with Vulkan and layer split, I’m not using an MTP model at the moment. You’re getting 120t/g TG while I get 70. But otherwise, in relative terms I see basically the same results you do, just slower. I will have to see about the VRAM overclocking you mentioned as I have not even thought about that. With the power consumption at under 50% on these cards, I have plenty of headroom…

According to your statement here, I’m probably doing just fine. Maybe a bit mysteriously slow, but without an overclock, and limited to PCIe gen4, maybe that explains it?

[digitalscream]

@Strawberry

I’d like to run Qwen3.6 or future models in that size range.

Size isn’t everything, active parameters are. Dense models like Qwen 3.6 27B activate all the parameters for every token, whereas Mixture of Experts models like Qwen 3.6 35B A3B only activate the “A3B” part - so 3B parameters. That’s why they run faster with a higher total parameter count.

@Strawberry

My reasoning is (and I might be wrong) that using tensor parallelism, you can fully utilize the hardware because you don’t need the cards to constantly back and forth, you front load the memory division.

The logic is based on a faulty assumption - that all models and architectures behave the same way, and therefore tensor parallelism will always be faster. The fact is that llama.cpp layer split with Vulkan is faster than any implementation that uses ROCm…for MoE models and smaller dense models, basically up to around 20B active parameters. Past that point, ROCm with tensor parallelism tends to take over.

I honestly wouldn’t bother with ROCm - for your uses, most of the time, Vulkan will likely be faster if you’re using the 35B MoE model. If you want to experiment with ROCm, then remove it from your system (and reinstall Mesa), and then use the Lemonade llamacpp-rocm portable distribution instead. Much easier to manage.

@Strawberry

I have not tried llama.cpp outside of LM-Studio because from my understanding it’s the same thing under the hood, just without an easy gui to configure it. (I may also be wrong here)

Again, I’m afraid that’s a faulty (albeit totally understandable) assumption. The GUI abstracts away a lot of the tuneable parameters of llama.cpp, to the point where it’s almost always ~40% slower than llama.cpp running raw. It’s always a fair bit behind the latest nightly builds of llama.cpp, and they release 3-4 times per day. A lot of optimisations go in every week.

@Strawberry

According to your statement here, I’m probably doing just fine. Maybe a bit mysteriously slow, but without an overclock, and limited to PCIe gen4, maybe that explains it?

Not really - I’m running my R9700s on PCIE 4.0 x8 each. The overclocks help by around 10%, but that’s it.

My advice - download the latest Ubuntu/Vulkan release of llama.cpp from their GitHub repo. Run llama-server in router mode (ie don’t point it at a model, point it at a directory containing your GGUFs), and see what that gives you - router mode comes with a web UI built in. I run it as a systemd service:

[Unit]
Description=Start llama.cpp service
After=network.target

[Service]
Type=simple
User=lj
Environment="RADV_DEBUG=nocompute"
WorkingDirectory=/path/to/llama.cpp/directory
ExecStart=/path/to/llama.cpp/directory/llama-server --host 0.0.0.0 --port 8080 --models-dir /path/to/models/directory --models-preset /path/to/models/directory/config.ini --models-max 1 --cache-ram 16384
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target

The RADV_DEBUG=nocompute is really important if you’re on Mesa 26.0 or above and you’re not using either of the GPUs for actual graphics - the graphics queue on the R9700 is faster than the compute queue. That’s usually worth a good 5-10% boost in TG, but if you’re using either GPU for a GUI then you’ll find it stutters like hell if you use this option.

The config.ini is where you configure all the model-specific optimisations (you can select the loaded model from within the web UI). It basically consists of repeated sections that look like this, which is basically just the command line arguments for llama-server organised by model:

[agent_35B]
mmap=0
ctx-size=262144
flash-attn=on
batch-size=2048
ubatch-size=1024
model=/path/to/models/directory/Qwen3.6-35B-A3B-UD-Q6_K_XL.gguf
n-predict=-1
n-gpu-layers=999
threads=1
top-p       = 0.95
top-k       = 20
temp        = 0.6
repeat-penalty = 1.0
split-mode = layer
reasoning = off
spec-type = draft-mtp
spec-draft-n-max = 3

(you probably don’t want to use MTP, since you want parallel execution, so get rid of the last two lines)

Finally…don’t forget that LACT can be managed remotely. Very handy…but can also result in doing silly things and not being able to immediately recover. I tripped the protection in my 850W power supply a few times before I realised that running in tensor split mode pushed both GPUs to the limit…and the actual usage is usually 70W+ past what nvtop is telling you (upwards of 400W each when the limit’s set at 330W). Given that you’re running a Threadripper 3000 series and a pair of R9700s, you absolutely don’t want to do that unless you’ve got a 1200W+ PSU.

Incidentally, I found that one of my GPUs could do 2800MHz VRAM, but the other can only manage 2750MHz. I erred on the side of caution and went with the lower value, as well as a 50mV undervolt.

Hope that’s enough to get you started

[digitalscream]

The big tl;dr - there’s a price to be paid for everything. If you cheap out on GPUs, like I did, then you’ll pay for it with the time involved in learning how to get the best out of them.

I learned this the hard way…with Intel GPUs

[Strawberry]

@digitalscream

Size isn’t everything, active parameters are.

Noted! I have noticed that I get way more consistent results from the 35B-A3B when activating more of the “experts”, up to a point.

@digitalscream

The fact is that llama.cpp layer split with Vulkan is faster than any implementation that uses ROCm…for MoE models and smaller dense models

This should be more common knowledge than it is, I think.

@digitalscream

If you want to experiment with ROCm, then remove it from your system (and reinstall Mesa), and then use the Lemonade llamacpp-rocm portable distribution instead.

This should also be on page one of the “So you bought AMD cards for AI” quickstart guide (that IIRC does not exist)

@digitalscream

it’s almost always ~40% slower than llama.cpp running raw.

120 tokens/sec
70 tokens/sec
~napkin math

@digitalscream

router mode comes with a web UI built in. I run it as a systemd service:

Now THAT is something I was not aware of. Definitely giving this a try today.

@digitalscream

The RADV_DEBUG=nocompute is really important if you’re on Mesa 26.0 or above and you’re not using either of the GPUs for actual graphics

Both of those conditions are true; this is supposed to be a headless server after all.

@digitalscream

the actual usage is usually 70W+ past what nvtop is telling you (upwards of 400W each when the limit’s set at 330W). Given that you’re running a Threadripper 3000 series and a pair of R9700s, you absolutely don’t want to do that unless you’ve got a 1200W+ PSU.

I have a 1200w PSU, but one correction, it’s a threadripper 5975wx. I also have monitoring at the wall, so if the power consumption is not recorded correctly, that’s fine.

Thank you so much for this fantastic guide! I will be trying all of this tonight, and hopefully it helps more people too, this is so plainly laid out, and I’m sure I’m not the only person confused by AMD’s software zoo.

[digitalscream]

@Strawberry

I have a 1200w PSU, but one correction, it’s a threadripper 5975wx. I also have monitoring at the wall, so if the power consumption is not recorded correctly, that’s fine.

D’oh, fair point - I forgot the 5xxx series used PCIE 4.0 too.

Still, baller rig you’ve got there

@Strawberry

@digitalscream

If you want to experiment with ROCm, then remove it from your system (and reinstall Mesa), and then use the Lemonade llamacpp-rocm portable distribution instead.

This should also be on page one of the “So you bought AMD cards for AI” quickstart guide (that IIRC does not exist)

One thing you should know about the Lemonade build is that it doesn’t include RCCL (AMD’s version of NCCL for P2P), which should - in theory - allow P2P communication when running in tensor split mode. However, I’ve not seen anybody manage to demonstrate any additional performance out of it with or without RCCL, at least not with llama.cpp. The build notes for llama.cpp + ROCm state that it’s hidden behind a compilation flag for that very reason - often it hurts performance rather than helps.

@Strawberry

Thank you so much for this fantastic guide! I will be trying all of this tonight, and hopefully it helps more people too, this is so plainly laid out, and I’m sure I’m not the only person confused by AMD’s software zoo.

No worries, happy to help. I wrote up something similar in another thread, I just couldn’t find it. In a couple of weeks (after I leave my current job) I’ll have a few weeks off, so I might just put something together to try to document everything I’ve got stored in my noodle about this.

One thing you’ll find is that not being in the Nvidia ecosystem can be really annoying, but also much cheaper. However…if you think the AMD world is a zoo, you should be really happy you didn’t buy Intel cards

[Strawberry]

@digitalscream

you should be really happy you didn’t buy Intel cards

Oh, uh, I did buy quite a few of them actually, but those are mostly in use on Windows systems. The only one I’m using for AI is a single b70, and, as long as I stick with Windows it’s downright mostly okay. That’s a different problem for a different day.

[digitalscream]

Well, to be fair, I’ve got an A770 16GB in my main desktop - and it’s genuinely great at the job of being a graphics card. It was one of a pair that I bought for AI purposes, and I gave up after Intel abandoned their third AI framework.

Sold one, put the other in my daily machine, and it’s been flawless ever since.

[Strawberry]

An update. This is working amazingly well. Performance is up, issues are down. Compiling for CUDA means I can run a second instance headless using a spare A4000. I’ve also discovered that I can run deepseek v4 flash purely in system RAM/CPU if I don’t need an answer right away using a third instance. And my PSU can only barely keep up with that. I may end up needing to upgrade it, which is a great problem to have.

@digitalscream

Still, baller rig you’ve got there

It wasn’t very useful before your advice. But yes, I am now VERY happy with how this part turned out. Now to set up Turnstone and all the other goodies!

[digitalscream]

@Strawberry

It wasn’t very useful before your advice. But yes, I am now VERY happy with how this part turned out. Now to set up Turnstone and all the other goodies!

I won’t lie…I’m significantly relieved. There was a small-but-non-zero chance that, after typing all of that opinionated stuff out, it still wouldn’t work for you…and then I’d probably just have to delete my account

Glad you’re up and running, though. Now the hard part happens…trying to do something useful with it

Don’t forget to stay up-to-date with the llama.cpp releases. I make a point of regularly checking current PRs and discussions on the repo just to see what’s coming down the pipe. I also keep all the releases in a directory, and then just symlink latest to the most recent one then point the service at the latest directory; with 3-4+ releases per day, performance and/or stability does drop occasionally, so it’s handy to have a binary history kicking around to revert if necessary.

[Waylon_R]

Claude says (we fought this fight together):

You’re not missing something obvious, and it’s not a marketing lie. You walked into a real, specific, still-unfixed bug — and then a second one on top of it. I run the same two cards (dual R9700, Z690), so this is all first-hand.

Problem 1: you’re on Ubuntu 26.04.

ROCm 7.2.x supports 24.04 LTS. It does not support 26.04. If you’ve been on 26.04 this whole time, you likely never had a working ROCm userspace at all — which means vLLM was structurally impossible no matter what else you did. LM-Studio “working” is consistent with this: it’s running llama.cpp on the Vulkan/Mesa backend, which doesn’t touch ROCm. That’s why it’s the only thing that works.

Go to Ubuntu 24.04.4, install ROCm 7.2.x via amdgpu-install, and confirm rocminfo lists two gfx1201 agents before you try anything else. Do not set HSA_OVERRIDE_GFX_VERSION — the R9700 is natively supported and overriding it breaks things.

Kernel cmdline (Threadripper): amd_iommu=on iommu=pt pcie_aspm=off amdgpu.runpm=0. The last one keeps the second card from dropping into a low-power state where it stops enumerating properly.

Problem 2: the RCCL tuning-index deadlock. This is the one that ate your week.

RCCL in ROCm 7.2.x (librccl.so.1.0.70201) is missing the tuning-table mapping for gfx1201. It silently falls back to tuning index 0 — which is the table for MI50/gfx906 — and that selects an LL-protocol AllReduce that deadlocks forever on RDNA4. Both GPUs pin at 100% and nothing ever comes back. Community bisect confirms it: librccl.so.1.0.70101 (ROCm 7.1.1) works, 70201 (7.2.x) hangs.

The fix is one environment variable:

-e NCCL_PROTO=Simple

That’s it. That’s the week.

Important details people get wrong:

• NCCL_P2P_DISABLE=1 works for rccl-tests but not for vLLM. Different bug.
• --enforce-eager, --disable-custom-all-reduce, and VLLM_WORKER_MULTIPROC_METHOD=spawn do not help the deadlock.
• Alternative if you’d rather not carry an env var: run a pre-7.2 image such as rocm/vllm-dev:upstream_preview_releases_v0.17.0_20260303 (RCCL 2.26.6), which doesn’t have the regression.

Status as of today: the permanent fix is RCCL PR #2166 (“Navi4 default tuning enablement”). It was opened March 4, approved by a maintainer March 12, and is still open and unmerged. Production ROCm is 7.2.4 and still ships RCCL 2.27.7. So assume every 7.2.x release has this bug until you see it in a changelog.

The guide you need is already on this forum: Wendell’s AMD Radeon R9700 Linux TP2 and TP4 — the Definitive Guide (/t/amd-radeon-r9700-linux-tp2-and-tp4-the-definitive-guide/250504). TP=2 and TP=4 working switch-free on plain root-complex topology, ~65 tok/s on Llama-3.1-8B at TP=2. Also worth reading: Dual R9700’s AMD VLLM-Container woes (/t/dual-r9700s-amd-vllm-container-woes/250053).

Two things that aren’t bugs, so you stop chasing them:

1. “70 tok/s, GPU utilization 33%.” That’s normal. Single-stream autoregressive decode is memory-bandwidth-bound, not compute-bound. 30–40% utilization is what a healthy card looks like doing that work. It is not a symptom of anything.

2. Tensor parallelism is not the speedup you think it is. TP=2 on an 8B gets you ~65 tok/s vs ~61 at TP=4 — the cross-card traffic eats most of the win on a root-complex topology. TP’s real value is fitting a model that doesn’t fit on one card. If your actual goal is “serve an OpenAI-compatible endpoint at full context for cline/cursor/etc,” strongly consider llama.cpp + llama-swap with one model per card instead. Less exotic, doesn’t touch RCCL at all, and for a single-user endpoint it will likely feel the same or better.

One warning before you get TP working: tensor-split is the only mode that drives both cards to full power simultaneously. R9700 is rated 300W TBP but transients spike toward ~400W. People on this forum have tripped OCP on an 850W PSU doing exactly this, right at the moment TP finally started working — and then blamed the software. If you get a hard reboot the first time TP succeeds, look at the PSU, not RCCL. LACT can cap power if you need a stopgap.

Good luck. The frustration is warranted; the conclusion isn’t.

[digitalscream]

@Waylon_R

People on this forum have tripped OCP on an 850W PSU doing exactly this, right at the moment TP finally started working — and then blamed the software

Not true - that was me, and I didn’t blame the software. I upgraded the PSU to 1000W, and the problem went away

@Waylon_R

“70 tok/s, GPU utilization 33%.” That’s normal. Single-stream autoregressive decode is memory-bandwidth-bound, not compute-bound. 30–40% utilization is what a healthy card looks like doing that work. It is not a symptom of anything.

That’s if you’re using vLLM. If you’re using llama.cpp with Vulkan, performance in both PP and TG is significantly higher (like, +40%) than anything you’ll get out of vLLM + ROCm up to somewhere between 4 and 8 parallel requests, even when you’re comparing tensor split on vLLM with layer split on llama.cpp.

One interesting thing…

@Waylon_R

You’re not missing something obvious, and it’s not a marketing lie. You walked into a real, specific, still-unfixed bug — and then a second one on top of it. I run the same two cards (dual R9700, Z690), so this is all first-hand.

If you’re running a Z690, I’m guessing one of your cards is in an x4 slot hanging off the chipset? AMD cards really don’t like the additional latency of hopping through the chipset; at least, the R9700s don’t. I swapped the board and CPU out to get x8/x8, and my second card problems went away.

[Waylon_R]

Howdy,

Claude wrote that and he’s harsh some times. He apologized: “He’s right on the PSU line, and that one’s on me — “and then blamed the software” was an editorializing flourish I added that wasn’t in the source material. Concede it cleanly. His 1000W upgrade is also independent confirmation that your own 1000W GPU rail sizing was load-bearing, not paranoid.”

I’m using dual psu: 1000w for the gpus. the board i’m using is msi unify, so it’s pcei gen 5 x8x8. I have a new am5 board, but that won’t work with 2 cards (rccl/p2p). i bought a pex88048 off amazon or somewhere and the firmware was wrong. so, i have the bases and cables for gen 4 x16x16, just need a card.

Now the part I actually want to dig into. +40% for llama.cpp/Vulkan over vLLM/ROCm on both PP and TG, holding up to 4–8 parallel requests, even against tensor split — that’s a much bigger claim than “llama.cpp is fine for single-user,” and it matches what I’ve been seeing. I’ve been running llama.cpp/Vulkan single-card and had assumed I was trading throughput for stability. If the Vulkan path is actually faster out to 8 concurrent, then the whole “you need vLLM for real work” premise the OP is operating under is just?

Can you say more about what you measured?

• Model + quant (I’m mostly on Qwen3 14B-Q8 / 32B-Q6)
• Where the crossover actually lands — is 8 the point where vLLM pulls ahead, or is that just where you stopped testing?
• llama.cpp build/backend (RADV vs amdvlk, and whether you had -fa on)
• PP and TG separately if you have them — I’d expect vLLM’s chunked prefill to be its strongest card, so PP is the number I find most surprising.

And one clarification on my own point, since I said it badly: I still don’t think 33% utilization means anything diagnostically under either backend — it’s a bandwidth-bound workload, the number is going to look low regardless. But you’re right that I used it to wave off a performance ceiling that’s real. The useful framing is the tok/s, not the utilization, and if llama.cpp gets him materially more of them then he’s already on the faster path and doesn’t know it.

[Waylon_R]

Worth putting the reasoning on the record, because I think it reframes the OP’s whole problem.

When I built my stack (dual R9700, RAG pipeline, OpenAI-compatible endpoint — basically the OP’s stated goal) I went Vulkan-only for decode and deliberately did not use ROCm for serving. Four reasons, in descending order of how load-bearing they turned out to be:

1. Vulkan is just faster on RDNA4. ~183 tok/s Vulkan vs ~150 tok/s ROCm HIP on Qwen3-30B-A3B Q4_K_M (llama.cpp GitHub Discussion #19890), corroborated by Phoronix’s “AMD ROCm 7.1 vs. RADV Vulkan For Llama.cpp With The Radeon AI PRO R9700.” I’ve been carrying 35–45% on decode as the working figure. @digitalscream’s +40% lands right on top of that, from an independent stack.

2. Flash Attention works on the Vulkan path. The Vulkan FA shader path is independent of the ROCm CK kernels that HIP uses, so -fa is available and validated on gfx1201. Non-trivial: it cut my 16K compute buffer by ~86%, +17% decode, +27% prefill. That’s the difference between fitting a model at full context and not.

3. FP8 is broken on gfx1201 in the 7.2.x stack. vLLM mis-detects it and silently dequantizes to FP32 (vllm #28649); AITER/TransformerEngine silently falls back rather than erroring (ROCm #520). Note silently — you don’t get told, you just get a slow model and no explanation. If you’re chasing vLLM specifically to get FP8 throughput on these cards right now, you’re chasing something that isn’t there.

4. No collectives = no RCCL = no bug. Every model I run fits on one 32GB card, so nothing ever crosses cards. Which means the tuning-index deadlock that ate the OP’s week isn’t something I work around — it’s out of scope by construction. That’s an architectural choice with real leverage: you can design the entire class of problem out of existence rather than env-var your way past it.

[Strawberry]

From what I am seeing this is pretty much correct. For the vast, vast majority of workloads you’d see in the real world in a labbing context, Vulkan + llama.cpp is THE way to go. At the moment it is just massively faster, and works with multiple GPUs without the headaches vLLM causes.

I’m getting 100 tg/sec and almost 5k pp, with two 9700 pro cards running Qwen3.6-35B-A3B at Q8 with unquantized context, it feels quite good to work with. Everything else was either slower, a headache to manage, or just did not work outright.

[RealPjotr]

Just to let you know what I run…

I got an R9700 for my main Proxmox host, an Epyc system. I passed the card to a VM specifically for all my ai applications (I have run models on CPU to dip my toes first). The VM runs Ubuntu 24.04, has 128 GB RAM. I used Gemini 3.1 Pro to help me set everything up.

I got ROCm installed, 7.2.2. I tried some llama.cpp images, then moved to vLLM, but it kept failing. In the end I built my own llamap.cpp image (for ROCm with AVX512) and have a compose run the container. I mainly use Qwen 3.6 27B Q6_K MTP with Q8 caches and 140k context.

It works really well. I run my own Forgejo instance for images, code and everything. I’ve setup workflows for PR Review. I have a separate docker swarm for “production” where I setup an Opencode container. I made a workflow that picks up any issue assigned to the BotOpenCode account and implements the issue, it ends up making a PR with explanation and logs. It gets PR reviewed, I’m working on followup workflows. I use Zoo Code agentic in Visual Code for hands on development, with a Searxng MCP container for web searches.

I also setup Hermes in a separate small development VM. Hermes has full access to my forge via its own BotHermes account. It can also make PRs, review code, etc. I talk to Hermes via Slack integration. I might try Pi.dev later.

I also started playing with ComfyUI for image and video generation.

I’ve now ordered a second R9700!