i was just wondering, it seems strange then that we don’t encourage it more/treat it as the default then, i mean it does save around 6-8 gbyte
Well, as I said…unquantised context tends to be faster, and a lot of folk prefer performance over capacity. Ultimately, though, I think it comes down to “default == raw” being the standard across the whole shebang, and then degrading data through compression/quantisation being an active choice.
Which makes sense, really.
Having just discovered this thread after pulling the trigger on two 9700 pros, I’m fascinated by the discussion. However, I’m rather out of my depth when it comes to AMD.
For context, so far, the best performance I have been able to get is using lm studio in windows. For whatever reason, it’s significantly faster than anything I’ve been able to run in Linux. But I’m only getting about 40% utilization on each card at that; a single b70 would be faster if it only had the vram for full context with qwen 3.6. Rocm and Vulcan on Linux both give about 55tps while Vulcan on windows nearly hits 80.
Is there a good guide or a starting place where I can really RTFM on RDNA for AI? I would love to be able to pull anywhere close to these numbers. I did try the custom vllm image above, but I have not been able to get that working either…
Which model are you using?
ROCm vs Vulkan performance depends heavily on the model used (ROCm will be faster for dense models, Vulkan generally faster for MoEs). There are a bunch of optimisations you can do for it too, and there’s a llama.cpp PR in the works to improve the performance of tensor split (although it has some significant bugs at the moment).
One thing worth bearing in mind - the default ROCm installation on Linux replaces Vulkan’s amdgpu driver, which knackers the performance. I’ve found that the best way to run both is to hose everything, install Mesa 26.0 (or 26.2, if you can), and then use Lemonade’s portable llama.cpp distro that includes ROCm. That way, you get the optimal performance from both…apart from the fact that Lemonade’s packages don’t include RCCL.
It depends on the workload. Considering how much faster MoE is generally, I’d lean towards it to start with. I was using qwen 3.6 35b a3b q8_k_p but dropped to standard q6 for testing when I started questioning whether two cards was causing an issue or if it was something more general going on.
I tried a lot of different things just to set a baseline, but there must be something I am missing here. Between Lm Studio/llmster and lemonade, every runtime and software combination gives approx. 55tps on Linux, only Vulkan on Windows is any faster than that, but after doing a quick survey of people I know with 9700 pros, a single card by itself should be faster than this. This was tested with qwen 3.6 35b a3b with no modifications. With the same model, I am getting about half the measured performance of anyone else.
I have not been able to get vLLM to work at all, but even if I could, the general limitations would kind of dissuade me from using it, so I have not been trying too hard to get it working, more of a “oh, well if this works, then maybe”, but RCCL is totally broken even with all the patches floating around to get around that issue, I have not gotten it to work even once.
OK - to give you an idea, with default clocks I get around 5200t/s prefill and 110t/s decode for Unsloth’s Qwen 3.6 35B A3B Q6_K_XL, using llama.cpp with Vulkan (Mesa 26.0).
If you’re not getting near that, then my first questions would be…
- Are the cards on x8 (CPU)/x8 (CPU) slots, or x16 (CPU)/x4 (chipset) slots? If the latter, you’re going to be sacrificing a ton of performance. R9700s (and possibly other GPUs) hate chipset-connected PCIE slots with llama.cpp, losing about 35-40% performance.
- Is your Mesa installation too old, say 25.0? 26.0 has a bunch of optimisations which really help.
- If you run llama.cpp (which you should), then run it with
RADV_DEBUG-nocomputeif you’re not using either card for graphics, it’ll add 5-10% performance.
I’m not getting anywhere near that. It’s 70 tokens/second give or take one. That’s with one or two cards; adding a second does not seem to change anything.
As far as slots, this is a WRX80 platform. All cards are in x16 slot at gen 4, connected to the CPU. I have other GPUs I can add to use for display out if that will help. I had not even thought about that.
I tried a standard Ubuntu 26 installation, as well as windows. Windows gave the best performance, but I did not go fiddling with mesa or anything else in Linux, so I am likely missing quite a few setup steps.
In the BIOS, have you enabled “Above 4G Decoding”? That will enable another feature you need, called ReBAR (make sure it’s enabled).
I also found I needed to disable CSM and force 100% UEFI for the options to appear in Asus WRX80.
You know, I was pretty certain, but you convinced me to go digging in the bios. TL;DR, ReBar was actually off. With it on, I can get 120 tokens/sec at least in Windows, which is a huge improvement. I’m pretty sure this is what I was missing in the end. Thank you!
Only semi-related though, with ReBar on, one GPU consistently crashes while loading or unloading models. If I can get something loaded, it will run, but I have isolated the crash to the specific GPU, it’s not related to OS, driver, slot… So, I will need to RMA this one. Thanks to everyone for all of the help! Once I get this hardware hiccup handled, I should be ready to get back to it.
1 Like
On Linux, I’ve seen others mentioning the kernel parameter: iommu=pt to set it to passive.