I am currently running Muse-Glimmer-30B-UD-Q2_K_XL.gguf using llama.cpp inside of a Proxmox LXC container with GPU pass through for my RX 6800 and so far it has been pretty good.
I have gotten a taste and would like to step up a level to be able to run bigger models on my own hardware but I am unsure what GPU or GPUS I should be aiming for?
A quick lookand it looks like the Intel Arc B60 at $999 AUD is probably the next step to being able to run a bigger model.
Is it worth getting the Intel Arc B60? Are there better step up in terms of price to performance that I should be looking at?
Depends on your other components, but I wouldnāt.
If you have a motherboard that has other x16 slots that wire up x4 and above, then it may be worth running a second 6800, just so you spend less money, have more VRAM, still use the other GPU instead of tossing it aside, and stay on AMD hardware, which is probably much better for support than Intel.
Hell, upgrading to a board with with x16 and x8 electrical slots and getting that 6800 will probably end up way under 1000. I get by with slower token generation, but you may not. Depends on what CPU you have. And the PSU.
A whole grand for a 24GB passive card is a bitch, and checking out Australian eBayā¦ How do you do it, manā¦
24GB is sort of an awkward middle ground. But Q2 to maybe Q5 would be a decent step up I guess. I just upgraded from a 6900 to 9700 & the difference is very noticeable, double the price of the B60 though.
Intel still lags behind quite a lot, especially software support. Nvidiaās modern architecture is the best, then modern AMD, then Intel (a ton of progress has been made, but itās not there yet for Intel).
If you have a kinda tight budget with a real use for local AI, like a fancy local chat bot or coding assistant, or image, video & audio generation, then you should look at used slightly older generation GPUs, or simply upping your budget. If you are going to sink most of a grand into a GPU for local AI I personally think a modern 32GB Nvidia or more likely an AMD card makes more sense even if it means saving up longer or selling some stuff, especially if this is a primary machine and not a 2nd, 3rd or 4th rig.
It really comes down to software you use now and want to use in the near future, Comfy UI, LLMs and other local AI want a lot of fast VRAM as well as fast GPU cores and accelerators, plus every other aspect of a system to a certain extent (SSD speed, PCIe speed & lanes, RAM capacity and speed, CPU cores & speed). Not all of them benefit from multiple GPUs, usually best to have one powerful and new as possible GPU with as much fast VRAM as possible, but budget constraints will dictate which features to prioritize.
16ā24GB of VRAM isnāt going to open up a world of options, especially when a lot of slightly larger models will āfitā with not a lot of room for context, as SUPEROXIDES said a 2nd R6800 may well make more sense if you just want larger LLMs with more room for context and donāt care about other AI uses, plus a B60 while newer is a weaker performer with a bit less memory bandwidth, lower TFLOPS ectā¦ (it is mostly a downgrade or side grade other than VRAM capacity)
I have an Asus TUF GAMING X570-PLUS (WI-FI) with an AMD Ryzen 9 5900X 12-Core Processor and 32GB of DDR4. The motherboard has two PCIe 4.0 x16, one runs at x16 mode and the other at x4 mode. It does look like it supports bifurcation but that appears to only be for M.2 SSDs so I will have to make do with two GPUs at most.
We are a small market so it is what it is. I can see a couple of second hand RX 6900 for less than $400 AUD so getting another one does sound very appealing to the budget side of things.
The R9700 at $2400 is about $500 more than the $1900 B70 which is nearly double the price of the B60 itself. I am wondering if getting two B60 is worth considering or if 48GB likewise becomes another awkward middle ground.
Iāve been using llama.cpp (Vulkan) with Arc B60 for a while. llama.cpp battlemage support is actively developed, the performance and stability is improved over the recent weeks, now it has usable level of stability. Its performance is approx. 1/4 of RTX5090 and IMO it is reasonable for its memory bandwidth.
Well this all sounds like good news. The cheapest RTX 5090 goes for $6875 so I could get nearly 7 B60 for the same price of a single RTX 5090 and have approximately 1.5X the performance.
48GB is a weird VRAM area too unfortunately, itās been whatās stopping me from getting another Intel GPU, ācause Iād have to jump to 64GB from 32 to actually experience some benefits.
When I go by the usability of models, lower quants are viable at higher parameter counts, but Q4KM with high-precision output and embeddings (Q4KL or Q4KXL) are a thumbs up all the time. With 48GB: you canāt fully offload Qwen 3.8 27B at full precision, which some like to do for guaranteed syntax retention for coding and other loss-sensitive tasks; ~100B flash models like Laguna 2.1 S, Mistral 4, Ling 3.0 Flash VL, and Qwen 3.8 Flash Next (when n-gram tables are offloaded to RAM, which you cant do on Intel GPUs on llama.cpp for Vulkan, Iām trying to do it for SYCL right now) will only fully offload at IQ2 if you wanted wanted some context, and even then youād have to get display from onboard memory to ensure you donāt freeze; and finally, the latest models big enough to benefit from the increased memory are Qwen 3 Coder Next (old but gold), andā¦ Kimi Linear (should stay dead).
All of this is assuming you donāt mind offloading some layers to RAM. I do it all the time with those 100B-ish models, and I have no problem using it. You could also check out the GSQ-RCO models, they claim to maintain quality at ~IQ3S sizes, so itās worth a shot. I havenāt done extensive testing with it though.
And a note about the llama.cpp version you use - Vulkan may be the easiest, but itās the slowest in the Intel-compatible selection at prompt processing and token generation. SYCL decodes faster, and OpenVINO processes faster. Iāve found that with the latest versions, SYCL is about the same for processing as Vulkan, and OpenVINOās generation is in the sewer. Itās gonna get better, butā¦ It only just got Qwen 3.X architecture working on it.
Figure out what you want the model to do and how you are going to run it. Iām running Qwen 3.8 27B on an Arc A770 and RX 3060 with llama.cpp. I get about 8 tok/sec. Faster would be nicer but it is functional for my use cases.
The big limitation with running multiple cards is the comparative delay in prompt processing. In this case a vulkan backend and hetrogenious environment mostly works. My experience is limited by PCI bandwith for prompt processing, memory bandwith for inference, total VRAM available for KV cache quality.
