Arc Pro B60 for Local LLMs

long.barn4823 · March 4, 2026, 12:15pm

A thread for me to throw some benchmarks of the performance I’m getting from this card. I purchased it to experiment with LLMs and SRIOV. I’ve seen a few questions about this in other places. I’ll put this here so that its separate from the discussions about the BattleMatrix setup.

System:

Using llama.cpp build 8175 SYCL I get the following numbers

Qwen3.5 9B Q4 K XL

llama-cpp-sycl
–bench -m /models/Qwen3.5-9B-UD-Q4_K_XL.gguf -ngl 100 -r 3 -p 2048,16384 -o md
load_backend: loaded SYCL backend from /app/libggml-sycl.so
load_backend: loaded CPU backend from /app/libggml-cpu-alderlake.so

model	size	params	backend	ngl	test	t/s
qwen35 ?B Q4_K - Medium	5.55 GiB	8.95 B	SYCL	100	pp2048	328.71 ± 11.00
qwen35 ?B Q4_K - Medium	5.55 GiB	8.95 B	SYCL	100	pp16384	317.41 ± 1.76
qwen35 ?B Q4_K - Medium	5.55 GiB	8.95 B	SYCL	100	tg128	23.01 ± 0.03

Qwen3.5 35B-A3B Q4 K XL

llama-cpp-sycl
–bench -m /models/Qwen3.5-35B-A3B-UD-Q4_K_XL.gguf -ngl 100 -r 3 -p 2048,16384 -o md
load_backend: loaded SYCL backend from /app/libggml-sycl.so
load_backend: loaded CPU backend from /app/libggml-cpu-alderlake.so

model	size	params	backend	ngl	test	t/s
qwen35moe ?B Q8_0	19.16 GiB	34.66 B	SYCL	100	pp2048	112.62 ± 0.93
qwen35moe ?B Q8_0	19.16 GiB	34.66 B	SYCL	100	pp16384	111.61 ± 1.11
qwen35moe ?B Q8_0	19.16 GiB	34.66 B	SYCL	100	tg128	8.34 ± 0.22

For reference of performance here is Qwen3.5 35B-A3B running on my RTX Pro 4500 (obv more expensive card but both nominally 200W GPUs)

local/llama.cpp:full-cuda
–bench -m /models/Qwen3.5-35B-A3B-UD-Q4_K_XL.gguf -r 3 -p 2048,16384 -o md
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA RTX PRO 4500 Blackwell, compute capability 12.0, VMM: yes
load_backend: loaded CUDA backend from /app/libggml-cuda.so
load_backend: loaded CPU backend from /app/libggml-cpu-alderlake.so

model	size	params	backend	ngl	test	t/s
qwen35moe ?B Q8_0	19.16 GiB	34.66 B	CUDA	99	pp2048	3807.62 ± 143.98
qwen35moe ?B Q8_0	19.16 GiB	34.66 B	CUDA	99	pp16384	3185.18 ± 15.34
qwen35moe ?B Q8_0	19.16 GiB	34.66 B	CUDA	99	tg128	133.47 ± 0.33

(I’m doing intentionally default config here for easy comparisons, I can tweak the batch/microbatch configs and get more t/s out of the nvidia card, but it wouldn’t be a reasonable comparison)

My experience is that the B60 is ok for a chat length context, on smaller models. A card like this with opencode/claude code ends up with you waiting 5 or 10mins for it to start responding to the initial prompt.

I am happy to run some other llama-bench on the card if folks have something of interest.

doodle-robot · March 10, 2026, 10:43pm

have you tried vulkan backend for the intel card? it seems much better for my a750 ive been testing. im a rookie though

long.barn4823 · March 12, 2026, 1:16pm

Qwen3.5 9B Q4 K XL - Vulkan

llama-cpp-vulkan-bench
–bench -m /models/Qwen3.5-9B-UD-Q4_K_XL.gguf -ngl 100 -r 3 -p 2048,16384 -o md
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Arc™ Pro B60 Graphics (BMG G21) (Intel open-source Mesa driver) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 131072 | int dot: 0 | matrix cores: KHR_coopmat
load_backend: loaded Vulkan backend from /app/libggml-vulkan.so
load_backend: loaded CPU backend from /app/libggml-cpu-alderlake.so

model	size	params	backend	ngl	test	t/s
qwen35 ?B Q4_K - Medium	5.55 GiB	8.95 B	Vulkan	100	pp2048	559.62 ± 0.35
qwen35 ?B Q4_K - Medium	5.55 GiB	8.95 B	Vulkan	100	pp16384	517.64 ± 0.14
qwen35 ?B Q4_K - Medium	5.55 GiB	8.95 B	Vulkan	100	tg128	17.82 ± 0.00

build: d903f30e2 (8175)

So it looks like vulkan backend for llama.cpp runs prefill faster but token generation slower relative to SYCL backend for Qwen3.5-9B Q4 K XL. I’d really love to be able to run this model faster than this so that the 2nd GPU has some other purpose than ‘just’ 1 or 2 SRIOV VMs. This vulkan backend is getting closer to what I’d like to see. If token generation was 25+ t/s I’d be a little happier. Worth exploring some more though!

Eyoldaith · March 12, 2026, 1:35pm

SYCL works now? How complicated was the setup?

long.barn4823 · March 12, 2026, 1:42pm

I cloned the llama.cpp repo and built docker container with:

docker build -t llama-cpp-sycl --build-arg=“GGML_SYCL_F16=ON” –target full -f .devops/intel.Dockerfile .

wimm · March 16, 2026, 3:28pm

Intel also merged their OpenVINO backend just a few days ago, which in theory should provide the best experience for Intel integrated graphics as far back as 6th gen and up to the latests Intel dGPUs and iGPUs. I haven’t been able to get it to work on my 9900K (not for lack of trying!)

But I’d love to see some numberes on these. Build instructions can be found here.

Eyoldaith · March 16, 2026, 3:32pm

Would be interesting to see if performance could be improved by processing the context on the iGPU on a model that only barely fits in VRAM. (265k iGPU+B60)

I’ll get back to you on that lmao

Eyoldaith · March 16, 2026, 3:33pm

Would be interesting to see if performance could be improved by processing the context on the iGPU on a model that only barely fits in VRAM. (265k iGPU+B60)

I’ll get back to you on that lmao

Does the 9900K iGPU support the intel xe driver?

wimm · March 16, 2026, 6:32pm

No, but I believe OpenVino is supported on all Intel graphics which are supported by Intels OpenCL icd. I could be wrong though.

Edit: I was wrong. It seems to be restricted to more modern GPUs and their support for older CPUs is AVX2 only.

long.barn4823 · March 19, 2026, 11:50am

Interesting to see the openvino developments, looks like a few rough edges still, to be expected given how fresh this is!

I tried to follow the docker build instructions and it built without any errors.

I tried to run with qwen3.5 9B and it threw an error and I found this - Eval bug: OpenVino: Cant load Qwen3.5 · Issue #20562 · ggml-org/llama.cpp · GitHub

long.barn4823 · March 26, 2026, 12:17pm

I saw a reply in the B70 thread about Qn_0 quants potentially being faster on intel GPU.

Gave it a go just now and it seems to have some merit, mostly on token generation:

model	size	params	backend	ngl	test	t/s
qwen35 ?B Q4_K - Medium	2.54 GiB	4.21 B	Vulkan	100	pp2048	821.08 ± 0.28
qwen35 ?B Q4_K - Medium	2.54 GiB	4.21 B	Vulkan	100	tg512	30.87 ± 0.03
qwen35 ?B Q4_K - Medium	2.54 GiB	4.21 B	Vulkan	100	pp2048+tg512	132.27 ± 0.70

model	size	params	backend	ngl	test	t/s
qwen35 ?B Q4_0	2.40 GiB	4.21 B	Vulkan	100	pp2048	812.10 ± 0.80
qwen35 ?B Q4_0	2.40 GiB	4.21 B	Vulkan	100	tg512	42.89 ± 0.07
qwen35 ?B Q4_0	2.40 GiB	4.21 B	Vulkan	100	pp2048+tg512	174.50 ± 0.43

llama.cpp build: d903f30e2 (8175) with vulkan backend in docker on B60

I now have my opencode setup running with qwen3.5-27B running on RTX Pro 4500 and with the small model that opencode uses for session name generation and other small tasks using the qwen3.5 4B model on the B60 while the B60 also runs SRIOV for a windows VM.

digitalscream · March 28, 2026, 2:16pm

They tend to run significantly faster on AMD GPUs, too. The trouble is, those quants are legacy for a reason - the quality is quite a bit lower than _K and _K_M variants etc.

long.barn4823 · April 7, 2026, 11:51am

long.barn4823:

Using llama.cpp build 8175 SYCL I get the following numbers

Qwen3.5 9B Q4 K XL

llama-cpp-sycl
–bench -m /models/Qwen3.5-9B-UD-Q4_K_XL.gguf -ngl 100 -r 3 -p 2048,16384 -o md
load_backend: loaded SYCL backend from /app/libggml-sycl.so
load_backend: loaded CPU backend from /app/libggml-cpu-alderlake.so

model size params backend ngl test t/s

qwen35 ?B Q4_K - Medium 5.55 GiB 8.95 B SYCL 100 pp2048 328.71 ± 11.00

qwen35 ?B Q4_K - Medium 5.55 GiB 8.95 B SYCL 100 pp16384 317.41 ± 1.76

qwen35 ?B Q4_K - Medium 5.55 GiB 8.95 B SYCL 100 tg128 23.01 ± 0.03

The latest 8685+ llama.cpp has some nice SYCL improvements. Q8_0 support was added by - PR#21527

llama-cpp-sycl
–bench -m /models/qwen3.5_9B/Qwen3.5-9B-UD-Q4_K_XL.gguf -ngl 100 -r 3 -o md -p 2048,16384
load_backend: loaded SYCL backend from /app/libggml-sycl.so
load_backend: loaded CPU backend from /app/libggml-cpu-alderlake.so

model	size	params	backend	ngl	test	t/s
qwen35 9B Q4_K - Medium	5.55 GiB	8.95 B	SYCL	100	pp2048	1633.72 ± 0.59
qwen35 9B Q4_K - Medium	5.55 GiB	8.95 B	SYCL	100	pp16384	1464.48 ± 0.36
qwen35 9B Q4_K - Medium	5.55 GiB	8.95 B	SYCL	100	tg128	33.41 ± 0.00

build: 71a81f6fc (8688)

On my B60 Pro with qwen3.5 9B Q4_K_XL I’m not getting the full 3x TG speedup but definitely prefill speeds are up a long way which is very handy for interactivity with long system prompts like when using claude code or opencode etc

Very cool to see these optimizations still coming in!

long.barn4823 · April 7, 2026, 12:14pm

long.barn4823:

Qwen3.5 35B-A3B Q4 K XL

llama-cpp-sycl
–bench -m /models/Qwen3.5-35B-A3B-UD-Q4_K_XL.gguf -ngl 100 -r 3 -p 2048,16384 -o md
load_backend: loaded SYCL backend from /app/libggml-sycl.so
load_backend: loaded CPU backend from /app/libggml-cpu-alderlake.so

model size params backend ngl test t/s

qwen35moe ?B Q8_0 19.16 GiB 34.66 B SYCL 100 pp2048 112.62 ± 0.93

qwen35moe ?B Q8_0 19.16 GiB 34.66 B SYCL 100 pp16384 111.61 ± 1.11

qwen35moe ?B Q8_0 19.16 GiB 34.66 B SYCL 100 tg128 8.34 ± 0.22

running Qwen3.5 35B A3B again, with newer build. New build is the real deal.

llama-cpp-sycl
–bench -m /models/Qwen3.5_35B/Qwen3.5-35B-A3B-UD-Q4_K_XL.gguf -ngl 100 -r 3 -o md -p 2048
load_backend: loaded SYCL backend from /app/libggml-sycl.so
load_backend: loaded CPU backend from /app/libggml-cpu-alderlake.so

model	size	params	backend	ngl	test	t/s
qwen35moe 35B.A3B Q4_K - Medium	20.70 GiB	34.66 B	SYCL	100	pp2048	400.93 ± 1.63
qwen35moe 35B.A3B Q4_K - Medium	20.70 GiB	34.66 B	SYCL	100	tg128	37.24 ± 0.01

build: 71a81f6fc (8688)

long.barn4823 · April 21, 2026, 9:11am

Stress tested things by running a benchmark in windows via sriov and llama.cpp in Linux at the same time. This card is only on pcie 4.0 4x slot so things aren’t exactly smooth sailing when loading llm and running inference at the same time.

I bumped up the vram for QXL output and things are quite usable for light desktop apps in the VM at 4k. Pretty happy with the setup with the B60 running VMs and smaller LLMs to offload things.

hoborific · April 24, 2026, 10:32pm

Here’s two PRs to track or try early

github.com/ggml-org/llama.cpp

SYCL: add BF16 to DMMV kernel path (~4x tg speedup on Intel Arc)

master ← PMZFX:sycl-bf16-dmmv

opened 07:53PM - 07 Apr 26 UTC

PMZFX

+45 -1

## Summary BF16 models currently have no dedicated token generation (tg) kernel… in the SYCL backend. During single-token generation, BF16 falls through to the generic `ggml_sycl_op_mul_mat_sycl` GEMM path, which dequantizes to FP32 and runs a full matrix multiply — far too heavy for a memory-bound batch=1 operation. This adds BF16 to the DMMV (dequantize mul-mat-vec) path, following the existing F16 pattern. ## Changes **`ggml/src/ggml-sycl/dmmv.cpp`:** - `convert_bf16()` — reads `sycl::ext::oneapi::bfloat16`, casts to float (mirrors `convert_f16`) - `convert_mul_mat_vec_bf16_sycl()` — kernel launcher (mirrors F16 version) - Added BF16 to the DMMV dispatch switch - Added BF16 to the `src1_convert_f16` list for half-precision intrinsics when `GGML_SYCL_F16` is enabled - All BF16 code guarded behind `GGML_SYCL_DMMV_HAS_BF16` (compile-time bfloat16 header detection) **`ggml/src/ggml-sycl/ggml-sycl.cpp`:** - Added `GGML_TYPE_BF16` to `ggml_sycl_supports_dmmv()` ## Benchmark — Qwen2.5-1.5B, Intel Arc Pro B70 (Xe2), single GPU | Format | Size | pp512 (before) | pp512 (after) | tg128 (before) | tg128 (after) | tg speedup | |--------|------|---------------|--------------|---------------|--------------|------------| | Q4_K_M | 1.04 GiB | 8777 | 8778 | 202.6 | 202.6 | — | | Q8_0 | 1.76 GiB | 9304 | 9304 | 180.6 | 180.6 | — | | BF16 | 2.88 GiB | 2580 | 4887 | 29.7 | 123.9 | **4.2x** | BF16 bandwidth utilization goes from ~14% to ~58% of theoretical (608 GB/s). ## Testing - Builds cleanly with `-DGGML_SYCL=ON -DGGML_SYCL_F16=ON` - Token generation produces correct output (verified text coherence) - No regressions on Q4_K_M, Q8_0, or larger 9B models - Tested on Qwen2.5-1.5B and Qwen3.5-9B - Not yet tested on Intel Arc A-series (Alchemist) — would appreciate community testing ## Hardware - Intel Arc Pro B70 (BMG-G31, 32 GB GDDR6 ECC, 608 GB/s) - Driver: libze-intel-gpu1 26.09.37435.1, IGC 2.30.1 - oneAPI DPC++ 2025.3.3 ## Note This addresses the tg (token generation) path only. BF16 is still not included in the F16-specific special paths for permuted/batched operations (KQ, KQV). Those are separate and would be a broader change. Fixes #20478 ## AI Disclosure AI (Claude) assisted with investigating the dispatch path and drafting the kernel code. All code was human-reviewed, tested, and benchmarked on real hardware.

github.com/ggml-org/llama.cpp

[SYCL] Use native subgroup size for K-quant DMMV kernels on Intel

master ← PMZFX:opt/kquant-dmmv-subgroup16

opened 11:43PM - 09 Apr 26 UTC

PMZFX

+301 -322

## Summary Use `WARP_SIZE` (16) instead of `QK_WARP_SIZE` (32) for all K-quant …DMMV kernels (Q2_K through Q6_K), including the reorder variants added in #21638. These kernels were migrated from CUDA via DPCT and kept a 32-wide subgroup size. On Intel targets, native subgroup size is 16. DPCT itself flagged all five original kernels with register pressure warnings recommending a smaller sub-group size. The non-K-quant DMMV path already uses `WARP_SIZE` (16). Each thread now processes both halves of the QK_K=256 block via a `for (int im = 0; im < 2; ++im)` loop. The inner dot-product computation is unchanged. ## Updated scope (rebase) The original version of this PR covered only the five non-reorder K-quant kernels. Since then, #21638 added `dequantize_mul_mat_vec_q4_k_reorder` and `dequantize_mul_mat_vec_q6_k_reorder`, which were written against the same QK_WARP_SIZE=32 code. On Intel GPUs, these new reorder kernels cause a hard hang during warmup on dense Q4_K models (tested with EVA-Qwen2.5-72B Q4_K_M on Battlemage B70). I wrote the original fix here before writing the reorder kernels in #21638. The reorder kernels were copied from the unfixed upstream code because this PR hadn't merged yet. Now that #21638 has landed, the hang is present on current master for dense Q4_K/Q6_K models. This rebased version covers the reorder variants with the same fix. ## Changes **`ggml/src/ggml-sycl/dmmv.cpp`:** Each of the 7 affected kernels (5 original + 2 reorder) is restructured so that each thread processes both halves of the QK_K=256 block via a `for (int im = 0; im < 2; ++im)` loop, keeping total work identical. The `im`-dependent offsets move inside the loop. Dispatch functions and warp reductions switch from `QK_WARP_SIZE` to `WARP_SIZE`. No other files changed. ## Testing Intel Arc Pro B70 (Xe2/Battlemage, 32 GB), dual GPU, oneAPI 2025.3.3, Ubuntu 26.04. **Hang fix:** - EVA-Qwen2.5-72B Q4_K_M (dense, 80 layers): hangs on master during warmup, loads and serves correctly with this PR. 3 sequential prompts, all coherent. **Correctness:** - `test-backend-ops -o MUL_MAT -b SYCL0`: 911/911 passed, 0 failures **Performance (single GPU, sequential, Release, JIT, GGML_SYCL_F16=ON):** | Model | Metric | Master | PR | Delta | |-------|--------|--------|-----|-------| | Qwen3.5-9B Q4_K_M | pp128 | 1040 t/s | 1128 t/s | +8.5% | | Qwen3.5-9B Q4_K_M | tg32 | 54.6 t/s | 55.3 t/s | +1.3% | | Qwen3.5-9B Q8_0 | pp128 | 1133 t/s | 1079 t/s | -4.8% | | Qwen3.5-9B Q8_0 | tg32 | 47.6 t/s | 47.5 t/s | -0.2% | | Gemma4-31B Q6_K | pp128 | 327 t/s | 327 t/s | 0% | | Gemma4-31B Q6_K | tg32 | 13.4 t/s | 13.4 t/s | 0% | Q8_0 is unaffected by this PR (already uses WARP_SIZE). pp variance is normal run-to-run variation. No regressions. Previous testing by @arthw across Arc770, BMG580, iGPU (UHD), and PVC confirmed the original non-reorder changes were acceptable. The reorder kernels follow the identical pattern.