Llama.cpp for OpenVINO. How does a half-baked build stack up against Vulkan?

High-Performance Computing Machine Learning, LLMs, & AI
1

Bottom line up front: Even with its variance, OpenVINO backend has light-speed prompt processing compared to Vulkan. According to my small dataset, the larger the model gets, the faster OpenVINO can generate tokens compared to Vulkan.

On my way to the llama.cpp Github to make sure I wasn’t hallucinating the SYCL speeds being slower than the Vulkan speeds, I saw that an OpenVINO build has been whipped up. So, naturally, I dropped what I was doing to chase the mere thought of faster generation speeds on Intel GPUs.

I should’ve known, like getting anything to work with Intel Arc, it was gonna put up a fight.

I won’t bore any of you with the details, but here’s some things you’d wanna know if you’re gonna try this out:

  • openVINO wasn’t compatible with Ubuntu-based distros out the box. It’d point that out and refuse to proceed with OpenVINO installation - just install through APT. Just be sure to install libtbb-dev, and replace include(“${OpenVINO_DIR}/../3rdparty/tbb/lib/cmake/TBB/TBBConfig.cmake”) with find_package(TBB REQUIRED) inside ggml/src/ggml-openvino/CMakeLists.txt within the llama.cpp source that you pulled.
  • You may find OpenVINO is in the wrong place for you to find your setupvars.sh, so you’ll have to find them yourself. Instead of source /opt/intel/openvino/setupvars.sh where the docs think it should be, it may specifies its version in the path: source /opt/intel/openvino_2026.0/setupvars.sh, and you’ll have to account for that.

After not being able to build with –parallel and having to keep threads at 2 (cmake --build build/ReleaseOV -j <not a lot.>), I’m finally in! Works just how it should, models-dir and everything.

Now that everything’s in place, I’m gonna fly off the rails.*

I’m gonna test a few of these validated models, Qwen2.5 1.5b and Llama 3.1 8b, both at Q4_K_M. The following are with all default options except flash attention set to 1 for both Vulkan and OpenVINO, as per the docs. Note, you have to set your GPU use with export GGML_OPENVINO_DEVICE=GPU, if you have multiple GPUs, it’s gonna be GPU.n, where “n“ is the index from 0, 0 for me was integrated graphics. Mayble I’ll throw in some SYCL tests for the hell of it.

With one Intel Arc A770 16GB, and all models at Q4_K_M quantization:

LLAMA 3.2 1B FOR OPENVINO: pp512 = 16305.12 ± 641.85 / tg128 = 88.67 ± 0.62

LLAMA 3.2 1B FOR VULKAN: pp512 = 6233.56 ± 75.12 / tg128 = 119.60 ± 9.12

LLAMA 3.2 1B FOR SYCL: pp512 = 3886.68 ± 16.82 / tg128 = 67.93 ± 0.10

QWEN2 1.5B FOR OPENVINO: pp512 = 11503.96 ± 203.70 / tg128 = 65.00 ± 2.15

QWEN2 1.5B FOR VULKAN: pp512 = 4198.28 ± 27.11 / tg128 = 89.93 ± 0.14

QWEN2 1.5B FOR SYCL: It legit failed.

HUNYUAN 7B FOR OPENVINO: pp512 = 4078.31 ± 24.72 / tg128 = 34.54 ± 0.50

HUNYUAN 7B FOR VULKAN: pp512 = 952.66 ± 2.70 / tg128 = 31.41 ± 0.38

HUNYUAN 7B FOR SYCL: pp512 = 952.66 ± 2.70 / tg128 = 31.41 ± 0.38

MISTRAL 7B FOR OPENVINO: pp512 = 3960.63 ± 21.57 / tg128 = 34.83 ± 0.23

MISTRAL 7B FOR VULKAN: pp512 = 969.09 ± 2.67 / tg128 = 36.16 ± 0.57

MISTRAL 7B FOR SYCL: pp512 = 692.41 ± 1.82 / tg128 = 24.61 ± 0.02

LLAMA 3.1 8B FOR OPENVINO: pp512 = 11503.96 ± 203.70 / tg128 = 65.00 ± 2.15

LLAMA 3.1 8B FOR VULKAN: pp512 = 3930.39 ± 36.31 / tg128 = 33.87 ± 0.08

DEEPSEEK-LLAMA 8B FOR OPENVINO: pp512 = 3676.43 ± 44.08 / tg128 = 33.56 ± 0.48

DEEPSEEK-LLAMA 8B FOR VULKAN: pp512 = 966.74 ± 3.54 / tg128 = 31.79 ± 0.46

DEEPSEEK-LLAMA 8B FOR SYCL: pp512 = 691.32 ± 1.97 / tg128 = 23.71 ± 0.01

QWEN3 8B FOR OPENVINO: pp512 = 3844.23 ± 21.79 / tg128 = 32.61 ± 0.07

QWEN3 8B FOR VULKAN: pp512 = 928.37 ± 2.43 / tg128 = 30.23 ± 0.42

One thing both Vulkan and SYCL have over OpenVINO here is that they can run multiple GPUs. I checked the docs and even consulted with the robot (Gemini), according to it the device option was export GGML_OPENVINO_DEVICE="MULTI:GPU.n,GPU.m,...", but that didn’t work nor was any evidence of that found in the OpenVINO docs the llama.cpp Github directed me towards. Whatever, search party’s over for now.

*I would have tested more models, but that list of validated models doesn’t seem to be a suggestion. Any model I tried to run on OpenVINO outside of that list would spit out “test_prompt: failed to decode prompt batch, res = -3 / main: error: failed to run prompt warmup“, even when dropping the batch size on the floor. I just wanted to see a larger model run and thought “hey, if Mistral 7B runs, what about Ministral 14B?“.

No.

I thought it was just the GPU, and switched to the CPU - nope. I then ditched my hopes for running a larger model and tried LFM2.5 350M. Million with an M. That made the terminal start yelling even louder with a “ggml/src/ggml-backend.cpp:809: pre-allocated tensor (cache_r_l0 (view) (copy of )) in a buffer (OPENVINO0) that cannot run the operation (CPY)“.

The OpenVINO build is still a work in progress, so I’m not gonna run with these conclusions just yet, there’s still a lot of development to be done… Promising… It’s promising. My CPU is a 13th gen, so I don’t have any NPU to test, and there is an option to select it (export GGML_OPENVINO_DEVICE=NPU) so I assume there’s support. If anyone has a Core Ultra to test, socket or mobile, I’d like to see that.

4 Likes
2

That’s beyond weird. Qwen3 8B hits PP numbers approaching that of an R9700 (~4kt/s) with Vulkan, indicating similar processing throughput, but TG is 71% lower since the R9700 hits around 110t/s (albeit with no real optimisation).

The weird part is that TG is supposed to be bound by VRAM bandwidth, and the two cards aren’t that far apart - the A770 16GB has 560GB/s vs the R9700’s 640GB/s. Based on that and the PP result, you’d expect the A770 with OpenVINO to have a ceiling of around 95t/s.

Now, that would admittedly be an astonishing result for a card like the A770, but it’s always struck me as phenomenal hardware that was just waiting for some decent software to unlock it. The bones are there, Intel just did it dirty with the lack of support…and I suspect they’ll continue to leave it in the dusty corner while they focus on Battlemage.

Any chance you could try firing up gpt-oss:20b on there to see how MoE models fare? If it can hit 60t/s, I might grab one so the wife can use it for her AI stuff (her work’s sensitive, so she can’t use cloud services).

1 Like
3

When it comes to GPT-OSS, I tried changing the size of the batch, prompt, and generation - I even tried integrated graphics offloading. No dice. It loudly objected to Qwen3.5 35B and Gemma 4 24B as well, RAM-only or otherwise. It seems like despite being validated on NPUs, for now only the listed models work, maybe some edge cases too, I dunno.

I don’t know if this is relevant to you, but while looking through it, some pre-compiled binaries for llama.cpp for OpenVINO were published too. Which would be good if (apparently) requires certain instruction sets and I have a sneaking suspicion that it’s AVX-512, else you’re booted out with an “Illegal instruction“ error. So old CPUs or low-power processors like mine are build from source only.

4

A small addendum:

If you don’t want to run models other than those listed on the OpenVINO readme.md, as of right now you can run custom models based on the validated models’ architecture. If you see one of the validated models in the model tree, or see in the config that the architecture is:

Qwen3ForCausalLM,

LlamaForCausalLM,

Phi3ForCausalLM,

Qwen2ForCausalLM,

HunYuanDenseV1ForCausalLM,

MistralForCausalLM,

or MiniCPMForCausalLM (according to them, but I cant get the model to load, it straight up fails),

It will work. I suppose they’re focusing on getting dense architectures working first. The largest model I’ve been able to get running is Phi 4 Reasoning/Vision with 15B reported parameters, though llama-bench detects 14B. Phi 3.5 also works. With the same benchmarking config:

PHI 4 REASONING/VISION 15B FOR OPENVINO: pp512 = 2208 ± 5.47 / tg128 = 22.20 ± 0.22

PHI 4 REASONING/VISION 15B FOR VULKAN: pp512 = 507.88 ± 0.28 / tg128 = 17.11 ± 0.08

PHI 4 REASONING/VISION 15B FOR SYCL: pp512 = 444.03 ± 2.11 / tg128 = 13.43 ± 0.12

PHI 3.5 MINI 4B FOR OPENVINO: pp512 = 5622.92 ± 25.78 / tg128 = 48.46 ± 0.12

PHI 3.5 MINI 4B FOR VULKAN: pp512 = 1390.95 ± 0.91 / tg128 = 65.17 ± 0.09

PHI 3.5 MINI 4B FOR SYCL: pp512 = 1428.62 ± 14.17 / tg128 = 26.72 ± 0.05

I’ve got my finger on the pulse right now, wanna get the most out of my hardware of course, so if I see any updates to what’s working I’ll add it to this thread.

5

Thanks for taking the time to write this up.

As a a770 owner in always trying to work out the balance of best performance whilst having access to the latest models.

What’s your default go to?

For me.

Easy of use and best model access has been llmstudio. Was able to to run small sized gemma4 and qwen3.5.

To run larger versions of either it’s where the pain and hours of hours or tests come into play. The best results have been with openvino.genai using pr’s from GitHub and pulling Intel-ov from hugging face. They’re far from perfect but have been able to get them running.

Hopefully we get Gemma and qwen3.6 native support soon through the openvino stack soon

6

I use pre-compiled Vulkan binaries on llama.cpp’s Github. If you’re on anything Ubuntu/Debian based right now, it’s worth checking out - they’ve added pre-compiled SYCL AND OpenVINO builds now.

LMStudio’s your best bet right now though, especially now they’ve added LMLink. Since they pull builds from llama.cpp’s Github to use, there should be one for OpenVINO option coming as soon it’s fully ready for Intel Arc, like they have CUDA and ROCm builds for NVIDIA and AMD.

Some others on the forum and I have had success using Ollama-Intel-Arc, which is a set of Docker builds and it’s llama.cpp build uses the SYCL backend and flash attention off. Though performance using multiple GPUs gets dicey on it, and RAM offloading simply doesn’t work at all.

7

What issues have y’all ran into using the OpenVINO llama.cpp backend? Particularly in the listed supported models from the OPENVINO.md doc file in that repo

8

Here’s a lil’ list of what I’ve found:

  • OpenVINO’s multi GPU protocol isn’t supported, you have to pick a device and stick with it.
  • Performance doesn’t seem to scale between CPU and GPU for now - using llama 3.2 1B, I got higher prompt processing and token generation using an i5-13500T (not even the iGPU, just straight core) compared to an overclocked A770 16GB. How. Granted, this is using Unsloth’s ultra-dynamic Q6_K_XL, so…
  • No quant below 4 bits are working, and you can give up now if you wanted to use I-quants. BUT…! Any K-quant or standard, 4-bit and above works, including Q8_K_XL and BF16 on CPU only!
  • Flash attention on is a hard requirement not just on llama-bench, but for all of it. Without it, it’ll boot you out with a seg fault.
  • MiniCPM is said to be validated… They’re lying. It doesn’t even see the .gguf as a .gguf: invalid magic characters: ‘PWRI’, expected ‘GGUF’
1 Like
9

I’ll see if I can reproduce the MiniCPM issue on my end

10

Some benchmarks from the B70 for anyone interested. Faster on the pp512 test, slower on the tg128. Also the setup time is much longer on the OpenVINO version.

model size params backend ngl fa test t/s
gemma4 31B Q4_K - Medium 17.39 GiB 30.70 B Vulkan -1 1 pp512 840.56 ± 2.53
gemma4 31B Q4_K - Medium 17.39 GiB 30.70 B Vulkan -1 1 tg128 25.23 ± 0.07
model size params backend ngl fa test t/s
gemma4 31B Q4_K - Medium 17.39 GiB 30.70 B OPENVINO -1 1 pp512 1704.43 ± 3.37
gemma4 31B Q4_K - Medium 17.39 GiB 30.70 B OPENVINO -1 1 tg128 22.27 ± 0.08

Running on Win11 26H2 with the 32.0.101.9150 driver on the B70.

Overlap result to a model run above to give some idea of relative performance between the A770 and B70.

model size params backend ngl fa test t/s
phi3 3B Q4_K - Medium 2.31 GiB 3.84 B Vulkan -1 1 pp512 6953.79 ± 78.25
phi3 3B Q4_K - Medium 2.31 GiB 3.84 B Vulkan -1 1 tg128 156.16 ± 0.14
model size params backend ngl fa test t/s
phi3 3B Q4_K - Medium 2.31 GiB 3.84 B OPENVINO -1 1 pp512 10386.71 ± 50.28
phi3 3B Q4_K - Medium 2.31 GiB 3.84 B OPENVINO -1 1 tg128 113.57 ± 0.76