Intel Battlematrix B60 Machine

wendell · February 22, 2026, 9:57pm

it’s here!!!

intel@intel-battlematrix:~$ lspci |grep VGA
05:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 52)
18:00.0 VGA compatible controller: Intel Corporation Device e211
36:00.0 VGA compatible controller: Intel Corporation Device e211
54:00.0 VGA compatible controller: Intel Corporation Device e211
72:00.0 VGA compatible controller: Intel Corporation Device e211

            .-/+oossssoo+/-.               intel@intel-battlematrix
        `:+ssssssssssssssssss+:`           ------------------------
      -+ssssssssssssssssssyyssss+-         OS: Ubuntu 25.04 x86_64
    .ossssssssssssssssssdMMMNysssso.       Kernel: 6.14.0-1008-intel
   /ssssssssssshdmmNNmmyNMMMMhssssss/      Uptime: 1 hour, 25 mins
  +ssssssssshmydMMMMMMMNddddyssssssss+     Packages: 1636 (dpkg), 12 (snap)
 /sssssssshNMMMyhhyyyyhmNMMMNhssssssss/    Shell: bash 5.2.37
.ssssssssdMMMNhsssssssssshNMMMdssssssss.   Resolution: 1024x768
+sssshhhyNMMNyssssssssssssyNMMMysssssss+   Terminal: /dev/pts/4
ossyNMMMNyMMhsssssssssssssshmmmhssssssso   CPU: Intel Xeon w5-3435X (32) @ 4.500GHz
ossyNMMMNyMMhsssssssssssssshmmmhssssssso   GPU: Intel Device e211
+sssshhhyNMMNyssssssssssssyNMMMysssssss+   GPU: Intel Device e211
.ssssssssdMMMNhsssssssssshNMMMdssssssss.   GPU: Intel Device e211
 /sssssssshNMMMyhhyyyyhdNMMMNhssssssss/    Memory: 88844MiB / 257243MiB
  +sssssssssdmydMMMMMMMMddddyssssssss+
   /ssssssssssshdmNNNNmyNMMMMhssssss/
    .ossssssssssssssssssdMMMNysssso.
      -+sssssssssssssssssyyyssss+-
        `:+ssssssssssssssssss+:`
            .-/+oossssoo+/-.

gpt oss120b


TORCH_LLM_ALLREDUCE=1 VLLM_USE_V1=1 CCL_ZE_IPC_EXCHANGE=pidfd VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 VLLM_WORKER_MULTIPROC_METHOD=spawn python3 -m vllm.entrypoints.openai.api_server   --model openai/gpt-oss-120b   --dtype=bfloat16   --enforce-eager   --port 8000   --host 0.0.0.0   --trust-remote-code   --gpu-memory-util=0.95   --no-enable-prefix-caching   --max-num-batched-tokens=8192   --disable-log-requests   --max-model-len=8192   --quantization mxfp4   --block-size 64  --tensor-parallel-size 4


Waiting for endpoint to become up in 600 seconds
 |                                                                                                                                                                                        | 00:06 elapsed, 1397:02:30 remaining
Initial test run completed. Starting main benchmark run...
Traffic request rate: inf
Burstiness factor: 1.0 (Poisson process)
Maximum request concurrency: None
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:25<00:00,  1.93it/s]
============ Serving Benchmark Result ============
Successful requests:                     50
Benchmark duration (s):                  25.95
Total input tokens:                      51200
Total generated tokens:                  25600
Request throughput (req/s):              1.93
Output token throughput (tok/s):         986.61
Total Token throughput (tok/s):          2959.84
---------------Time to First Token----------------
Mean TTFT (ms):                          3920.69
Median TTFT (ms):                        4348.54
P99 TTFT (ms):                           6626.93
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          42.88
Median TPOT (ms):                        42.08
P99 TPOT (ms):                           48.93
---------------Inter-token Latency----------------
Mean ITL (ms):                           42.88
Median ITL (ms):                         38.10
P99 ITL (ms):                            41.10
==================================================

(APIServer pid=895) INFO 02-22 19:32:07 [loggers.py:123] Engine 000: Avg prompt throughput: 60.4 tokens/s, Avg generation throughput: 36.5 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=895) INFO:     10.200.0.55:60966 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=895) INFO 02-22 19:32:17 [loggers.py:123] Engine 000: Avg prompt throughput: 10.3 tokens/s, Avg generation throughput: 28.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 0.0%
(APIServer pid=895) INFO 02-22 19:32:27 [loggers.py:123] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 38.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.8%, Prefix cache hit rate: 0.0%
(APIServer pid=895) INFO 02-22 19:32:37 [loggers.py:123] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 38.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.2%, Prefix cache hit rate: 0.0%
(APIServer pid=895) INFO 02-22 19:32:47 [loggers.py:123] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 38.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.5%, Prefix cache hit rate: 0.0%
(APIServer pid=895) INFO 02-22 19:32:57 [loggers.py:123] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 38.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.9%, Prefix cache hit rate: 0.0%
(APIServer pid=895) INFO 02-22 19:33:07 [loggers.py:123] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 38.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.2%, Prefix cache hit rate: 0.0%
(APIServer pid=895) INFO 02-22 19:33:17 [loggers.py:123] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 38.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.6%, Prefix cache hit rate: 0.0%
(APIServer pid=895) INFO 02-22 19:33:27 [loggers.py:123] Engine 000: Avg prompt throughput: 299.3 tokens/s, Avg generation throughput: 36.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 0.0%
(APIServer pid=895) INFO:     10.200.0.55:60966 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=895) INFO 02-22 19:33:37 [loggers.py:123] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=895) INFO 02-22 19:33:47 [loggers.py:123] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%

This is both excellent scaling and excellent parallel throughput.

However, for a single request, with gpt-oss120b we’re in that roughly 20-40t/s performance window.

LLama 70b

So I ran into a problem and I thought --max-num-batched-tokens or no-enable-batched would help me diagnose. Turns out V1 is designed ONLY for batching.


^Cchunked_prefill: Unsupported dtype!

(Worker_TP3 pid=5173) INFO 02-22 21:17:20 [multiproc_executor.py:546] Parent process exited, terminating worker
(APIServer pid=4954) WARNING 02-22 21:17:20 [_logger.py:68] port 8000 is used by process psutil.Process(pid=4954, name='python3', status='running') launched with command:
(APIServer pid=4954) WARNING 02-22 21:17:20 [_logger.py:68] python3 -m vllm.entrypoints.openai.api_server --model meta-llama/Llama-3.1-70B --served-model-name meta-llama/Llama-3.1-70B --enforce-eager --host 0.0.0.0 --port 8000 --trust-remote-code --gpu-memory-utilization 0.90 --no-enable-prefix-caching --no-enable-chunked-prefill --max-model-len 2048 --block-size 16 --quantization fp8 --tensor-parallel-size 4
(APIServer pid=4954) INFO 02-22 21:17:20 [launcher.py:101] Shutting down FastAPI HTTP server.
(Worker_TP0 pid=5170) INFO 02-22 21:17:20 [multiproc_executor.py:546] Parent process exited, terminating worker
(Worker_TP1 pid=5171) INFO 02-22 21:17:20 [multiproc_executor.py:546] Parent process exited, terminating worker
(Worker_TP2 pid=5172) INFO 02-22 21:17:20 [multiproc_executor.py:546] Parent process exited, terminating worker
^C(APIServer pid=4954) INFO:     Shutting down
(APIServer pid=4954) INFO:     Waiting for connections to close. (CTRL+C to force quit)
(APIServer pid=4954) INFO 02-22 21:17:24 [loggers.py:123] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 206.4 tokens/s, Running: 20 reqs, Waiting: 30 reqs, GPU KV cache usage: 96.3%, Prefix cache hit rate: 0.0%

So vllm bench serve is broken here with the spam of chunked_prefill: Unsupported dtype! ?

however


TORCH_LLM_ALLREDUCE=1 VLLM_USE_V1=1 CCL_ZE_IPC_EXCHANGE=pidfd VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 VLLM_WORKER_MULTIPROC_METHOD=spawn python3 -m vllm.entrypoints.openai.api_server --model meta-llama/Llama-3.1-70B --enforce-eager --host 0.0.0.0 --port 8000 --trust-remote-code --gpu-memory-utilization 0.90 --no-enable-prefix-caching --no-enable-chunked-prefill --max-model-len 2048 --block-size 64 --quantization fp8 --tensor-parallel-size 4

# Readability
TORCH_LLM_ALLREDUCE=1 \
VLLM_USE_V1=1 \
CCL_ZE_IPC_EXCHANGE=pidfd \
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
VLLM_WORKER_MULTIPROC_METHOD=spawn \
python3 -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-70B \
  --enforce-eager \
  --host 0.0.0.0 \
  --port 8000 \
  --trust-remote-code \
  --gpu-memory-utilization 0.90 \
  --no-enable-prefix-caching \
  --no-enable-chunked-prefill \
  --max-model-len 2048 \
  --block-size 64 \
  --quantization fp8 \
  --tensor-parallel-size 4

–block-size 64 is the solution to this problem.

============ Serving Benchmark Result ============
Successful requests:                     50
Benchmark duration (s):                  110.22
Total input tokens:                      51150
Total generated tokens:                  20526
Request throughput (req/s):              0.45
Output token throughput (tok/s):         186.23
Total Token throughput (tok/s):          650.29
---------------Time to First Token----------------
Mean TTFT (ms):                          34482.42
Median TTFT (ms):                        19422.21
P99 TTFT (ms):                           76096.98
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          310.94
Median TPOT (ms):                        107.19
P99 TPOT (ms):                           1444.64
---------------Inter-token Latency----------------
Mean ITL (ms):                           104.44
Median ITL (ms):                         67.47
P99 ITL (ms):                            1439.60
==================================================

Not bad! TTFT is, as expected, a bit rough with the dynamic quant.

If you set this up in openwebui it’ll give you:

you need to create a chat template like llama3.jinja

{% set loop_messages = messages %}
{% for message in loop_messages %}
{% if loop.index0 == 0 %}<|begin_of_text|>{% endif %}<|start_header_id|>{{ message['role'] }}<|end_header_id|>

{% if message['content'] is string %}
{{ message['content'] }}
{% else %}
{% for part in message['content'] %}
{% if part.get('type') == 'text' %}
{{ part.get('text', '') }}
{% endif %}
{% endfor %}
{% endif %}<|eot_id|>
{% endfor %}
{% if add_generation_prompt %}<|start_header_id|>assistant<|end_header_id|>

{% endif %}

then use --chat-template /path/to/llama3.jinja as part of the startup.

TORCH_LLM_ALLREDUCE=1 VLLM_USE_V1=1 CCL_ZE_IPC_EXCHANGE=pidfd VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 VLLM_WORKER_MULTIPROC_METHOD=spawn python3 -m vllm.entrypoints.openai.api_server --model meta-llama/Llama-3.1-70B --enforce-eager --host 0.0.0.0 --port 8000 --trust-remote-code --gpu-memory-utilization 0.90 --chat-template ./llama3.jinja --max-model-len 2048 --block-size 64 --quantization fp8 --tensor-parallel-size 4

It is also helpful to set the temperature to 0.2, max_tokens and a repetition penalty.. but that tends to be more on the openwebui side of things.

When using this model with just one client via Openwebui I get

(APIServer pid=2023) INFO 02-24 19:44:11 [loggers.py:123] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.1 tokens/s, Running: 1 reqs, Wait

In general Llama 70b-fp16 (in dynamic fp8) behaved a bit weirdly. It can work as well as this however:

Qwen3-30B-A3B


TORCH_LLM_ALLREDUCE=1 VLLM_USE_V1=1 CCL_ZE_IPC_EXCHANGE=pidfd VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 VLLM_WORKER_MULTIPROC_METHOD=spawn python3 -m vllm.entrypoints.openai.api_server --model Qwen/Qwen3-30B-A3B --enforce-eager --host 0.0.0.0 --port 8000 --trust-remote-code --gpu-memory-utilization 0.90  --max-model-len 2048 --block-size 64 --quantization fp8 --tensor-parallel-size 4

For the benchmark:

(APIServer pid=2587) INFO 02-24 20:00:13 [loggers.py:123] Engine 000: Avg prompt throughput: 3166.1 tokens/s, Avg generation throughput: 881.1 tokens/s, Running: 50 reqs, Waiting: 0 reqs, GPU KV cache usage: 12.4%, Prefix cache hit rate: 1.8%

============ Serving Benchmark Result ============
Successful requests:                     50
Benchmark duration (s):                  25.82
Total input tokens:                      51034
Total generated tokens:                  25600
Request throughput (req/s):              1.94
Output token throughput (tok/s):         991.66
Total Token throughput (tok/s):          2968.56
---------------Time to First Token----------------
Mean TTFT (ms):                          2858.24
Median TTFT (ms):                        2768.20
P99 TTFT (ms):                           5462.78
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          43.97
Median TPOT (ms):                        44.12
P99 TPOT (ms):                           48.14
---------------Inter-token Latency----------------
Mean ITL (ms):                           43.97
Median ITL (ms):                         39.65
P99 ITL (ms):                            219.20
==================================================

Single client openwebui metrics:

(APIServer pid=2587) INFO 02-24 20:01:33 [loggers.py:123] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 29.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU 


KV cache usage: 0.1%, Prefix cache hit rate: 1.8%

Deepseek R! Distill Llama 70B

I was curious about this llama-based deepseek distillation given the hiccups on Llama 70b, so I tried it and it worked fine.

Command I used (no chat template needed)

TORCH_LLM_ALLREDUCE=1 VLLM_USE_V1=1 CCL_ZE_IPC_EXCHANGE=pidfd VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 VLLM_WORKER_MULTIPROC_METHOD=spawn python3 -m vllm.entrypoints.openai.api_server --model deepseek-ai/DeepSeek-R1-Distill-Llama-70B --enforce-eager --host 0.0.0.0 --port 8000 --trust-remote-code --gpu-memory-utilization 0.90 --max-model-len 2048 --block-size 64 --quantization fp8 --tensor-parallel-size 4

(APIServer pid=4414) INFO 02-24 20:35:36 [loggers.py:123] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.4%, Prefix cache hit rate: 0.0%

============ Serving Benchmark Result ============
Successful requests:                     50
Benchmark duration (s):                  143.17
Total input tokens:                      51150
Total generated tokens:                  25600
Request throughput (req/s):              0.35
Output token throughput (tok/s):         178.80
Total Token throughput (tok/s):          536.06
---------------Time to First Token----------------
Mean TTFT (ms):                          43833.33
Median TTFT (ms):                        38216.57
P99 TTFT (ms):                           113761.08
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          99.47
Median TPOT (ms):                        98.13
P99 TPOT (ms):                           165.66
---------------Inter-token Latency----------------
Mean ITL (ms):                           99.47
Median ITL (ms):                         67.58
P99 ITL (ms):                            1436.97
==================================================

And the actual openwebui results are coherent!

Battlematrix (4× Intel B60) vLLM caveat for Llama-70B / DeepSeek-R1-Distill-Llama-70B FP8 benchmarks

I finally got Llama-3.1-70B and DeepSeek-R1-Distill-Llama-70B running on the 4×B60 “Battlematrix” setup using Intel’s XPU vLLM stack, but there’s an important caveat for anyone trying to reproduce results.

TL;DR

In this Intel vLLM XPU build, when VLLM_USE_V1=1 is enabled, chunked prefill appears to be forced on, even if you pass:
- --no-enable-chunked-prefill
The CLI arg is parsed (shows as false in logs), but the scheduler still reports chunked prefill enabled.
I was hitting chunked_prefill: Unsupported dtype! with FP8 until I changed:
- --block-size 64
With --block-size 64, both:
- meta-llama/Llama-3.1-70B
- deepseek-ai/DeepSeek-R1-Distill-Llama-70B
  work in FP8 with tensor parallel = 4.

Tango down, we’ve lost a gpu

[ 1406.844376] hid-generic 0003:3384:0007.000A: hiddev2,hidraw5: USB HID v1.11 Device [System76 Launch Heavy Configurable Keyboard (launch_heavy_1)] on usb-0000:00:14.0-6.3.6/input1
[ 5818.822901] {1}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 0
[ 5818.822912] {1}[Hardware Error]: It has been corrected by h/w and requires no further action
[ 5818.822916] {1}[Hardware Error]: event severity: corrected
[ 5818.822920] {1}[Hardware Error]:  Error 0, type: corrected
[ 5818.822925] {1}[Hardware Error]:   section_type: PCIe error
[ 5818.822928] {1}[Hardware Error]:   port_type: 5, upstream switch port
[ 5818.822932] {1}[Hardware Error]:   version: 3.0
[ 5818.822935] {1}[Hardware Error]:   command: 0x0147, status: 0x0011
[ 5818.822940] {1}[Hardware Error]:   device_id: 0000:34:00.0
[ 5818.822945] {1}[Hardware Error]:   slot: 0
[ 5818.822948] {1}[Hardware Error]:   secondary_bus: 0x35
[ 5818.822951] {1}[Hardware Error]:   vendor_id: 0x8086, device_id: 0xe2ff
[ 5818.822955] {1}[Hardware Error]:   class_code: 060400
[ 5818.822959] {1}[Hardware Error]:   bridge: secondary_status: 0x0000, control: 0x0013
[ 5818.822963] {1}[Hardware Error]:   aer_cor_status: 0x00002001, aer_cor_mask: 0x00000000
[ 5818.822967] {1}[Hardware Error]:   aer_uncor_status: 0x00100000, aer_uncor_mask: 0x00180000
[ 5818.822971] {1}[Hardware Error]:   aer_uncor_severity: 0x00463030
[ 5818.822974] {1}[Hardware Error]:   TLP Header: 00000000 00000000 00000000 00000000
[ 5818.823043] pcieport 0000:34:00.0: AER: aer_status: 0x00002001, aer_mask: 0x00000000
[ 5818.823056] pcieport 0000:34:00.0:    [ 0] RxErr                  (First)
[ 5818.823063] pcieport 0000:34:00.0:    [13] NonFatalErr
[ 5818.823069] pcieport 0000:34:00.0: AER: aer_layer=Physical Layer, aer_agent=Receiver ID
[ 7526.844152] audit: type=1400 audit(1771790400.347:224): apparmor="DENIED" operation="open" class="file" profile="snap.firmware-updater.firmware-notifier" name="/proc/sys/vm/max_map_count" pid=15365 comm="firmware-notifi" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
[ 9143.258478] xe 0000:54:00.0: [drm] VM worker error: -12

Helpful script for diagnosing intel gpu down

for i in 0 1 2 3; do
  echo "=== GPU $i ==="
  ZE_AFFINITY_MASK=$i python3 -m vllm.entrypoints.openai.api_server \
    --model Qwen/Qwen3-0.6B \
    --served-model-name qwen-test-$i \
    --host 0.0.0.0 --port $((8100+i)) \
    --dtype bfloat16 \
    --gpu-memory-utilization 0.8 \
    --max-model-len 1024 \
    --max-num-batched-tokens 1024 \
    --enforce-eager &
  sleep 20
  curl -s http://127.0.0.1:$((8100+i))/v1/models || true
  pkill -f "port $((8100+i))" || true
  sleep 5
done

Deepseek Distill 70b

TORCH_LLM_ALLREDUCE=1 \
VLLM_USE_V1=1 \
CCL_ZE_IPC_EXCHANGE=pidfd \
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
VLLM_WORKER_MULTIPROC_METHOD=spawn \
python3 -m vllm.entrypoints.openai.api_server \
  --model deepseek-ai/DeepSeek-R1-Distill-Llama-70B \
  --served-model-name dsr1-70b-fp8 \
  --enforce-eager \
  --host 0.0.0.0 \
  --port 8000 \
  --gpu-memory-utilization 0.90 \
  --no-enable-prefix-caching \
  --no-enable-chunked-prefill \
  --max-model-len 2048 \
  --block-size 16 \
  --quantization fp8 \
  --tensor-parallel-size 4

vllm bench serve \
  --backend vllm \
  --endpoint-type openai \
  --base-url http://127.0.0.1:8000 \
  --endpoint /v1/completions \
  --model dsr1-70b-fp8 \
  --tokenizer deepseek-ai/DeepSeek-R1-Distill-Llama-70B \
  --dataset-name random \
  --num-prompts 50 \
  --random-input-len 1024 \
  --random-output-len 512 \
  --percentile-metrics ttft,tpot,itl \
  --metric-percentiles 99

(APIServer pid=8919) INFO 02-22 21:57:24 [loggers.py:123] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 272.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%

Why short synthetic benchmarks look fast, but agentic coding (OpenCode, etc.) feels slower

This is the big gotcha.

A lot of benchmark runs (including what I’m posting below) use something like:

~1K input tokens
a few hundred output tokens

That’s useful for comparison, but it doesn’t fully represent agentic coding workloads.

What changes with tools like OpenCode / agentic coding workflows

Agentic coding often carries:

very large context windows (e.g. 10K+ tokens)
prior turns
file excerpts
tool outputs
diffs
system prompts / planning traces

That makes the prefill phase much heavier.

wendell · February 22, 2026, 10:04pm

So – what do you wanna bench on it?

We’re prettymuch relegated to running llm-scaler:

github.com/intel/llm-scaler

README.md

main

# LLM Scaler

LLM Scaler is an GenAI solution for text generation, image generation, video generation etc. running on [Intel® Arc™ Pro B60 GPUs](https://www.intel.com/content/www/us/en/products/docs/discrete-gpus/arc/workstations/b-series/overview.html). LLM Scalar leverages standard frameworks such as vLLM, ComfyUI, SGLang Diffusion, Xinference etc and ensures the best performance for State-of-Art GenAI models running on Arc Pro B60 GPUs.

---

## Latest Update
- [2026.01] We released `intel/llm-scaler-vllm:1.3` (or, `intel/llm-scaler-vllm:0.11.1-b7`) for vllm 0.11.1 and PyTorch 2.9 support, various new models support and performance improvements.
- [2026.01] We released `intel/llm-scaler-omni:0.1.0-b5` for Python 3.12 and PyTorch 2.9 support, various ComfyUI workflows and more SGLang Diffusion support.
- [2025.12] We released `intel/llm-scaler-vllm:1.2`, same image as `intel/llm-scaler-vllm:0.10.2-b6`. 
- [2025.12] We released `intel/llm-scaler-omni:0.1.0-b4` to support ComfyUI workflows for Z-Image-Turbo, Hunyuan-Video-1.5 T2V/I2V with multi-XPU, and experimentially support SGLang Diffusion. 
- [2025.11] We released `intel/llm-scaler-vllm:0.10.2-b6` to support Qwen3-VL (Dense/MoE), Qwen3-Omni, Qwen3-30B-A3B (MoE Int4), MinerU 2.5, ERNIE-4.5-vl etc. 
- [2025.11] We released `intel/llm-scaler-vllm:0.10.2-b5` to support gpt-oss models and released `intel/llm-scaler-omni:0.1.0-b3` to support more ComfyUI workflows, and Windows installation.
- [2025.10] We released `intel/llm-scaler-omni:0.1.0-b2` to support more models with ComfyUI workflows and Xinference.
- [2025.09] We released `intel/llm-scaler-vllm:0.10.0-b3` to support more models (MinerU, MiniCPM-v-4.5 etc), and released `intel/llm-scaler-omni:0.1.0-b1` to enable first omni GenAI models using ComfyUI and Xinference on Arc Pro B60 GPU.
- [2025.08] We released `intel/llm-scaler-vllm:1.0`.



## LLM Scaler vLLM

This file has been truncated. show original

… and the models there (with caveats noted above).

ComfyUI is up next:

https://github.com/intel/llm-scaler/blob/main/omni/README.md/#getting-started-with-omni-docker-image

long.barn4823 · February 22, 2026, 11:22pm

When using opencode or similar the initial request is more like 15k tokens. The benchmark looks like its requests are around 1k tokens. When I use a single B60 it struggles with that prefill stage when processing a longer context. I found single B60 acceptable performance for interactive chat but not up to a full local dev setup with opencode.

Would be fun to see how the full Battlematrix setup goes in something similar to that environment.

HappyPie · February 26, 2026, 5:56pm

Around 900-1000 tokens / second on both gtp oss 120b and qwen 3 30b?! I get “only” around 47 when running these two models on Strix Halo. (LM Studio as provider, trying to setup vLLM but takes time )
I do love my Strix but if I can get these 1000 tokens/second (and low TTFT) then this type of performance would be a DREAM for agentic use with like OpenCode.

Would love to see more testing for agentic use and OpenCode.

Michael_m · February 26, 2026, 6:06pm

can you bench it on yolo or object detection training please? inference as well if possible too. Its impossible to find good data on so many gpus when it comes to inference/object detection.

SUPEROXIDES · February 26, 2026, 8:32pm

Outside of that LLM Scaler suite, I’d wanna see how Proxmox handles SR-IOV, there was a YouTube comment saying it was zero-effort. It says the same on the article too (along with using Intel Extension for Torch, which is dead? So I dunno why), but no mention of a timeline for B50 support.

wendell · February 26, 2026, 9:08pm

ah check the thread above. that’s with massive parallelism i.e. lots of requests running at once.

for the openwebui test that’s more like what you’re doing on your strix halo and so it’s slower on Intel, too

i.e. Intel can to 50x requests at 20 tokens per second of the benchmark is 1000t/s total for example

edit: forgot, thats why I added the “Why short synthetic benchmarks look fast, but agentic coding (OpenCode, etc.) feels slower” section too – anticipating this line of reasoning

hoborific · February 26, 2026, 11:13pm

I bought 4 of these to play with at home and have had a surprisingly good time with intel-autoround and doing W4A16 quants to squeeze more context out of dense models.

couple jank home solutions below.

high cpu idle

VLLM_SLEEP_WHEN_IDLE=1 has a few downsides however stops a cpu thread per gpu being pinned at 100% which is nice for power and noise at home at the cost of latency when receiving a request after being idle.

high gpu fan idle

the fans still don’t ramp down when idle for me even with updated firmware and intel’s new BKC installer revision (6.14/1011-intel)

I ended up having to jank this together:

VLLM_CONFIGURE_LOGGING=1
VLLM_LOGGING_CONFIG_PATH=/home/user/scripts/logger.json

logger.json{
“formatters”: {
“json”: {
“class”: “pythonjsonlogger.jsonlogger.JsonFormatter”
}
},
“handlers”: {
“file”: {
“class”: “logging.FileHandler”,
“formatter”: “json”,
“level”: “INFO”,
“filename”: “/tmp/vllm.log”
}
},
“loggers”: {
“vllm”: {
“handlers”: \[“file”\],
“level”: “INFO”,
“propagate”: false
}
},
“version”: 1
}

#!/bin/bash

LOG_FILE=“/tmp/vllm.log”
SEARCH_STRING=“Avg prompt throughput”
COMMAND_TO_RUN=“sleep 15 && xpu-smi discovery > /dev/null”

tail -F “$LOG_FILE” | while IFS= read -r line; do
if \[\[ “$line” == *“$SEARCH_STRING”* \]\]; then
eval “$COMMAND_TO_RUN”
fi
done

you could probably get away with something as simple as triggering xpu-smi on a timer, the user needs video and render groups for this to work.

kewiha · February 26, 2026, 11:52pm

RE: use cases, an ollama or similar setup that RAGs linux man pages (perhaps even the ones on your system?) would be lovely if it improves the quality of results. I don’t get much value out of my P100 ollama+openwebui setup because models that fit in VRAM lie so much, and the big models that don’t take so long to run (partly because of low VRAM and partly because of obnoxious verbosity) that I just RTFM. It’s a shame because when manually tee’d up with the relevant man page as context the small models tend to give solid concise enough answers quickly.

Kimmono · February 27, 2026, 2:05am

I would also love a more detailed video or instructions on the forum for setting up a solution based on these cards?

Very interesting that three cards did not work?

Jimster480 · February 27, 2026, 2:36am

I have seen some of these new “on the fly” quantization but I am not sure what the downside to this is. I try to run BF16 models when I can.
Which version of Qwen did you use? Which Quant? I specifically typically run Q8 or BF16.
Your post suggests that the Qwen-30B-A3B is 29.1 Tokens/Sec for generation. For example my W7900 does 82 Tokens/Sec on Qwen-30B-A3B-Q8 from Unsloth with LM Studio’s LLama RoCM server.
I haven’t really played around with vLLM because it has been so good.

Have you done something with a large context that is Multi-Modal? I have a software I built which grades worksheets by generating answer keys + the kids answers.

I am thinking about launching a service with my software sometime this year targeted at Home Schooling families like mine.

hoborific · February 27, 2026, 3:01am

generally the lm head can’t be evenly divided, you’ll be stuck with 1,2,4,8 card splits and so forth for tensor parallel.

wendell · February 27, 2026, 3:04am

not a quant, kinda,

TORCH_LLM_ALLREDUCE=1 VLLM_USE_V1=1 CCL_ZE_IPC_EXCHANGE=pidfd VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 VLLM_WORKER_MULTIPROC_METHOD=spawn python3 -m vllm.entrypoints.openai.api_server --model Qwen/Qwen3-30B-A3B --enforce-eager --host 0.0.0.0 --port 8000 --trust-remote-code --gpu-memory-utilization 0.90 --max-model-len 2048 --block-size 64 --quantization fp8 --tensor-parallel-size 4

the quants would likely be faster, this is doing on the fly fp8 from the actual Qwen/Qwen3-30B-A3B hf repo – not unsloth. So higher quality but slower

yoink, gonna add these to the next video

Echo9Zulu · February 27, 2026, 3:34am

Hello Wendell!

Thanks for the video, great to see thorough coverage of Arc GPUs.

You asked for things to try, so I would invite you to test SearchSavior/OpenArc (on Github), my project which leverages OpenVINO runtime and includes features not mentioned in the release notes like pipleine paralell multi gpu and tensor parallel for dual socket CPU.

You could start with [Echo9Zulu/Hermes-4-70B-int4_asym-ov] or any of the other models on my HF. AFAIK, moe models are not yet supported with the HETERO device plugin meaning even the supported architectures wouldn’t allow you to use all 4 B60s. However, large dense models based on llama, qwen and mistral will work (off the top of my head). Like you mention in the video, Intel provides a “sandbox” where performance is fantastic, especially for prefill.

Keep in mind, this is a totally different stack from llm-scaler. OpenVINO GPU acceleration is based on OpenCL and oneDNN, not SYCL. For really broad context (from memory), when I did an architecture analysis on Kokoro, the pytorch jit trace reveals ~20 operations; OpenVINO graph covers ~60 different ops filled with kernel fusions, subgraphs and other tricks which leverage CPU/GPU features like AVX512, VNNI, XMX cores depending on what the target device supports. I would be interested to see how [VTune Profiler XMX usage compares in llm-scaler vs OpenVINO on B60.

I own 3x A770 and did some benchmarks using the tool I made for OpenArc, openarc bench meant to establish parity with llama-bench for comparing llama.cpp backends to OpenVINO. input tokens are sampled pseudo randomly from the target models vocabulary.

Hermes-4-70B-int4_asym-ov
Xeon W2255, 3x A770 running i915, Ubuntu 24.04, kernel 6.19.3-061903-generic
┏━━━━━┳━━━━━┳━━━━━┳━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ run ┃   p ┃   n ┃ ttft(s) ┃ tpot(ms) ┃ prefill(t/s) ┃ decode(t/s) ┃ duration(s) ┃
┡━━━━━╇━━━━━╇━━━━━╇━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│   1 │ 512 │ 128 │    1.49 │   185.94 │        344.3 │         5.4 │       25.10 │
│   2 │ 512 │ 128 │    1.01 │   180.61 │        506.8 │         5.5 │       23.95 │
│   3 │ 512 │ 128 │    1.00 │   177.63 │        511.9 │         5.6 │       23.56 │
│   4 │ 512 │ 128 │    1.01 │   176.34 │        507.6 │         5.7 │       23.40 │
│   5 │ 512 │ 128 │    1.01 │   175.81 │        508.1 │         5.7 │       23.34 │
└─────┴─────┴─────┴─────────┴──────────┴──────────────┴─────────────┴─────────────┘

I wanted to add some tests against vulkan llama.cpp on my a770s but was having some issues downloading Hermes-4 weights … and it’s getting late .

Thanks again for the great video!!!

I saw after writing that level1techs does not allow links.

For some NPU benchmarks, see issue 43 and 50

Jimster480 · February 27, 2026, 10:34am

I thought the full model is BF16 as is available from Qwen and unsloth? If this is running FP8, isn’t that the same as a Q8_0 Quant, its just on the fly as you mentioned?

Jimster480 · February 27, 2026, 10:34am

I only have 1x W7900 so I am not worried about dividing it at the moment.

Gordo_Gato · February 27, 2026, 11:27am

Hi Wendell,
I am a researcher building a database of key information regarding microbial species (fungi, bacteria) from scientific literature via a RAG pipeline. DNA analysis of environments often results in massive lists of microbial species, which is a challenge for researchers to make sense of, as information about each species is often sparse. We are aiming at making this database reliable enough to be able to perform large scale text-mining on these species lists to better our understanding of these environments.

For this purpose, we are using a fine-tuned model, google/medgemma-27b-text-it, which we have so far run on H100s in Google Cloud via the Vertex AI Batch Predictions API to process ~800m input tokens. This took about 38 hours of H100 compute time, according to my billing. Unfortunately, Google Cloud is quite opaque about what exactly happens in its batch API, so I can’t say for sure if that’s active processing time or if the VM/Model spin-up time is included in that, what the actual throughput in TP/S is, etc.

We are looking at trying prompt repetition, as Google has recently shown this massively improves non-reasoning model’s performance. This task could be used to test the batch throughput capabilities this hardware, although it will be very heavy on the prompt-preprocessing side. If you would like a dataset of these prompts, we would be happy to share this with you.

I can also DM you a link to this database, unfortunately I can’t share this publicly yet. Thank you for your insightful content!
Cheers

grimm · February 27, 2026, 1:04pm

Seconding this, I’ve used OpenArc for benchmarking on dual Xeon workstations and it’s reasonably performant with a good quant - here’s a dual Cascade Lake system for example (not even AMX… Phoronix has some very impressive Granite Rapids examples posted though) with a larger MoE:

ov_qwen3_235b_a22b_int4

┏━━━━━┳━━━━━┳━━━━━┳━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ run ┃   p ┃   n ┃ ttft(s) ┃ tpot(ms) ┃ prefill(t/s) ┃ decode(t/s) ┃ duration(s) ┃
┡━━━━━╇━━━━━╇━━━━━╇━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│   1 │ 512 │ 128 │   48.59 │   244.65 │         10.5 │         4.1 │       79.66 │
│   2 │ 512 │ 128 │   48.45 │   238.26 │         10.6 │         4.2 │       78.71 │
│   3 │ 512 │ 128 │   47.82 │   236.57 │         10.7 │         4.2 │       77.42 │
│   4 │ 512 │ 128 │   50.17 │   235.14 │         10.2 │         4.3 │       64.51 │
│   5 │ 512 │ 128 │   47.63 │   251.52 │         10.8 │         4.0 │       54.93 │
└─────┴─────┴─────┴─────────┴──────────┴──────────────┴─────────────┴─────────────┘

It can be a little rough with MoE architectures at times in terms of compute and memory efficiency, I expect there’s probably another 50% in optimization left on the table, but it’s pretty fun to be able to run “big Qwen” at better than dial up terminal speeds on a $500 server even if the prefill performance is really poor. If I had a device to pair with this to boost prefill in particular it’d even be somewhat useful for online workloads

progressEdd · February 28, 2026, 2:03am

More agentic coding experiments with comparisons between models. I want to replace github copilot and chatgpt codex.

Some content around fine tuning would be cool. Fine tuning a llm model act like Frank Renolds from always sunny would be a funny benchmark.

A shootout between, the top performing DGX Spark, a 7900 xtx, a 3090, and Blackwell would be also cool. I remember you mentioned the DGX Spark video that the real power is using the spark to fine tine a model.

Benchmarking a older project with newer hardware would be cool.

Following the Unsloth guides would be cool too

musicalderp · March 1, 2026, 6:38pm

Yeah, I was thinking about a comparison between this and a dgx spark