I’m not sure how to begin this so here’s my life story and ramblings as I try and justify my intel purchase.
I feel I need to get this information out to a place that can be scraped by agents or found by future adventurers rather than lost in scattered thoughts and sporadic messages.
I’ll hopefully keep updating this as I go
The build began in December 2025, an awful time to start, made do with what I had already laying around.
A Maxun Z890, an intel 225F and 64gb of questionable latency DDR5.
I bought two Intel B60s around Christmas just to see what it’s like, no way you can buy brand new intel cards cheaper than a used 3090 and run AI well, right? turns out you can, but you really have to work for it.
I had four cards by January, and now in July a PEX88096 10-port switch, 11-slot pcie “backplane” with plans to bring my total up to 8 cards by August.
Platform
Naively I began with the official intel endorsed config which was not a good time, Intel’s BKC/platform at that time was Ubuntu Server 25.04 with pinned dependencies, an offline installer and overall it was just a bad experience, I should’ve known better, they themselves would have known better, no one did better.
currently my platform and recommendation is kernel 7.1+ and compute-runtime being 26.22.38646.4 or newer and realistically any OS where you can install the dependencies and a modern kernel.
(GitHub - intel/compute-runtime: Intel® Graphics Compute Runtime for oneAPI Level Zero and OpenCL™ Driver · GitHub)
if you’re lazy or starting out, with Ubuntu 26.04 LTS you can follow:
Which is relatively new documentation and was not what intel were preaching earlier this year or even what is probably still in the llm-scaler documentation, avoid that.
Current Hardware
OS: Ubuntu 26.04 LTS (Resolute Raccoon) x86_64
Host: X870 Pro-A WiFi
Kernel: Linux 7.1.3-070103-generic
CPU: AMD Ryzen 9 9900X (24) @ 5.66 GHz
GPU 1: Intel Arc Pro B60 @ 2.00 GHz [Discrete]
GPU 2: Intel Arc Pro B60 @ 2.00 GHz [Discrete]
GPU 3: Intel Arc Pro B60 @ 2.00 GHz [Discrete]
GPU 4: Intel Arc Pro B60 @ 2.00 GHz [Discrete]
GPU 5: Intel Arc Pro B60 @ 2.00 GHz [Discrete]
GPU 6: Intel Arc Pro B60 @ 2.00 GHz [Discrete]
GPU 7: Intel Arc Pro B60 @ 2.00 GHz [Discrete]
GPU 8: Intel Arc Pro B60 @ 2.00 GHz [Discrete]
I recently bought 4 more cards bringing my total to 8, it required swapping the 265k plus/Z890 for a 9900x/X870.
having multiple cards on a consumer platform is a bad time, before buying the pcie switch it was running four cards, two at 5.0 x8 and two at 4.0 x4 across different root ports, the performance was awful and the latency was even worse.
but it ran and it was cheap.
I eventually purchased a PEX88096 10-port pcie switch and an 11-slot 8654 to PCIE board off aliexpress, with the current price of ECC memory and server platforms it was about a 10th the price of doing things properly.
example listings of the parts I bought, I did not use these stores, this is an example:
https://www.aliexpress.com/item/1005011868399389.html
https://www.aliexpress.com/item/1005012272021905.html
It’s worth noting that battlemage is pcie 5.0 and I halved my theoretical bandwidth using a 4.0 switch, however the results seem great and I can’t justify the price of 5.0 switches with the cards themselves being so cheap.
Added bonus of the B60 in particular is it’s wired x8, I only had to run a single 8654 8i cable per slot, per card, that’s at least 20 bucks of savings each.
With the particular 10 port switch I bought, it conveniently gives me the possibility of up to 10 x8 cards with dip switches on the pcie card itself for x16, x8/x8, 4*x4
xpu-smi topology -m
GPU 0/0 GPU 1/0 GPU 2/0 GPU 3/0 GPU 4/0 GPU 5/0 GPU 6/0 GPU 7/0 NIC0 CPU Affinity
GPU 0/0 S PIX PIX PIX PIX PIX PIX PIX PHB 0-23
GPU 1/0 PIX S PIX PIX PIX PIX PIX PIX PHB 0-23
GPU 2/0 PIX PIX S PIX PIX PIX PIX PIX PHB 0-23
GPU 3/0 PIX PIX PIX S PIX PIX PIX PIX PHB 0-23
GPU 4/0 PIX PIX PIX PIX S PIX PIX PIX PHB 0-23
GPU 5/0 PIX PIX PIX PIX PIX S PIX PIX PHB 0-23
GPU 6/0 PIX PIX PIX PIX PIX PIX S PIX PHB 0-23
GPU 7/0 PIX PIX PIX PIX PIX PIX PIX S PHB 0-23
NIC0 PHB PHB PHB PHB PHB PHB PHB PHB S 0-23
lspci -tv
-[0000:00]-+-00.0 Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Root Complex
+-00.2 Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge IOMMU
+-01.0 Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Dummy Host Bridge
+-01.1-[01-2f]----00.0-[02-2f]--+-00.0-[03-0c]----00.0-[04-0c]--+-10.0-[05-08]----00.0-[06-08]--+-01.0-[07]----00.0 Intel Corporation Battlemage G21 [Arc Pro B60]
| | | \-02.0-[08]----00.0 Intel Corporation Device e2f7
| | \-18.0-[09-0c]----00.0-[0a-0c]--+-01.0-[0b]----00.0 Intel Corporation Battlemage G21 [Arc Pro B60]
| | \-02.0-[0c]----00.0 Intel Corporation Device e2f7
| +-04.0-[0d-1e]----00.0-[0e-1e]--+-00.0-[0f-12]----00.0-[10-12]--+-01.0-[11]----00.0 Intel Corporation Battlemage G21 [Arc Pro B60]
| | | \-02.0-[12]----00.0 Intel Corporation Device e2f7
| | +-08.0-[13-16]----00.0-[14-16]--+-01.0-[15]----00.0 Intel Corporation Battlemage G21 [Arc Pro B60]
| | | \-02.0-[16]----00.0 Intel Corporation Device e2f7
| | +-10.0-[17-1a]----00.0-[18-1a]--+-01.0-[19]----00.0 Intel Corporation Battlemage G21 [Arc Pro B60]
| | | \-02.0-[1a]----00.0 Intel Corporation Device e2f7
| | \-18.0-[1b-1e]----00.0-[1c-1e]--+-01.0-[1d]----00.0 Intel Corporation Battlemage G21 [Arc Pro B60]
| | \-02.0-[1e]----00.0 Intel Corporation Device e2f7
| +-08.0-[1f-2a]----00.0-[20-2a]--+-00.0-[21]--
| | +-08.0-[22-25]----00.0-[23-25]--+-01.0-[24]----00.0 Intel Corporation Battlemage G21 [Arc Pro B60]
| | | \-02.0-[25]----00.0 Intel Corporation Device e2f7
| | +-10.0-[26-29]----00.0-[27-29]--+-01.0-[28]----00.0 Intel Corporation Battlemage G21 [Arc Pro B60]
| | | \-02.0-[29]----00.0 Intel Corporation Device e2f7
| | \-18.0-[2a]--
| +-0c.0-[2b-2e]----00.0-[2c-2e]--+-14.0-[2d]--
| | \-15.0-[2e]--
| \-1c.0-[2f]----00.0 Broadcom / LSI PEX880xx PCIe Gen 4 Switch
nvtop
Device 0 [Battlemage G21 (Arc Pro B60)] PCIe GEN 4@ 8x RX: N/A TX: N/A
GPU 2000MHz MEM N/A MHz TEMP 33°C FAN 0RPM POW 7 / 400 W
GPU[ 0%(eff 0%)] MEM[ 0.039Gi/23.906Gi]
Device 1 [Battlemage G21 (Arc Pro B60)] PCIe GEN 4@ 8x RX: N/A TX: N/A
GPU 2000MHz MEM N/A MHz TEMP 31°C FAN 1305RPM POW 9 / 400 W
GPU[ 0%(eff 0%)] MEM[ 0.039Gi/23.906Gi]
Device 2 [Battlemage G21 (Arc Pro B60)] PCIe GEN 4@ 8x RX: N/A TX: N/A
GPU 2000MHz MEM N/A MHz TEMP 30°C FAN 1305RPM POW 9 / 400 W
GPU[ 0%(eff 0%)] MEM[ 0.039Gi/23.906Gi]
Device 3 [Battlemage G21 (Arc Pro B60)] PCIe GEN 4@ 8x RX: N/A TX: N/A
GPU 2000MHz MEM N/A MHz TEMP 33°C FAN 0RPM POW 8 / 400 W
GPU[ 0%(eff 0%)] MEM[ 0.039Gi/23.906Gi]
Device 4 [Battlemage G21 (Arc Pro B60)] PCIe GEN 4@ 8x RX: N/A TX: N/A
GPU 400MHz MEM N/A MHz TEMP 32°C FAN 0RPM POW 9 / 400 W
GPU[ 0%(eff 0%)] MEM[ 0.039Gi/23.906Gi]
Device 5 [Battlemage G21 (Arc Pro B60)] PCIe GEN 4@ 8x RX: N/A TX: N/A
GPU 400MHz MEM N/A MHz TEMP 32°C FAN 0RPM POW 8 / 440 W
GPU[ 0%(eff 0%)] MEM[ 0.039Gi/23.906Gi]
Device 6 [Battlemage G21 (Arc Pro B60)] PCIe GEN 4@ 8x RX: N/A TX: N/A
GPU 400MHz MEM N/A MHz TEMP 32°C FAN 0RPM POW 9 / 400 W
GPU[ 0%(eff 0%)] MEM[ 0.039Gi/23.906Gi]
Device 7 [Battlemage G21 (Arc Pro B60)] PCIe GEN 4@ 8x RX: N/A TX: N/A
GPU 400MHz MEM N/A MHz TEMP 32°C FAN 0RPM POW 9 / 440 W
GPU[ 0%(eff 0%)] MEM[ 0.039Gi/23.906Gi]
old intel topology:
I hit the limitation of this intel consumer platform and only 6 cards will post, any more throws a D4 qcode and fails to post, MMIO space can’t be user adjusted on intel consumer platforms, at least not mine. 6/8 was the final total.
xpu-smi topology -m
GPU 0/0 GPU 1/0 GPU 2/0 GPU 3/0 GPU 4/0 GPU 5/0 NIC0 NIC1 CPU Affinity
GPU 0/0 S PIX PIX PIX PIX PIX SYS SYS 0-19
GPU 1/0 PIX S PIX PIX PIX PIX SYS SYS 0-19
GPU 2/0 PIX PIX S PIX PIX PIX SYS SYS 0-19
GPU 3/0 PIX PIX PIX S PIX PIX SYS SYS 0-19
GPU 4/0 PIX PIX PIX PIX S PIX SYS SYS 0-19
GPU 5/0 PIX PIX PIX PIX PIX S SYS SYS 0-19
NIC0 SYS SYS SYS SYS SYS SYS S PHB 0-19
NIC1 SYS SYS SYS SYS SYS SYS PHB S 0-19
lspci -tv
-+-[0000:00]-+-00.0 Intel Corporation Device 7d1b
| +-04.0 Intel Corporation Device ad03
| +-06.0-[01-29]----00.0-[02-29]--+-00.0-[03-0c]----00.0-[04-0c]--+-10.0-[05-08]----00.0-[06-08]--+-01.0-[07]----00.0 Intel Corporation Battlemage G21 [Arc Pro B60]
| | | | \-02.0-[08]----00.0 Intel Corporation Device e2f7
| | | \-18.0-[09-0c]----00.0-[0a-0c]--+-01.0-[0b]----00.0 Intel Corporation Battlemage G21 [Arc Pro B60]
| | | \-02.0-[0c]----00.0 Intel Corporation Device e2f7
| | +-04.0-[0d-1b]----00.0-[0e-1b]--+-00.0-[0f-12]----00.0-[10-12]--+-01.0-[11]----00.0 Intel Corporation Battlemage G21 [Arc Pro B60]
| | | | \-02.0-[12]----00.0 Intel Corporation Device e2f7
| | | +-08.0-[13-16]----00.0-[14-16]--+-01.0-[15]----00.0 Intel Corporation Battlemage G21 [Arc Pro B60]
| | | | \-02.0-[16]----00.0 Intel Corporation Device e2f7
| | | +-10.0-[17-1a]----00.0-[18-1a]--+-01.0-[19]----00.0 Intel Corporation Battlemage G21 [Arc Pro B60]
| | | | \-02.0-[1a]----00.0 Intel Corporation Device e2f7
| | | \-18.0-[1b]--
| | +-08.0-[1c-24]----00.0-[1d-24]--+-00.0-[1e]--
| | | +-08.0-[1f-22]----00.0-[20-22]--+-01.0-[21]----00.0 Intel Corporation Battlemage G21 [Arc Pro B60]
| | | | \-02.0-[22]----00.0 Intel Corporation Device e2f7
| | | +-10.0-[23]--
| | | \-18.0-[24]--
| | +-0c.0-[25-28]----00.0-[26-28]--+-14.0-[27]--
| | | \-15.0-[28]--
| | \-1c.0-[29]----00.0 Broadcom / LSI PEX880xx PCIe Gen 4 Switch
| +-06.1-[2a]----00.0 INNOGRIT Corporation NVMe SSD Controller IG5236 [RainierPC]
| +-07.0-[2b-54]--
| +-07.1-[55-7e]--
nvtop
Device 0 [Battlemage G21 (Arc Pro B60)] PCIe GEN 4@ 8x RX: N/A TX: N/A
GPU 400MHz MEM N/A MHz TEMP 37°C FAN 0RPM POW 6 / 400 W
GPU[ 0%(eff 0%)] MEM[ 0.039Gi/23.906Gi]
Device 1 [Battlemage G21 (Arc Pro B60)] PCIe GEN 4@ 8x RX: N/A TX: N/A
GPU 400MHz MEM N/A MHz TEMP 36°C FAN 0RPM POW 8 / 400 W
GPU[ 0%(eff 0%)] MEM[ 0.039Gi/23.906Gi]
Device 2 [Battlemage G21 (Arc Pro B60)] PCIe GEN 4@ 8x RX: N/A TX: N/A
GPU 400MHz MEM N/A MHz TEMP 34°C FAN 0RPM POW 11 / 400 W
GPU[ 0%(eff 0%)] MEM[ 0.039Gi/23.906Gi]
Device 3 [Battlemage G21 (Arc Pro B60)] PCIe GEN 4@ 8x RX: N/A TX: N/A
GPU 400MHz MEM N/A MHz TEMP 36°C FAN 0RPM POW 8 / 400 W
GPU[ 0%(eff 0%)] MEM[ 0.039Gi/23.906Gi]
Device 4 [Battlemage G21 (Arc Pro B60)] PCIe GEN 4@ 8x RX: N/A TX: N/A
GPU 400MHz MEM N/A MHz TEMP 37°C FAN 0RPM POW 9 / 400 W
GPU[ 0%(eff 0%)] MEM[ 0.039Gi/23.906Gi]
Device 5 [Battlemage G21 (Arc Pro B60)] PCIe GEN 4@ 8x RX: N/A TX: N/A
GPU 400MHz MEM N/A MHz TEMP 38°C FAN 0RPM POW 9 / 400 W
GPU[ 0%(eff 0%)] MEM[ 0.039Gi/23.906Gi]
P2P isn’t working on this awful intel platform existence
This one was rough, I ended up having to do all the ACS in grub and escape them for some reason, confirm they’re all there after a reboot in /proc/cmdline
GRUB_CMDLINE_LINUX_DEFAULT="pci=disable_acs_redir=0000:07:01.0\\;0000:05:10.0\\;0000:03:00.0\\;0000:15:01.0\\;0000:0f:08.0\\;0000:03:04.0\\;0000:11:01.0\\;0000:0f:00.0\\;0000:0b:01.0\\;0000:05:18.0"
the addresses will be different, easiest way is to fire up VLLM or another backend that’ll attempt P2P and extract the system’s cries from dmesg.
may also not hurt to try intel_iommu=off while you’re at it.
There’s probably a better way to do this than manually adding all the addresses like this, but it worked.
I have a feeling this just works on AMD but I had intel.
tips, tricks, problems and solutions
underclocking and performance
These cards don’t draw a lot of power individually and the 220wish TDP is quite extreme for AI workloads.
Under load/decoding it’s about 100w, 150w doing batches quite aggresssively.
I’ve found reducing maximum clock from 2400mhz to 2000mhz nets around a 15% (80-85w vs 100w) reduction in power for a negligible loss in single request LLM performance, haven’t tested further, should be explored.
idle without working ASPM can be rough, around 35w per card.
With working ASPM and an active process such as nvtop in the backround they’re closer to 10w, without a process they end up closer to 35w again.
cards don’t show up in xpu-smi discovery | dmesg complains about bar space/allocations
GRUB_CMDLINE_LINUX_DEFAULT="pci=realloc=off"
turns out this is an XE quirk that I don’t care to investigate further, likely related to the XE VF passthrough mode, didn’t go away with sr-iov enabled either.
this relates to: a vbios update broke my cards from showing up in xpu-smi/xpumanager.
There’s likely performance implications to this but implications are better than a non-functioning system.
P2P isn’t working AMD edition
This one was rough, I ended up having to do all the ACS in grub and escape them for some reason, confirm they’re all there after a reboot in /proc/cmdline
GRUB_CMDLINE_LINUX_DEFAULT="pci=disable_acs_redir=0000:07:01.0\\;0000:05:10.0\\;0000:03:00.0\\;0000:15:01.0\\;0000:0f:08.0\\;0000:03:04.0\\;0000:11:01.0\\;0000:0f:00.0\\;0000:0b:01.0\\;0000:05:18.0"
the addresses will be different, easiest way is to fire up VLLM or another backend that’ll attempt P2P and extract the system’s cries from dmesg.
There’s probably a better way to do this than manually adding all the addresses like this, but it worked.
I had a feeling this was an intel quirk but it happens on AMD, I likely need to configure the switch itself but it works for now so good enough.
fans don’t update when the cards are idle, temperature doesn’t update while idle
For the last 7 months this has remained a problem for me, I currently run nvtop in a screen as part of my startup jobs, however the most reliable way to update the cards is to simply probe them by running something such as xpu-smi discovery.
your user will require video/render groups for this to work sudoless.
high power idle after a load, cards wedge, errors when waking cards
this one was relatively common sense but I forgot at some point during my platform migration and noticed they were idling at 31w each, fix was enabling
iommu=pt
to stop them getting wedged when P2P is enabled, I also noticed errors reported with this on/full.
another problem I had was any kind of load caused the cards to wedge and never return to a proper idle, “solution” was to disable d3cold
#!/bin/bash
# Find all BDFs for Intel Arc B60 (8086:e211)
GPU_BDFS=$(lspci -d 8086:e211 -D | awk '{print $1}')
if [ -z "$GPU_BDFS" ]; then
echo "No Intel Arc GPUs (8086:e211) found."
exit 1
fi
for bdf in $GPU_BDFS; do
echo "Pinning $bdf out of D3cold..."
echo 0 | sudo tee /sys/bus/pci/devices/$bdf/d3cold_allowed
echo on | sudo tee /sys/bus/pci/devices/$bdf/power/control
done