One week of Local AI on Strix Halo

[felix920506]

This is my experience after one week of running local AI on Strix Halo, in mid-August, 2026.
I’ll put the TL; DR up front: The hardware is fine, but the software stack is far from there yet.

I have 2 setups under my control, one at work and one at home, with a bit different hardware.

Home Hardware Setup

• 2x FEVM FAEX1 128GB nodes
• USB4 interconnect
• YMTC ZhiTai TiPlus 7100s 2TB SSD in each node
• Chicony Power Bricks (because the factory Huntkey ones melted their casings)

This one is sitting there doing nothing right now as I am trying to come up with things to run on there

Work Hardware Setup

• 2x Framework Desktop 128GB nodes
• 5Gbps Ethernet Interconnect (Via TPLink Switch)
• YMTC ZhiTai TiPlus 7100s 1TB SSD in each node

Work Software Setup

• Deepseek V4 Flash 0731 model
• ds4 inference engine
• Open-webui frontend (I wanted to do agents but management is afraid they will nuke all our stuff)

Backstory

Business ordering Framework turned out to be a complete pain and it took about a month to get all the crap sorted and for us to receive the machines. However since this is about strix halo I’m not going to elaborate here.

The experience

Knowing that support for Deepseek V4 on vLLM is nonexistent on AMD, I started with llama.cpp. However with llama.cpp and ROCm backend I was hitting a bug where it crashes at about 4000 tokens input when using rpc (basically useless). Vulkan crashed at about 6000 ish, still basically useless.

I then switched to ds4 (purpose built engine for deepseek v4), which doesn’t crash but since it doesn’t support batching on AMD or true pipeline parallelism it means one node is always sitting there doing nothing at any given moment.

Performance

Take these with a grain of salt, they weren’t properly benchmarked

llama.cpp rocm:
Prefill I forgot, Decode about 12.5t/s single stream, 16t/s aggregate (unsloth Q8 + BF16 Dspark)
though this one is basically useless since it crashes at ~4000 tokens context

llama.cpp Vulkan:
Prefill I forgot, Decode about 8t/s single stream, 11t/s aggregate (unsloth Q8 + BF16 Dspark)

ds4:
Prefill 60tok/s, Decode 12.5t/s (the ds4 specific weights they provide)

Why buy AMD

Because we weren’t completely sure if we wanted this or not when we ordered these. AMD gave us the best exit strategy as these are very useful as general purpose workstations, unlike the main other options (Apple M5 Max and Nvidia GB10) which are basically useless for us outside AI inference). If I were to pick something to run local LLMs on today I definitely would not pick AMD for sure.

General teething issues / ecosystem issues

I’m not blaming one but here are the stupid things I had to deal with along the way

• No prebuilt rocm docker images, I have to build the rocm images myself
• One both FEVM systems one of the USB4 ports doesn’t work
• Broken llama.cpp rocm implementation for deepseek 4
• Broken llama.cpp Vulkan implementation
• Worse support in DS4 of the platforms it targets
• Broken llama.cpp rocm implementation for gemma 4 (tried to run it on home setup to test)

It genuinely feels like AMD is being treated as a 3rd class citizen on all the major AI stuff, after Nvidia and Apple.

Conclusion

Nice hardware being held back severely by software.

What next

I don’t know what I should run on the home setup currently, maybe throw some claude tokens at some of the problems I faced since I have some free tokens. Or maybe you guys can suggest things to do.

[mashie]

Shame you didn’t go down the DGX Spark route, no exit strategy would have been needed then as they work really well.

[felix920506]

should have gone Apple tbh, it is still easier to sell macbooks than DGX sparks

[mashie]

True, but with a DGX Spark you don’t need to sell them in the first place. Just use them as intended.

[felix920506]

At least at work where higher ups aren’t even sure if AI is useful or not to us Strix halo is the best option since the GB10 systems are basically expensive paperweights if we don’t run AI on them.

[felix920506]

I decided to just throw my unused Claude and ChatGPT quotas which I generally don’t completely use up and see if it can improve ds4 better on strix halo

[lhl]

The state of vLLM on Strix Halo is pretty rough. So rough that I consider it mostly broken. There are AMD devs working on it atm, but the state is glacial/generally bad. The best bet for a “working” vLLM w/ a validated set of models is here: GitHub - kyuz0/amd-strix-halo-vllm-toolboxes · GitHub

If you get it working though, it’s still quite slow/underoptimized. I will say from first-hand experience that if your goal is running DS4 Flash, to forget about vLLM - it runs terribly even on Nvidia cards without DeepGEMM. The difference between running the default Triton and the DeepGEMM kernels on TP=2 PRO 6000s is 5-10x.

There are many Strix Halo focused forks of llama.cpp (rotating cast) - these basically will never make their way upstream, but some are very active, like Ciru’s stuff, which you might want to check out: GitHub - ciru-ai/ROCmFPX: ROCmFPX Family for AMD Hardware and Processors. More quants and special agent quants · GitHub

For “playing” around (read: burning frontier tokens), I’ve spent a few months on custom kernels for c=N/MTP friendly running of various models on gfx1100 and gfx1151 (Strix Halo): GitHub - shisa-ai/hipEngine · GitHub

There are a few other projects like GitHub - warpfront/hipfire: RDNA-native LLM inference engine in Rust. · GitHub and I just spotted jcbtc/Qwen3.8-27B-IU4-Kairic-Edge · Hugging Face which looks interesting (haven’t tried either, so YMMV).

I’ve been running my own evals (this is a mix of various agentic/coding benchmarks including BigCodeBench, Terminal Bench, and DeepSWE / SWE-bench subsets running in Pi as the harness) and surprisingly, Qwen 3.8 27B outscores DS4 Flash. It probably runs faster as well. Also Ornith 1.5 35B-A3B score pretty decently and is also quite fast (and is much more token efficient!):