Overall, I’ve been pleased with my purchase of a 128GB Strix Halo (Bosgame) but the prefill has been killing me. So when Qwen3.8 27B dropped, I blew the last week or so of my life in front of Claude trying to make it usable for agentic workflows.
I honestly had more success than expected.
I ended up building an inference engine tailored to the Strix Halo and Qwen 3.8 architecture… It turned out being about 2x as fast at prefill as the next closest engine I could find at higher model weights. >550t/s @32k depth. There might be a little more room to go but it is approaching the wall of physics and my hardware can’t run past 2400Mhz under load. If you have good cooling and can hit closer to the 2900MHz clock, you can probably push 600+ @32k context depth.
Anyway, I wasn’t going to release this, but decided if even a few people find it useful and it saves them time its worth it.
I tried to make it as simple as possible to just fire up a server and run it. The instructions are in the repo. Model weights on hugging face.
I’m using this with my own harness (in development) and it is definitely usable. Is it Deepseek v4 flash fast? No. But once you warm it up and you’re careful with context its been extremely useful and enjoyable to work with. The engine does prompt caching so it feels pretty good.
If you have a Strix Halo, try it out and let me know what you think. I hope it saves you time.
Thanks for looking.
P.S - I don’t have social media or reddit so this is probably the only place I’m going to post this. If you want to leave feedback just put it in this thread, or as an issue on github.
It’s definitely cool. Thanks for this. I will recommend it to some other people that I know to have Strix Halo. Personally, I have already an A3 and I’m about to have an A100 setup as well, but I think pushing the envelope on sTrix Halo is very cool and I have definitely considered purchasing one of these, but we’ll probably hold off until next generation.
I appreciate you taking the time to check it out. Thank you. …jealous of your A100
I was on the fence about the Strix Halo as well, but ultimately ended up pulling the trigger on one because it was (at the time) the cheapest way to get 128GB VRAM to play around with. The plan was to wait for the new Mac Studio and use this machine as a stop gap. I don’t regret buying one.
I have a soft spot for AMD and really want them to succeed. It’s just annoying that RDNA3.5 is a “near miss” and we are unfortunately stuck with this architecture for at least another year. Gorgon Halo or whatever they are calling it is incremental…
Next generation / RDNA5 or Medusa Halo (I think thats the name) is going to be good. Until then I hope the community can figure out ways to squeeze a little more performance out of what we have.
There are NPUs sitting around doing absolutely nothing on Strix Halo because turning them on immediately causes you a 15% performance hit. I didn’t spend too much time investigating, but my guess is this will need kernel driver work in iommu upstream.
Unfortunately NPU’s are actually not really made for inference. There are ways to integrate them but typically like you said they slow down GPUs when combined together.
The A100s will be deployed this weekend using a trick. I might make a thread about it here once it is all done.
The instructions seem clear enough, but how are you achieving these improvements on prefill? Could the same changes be ported to other platforms in similar ways?
Im sure they could be, if someone wanted to take the time. Its a lot of work that consumes wall clock. There is no way around that. The core idea here is single hardware architecture single model architecture single quant single engine. Whether this could be generalized was beyond the scope of this work. IMO, the future is kernel auto tuning and releasing dedicated inference engines along side model releases.
Hi! Running halogen-flash 0.2.0 on a Bee-Link GTR9 128GB and it’s
been great. Over 135 requests my session grew from ~50 to 90K tokens of context and decode stayed at
33–53 t/s (mean ~39) the whole way. No slowdown with depth, which my old llama.cpp setup just couldn’t do (it dropped to ~15 t/s and OOM’d at 133K). 94% of turns hit the prompt cache, so
follow-ups at 90K context take ~2s. Prefill on big cold turns runs at 535–800+ t/s, also flat with
depth. Best experience on this hardware I’ve found, hands down.
One thing I’m missing: image support. The model family is multimodal and I’d love to feed it
screenshots/docs, but there’s no mention of vision in the repo or /health. Is it on the roadmap?
Hi! You’re welcome. I’m glad you’re having a good experience with halogen! I just pushed 0.3.1 of flash-server this morning, which is a concurrency update… On your hardware, you can probably run up to 3 full context conversations simultaneously. Check out the readme.
Regarding multimodal / vision, you’re the second person to ask about it, its in the queue, and its a lot of work.
Hey, same here. Tried various different ways to run Qwen3.8 Flash Next and the halogen-flash has been fantastic. Absolutely stable and fast. By far the best experience on my Bosgame M5 so far.
Just like @kx1 I’d love to see multi-modal input and a more rich API.
And I would also appreciate it being open sourced. For community contribution but also for security reasons. Would feel more at ease to know what code I’m executing
Either way, thank you @wenis for this. It has made both qwen3.8 27b and qwen3.8-flash-next actually usable on my strix halo.
I appreciate the feedback. Thanks for checking it out and I’m glad halogen is working well for you. Multi-modal is next. Sorry its taking so long. I only have one Strix Halo machine and it has been in use 24/7. I’m a little worried about regression with the vision kernels so I want to take the time to get it right. The whole point of halogen is so people don’t have to think about the inference layer… just fire it up, point your agent at it, and go do something awesome. Quick question for you. How are you supporting vision now? Do you use hermes agent? Or do you not use vision at all?
Hi! I tried your engine today on my ASUS Strix tablet/PC powered by 395+. It worked perfectly well (this device is a little slower because of thermal specs). But I have 6 more Strix Halos here (4 Minisforum Max, an Evo x2 and a generic box from a discontinued brand. Great work, it is really fast as advertised. Looking forward to having vision support, or in case you open source this I would gladly try to help and PR if it works reasonably well (have Claude Max and Codex Pro plans). Thanks again!
Great work for the halo community and please ignore the comments from people incapable of reading. Appreciate the time and effort (and electricity ) you spent on this.
I’m eagerly waiting for the second small box I purchased to arrive, then I will offload all services there, install arch and run halogen flash on the halo.
I’m also interested in your design and development workflow. Since you are using qwen 3.8 to do all these on the halo, would be interesting to read the journey as a separate blog post when you are done.
(deleted the previous post while trying to fix a typo by mistake)
Thank you! The TLDR is a lot of late nights, and a lot waiting on tests.
I am working on the harness, which will include all of my findings across the 27B and flash model for qwen 3.8. I think this is the important work that resulted from halogen and not necessarily the kernels.
