Mainly asking for local LLM use. I have both machines already.
Is there actually anything useful I could do with a 64GB Strix Halo box that I can’t already do better on the 5090?
I’m guessing the main advantage would be loading models larger than 32GB into memory, even if inference is slower. But does that make it worth playing with one for local LLMs? It seems like there is no model better than Qwen 3.8 27B for this hardware anyway?
I’m not sure what your use case is, but I think Q8_0 would still be more accurate than Q4 turbo quant. However, I don’t have personal experience to compare those.
I don’t have Strix Halo, but I do run a mix of models across hardware. Even if it isn’t as performant, it can still be really handy to just have more capacity to do work in parallel.
For example, with Turnstone I have an A6000 system with 3.8 27B 262K context acting as my coordinator while I have my 4090 system with 3.8 27B 131K context but double the performance handling the child workstreams. I also have an RTX 5000 Mobile system running Gemma 4 12B QAT for the judge role (thinking about switching this out, it’s a bit slow for what it is) and a Mac Studio M1 Ultra 64GB on 3.6 35B A3B 262K for simpler child workstream tasks (the Turnstone coordinator can specify an agent alias when dispatching a workstream and can be pretty good at routing workloads based on needs). I can’t wait for 3.8 35B A3B to drop.
Here’s a super quickstart - may need some troubleshooting?
git clone github.com/turnstonelabs/turnstone.git
cd turnstone
cp .env.example turnstone/deploy/.env
cd turnstone/deploy
[nano|vim] .env
# Set at minimum TURNSTONE_JWT_SECRET, POSTGRES_PASSWORD, WORKSPACE_MOUNT following the instructions
# You can configure the LLM endpoints and stuff from web gui once running
docker compose pull && docker compose build && docker compose up -d
# Wait a min for it all to complete
# From another computer browse to `https://<host-ip-address>:8443
# Setup your initial login
# Go under Manage > System > Models and add your endpoints (put "dummy" or w/e for the key even if one isn't required and don't miss the "Detect" button at the bottom once you've added the Base URL (e.g. http://192.168.1.2:8000/v1) and API key at minimum - this will let you select your Model ID from a dropdown)
# Click "Sync to nodes"
# Go to the Roles tab and assign your different models to different roles, namely Coordinator, Judge, and Task Agent
# Back on the Dashboard tab you can create either a Coordinator stream or Interactive stream
# Coordinator can't access your workspace mount directly, but rather it dispatches child processes (task agents) to do the work while it keeps everything more or less on the rails
# Interactive stream works 1-on-1 with what's effectively a child workstream, and can be better when you need something simpler done quickly
Notice above that we are diving into the more “production-ready” compose file within ~/turnstone/turnstone/deploy rather than base ~/turnstone. This gives you a simpler (to start) 1 console, 1 node config whereas ~/turnstone is more dev oriented and spawns like 10 nodes by default which I don’t see the point of. When you want to get fancy you can dive into the bare-metal node options in the documentation and give Turnstone direct control over a server (the included nodes run in a docker container).
Good luck!
EDIT oh and make sure you’re on either main (v1.8) or another specified stable branch, not dev
Performance wise not at all, but in pratice many starters underestimate the importance of context.
So the 64 GB of the halo can give you that, and since you have both, just experiment and come back to us.
+1 for @ChrisL1T setup, and thank you for the turnstone reminder I was waiting it to mature and look at the orientation and then I forget to check.
Seems more straigth forward than Hermess implementation.
Lemme know if I gave any misinfo there - just going off memory. I also have some compose.override.yaml and Dockerfile.custom in my config to add stuff like Go to the turnstone-server-1 container, and a couple bare-metal hosts added which need POSTGRES_BIND=0.0.0.0 in the .env and at one point had the Discord channel configured as well…probably need to reimplement that since I think recent updates let you have it churn on work in the background then ping you on Discord when finished //TODO