I am running Swift Qwen3.8-27b FP8 with FP8 KV. I am using RoPE/YaRN to extend context to 3x 512k for on both of my systems and I use LiteLLM to distribute load between them. After some tuning in Turnstone and Kilo I am finding a whole lot of successfully completed programming tasks that just work. They are making my Hermes agent that I started off with look like frustrated toddler surrounded by too many toys to pay attention to any of them for more than a couple seconds
Just a heads up if this isn’t a typo: if you’re shooting for 512k you’ll have to use a rope multi of 2x (2x 256) - It will work with higher / highest multi of 4x with lower context lengths but you will loose some rotational resolution (recall ability) when doing so. Best to keep it directly inline with your context window. Also remember the differece between Kibibytes and Kilobytes - If wanting 512k - the amount to enter will be: 512x1024=524288.
I know you and most know all this, just a PSA.
Well last night, I decided to upgrade my Swift models to swift 1.5 with unquantized KV cache. I actually had Hermes make the change for me, when you have two different computers serving models to LiteLLM and then using that to serve your services it enables self upgrades without interrupting. I actually gave it a very low probability of being successfully implemented but I was supposed to find 2 healthy copies of swift 1.5 being served this morning. If the reasoning tokens stay low, I won’t miss the extra headroom the quantized KV cache was providing.
1 Like
Interesting you mention that - I’m stretching the legs of swift 1.5 into fp8 kvcache and was able to tweak all the way up to 1m context with 400mb left per card! lol - But it’s working amazingly and honestly very competent. Making sure that ‘preserve_thinking’ and ‘enable-prefix-caching’ (which the latter is enabled by default on v1 vllm i found out now that I was able to get v1 working) are enabled allows the workflow access to prior work and cross agent information - I’ve enabled work-trees in my workflow as well and OMG that has enabled some very interesting and useful sharing of information among the agents. - I’m now experimenting with MTP and have found that with v1 engine in vllm that the slow-downs across multiple agents is no longer there - so where i was seeing ~30tps per agent before I’m now seeing ~55tps … I’m still putting it through it’s paces, but it all looks promising.
But yes, Swift 1.5 is really good and I feel much better at just about every task than the qwen base or 1.0.
1 Like
I think I am giving up on Kilo Code. It just isn’t working with my workflow since it runs in my Code Server but the latest version no longer has persistance across devices. I like Kilo and it is easy to use but after losing my sessions multiple times I am moving on. I get other things done during the day while I let the agents churn on my project but I monitor their progress. In the mornings I like to monitor their progress from my phone, and with Kilo this doesn’t work well. I find that Turnstone works well but it doesn’t learn as well as Kilo or Hermes so I am have been working on improving the automation for creating and updating skills in Turnstone. I went back to using Swift 1.5 Qwen3.8-27b in FP8 with 512k context for all of my sessions, which helps keep things running without the constant session compaction and allows the scope of the sessions to be quite a bit larger.
1 Like
I get this. Kilo is for sure more of a sit and use it tool. That is how I use it. I’m coding along side it, it’s more of my assistant than a coworker. I completely get the need for a different harness if that’s not your specific activity at any one point in time and need something to be more autonomous. I have also found swift 1.5 27b @ fp8 with preserve_thinking to be awesome while allowing me to push the context window up to crazy amounts I never thought I’d be able to obtain / utilize. I’m also using MTP since vllm 30 it now works with multiple streams. I’ve even found that MTP-4 is my sweet spot so far and I can push well over double the decode I was previously capable of.
I’ve been using a tool I’ve been building to tune in my ai server that has been very helpful, you might be interested in: Crucible LLM :: Bench your LLM and server. Is it fast enough… for everyone?
Let me know what you think as I’m looking for all forms of feedback.
2 Likes
I wish I could hit a button that delivered you a beer, dude. Been loving just about everything you post tbh
1 Like
Is anyone running a lightweight judge model for turnstone? I am curious if anyone is using one. My workflow has been so much better since switching from Kilo Code to turnstone but turnstone takes a lot of tuning and optimizaiton work to so far in my experience. Kilo was much faster at getting up and running, I just couldn’t monitor the work from my phone, tablet, and computer as each device would have a separate session with no continunity between them.
1 Like
I am. I use Gemma e4b on an Nvidia A4000 as a judge, but it can’t really keep up with radiance, so I am looking to upgrade to a spare 3080ti I had lying around… benches look solid and only 7G of ram at full context.
2 Likes
I setup a tiny judge model today and now turnstone is so much faster. I was using my primary Qwen3.8-27B models for the Judge process and the latency for the reasoning was slowing everything down. The tiny model that I used is https://flow-ai.com/judge and it is super fast in FP4 on my RTX 6000 Pro Blackwell. Long story short - maybe try a small cheap model as the judge to speed up the process. Now I can’t really say if I have comprimised the judge functions but on a speed basis this is a huge improvement.
1 Like
I took another step in the AI Evolution of self hosted turnstone driven AI. I added in Google Gemini servers to manage a couple critical projects. It is crazy how few tokens Gemini Argon uses in comparison to Qwen3.8-27B. I am using Gemini as the Executive and Qwen as the muscle to do all of the tasks and I will get millions of token of work done from Qwen for only 30-100k token from Gemini, and I am hitting my project goals faster, which is great for business and easier on the bank account than going full cloud or full metal.
1 Like
Here is my opinion of using a frontier model as the turnstone coordinator. I am getting more done faster but I have lost a lot of visibility on what is actually going on. The frontier model, in this case Gemini Argon, is essentially commanding nodes in a non-human readable format. I appreciate that this minimizes output tokens and cost but I feel left out of the conversation / process now. Previously with QWEN3.8-27B I monitored the work flow and understood all of the pain points and issues that needed to be resolved. But if it is able to solve the math for the custom 5 level dynamic AMR meshing I need for my plasma simulations in WarpX then it will be worth every penny as I have already spent more than a month trying with Qwen fully local with very little progress.
1 Like
300k output tokens from a frontier model directing my local models to do all of the work and my simulations are up and running…
1 Like