Local LLM Based Coding on HPC Equipment

I get this. Kilo is for sure more of a sit and use it tool. That is how I use it. I’m coding along side it, it’s more of my assistant than a coworker. I completely get the need for a different harness if that’s not your specific activity at any one point in time and need something to be more autonomous. I have also found swift 1.5 27b @ fp8 with preserve_thinking to be awesome while allowing me to push the context window up to crazy amounts I never thought I’d be able to obtain / utilize. I’m also using MTP since vllm 30 it now works with multiple streams. I’ve even found that MTP-4 is my sweet spot so far and I can push well over double the decode I was previously capable of.

I’ve been using a tool I’ve been building to tune in my ai server that has been very helpful, you might be interested in: Crucible LLM :: Bench your LLM and server. Is it fast enough… for everyone?

Let me know what you think as I’m looking for all forms of feedback.

2 Likes