Hello, I’m new to self hosted AI workloads and wanted to know what the recommended specs for decent performance with something like odysseus. I currently have a 30090ti to spare but not sure if that’s gonna be enough. Thanks in advance!
3090 Ti should do fine. A Qwen3.8-27b quant should fit in the VRAM on that and its a wickedly good small model. Whether you can fit an AWQ quant or a flat 4-bit depends on how large a context window you need. I haven’t fooled with Odysseus so I don’t know what it specifically needs.
1 Like
Hi! I’m a newbie myself with a setup probably similar to yours, just two 3090s. I’ll agree with @ZandoBob’s thoughts but you should be aware - especially for use with a harness such as Hermes or Odysseus - that the newer models have massively increased their context size that also uses memory. You really want to maximize context when using a harness so it doesn’t lose track of what’s it’s doing. Because of this, you’ll likely have some tradeoffs to explore.
Depending on the model runner you’re using (Ollama, llama.cpp, vLLM), the default configuration may either overflow the context space to system RAM making it much slower, automatically size the context to GPU memory, or simply failing to load the model. If you have problems, try specifying a low context size when loading and run nvtop to watch how high GPU memory goes.
I’m using the same Qwen 3.8 27B that @ZandoBob recommends and it’s great, but you may also want to experiment with a mixture of experts model like Qwen 3.6 35B A3B because those supposedly only use a fraction of memory at any given time. I’m not super familiar with the details there, so someone else well have to add some nuance to my statement.
All that being said, best wishes on your new setup! This has been the most exciting computing project of my life and I hope you have a similarly great experience. Reach out with questions, especially if you ever decide to move to multiple GPUs, as that’s it’s own animal.
Edit: corrected version of Qwen A3B model.
1 Like
If you mean an RTX 3090 Ti, you’re actually starting from a pretty good place. The 24GB of VRAM should be enough for a lot of local AI workloads, although the exact performance will depend heavily on the Odysseus model size, quantization, and context length. I would try the 3090 Ti first before buying anything. For self-hosted AI, VRAM is often the biggest constraint, so having 24GB gives you considerably more flexibility than most consumer GPUs. If you run into memory limits, reducing the quantization or context size can often help before upgrading hardware.
1 Like
Highly recommend starting with ramalama and it manages a lot of things for you to start
Qwen3.8 is only available as the dense, and it’s much improved over 3.6 (the 35B MoE, and the older 27B dense). I’ve found in general with small MoEs that they run very fast, but they make mistakes and have to redo work, leading to them being slower across a given session than a dense model. 3.8 is… I forget the term, it’s a mixed-attention architecture or something. Not all the heads need to talk to the KV cache, so its KV Cache size balloons much less than a traditional dense model. You can quantize KV cache to 8-bit to save a bit more RAM too, without a measurable (in my testing so far, at least) quality drop. But I haven’t tried that out to the full native 256K context. And quantizing KV cache will make it run a little slower, full BF16 gets a few more tokens/sec at the cost of using more VRAM.
2 Likes
Oh wow, you’re right. I could’ve sworn that 3.8 had a 35B A3B model, but I must’ve been seeing references to the 35B model for version 3.6. Sorry about the mistake!
That’s an interesting take on the MoE models. I’m curious if you’ve had any experiences with context limitations vs smaller quantizations. Someone told me in another thread that Q6 was a good sweet spot (paraphrasing by memory), but I’ve been running happily on a Q8 model with MTP and a slightly constrained context size.
It depends on the model, the quant, and what you’re doing. Some models go insane when crunched too small, but other versions of them don’t. There’s different ways to quantize models, if you have a smidge more space you can usually go for a mixed-bit quant, where weights are at 4, 6, 8, or full 16 bit depending on how much they influence the model’s output. There’s Quantization Aware Training too, but I’ve only seen Google give out models using that. The QAT versions of Gemma4 didn’t have any quality drop in my testing, but they used way less space (and thus RAM). I don’t use Gemma overall because their reasoning is pretty insane, and Qwen models perform more consistently (once you have them set up correctly so they don’t think-loop).
The testing I do is a bunch of python scripts I had Claude put together, and I usually have Claude or a large open-weight model on Ollama Cloud (GLM5.2 usually, I find 5.3 goes insane and starts speaking chinese after a bit, and I only speak english so it’s not terribly useful then) run it against local models running on LAN at work, or LMStudio/Ollama/MTPLX on my own machine. It’s based on local projects instead of generic public benches (I don’t trust those, see above where 5.3 supposedly scores super high, but in my experience goes off its rocker). I’ll see about cleaning them up and whacking them on github, you’d want to tweak them for whatever you actually do, but it’d get you there a lot faster vs having an LLM re-build it from scratch.
The TLDR is, at least in my experience, you want a consistent “plodding along” model to get work done. Models that are too eager/hasty tend to rush ahead, which results in a lot of time/token burn to fix all their mistakes. Tortoise beats the hare, I guess.
3 Likes