Looking for advice on a low-end AI coding setup that keeps timing out (Radeon VII + RX 5500 XT)

nfriedly · August 24, 2026, 2:57am

Hi, I have a home file server that is mostly just leftover parts + extra hard drives. I’m trying to get to a point where I can give it a task for one of my open source projects and check back in on it a day or so later to see what it’s accomplished. (Or maybe have it open a pull request when it’s done.) The problem is that most model + harness combos I’ve tried seem to be timing out or hitting other errors.

This is the full server specs - it’s mostly leftover parts + extra hard drives, but it does have 24gb vram, which ought to be able to run most of these models:

# ollama list

NAME                                                        ID              SIZE      MODIFIED
hf.co/unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q8_K_XL               b002f560a9a2    39 GB     6 hours ago
hf.co/JonathanColetti/Qwen3.8-27B-Uncensored-GGUF:IQ4_XS    49d2e81026ce    15 GB     2 days ago
gemma4:latest                                               c6eb396dbd59    9.6 GB    4 days ago
hf.co/JonathanColetti/Qwen3.8-27B-Uncensored-GGUF:Q6_K      79b2ee0918f7    22 GB     8 days ago
muse-glimmer:30b                                            de878ce33ad8    18 GB     8 days ago
hf.co/unsloth/Qwen3.6-27B-GGUF:Q6_K                         9bcf9b6496ac    23 GB     8 days ago
hf.co/unsloth/Qwen3.6-27B-GGUF:Q3_K_M                       b0fa9f8afbca    14 GB     8 days ago
hf.co/unsloth/Qwen3.6-27B-GGUF:IQ4_XS                       57c0a9428046    16 GB     8 days ago
hf.co/unsloth/Qwen3.6-27B-GGUF:UD-Q4_K_XL                   d4ddf6641ed2    18 GB     10 days ago

These all work fine for basic things like “tell me a joke”, but when I give it a larger task like reviewing the changes in this PR, most of the setups I’ve tried have either hit an error or just stopped before finishing.

The one success I’ve had was pi + muse glimmer: over the course of 2 sessions, it was able to first write a plan to a markdown file, and then follow through with the plan and give me a list of findings. However, in the third session, where I asked it to fix the problems it found, it too hit errors, first spinning in a loop of trying to read past the end of a file 3 times before giving up and stopping, and then finally the dreaded “Response was truncated before completion.” Once that happens in a session, it will keep happening, with the model only able to get out a few sentences, or sometimes only a few words before getting cut off.

I tried several more models in pi with the plan that muse-glimmer created, but none of them could even finish the review. Mostly they work for a bit and then hit the same “Response was truncated before completion." error. A closed github issue suggested bumping up the max_completion_tokens, but I set it to 128k and it was still hitting the error before getting anywhere near that many tokens.

I’ve also tried Open WebUI + Open Terminal, but gave up when I realized that it stops working as soon as the connection breaks - I want to be able to let my laptop go to sleep and have the server keep working.

Additionally I tried Open Claw, but any time I give it a larger task, it seems to just stop after a while with no apparent error. I’m guessing it’s the same root cause as Pi’s issues.

Oh, and I tried Open Code, but it mostly just blows up with too much context before even starting or, when I make a modelfile with a larger context, it spends most of it’s time compacting and never gets around to actually processing my prompt. (I watched it go through 3 compactions before getting anywhere, and then gave up on it.)

I tried watching ollama’s logs, but nothing stands out at me. It seems like it’s working normally and then just stops, but maybe I just don’t know what I’m looking for.

The server is running unraid, with everything else in docker images. The models are run in ollama in vulkan mode (rocm used to work with the radeon vii, but they dropped support somewhere along the line, and I don’t think the 5500 xt was ever supported.)

I know this is all old hardware, and the 5500 XT is dragging the Radeon VII down to a quarter of the speed it could run at on it’s own (I’m getting 3~5t/s with both cards), but I feel like it ought to be able to keep going slowly, and I’m just not sure where it’s failing.

Do you all have any advice for me?

lambda · August 25, 2026, 3:19pm

Hello,
Have you tried using just llama.cpp for the inference engine? Without any logs or related stuff I’m not sure if it’s an inference problem or an issue with your coding harness (or whatever it’s called).

I’ve had good luck with just llama.cpp and using ZooCode plugin for VS Code.

As for the speed, use a Q4 quant of Qwen 3.6 35B A3B. The dense models are too slow on your current setup.

s1ugh34d · August 25, 2026, 3:27pm

So for full on conversation guided interaction, these sub 16GB models are tough. I find the best use from single task/subtask one-shots, with eval passes. I also have been developing a harness to utilize these smaller context windows in a collective. All on my RVII

Right now I have a SKILL.md with prompt guiding the llama-swap driven qwen3.8-9b-distill model to eval files, one by one. The script is just a simple python wrapper with resuming and a simple json index to log to.

Using the gguf models for one to one coding tasks seems inefficient for me at least. Context is the killer feature, longer context equate to worsening performance with these heavy quants. I get best performance from the first and second response to any context, thus plan and work around that limitation.

nfriedly · August 25, 2026, 6:32pm

Thanks for the advice. I also wanted to follow up with a few things I’ve figured out:

I bumped up all of my timeouts in pi and openclaw. I think part of the problem I was hitting was with these large models on my weak hardware was that just loading the model and processing the system prompt could take long enough that the harness gave up and closed the connection.

So, I think that explains some of the times where it just sat there and never responded.
When pi claims to automatically manage context, what that means is that it compacts when the context gets close to the limit. But unless you explicitly configure it, pi just assumes that the limit is whatever the model was trained on, so 128-262k for the models I was testing on. Ollama, on the other hand defaults to 2-4k and apparently has no way of conveying what the actual limit is to pi (?)

I had bumped the ollama default to 16k with the env var OLLAMA_CONTEXT_LENGTH=16384, and bumped it up a little higher for specific models by creating a modelfile.

So, at least some of the “Response was truncated before completion.” errors were because it was hitting the limit of the context window.
I was choosing models that fit into my vram, with a little room left over (this seems to be what ollama does by default if you don’t specify a specific quaint). But I didn’t realize just how memory intensive context was. It turns out that most of the models I was using would need more than all of my VRAM for their full context.

So I switched to a couple of MoE models that can offload a lot of the experts to system RAM (the aforementioned Qwen 3.6 35B-A3B and I’m also trying Gemma 4 26B-A4B, both in the unsloth UD-Q8_K_XL variations).

I also changed OLLAMA_KV_CACHE_TYPE from f16 to q8_0 to cut the memory size in half for a given amount of context, and then had Gemini calculate how much context I could use each of those (~56k for Qwen 3.6, or the full 256k for Gemma 4 - apparently it’s more efficient with cache).

With those changes, things seem to be going a little better. I might yet try running llama.cpp directly, but I’m happy to at least have this thing working now.

Hopefully this will be helpful to someone else; I might post another followup eventually.