Handling shortcomings of small local AI models?

I have been playing with small local models from 8b on CPU to 27b on GPU. Gemma, Qwen, etc.

I set temperature = 0 to achieve some consistency on my tests, and I found repeating the same prompts on clean context windows can fail in predictable ways.

As expected I found more errors in the answers of smaller models of the same family but sometimes they fail in unexpected ways.

For example generating thousands of XML with a specific format but different contents using 3 different sizes of the same model, the smaller one made more errors, the middle one made less and the bigger none. However the errors made by the middle and the small were not the same. The errors were always the same for the same prompt/model, in this case missing or not closing properly a tag. About 15k prompts, a dozen errors for the small, a couple for the middle.

Generating small C++ snippets of less than 20 lines there were sometimes logic errors like missing a required keyword to achieve the goal on the 12b model, but not on the 27b. This was a conceptual issue and no matter what, even pointing the error, the 12b was unable to fix it.

How to handle those shortcomings without going bigger?

Iteration? asking for validation? On the same context? on a new one? Something else? Same model? Another model? Agreement of three aviation style? What about minority reports?

What strategies work for you?

For my 4-14b range of models, I typically bound the task, atomicize it through decomposition and then delegate it out through the harness with deliberation. This produces a tremendous amount of calls but the results are much better than less.

I do not converse or directly interact with small models, they are tools, task agents, and the testing ground for various project I’m assembling. For my dataset inspector, the same prompt is used with seperate files, and there is a difference in response output we judge before an inspector pass is started. Some reasoning models overthink and break the budget, triggering no response warnings. Sometime you need just the right model.

The intent pipeline in my harness uses three signals for classification as we have specialized intent and classify models, but they are drab, leaving out qualities and factors that actually matter(over 1000 prompts over half were too concise.) So I added a third pass by a ~9b+ model to get a grounding, three calls one classification, with parallel lanes in llama.cpp this all happens very fast.

CPU? That is a different world, bounding the models capability with tools skills and structure helped me the most. I’m still building the harness, but it’s deliberation stage we adopted in dev. Local models under my 16GBVRAM need verification passes. “Personal” solutions seem to be pretty common in the modern cheap code age.

You can also utlize various “free” cloud models to add a seat in the agent review pass. Depends on your level of scrutiny on where data flows.

1 Like