Most discussion on AI models seems to be about how to fit them in memory and it sounds like most AI performance is bottle-necked purely on bandwidth.
This is a bit of a “shower thought” but I was wondering if anyone had thought about or seen research into making smaller models that do more number crunching for the parameter count. Basically thinking harder with a smaller brain.
I could see one possibility be like how diffusion models worked for image generation. You could have a smaller model designed to run multiple times on the same input alongside the previous output until it generates a better final output. It’s like how a person might re-read what they’ve written and edit it as they go.
“smaller” and “bigger” are becoming increasingly ambiguous terms since you’ll need to compare MoE spare and dense models. They mostly all have different architectures, parameter activation strategies etc. And doubly so on the inference side where you have to consider expert parallelism, data parallelism, kv offloading etc. I don’t understand the depths nearly as much as others here, but “bigger” and “smaller” are difficult terms.
Where I’m personally interested atm is at the harness layer “above” training and inference for multi-agent organization. It also sounds like this is more aligned with your Q.
You might be interested in Anthropic’s advisor patterns, which pairs a fast, lower-cost model like Sonnet or Haiku with a powerful reasoning model like Opus to provide guidance mid-task.
OpenAI’s flavor is multi-agent which they claim made a pretty big difference when solving navier stokes. It’s expensive though to push this beyond 8 or 16 models or ordinary tasks. And of course you need the inference infra to support it.
Edit:
Since you mentioned diffusion, you should also checkout Stefano Ermon’s research and Inception Labs. Their Mercury model is taking diffusion learnings from image gen an applying them to language gen!
“Large language model (LLM) agents have demonstrated strong capabilities in complex interactive decision-making tasks. However, existing LLM agents typically rely on increasingly long interaction histories, resulting in high computational cost and limited scalability. In this paper, we propose STEP-HRL, a hierarchical reinforcement learning (HRL) framework that enables step-level learning by conditioning only on single-step transitions rather than full interaction histories. STEP-HRL structures tasks hierarchically, using completed subtasks to represent global progress of overall task. By introducing a local progress module, it also iteratively and selectively summarizes interaction history within each subtask to produce a compact summary of local progress. Together, these components yield augmented step-level transitions for both high-level and low-level policies. Experimental results on ScienceWorld and ALFWorld benchmarks consistently demonstrate that STEP-HRL substantially outperforms baselines in terms of performance and generalization while reducing token usage.”
That is basically what the whole concept of reasoning was about. You can’t really make it think “harder” but you can make it think longer - reasoning tokens.
Better prompts & inputs help a lot. If you have a large context problem, get the smaller model to read through, summarize, note key points, output a smaller document, then feed that back into the same model so you have a better starting point. Handy when you have been working on something for a while & are approaching context limit too, get all the previous steps in a nice little list so you aren’t going over the same ground.
In a very practical sense that’s not going to work right now because smaller models literally have less “fire power” to go on extensive reasoning sessions. If you ever let a small model reason on a task it will most likely start to loop because it’s not capable of keeping track of everything it needs to close it’s reasoning. So it ends up burning the thinking context and never come out with an answer.
What drbsg posted from Reddit is really interesting because it suggests that models, like humans, should reason in a space that’s not constraint by words. That might be something close to “thinking harder” but still stumbles on the premise that if there isn’t a lot to work with there won’t be a lot at the end to get out of it.
If you ask a kid to rewrite their homework describing their house every version of it will most likely by a variation of the same description without much improvement between rewrites.
If you ask a well educated adult to do the same they most likely will keep improving the homework adding details and intricate descriptions, while having an headstart on the kid.