Your company is probably spending too much on AI: Switchyard, local models, routing, and the data flywheel
So I have been working on an enterprise consulting project where the plan is “adopt AI” but my go-to IT toolbox is monitoring and management of “how” and “what” the users are doing (before “AI” even enters the picture).
The changes/improvements to SwitchYard and the release of Nemoton Lightning are two missing puzzle pieces that, imho, will solve the last missing bits for managing “responsible” and non-IT AI use cases in an organization. Or, at least, give you ammo and documentation to push back against misunderstanding how to apply the tools at the c-suite level.
Look… this isn’t complicated. I want away from the AI hype tulip bulb mania. Management has wildly wildly different ideas about how AI might be used in an organization than the reality of how it will actually be used. I want to capture the reality data to push back on the hype/hallucinations/ai psychosis/whatever you want to call it.
Finally, we have effective tools to take action in a meaningful way with only a tiny fraction of the effort a year ago.
That data is a shield to protect my sanity for two reasons.. re-alignment with reality and to have the foundations of a process where we actually get measurable artifacts to know the efficacy of the tools we’re deployng.
The short version:
Commodity intelligence → routing → observable execution → evaluation → data exhaust → specialization → cheaper local intelligence → fewer frontier calls → repeat.
We did a video a while back on using HAProxy for HA but… expand that to “model router” and then telemetry on what users are trying to do.
I think this is where enterprise AI is heading, and NVIDIA’s Switchyard update plus Nemotron 3.5 Lightning makes the whole thing much more accessible.
It’s not another model release – it’s the last piece of software you need for this whole ecosystem that’s required to stop sending every stupid little task to a frontier model.
It is finally becoming paint-by-numbers enough for normal IT people to deploy, measure, customize and improve.
That is going to have two consequences:
Economic: frontier inference becomes an exception rather than the default.
Organizational: the company stops renting all of its intelligence and starts accumulating some of its own.
What’s Running In The Video
My demonstrator is an HP Z8 Fury with two RTX PRO 6000 Blackwell GPUs.
On it I have:
Switchyard as the model router.
A small local model.
A much larger local model, currently DeepSeek-V4-Flash-0731.
RAG/vector infrastructure.
Evaluation and logging.
(The frontier API is still available. It just isn’t the first thing every request gets thrown at.)
Switchyard sits between the application and all of those resources. The current public implementation can translate OpenAI Chat, OpenAI Responses and Anthropic Messages formats, then route requests to things like vLLM, NIM, Ollama and OpenAI-compatible endpoints. It supports classifier routing, stage routing, escalation and custom routing policies. Claude Code and Codex can both be launched through it. (Switchyard Magic!)
So the application does not need to particularly care whether the answer came from a 3B-active local model, a big local model, a server elsewhere in the building, or Opus.
That is the point.
This idea did not appear suddenly on a crisp Tuesday afternoon
This wasn’t really in any of NVIDIA’s press release stuff, but in my mind (and I could be a crazy person) there is a pretty obvious research trail here:
In 2023, FrugalGPT showed that cascades of models could match the best individual model with dramatically lower cost, or improve accuracy at a fixed cost. (FrugalGPT)
In 2024, RouteLLM formalized learning when to send a request to a strong or weak model. (RouteLLM)
NVIDIA followed with its own cost-efficient LLM Router Blueprint. (NVIDIA LLM Router Blueprint)
Then NVIDIA Research published a paper with the wonderfully unsubtle title Small Language Models are the Future of Agentic AI . Their recommendation is almost exactly what I am arguing here: use small models for repetitive narrow agent work, specialize them, and build heterogeneous systems that reserve large general-purpose models for the difficult stuff. (paper)
Then came ToolOrchestra and Nemotron-Orchestrator-8B .
An 8B orchestrator was trained to decide when to use other models and tools. In NVIDIA’s evaluation, the resulting system scored 37.1% on Humanity’s Last Exam versus 35.1% for GPT-5, at about 30% of the monetary cost and 2.5x the speed. (ToolOrchestra paper, NVIDIA project page)
That’s not “an 8B model is smarter than GPT-5” but the system was better at the task.
The orchestrator could assemble the resources actually required instead of asking one enormous model to cosplay as omniscience.
(This is also a bit of the philosophy in our own Turnstone project with MCP, Skills, tool calling, etc)
Nemotron 3.5 Lightning
NVIDIA’s prerelease material for Tuesday describes Nemotron 3.5 Lightning as a 30B MoE with 3B active parameters , aimed at high-volume agent execution and low latency. Long-running agentic tasks.
It includes multi-token prediction and speculative decoding support, plus DFlash and DSpark inference optimization. NVIDIA says post-training uses Multi-Teacher On-Policy Distillation, or MOPD, and that the model was deliberately trained across multiple open-source agent harnesses.
Sure okay, long-running agent tasks BUT it’s got another aspect that everyone’s missing earnestly trying to “do right by their organizations” with AI – customization with low effort and good documentation around that.
NVIDIA says Lightning will be customizable with LoRA or full SFT using NeMo AutoModel and Megatron Bridge, plus reinforcement learning through NeMo RL and NeMo Gym. Weights, training data and recipes are being released under OpenMDW-1.1.
That licensing direction is also public. The Linux Foundation announced in May that NVIDIA would adopt OpenMDW-1.1 for future Nemotron releases. (OpenMDW-1.1 announcement)
Here’s their blog post about it
Small models are no longer cheap generic models. They’re becoming things you are expected to modify and should modify in this use-case ecosystem.
Qwen 3.6 is another version of the same story
Everyone loves Qwen3.6 and everyone runs it through a battery of tests and asking whether the 27B or 35B-A3B can beat model X on benchmark Y.
Fine.
Qwen3.6-27B is a 27B open-weight model. Qwen3.6-35B-A3B is a 35B MoE with only about 3B active parameters. The Qwen model cards document 262K native context and support through the usual open inference ecosystem. (27B, 35B-A3B)
I’m not talking about any of that – I’m telling you to go look at the Qwen training ecosystem… This is, and has been all along, the disruptor.
Again, wearing my computer janitor hat here and I have been trying to help with this on some enterprise projects.
So for NVIDIA and Nemotron.. look at where they are now and where they’re going.. Full fine-tuning. LoRA. QLoRA. SFT. GRPO. DPO. PPO. Megatron. Distributed training. Custom dataset examples. Runnable commands.
The MS-SWIFT docs literally walk through dataset formatting and training commands. (Qwen MS-SWIFT docs)
We, the unwashed IT masses, can increasingly do this ourselves.
For me this has changed the question from:
“Which company has the smartest generic model we can just buy in the cloud?”
to:
“Which model can I cheaply turn into the best model at our boring repetitive thing?”
That’s what I’m talking about in those diagrams. Specialization does not have to beat omniscience generally. It only needs to beat it on that crispy Tuesday afternoon, just for accounts receivable.
LoRA is where this becomes operationally reachable to “ordinary” org IT
LoRA is basically a small patch for a big model.
Technically, it freezes the base model and trains low-rank matrices instead of updating every parameter. The original LoRA paper demonstrated enormous reductions in the number of trainable parameters. (LoRA paper)
NVIDIA NIM can keep a base model resident while loading different LoRA adapters. Current NIM supports runtime loading and unloading, and multiple adapters can be served at the same time subject to GPU memory. (current NIM LoRA docs, NVIDIA multi-LoRA article)
So your job is to think in these terms: A base model can become:
Accounting model.
Software engineering model.
Support model.
Internal-product-nobody-documented-since-2017 model.
Gary’s dark institutional knowledge model. (Gary might even get a vacation.)
So that’s why I’m saying whatever model you’re running is becoming less interesting than the system around it that captures and logs how users are trying to solve their own problems with the tools. Notice what they’re trying to do, and see if you can help do that (or steer them toward stuff AI can actually do.. like help your customers).
This mindset is an important starting point for getting away from “AI builds another kafka-trap for our customers, driving them away eventually!” and “AI actually helped us be better at what we do.”
RAG and fine-tuning solve different problems
People still tend to lump these together as “teaching the AI about our company.”
RAG is for what the model needs to know right now.
Changing documentation. Current inventory. Customer state. Source code. Contracts. Policies. Anything that needs provenance or a hard reference.
The original RAG paper is still worth reading because updating knowledge and providing provenance were explicitly part of the problem it was designed to address. (RAG paper)
Fine-tuning is primarily for what the model needs to learn how to do.
Terminology. Conventions. Repeated workflows. Tool use. Classification. Output structure. Coding practices. Organizational behavior.
Stable domain knowledge can absolutely end up in weights, but if a fact needs a citation, retrieve it.
If a behavior happens ten million times, train it.
So what’s changing here is that the retrainability via fine-tuning is increasingly more and more reachable without deep domain knowledge and a highly specialized toolbox. Scripts and examples from the websites!
If your vendor manifest changes every Tuesday, retraining the model Wednesday is technically a solution in roughly the same sense that rebuilding the library is a solution to swapping in a second edition of a book.
The secondary dataset may be the most valuable output
So the whole “price of admission” on this type of project is entirely worth it just to have telemetry on how people are using AI (trying to use AI) and the results. It is turning into my best tool to keep executives accountable and to tamp down “ai psychosis” because it is a solid accountability tool, but in an AI context.
Track: What was requested? Which model was selected? What context was retrieved? Which tools were called? Did evaluation pass? Did the small model fail? Was the request escalated? Did the expensive model succeed?
Did a human correct it? What did the whole thing cost?
That isn’t merely telemetry.
That’s a training dataset about how your company actually uses AI.
NVIDIA’s older Data Flywheel Blueprint explicitly worked this way. Production agent traffic became evaluation and fine-tuning data, candidate smaller models were tested, customized and promoted back into production. NVIDIA demonstrated a fine-tuned Llama-3.2-1B retaining 98% of the tool-calling accuracy of the original 70B model. (Data Flywheel article, GitHub)
The old Blueprint repository itself is now deprecated for new production deployments. The playbook is still useful for understanding the points I’m trying to drive home and what I’ve been experiencing “in the enterprise” the last few months.
NVIDIA has since published a very practical synthetic-data/distillation pipeline showing how to seed domain data, generate Q&A examples, evaluate and filter them, then produce training data suitable for fine-tuning or distillation. (synthetic data pipeline)
And this is not just NVIDIA’s theory.
A July 2026 paper called Agentic Routing: The Harness-Native Data Flywheel makes almost exactly this argument. Each routing decision naturally creates a record containing the request, harness state, model choice, execution trace, outcome and cost. Those records then train better routers and models. (paper)
That’s the flywheel I’m talking about here, and what I tried to show in the diagram:
Route → observe → evaluate → identify failures → specialize → deploy → route again.
Maybe I start with 80% of requests reaching a frontier model.
If my workloads repeat and I actually do the engineering, that percentage should fall.
The expensive model becomes the teacher and exception handler instead of the employee answering every ticket forever.
Long-running agents make routing more important, not less
This is also where I disagree with the current trend toward “give one enormous model a giant context window and let it disappear for several hours.” a la “long running agentic tasks”
That can be useful, sure, okay. But I don’t want Ted from accounting doing that to generate his quarterly status PowerPoint. Really.
Recent routing research increasingly treats agentic routing as a step-level problem because the right model may change during a trajectory. (I had been focused on sub-tasks being routed wholistically but that’s not not really needed the way I had been thinking about it).
TwinRouterBench specifically evaluates routing inside coding, RAG and other long-running agent workflows. (TwinRouterBench)
Budget-Aware Agentic Routing treats cheap-versus-expensive model selection as a sequential decision because early mistakes can compound. (paper)
Agent-as-a-Router goes further and learns from execution feedback over time. (paper)
There is also useful contrary evidence. LLMRouterBench , with more than 400,000 instances across 21 datasets and 33 models, confirms that models have complementary strengths but finds that fancy routers often fail to reliably beat simple baselines. There remains a large gap to an oracle router. (LLMRouterBench)
I still think that’s good, though, for this enterprise scenario I’m describing because routing isn’t magic.
If you have visibility into what’s happening with the system evaluation and escalation and those smaller local models the router has access to get better. These modern model routers can also simultaneously route requests and then evaluate responses that come back in parallel before picking and returning one.
Switchyard’s escalation router starts with the cheaper model and watches whether the run is actually going badly before latching the session onto the stronger model. That is much more interesting to me than trying to perfectly predict task difficulty from the first prompt. (Switchyard escalation routing)
Observability beats blind autonomy
I have also been increasingly unhappy with AI coding tools that surface less of what is happening and encourage you to just let the machine disappear into the task; I mentioned Codex CLI in the video.
To be precise, I do not need a model’s private internal monologue.
Anthropic’s own research shows chain-of-thought is not necessarily a faithful explanation of why a model reached an answer. (Anthropic)
What I want is the engineering trace: What information did it retrieve? What tool did it call? What changed? What failed? What model performed the step?
Why did we escalate? What did evaluation say? Where did the human intervene? (oh, this sounds suspiciously like a Dune quote! haha)
OpenAI itself argues that reasoning monitoring can provide useful safety signals, while its Codex material is simultaneously pushing toward very long-running delegated work. OpenAI recently showed a 25-hour Codex task and described a future with humans steering at milestones rather than line by line. Importantly, their own example retained a live audit log. (Codex long-horizon example, OpenAI reasoning monitoring)
Qwen Code recently went the other direction in one useful respect and added real-time steering so you can correct an agent while it is working. (Qwen Code real-time steering)
More capability should demand better supervision, not less.
Yes, distillation is part of the economics
I have a hypothesis that one reason commercial tools may be reluctant to expose increasingly useful reasoning/task traces is that those artifacts are extremely useful for training their cheaper replacements. You want to plug it into the system I’ve outlined above? It works great for improving the local models I’ve found!
I cannot prove that’s why any particular UI changed, though. Just a guess.
The economic incentive unquestionably exists though.
OpenAI itself offers Model Distillation , explicitly using outputs from expensive models to fine-tune cheaper models that approach their performance on specific tasks. (OpenAI Model Distillation)
NVIDIA’s Nemotron-Cascade 2 uses multi-domain on-policy distillation from strong teacher models. It is itself a 30B MoE with 3B active parameters. (Nemotron-Cascade 2)
The economics on my desk
My large local model right now is DeepSeek-V4-Flash-0731 . It is available as open weights and runs through the normal local serving ecosystem. (model)
On the workloads I’ve tested, I would put it broadly in the Opus 4.6 to 4.8 neighborhood. That is my workload result, not a claim that it universally beats Opus.
My measured marginal electricity cost (once the workstation is already paid for) is around $0.015 per million tokens .
My practical frontier-model cost basis for the workloads I am comparing is roughly $12 to $25 per million tokens .
That is roughly an 800x to 1,667x difference in marginal token cost .
For another reference point, Anthropic’s list pricing for Opus 4.8 was $5/M input and $25/M output. Current Opus 5 remains $5/M input and $25/M output. (Opus 4.8, Opus 5)
This is not TCO .
The HP workstation did not grow naturally in the server room after that particularly crisp Tuesday morning creating just the right soil moisture conditions…
Once the infrastructure exists and is kept busy, the marginal cost of local intelligence gets absurdly low and that is why blindly sending ordinary corporate work to the most expensive model available increasingly looks insane.
I’m not even really arguing the cloud goes away. I’m not even arguing AI capex slows down because probably Robotics, physical AI, science and entirely new workloads will happily consume every GPU humanity can fabricate.
Focus on something narrower, fellow computer janitors:
Most ordinary corporate tokens do not require frontier intelligence.
And increasingly we have the research, open models and software tooling to prove it.
The days of paying Opus to remind Gary which order the arguments go in should be long over.
Papers, software and further rabbit holes
- Switchyard: GitHub, routing overview, escalation routing, skill distillation. Note that the public repository has been labeled experimental/pre-alpha during development.
- NVIDIA prerelease material: Nemotron 3.5 Lightning and Switchyard launch/update article. Add public NVIDIA URL here when the launch embargo lifts.
- NVIDIA LLM Router Blueprint: Deploying the NVIDIA AI Blueprint for Cost-Efficient LLM Routing.
- Small Language Models are the Future of Agentic AI: arXiv 2506.02153, NVIDIA Research.
- ToolOrchestra: Elevating Intelligence via Efficient Model and Tool Orchestration: arXiv 2511.21689, NVIDIA Research, GitHub.
- FrugalGPT: arXiv 2305.05176. Early demonstration of model cascades as an accuracy/cost optimization.
- RouteLLM: arXiv 2406.18665. Learned strong/weak model routing.
- RouterBench: arXiv 2403.12031. Earlier benchmark for multi-LLM routing.
- LLMRouterBench: arXiv 2601.07206. More than 400K instances, 21 datasets and 33 models. Good reality check on router performance.
- The Routing Plateau: arXiv 2606.07587. Useful caution that many router designs converge to similar performance well below an oracle.
- Budget-Aware Agentic Routing via Boundary-Guided Training: arXiv 2602.21227. Step-by-step cheap/expensive routing for long-horizon agents.
- TwinRouterBench: arXiv 2605.18859. Agentic step-level routing with end-to-end task success and actual spend.
- Agentic Routing: The Harness-Native Data Flywheel: arXiv 2607.11399. Probably the paper closest to the central argument in this post. Routing traces themselves become the training flywheel.
- Agent-as-a-Router: arXiv 2606.22902. Uses context, action and execution feedback to improve routing over time.
- WISERouter: arXiv 2607.23765. Workload-level budget constrained routing with online learning.
- Nemotron-Cascade 2: arXiv 2603.19220, NVIDIA Research. Multi-domain on-policy distillation in a 30B/3B-active MoE.
- Nemotron-Cascade: arXiv 2512.13607. NVIDIA reports its 14B post-trained model outperforming its SFT teacher DeepSeek-R1-0528 on several LiveCodeBench variants.
- Data Flywheel Blueprint: NVIDIA article, GitHub. The old implementation is deprecated, but the architecture is worth studying.
- NVIDIA synthetic data/distillation pipeline: How to Build License-Compliant Synthetic Data Pipelines for AI Model Distillation. Particularly relevant for turning small amounts of domain knowledge into evaluated training data.
- LoRA: Low-Rank Adaptation of Large Language Models. The paper behind the adapter discussion.
- NVIDIA NIM dynamic LoRA: current docs, Swarm of LoRA Adapters article.
- RAG: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Still the useful conceptual starting point for retrieval, updateable knowledge and provenance.
- Qwen3.6-27B: model card.
- Qwen3.6-35B-A3B: model card. 35B total, approximately 3B active.
- Qwen training documentation: MS-SWIFT, Unsloth. This is worth browsing even if you have no immediate plans to fine-tune anything.
- Qwen Code real-time steering: July 23, 2026 update. Useful counterexample to the idea that increased autonomy requires decreased human steering.
- DeepSeek-V4-Flash-0731: official model card.
- OpenAI Model Distillation: official announcement. Useful evidence that frontier-to-small-model distillation is an ordinary optimization strategy, not an open-model conspiracy theory.
- OpenAI long-horizon Codex: 25-hour task writeup. Interestingly, OpenAI’s own example includes explicit milestones, verification and an audit log.
- OpenAI chain-of-thought monitorability: Reasoning models struggle to control their chains of thought.
- Anthropic chain-of-thought faithfulness: Reasoning models don’t always say what they think. Important qualification for anyone treating displayed reasoning as an audit log.
- OpenMDW-1.1: Linux Foundation announcement.
