Stop Wasting Cloud Tokens - Local AI FTW

For me the only missing puzzle piece is then some sort of queuing for longer running tasks. If we get to a world where local AI can behave roughly like AI was a year ago, this will lead to long running tasks that will pile up in the organisation. I guess software teams will then use light “prompts” during the day to setup longer running tasks for the times where no utilization would occur anyway; e.g. the night time.

Or if the bubble pops, there will be still some demand for cloud AI, but without the huge data-center build outs, a market can emerge where depending on your location and time you will pay different amounts / token. There this would also help to keep costs down.

Hey Wendell,

Thanks for your time (& money) on this. I work for a big cloud company, and I hear the “we want to use local AI” models more and more frequently from real, large enterprises every day and I have been writing a presentation on this topic - to help steer organisations down this path if that’s what they really want to do - albeit my findings are a bit different than yours.

I am curious to give the new team green model a try, and I am eagerly awaiting qwen3.8-27B also…

I currently have a workstation/server, funded personally with 2 x R9700 AI Pro’s (64gb vram) which is about the limit of my willingness to spend (2 x rtx 6000’s is like the price of a brand new Tesla here…) - and unfortunately leaves me in the dreaded “model canyon”. I did try DeepSeek V4 Flash on this rig, with MoE RAM spillover, but at 3tok/sec, it wasn’t very fun.

I am fortunate to have access to lots of free tokens through work, and use it to experiment and benchmark repeatedly. Not just on things like polygot, but on real developer specs, features, and agentic workflows that mimic what a Software Engineer would be doing today.

My current concerns with Local AI are:

  • It is mostly a distraction for large enteprises. Yes there is a huge gap in $/token for qwen3.6-27B on my home server vs Opus 5 or Fable, but in most Enterprises, this isn’t where the true, systemic problems are and cost drain is
  • It’s a field where there is no tech talent. Most orgs DevOps & Security teams are stretched so thin - that adding “AI Operations Engineers” into the mix is a talent that (maybe outside L1Techs :slight_smile: ) doesn’t really exist at scale - and no Enterprise want’s to pin their entire local AI fleet on “Dave”.
  • Local AI at scale for internal orgs may just be more expensive to maintain than SaaS.

I think if the RAM apocalypse wasn’t with us, this would be a very different story. Would love to chat to you on some of this in depth if/when you have the time, and share some of my own findings and more detail of what I am hearing in customer land.

2 Likes

Yes, but only if

100% – but there’s two threads here. And that lack-of-labor is why I’m so insanely frustrated with the way we see many CIO/CTOs deploying AI “We can do more with less people!” Noooooooooo!!! There was never enough labor to begin with – to do things the way we want to the way things ought to be done. We can use this as a labor augmentation tool and we must not lose the people that know what remains to be done.

So I think based on discussions I can better articulate what’s crystalizing in my mind for enterprise (even small/medium 100-1000 staff organizations):

If you can get deepseek v4 flash for $1 per million (or $5 per mil) from a cloud provider you trust then yes, I agree with you, don’t bother with local AI (yet, let me bring it back there).

Use that to get your dataset. Or don’t and just use the frontier SaaS model for everything. BUT you MUST build the dataset. If you have the dataset THEN
you can customize/fine-tune the small model. THAT is the benefit.. that’s the juice that’s worth the squeeze.

Your argument holds up if you’re just deploying vanilla qwen or vanilla deepseek and “everything else” around that is a heavy lift for your organization.

However, if you build the dataset then spend some time experimenting with the dataset and customizing small models THEN, I think, you’ll end up with something that beats frontier models AND protect somewhat against competitors that get a Claude account and say “Hey Claude, using everything you’ve learned from your biz clients in $YOUR_INDUSTRY help me in my $COMPETITOR_COMPANY” – your biz gets to keep their process/IP moat the more you “obscure” from conglomerated cloud models how/what exactly you’re doing.

So to recap: Maximal Small Model Value comes from customization. The biggest-bang-for-buck from customization comes from gathering data on how the org uses the tools (fancy, non-local, etc) and customizing a “small” model.

Your customized “small” model that is “local” may be cloud hosted (private cloud, neo cloud) or on prem. Or, by then, maybe we have better local hardware cheaper. Local here might be 27b parameters of 270b or 1t by then.. who knows. This is when it becomes not a distraction for enterprises, I think, is the level of customization and specialization.

Tools to do this will emerge. You’re right everyone’s stretched too thin BUT whats exciting here is that this is at least two orders of magnitude less effort in the tooling for customization.

If you step back on a historical timeline a lot of the “no one has time to become an expert in ____” was true for everything in the history of compute. Networking. Databases. Directory services. Novell/Directory being too complicated and then LDAP and then LDAP being too complicated still is how we ended up with Active Directory kind of thing (lol not really but sorta kinda).

How complicated “the flywheel” as a tool for improving how your org can use the tool was a couple years ago was totally unreachable, I agree, but at this point it can almost (not quite, maybe another year away) be packaged into a product that sits in front of your cloud usage.

I’m pretty sure that is exactly what some firms in Asia are already doing and they’re so far ahead that’s why OpenAI and Anthropic said it was a distillation attack they didn’t detect.. not really? It was them doing this exact thing on a grand, grand scale. Maybe. I’m not sure.

Eventually with enough of this data “the flywheel” can lead to recursive self improvement of these algorithms and structures for your organization. Probably. But that will take a while…

5 Likes

what @wendell say is true on paper but not in practice, why? easy to build the “moat” takes time and skill the solutions are more on the prototype side that everything else and unless the person doing the moat is own part owner of the business there is little to no incentive to do build the “moat”

i invite anyone to do the math of how much they need to get paid to se this up and make sure that if they get fired can live 10 year of that money then see who is willing to pay that

Was really interested by this video and I am looking to get started trying to implement this, but I’m still stuck trying to understand the architecture and actual moving components in this solution. The deprecation on the whole data flywheel model (also the data flywheel was this offline self-evaluating system???), the upgrading of NeMo to v0.3, the fact that switchyard only handles the routing piece and not the other data collection and training pieces, and the sheer amount of things labeled “NeMo” is confusing.

the “enterprise” implementation is probably overkill tbh but start here:

This is a proxy that can sit in between your API(s) and model. Which you can use to log what users are doing.

The model router can aggregate so you can go to like

http://ip/v1/models and get a list of models.. where some of those models are cloud (or all of them if you dont have the bandwidth to manage small/local models).

If you really get into it, the flywheel can query multiple models at once and a judge model can judge if the output is any good to help.. but I wouldn’t go for the “full automatic” model selection out of the box as others have rightly pointed out your users might not understand why sometimes the model is really stupid and sometimes not.

1 Like

Day0 qwen3.8, and tensor split doesnt work under 2 x R9700’s :melting_face:

This is where I worry about the “tinkering” time, and resources that when a decision goes to a CFO/CIO - “the build v buy” - it will split.

The buyers will keep their Cloud provider spend as is, the builders will go down this path - but when you add everything up int the wash - is cost savings going to be alone the driver?

I think some of the other elements around data sourverignty , trade secrets, etc, and companies that want to build a genuine engineering moat - and something that can withstand a provider outage, a frontier model being yanked or tripled in price $/tok - are going to be the bigger triggers that make this stick.

Sure it does :slight_smile: At least, it’s working on mine with ROCm…although interestingly, it’s about the same speed as running layer split under Vulkan. I’m waiting for the 35B A3B, though (ModelScope accidentally leaked that it’s coming on their GitHub repo).

Well, for what it’s worth, I think it’s unlikely that a company hosting their own AI services will be trying to implement new models on day 0.

Thanks, sent you a DM to not take this further OT.

On my system (2x R9700) ROCm tensor split-mode makes pp 50% slower and tg 50% faster, at least that is the values I’m seeing with llama-bench.

Well, once MTP is involved, the numbers on both sides are roughly comparable between ROCm tensor split and Vulkan layer split. I wouldn’t call either particularly usable for my uses (~1100t/s prefill, 65t/s decode), but far from the worst I’ve seen.

In particular, prefill doesn’t take the catastrophic nosedive that it did with Glimmer and DFlash.

I am probably missing something, but switchyard natively doesn’t actually seem to log the data around the prompt request and output, the transaction, or any other details that could be used to train a larger model. It only seems to log regular token counts, health, and other normal metric data, and the pieces that did that were only part of the “deprecated” flywheel model.

Still trying to figure out what the proposed architecture is with switchyard in the loop…

ah, was building on our haproxy vid there :smiley:

If you want full session traces you will want a model router / llm gateway.

They let you hook up every model provider once centrally, assign virtual model aliases like “vision” that routes to whatever real model on the backend (or different providers for failover)

Then you register your own applications and issue api keys (or use oauth :D) so you can centrally track usage.

Finally just let the model router live debug the sessions so you can see whats going on, or go back and reference problematic sessions later.

2 Likes

Well @wendell i just bit the bullet and purchased 2 x 192gb kits of Team Group Master RDIMM’s from B&H - now i have 384gb of 8-channel DDR6400 RDIMM’s heading my way to go in my ASUS Sage SE board with a Xeon 658X.

Hoping to have it all together in late October (work travel over the next month is brutal) and see how I go with 128GB of VRAM (4x R9700’s) and then spillover to the 384GB of 8-chan DDR5.

I’m hoping to have this running a mid-tier model, and then I have a couple of smaller systems with a single R9700 and another with 2 B60’s to see how I go.

I have about 4 real world codebases I can run some exercises (not benchmarks) on to see how good this system is. I’ll even try and throw in Spotify’s just announced ( GitHub - spotify/portal-ai-plugins · GitHub ) Portal plugins - and I plan to use Claude Code (wrapped by Switchyard to send my Claude requests via Qwen/Deepseek).

Curious to know how this turns out. I also have 4x 9700s but not having a lot of luck with model configs exceeding that 128g vram.