Precision-Maxxed Local AI: Viable to run full models from flash?

High-Performance Computing Machine Learning, LLMs, & AI
1

I am developing an idea for a security focused local AI project.

I would like to set up an AI that is maxxed for precise analysis and bug/flaw detection. I plan to give it a set of code and ask it to find network and cryptographic security design flaws and vulnerabilities. Per my research, looks like the best way to do this is run the best models with full parameters on the best GPUs. (So do something like rent 8xB300 and run the full parameter GPT-5.2, Kimi K2.5, etc. with full context or whatever [still learning] and compare results.)

The catch is I would also like to maxx leak security. I understand that AI models can be trained to exfiltrate data, etc. and that detecting if a model would do something like that beforehand is nigh impossible in 2026. Another catch is that despite having consiterable resources (see rig below), I do not have a GB200 NVL72 system lying around so I can wipe the hard drives after the analysis is done.

So here’s my idea: Can I run a full model from flash storage while keeping code analysis precision-maxxed?

I do not care if it takes all day to perform an analysis. I just need it to find flaws at an elite (hopefully PhD-like) level.

If we lose code analysis precision on flash for whatever reason, would running a model on, say, 1TB system memory work for precisionmaxxing?

I am continuing to learn about all this so if any of the above ideas are wrong (or imprecise ;- ) please correct me.

Here’s the rig:

rig0
rig01920×3413 1.48 MB

rig1
rig11920×3413 1.46 MB

Spexxs:

  • ASRock TRX50 WS
  • 7975wx Threadripper Pro 32-Core
  • 256GB 5600MHz CL32 DDR5 System Memory
  • 2x RTX PRO 6000 - 192GB VRAM
  • 1x Solidigm D7-PS1010 - 7.68 TB MCIO —> U.2
  • NZXT C1500 Platinum
  • Antec Flux Pro

For this system, the idea would be to run full (~1T parameter or whatever) models from the D7 if flash is viable for the precisionmaxxed use case.

Thanks for the help everyone and thanks for everything that you do and stand for Level1.

2

Models are just weights, long lists of numbers. So the weights cant do anything on their own.

The software that execute the weights are inference engines. And if you poke around some of my recent writeup’s on those and their behavior, they are made up of often hundreds of discreet pieces of software. All software contain vulnerabilities, and in the case of the VLLM pile i dug through, multiple instances of them tried to phone home at runtime.

So, what is a practical recipe for security?

Separate your GPU, storage, and compute onto different hardware.

Isolate the GPU system so that the only 2 things it can talk to are your NFS mount for weights/datasets, and your local container repo. It doesnt need any connectivity to the internet or other systems.

Configure your NFS storage target and container repo for single purpose work, the only thing that should write to them is your compute/workstation and the reader is the GPU rig.

Dont run untrusted code on your workstation. It can talk to the internet, download weights and datasets from HF and write them directly to your NFS target. It can snag your favorite container images and jam them into the container repo.

You probably want separate cheaper compute to run non-gpu workloads, hopefully likewise restricted, for things you intend to connect to the GPU system, like your model router, openwebui, etc. grab their containers from the internet via the workstation, publish to the repo, mount filesystems to nfs, etc.

The result

No untrusted trash that gets loaded on your gpu box can ever go anywhere and exfiltrate your data.

No untrusted trash you download needs to run on your workstation.

Storage is a single purpose thing, never mixing compute and storage roles simplifies security dramatically.

If you run a harness anywhere, like claude code/codex/opencode/etc. make sure its in an unprivileged VM and its only doing work in a restricted environment like your gpu or non-gpu compute nodes, never your workstation.

Supply chain attacks are profoundly common right now, and any credentials on either your code harness vm or untrusted compute hosts should expect to be harvested lol

1 Like
So, like, how do I measure my local LLM environment and expand it?
3

Thanks for the reply, security ideas, and your recent posts offering in-depth analysis. I am in the process of digesting them. I’m sure I’ve seen documentation talking about a local container repo but this is the first I’ve thought about it as part of security infrastructure so especially thanks for that idea.

Am I accurate in gathering that my target of effective code analysis precision (AI quality I guess I’m talking about?) relies on many things especially the software stack implementation as you talk about in the Why your local LLM feels dumber than it is post?

So in addition to paying close attention to the software stack, what are the primary things I should focus on if AI code analysis quality is my goal? Is this like, kinda unclear and what everyone is working on? Is this the frontier?

4

That depends on a whole lot of factors. At the most foundational level, model capability is at the heart of it. Even if you sat there and ran a perfect reference inference stack at half precision, the best any of us can hope to accomplish at home, you will be bottlenecked by the model capabilities.

There are too many dimensions to list but, the size of your codebase is going to be a limiting factor. You will not hold a 800,000 token codebase in a 128k ctx trained dense model. That’s just not a thing.

What language/stack/technologies are used, different models have better/worse understanding of those things as well. GLM 5.3 gaslit me for hours this weekend getting into the AMD Epyc Platform Security Processor and secure boot. I ended up writing my own benchmark and running it against the latest/largest frontier models to find out why. GLM 5.3 literally lacks the knowledge of when it was added or how it behaves in different generations of chip. That’s not just a model limitation, it affects the whole family of GLM5 models across any provider.

You will need to test your code base or known sample sets of it against different models to figure out which ones understand it well and can make good quality recommendations (Security is just one aspect of quality). Most SWE benchmarks are kind of broad and focus on popular areas of technology more than isoteric. If you are planning to unleash it on an archaic JOBOL rewrite for your IBM midrange gear (dont call me, really, i dont want that kind of work) odds are you might need something a little more specialized.

There are models built specifically for what you want. The problem is, they are not for sale unless you are part of a specific community. Daybreak and Mythos (if you can get access to those through work) might surprise you, or disappoint. It all depends.

So, sorry there are no simple quick or easy answers here. But its an hoenst take of the world as of 9am this morning. Check back at 9:30 and everything might have changed, thats how fast things are moving.

2 Likes
5

Check back at 9:30…. Truer words ….

Run the largest model that you can on the 192 gig. Open up the context window as far as it goes for your model.

Maybe try several models. Hopefully you have some devious samples for test cases.

I was working on a parser the other day. I use a 90+ gig qwen3-coder that has 30B parameters. It spat out some really nice code, but the actual parser had hard wired numbers in it. I tried and tried to get it to fix it. I finally had to tell it to strip out the spaces, split on ‘,’ then split on ‘=’. It still assumed that the tokens always came in the same order. I told it not to. The final code was really very nice. But not exactly PHD bug finding level.

Having said that, I was pulling my hair out the other day on a random crash. I was in claude running locally and I told it that there has to be a memory corruption somewhere. It did just what I had done. It checked memory bounds and the like. After about 10 minutes, it found a stack overflow in one of the libraries. That’s probably the best bang for the buck so far.

Your mileage may vary… As the man says, check back at 9:30…

1 Like
6

Wouldn’t be as much fun if it was easy right?

Yeah! Using several models and comparing them sounds promising.

Another idea I had was to feed each of the set of models only specific subsets of code that do specific things. So instead of feeding the entire codebase, just feed it say, the parts that handle networking and ask it to find network flaws. Same with cryptographic analysis. Instead of asking ‘How do I build a rocket to get to the moon?’ ‘How do I build a rocket engine that produces this much thrust while connected to the chassis in this way?’

Perhps a better metaphor would be instead of “Here are the entire plans of the Apollo 13 mission including all electrical and computer designs, etc. Find flaws please.” Something like; “Here are the plans for the Service Module’s electrical and life support systems, what could be a problem here?”

Ya know, how ya might ease an intern onto the team by giving them tasks they are likely to be able to accomplish.

In anycase I think it’s time to start building out a secure infrastrcture (thank you very much for the design thr3e-san!) and gathering expiramental data. We’ll let ya know how it goes!

7

Found an answer to my question I think: Qwen3.8 Max - 2.4 Trillion parameters. So 8xB300 actually.

Can’t wait for the bubble to burst in 3 years so I can get one!

8

Not to put too fine a point on it… But if anybody is looking to beat the future rush on home lab equipment now, there might be good pieces to start assembling your basement liquid helium plants now.

Quantum is gonna be a wild ride.

9

I think your time would be way better spent fine tuning as much as possible the best™ model you can run directly off your RTX PRO 6000.

There’s a ceiling to this, of course, but it’s much higher than you might think and it would be much more efficient instead of bringing in a 2.4T model that knows how to do EVERYTHING well. You don’t need that from what I understand.

10

Training requires orders of magnitude more hardware and time than quantizing or distilling something though, not to mention you need a really strong dataset to do the training. But everybody has huge frontier reinforcement learning traces and logit captures sitting in their basements 50tb flash storage arrays right?

Tuning an MoE model sized to run inference on a 6k (lets say 65B+ neighborhood) requires ~a couple B200 nodes to run the fine tune for days, maybe ~1 epoch per day? and moe is really tricky to get right so assume 3 months in the training loop to get close.

Let me know if u want to borrow my data curation pipeline though, human hand labeling and scoring took me months of listening to audio books while clicking my touchpad.

Edit: Here, I have a simple 1 dimension gradient descent simulator posted. I was working on to help explain the concept of tuning hyper parameters to someone:

Now multiply the complexity by 5000 dimensions to spot the challenges :smiley:

1 Like
11

Yes, it is indeed quite the expense. But you can scaffold your fine tuning over time and for the scope of OP seems more reasonable rather than trying to throw as much compute as possible at the problem.
The barrier might be finding reliable datasets, that’s absolutely true.

I’ve not dabbled in MoE fine tuning yet so I take your word for it.
I think three months is a reasonable time to get something decent out of it that will perform and be decently fast.

That sounds amazing, thank you! You can DM me if you want.
I’ve done something similar for voice training and I still feel my stomach turn if I think about it.