I had $45k budgeted for contractor help this year, and I asked if I could put it toward a local AI setup instead. They said yes, but I have to spend it all in 2026. Now I just want to make sure I’m not about to do something dumb with it.
Where I think I landed is six DGX Sparks and a MikroTik CRS804. For now, I’m thinking four together on the switch and two off on their own, but the nice thing is I can shuffle them around later. 4+2, 6, 2+2+2, whatever I end up needing. In theory you could run 8 at 200G off the switch too if I run breakout cables, but I haven’t seen anyone do that yet.
Honestly I mostly want something that’s supported and not a pain to set up or keep updated. Part of why I went with the Sparks is there’s a big community of folks sharing optimised clustering setups, and I like the option of not needing to fully be off in my own world doing all the optimisation myself.
I work remotely, and this would only be accessed by me and my agents. I have a fair amount of networking and datacenter experience, so I’m open to running just about anything and could put up a rack in the basement if I needed to, but I like the idea of a mini rack for a setup like this in my office.
Is this the route you’d go with this kind of money?
Using that many sparks would get you a lot of vram and load a very large model, but any inference on a model that fills that vram would be unusably slow because you’ll be bottlenecked by memory bandwidth. Single digit t/s inference is not where I would want to be.
But I have interactive use cases. Maybe you don’t. It would be easier to give you suggestions if you’re able to give a bit more description about your use cases. Concurrency? Long horizon? Coding? Chat? Image / video? Do you have a particular model already in mind?
Like Honeybadger said, speed of token generation is almost always more important (to an extent) vs “running” a massive model. Being able to run a massive model is alright but in a lot of cases you can run a still very large model either partly or mostly on VRAM with a more traditional Motherboard, CPU & GPU build. Personally output slower than 30Tokens/s feels too much like waiting for dial up internet or games from the late 90 to load on old computers, 20T/s sucks, 10T/s is brutal, and around 1-2T/s is an exercise in patience I don’t have.
One massive problem people forget about with slow token generation speed is how a particular output will require X amount of tokens, so let’s say a whole project takes something like 2.5 million tokens over several weeks, at 10T/s it’s probably around 70 days of generation time, at a more reasonable 60T/s 2.5 million tokens would probably take less than 12 days of generation time (pretty sure I got that estimate close enough). Generally the cloud API cost for something like Deepseek is maxing out around a couple of dollars for 1 million tokens, not that a lot of data can/should be offloaded to Chinese servers.
If you can completely avoid the sub 4bit quantized models you will probably have a good time in terms of output quality.
Or you can go kinda hybrid and build a 2-4 Spark cluster and see how it goes, then use the rest of the money for something like a PCIe 5.0 x8 x8 double AMD R9700 build on higher end consumer hardware (that’s 64GB of decently fast VRAM on a modern architecture that is decently well developed for, then have a pile of RAM on it to open up running pretty big MOE models). Can also just use a chunk of the budget on one or two Nvidia RTX PRO 6000 cards with 96GB of fast VRAM each.
I looked at the Spark & Strix Halo options and went with a Threadripper & eventually 4x R9700 build due to token generation speed, memory bandwidth speed, upgrade options, and fact that a component failure means I can swap a part in easily.
Surely a Threadripper or Xeon workstation would be much more appropriate with that budget? Then it’s PCIE 5.0 x16 * 2.
Given that it’s a work machine, though, you have to factor in the maintenance overhead. Managing a Spark cluster, as well as updating it as the niche tech develops at the bleeding edge, is going to become a bigger and bigger headache which drags away from the actual work being done.
Personally, I’d just jump straight to a mid-range workstation with 128-256GB RAM from HP or Lenovo for $15k with a pair of RTX Pro 6000s, though. 192GB VRAM on two flagship-class cards with PCIE 5.0 x16 connections is about as good as anybody’s going to need these days and for the foreseeable future.