good timezone all,
my company is building a machine for local ai workloads. really just looking for any obvious issues with the below specs,
1-2x RTX Pro 6000
1x Threadripper Pro 9955WX
64GB DDR5
Asus Pro WS WRX90E-SAGE SE
Seasonic Prime 2200W
2x Samsung 9100 Pro 1TB
Get a better CPU with more CCDs, it’ll have higher actual memory bandwidth. Shoot for the 32c part although 24c should work fine.
Why would he need more memory bandwidth, he plans like 64GB memory, which is the bare minimum, it does not look like as if he would want to use CPU offloading or CPU inference. The two RTX Pro 6000 will do absolutely fine on a smaller processor. He would not even need a system that new for it.
@Reflected3771 maybe you can tell us more what kind of models you want to use and what you plan to do with it in detail.
Off the cuff, larger storage might be useful depending on the models and quants you’re looking to use. Like H-i-v-e mentioned, sharing what models and plans you have in mind will certainly help.
If there are no plans for the CPU then yes, the current config seems fine.
Seconded, we need to know what models and workloads this will be running to assess the build. 
Thanks all, storage isn’t a concern as we plan to use a few specific models rather than a variety. Atm we’re using the midsized Gemmas (~27b) but that may change with the better hardware capability. The aim is for consistent large volumes of output rather than biggest possible models.
What’s everyones recommendations regarding memory? I’ve looking at 2x Kingston KP8XPW-MID-IIMI 32GB 5600MHz as stock for 32GB dimms in 6400MHz was limited and i think dual 32 will be better overall than single 64 - the board gives us plenty of room to expand with 32GB modules.
Assuming you’re planning to use the full BF16 weights for either the 26b or 31b models, the weights are ~50-60 GB on disk. Those need to load from disk to the GPU VRAM. If the model is planned to always be in VRAM, then it’s a startup cost to load in with system RAM less than the model weights, but if it’ll be loaded and unloaded regularly, that’ll likely be a bottleneck. Dual channel memory is generally better, but likely more specific for the CPU/MOBO. Someone else more experienced on model loading and infra could correct me, but that’s my understanding.
2x 1TB for storage will be okay if there are no plans for fine-tuning custom models or trying out other specialized fine-tuned models. Even with several fine-tuned model you’ll likely be okay, but just something to be aware of as usage increases. My personal experience is that when a new model drops that is reasonably sized for my GPU, I tend to want to download it and try it out. My primary drive filled up quite quickly bc of this, so I had to get a secondary drive to migrate other data off the primary drive. Certainly could go differently for an enterprise rollout, but figured it was worth mentioning that storage could become an issue if/when other models are desirable for testing or deploying.
With memory engram offloading to ram working on b12x, you are going to want to budget 150-300gb per model weights on disk to feed 2x rtx6k cards at 192gb vram. 1tb will feel awful snug real fast, or you can offload model storage to slower storage and only keep the 2-3 you are actively swapping on nvme. but you really need more gen5 nvme drives to feed models into vram with instanttensors. running 4 m.2 in a zfs stripe, vllm will absolutely pull 50GB/s. plus if you wind up having to quantize your own, you need moar. always moar.
Interesting. This machine will be for production use so won’t get much model testing done on itself - we have other systems for that. But I may suggest we go for dual 2TB NVMe instead of the 1TB just to be fully comfortable with storage. Considering the rest of the component cost, this shouldn’t be a hard sell. We do plan on adding more RAM to the system eventually, but trying to get away with the minimal amount in the current market.
Okay last question on this I think.
What OS would everyone recommend?
Primary workload will be containers but we will need some VM support. Considering either TrueNAS or Proxmox for for Appliance-like experience, but also thinking about just doing it vanilla debian. Not sure how well TrueNAS and Proxmox will support the bleeding edge hardware?