You missed the ideal purchase window by about a year, price & availability on a lot of hardware is way worse than it was, you may need to cut back hardware performance or increase budget, or run MOE type models on RAM & VRAM instead of just loaded in VRAM. A DIY build can make a lot of sense if you want performance and are willing to tinker and diagnose, otherwise a “unified memory off the shelf box" is your best bet for good enough performance (should be easy enough to see what people run and how fast it is)
Should mostly consider 32GB cards, 16GB GPUs are OK but it’s not dense enough for the cost of the GPU & boards or splitters to run a pile of 16GB cards. The AMD R9700 was pretty solid at $1300 but is more now, still a newer architecture and one of the fastest commonly supported 32GB GPUs for the best price (1 Alphacool waterblock option available in case you want to water cool the rig later, avoid the Gigabyte cards in that case as they aren’t compatible). 5090 was the next big step up with faster memory and CUDA which is well supported at 2,500-3,000 but it’s not even available anymore. A step down is the Intel B70 & B65 32GB cards, the software support isn’t quite there but it is alright from what I have seen.
Depending on your use case it might make sense to run 2x or 4x the same 32GB GPU for 64-128GB of total VRAM and fit your models into that with maybe a previous generation Threadripper/ Epyc motherboard to give the GPUs as many direct to CPU PCIe lanes as possible for the money. Or maybe a kinda modern AMD/ Intel consumer CPU & Motherboard with half decent double PCIe 4.0 x8 or double PCIe 5.0 x8 slots so you can run 2 GPUs at modest interconnect speeds on a more affordable platform that hopefully let’s you afford more RAM to help fit larger MOE models because you will only have 64GB VRAM.
You can go multiple generations old and get a lot more VRAM for your money, but performance will be kinda poor if the hardware is kinda affordable. Plus you need to plan for supplying enough power and dealing with the heat of a box full of older gen stuff that will run longer for the same result (depending on the hardware). Should really avoid P or Pascal generation stuff, missing too many instruction sets and modern features or support.
Otherwise you should avoid doing a self built CPU & RAM only inference machine as RAM bandwidth even with an bunch of channels isn’t great plus the CPU won’t be able to number crunch like a half decent GPU can. Technically it can let you run very large models, but it will be so slow it is probably unusable. This is different from the new unified memory boxes.
If there is a Microcenter near you it could be worth buying a Mac, Spark or Halo device, you can return most things for any reason within 30 days, so you can buy it, set it up and make sure it’s good enough for your needs. The sorta powerful iGPU in them is alright enough at decode/ number crunching, but memory bandwidth is a limiting factor for token generation speed, don’t bother with the versions that have less than 128GB on the AMD or Nvidia boxes, for Apple M-series good luck finding anything in stock with a lot of unified memory, probably several month lead time if it’s not immediately available.
I was strongly considering a 128GB Strix Halo box when they were $2k, instead I built a $12k Threadripper box with 4x R9700 GPUs (128GB RAM & 128GB VRAM). Just easier or possible at all to upgrade it over time, plus is more capable, and if you have a board or major component failure on a unified memory box it becomes a paperweight if the manufacturer won’t replace or repair it.