I am looking to revamp my AI workstation with 4 3090s in it.
with a budget of around 10k, should I go with 4x 5090s (when the price stabilizes) or go with some other GPU like a used A100?
I am seeing that most of my workloads dont use multiple GPUs very well. Training does well, inference does not. And this workstation is not serving multiple people, I have a server for that. So for single user AI dev, I am wondering if I can get more bang for my buck with a single card vs 4.
It’s a worthy question, some of the answers lie in your environment
Is the software you use able to scale 2 GPUs but not 4? Does it benefit from a singular non distributed pool of vram? Are you on a multi processor? Numa zones set up across the correct channels? Are you virtualizing?
Without any other data elements, a go to suggestion would be for your budget to stick to 1 or 2 high vram beefy cards. It also depends on cuda, tensors, CL, etc.
Will your software change with updates, or is it strictly you writing it and doing your own algorithms?
I develop based on others’ tools. Sometimes i train models using python and wrapping the training loop for multiple gpus. My workstation is a single 64 core tr pro with 1tb ram. Right now i have dual 1600w psus. So putting 4 5090s at 600w plur rhe cpu at 400w and 21 fans, 2 pumps, etc. might work.
I have yet to see a tool distribute vram effectively. Even llama.cpp doesnt seem to currently do that well. If it does distribute, because we cant nvlink anymore im fairly sure that the pcie lanes are bottlenecking enough that the gpus can run full tilt. Nvtop has shown a max of about 1.3GBps.
You say you use python, do you mean you use a framework like pytorch, tensorflow, or another machine learning framework? Or just python itself with a CLI program?
With 4 3090s, you have about 96GB of VRAM but at the cost of about 1400 watts. With this in mind, what do you want more in terms of performance and efficiency?
An example would be a single RTX A6000 that has 48gb at 300 watts but similar speeds to a 3090. The cost for one of these is much more expensive than two 3090s, but you do save 400 watts for other gpus (including physical space of the gpu slot).
Also, I don’t think a used A100 80GB is under $10k, maybe an A100 40GB. But are the additional things that comes in an A100 is worth more than a workstation card like the RTX A6000 (ampere gen) to your needs?
I do both, I use frameworks and sometimes I might get into the weeds with straight python (still using modules and such) but at a much smaller level. Multi-GPU really only comes in with TF, pytorch, and tensorrt.
Typically when I am loading into VRAM when using multiple GPUs, the data is copied from each GPU so for 4 GPUS, I have 4 copies of data in VRAM. So my 96GB VRAM doesn’t work as one might think. That may also be because I am not as strong as a developer as others and just don’t know what I am doing yet.
power doesn’t bother me, heat does. but currently I have enough cooling to let all them rip and it is fine. My office however, gets pretty toasty.
The RTX A6000 Ada 48GB is about 8k
The A100 40GB PCIE is about 4.5k
The A100 80GB PCIE is about 17k
The RTX 5090 32GB is supposed to be about 2k. (I can wait till prices stabilize).
I’ll be honest—we have a couple of workstations in the office and a small training server.
For workstation purposes, the A6000 or A100 just isn’t worth it. It’s a ton of money for what it is. If you really need it, you’ll know.
Get the best GPU you can afford, which is most likely a 4090, and call it a day. I have no idea what kind of use cases you’re dealing with, but if you need multi-GPU and high-end hardware, then just rent it. It’s not worth owning.
I have a Dell C4140 with V100s in a rack. I am looking at getting that upgraded in the future once I have a business case for it. For now, I am just doing development so funds are going into this sort of thing. I am seriously leaning toward the 5090s now. I just stumbled across the fact that Alphacool has an ES block for the 5090 that is single slot. That offers workstation density that I did not think exists. In theory I could run up to 224GB Local VRAM. with 1TB system RAM and hopefully a CPU/Mobo that I get fixed in the next week or so.
This topic was automatically closed 273 days after the last reply. New replies are no longer allowed.