I am in the beginning process of starting up an AI server to train an AI, use it to generate code, images, video, and do research for me. I am considering 4 3090s and I wanted to know how they would compare in scope to the 5090. I can buy 4 3090s for the price of a single 5090, so that seems really appealing. I currently have a single 3090 FE and an MSI 5090 I could use, and I would either buy one more 5090 or 3 more 3090 to put in the build.
What I want to know is how much the actual GPU chip affects the speed of AI processing. I know tensor cores make a huge difference in how fast something is processed, and would the total count of tensor and cuda cores being more similar in a 4x 3090 build vs a 2x 5090 build result in similar performance?
If it is a big enough difference, I will get the 5090. I would rather do the 3090s due to TDP, but I value performance over TDP.
If someone has a video comparing the two in the configuration I am looking at, please let me know!
The right answer depends on your budget and what you want to do.
Realistically nobody is training on 5090s. PCIE is too slow and the models are too big.
The fastest inference solution is the largest single card you can afford. 1500 GB/s of memory bandwidth is gonna scream with a rtx 6k. That bandwidth between cards at pcie gen5x16 drops to 52 GB/s. That’s a 30x bandwidth drop moving between cards, and prefill will be bandwidth bound on a properly optimized stack.
Depending on your software side though, you might shockingly find an Ada generation card “faster”. The SM120 performance is not nearly what it should be almost a year later, because the libraries are built for datacenter cards, not workstation GPUs.
Everything is a trade off. And every use case is different.
I am very heavily considering this specifically because of the VRAM and getting 96GB for half or a third of a single RTX 6000. Wendell needs to make an ELI5 series for people like me
I really want to get a 6000, but I am about to put a huge chunk of change replacing my media server drives since they are all at 50k-65k hours. Plus I need to figure what I want to do with networking, I need to get an electrician out, and just a bunch of other stuff. Long term, I want to go threadripper or epyc and do at least one RTX 6000, but that is a down the road project and I am trying to see what I can do now with one of my spare 9950x3d CPUs. I unfortunately am very new to trying to understand things like PCIe in technical detail and how hardware communicates with each other to know what is and is not worth doing. I need a good ELI5 series to watch since I retain info better when watching something vs reading it.
if you go with a single 6k and whatever desktop chip you have laying around, that is likely a better use of cash than a threadripper build. especially with the price of ecc ddr5 rdimms.
I had the same idea, and though I may have gotten a bit sidetracked, the savings will not be as big as imagined. I can second what thr3e is getting at.
If you like tinkering (read: are ready for your next time sink), for sure, I had a lot of fun so far. Just know, on consumer hardware, it’s not quite easy mode.
I would consider using what you have now to do Agentic workflows if that peaks your interest. A 3090 can run a gaggle of 1bit Bonsai 8B models and not flinch. Use the 5090 to run the biggest, smartest model for orchestration you can.
You can build massive systems that do data transformation, ETL pipelines from historical data sets to ai sympathetic data sets, use tool calls for specific things the Unix way, and spend the rest of the money on Storage and clean power delivery for when you decide to take the next step.
I think I need to take a step back and really figure out and be able to articulate what I want to achieve and see what I need to get there. From what research I have done, some models are huge and cannot run on a single 5090. It appears that the more parameters a model has, the better it will be at providing good, accurate data in the output, and I don’t want to spend all of my time fact-checking something, especially if I want it to be able to handle researching various subject matters.
For an AI build, this is something you need to answer intrinsically.
It sounds like you’re heavily interested in inference. If you’re looking at pure inference speeds, the notable problem with the 5090 specifically is that it sits in a really awkward middle ground, where specifically the RTX 6000 Pro Blackwell 96GB is an extreme value play for inference.
The other option is to build out a thread-over system with many PCIe lanes and source 3090s, 3090 Ti’s ideally, and get them used for a reasonable price. Then put them onto a system with a pretty large amount of DDR4, which is a problem right now.
Notably, you want to have NVLink between your 2 3090s in this case to actually run models of a decent size. The NVLink bridges, if you look on eBay, are obscenely priced.
For me personally, I use three systems.
One which is basically a training system with four 3090 Ti FEs with NVLink bridges in between them And with a ThreadRipper Pro with a full terabyte of DDR4 ECC LRDIMMs. If you’re willing to go through all that pain and have the power, which is 240 volts, I had to have a NEMA 14-50 plug put in. It’s going to probably outperform the single RTX 6000 Pro Blackwell for training.
Second is an AM5 machine that is currently running on 96 gigabytes of DDR5. It holds a single RTX 6000 Pro Blackwell. The main important thing here is that if you want a card with a single slot and have everything, the RTX 6000 Pro Blackwell is basically better than four 3090s in a trechcoat and can fit on an AM5 board.
Third is simply a system that runs multiple GPUs as my home lab/infrastructure with Pascal-grade GPUs.
It really depends on your use case “AI” is very broad. I have both cards 3090 and 5090. For image/video models 5090 is much faster but for the price 3090 is pretty good still until you need bigger VRAM. Unfortunately most fine tuning now would run on 32GB, 24GB is still possible but it is getting harder. You also get faster inference on FP8 and FP4 with 5090 as it supports them natively. If you just plan to run LLMs on them multiple 3090s is probably better value. Or in my case I have started to experiment on V100. If money is not a considering get the biggest single VRAM like RTX Pro 6000 you will have more flexibility. The cheaper options like I have can eventually be expensive if my use case changes or software support changes.
AMD fanboy here.
I was so obsessed with memory capacity that I ended up buying a couple of R9700s. I only installed one so far. In theory the specs look good, more memory, more compute, support for FP8, but the memory bandwidth is 32% lower than the 3090. Software is supposed to be a nightmare, but what other better way to learn. I did manage to run llama.cpp with gpu support.
Intel supposedly launched the B70, also with 32GB. I see it available to order at ~10% lower price than AMD. As far as i saw in specs, AI compute should also be 10% lower, but gaming 50% lower. I guess nobody buys it for gaming. Looking at the prices I’d say it’s almost AI compute arbitrage.
Interestingly, even though the bandwidth is much lower than the 3090, I’m getting roughly 3090-level performance from my R9700s (I had a pair of 3090s in that machine before).
I still don’t get why people are buying the B70 for inference. Yes it’s a bit cheaper, and yes it’s got the theoretical hardware specs to compete with the R9700, but in the real world it’s at least 50-60% slower even when running in its best-case conditions.
As for the R9700s, I’m finding them to work well - they’re always behind the curve when it comes to optimisations because CUDA comes first and proper ROCm support is nowhere to be found relative to Vulkan, but the new model support usually only lags by a week or so.
All in all, they’re a pretty well-balanced card. The 3090 feels like it should have 32GB VRAM, and the 5090 is blazing fast beyond the point where it’s useful; all that horsepower would be more useful in a card with 64GB.
