Advice for a GPU for a Workstation for AI

[Loki64]

I’ve recently ordered an Ampere Altra Developer PlatformAmpere Altra Developer Platform and I’m anticipating it’s arrival in about 5-7 weeks.

Since I ordered one anyway, I might as well use it. I want to use it as a workstation where I could play around with AI such as Stable Diffusion and LLM’s.

Here’s what I already figured out:
According to this blog I should look for a GPU with lots of Tensor Cores and Memory Bandwith and VRAM

This Stable Diffusion benchmark seems to suggest that when using xformers, the 7900xtx is slower than even the RTX 4070 for Stable Diffusion. And it’s 82% the performance of the RTX 4080.

Without it, the 7900xtx is about 110% the performance of an RTX 4080.

Potential options:
I could buy four AMD Radeon Instinct MI25 16GB along with a Bifurcated Riser X16 To 4X4 for 507 Euro total.
Or I could buy a NVIDIA Tesla V100 16GB for 587,09 Euro
Or I could buy a brandnew 7900XTX for 1000 Euro

Anyway, here are my two questions:
Could I get links to additional resources that could help me make a better informed decision?
Could I get some advice on what GPU to buy?

I’m ready to pay about 1200 euro’s max.

Thanks |:-}

[lemij31400]

1200 should be able to get you 1x RTX A5000/ 2x RTX A4000s on the used markets.

I like AMD cards but for ML they arent the best unless ROCm plays nice. and for me on a MI100 i tested it was a pain to get 100% working. So i would still go NVIDIA all the way.

I have 1x RTX A5000 in my machine right now if you are curious and i can do some benchmarks

[Loki64]

Thanks for the quick response.

Yeah, if you have some benchmarks, that would be great. Especially if I could compare it to others.

[lemij31400]

Sure give me a few hours to escape the GF and i can benchmark, any projects/programs u want me to try?

[Loki64]

Well… I’m not sure about any good projects/programs for benchmarking. So a quick google search yields https://github.com/AUTOMATIC1111/stable-diffusion-webui.

Whatever is handy for you is fine as well.

[gc71]

I am not familiar with the workstation in question (Ampere Altra Developer Platform – I-Pi SMARC)
however one thing sticks out to me;

750W power unit

is this saying that the system is expected to use 750W when running, or that the PSU is rated for 750W? I am assuming its the latter. Honestly that feels kinda low for a system you want to load up with GPUs for AI stuff, so I would probably agree with @lemij31400 regarding the RTX A5000, which was running about $1200 USD used on eBay last I checked. You could get something like a RTX 3090 for half that much (used on eBay) however the power draw is significantly higher.

Note that VRAM requirements for Stable Diffusion is far less than for LLM’s; SD can get by just fine with as low as 12GB, but afaik atm you will want about 24GB VRAM for some of the larger LLM’s, which would put you in RTX 3090 / 4090 / A5000 and up territory.

also I am not sure at the moment how useful multiple GPU’s are for these purposes (can anyone weigh in on this?); I have 2x RTX 3090’s in NVLink in my system but have been too lazy to actually try utilizing it with LLM. I think @wendell mentioned recently that SD was only running on a single GPU with A1111.

if multiple GPU’s are indeed useful for you, well a few months ago you could get RTX 3090 used on eBay for about $700-800 USD, so a dual 3090 setup would run you about $1600 + $200 for the NVLink bridge; however it would blow out your PSU’s 750W rating. I think you could do an analagous setup with dual A5000’s within your PSU power rating but it would cost about twice as much.

just some thoughts, others please weigh in as well

[Loki64]

Looking at this picture of the system, it appears to be a standard Power Supply, so I should be able to replace it with a more powerfull one.

[m0nkey]

Any of the Pro cards from AMD or Nvidia that have ECC memory are necessary to avoid data corruption when handling AI workloads.