I’m struggling on making a decision to buy an AMD 7900 XT or an NVIDIA 4070 Ti Super. I will be doing some light gaming and video editing (Davinci Resolve), but really want to start playing around with running local AI. Team green seems to be the current leader in the AI space, but for $100 less I get 20 GB vs 16 Gb. Has ROCm matured enough where it levels the playing field from an ease of use and performance perspective?
Specs:
7700x
32 GB
Dual boot Windows/Linux
AFAIK you will be able to use most open source stuff. Some comercial stuff like Mathlab are CUDA only.
For the open source stuff a slight degree of suffering might be necessary like using a different version of torch and so on.
You will be fine under linux and it might work okay under windows. Also, with Vulkan support from llama.cpp it is really easy to get things working under both windows and linux just not as fast. If you are running a model that fits in the 20gb vram but not 16gb, you will get better performance than dual gpus also. Vulkan multi gpu also works quite well right now.
The simple answer is no! I would encourage you to research compatibility for specific packages before pulling the trigger on an AMD card.
Users’ horror stories are endless, even for packages that go out of their way to support AMD, like llama.cpp through hipBLAS.
Conversely, Level1Techs is an AMD-centric community so I’m sure there are users who will help you through the inevitable pain points.
Bruh, I dont know if I (we?) should feel attacked? 
Back on track, I feel interested in this as well as I already have the card (but not the time) to do this.
Not at all, I was stating that there are a lot of users on this forum with AMD GPUs. Wendell included.
Its a joke I use both, but you are right, I do love AMD more 
Could you elaborate on such horror stories? I’ve been using HIP on llama.cpp from even before it was merged and never had any issues besides some OOMs while “pushing a bit too hard” with context size.
I also don’t think they go out of their way to support it. Some dude forked the project and HIPified the code. Gerganov didn’t actually touched it much. The project nowadays is open to many kinds of backends. If anything I would say the backend they go all of the way to support is Metal.
Yeah Wendell mentioned in his latest 7900 GRE review that he would also post a review on Level1Linux soon. I’m hoping he talks about AI on AMD.
Flash attention barley works for rdna3 so good luck
Is it even merged on llamacpp? I’ve seen some discussions but not much beyond it.
I’m talking about the library, which also can’t do training because there’s no backwards pass functions.
My impression is that while specific projects work with rocm/amd, NVIDIA is still the industry standard. If you have a single use case and and is supported, go for it. If you want to be sure pretty much anything will run on your gpu (provided you have the vram), go with nvidia.
It is what it is. But to be fair, 20 years ago NVIDIA started investing in software support and documentation for cuda, it’s no wonder they are leading right now.
This really all comes down to your use case and how comfortable you feel.
I will put it in terms like this.
If you are a tinkerer, if you like to pull things apart and rebuild them constantly, if you own a classic car that you drive on a Sunday and then spend all week fixing…
Then buy the AMD.
If you like things that just work, that you can get up at any moment and say - oh I want to go for a drive to the supermarket, I need some things, if you own a modern car with all the ease that comes with it.
Then by the Nvidia.
Both can work, both will work, one will work with less pain points than the other.
Well, like others here have said - Nvidia is, hands down, the easiest way to run AI today. It is because Nvidia bet on AI while AMD was in the gutter trying to fish up their lost Radeon RX 380 glory and could barely spend a penny on other stuff.
This has led to Nvidia being the clear choice today. However, there are signs pointing to AMD making a strong comeback, because as AI becomes more important more and more customers realises that by relying on CUDA, they are essentially giving Nvidia the keys to the kingdom - and this might not be in their long term interest. Especially given the recent tales of how Nvidia treat their partners (hint: borderline abusive). That coupled with previous tales of Google, IBM and Microsoft, make many people question whether they want to be in Nvidia land with no way out.
Why is it that every tech company, when they get to control a big monopoly, inevitably turn into narcissistic sociopaths? Must be something in the water, I guess… 
Anyway, my point here? AMD is the secondary option today, but is picking up the pace and might leapfrog Nvidia within two years or so. This is all currently speculation however. The safe bet is Nvidia now and then re-evaluate in three years or so.
I decided on the Nvidia 4070 Ti Super. I got it installed in Windows without any issues and it’s running great.
When I boot into LMDE 6 I’m greeted with…
“[FAILED] Failed to start lightdm.service - Light Display Manager”.
If I plug it into the onboard DVI port, it boots normally. I tried a few Google suggestions but couldn’t get it to work. I’ll roll it back today and poke around a bit more, but I think I’m going to have to do a fresh install. Most of the LLM how-to articles seem to focus on Ubuntu 22.04, so I may go that route.
Yeah, you probably need to install the proprietary Nvidia drivers manually on Linux, or do a fresh install with the 4070 Ti Super in the computer.
Nvidia really do not care about non-KDE, non-Gnome desktops, so unless your distro is specifically catering towards making it work with your favorite DE, then there will be breakage.
Especially on Wayland.
Why do you think a lot of Linux people recommend Radeon over Nvidia for? It’s not just software purity, Radeon integration is de facto better with Linux.
geohot has some interesting points regarding the quality and current state of the AMD drivers:
