So my 6900xt died in my homelab, and while prices are reasonable…ish, I am looking for a replacement, I’m leaning towards 9070xt or 7900xt. They’re at pretty much price parity (at Microcenter.) I primarily focus on worrying about inference, not necessarily training. And for the moment I only really care about text, not audio, images, or video.
The thing I’m really struggling with is the fact that the 9070xt is PCI gen 5, card and I have an AM4 system, so it’ll run at PCIe 4 speeds, so I’m not sure if that, combined with the 256 bit memory bus (vs the 7900xt’s 320bit) and less memory bandwidth on the 9070xt is also less.
The machine is entirely on Unraid, so linux.
I do game, on bazzite, but my old 6900xt was more than enough for everything I do, so no worries on that front.
Now, I am curious, some other really “bad ideas” ideas that I am thinking on, what about doing something like 2x intel B580s for inference (and just use one for gaming)? I’m open to anything in this price range.
The AM4 and pcie gen4 things on this are the main questions I have outstanding. I’m not quite sure how much of an impact it will have.
For inferencing within VRAM limits the difference between PCI gen 4 and 5 would only be noticed while loading models. Once loaded there will be virtually no difference.
Nominally 645 GB/s versus 800. The cache layout also shifts. I don’t know of a good comparative LLM benches between the two but AMD’s put out some first party numbers comparing to the 7900 GRE, which gives an idea. Absent reason to expect regular use of 16-20 GB GDDR, personally I’d bet on the 9070 XT.
And probably not even then. 4x16 offers 31 GB/s bandwidth while 5x16 is 63 GB/s. So, unless there’s a high end drive array, a very optimized loading stack, and DDR4-4000+ or DDR5-5600+, PEG isn’t going to be the bottleneck on dual channel DDR. We have a whole thread about how LLM loading speeds tend to get stuck around 2 GB/s.
If it’s transfer of cached coefficients from high overclock DDR4 to GDDR then PEG’s a potential bottleneck. But I don’t know of any data showing it’d actually be hardware limited rather than thread bound in torch or whatever. If it is hardware constrained I’d expect DDR5 to make more of a difference than 5x16 PEG.
thank you wildly for your input. This is eye opening.
The main reason why I am waffling as much as I am is the idea of that extra 4gb of vram enabling larger models. It gives me more context headroom with something like say the GPT-OSS20b or Gemma or any number of the 32b models that are 4bit quant. THAT is the main reason why I’m waffling (and living up to my name.)
It does sound like my off the wall ideas of multiple intel cards though is just… a bad idea?
I don’t know enough about dual bonding and Battlemage support to speculate usefully about it as a path to 24 GB. Like if you have x8+x8 muxed PEG, or maybe PEG + X570 x4 slot, it’s not my sense it’s not worth at least looking into. But I’d guess it’d probably be involved compared 7900 XTX or R9700, despite those being out of budget. Maybe also B60, but seems like they’re pretty much unobtainium, definitely at least around here.
yea, there was an open box xtx at my local (ish, 3 hours away but have friends that can pick it up and I was visiting there later this month anyway) that I was willing to push my budget for, but it seemed to have sold and now it seems the next up xtx is rapidly approaching “hundreds more” rather than “an additional $50 I can stretch to.” I’ll play with some of the “can run on 16 gigs” models to see if they suit my needs on openrouter and see if they satisfy, cause it sounds like the other technical upgrades are definitely worth it. I just want that model size dangit!
I’m lazy today, so here’s my quick google synopsis. RDNA 4 introduces dedicated, specialized hardware units called Matrix Acceleration Engines, often referred to as “Tensor Cores,” for significantly boosting AI and machine learning performance, a major upgrade from RDNA 3’s less specialized approach. These cores support new data formats like FP8 and BF8 crucial for FSR 4 upscaling, offering massive throughput gains (2x-4x) over RDNA 3 for AI workloads by handling matrix math more efficiently, bringing AMD closer to Nvidia’s AI capabilities.
@lemma dropped a really useful link in their first post, of some first party benchmark differences its a huge uplift, I’m just worried that I’ll be majorly offloading a bunch of stuff onto system ram for models like the GPTOSS 20b.
Probably also something in there about optimization, what with the AI craze and RDNA4 being when AMD finally acted on its need for a tensor core compete.
Having built both, I’m quite happy with the rasterization and GDDR compete 7000 series offered to 40 series but that’s only half the use case here.
ok, to throw a wrench into things… 9070xt or possibly a 7900xtx from facebook marketplace that looks like its only 6 months old stretching my budget an itty bitty bit. My main concern is the one I think I would be able to get off FB marketplace suggests a 1000w PSU and I only have an 800w.
The trouble is…Intel’s software support is awful. Like, worse than useless, because a) it’s restricted to old versions of llama.cpp, vLLM etc, so you’ll never be able to run the latest models, b) they have no qualms about breaking their software stack from day to day, to the point where you can’t even rely on building a Docker image two days in a row, and c) if you want to avoid all that, the only way is to use llama.cpp’s native SYCL or Vulkan support, which - on Intel GPUs - is functionally no faster than using a 14600k for CPU inference.
As to the original question…a used 3090 is the answer. I just bagged a second Founders Edition here in the UK for £450 (although I’m not convinced the guy’s actually going to go through with the deal…I’ll update tomorrow if it’s real), and they regularly go for around the £500 mark.
I’d buy soon, though, because we’re probably about to see a step-change in prices on the used market…
is a 3090 preferable to a 7900xtx? I have a deal in the works for a 7900xtx, its a newer card, I know rocm is still not as good as CUDA, but its still good enough yea? And the card itself is 6 months old so I’d get some warranty left in it. And while I don’t have a huge emphasis on this fact in this thread cause all the cards we’ve talked about will be more than sufficient for my usecase for it, I do game, so having a newer card would help with that too.
Well, if you’ve got a job to do then the best card is one that’s in your hands rather than one you’re hunting for, so…if you’ve got a good deal on the 7900XTX, and you can’t get hold of a 3090 for the same (or less) money, then it’s the one to go for. It probably won’t perform quite as well as a 3090 given the weirdness in AMD’s ROCm stack, but it’ll get you most of the way there (most benchmarks have it at around 90-95% the performance of the 3090 in AI workloads).
The only remaining question to ask yourself is…is the convenience of getting one right now enough to soothe the friction of not having CUDA available?
The good news is that the 7900XTX tends to sell for more than a 3090 on the used market (at least here in the UK), so it’s not going to hurt too much if you change your mind later on.
AMD’s supported much of CUDA for some time and finally got not stupid about desktop GPU support with RDNA3. With last month’s main channel drivers AMD caught up enough torch even runs natively on Windows. Linux is, hmm, a couple years ahead of that?
Given Nvidia’s turn away from client and AMD’s interest in occupying the space I’m not sure I’d bet 3090 over 7900 XTX over the prospective use lifetime here. Personally my own builds hedge the question with about half being Nvidia and half being AMD.
So, this is at the end of the day, my homelab. I also am inclined to go with AMD cause better linux gaming drivers. 3090 would be better on the AI for sure, and I do run bazzite on my gaming machines that can be bundled in on the OS, but the unraid nvidia setup is annoying. I think I am gonna be quite content with the AMD. Again, I am not running a business off of this, I just like the idea of getting the most LLM I can for my budget while keeping the fact that I like gaming in the loop. I was quite content with my 6900xt, and would’ve kept it indefinitely had it not kicked the bucket. Ran the models it ran more than well enough. I’m mostly running ollama, I may try llama.cpp with vulkan at some point cause its faster? But /shrug, ollama is good enough.
That’s fair enough - I hadn’t clocked that you were planning to do all this on your daily driver; in that case, AMD is by far the better choice for desktop experience (I use an A770 on my desktop for that very reason - Nvidia drivers make the Linux desktop feel like you’ve downgraded by a decade).
