Considering a second (and fourth...) R9700 for my 9950X build

Hardware Hub GPU
,
1

Hello everyone. I am brand new to this forum and was not sure where best to post it.

After living years off my old budget laptop, I have recently built my first PC in like 15 years, and I might have caught a hardware-buying virus.

My build so far is:

  • AMD Ryzen 9 9950X 4.3 GHz 16-Core Processor
  • be quiet! Silent Loop 3 78.4 CFM Liquid CPU Cooler
  • Asus ProArt X870E-CREATOR WIFI ATX AM5 Motherboard
  • Corsair Vengeance RGB 96 GB (2 x 48 GB) DDR5-7000 CL40 Memory (Running 6000 CL28)
  • Samsung 9100 PRO 2 TB M.2-2280 PCIe 5.0 X4 NVME Solid State Drive
  • ASRock Creator Radeon AI PRO R9700 32 GB Video Card
  • Asus ProArt PA602 Wood Edition ATX Mid Tower Case
  • be quiet! Power Zone 2 1000 W 80+ Platinum Certified Fully Modular ATX Power Supply

All running on a pretty up-to-date Arch Linux.

Overall due to the overkill cooling situation, the build is practically silent, that is unless the GPU is pushed to the limit, then somehow the sleek ASRock card can make the earth shake, but such is the blower form factor tax.

Generally it was meant to be a productivity/fun build, to help me with personal software development projects (big C/C++ codebases, FPGA).
As a bonus I thought it would be nice to try out some virtualization experiments to run multiple systems like Windows if some of my tools or SW requires it, set up some servers, maybe try stuff like locally streaming 2 “gaming” instances to my 4K TV for coop on the couch etc.

I have also been using LLMs to accelerate my learning and development a lot. I consider them to be interesting tools, but even the SOTA stuff like Opus 4.5 feels lackluster at times. I know it will not be feasible to have something of this sort running locally, but I would appreciate something usable for deeper reasoning and analysis of information across various documents and sources, with the ability to work with decent context window, as a lot of the information present in the model itself would work against me (one of the aspects of the project is my personal programming language, so a lot of spec and sources need to fit in the window).

For this reason I am considering expanding to 64GB with a second R9700 to be able to run smarter models with more context. (would unfortunately need a PSU update as well)

More than anything, this question might be about the current state of the LLMs available at this scale. I am not made out of money but would survive this purchase, if it could bring some decent benefits to my projects. It is a great time to be alive in terms of the variety of nerdy topics worthy of exploration (apart from the current HW pricing…).
I would still hit a pretty hard wall with PCI lanes here though with 2x setup, and I cannot see myself upgrading to threadripper in the short term.

Long story short, questions on my mind are:

  • how useful are LLMs that can run on 2xR9700?
  • for a person that is generally interested in various software and research projects, what fascinating doors could be open by expanding on the setup a little bit?
  • finally the value of getting these cards… I live in Japan, where they are a bit pricier than some other countries, but they have not exploded yet like RAM, Nvidia and SSDs. Do you think they would generally keep their value, or they could be made “obsolete” pretty soon? (not literally, but for example perhaps some of the shared memory platform with higher memory ceiling could be a better choice?)

Sorry for this post being a little bit vague and all over the place. I thought this forum might be a place where people with similar hardware or interests would be kind enough to give me some feedback on this considering the present state of this domain.

Thank you very much in advance.

2 Likes
2

Considering something similar. In terms of value for local exploration, besides the VRAM dont forget 2x R9700 also gives you 2x the (somewhat limited) TOPS of a single 9070XT/R9700 too.

And yes the cost bump to go to Threadripper is very high merely for the extra PCIe lanes. (mb, cpu, case, ram are all much pricier.)

Other questions to consider:

  1. For AM5, 2x R9700 means running each in 8x+8x mode. How does this impact performance vs. published 2x R9700 benchmarks from Threadripper systems running these in 16x+16x mode?

  2. If there is a serious impact from running 8x+8x, is it worth trying an external PCIe switch which would allow full 16x p2p for each? (e.g. those from PCIe Packet Switch Adapters - gen5 – c-payne.com ) What is the cost/benefit of that vs. going to Threadripper?

2 Likes
3

@CircuitSage I pulled the trigger, couldnt help myself. :wink:
I was under the impression that PCIe5 x8 per card should be quite enough for these cards.

If there is some stuff you would like to me try or benchmark let me know. I should have it running on Wednesday.

4

That setup of yours is somewhat similar to mine (9950x + 4x64GB + ProArt x670e + 2x3090 + 1000W PSU), so I’ll chime in with my 2 cents.

I run both my 3090s on a 1000W PSU. I used to run them on a 850W PSU as well without issues (with the proper power limits, of course). You should be fine and not even need to power limit those GPUs, given that a single 9700 draws up to 300W (if i’m not mistaken), so with 2 of those (600W) you’d still have 400W to spare, which is more than enough for your 9950x and peripherals.

x8/x8 is totally fine for only a pair of GPUs, specially given that you’re only doing inference.

Far from being as good as the huge frontier models for general tasks, but you can find smaller, purpose-specific models that are really good. Really depends on your use case.

1 Like
5

Might be worth actually looking at the models you want to run, rather than just “can run bigger stuff”. I looked around, and I couldn’t find much that I actually wanted to run which would need 64GB vs 48GB in my dual 3090s when I had them. Even then, I was absolutely fine stepping down to a single R9700 - performance is similar enough to 3090s on the models I run, and I was grateful to run with less power.

IMO, the next properly useful step up from 32GB is 96GB, if you’re looking at mainly inference workloads - that’s when you can start running 70B and 120B models with useful context sizes.

If I was running those models, though, I’d definitely be wanting to upgrade to something with more grunt than the R9700. They’re great cards, but maintaining usable performance would need something with more kick to it, at least for me.

1 Like
6

Thank you for the responses.

Upon looking a bit more into the power supply a couple days back, it seemed like while it looked doable in terms of the pure power budget, providing power to both cards seemed a bit dodgy (adapter was 3x8 pcie, whereas I only has 2 connectors left). I ended up getting a PSU that can do 2x 12V with native connectors. (Taichi 1300)

I was hoping so. I might want to try do a little bit of fine tuning for my specific applications and data sets, but its not the “core task” for me.

It did seem like it might be the case. I thought to try experimenting with offloading some extra context onto system RAM to “benchmark” how the quality would differ to me. Of course, that will be incomparably slower compared to being able to fit everything into VRAM, but maybe at least plausible as a background task.

Yeah, at the end of the day, while not cheap, R9700 are pretty entry level still for bigger models. AI is an expensive hobby. Thankfully this is just a personal project, so I think the performance here should be fine with me. At this point I think I might have maxed out what I can get from my AM5 platform at reasonable cost. (wouldnt it be nice if threadripper prices scaled up a bit less :wink: )

7

Another way to go around it, if you have enough RAM, would be to go with a MoE model, leave the shared weights loaded in VRAM and do prefill with the GPUs, but do the expert forward passes in your CPU. Most MoEs should give usable speeds this way since the experts themselves are often small enough.

1 Like
8

Just as another, much less expensive, option…if you’re using an R9700, then there’s a reasonable chance you’re using Vulkan (as I am, with llama.cpp). That brings up the possibility of using mismatched GPUs, which means that - for example - you could use a 7800 XT, which has similar raw compute and VRAM bandwidth to the R9700. That would give you the ability to run slightly larger models and/or larger context, as well as dipping your toe into the multi-GPU world without the jeopardy of an eye-watering expense that could turn out to be useless to you.

If it doesn’t give you any benefit, you’ll be able to sell pretty quickly for no loss.

9

Thank you for the ideas!

Thanks, I will look into this more! I should have about 80GB of free system RAM most of the time. (Total currently will be ~64GB VRAM and ~96GB RAM)

In terms of the current market, I felt somewhat ok buying a VRAM-heavy card for approx. MSRP price. Hopefully they will keep their value for the next few months if I change my mind.

The new GPU and PSU is here. I went with Taichi 1300W to make things simple.
It might have to wait until the weekend before I have some time to start testing some workloads. I will share my findings in the forum if I come across any interesting results.

1 Like
10

Similar boat as yours, slowly moving from an old ThreadRipper to a new one. Still ordering parts. but with the existing setup - TR 3960X + R9700 + W7800 48GB and a 1300W PSU, things are under control. I’m planning to just migrate the MB + CPU and keep the RAM (256GB UDIMM / as the current RAM prices are rather too rich for my taste) + GPUs. When I do move to 5995X I’m hoping that the PCIe lanes and octa-channel TR-Pro would be a meaningful upgrade. because of the number of PCIe lanes, upgrade option would be open with another R9700 or anything that might come later. If and when a bubble bursts, I might luck out on a TB of RAM and stick it in. With the mem bandwidth I hope to run some models/layers on the CPU as well.

looking at the most efficient high watt PSU out there right now - the SuperFlower 2800W (with 4 X 12V-2x6) which has a good Titanium rating from Cybernetics, seems to be an excellent option.

Do keep us updated on your progress, I’m sure it will benefit others here - thank you.

1 Like
11

Thanks for the reply @DanceOfLight !

I was a bit curious, the threadripper platforms you mentioned are the older generations - is it because they are more approachable price wise?
I was honestly hoping to try out some pci developments boards at some point but with the native wifi chipset on the mobo not being supported by Linux, I needed to buy another pci card for connectivity. So from now on I might have to juggle more adapters and m2 devices if further expansion will be needed. Threadripper would be a nice quality of life upgrade…

On the PSU side, do you live in a 220V country? I moved to a 110V one some time ago and it is surprising how little power you can get out of an outlet here sometimes.

1 Like
12

you are right, the older ones are cheaper in some countries the 12 and 16 core 3 and 5 series TR-Pro are being sold under 200 new. I choose this because I could still use my old DDR4 UDIMMs until the RAM prices cool down (if at all) and have a upgrade path to 1TB (realistically, the MBs claim to support to 2TB) in the future. The CPU upgrade path is also pretty nice (some boards support 7xPCIex16) which could culminate in a 5995WX with 64 core monster which for my use-cases are plenty. A 5995WX + 1TB + 2x R9700 will make very nice local inference and to a certain extent extend training machine as well. Tons of PCIe lanes would still exist if expansion is needed. The same setup is likely to munch on anything that you throw at it from the photography workflow (Linux, I have less exp on win)

Yes I live in the EU so it is indeed better WRT to power from the mains/wall. 110V is a difficult, unfortunately, but not impossible AFAIK. The coin miners have been doing it for sometime now.

13

Hopefully DDR4 RDIMMs will be cheap in the used market by then. Max you’d be able to go with UDIMMs on DDR4 is 256GB (8x32GB, which I assume is what you currently have).

14

Correct - 256GB (32GB X 8) UDIMMs

15

Did not find the time to do anything in-depth, but I was happy to see that on linux llama.cpp worked right from the start!

I tried the recently released Qwen3-Coder-Next.

$ HIP_VISIBLE_DEVICES=0,1 HSA_OVERRIDE_GFX_VERSION=12.0.0 ./build/bin/llama-server   -m ../models/Qwen3-Coder-Next-Q4_K_M.gguf   -ngl 999   --split-mode layer   -fa on   -c 300000   --port 8080   --host 0.0.0.0

ggml_cuda_init: found 2 ROCm devices:
  Device 0: AMD Radeon AI PRO R9700, gfx1200 (0x1200), VMM: no, Wave Size: 32
  Device 1: AMD Radeon AI PRO R9700, gfx1200 (0x1200), VMM: no, Wave Size: 32
main: n_parallel is set to auto, using n_parallel = 4 and kv_unified = true
build: 7936 (8abcc70a7) with GNU 15.2.1 for Linux x86_64
system info: n_threads = 16, n_threads_batch = 16, total_threads = 32

Was able to feed it a pretty big amount of data in the layer mode. (150k tokens)
It takes quite a while, like 5-6 minutes, to process the input. Looking at the GPU load you can notice that only one of them is working at a given time, so it could probably be optimized better in the row mode, although this one has been crashing for me.
The token output speeds I have seen were between 20-40 tps.

 prompt eval time =  373320.19 ms / 159686 tokens (    2.34 ms per token,   427.75 tokens per second)
       eval time =   51724.33 ms /  1222 tokens (   42.33 ms per token,    23.63 tokens per second)
      total time =  425044.51 ms / 160908 tokens

Maybe I will open another thread to share more findings from this setup. The new Qwen model is quite impressive considering its size.

If anyone has some suggestions or things they would be curious to see me try out, let me know!

2 Likes
16

Have you made a new thread yet?

A note about Qwen in my research:

There doesn’t seem to be a Qwen3.5 “Coder” model but when looking at the coding test results for Qwen3.5, they were all higher and nearly matched some frontier models. That alone was good enough for me.

Then I found Unsloth’s re-trained variants on HuggingFace. Specifically, their UD models where they dynamically increase the weights of “what really matters” that bumped several coding results even higher. Now what gets reprioritized, I can’t tell ya though.

I ask as because your usecase and system build seems very close to mine - just needing a bit more VRAM. Happy to follow along with your findings.

17

It seems, from Alibaba’s description, that Qwen 3.6 is designed to fill the same gap as the Coder variants did. They’ve certainly focused on the agentic response and tool-calling capabilities, and it beats Qwen3-Coder-Next-80B for me.

EDIT: Albeit using a different quant. I’m using Qwen 3.6 UD Q8_K_XL (36GB), versus Qwen 3 Coder Next Q4_K_M (44GB, I think).

1 Like
18

Been slogging through vllm rocm setup for 2x R9700.

gemma-4-26b-a4b-it:

Throughput: 2.95 requests/s, 3394.65 total tokens/s, 377.18 output tokens/s
Total num prompt tokens: 1024000
Total num output tokens: 128000

Need to run benchmark_moe to build targeted config enabling fused kernels. This ought to deliver 10-15% performance improvement. Prototyping that workflow on Blackwell, I noted insane performance.

Throughput: 27.67 requests/s, 31878.65 total tokens/s, 3542.07 output tokens/s
Total num prompt tokens:  1024000
Total num output tokens:  128000

My aim is Qwen3.6. Rocm side has been an adventure to say the least. I’m fortunate to have other hardware to validate against.

19

Hi!
For a while I have not been doing too much with the setup, as the project I was focusing on was a pretty deep research issue regarding a lot of obscure knowledge, so I opted for mainly working using cloud providers.

Now, with the specification and basic architecture being better defined, I could finally start using my local models more. With the very recent release of Qwen 3.6 which seems impressive, I think the local environment is more ready for use.

For the runtime, I experimented with vLLM for a bit but I did not quite like the way it performed on my machine. Firstly, somehow the load on GPU was very big even when nothing was being processed. Because my deployment did not need the benefits of vLLM, I opted to use llama.cpp in my experiments. I tried a number of models ranging from 9B to 122B. I found that the 27B options are the more reliable ones for harder questions, while the 35B model provides much better performance. I ran some benchmarks to compare different batching sizes and drivers, and generally had the best results with -b 8192 -ub 512. Other options resulted in lower GPU utilisation and performance. Situation could vary now with more up-to-date implementations and models but the Vulkan build seemed to outperform the ROCM one.

Due to some unresolved issues with my mobo one card runs at gen5 x8 while the second one runs at gen5 x4.

As an example, I ran a query with existing context running the model like so:

./build/bin/llama-server \
  -m /fstorage/models/Qwen3.6-27B-UD-Q8_K_XL.gguf \
  --host 0.0.0.0 \
  --port 8000 \
  -c 262144 \
  -ngl 99 \
  -fa 1 \
  -b 8192 \
  -ub 512 \
  --temp 0.7

Which resulted in the following performance:

VRAM usage: 26/32GB + 27/32 GB.
During pp, the utilization of both GPUs was around 80/90 percent, and the pull was 300W each.
During token output, the usage varies between 40/60 percent, and pull for both ranges in 80/110W.
Not much happening on CPU in both cases.

prompt eval time =   54253.14 ms / 72075 tokens (    0.75 ms per token,  1328.49 tokens per second)
       eval time =  319165.40 ms /  4509 tokens (   70.78 ms per token,    14.13 tokens per second)
      total time =  373418.55 ms / 76584 tokens

I would be curious to see how the model performs for you, especially if we could compare the following axis: vLLM/llama.cpp; Vulkan/ROCM; x8+x4 PCI vs a different number.

20

Well, I can answer a couple of those off the top of my head.

  • I don’t use vLLM; I have no use for it, and I swap models in and out quite a lot so llama.cpp’s new router mode is far more appropriate (not to mention far more performant when using Vulkan). Also, when I did manage to get it working, it was noticeably slower than llama.cpp + Vulkan for all the models I was interested (primarily the Qwen3+ models).
  • In all cases, for the R9700, ROCm is at least 30% slower in TG than Vulkan. PP is usually competitive-ish, but even when using -sm tensor in ROCm versus -sm layer in Vulkan, Vulkan is always significantly faster.
  • PCIE lanes don’t necessarily make a difference, but what they’re attached to really does. I originally had my cards running in a standard B760 Intel board, with one on PCIE 4.0 x16 from the CPU and the other on PCIE 4.0 x4 from the chipset; the x4 card underperformed by 25-40%, depending on the workload. I’ve since “downgraded” to a 5700X on an X570 board which can do x8/x8 split on the CPU lanes, and the cards both now perform identically to the one in the x16 slot originally. This has given a +25% boost to the dual-card performance.
1 Like