I would need some advice for the fan layout and configuration for one of our self-made workstations. System specs in brief:
Case: Corsair 7000D AIRFLOW, 4x 140 mm fans in the front, 3x 140 mm fans on the top, 4x 120 mm fans on the side, 1x 140 mm fan in the read. All of them are corsair fans but I need to check which model.
Motherboard: Pro WS WRX90E-SAGE SE
CPU & RAM: AMD Ryzen Threadripper 7960X @stock 350W and 8x 64 GB ( Kingston Server 64GB DDR5 RDIMM 5600MHz CL46 ECC) running at 5600MHz
CPU cooler: Noctua DX series NH-U14S DX-4677
Storage: 4x Corsair MP700 Pro SSD 2TB installed on the motherboard
We are using the system for mixed scientific computational workloads with GPU acceleration. Usually the GPUs use around 300W, but we have workloads that push both at them same time to 450W. This creates naturally some cooling challenges, especially for the CPU. Both GPUs run under full load usually around 65°C, however, just the GPU load can increase the CPU, and I assume also RAM temperature, by around 20°C since the RTX4090 blow directly towards CPU and RAM. That means under CPU only loads we reach temperatures of around 75°C and when pushing both GPU and CPU we run into slight thermal throttling (base frequency, no boosting anymore).
Our attempt to counteract this issue was to run all fans at full speed and to install as many fans as possible in the following configuration:
Front and side intake
Top and back out
However, I am not sure if this is the smartest configuration and I would be happy if anyone could recommend me something? I was thinking of two top fans in and only one out?
Or is this a “hopeless” case without water-cooling?
Yes, assuming a standard front to back airflow setup top mid-front intake is what’ll help with mixing down dGPU passthrough. With ~300 W of dGPU I’ve seen only minor differences but with ~900 W something meaningful wouldn’t surprise me. Running the case rear to front, rather than front to rear, may be more effective if dust filtration isn’t much of a consideration.
When I’ve measured top rear exhaust it’s typically been mildly detrimental, lowering VRM temperatures slightly at the expense of upper M.2 increases. Very popular to assume it makes a big improvement in ATX desktops, though, and you’ll also find a lot of people insisting in and out next to each other just produces useless recirculation but I haven’t particularly seen that in measurement. In this config the main question I’d have is whether top rear intake drops RDIMM temperatures any, though probably the hot ones are up against the top dGPU and probably trying to get crossflow past the U14S won’t be very effective.
In some situations both adding more fans and spinning them up increase temperatures compared to fewer fans or slower spin. So question those assumptions as well. Depends on the fans and thermal load but often I don’t see temperatures drop much past 45-50% PWM.
Even by Noctua’s usually delusionally optimistic compatibility marketing ratings the U14S is only barely adequate. Throttling onset near 75 °C sounds odd, though, as Zen 4 CCDs are rated for 95. Soft throttle starting from 70 °C was a Zen 2 thing removed in Zen 3. Makes me wonder if maybe it’s actually RDIMM temperatures backing up into the cores or just something going on with the workload.
That said, if you’re not already doing so, look at per core power and temperature distribution rather than headline monitoring numbers. Throttling’s a function of power concentration on active CCDs, not the average across all four.
Sorry, I was not clear: with CPU only load, the CPU gets only around 75°C on the hottest core at 350W default (no PBO). The delta between the cores is around 10°C, which should be fine. Only once the GPU kicks in and blows hot air into the CPU, it reaches up to 95°C and as a result mild throtteling.
I give it a shot, let’s see if that improves the temperature.
The workstation is not in a dedicated server room with lots of dust around, I will try this only as a last resort.
Okey good to know! I assume more turbulence and less laminar flow. Even though I think with the two 4090s there is not much laminar flow once they start ramping up. The fans on the side improved the NVMe Gen5 temperatures a lot, I assume by cooling the motherboard.
I guess I will just have to test around a bit, and the lesson learned next time we will opt for blower fan designs again. Noise is not much of an issue.
Thanks a lot I will keep testing, luckily with IPMI setting fan speeds is quite easy on the fly.
It’s all turbulent. I don’t know of sTR5 data but with AM5 there’s always workloads that’ll concentrate sufficient power into a CCD to hit 95 °C under the IHS. The advantage of AIOs is it’s easier to hold to +60 °C ΔT or so with fairly well balanced all core loads at dGPU idle. But there’s little sTR5 data there either.
Yeah, wouldn’t expect Nvidia to stop being thermally stupid about transverse fins. It’s an inexpensive way of fitting a single dGPU for desktop gaming but I can’t say I’m enthusiastic about use of cost down design to hold up a ~75% overall profit margin at US$ 2000+ price points.