Threadripper 7000 Idle Wattage - TRX50 Motherboard

Hello,

I have been on this site for years, but never created an account until now. Please let me know if I am posting in the wrong forum.

I recently built a new, way overkill, Threadripper machine that is a cross between a gaming server and a workstation. I have local and remote access to the VMs it runs.
My issue is regarding the idle power usage and what I can change in the BIOS or in Proxmox to help lower it, but still get decent boost clocks for my gaming VMs. Idle wattage right now is 230 watts with 1 VM sitting at idle. I had an Epyc 7443p machine that idled at basically half of that with 6 VMs running.

I am using the amd_pstate driver in active mode and Iā€™ve played around with the Proxmox scaling governors and EPP hint, but they donā€™t seem to do much for idle power. I do see idle wattage at 180watts after rebooting with no VMs running . On boot, the VFIO driver is being used on my 7900xtx and Nouveau is used on the gt1030. Booting the VM the 7900xtx is passed through to bumps the wattage up to 230 watts despite the card getting itā€™s driver and in the guest OS and the GPUs fans spinning down. Radeon software says the GPU is at like 30 watts, but my experience with prior builds has shown the system wattage goes down after the GPU is passed through and gets itā€™s actually driver in the guest OS.

Any help would be much appreciated.

Hardware:
CPU - 7960X
Mobo - Asrock TRX50 WS
RAM - 2x32GB 6000MHz ECC (soon to be 4x32GB)
OS Drive - Crucial 250GB Sata (Proxmox, LVM)
VM OS Drives - Samsung 960GB PM9A3 x2 (Windows 11 x2 and EndeavourOS, LVM)
VM Storage Drives - Samsung 2TB 980 Pro (LVM), 1TB 860 Evo, 1TB 870 Evo
GPU1 - 7900XTX passed through to Windows 11 #1 (for gaming)
GPU2 - gt1030 passed through to Windows 11 #2 (audio production, editing)
GPU3 (currently not installed) - 7700xt passed through to EndeavourOS (for gaming)
PCIe device - Startech PEXUSB3S44V (4 USB host controllers for passthrough)

Maybe itā€™s overhead of the VM itself? What happens if you boot up a similar VM but with no passed GPU?

Overall Iā€™m not too surprised with the idle power. My 7950x box takes 100W idle, 120W with light load (like playing youtube or so). This is already down 30W from what it was recently ā€“ I think thanks to kernel upgrades. The more chiplets the higher the idle use AFAIK.

1 Like

I just booted up a second VM with the gt1030 passed through and it went from 230 watts to basically 240 watts. I can try rebooting the host and then booting one of the VMs without a GPU, but even 180 watts with no VMs running seems high to me. Like I said, I had a lot of this hardware on an Epyc 7443p platform and it idled at like 120 to 130 watts with a 6 HDDs spun up.

Think it might be possible that running the 6000MHz EXPO profile on my RAM is hogging power? I didnā€™t notice it being so high until after I enabled it and I donā€™t have the power usage logged yet on this build to go back and look.

1 Like

It is high but in line with my expectations of a 7950x pulls 100W. That is with 3 m.2ā€˜s one 3060 and a connectx-3 nic.

It might a bit. I have no registered memory but on my system there is a small difference with 6000MT vs 5200 jedec speeds. Also VSOC will be raised with expo, which will increase power a bit.

But mainly the chiplet architecture is to blame.

I just rebooted and have no VMs running. Idle power is 180 - 186 watts. Prior to rebooting, I enabled the multi-refresh rate thing in Windows so it renders at 60Hz on the 7900xtx instead 120Hz when there is nothing going on. It dropped about 12 watts with the VM running.

I really feel like something isnā€™t tuned properly here. I am not all that versed on how the C-states and stuff work, but I just noticed when running powertop that I only have C1 and C2 as states on the Idle Stats tab. Shouldnā€™t there be more than that?

Just ran the gaming VM without the 7900xtx passed through or the USB controller. Idle wattage is 330+ now.

WTH?

How is the VM configured? Is it possible itā€™s pushing the CPU into a higher power state, keeping all cores active, or pegging the frequency high?

But this does explain something IMO. The VM running is causing high power use elsewhere, so high it offsets the savings on the GPUā€¦

On my AM5 system I do see high percentage C3 states (>90% now just browsing).

Proxmox VM Config:

#hostpci0%3A 0000%3A03%3A00,pcie=1,x-vga=1,romfile=7900xtx.rom
agent: 1
balloon: 0
bios: ovmf
boot: order=scsi0
cores: 12
cpu: host
efidisk0: PM9A3_960GB2:vm-103-disk-1,efitype=4m,pre-enrolled-keys=1,size=4M
machine: pc-q35-9.0
memory: 24576
meta: creation-qemu=9.0.2,ctime=1732748150
name: Win11-AMD
net0: virtio= ,bridge=vmbr1,firewall=1
numa: 0
ostype: win11
scsi0: PM9A3_960GB2:vm-103-disk-0,discard=on,iothread=1,size=128G,ssd=1
scsi1: Samsung_2TB:vm-103-disk-0,backup=0,discard=on,iothread=1,size=752G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=dd843db3-8e8f-49d3-93d6-ba6c1a615287
sockets: 1
tpmstate0: PM9A3_960GB2:vm-103-disk-2,size=4M,version=v2.0
vga: none
vmgenid: 88de766e-1a39-451f-b252-36c28f6e4c12

I donā€™t know what is different now, but the one VM is pulling 300 watts at idle with the 7900xtx added back in. Note, the above config was prior to adding it and the USB controller back in, but the wattage was super high both ways. The Windows guest is in Balanced power mode and it appears that the GPU is clocking down. Task manager shows very little usage with Steam being the main thing using RAM, but next to no CPU usage.

Check on the host. What does cpupower frequency-info say before/after launching the VM? s-tui? powertop?

See below. Note that powertop is just showing the top ~10 rows.

VM Running

root@pve3:~# cpupower frequency-info
analyzing CPU 0:
driver: amd-pstate-epp
CPUs which run at the same hardware frequency: 0
CPUs which need to have their frequency coordinated by software: 0
maximum transition latency: Cannot determine or is not supported.
hardware limits: 545 MHz - 5.67 GHz
available cpufreq governors: performance powersave
current policy: frequency should be within 545 MHz and 545 MHz.
The governor ā€œperformanceā€ may decide which speed to use
within this range.
current CPU frequency: Unable to call hardware
current CPU frequency: 545 MHz (asserted by call to kernel)
boost state support:
Supported: yes
Active: yes
Boost States: 0
Total States: 3
Pstate-P0: 4200MHz
Pstate-P1: 2200MHz
Pstate-P2: 1500MHz

powertop

Summary: 3936.6 wakeups/second, 0.0 GPU ops/seconds, 0.0 VFS ops/sec and 8.4% CPU use

            Usage       Events/s    Category       Description
         10.4 ms/s     1270.2       Timer          stimer_timer_callback
          1.2 ms/s     377.2        Timer          tick_nohz_highres_handler
         10.2 ms/s     295.0        Process        [PID 12345] /usr/bin/kvm -id 103 -name Win11-AMD,debug-threads=on -no-shutdown -chardev sock
          6.5 ms/s     245.4        Process        [PID 12344] /usr/bin/kvm -id 103 -name Win11-AMD,debug-threads=on -no-shutdown -chardev sock
          7.3 ms/s     226.1        Process        [PID 12346] /usr/bin/kvm -id 103 -name Win11-AMD,debug-threads=on -no-shutdown -chardev sock
          4.3 ms/s     182.1        Process        [PID 12336] /usr/bin/kvm -id 103 -name Win11-AMD,debug-threads=on -no-shutdown -chardev sock
          3.8 ms/s     154.0        Process        [PID 12347] /usr/bin/kvm -id 103 -name Win11-AMD,debug-threads=on -no-shutdown -chardev sock
          2.2 ms/s     141.8        Process        [PID 12337] /usr/bin/kvm -id 103 -name Win11-AMD,debug-threads=on -no-shutdown -chardev sock
          3.1 ms/s     138.4        Process        [PID 12339] /usr/bin/kvm -id 103 -name Win11-AMD,debug-threads=on -no-shutdown -chardev sock
          2.9 ms/s     131.4        Process        [PID 12338] /usr/bin/kvm -id 103 -name Win11-AMD,debug-threads=on -no-shutdown -chardev sock
          4.6 ms/s     129.1        Process        [PID 12341] /usr/bin/kvm -id 103 -name Win11-AMD,debug-threads=on -no-shutdown -chardev sock
          3.6 ms/s     128.1        Process        [PID 12342] /usr/bin/kvm -id 103 -name Win11-AMD,debug-threads=on -no-shutdown -chardev sock
          2.3 ms/s     128.4        Process        [PID 12340] /usr/bin/kvm -id 103 -name Win11-AMD,debug-threads=on -no-shutdown -chardev sock
          2.8 ms/s     128.0        Process        [PID 12343] /usr/bin/kvm -id 103 -name Win11-AMD,debug-threads=on -no-shutdown -chardev sock
          2.0 ms/s      81.3        Interrupt      [7] sched(softirq)

Every 2.0s: grep ā€œcpu MHzā€ /proc/cpuinfo pve3: Thu Dec 19 14:08:09 2024

cpu MHz : 545.000
cpu MHz : 4799.856
cpu MHz : 4785.872
cpu MHz : 545.000
cpu MHz : 545.000
cpu MHz : 545.000
cpu MHz : 4794.211
cpu MHz : 545.000
cpu MHz : 545.000
cpu MHz : 545.000
cpu MHz : 545.000
cpu MHz : 545.000
cpu MHz : 4755.990
cpu MHz : 545.000
cpu MHz : 545.000
cpu MHz : 545.000
cpu MHz : 4694.330
cpu MHz : 4767.088
cpu MHz : 4771.754
cpu MHz : 4779.843
cpu MHz : 545.000
cpu MHz : 4785.342
cpu MHz : 545.000
cpu MHz : 545.000
cpu MHz : 545.000
cpu MHz : 545.000
cpu MHz : 545.000
cpu MHz : 545.000
cpu MHz : 4799.805
cpu MHz : 545.000
cpu MHz : 545.000
cpu MHz : 545.000
cpu MHz : 4796.127
cpu MHz : 4790.330
cpu MHz : 4798.965
cpu MHz : 545.000
cpu MHz : 545.000
cpu MHz : 4783.829
cpu MHz : 4781.904
cpu MHz : 4776.916
cpu MHz : 545.000
cpu MHz : 4762.992
cpu MHz : 4780.768
cpu MHz : 4773.240
cpu MHz : 545.000
cpu MHz : 4785.842
cpu MHz : 545.000
cpu MHz : 545.000

VM not running

root@pve3:~# cpupower frequency-info
analyzing CPU 0:
driver: amd-pstate-epp
CPUs which run at the same hardware frequency: 0
CPUs which need to have their frequency coordinated by software: 0
maximum transition latency: Cannot determine or is not supported.
hardware limits: 545 MHz - 5.67 GHz
available cpufreq governors: performance powersave
current policy: frequency should be within 545 MHz and 545 MHz.
The governor ā€œperformanceā€ may decide which speed to use
within this range.
current CPU frequency: Unable to call hardware
current CPU frequency: 545 MHz (asserted by call to kernel)
boost state support:
Supported: yes
Active: yes
Boost States: 0
Total States: 3
Pstate-P0: 4200MHz
Pstate-P1: 2200MHz
Pstate-P2: 1500MHz

powertop

Summary: 207.1 wakeups/second, 0.0 GPU ops/seconds, 0.0 VFS ops/sec and 5.4% CPU use

            Usage       Events/s    Category       Description
        292.0 Āµs/s      45.2        Timer          watchdog_timer_fn
        340.6 Āµs/s      41.4        Timer          tick_nohz_highres_handler
         45.4 ms/s       1.9        Process        [PID 44747] powertop
        508.1 Āµs/s      11.3        Process        [PID 1841] pve-ha-lrm
        102.9 Āµs/s      11.3        Process        [PID 17] [rcu_preempt]
        100.0 Āµs/s       9.4        kWork          psi_avgs_work
        461.6 Āµs/s       6.6        Interrupt      [7] sched(softirq)
        135.3 Āµs/s       5.6        Process        [PID 1776] /usr/bin/pmxcfs
         75.9 Āµs/s       5.6        Interrupt      [3] net_rx(softirq)
        311.7 Āµs/s       4.7        Process        [PID 1778] /usr/bin/pmxcfs
        167.2 Āµs/s       4.7        Process        [PID 1830] /usr/bin/pmxcfs
         67.5 Āµs/s       4.7        kWork          fb_flashcursor
         26.9 Āµs/s       4.7        Interrupt      [77] ahci[0000:4a:00.0]
        166.6 Āµs/s       3.8        Process        [PID 1777] /usr/bin/pmxcfs
         69.4 Āµs/s       3.8        Process        [PID 1426] /usr/sbin/pvefw-logger
          1.8 ms/s       2.8        Process        [PID 19170] pveproxy worker
         89.5 Āµs/s       2.8        Process        [PID 804] [jbd2/dm-1-8]
         46.6 Āµs/s       2.8        Process        [PID 1915] /usr/bin/rrdcached -B -b /var/lib/rrdcached/db/ -j /var/lib/rrdcached/journal/ -p
         11.7 Āµs/s       2.8        kWork          blk_mq_requeue_work

Every 2.0s: grep ā€œcpu MHzā€ /proc/cpuinfo pve3: Thu Dec 19 14:06:31 2024

cpu MHz : 545.000
cpu MHz : 545.000
cpu MHz : 545.000
cpu MHz : 545.000
cpu MHz : 545.000
cpu MHz : 545.000
cpu MHz : 545.000
cpu MHz : 545.000
cpu MHz : 545.000
cpu MHz : 545.000
cpu MHz : 545.000
cpu MHz : 545.000
cpu MHz : 545.000
cpu MHz : 545.000
cpu MHz : 545.000
cpu MHz : 2599.967
cpu MHz : 545.000
cpu MHz : 545.000
cpu MHz : 2597.811
cpu MHz : 545.000
cpu MHz : 545.000
cpu MHz : 545.000
cpu MHz : 545.000
cpu MHz : 545.000
cpu MHz : 545.000
cpu MHz : 545.000
cpu MHz : 545.000
cpu MHz : 545.000
cpu MHz : 545.000
cpu MHz : 2597.146
cpu MHz : 545.000
cpu MHz : 545.000
cpu MHz : 545.000
cpu MHz : 545.000
cpu MHz : 545.000
cpu MHz : 545.000
cpu MHz : 545.000
cpu MHz : 545.000
cpu MHz : 545.000
cpu MHz : 545.000
cpu MHz : 545.000
cpu MHz : 545.000
cpu MHz : 545.000
cpu MHz : 545.000
cpu MHz : 545.000
cpu MHz : 2666.895
cpu MHz : 545.000
cpu MHz : 545.000

Yeah, clearly running the VM is making the CPU run at high frequencies more often. I see this on my system to (120 ā†’ 170W). Switching to the powersave governor may help a bit, cpupower frequency-set -g powersave. Maybe someone with a TRX50 system can chime in but compared to my 7950X system, I think your power use makes sense. With a VM running I get 170W, you get 230W but on a 4 CCD vs. 2 CCD CPU. It is what it is. Idle power is the achilles heel of AMDs current architecture.

Damn, I really hope I can find a way to drop it down. Iā€™m already on the powersave governor.

I am just doing things remotely right now, but I am going to poke around the BIOS tonight to see if anything looks off. I did a BIOS update yesterday, which seemed to set a lot of stuff to default. I went back through and changed the virtualization stuff, along with enabling EXPO, which I never had enabled prior to the BIOS update. Iā€™m wondering if I left C-States disabled or something 'cause I swear it was idling lower than this a few days ago.

FWIW, I briefly had a 7950X and Supermicro H13SAE-MF board prior to this build, but it was using the iGPU in place of the gt1030. I had the 7700xt and 7900xtx both passed through at PCIE4 x8 each. That was idling at less than 100 watts, so you might still have some a little efficiency on the table. I never ran R23 on the host, but 6c12t in the Windows VM would still hit 5.4Ghz and was getting a 20k score, so it was doing well at idle and load with all the powersave stuff set up and a lower ppt set in the BIOS.

I was getting frustrated trying to make things work with not enough PCIe lanes on AM5, so I requested a return and went with Threadripper.

Yeah I could optimize power use for sure. Mostly I have 2 crappy Kingston NV2 SSDs that seem to not support ASPM and have high power use in idle. Plus a pretty old connect-x 3 NIC which does support ASPM but is probably still pretty inefficient. Just one device without ASPM will keep the CPU from idling very deep AFAIK.

Your cpupower output shows governor performance though?

Ah, yep, it does. I have a crontab that sets it to powersave at boot. I have been messing with it and mustā€™ve forgot to swap back. Changing it back did drop the wattage by 20 watts, which is about what Iā€™ve seen on multiple platforms. For some reason, after booting the VM with the GPU removed, though, it is no longer at 230 watts when the VM was running. It was at like 300 prior to powersave mode and 280 afterwards. That is with or without the GPU passed through. I rebooted Proxmox and itā€™s still doing the same thing. I just really feel like something is wonky.

Anyway, thanks for your help.

1 Like

Ive recently upgraded to same hardware as you are, except I have 128Gb ram and 4090. I donā€™t run Proxmox on it, just Windows 11. I can try to measure idle power usage on mine if you tell me how you measure it ? Do you have an energy power meter in the socket from where you connect to the PC ?

I have home assistant running on one of my servers with an MQTT broker that can bring in data from Zigbee devices. The Threadripper machine is plugged into a smart Zigbee outlet, which sends power stats back to Home Assistant for viewing and trending.

If youā€™re on Windows, you may be able to just look at package power in software like HwInfo, but yeah a smart outlet or something works really well for machines that are on 24/7. Thatā€™s why I am so concerned with power usage.

On windows itā€™s pretty easy to see package power use with e.g. HWInfo.

On linux you can see this with sensors or s-tui. In absence of measuring at the outlet this would at least be some comparison point.

Youā€™re able to get wattage with sensors?

Iā€™ve only ever been able to get some thermals and even then, not on all devices. Thatā€™s across multiple boards, both AMD and Intel. I havenā€™t even been able to get fan speeds or DIMM temps.

hmm, scratch that. It only shows amdgpu PPT on my system. But s-tui does.

Is this correct sensor from HWinfo ?