After some inspiration from @wendell low power video; I decided to get myself an alder lake core i3 12100 and some motherboards to try to get to the lowest possible idle power draw in proxmox/linux for my home server.
I think @wendell recommended the Asrock Steel Legend PCI 5.0 motherboard in that video - but I need dual x16 slots for my HBA and some other nvme so I ended up testing a few motherboards but the one that came out ahead in my testing (and that I wanted to share here) was TUF GAMING B660M-PLUS WIFI D4 at 22.5 watt idle on i3-12100 with dual nvme drives and proxmox!
If I load up my HBA SAS3008 and 8 spinning rust disks onto it the total system idle was 56 watts.
This was amazing as I am coming from a Ryzen 3700X that sitting at idle with only 4 spinning rust disks consumes 110 watts.
I forget exactly what was the lowest idle watts that @wendell achieved with the Asrock b660 steel legend but since the Asus has more slots for my use case in matx format I think I may stick with it.
Thanks for sharing the results of your endeavor with some interesting tidbits.
I have been actively managing the power consumption of my residence (bigger scope, possibly another post) including my 24x7 on server for many years. The latest configuration also consumes ~60 watts at rest (itâs never quite idle, but thatâs the avg consumption it drops down to).
The âserverâ is based on an Asrock Z170 Pro4S with an Intel i5-6500 (I said itâs a few years old) and 4x 16GB DDR4 2400 RAM sticks. It has 5x 8TB HGST spinning rust (off the mb), 1x Samsung 970 NVMe, 1x Optane 905 for zfs acceleration, an Aquantia 10G NIC.
It obviously pre-dates USB 3.2 but has a bunch of devices running off of it, including a Pulse-Eight CEC adapter so I can control it from my living room TV.
I started to look into modern options for upgrades, so your post comes timely for me.
I will now look at the USB 3.2 settings on my other mainboards to see if I can reduce the power consumption âŚ
thanks! I think this ASUS board is almost perfect and the lowest consumption vs. another MSI and gigabite B660 boards I tested. There is one issue I am having and that is related to IOMMU groups - I canât get the dual m.2 nvme slots to be in its own group in proxmox. Tried all the settings possible.
Since I need one of these M.2 nvme slots to be able to be passthru to my TrueNAS VM (and I would hate to do direct-disk passthru via proxmox host) I decided to also try a board on sale by newegg this weekend for $90 - it doesnât have PCIe 5.0 or 2.5GbE but realistically donât need those.
AsRock doesnât seem to have advanced power settings; in contrast to ASUS - there are some settings in BIOS to do âextreme power savingâ - however I did apply the same settings I did on the ASUS in terms of the CPU, PCIe, DMI power management.
Here are the IOMMU groups on the AsRock B660M steel legend (good news here is that m.2 NVME slots have their own unique groups (10 and 11):
IOMMU Group 0:
00:00.0 Host bridge [0600]: Intel Corporation Device [8086:4630] (rev 05)
IOMMU Group 1:
00:02.0 VGA compatible controller [0300]: Intel Corporation Device [8086:4692] (rev 0c)
IOMMU Group 2:
00:06.0 PCI bridge [0604]: Intel Corporation Device [8086:464d] (rev 05)
IOMMU Group 3:
00:14.0 USB controller [0c03]: Intel Corporation Device [8086:7ae0] (rev 11)
00:14.2 RAM memory [0500]: Intel Corporation Device [8086:7aa7] (rev 11)
IOMMU Group 4:
00:15.0 Serial bus controller [0c80]: Intel Corporation Device [8086:7acc] (rev 11)
IOMMU Group 5:
00:16.0 Communication controller [0780]: Intel Corporation Device [8086:7ae8] (rev 11)
IOMMU Group 6:
00:1a.0 PCI bridge [0604]: Intel Corporation Device [8086:7ac8] (rev 11)
IOMMU Group 7:
00:1c.0 PCI bridge [0604]: Intel Corporation Device [8086:7abd] (rev 11)
IOMMU Group 8:
00:1c.6 PCI bridge [0604]: Intel Corporation Device [8086:7abe] (rev 11)
IOMMU Group 9:
00:1f.0 ISA bridge [0601]: Intel Corporation Device [8086:7a86] (rev 11)
00:1f.4 SMBus [0c05]: Intel Corporation Device [8086:7aa3] (rev 11)
00:1f.5 Serial bus controller [0c80]: Intel Corporation Device [8086:7aa4] (rev 11)
IOMMU Group 10:
01:00.0 Non-Volatile memory controller [0108]: Phison Electronics Corporation E16 PCIe4 NVMe Controller [1987:5016] (rev 01)
IOMMU Group 11:
02:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM951/PM951 [144d:a802] (rev 01)
IOMMU Group 12:
03:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 02)
IOMMU Group 13:
04:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller [10ec:8125] (rev 05)
If youâre running stock you can generally get away with a -50mv offset undervolt no matter what silicon quality you have, possibly even -100mv
If you disable turbo bins youâll likely get away with even more undervolting
You can really take advantage of the behavior of frequency and voltage requirements
See the higher your frequency the higher the bump in voltage needed to sustain it is from the last bump, sort of a diminishing returns deal
Same deal with amperage but nothing we can do about that
So the lower the frequency, the higher the effecincy is, so by disabling turbo or at least lowering it, you can really undervolt it and get incredible results
Vrm switching frequency is another factor of power consumption
While higher frequency allows for better overclocks it is also more inefficient
You can usually manually set these in bios, lower frequency also has the benefit of a cooler vrm
Also undervolting makes the CPU cooler meaning lower fan speeds which is a tiny boost to power savings
thanks for the pointer. I will experiment with downvolting the CPU.
Right now I am trying to be consistent in my testing methodology, which is to only enable BIOS features that define âpower savingsâ - because I think the BIOS and motherboard is what drives most of the idle power draw and common motherboards on sale today are not focused on power savings like server boards like AsRockRack - but I am hoping thru my testing some of them come to light.
Hereâs some results for the B660M AORUS PRO AX DDR4 - lowest 24.5 watts idle (2 nvme, proxmox, âpowersaveâ cpu scaling governor, âpowertop --auto-tuneâ plus all BIOS settings except downvolting).
TL;DR: 45 watt idle with (LSI HBA + 4 x 3.5" spundown + 2 x NVME + powertop auto + âpowersaveâ scaling governor). This is the lowest idle I have been able to achieve so the board is a keeper!
By returning the 2nd best board tested B660M-PLUS WIFI D4 I am giving up the 2.5GB NIC and the PCIe 5.0 slot. But also that board is almost twice the cost ($90 vs. $160). I will likely build a new NAS server in 2 years when PCIe 5.0 is more mainstream so I rather take the power savings now and get what I really need.
Snapshot of my testing spreadsheet to help folks. I did not record some of my earlier test attempts with other boards.
After reading this post I was irked that voltage offset was locked on my 12100f & B660M-A Pro Wifi, but decided to yolo it and enable all of the settings I could to see what would do anything.
The singular setting that does anything at all for me is setting the CPU Lite Load to Mode 1 from Auto (12). This only dropped core voltage around ~25mv, but max CPU draw under all-core load dropped to 48w from 53w with identical R23 scores. (locked 4.1ghz AC and 4.3 ST)
That mixed with enabling âallâ of the the c-states has dropped total desktop idle consumption down to 40w from 50w on an 80+ bronze with 4 case fans, a spinning rust game drive , and some mild RGB.
Iâll post an update if it crashes or anything similar.
Iâm not sure I follow, I was under the impression that CPU Lite Load affected the standard vcore curve for binning purposes. Or are you referring to the C-states?
Some boards let you change the switching frequency in a menu like digi+ or digiVRM higher frequency means better overclocks lower frequency means the vrm become more effecient, kinda like 80+ ratings in PSUs
When a vrm provids say 120w of power to CPU it might use 130w +/- depending on how high/low the vrm switching frequency is
The inefficiency also turns into heat on the vrm, so lower vrm freq also means cooler vrm
In your table you show 24.5W for the PRO B660M-A DDR4. Two questions: is this the MSI PRO B660M-A DDR4 and what was the configuration with which you achieved the 24.5W? (memory, drives, PSU, etc.)
Itâs most likely an M.2 card under the IO shield. If you can get to it, you can physically remove it from the motherboard. Or get the non-wireless variant in the first place.