In the pursuit of low-power consumption NAS with alder lake (12th gen)

After some inspiration from @wendell low power video; I decided to get myself an alder lake core i3 12100 and some motherboards to try to get to the lowest possible idle power draw in proxmox/linux for my home server.

I think @wendell recommended the Asrock Steel Legend PCI 5.0 motherboard in that video - but I need dual x16 slots for my HBA and some other nvme so I ended up testing a few motherboards but the one that came out ahead in my testing (and that I wanted to share here) was TUF GAMING B660M-PLUS WIFI D4 at 22.5 watt idle on i3-12100 with dual nvme drives and proxmox!

If I load up my HBA SAS3008 and 8 spinning rust disks onto it the total system idle was 56 watts.

This was amazing as I am coming from a Ryzen 3700X that sitting at idle with only 4 spinning rust disks consumes 110 watts.

I forget exactly what was the lowest idle watts that @wendell achieved with the Asrock b660 steel legend but since the Asus has more slots for my use case in matx format I think I may stick with it.

Also in order to achieve this low power consumption you must disable:

  • audio controller
  • USB 3.2
  • WIFI and bluetooth
  • enable all power saving features in BIOS (should be a given but being explicit)

If you definetely want to have USB 3.2 enabled in BIOS this will add 4-6 watts 24/7 to usage. I don’t need that for my use case.

3 Likes

Nooiiice!

Thanks for sharing the results of your endeavor with some interesting tidbits.

I have been actively managing the power consumption of my residence (bigger scope, possibly another post) including my 24x7 on server for many years. The latest configuration also consumes ~60 watts at rest (it’s never quite idle, but that’s the avg consumption it drops down to).

The “server” is based on an Asrock Z170 Pro4S with an Intel i5-6500 (I said it’s a few years old) and 4x 16GB DDR4 2400 RAM sticks. It has 5x 8TB HGST spinning rust (off the mb), 1x Samsung 970 NVMe, 1x Optane 905 for zfs acceleration, an Aquantia 10G NIC.
It obviously pre-dates USB 3.2 but has a bunch of devices running off of it, including a Pulse-Eight CEC adapter so I can control it from my living room TV.

I started to look into modern options for upgrades, so your post comes timely for me.

I will now look at the USB 3.2 settings on my other mainboards to see if I can reduce the power consumption …

2 Likes

thanks! I think this ASUS board is almost perfect and the lowest consumption vs. another MSI and gigabite B660 boards I tested. There is one issue I am having and that is related to IOMMU groups - I can’t get the dual m.2 nvme slots to be in its own group in proxmox. Tried all the settings possible.

Since I need one of these M.2 nvme slots to be able to be passthru to my TrueNAS VM (and I would hate to do direct-disk passthru via proxmox host) I decided to also try a board on sale by newegg this weekend for $90 - it doesn’t have PCIe 5.0 or 2.5GbE but realistically don’t need those.

Here’s the board that’s onsale, depending on test outcome (if its lower power and no IOMMU issues compared to the ASUS I may keep): ASRock B660M PRO RS LGA 1700 Intel B660 SATA 6Gb/s DDR4 Micro ATX Motherboard - Newegg.com

will report next weekend once it arrives.

1 Like

This is same CPU, two NVMEs on B660 AsRock Steel Legend.

The lowest idle power draw I was able to get with this board 28 watts. ASUS still beats it.

AsRock doesn’t seem to have advanced power settings; in contrast to ASUS - there are some settings in BIOS to do ‘extreme power saving’ - however I did apply the same settings I did on the ASUS in terms of the CPU, PCIe, DMI power management.

Here are the IOMMU groups on the AsRock B660M steel legend (good news here is that m.2 NVME slots have their own unique groups (10 and 11):

IOMMU Group 0:
        00:00.0 Host bridge [0600]: Intel Corporation Device [8086:4630] (rev 05)
IOMMU Group 1:
        00:02.0 VGA compatible controller [0300]: Intel Corporation Device [8086:4692] (rev 0c)
IOMMU Group 2:
        00:06.0 PCI bridge [0604]: Intel Corporation Device [8086:464d] (rev 05)
IOMMU Group 3:
        00:14.0 USB controller [0c03]: Intel Corporation Device [8086:7ae0] (rev 11)
        00:14.2 RAM memory [0500]: Intel Corporation Device [8086:7aa7] (rev 11)
IOMMU Group 4:
        00:15.0 Serial bus controller [0c80]: Intel Corporation Device [8086:7acc] (rev 11)
IOMMU Group 5:
        00:16.0 Communication controller [0780]: Intel Corporation Device [8086:7ae8] (rev 11)
IOMMU Group 6:
        00:1a.0 PCI bridge [0604]: Intel Corporation Device [8086:7ac8] (rev 11)
IOMMU Group 7:
        00:1c.0 PCI bridge [0604]: Intel Corporation Device [8086:7abd] (rev 11)
IOMMU Group 8:
        00:1c.6 PCI bridge [0604]: Intel Corporation Device [8086:7abe] (rev 11)
IOMMU Group 9:
        00:1f.0 ISA bridge [0601]: Intel Corporation Device [8086:7a86] (rev 11)
        00:1f.4 SMBus [0c05]: Intel Corporation Device [8086:7aa3] (rev 11)
        00:1f.5 Serial bus controller [0c80]: Intel Corporation Device [8086:7aa4] (rev 11)
IOMMU Group 10:
        01:00.0 Non-Volatile memory controller [0108]: Phison Electronics Corporation E16 PCIe4 NVMe Controller [1987:5016] (rev 01)
IOMMU Group 11:
        02:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM951/PM951 [144d:a802] (rev 01)
IOMMU Group 12:
        03:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 02)
IOMMU Group 13:
        04:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller [10ec:8125] (rev 05)
1 Like

If you’re running stock you can generally get away with a -50mv offset undervolt no matter what silicon quality you have, possibly even -100mv

If you disable turbo bins you’ll likely get away with even more undervolting

You can really take advantage of the behavior of frequency and voltage requirements

See the higher your frequency the higher the bump in voltage needed to sustain it is from the last bump, sort of a diminishing returns deal

Same deal with amperage but nothing we can do about that

So the lower the frequency, the higher the effecincy is, so by disabling turbo or at least lowering it, you can really undervolt it and get incredible results

Vrm switching frequency is another factor of power consumption
While higher frequency allows for better overclocks it is also more inefficient

You can usually manually set these in bios, lower frequency also has the benefit of a cooler vrm

Also undervolting makes the CPU cooler meaning lower fan speeds which is a tiny boost to power savings

3 Likes

thanks for the pointer. I will experiment with downvolting the CPU.

Right now I am trying to be consistent in my testing methodology, which is to only enable BIOS features that define ‘power savings’ - because I think the BIOS and motherboard is what drives most of the idle power draw and common motherboards on sale today are not focused on power savings like server boards like AsRockRack - but I am hoping thru my testing some of them come to light.

Here’s some results for the B660M AORUS PRO AX DDR4 - lowest 24.5 watts idle (2 nvme, proxmox, ‘powersave’ cpu scaling governor, ‘powertop --auto-tune’ plus all BIOS settings except downvolting).

This board is in the same league as the ASUS from the start, two things I noticed in the AORUS BIOS were:

  • you can’t disable WIFI/Bluetooth (likely costing some 1-2watt idle efficiency)
  • can’t disable USB 3.2 controller
  • disabling built-in soundcard does not seem to do anything, linux still sees the device when it should be off.
  • IOMMU groups are not an issue and well separated m.2 slots.
IOMMU Group 0:
        00:00.0 Host bridge [0600]: Intel Corporation Device [8086:4630] (rev 05)
IOMMU Group 1:
        00:02.0 VGA compatible controller [0300]: Intel Corporation Alder Lake-S GT1 [UHD Graphics 730] [8086:4692] (rev 0c)
IOMMU Group 2:
        00:06.0 PCI bridge [0604]: Intel Corporation 12th Gen Core Processor PCI Express x4 Controller #0 [8086:464d] (rev 05)
IOMMU Group 3:
        00:0a.0 Signal processing controller [1180]: Intel Corporation Platform Monitoring Technology [8086:467d] (rev 01)
IOMMU Group 4:
        00:14.0 USB controller [0c03]: Intel Corporation Alder Lake-S PCH USB 3.2 Gen 2x2 XHCI Controller [8086:7ae0] (rev 11)
        00:14.2 RAM memory [0500]: Intel Corporation Alder Lake-S PCH Shared SRAM [8086:7aa7] (rev 11)
IOMMU Group 5:
        00:14.3 Network controller [0280]: Intel Corporation Alder Lake-S PCH CNVi WiFi [8086:7af0] (rev 11)
IOMMU Group 6:
        00:15.0 Serial bus controller [0c80]: Intel Corporation Alder Lake-S PCH Serial IO I2C Controller #0 [8086:7acc] (rev 11)
        00:15.1 Serial bus controller [0c80]: Intel Corporation Alder Lake-S PCH Serial IO I2C Controller #1 [8086:7acd] (rev 11)
        00:15.2 Serial bus controller [0c80]: Intel Corporation Alder Lake-S PCH Serial IO I2C Controller #2 [8086:7ace] (rev 11)
        00:15.3 Serial bus controller [0c80]: Intel Corporation Alder Lake-S PCH Serial IO I2C Controller #3 [8086:7acf] (rev 11)
IOMMU Group 7:
        00:16.0 Communication controller [0780]: Intel Corporation Alder Lake-S PCH HECI Controller #1 [8086:7ae8] (rev 11)
IOMMU Group 8:
        00:19.0 Serial bus controller [0c80]: Intel Corporation Alder Lake-S PCH Serial IO I2C Controller #4 [8086:7afc] (rev 11)
        00:19.1 Serial bus controller [0c80]: Intel Corporation Alder Lake-S PCH Serial IO I2C Controller #5 [8086:7afd] (rev 11)
IOMMU Group 9:
        00:1a.0 PCI bridge [0604]: Intel Corporation Device [8086:7ac8] (rev 11)
IOMMU Group 10:
        00:1c.0 PCI bridge [0604]: Intel Corporation Alder Lake-S PCH PCI Express Root Port #1 [8086:7ab8] (rev 11)
IOMMU Group 11:
        00:1f.0 ISA bridge [0601]: Intel Corporation Device [8086:7a86] (rev 11)
        00:1f.3 Audio device [0403]: Intel Corporation Alder Lake-S HD Audio Controller [8086:7ad0] (rev 11)
        00:1f.4 SMBus [0c05]: Intel Corporation Alder Lake-S PCH SMBus Controller [8086:7aa3] (rev 11)
        00:1f.5 Serial bus controller [0c80]: Intel Corporation Alder Lake-S PCH SPI Controller [8086:7aa4] (rev 11)
IOMMU Group 12:
        01:00.0 Non-Volatile memory controller [0108]: Phison Electronics Corporation E16 PCIe4 NVMe Controller [1987:5016] (rev 01)
IOMMU Group 13:
        02:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM951/PM951 [144d:a802] (rev 01)
IOMMU Group 14:
        03:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I225-V [8086:15f3] (rev 03)

1 Like

I received my $90 ASRock B660M PRO RS today. Time for sharing some results.

TL;DR: 45 watt idle with (LSI HBA + 4 x 3.5" spundown + 2 x NVME + powertop auto + ‘powersave’ scaling governor). This is the lowest idle I have been able to achieve so the board is a keeper!

By returning the 2nd best board tested B660M-PLUS WIFI D4 I am giving up the 2.5GB NIC and the PCIe 5.0 slot. But also that board is almost twice the cost ($90 vs. $160). I will likely build a new NAS server in 2 years when PCIe 5.0 is more mainstream so I rather take the power savings now and get what I really need.

Snapshot of my testing spreadsheet to help folks. I did not record some of my earlier test attempts with other boards.

After reading this post I was irked that voltage offset was locked on my 12100f & B660M-A Pro Wifi, but decided to yolo it and enable all of the settings I could to see what would do anything.

The singular setting that does anything at all for me is setting the CPU Lite Load to Mode 1 from Auto (12). This only dropped core voltage around ~25mv, but max CPU draw under all-core load dropped to 48w from 53w with identical R23 scores. (locked 4.1ghz AC and 4.3 ST)

That mixed with enabling “all” of the the c-states has dropped total desktop idle consumption down to 40w from 50w on an 80+ bronze with 4 case fans, a spinning rust game drive , and some mild RGB.

I’ll post an update if it crashes or anything similar.

The switching frequency of VRMs won’t show power savings on the info it’s more of a kill-a-watt thing

I’m not sure I follow, I was under the impression that CPU Lite Load affected the standard vcore curve for binning purposes. Or are you referring to the C-states?

Some boards let you change the switching frequency in a menu like digi+ or digiVRM higher frequency means better overclocks lower frequency means the vrm become more effecient, kinda like 80+ ratings in PSUs

When a vrm provids say 120w of power to CPU it might use 130w +/- depending on how high/low the vrm switching frequency is

The inefficiency also turns into heat on the vrm, so lower vrm freq also means cooler vrm

1 Like

I didn’t notice any settings like than in MSI’s UEFI, but I also wasn’t looking for it either. That all makes sense though.

In your table you show 24.5W for the PRO B660M-A DDR4. Two questions: is this the MSI PRO B660M-A DDR4 and what was the configuration with which you achieved the 24.5W? (memory, drives, PSU, etc.)

It’s most likely an M.2 card under the IO shield. If you can get to it, you can physically remove it from the motherboard. Or get the non-wireless variant in the first place.