Improving power efficiency and configuring ASPM

Operating Systems & Open Source Linux
, ,
1

I have a Debian based Proxmox 8.0.4 server running kernel 6.2.16-12-pve and I’ve been working on improving it’s power efficiency.
I need help to optimally configure the ASPM particularly for the Intel Arc A380 and my Dell H310 HBA card (should ASPM be enabled for this device?)
When looking at the current ASPM status, some devices are still disabled. How do I enable ASPM for these devices and which ones should I keep disabled?

Specs

Summary 
CPU:  AMD Ryzen 5900x
MB: Asus ROG Strix X570-E Gaming WiFi II (4602XII Modded BIOS)
RAM:  Corsair Vengeance 2x32GB 3600MT C18 DDR4
GPU: ASRock Intel Arc A380 Challenger
PSU: Seasonic G550 Modular 550w
HBA: Dell H310 (LSI SAS2008)
Remote Access: Geekworm PiKVM-A8
Boot: 128GB Samsung 840 Pro
SSDs: 2x 1TB WD WDS100T2B0A
HDDs: 2x4TB in mirror (HGST Ultrastar); 3x10TB in RAIDZ1 (HGTST Ultrastar); 3x18TB in RIADZ1 (Seagate Exos x18)

AMD Pstate driver is enabled by configuring /etc/defaultgrub as follows:

GRUB_CMDLINE_LINUX_DEFAULT=“quiet amd_pstate=passive initcall_blacklist=acpi_cpufreq_init amd_pstate.shared_mem=1 cpufreq.default_governor=powersave”

CPU Status:

kristian@pve:~$ cpupower frequency-info
analyzing CPU 0:
  driver: amd-pstate
  CPUs which run at the same hardware frequency: 0
  CPUs which need to have their frequency coordinated by software: 0
  maximum transition latency: 20.0 us
  hardware limits: 550 MHz - 4.98 GHz
  available cpufreq governors: conservative ondemand userspace powersave performance schedutil
  current policy: frequency should be within 550 MHz and 4.98 GHz.
                  The governor "powersave" may decide which speed to use
                  within this range.
  current CPU frequency: Unable to call hardware
  current CPU frequency: 550 MHz (asserted by call to kernel)
  boost state support:
    Supported: yes
    Active: yes
    AMD PSTATE Highest Performance: 166. Maximum Frequency: 4.98 GHz.
    AMD PSTATE Nominal Performance: 130. Nominal Frequency: 3.90 GHz.
    AMD PSTATE Lowest Non-linear Performance: 61. Lowest Non-linear Frequency: 1.83 GHz.
    AMD PSTATE Lowest Performance: 20. Lowest Frequency: 550 MHz.

BIOS Settings

image
image1632×1530 292 KB

Current ASPM Status:

kristian@pve:~$ sudo dmesg | grep ASPM
[    0.988560] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI EDR HPX-Type3]
[    1.002169] pci 0000:09:00.0: ASPM: overriding L1 acceptable latency from 0x0 to 0x7
kristian@pve:~$ sudo lspci -vv | awk '/ASPM/{print $0}' RS= | grep --color -P '(^[a-z0-9:.]+|ASPM )'
00:01.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge (prog-if 00 [Normal decode])
                LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L1 <32us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge (prog-if 00 [Normal decode])
                LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L1 <64us
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
00:03.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge (prog-if 00 [Normal decode])
                LnkCap: Port #2, Speed 5GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] (prog-if 00 [Normal decode])
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk+
00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] (prog-if 00 [Normal decode])
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk+
01:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse Switch Upstream (prog-if 00 [Normal decode])
                LnkCap: Port #0, Speed 16GT/s, Width x8, ASPM L1, Exit Latency L1 <32us
                LnkCtl: ASPM L1 Enabled; Disabled- CommClk+
02:05.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge (prog-if 00 [Normal decode])
                LnkCap: Port #5, Speed 16GT/s, Width x1, ASPM L1, Exit Latency L1 <32us
                LnkCtl: ASPM Disabled; Disabled- CommClk+
02:08.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge (prog-if 00 [Normal decode])
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM L0s L1 Enabled; Disabled- CommClk+
02:09.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge (prog-if 00 [Normal decode])
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM L0s L1 Enabled; Disabled- CommClk+
02:0a.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge (prog-if 00 [Normal decode])
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM L0s L1 Enabled; Disabled- CommClk+
03:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
                LnkCap: Port #5, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 <16us
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
04:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk+
04:00.1 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller (prog-if 30 [XHCI])
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk+
04:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller (prog-if 30 [XHCI])
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk+
05:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51) (prog-if 01 [AHCI 1.0])
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk+
06:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51) (prog-if 01 [AHCI 1.0])
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk+
07:00.0 PCI bridge: Intel Corporation Device 4fa1 (rev 01) (prog-if 00 [Normal decode])
                LnkCap: Port #0, Speed 16GT/s, Width x8, ASPM L1, Exit Latency L1 <64us
                LnkCtl: ASPM Disabled; Disabled- CommClk+
08:01.0 PCI bridge: Intel Corporation Device 4fa4 (prog-if 00 [Normal decode])
                LnkCap: Port #8, Speed 2.5GT/s, Width x1, ASPM L1, Exit Latency L1 <1us
                LnkCtl: ASPM Disabled; Disabled- CommClk-
08:04.0 PCI bridge: Intel Corporation Device 4fa4 (prog-if 00 [Normal decode])
                LnkCap: Port #8, Speed 2.5GT/s, Width x1, ASPM L1, Exit Latency L1 <1us
                LnkCtl: ASPM Disabled; Disabled- CommClk+
09:00.0 VGA compatible controller: Intel Corporation DG2 [Arc A380] (rev 05) (prog-if 00 [VGA controller])
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
pcilib: sysfs_read_vpd: read failed: No such device
0a:00.0 Audio device: Intel Corporation DG2 Audio Controller
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
0b:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
                LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s, Exit Latency L0s <64ns
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
0c:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk+
0d:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk+
0d:00.1 Encryption controller: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk+
0d:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller (prog-if 30 [XHCI])
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk+
1 Like
2

I wish ASPM could have helped to save a bit more power. Unfortunately it’s within measurement errors for desktop/workstation in my experience. For laptops, every watt counts.

So don’t bother if you’re aiming more than one watt or two. Otherwise, can play with “setpci”

On “PM L1 SS”. SS stands for sub-states. Newer PCie standards define additional states to 'L1" state. Some m.2 NVMe drives, for example, support the new sub-states. Can’t remember on top of my head which does what. It’s a trade off between latency and power saving.

Curiously, what are all the options for “ErP Ready” on your motherboard?

3

That’s disappointing to hear. For the Intel Arc cards though, Intel mentions that they recommend enabling ASPM to L1 to improve idle performance. According to Tomshardware, the power savings can improve by half. My current ASPM status indicates that it is disabled for the A380 though:

09:00.0 VGA compatible controller: Intel Corporation DG2 [Arc A380] (rev 05) (prog-if 00 [VGA controller])
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-

Do you know how I can enable it?

For “ErP Ready” the options are “Disabled, Enable(S4+S5), Enable(S5)”

For the “PM L1 SS” option, I have tried “L1.1_L1.2” but is causes my GPU not to work on boot so I reverted back to “Auto”.

When using powertop, it indicates the the CPU only goes as high as C2. From what I understand, the Ryzen 5000 chips can go as high as C6. I heard there is a script called zenstates which could let me manually enable the C6 state so I could also look into that.

4

As mentioned previously, you can use “setpci” [0]. There are many scripts online to get the task done less daunting by using “setpci”. Here is one example [1] from my quick search. You need to adapt for your device id & etc.

[0] setpci(8) — Arch manual pages
[1] Force enable ASPM L0 and L1 for all devices on Dell 9360 · GitHub

1 Like
5

Thanks. Will look into the links you provided.