ASRock Rack W680D4ID-2T/G5/X550 C-states / ASPM problem

Hardware Hub Motherboards
,
21

Hi all,

Let me join the party. :smiley:

I read through a lot of posts on dozens of forums related to power consumption, and regardless of an ASRock board’s model, they all have the very same issue. It’s kind of a sustainable fact: ASRock = energy inefficiency.

I have a consumer board: ASRock Z690M-ITX/AX. I coupled it with Intel i9-13900K, DDR4 3200, WD SN700 4GB, and PicoPSU 150W. Mainboard has RTL8125B and i226-V network interfaces which is important to know. I got my board back in December and have been tinkering with it since then.

I started my journey with PC3 and 15W. Obviously, all ASPM and power-saving features were enabled in the BIOS. I also disabled turbo and hyper-threading, as I need a low-power setup with an on-demand powerful CPU for occasionally intensive tasks.

I brought it down to 9W from 15W and to PC6 from PC3 by only doing the following:

  • Enable “Low Power S0 Idle Capability” in hidden BIOS settings using AMI’s SCELNX_64 tool
  • Disable RTL8125B in BIOS as it has a infamous ASPM bug so Linux’s driver disables ASPM by default.
  • Locking RAM at 2100MHz

One interesting observation that seconds others’ findings: PCIe slots attached to the CPU cause problems with package states. This is common to all 12-14th gen CPUs, regardless of the board vendor. For instance, my system reaches only PC6 when I use slots connected to the CPU, and I easily get C10 if I use only PCH-connected slots.

I also tried other things that did NOT contribute to the results:

  • Playing around with MSR_PKG_CST_CONFIG_CONTROL register by writing C-state and package (un)demotion
  • Disabling Multi-VC
  • Playing around with S0ix substates in BIOS hidden settings
  • Patching intel_idle driver to support C-states directly rather than through ACPI
  • Some other things I already don’t remember

I came across this thread once I tried to run Intel’s S0ixSelftestTool and it finished with an error that my PCIe bridge doesn’t support L1.1 and L1.2 substates. I started digging in this direction, and from reading Linux sources and other things, I got an understanding that this could be due to LTR and CLKREQ issues in PCIe. That’s how I found this thread.

All in all, it’s very likely that all ASRock boards do an improper configuration of PCIe, and this thread has moved far beyond any other one in understanding the root cause.

1 Like
22

I still think ASRock f*ked up with a PCIe configuration since PC10 shall do the difference to PC6 in terms of power consumption. I had Gigabyte H610I Mini-ITX board in my hands and with the very same components (nvme, ram, psu) the board easily reached PC10 and 4W, so PC6 vs PC10 does no difference only on ASRock boards.

23

Yes, for sure. In my opinion almost all vendors don’t do proper tests for low power consumption scenarios. In my desktop I have ASUS X670E-I which costs around 400 dollars, which is pretty premium I would say. But I cannot even achieve PC3. It has hidden Modern Standby toggle in BIOS, but it does nothing. I think they just cut this functionality entirely making it exclusively for laptops.

24

Though I would say that in my case I tested with Intel A380 (which is not very efficient in idle and even with ASPM enabled draws around 15W) and with 2 server NVMe drives which don’t support any energy saving states at all. And don’t forget my board has IPMI which could draw around another 4W constantly.

In the end I am kinda ok with that result. Though of course I am disappointed that after spending so much time I basically just figured out that most of the problems are with the board itself and they cannot be solved without vendor participation and probably some hardware fixes.

Oh, btw after upgrading to Fedora CoreOS 42 which runs most of my apps power draw dropped to 45-50W average for some unknown reason (fixes in Linux 6.14?). I run 46 containers for now and three VMs. So for me that’s totally fine.