Hello folks,
After watching wendell videos for home servers I decided to build my TrueNas but I’m having a problem with the network cards.
Below is the config:
- Core i5 12600K
- 64Gb
- Z690 Aorus Elite
- 4x 12Tb Seagate Exos x16
- 4x 1tb Nvme
- 2x 128Gb SSD (boot)
- 5 fans + 2 fans on the CPU Cooler
After configuring ASPM in the BIOS and running Powertop I got roughly 14W at iddle (from the wall) with the spinning rust stopped. Which is simply perfect for me.
I then got a pair of Mellanox Connectx-4 LX Model MCX4121A-ACAT so I could connect direct with my desktop using a DAC cable and use the other port as 10Gb.
I updated the firmware to the latest one and I already set the configuration to enable ASPM and activate power save for the port not used.
When I installed the card, my iddle went up to 42W and when I checked Powertop, my machine is only reaching Pstate 2 when in the past it was reaching Pstate 7.
What I already tried:
- pcie_aspm=force as kernel parameter on Ubuntu and TrueNas
- Testing the new Beta of Truenas
- Installing ubuntu with Mellanox drivers
- 3 ddiferent machines (Ryzen on X570, 2 HP Desktops using Ubuntu)
All the tests resulted in Maximum of Pstate 2.
Does anyone has a Connectx-4 and can provide any feedback if it is possible to reduce the power usage?
If not possible, is there any 10gb or 25gb cards which can reach PState 7 or 8?
I know that financially would not make any sense replacing the hardware for this difference in power, but I got really annoyed 
Thanks so much in advance.