Hi, for certain days in my homelab when not running in energy efficient mode, I would like to get max performance, but the BIOS settings of increasing cTDP to 225w seems to not improve things, seems I’m missing some setting. Any advice how to maximize performance for Proxmox NAS with 25 GbE in BIOS. Seems the EPYC CPU is struggling to get above 13Gb/s even with Gen5 NVME. Could it be because of single DDR5 stick, and not having filled all 6 DDR5 channels?
The hardware is Gigabyte ME03-CE0 and Epyc 8224P.
I’m open to OC a few cores if feasible.
Thanks in advance!
192.168.3.1 has 7950x up to 6.0GHz
192.168.3.3 has Epyc 8224P, up to 3 Ghz
Linked by Connectx-4 up to 25 Gb/s.
Have you tried playing with numactl to run iperf on the NUMA node that the memory is tied to? You may need to enable NPS4 with the NPS settings in the BIOS to have the OS see the separate NUMA nodes, then figure out which node the memory is tied to and then run iperf on that node.
It may basically just have to do with the overhead of shuffling that much traffic from the cores to the memory through the I/O die. That should tell you whether it’s a memory channel constraint.
Thank autoturk, I might try that. Never used numactl before.
I did however reinstall Proxmox on the EPYC 8224p changing the OS file format to EXT4, and switched to Gen5 NVME for the OS, to max IOPS. Then set up VMs with Windows Server 2022 Datacenter, and did a Passthrough of the Connectx-4 into the VM. Then activated the SMB Direct with RDMA successfully.
It seem to have improved.
From the Epyc out to 7950X:
first file transfer is around 1.3 GByte/s. The subsequent file transfer of the exact same file when cached, reached full speed of 2.6 GByte/s.
From the 7950X to EPYC:
first file transfer is around 2.0 GByte/s.
The subsequent file transfer of the exact same file when cached, reached full speed of 2.6 GByte/s.
Per today I have for High Perf on ME03-CE0 with 8224P on Bios F05 the follwing set:
It seems the absolute most important performance setting is NPS1
AMD CBS → DF Common Options → Memory Addressing → Numa nodes per socket = NPS1
Then under
AMD CBS → NBIO Common Options → SMU Common Options →
Power Policy Quick Setting = Best Performance
TDP Control = Manual
TDP 225
PPT Control Manual
PPT 225
Power Profile Selection = High Performance Mode
What score does your 7950x get, assuming this is somewhat relevant to your use case(s)?
This question may be relevant considering that the 7950x should consume much less than 225W.