Hi, I have a workstation/server that has extremely slow DRAM performance. What could be the cause? What can I test next to debug it?
Motherboard: Asus WRX80E-SAGE SE WIFI CPU: Threadripper Pro 3945WX RAM: Samsung M393A8G40CB4-CWEBQ 64GB (DDR4 RDIMM) OS: Proxmox 8.2
Passmark PerformanceTest shows extremely low “Memory Write” at 8158 MB/s and extremely low “Memory Threaded” 18725 MB/s. DRAM supports max 3200 but running at 2667 with default BIOS settings.
PassMark PerformanceTest Linux (11.0.1002)
AMD Ryzen Threadripper PRO 3945WX 12-Cores (x86_64)
12 cores @ 4425 MHz | 62.7 GiB RAM
Number of Processes: 24 | Test Iterations: 1 | Test Duration: Medium
--------------------------------------------------------------------------------
Memory Mark: 2527
Database Operations 5794 Thousand Operations/s
Memory Read Cached 27460 MB/s
Memory Read Uncached 14555 MB/s
Memory Write 8158 MB/s
Available RAM 61135 Megabytes
Memory Latency 47 Nanoseconds
Memory Threaded 18725 MB/s
Either timings are off (primary or secondary) or issues with either motherboard or ram. You can check the timings and then check the ram because my Ddr3 numbers blows yours out of the water.
Start there and see what comes up. If unsure then check back with us
I can not tell you what it is about your specific setup, but I am on a 3995WX with all 8 channels populated and these are my results for the same test.
I did a few tests by tweaking the DRAM timings and NUMA in BIOS. Two BIOS settings that helped improve the memory bandwidth:
set DRAM Frequency to DDR4-3200 (default is 2666MHz)
change NUMA domains per socket from AUTO to NPS2 (or NPS4). This setting alone lifted Memory Write bandwidth from 9GB/s to 14GB/s, and Memory Threaded from 22GB/s to 56GB/s.
Thank you for sharing. Comparing yours to my optimized result with BIOS tweaking, I see that the Memory Read and Memory Write are comparable now. But my Memory Threaded is only 56GB/s (6 channels used), while yours is 142GB/s (8 channels used).
Suppose I had 8 DIMMs populated, my Memory Threaded could go up to 56/6*8=75GB/s, which is only about 50% of 142GB/s. Could CPU be a major factor here? Your 3995WX has way more threads than my 3945WX.
Hold on a second, I have the same motherboard. Which Bios/UEFI Version do you have because I do not have an option to set anything in regards to NUMA as far as I can see. Can you let me know in which submenu the options for this are for you and which version you are on, please?
The idea with NUMA usually is that you can optimize either for latency (having things on one node that has fast access to memory) or for bandwidth (having and average latency but more total bandwidth) so I am not sure how accurate this specific benchmark is.
Passmark uses a fairly shallow memory depth for it’s memory benchmark and at this point, among the larger processors with lots of L3, is benchmarking L3 bandwidth more than it is DRAM bandwidth. This explains why NUMA is affecting the score so much.
@H-i-v-e thanks again for sharing your result. Glad you fount the NUMA setting.
I’m still puzzled that the Memory Read/Write at 3200Mhz is only around 15GB/s. The theoretical speed of 3200MHz should be 3200/1000*8=25.6GB/s.
@twin_savage thanks for chiming in. What benchmark tool would you recommend for DRAM speed testing?
I recommend using the Intel Memory Latency Checker; its basically the FIO of memory benchmarks, it takes some arguments to get it to do what you want but it is very customizable and it runs on Linux as well as Windows.