Extremely slow DRAM: 8GB/s Memory Write on Samsung DDR4 RDIMM

Hi, I have a workstation/server that has extremely slow DRAM performance. What could be the cause? What can I test next to debug it?

Motherboard: Asus WRX80E-SAGE SE WIFI
CPU: Threadripper Pro 3945WX
RAM: Samsung M393A8G40CB4-CWEBQ 64GB (DDR4 RDIMM)
OS: Proxmox 8.2

Passmark PerformanceTest shows extremely low “Memory Write” at 8158 MB/s and extremely low “Memory Threaded” 18725 MB/s. DRAM supports max 3200 but running at 2667 with default BIOS settings.

PassMark PerformanceTest Linux (11.0.1002)
AMD Ryzen Threadripper PRO 3945WX 12-Cores (x86_64)
12 cores @ 4425 MHz  |  62.7 GiB RAM
Number of Processes: 24  |  Test Iterations: 1  |  Test Duration: Medium
--------------------------------------------------------------------------------
Memory Mark:                       2527
  Database Operations              5794 Thousand Operations/s
  Memory Read Cached               27460 MB/s
  Memory Read Uncached             14555 MB/s
  Memory Write                     8158 MB/s
  Available RAM                    61135 Megabytes
  Memory Latency                   47 Nanoseconds
  Memory Threaded                  18725 MB/s

Either timings are off (primary or secondary) or issues with either motherboard or ram. You can check the timings and then check the ram because my Ddr3 numbers blows yours out of the water.

Start there and see what comes up. If unsure then check back with us

RAM: Samsung M393A8G40CB4-CWEBQ 64GB (DDR4 RDIMM)

Are you really running with only single dimm? Regardless of that, unchached read and write should be roughly symmetrical.

Here are rough benchmark numbers PassMark - Samsung M393A8G40AB2-CWE 64GB - Price performance comparison , your write is half ow what it should be.

  • double check bios settings, set to jedec defaults
  • try moving dimm around to different slots and retest, consult mb manual which are optimal
  • run memtest just to see if there are no stability issues
1 Like

I can not tell you what it is about your specific setup, but I am on a 3995WX with all 8 channels populated and these are my results for the same test.

Memory Mark:                       3273
  Database Operations              24090 Thousand Operations/s
  Memory Read Cached               31807 MB/s
  Memory Read Uncached             13834 MB/s
  Memory Write                     13974 MB/s
  Available RAM                    491342 Megabytes
  Memory Latency                   48 Nanoseconds
  Memory Threaded                  142900 MB/s

Thanks for the tips and quick replies, @Necrosaro, @greatnull , H-i-v-e .

I did a few tests by tweaking the DRAM timings and NUMA in BIOS. Two BIOS settings that helped improve the memory bandwidth:

  1. set DRAM Frequency to DDR4-3200 (default is 2666MHz)
  2. change NUMA domains per socket from AUTO to NPS2 (or NPS4). This setting alone lifted Memory Write bandwidth from 9GB/s to 14GB/s, and Memory Threaded from 22GB/s to 56GB/s.
Memory Mark:                       3384
  Database Operations              9842 Thousand Operations/s
  Memory Read Cached               30772 MB/s
  Memory Read Uncached             15041 MB/s
  Memory Write                     14588 MB/s
  Available RAM                    351021 Megabytes
  Memory Latency                   40 Nanoseconds
  Memory Threaded                  56536 MB/s

Note that in this test, I have 6 DIMMs in the system: 5x64GB and 1x32GB (total 352GB).

1 Like

I have more DIMMs. In my original test, I wanted to isolate and debug so I started with 1 DIMM.

Thank you for sharing. Comparing yours to my optimized result with BIOS tweaking, I see that the Memory Read and Memory Write are comparable now. But my Memory Threaded is only 56GB/s (6 channels used), while yours is 142GB/s (8 channels used).

Suppose I had 8 DIMMs populated, my Memory Threaded could go up to 56/6*8=75GB/s, which is only about 50% of 142GB/s. Could CPU be a major factor here? Your 3995WX has way more threads than my 3945WX.

1 Like

Hold on a second, I have the same motherboard. Which Bios/UEFI Version do you have because I do not have an option to set anything in regards to NUMA as far as I can see. Can you let me know in which submenu the options for this are for you and which version you are on, please?

Nevermind, I found it.

Numa set to auto:

Memory Mark:                       3273
  Database Operations              24090 Thousand Operations/s
  Memory Read Cached               31807 MB/s
  Memory Read Uncached             13834 MB/s
  Memory Write                     13974 MB/s
  Available RAM                    491342 Megabytes
  Memory Latency                   48 Nanoseconds
  Memory Threaded                  142900 MB/s

Numa set to NPS4

Memory Mark:                       3447
  Database Operations              24880 Thousand Operations/s
  Memory Read Cached               30094 MB/s
  Memory Read Uncached             15151 MB/s
  Memory Write                     13021 MB/s
  Available RAM                    507396 Megabytes
  Memory Latency                   43 Nanoseconds
  Memory Threaded                  174654 MB/s
1 Like

The idea with NUMA usually is that you can optimize either for latency (having things on one node that has fast access to memory) or for bandwidth (having and average latency but more total bandwidth) so I am not sure how accurate this specific benchmark is.

1 Like

Passmark uses a fairly shallow memory depth for it’s memory benchmark and at this point, among the larger processors with lots of L3, is benchmarking L3 bandwidth more than it is DRAM bandwidth. This explains why NUMA is affecting the score so much.

@H-i-v-e thanks again for sharing your result. Glad you fount the NUMA setting.
I’m still puzzled that the Memory Read/Write at 3200Mhz is only around 15GB/s. The theoretical speed of 3200MHz should be 3200/1000*8=25.6GB/s.

@twin_savage thanks for chiming in. What benchmark tool would you recommend for DRAM speed testing?

love that free money

I recommend using the Intel Memory Latency Checker; its basically the FIO of memory benchmarks, it takes some arguments to get it to do what you want but it is very customizable and it runs on Linux as well as Windows.

This topic was automatically closed after 270 days. New replies are no longer allowed.