TL;DR:
On my dual-socket server, Optane PMem (in App Direct mode) is assigned to separate, non-CPU proximity domains in the ACPI SRAT table (e.g., CPUs are domains 0/1, PMem is 2/3). This causes Linux (both old and new kernels) to fail in correctly identifying the PMem’s NUMA affinity. Performance tests show no NUMA effect—remote access is sometimes even faster than local. I suspect this is due to a BIOS HMAT policy, but I’m stuck on how to fix it.
Full Post:
Hello everyone,
I’ve run into a tricky PMem NUMA performance issue while setting up a server. After an in-depth investigation, I’ve hit a wall and would appreciate some advice.
Hardware and Configuration:
-
Server: A dual-socket server
-
CPU: 2 x Intel(R) Xeon(R) Gold 6330 CPU @ 2.00GHz (corresponding to NUMA nodes 0 and 1)
-
Memory: 8 x Intel Optane Persistent Memory 200 Series (PMem)
-
PMem Configuration:
-
Mode: App Direct
-
Interleaving: Intra-socket interleaving
-
Logical Devices: Presented to the OS as two block devices, /dev/pmem0 and /dev/pmem1, which are expected to be affiliated with NUMA nodes 0 and 1, respectively.
-
The Problem:
1. Incorrect NUMA Node Assignment:
-
On an older Linux kernel, the output of cat /sys/block/pmem0/device/numa_node was -1 .
-
After upgrading to Ubuntu 22.04 (with a newer kernel), the numa_node for both pmem0 and pmem1 shows 0 . I believe this is incorrect, as they can’t both belong to node 0. It seems to be a fallback default value.
2. Performance Tests Confirm the Issue:
-
I used the test [code][GitHub - hpides/pmem-olap: This repository contains the source code for our ACM SIGMOD '21 paper (Maximizing Persistent Memory Bandwidth Utilization for OLAP Workloads)] provided in a [research paper][https://lawben.com/publication/pmem-olap/pmem_olap_sigmod_2021.pdf] to benchmark the write bandwidth on both PMem devices.
-
The performance results show no significant difference, regardless of whether the test thread was pinned to the local NUMA node, a remote NUMA node, or not pinned at all.
-
In some cases, the bandwidth from a remote CPU accessing the PMem was even slightly higher than local access, which completely contradicts the principles of a NUMA architecture.
My Investigation Steps:
1. Checked the ACPI SRAT Table: I believe the root cause lies within ACPI. The SRAT table shows:
-
The two CPU proximity domains are correctly identified as 0 and 1 .
-
However, the two PMem logical devices are assigned to new, separate proximity domains: 2 and 3 .
2. Enabled SNC (Sub-NUMA Clustering): I tried enabling SNC in the BIOS, which expanded the CPU NUMA domains to four (0, 1, 2, 3). In response, the PMem devices’ proximity domains simply shifted to 4 and 5 . This indicates a deliberate BIOS policy to always isolate PMem devices in their own domains.
3. My Hypothesis: I’ve learned this might be an advanced BIOS feature related to HMAT (Heterogeneous Memory Attribute Table) . This feature reports PMem as a separate “Memory-only” domain to provide finer-grained information about the memory hierarchy. However, I strongly suspect this is what’s preventing the Linux kernel from correctly associating the PMem devices with their physically adjacent CPU NUMA nodes.
My Dilemma and Questions:
I’m now at a crossroads. Either my BIOS is misconfigured, or my newer OS is missing a step needed to correctly interpret this advanced HMAT information.
-
BIOS Level: What specific settings should I look for in the BIOS/UEFI to change this reporting behavior? For instance, is there a way to disable HMAT or force the PMem proximity domains to align with the CPU domains (e.g., pmem0 → domain 0, pmem1 → domain 1)?
-
OS Level: On a modern system like Ubuntu 22.04, are there any specific kernel parameters or advanced ndctl commands that can be used to manually associate these detached NUMA domains?
-
Other Possibilities: Have I overlooked any other potential avenues for investigation?
Thank you for any help or suggestions you can offer!