Proxmox on Ryzen 7950X - low memory performance in guests

Operating Systems & Open Source Virtualization
, ,
1

Hey guys,

I’ve got a Proxmox cluster with a few different nodes in it, and while trying to benchmark the VMs I noticed some weirdly inconsistent performance in VMs on the nodes running 7950X processors. Essentially, I’m seeing a fraction of the memory performance on those VMs compared to the hypervisor itself when performing memory-intensive benchmarks.

Ignoring real-world testing for a while, I can consistently reproduce the issue with sysbench with a small block size (1K usually). We’re talking ~230 MiB/s, compared to the ~6000 MiB/s I’m getting on the hypervisors (which itself is slower than I’d hope, but I’m running 192GB of memory in them so can’t get them to boot over 3600MT/s). The performance also scales basically linearly with the number of threads (so doubling sysbench threads on the VM will double the throughput without changing performance on the hyper).

The same tests on the older Xeon-based hypers (E5620, for example) show nearly identical performance between the hyper and VMs on it. Proxmox itself is installed identically on each of them - it’s all Ansibled and they’re all running Debian 12.

sysbench CPU benchmarks come within 5-10% of the hypervisor, so it really is just limited to memory.

A few things I’ve tried to see if I can optimise the issue:

  • NUMA enabled, core pinning set, Ballooning disabled, mitigations disabled (temporarily), CPU type set to host, Huge Pages enabled
  • Tried to increase the memory clock, no luck getting the DIMMs running past 3600MT/s (not surprising, really)
  • Checked for any obvious BIOS tweaks that might affect this, not seeing anything.
  • Update BIOS versions.
  • Compared a desktop motherboard with an Asrock Rack one - no difference between the two.
  • Tried a different set of memory, 128GB (4x 32GB instead of 4x 48GB)

At this point I’ve basically exhausted my options, and I don’t really have a way to isolate if this is a 7000-series issue, or a DDR5 issue, or just a classic PEBKAC. Hoping that someone here has come across this sort of issue and might have some pointers or ideas. Thanks in advance!

2

can you dump

cat /proc/meminfo | grep HugePages

and

the xml config for your vm (I am looking for memoryBacking entry specifically but all the info there might be handy).

3

Thanks for taking a look.

For the 7950X VM, HugePages output:

root@hpbench:/home/ubuntu# cat /proc/meminfo | grep HugePages
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
FileHugePages:         0 kB
HugePages_Total:    1536
HugePages_Free:     1536
HugePages_Rsvd:        0
HugePages_Surp:        0

Same setting for the VM on the Xeon system I’m using at the moment. I previously had 1GB Huge Pages enabled but deleted and recreated the VM after, so we’re back down to 2MB Huge Pages.

I’m just configuring it all with Proxmox interfacing directly with KVM/QEMU (no libvirt on these systems), so I don’t know of any XML config with memoryBacking in it. Not sure if the Proxmox .conf for the VMs helps?

balloon: 0
boot: c
bootdisk: scsi0
cores: 2
cpu: host,hidden=1,flags=+ibpb;+amd-no-ssb;+pdpe1gb;+aes
hookscript: local:snippets/vm-hook.py
machine: q35
memory: 4096
meta: creation-qemu=8.0.2,ctime=1715251786
name: hpbench
net0: virtio=BC:24:11:79:9D:CD,bridge=vmbr0,firewall=1
numa: 1
onboot: 1
ostype: l26
scsi0: vm-nvme:vm-154-disk-0,discard=on,size=60G,ssd=1
scsi1: vm-nvme:vm-154-disk-1,discard=on,size=1G,ssd=1
scsihw: virtio-scsi-pci
serial0: socket
sockets: 1
vga: type=std,memory=16

Compared to the Xeon (basically the defaults that the 7950X VM had before I started tweaking settings):

boot: c
bootdisk: scsi0
cores: 2
cpu: cputype=kvm64,hidden=1
hookscript: local:snippets/vm-hook.py
memory: 4096
meta: creation-qemu=8.0.2,ctime=1715251786
name: hpbench2
net0: virtio=BE:C9:79:9B:5B:53,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: l26
scsi0: vm-nvme:vm-183-disk-0,discard=on,size=60G,ssd=1
scsi1: vm-nvme:vm-183-disk-1,discard=on,size=1G,ssd=1
scsihw: virtio-scsi-pci
serial0: socket
sockets: 1
vga: type=std,memory=16