Sub-par frametime and lower framerates with Proxmox 7.4 Win10 Guest

Operating Systems & Open Source Virtualization
1

Hi Everyone.

I have a win10 guest VM on proxmox that I’ve GPU passthroughed to, but the frametimes are rough and the framerates are a bit low. Still above 60. I have followed this guide to get to where I am now: PCI Passthrough - Proxmox VE

I am looking for input from my esteemed vfio peers as to what to do next.

Frametime example and problem explanation

image

image
image766×140 2.42 KB

In games I have consistently “decent” framerates, but every say, 30 seconds to a minute or so I will have a rather large frametime jump where it’s quite noticable that it’s “stuttered” and the frametimes are in the 16-20ms range for some games, where as its normally a bit lower. It’s not very smooth either as you can see.

I haven’t yet got to benchmarking the framerate to see if I am getting the same, but I feel that the RX 6900 XT is producing lower framerates then baremetal Win11 as well (my previous setup)

Really I am just concerend about smoothing out this minor stuttering. I can accept “only” 170 frames instead of say 200.

Host Specs, GRUB Contents & Proxmox Version

Grub contents: GRUB_CMDLINE_LINUX_DEFAULT=“quiet intel_iommu=on iommu=pt”
^Just noticed this is INTEL IOMMU, switching it to AMD after posting this… probably wont help…

CPU: Ryzen 3900x
RAM: 64GB 3200mhz Corsair 15-15-15-36 (2x32GB Sticks)
MB: TUF GAMING B550M-PLUS
GPU: RX 6900XT (Navi 10, no reset bug)
GPU: Intel A380 (Unused and currently not showing up in lspci, to investigate later)
SSD: 2x 500GB samsung/crucial Sata Drives (raid 1 zfs A-SHIFT 13, 128GB passed to guest as VirtIO)
NVME: 2x 1TB Gigabyte Gen4 NVMe (raid 0 zfs A-SHIFT 13, 1.9TIB passed to guest as VirtIO)

Kernal: 6.2.6-1-pve #1 SMP PREEMPT_DYNAMIC PVE 6.2.6-1
PVE Manager Version: pve-manager/7.4-3/9002ab8a

root@maxwell:~# pveversion -v
proxmox-ve: 7.4-1 (running kernel: 6.2.6-1-pve)
pve-manager: 7.4-3 (running version: 7.4-3/9002ab8a)
pve-kernel-5.15: 7.4-1
pve-kernel-6.2: 7.3-8
pve-kernel-6.2.6-1-pve: 6.2.6-1
pve-kernel-5.15.104-1-pve: 5.15.104-1
pve-kernel-5.15.74-1-pve: 5.15.74-1
ceph-fuse: 15.2.17-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.4
libproxmox-backup-qemu0: 1.3.1-1
libproxmox-rs-perl: 0.2.1
libpve-access-control: 7.4-2
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.3-4
libpve-guest-common-perl: 4.2-4
libpve-http-server-perl: 4.2-1
libpve-rs-perl: 0.7.5
libpve-storage-perl: 7.4-2
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-2
lxcfs: 5.0.3-pve1
novnc-pve: 1.4.0-1
proxmox-backup-client: 2.4.1-1
proxmox-backup-file-restore: 2.4.1-1
proxmox-kernel-helper: 7.4-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.6.5
pve-cluster: 7.3-3
pve-container: 4.4-3
pve-docs: 7.4-2
pve-edk2-firmware: 3.20230228-1
pve-firewall: 4.3-1
pve-firmware: 3.6-4
pve-ha-manager: 3.6.0
pve-i18n: 2.12-1
pve-qemu-kvm: 7.2.0-8
pve-xtermjs: 4.16.0-1
qemu-server: 7.4-3
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+3
vncterm: 1.7-1
zfsutils-linux: 2.1.9-pve1

Guest VM Settings

agent: 1
balloon: 0
bios: ovmf
boot: order=scsi1
cores: 12
cpu: host
efidisk0: local-zfs:vm-107-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M
hostpci0: 0000:0a:00,pcie=1,x-vga=1
machine: pc-q35-7.2
memory: 24576
meta: creation-qemu=7.2.0,ctime=1680595875
name: Win10-Ligo
net0: virtio=4E:43:07:96:2B:CF,bridge=vmbr0,firewall=1,tag=50
numa: 1
ostype: win10
scsi0: Ligo:vm-107-disk-0,backup=0,discard=on,iothread=1,size=2000G,ssd=1
scsi1: local-zfs:vm-107-disk-1,cache=unsafe,discard=on,iothread=1,size=128G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=e1907010-f732-4853-8565-7c534d9f69c8
sockets: 1
tags: vlan50-mgt
usb0: host=8087:0029
usb1: host=1e71:170e
usb2: host=046d:0ab1
usb3: host=046d:c21f
usb4: host=046d:c33c
usb5: host=046d:c08b
vmgenid: 61845dc8-df21-433e-90c3-d40ef255420b

BIOS options I have knowingly changed

Above 4G decoding: On
Resizable Bar: Off (causes VM to crash with no display after loading windows if ON)
xAPIC: set to x2APIC
VT-D/IOMMU: Enabled
CSM: Disabled

Things I have tried so far

Played around with Resizable bar and found it crashed the VM.
Orginally I had the host OS drive as an IDE, instead of VirtIO, changing it to virtio improved the frametimes but not that much.

Tried allocating all 24 cores to the guest VPM, no dice. Tried 4 cores, 6 cores, and now 12 cores. Doesn’t appear to change anything with lower amounts obviously slowing it down further.

Tried enabling all relevent AMD CPU flags in the proxmox menu, no change really.

Tried disabling balooning ram, no real improvement.
Tried buggering around with BIOS settings are PCIe, no real improvement.

Went from Kernal 5.15 to 6.2, no real improvement.

Enabled CPU NUMA on proxmox UI, unclear if it helped.

Send help my fellow magicians! :slight_smile:

P.S, hope I made this nice and short for you… :heart:

2

What’s your time spy score? It should be on par with that under bear metal.

3

I’ll go ahead and start running that for you now.

4

image
image1419×984 117 KB

My results. AMD Radeon RX 6900 XT video card benchmark result - AMD Ryzen 9 3900X,?

5

It looks like your cpu score is for short.
I don’t know Proxmox.
For Linux, you can try following things mentioned in my post here.
https://forum.level1techs.com/t/7900x-vs-7950x-for-vfio/194648/6?u=jxdking

6

I dont really know how to start translating that to proxmox, but I’ll give it a go later on. CPU Pinning seems to be not required for some people, is there any way around that as I’m just concerned about my ability to replicate the setup if it catches fire etc :slight_smile:

7

Update!

Using Proxmox’s new CPU Affinity and isolcpus=6-11,18-23 as a cmdline default, I am seeing less stuttering over a given game then before. It’s still there but it’s improved. Unfortunetely MSI afterburner’s overlay is now crashing the VM/HOST due to some sort of AMD driver issue but that is a seperate problem.

I’m still looking to improve it to where there’s little to no consistent stuttering, but we’re getting there. Next I might look into huge pages and interrupt signaling as my next tests.

It’s a little frustrating to experience this on Proxmox as normally it’s a just works solution. I guess in this case it’s just too advanced to expect it to be perfect.

8

“mitigations=off” plays a big role on performance. Add this to your boot parameters if possible.
You only passthrough second CCD. Effectively, it is a 3600x. If there is no other things going on your system, why not using all available physical cores? Try passthrough 0-11, or even 0-23.
Also, you may enable resize bar in the bios to see if it improves.

9

Post the vm conf file also.

10

Hi,

It’s available in my post under Guest VM settings

11

If you have other load on your Proxmox (by other VMs or Containers), it kills latency-sensitive workloads like games. Even though the cores may be exclusive to that VM, the rest of the CPU, the memory and periphery like PCIe is still shared. Current architecture can’t really limit or isolate most of the system. If VM No. 34 wants memory bandwidth, everyone else has to deal with higher latency.

12

I only want to give the guest vm a 3600x worth of CPU as the other “half” is intended to be used by other VMs

13

Are you therefore saying it’s not possible to do VFIO gaming on Proxmox?

14

This has nothing to do with Proxmox or VFIO (which both work fine, using it myself), but with technical limitation of the architecture.
If you don’t have any load on the system and only your Gaming VM running, it should perform very well and within expected performance.
But if another VM is compressing 100s of Gigabytes on multiple cores (my Proxmox) or if you render stuff in blender (bare-metal native machine) in the background → Gaming performance will be bad because of shared ressources. And Gaming doesn’t like noisy neighbors.

15

I bet you earned levelone rewards for your fancy folded up post :wink:

I’ve just built a dual gaming system in a SFF case using proxmox, a B550 board and an A380 and an RTX3050. Both passed through after following the usual instructions.

It is vital to have resizable BAR turned on to get decent performance on the A380
Probably also for the RTX 3050 but I never tried without.

I’m delighted with how it’s working. Solve the resizable BAR problem and I think you’ll be fine.