Proxmox vs XCP-NG performance

davemcl · January 9, 2023, 2:10am

Ive been evaluating Proxmox & XCP-NG for MSSQL/Windows workloads and long story short I cant replicate the performance of XCP-NG with Proxmox.
Whilst fio, Geekbench & other benchmarks were fairly similar, when it got to actual real workloads that will eventually run on these servers the difference was staggering.

MSSQL ETL task
Current Xeon E5-2689 v4 on VMWare 6.7 (OVH)
40m 18s

Dell R740 2 x XEON Gold 6152, RAID 10 SAS/SSD on Perc H730
Proxmox 7.3-4 - VM 16 vCPU, 32GB RAM
41m 06s

XCP-NG 8.2.1- VM 16 vCPU, 32GB RAM
17m 20s

My Proxmox VM settings

agent: 1
bios: ovmf
boot: order=scsi0;net0;ide0;ide2
cores: 16
cpu: host
efidisk0: SSD-R10:vm-200-disk-1,efitype=4m,pre-enrolled-keys=1,size=528K
ide0: none,media=cdrom
machine: pc-q35-7.1
memory: 32768
meta: creation-qemu=7.0.0,ctime=1663398507
name: Win-Srv-22-01
net0: virtio=AE:9A:D1:19:00:E7,bridge=vmbr0,firewall=1,tag=25
numa: 0
ostype: win11
scsi0: SSD-R10:vm-200-disk-2,cache=writeback,discard=on,iothread=1,size=64G
scsi1: SSD-R10:vm-200-disk-3,discard=on,iothread=1,size=64G
scsi2: SSD-R10:vm-200-disk-4,discard=on,iothread=1,size=128G
scsi3: SSD-R10:vm-200-disk-5,discard=on,iothread=1,size=64G
scsihw: virtio-scsi-single
smbios1: uuid=38eb0316-63e2-4dbb-9e1a-e0f45bc1d220
sockets: 2
tags: windows
tpmstate0: SSD-R10:vm-200-disk-0,size=4M,version=v2.0
vmgenid: 5a31f7fb-22dd-4e1a-adcc-136fcba06c34

uname -a
Linux pve02 5.15.83-1-pve #1 SMP PVE 5.15.83-1 (2022-12-15T00:00Z) x86_64 GNU/Linux

Ive tried different CPU types in PVE with minimal changes in result.
Any recommendations on what to try next?

Dave

max1220 · January 9, 2023, 3:23am

I’ve not used PVE, but it uses QEMU/KVM, so have you tried to use a “host-passthough” option? This should pass along the host CPU flags, so you should have the same instructions available(well, detected, I think they’re always available).

Maybe it’s not the CPU though, so have you tried running an disk/IO benchmark, maybe? What’s your backing device for the VM storage on Proxmox?

davemcl · January 9, 2023, 3:48am

Backing storage is 6 X 960 GB SAS SSD connected to a Dell PERC H730 card in RAID10.
I’m digging into BIOS settings at the moment to see if Ive missed something obvious.

In Proxmox Im using raw format with LVM-thin.

davemcl · January 9, 2023, 7:25am

Have also now tried

disabling mem ballooning
changing cache on controller to writeback
setting trace flag for SQL to T8038
Disable use tablet for pointer

davemcl · January 9, 2023, 9:00am

Tried same ETL process on a different node that has 2 x Silver 4114 CPU’s.
Initial process took 46m, went down to 23m after disabling all CPU vuln mitigation.

davemcl · January 10, 2023, 2:37am

XCP-NG seems less affected by CPU vuln mitigation than Proxmox - at least for Windows workloads.

MadMatt · January 10, 2023, 5:57am

Have you enabled huge pages?
Is the test CPU or disk bound?

davemcl · January 10, 2023, 6:37am

Arent they enabled by default? Ill check later when Im doing more testing.
Oddly the task doesnt really max out CPU or disk.

MadMatt · January 10, 2023, 7:12am

Nope …
You need to configure them in your kernel boot params, then you can use them in your VMs:

Zaf9670 · March 10, 2023, 8:33pm

Any progress @davemcl or did you decide to stick with XCP-NG for performance at the end?

davemcl · July 8, 2023, 12:19pm

Went with Proxmox and did lots of test workloads, however theres nothing quite like production ramping up and Ive had lots of issues with VM’s locking up under load. Basically we get SCSI reset messages in the VM and obscure errors on the PVE host.
Theres a number of forum posts about it and similar issues are being experienced by others.

bambinone · September 22, 2023, 9:56pm

Did you install the virtio-win drivers in the VM?

Have you tried increasing the number of I/O threads? Have you tried pinning the CPUs?

You should probably be running with cache=none on all those volumes. Let Windows manage its own caching.

You should probably be running with hugepages as someone else mentioned. It’s most likely running with 2MiB transparent hugepages out of the box but you can switch to 1GiB static hugepages by adding default_hugepagesz=1G hugepagesz=1G hugepages=32 to your kernel commandline and this to your domain XML:

<memoryBacking>
	<hugepages/>
</memoryBacking>

I’m not familiar with Proxmox’s VM configuration file format but if you can figure out how to dump the domain XML I can take a closer look. Cheers.

davemcl · October 27, 2023, 4:29am

Yes, the latest Virt-IO drivers are installed, plenty of others experiencing the same issue.
Its being tracked on the virt-io side here:

github.com/virtio-win/kvm-guest-drivers-windows

viostor Reset to device, \Device\RaidPort3, was issued. VM is frozen.

opened 04:35PM - 05 Apr 22 UTC

mherlitzius

Bug

Unfortunately we are experiencing the same issue like described in #623. Erro…r message "viostor Reset to device, \Device\RaidPort3, was issued." appears in the log and the VM is frozen / unresponsive. Only cold reboot helps. This happened already several times. **Environment:** - KVM / libvirt 3.0.0-4+deb9u5 - Storage: Ceph Nautilus 14.2.21, SSD pool - Problem occurs on both Intel (Xeon E5-2643) and AMD (EPYC 7F72) compute nodes **Affected VMs:** Windows server version: Windows Server 2016 - Virtio driver version: build 215 Windows server version: Windows Server 2016 - Virtio driver version: 100.77.104.17100 Windows server version: Windows Server 2012 R2 - Virtio driver version: 62.74.104.14100 It seems primarily servers running MS SQL Server are affected. The error message means that the storage does not respond for 30 seconds. In Ceph we see nothing close to that, latency of the affected volume is good. Disk performance and throughput on the Ceph SSD storage pool is excellent when measured with CrystalDiskMark or the SQL Server-internal benchmark. It seems the issue may occur at arbitrary times, i.e. it is not related to the overall load on the storage cluster. The VM often runs for a few minutes after the first "viostor Reset to device" entries appear in the log. The first application-level problems can appear even if the entire VM is not yet frozen. On a volume on which this error occured once, we can reproduce it as follows: - run CrystalDiskMark on that volume (or just copy files from that volume), while the volume is attached to the original server. The error appears in the log and the VM freezes within seconds. When we can not reproduce this issue: - If the affected volume is attached to another VM, we can not reproduce this problem. Running the same operation (e.g. CrystalDiskMark or copying files) on the same volume when attached to another VM the error does not occur. - Running the same operation on a (newly created) volume on the initial VM does not reproduce the problem. This means that apparently "only" this combination of volume and VM causes the problem. We have been able to resolve this issue at least for some time, by copying the data from the affected volume to a new volume (while attached to another VM) and then attach the newly created volume to the initial VM. It is working for a while before the error reoccurs. It also seems that after the problem occurred for the very first time on a volume it then reoccurs more often (it is not always possible to immediately replace the affected volume as these are production servers). Do you have any idea what could be causing this problem and how we can work around it? Thank you very much.

davemcl · November 20, 2024, 12:22pm

This is finally resolved with the latest VirtIO Drivers (.266) for Windows VM’s and SCSI timeouts.

Zedicus · November 20, 2024, 1:57pm

i wrote a issue about this in 2017. it is actually a cloned hardware issue that has to do with a weird chain of events not exactly related to ProxMox. it is good to see an in-driver fix finally implemented.

it has to do with how LVM handles some of the requests, and yes, is only substantially reproducible when load is heavy. and will cause issues on any OS in the VM, not just windows.