[Build Log] Threadripper and Proxmox shenanigans

[Veeh]

I started this build 2 years ago, then the GPU shortage happened and I waited 2 years to buy my 6800XT at a decent price lol!

There is not a lot of room left!

TR 3960X
Asrock TRx40 Creator
128G 3200
6800XT
1080
5x1T NVMe
5x2T SATA 7.2k
1200W Platinum

Honestly, this PC is just a blast. Threadripper IOMMU support is awesome. I had no issue with the passthrough. The performances are great.
If anyone is interested I can post the conf. There is a small issue that has a workaround with the primary VM. Due to a kernel 5.15 bug. But just let me know.

Benchmark (With both VM running):
Cyberpunk 1440p Ultra: 85fps
Shadow of the TombRaider 1440p Ultra: 124ftp
Horizon ZeroDown 1440P Ultimate: 112fps
Borderland3 1440p Ultra: 108fps
And this is more like a real-life FPS take. I have 3 screens with discord on one and youtube videos running on the other one while running the benchmark.

The game on the NVMe zfs pool have basically no loading time (almost none :P)
Stuff running on the SATA spinning HD even if this is a ZFS pool, it’s more complicated.
If both VM are loading stuff from this HD it can get a bit long.
But it is a really small price to pay to have easy snapshots and full backup.

The only thing I was disappointed by is the Linux gaming part. Before getting the 6800XT, I was using the 1080, and even as VM I could game with it in linux. The perf was lower compared to the win11 VM. But still, it was not horrible.
So far I have not been able to get ok fps in Cyberpunk and Shadow of the tomb raider.
I tried Fedora, Garuda, OpenSUSE, and even RHEL9 with amdgpu-pro. Somehow the GPU frequency stays stuck at ~500MHz, and the GPU load does not ramp up. It is really weird.

Anyway, if you have a lot of PCI lanes and you are not using everything. Add more PC to your machine! lol
It’s working very well. For people who are used to dual booting, this is just for you.

R9 290 - 1080 - 6800XT

[Cheetobandit]

Nice setup! What was the 5.15 kernel bug you had to work around? I may be suffering from it too. And are you using any special vdevs on your zpools?

[Veeh]

Hey.

Thanks
Regarding the kenel bug.
From my understanding, this is happening when you do not have an onboard graphic with your CPU. The host when booting will use the first GPU available to display the boot process. And during this process at some point (with modprobe i think) it is supposed to release your gpu and pass it to vfio. Then you can boot your VM with the graphic card.
This is the bug with kernel 5.15, this release is not happening.

The workaround is to force the host to release the GPU. With proxmox you can do that 2 ways. With a cron @reboot or with a “hookscript”. Bash script that runs when the VM starts.

#!/bin/bash
if [ $2 == “pre-start” ]
then
echo “gpu-hookscript: Resetting GPU for Vitual Machine $1”
echo 1 > /sys/bus/pci/devices/0000:03:00.0/remove #My GC use 03:00, put the pci bus of your GC here
echo 1 > /sys/bus/pci/rescan
fi

People also recommend adding this to your kernel command lines
video=simplefb:off
But It did not help.

Nor did the vendor-reset kernel module
https://www.nicksherlock.com/2020/11/working-around-the-amd-gpu-reset-bug-on-proxmox/

I tried all options but the hookscript that force the pci rescan work great.

regarding the zpool. No fancy vdev setup.
I have 2 zpool, 1 vdev (RAIDZ) per disk type.