Iām a long-time VFIO user, recently with AM5 and a 7000-series AMD GPU. Passthrough works but with work-around for reset. Details below.
Hardware
Motherboard: ASUS X670E ProArt CREATOR WIFI
CPU: AMD 7950X3D
Memory: Kingston FURY Beast DDR5 6000MHz 64GB
dGPU #1: Acer Predator BiFrost Intel Arc A770 OC
dGPU #2: XFX MERC 310 AMD Radeon RX 7900 XTX
BIOS Configuration
iGPU disabled (I simply never tried using it)
Memory running stable at the 6000MHz EXPO profile
ReBAR enabled
IOMMU enabled (not auto, FWIW)
Software
Void Linux on kernel 6.2.13_1
qemu 7.1.0 with patch for āstatic ReBARā, libvirt 9.5.0
Looking Glass B6
Notes
Motherboard
I chose this motherboard because, AFAIU, the first two PCIe x16 slots are wired directly to the CPU, and can run at x8 gen 4 when both slots are occupied. I donāt know if, or how much, this really matters for performance.
CPU
Iām using vfio-isolate to dedicate 7/14 physical/logical cores to the VM. These are the 3D V-Cache cores, except core zero, because I read somewhere that Linux uses core zero even when isolated.
Arc GPU
The BIOS and the Linux host use the Arc GPU, inserted in the first PCIe slot. The BAR size really needs to be increased on this one - I was getting graphical glitches at the small BAR size. Also requires a recent kernel and Mesa 3D version for decent performance. With ReBAR enabled, the BIOS sets the maximum BAR size at boot and all is well. I havenāt tried PCI passthrough with this GPU.
The nice thing about this GPU is that itās only two-slots tall, giving it room to breathe when the second x16 PCIe slot is occupied by another GPU. With two dGPUs stacked, cooling can be a real problem, but this works well in my Define 7 XL case.
Radeon GPU
The Radeon GPU is for a Windows 10 guest VM.
Reset
The VFIO driver is attached at Linux boot via the kernel command line, but if I go straight to boot the VM, I get no graphics output from the VM on the GPU. The workaround is to suspend the Linux system to RAM (S3 sleep) then wake back up. Once it wakes up, however, the BAR size is the default small one, so a script sets it back to the maximum BAR size. Now I can boot the VM with working graphics - but see details about ReBAR below. If I shut down the VM, I have to suspend to RAM again before booting up the VM.
ReBAR
As noted above, Iām using ReBAR. With stock qemu 7.1.0, the GPU could not be successfully initialized in Windows when ReBAR is enabled in BIOS. With ReBAR disabled and using the default BAR size, Windows boots with graphics, but I wanted the big BAR in case it boosts performance.
I found the static ReBAR patch in this Reddit thread. With this patch, Windows boots with graphics even when the GPU has the maximum BAR size. Also, the Radeon software considers āAMD SmartAccess Memoryā enabled in the performance tuning menu.
Conclusion
Overall itās a success, and Iāve played a lot of games on the Windows VM with great performance. Suspending to sleep and waking back up only takes a few seconds and isnāt a great inconvenience, but it must be noted that Windows likes to reboot twice during system updates, and because of the aforementioned reset issue, I have no graphics while this happens. It would be nice to have a fix or better work-around to the reset issue.
edit: I had originally said that setting BAR size via sysfs worked while the vfio driver was loaded - this wasnāt the case, my script was unbinding the driver, setting BAR size, then re-binding the driver.
Scripts
reset-vfio-gpu
#!/bin/sh
gpuAddress="0000:08:00.0"
gpuAudioAddress="0000:08:00.1"
echo 1 > "/sys/bus/pci/devices/$gpuAddress/remove"
echo 1 > "/sys/bus/pci/devices/$gpuAudioAddress/remove"
echo "Suspending..."
rtcwake -m mem -s 10 # use zzz instead?
echo "Woke up, waiting..."
sleep 5s
echo 1 > /sys/bus/pci/rescan
sleep 2s
/usr/local/bin/set-pci-bar-size-7900xtx
echo "Reset done"
set-pci-bar-size-7900xtx
#!/bin/sh
# Point this to the Radeon RX 7900 XTX
gpuAddress="0000:08:00.0"
bar1Size="15" # for 32GB
bar2Size="8" # for 256MB
deviceId="1002 744c"
echo -n "$gpuAddress" > /sys/bus/pci/drivers/vfio-pci/unbind
echo "$bar1Size" > "/sys/bus/pci/devices/$gpuAddress/resource0_resize"
echo "$bar2Size" > "/sys/bus/pci/devices/$gpuAddress/resource2_resize"
echo -n "$deviceId" > /sys/bus/pci/drivers/vfio-pci/new_id || echo -n "$gpuAddress" > /sys/bus/pci/drivers/vfio-pci/bind
# Bit Sizes
# 1 = 2MB
# 2 = 4MB
# 3 = 8MB
# 4 = 16MB
# 5 = 32MB
# 6 = 64MB
# 7 = 128MB
# 8 = 256MB
# 9 = 512MB
# 10 = 1GB
# 11 = 2GB
# 12 = 4GB
# 13 = 8GB
# 14 = 16GB
# 15 = 32GB
isolate-windesktop-cores
#!/bin/sh
op="$1"
test -n "$op" || { echo 'usage: isolate-windesktop-cores <enable|disable>' >&2; exit 1; }
#undoDirPath="/tmp/vfio-isolate"
#mkdir -p -m go=r "$undoDirPath"
#undoFilePath="$undoDirPath/undo.bin"
allCores=0-31
hostCores=0,16,8-15,24-31
guestCores=1-7,17-23
case "$op" in
enable)
vfio-isolate \
drop-caches \
cpuset-create --cpus "C$hostCores" /host.slice \
move-tasks / /host.slice \
compact-memory
#irq-affinity mask "C$guestCores"
taskset -pc "$hostCores" 2 # kthreadd only on host cores
;;
disable)
#vfio-isolate restore "$undoFilePath"
vfio-isolate \
cpuset-delete /host.slice
#irq-affinity mask "C$allCores"
taskset -pc "$allCores" 2 # kthreadd reset
;;
*)
echo "unknown operation '$op'" >&2
false
;;
esac
windrive-bind-vfio
#!/bin/sh
op="$1"
test -n "$op" || { echo 'usage: windrive-bind-vfio <bind|unbind>' >&2; exit 1; }
vendorId="144d"
deviceId="a808"
deviceClass="67586"
pciAddress="0000:70:00.0"
driver_name(){
local address="$1"
basename $(readlink /sys/bus/pci/devices/$address/driver)
}
case "$op" in
bind)
if test -d /sys/bus/pci/devices/$pciAddress/driver; then
driver=$(driver_name $pciAddress)
if test "$driver" = "vfio-pci"; then
echo "vfio driver already loaded; doing nothing"
exit
fi
echo -n "unbinding current driver '$driver'... "
echo "$pciAddress" > /sys/bus/pci/devices/$pciAddress/driver/unbind || exit
echo "done"
else
echo "no driver loaded"
fi
echo -n "binding vfio driver... "
echo $vendorId $deviceClass > /sys/bus/pci/drivers/vfio-pci/new_id || exit
echo "done"
;;
unbind)
if test -d /sys/bus/pci/devices/$pciAddress/driver; then
driver=$(driver_name $pciAddress)
if test "$driver" = "nvme"; then
echo "NVMe driver already loaded; doing nothing"
exit
fi
echo -n "unbinding current driver '$driver'... "
echo "$pciAddress" > /sys/bus/pci/devices/$pciAddress/driver/unbind || exit
echo "done"
else
echo "no driver loaded"
fi
echo -n "binding NVMe driver... "
echo $vendorId $deviceClass > /sys/bus/pci/drivers/nvme/new_id || exit
echo " done"
;;
*)
echo "unknown operation '$op'" >&2
false
;;
esac
(card is louder and imo in general worse in every aspect).
But Good guys Corsair replaced then after I told them what happen - main part I fucked up and so on )
)