I wrote myself a little guide and a few scripts to manage GPU passthrough and thought I would share.
NOTE: This is for arch and seems to work for me, YMMV
Configuration for VMM PCI Passthrough with NVidia GPU for host and VM
Setup Host
mkinitpcio.conf
Early Load VFIO before Nvidia drivers
REF: https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF
MODULES=(vfio_pci vfio vfio_iommu_type1 vfio_virqfd nvidia nvidia_modeset nvidia_uvm nvidia_drm)
Boot Loader
Add Device ID(s) of passthrough device(s) to VFIO driver in options.
REF: https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF
options vfio-pci.ids=10de:2684,10de:22ba
Enable Hugepages
REF: https://looking-glass.io/wiki/Performance_Optimization
- create a sysctl entry (/etc/sysctl.d/10-hugepages.conf) to create the hugepages (see REF for sizing)
vm.nr_hugepages=33000
vm.hugetlb_shm_group=992
sys.kernel.mm.transparent_hugepage.defrag='never'
sys.kernel.mm.transparent_hugepage.enabled='never'
- add Hugepages to /etc/fstab to mount with appropriate permissions
# HUGEPAGES
hugetlbfs /dev/hugepages hugetlbfs mode=01770,gid=kvm 0 0
- Create a file /etc/tmpfiles.d/10-looking-glass.conf
# Shared Memory
f /dev/shm/looking-glass 0660 brian kvm -
Install Looking Glass Client on arch
REF: https://aur.archlinux.org/packages/looking-glass
Setup VMM (libvirt) VM
Pre-Install
After creating the VM update it’s XML
EDITOR=vim virsh edit VM_Name
# Change Domain tag to include Namespace
# REF: https://www.reddit.com/r/VFIO/comments/qx4rg7/can_anyone_else_confirm_that_vfio_doesnt_work_w/
<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
# Add qemu command line parameters to increase the mmio address space to 64MB -- ReBar Support (within Domain, after devices)
# REF: https://www.reddit.com/r/VFIO/comments/qx4rg7/can_anyone_else_confirm_that_vfio_doesnt_work_w/
<qemu:commandline>
<qemu:arg value='-fw_cfg'/>
<qemu:arg value='opt/ovmf/X-PciMmio64Mb,string=65536'/>
</qemu:commandline>
The following can also be done within VMM XML Editor
# Pin the VCPUs to PCPUs in the same CCD (within devices)
# REF: https://looking-glass.io/wiki/Performance_Optimization
<cputune>
<vcpupin vcpu='0' cpuset='8'/>
<vcpupin vcpu='1' cpuset='9'/>
<vcpupin vcpu='2' cpuset='10'/>
<vcpupin vcpu='3' cpuset='11'/>
<vcpupin vcpu='4' cpuset='12'/>
<vcpupin vcpu='5' cpuset='13'/>
<vcpupin vcpu='6' cpuset='14'/>
<vcpupin vcpu='7' cpuset='15'/>
<vcpupin vcpu='8' cpuset='40'/>
<vcpupin vcpu='9' cpuset='41'/>
<vcpupin vcpu='10' cpuset='42'/>
<vcpupin vcpu='11' cpuset='43'/>
<vcpupin vcpu='12' cpuset='44'/>
<vcpupin vcpu='13' cpuset='45'/>
<vcpupin vcpu='14' cpuset='46'/>
<vcpupin vcpu='15' cpuset='47'/>
</cputune>
# Create the shared memory space for looking glass (within devices)
# REF: https://looking-glass.io/docs/B5.0.1/install/
<shmem name='looking-glass'>
<model type='ivshmem-plain'/>
<size unit='M'>32</size>
</shmem>
# Enable HugePages memory backing (within devices)
# REF: https://looking-glass.io/wiki/Performance_Optimization
<memoryBacking>
<hugepages/>
</memoryBacking>
# Enable shared clipboard (within devices)
# REF: https://looking-glass.io/docs/B5.0.1/install/#client-determining-memory
<channel type="spicevmc">
<target type="virtio" name="com.redhat.spice.0"/>
<address type="virtio-serial" controller="0" bus="0" port="1"/>
</channel>
# Disable membaloon (within devices)
# REF: https://looking-glass.io/docs/B5.0.1/install/
<memballoon model="none"/>
Install Guest
Post Install
- Find your <video> device, and set <model type=‘none’/>
- Remove the <input type=‘tablet’/> device, if you have one
- Create an <input type=‘keyboard’ bus=‘virtio’/> device to improve keyboard usage
- This requires the vioinput driver from virtio-win to be installed in the guest
Automate GPU transition to/from host/guest
To simplify transitioning from GPU host usage to VM usage, a couple scripts may help. Should be easiy enough to modify for your use case. (review the VARS)
Assign Device to VFIO and Launch VM
#! /bin/bash
###############
# DEFINE VARS #
###############
PCI_ID='0000:4b:00.0'
VM_NAME='win11'
DRIVER_GPU='nvidia'
DRIVER_VFIO='vfio-pci'
REQUIRED_HUGEPAGES=32768
######################
# ASSIGN GPU TO VFIO #
######################
echo "Assigning ${PCI_ID} to VFIO:"
if [[ -e "/sys/bus/pci/devices/${PCI_ID}/driver" ]] ; then
CURRENT_DRIVER=`readlink "/sys/bus/pci/devices/${PCI_ID}/driver" | sed 's/.*\/drivers\///'`
case ${CURRENT_DRIVER} in
${DRIVER_VFIO})
echo " -- ${PCI_ID} is already assigned driver (${DRIVER_VFIO}). Nothing to do."
;;
${DRIVER_GPU})
sudo tee /sys/bus/pci/drivers/${DRIVER_GPU}/unbind <<< ${PCI_ID} > /dev/null || echo " -- ERROR: Unable to unassign ${PCI_ID} from ${DRIVER_GPU}."
sudo tee /sys/bus/pci/drivers/${DRIVER_VFIO}/bind <<< ${PCI_ID} > /dev/null || echo " -- ERROR: Unable to assign ${PCI_ID} to ${DRIVER_VFIO}."
echo " -- ${PCI_ID} successfully assigned driver (${DRIVER_VFIO})."
;;
*)
echo " -- ERROR: Unexpected driver (${CURRENT_DRIVER}) assigned, no action taken."
;;
esac
else
sudo tee /sys/bus/pci/drivers/${DRIVER_VFIO}/bind <<< ${PCI_ID} > /dev/null || echo " -- ERROR: Unable to assign ${PCI_ID} to ${DRIVER_VFIO}."
echo " -- ${PCI_ID} successfully assigned driver (${DRIVER_VFIO})."
fi
####################
# ASSIGN HUGEPAGES #
####################
echo "Assigning HugePages:"
FREE_HUGEPAGES=`cat /proc/meminfo | grep HugePages_Free: | grep --only-matching '[[:digit:]]\+'`
if [[ ${FREE_HUGEPAGES} -lt ${REQUIRED_HUGEPAGES} ]]; then
NEEDED_HUGEPAGES=$((${REQUIRED_HUGEPAGES} - ${FREE_HUGEPAGES}))
echo " -- Reserving an additional ${NEEDED_HUGEPAGES} HugePages ($((NEEDED_HUGEPAGES * 2))MB)"
sudo tee /proc/sys/vm/nr_hugepages <<< ${NEEDED_HUGEPAGES} >>/dev/null || echo " -- ERROR: Unable to assign HugePages."
else
echo " -- No additional HugePages required (${FREE_HUGEPAGES} available)"
fi
#####################################
# START VM AND LAUNCH LOOKING-GLASS #
#####################################
echo "Launching VM (${VM_NAME}):"
echo " -- Starting VM (${VM_NAME}):"
VM_STATUS=`sudo virsh list --all | grep 'win11' | colrm 1 16`
if [[ ${VM_STATUS} == 'shut off' ]]; then
sudo virsh start ${VM_NAME} > /dev/null 2>&1 || echo " -- ERROR: Unable to start ${VM_NAME}"
else
echo " -- ${VM_NAME} is already running."
fi
echo " -- Launching Looking Glass:"
looking-glass-client -F -k -Q -S spice:captureOnStart > /dev/null 2>&1
######################
# VERIFY VM SHUTDOWN #
######################
echo "Waiting on VM (${VM_NAME}) to shutdown (MAX 60 Seconds)..."
i=0
VM_ONLINE=`sudo virsh list --all | grep 'win11' | colrm 1 16`
while [[ ${VM_ONLINE} != 'shut off' ]]; do
if [[ $i == 60 ]]; then
echo " -- ${VM_NAME} has not stopped, skipping remaining tasks"
exit
fi
sleep 1
VM_ONLINE=`sudo virsh list --all | grep 'win11' | colrm 1 16`
((i++))
done
echo " -- VM (${VM_NAME}) is now off."
##################
# FREE HUGEPAGES #
##################
echo -n 'Release Reserved HugePages? [Y,N]:'
#echo 'Releasing Reserved HugePages:'
read RELEASE_HUGEPAGES
case ${RELEASE_HUGEPAGES} in
'y' | 'Y')
TOTAL_HUGEPAGES=`cat /proc/meminfo | grep HugePages_Total: | grep --only-matching '[[:digit:]]\+'`
FREE_HUGEPAGES=`cat /proc/meminfo | grep HugePages_Free: | grep --only-matching '[[:digit:]]\+'`
REMAINING_HUGEPAGES=$((TOTAL_HUGEPAGES - FREE_HUGEPAGES))
echo " Releasing ${FREE_HUGEPAGES} HugePages ($((FREE_HUGEPAGES * 2))MB), ${REMAINING_HUGEPAGES} ($((REMAINING_HUGEPAGES *2 ))MB) HugePages remain in use. "
sudo tee /proc/sys/vm/nr_hugepages <<< ${REMAINING_HUGEPAGES} >>/dev/null || echo " -- ERROR: Unable to Free HugePages."
;;
'n' | 'N')
echo " -- Skipping HugePage Release"
;;
*)
echo " -- Response not understood, skipping HugePage Release"
;;
esac
######################
# ASSIGN GPU TO HOST #
######################
#echo 'Reassigning GPU to host:'
echo -n 'Reassign GPU to host? [Y,N]:'
read ASSIGN_TO_HOST
case ${ASSIGN_TO_HOST} in
'y' | 'Y')
echo "Assigning ${PCI_ID} to Host"
if [[ -e "/sys/bus/pci/devices/${PCI_ID}/driver" ]] ; then
CURRENT_DRIVER=`readlink "/sys/bus/pci/devices/${PCI_ID}/driver" | sed 's/.*\/drivers\///'`
case ${CURRENT_DRIVER} in
${DRIVER_GPU})
echo " -- ${PCI_ID} is already assigned driver (${DRIVER_GPU}). Nothing to do."
;;
${DRIVER_VFIO})
sudo tee /sys/bus/pci/drivers/${DRIVER_VFIO}/unbind <<< ${PCI_ID} > /dev/null || echo " -- ERROR: Unable to unassign ${PCI_ID} from ${DRIVER_VFIO}."
sudo tee /sys/bus/pci/drivers/${DRIVER_GPU}/bind <<< ${PCI_ID} > /dev/null || echo " -- ERROR: Unable to assign ${PCI_ID} to ${DRIVER_GPU}."
echo " -- ${PCI_ID} successfully assigned driver (${DRIVER_GPU})."
;;
*)
echo " -- ERROR: Unexpected driver (${CURRENT_DRIVER}) assigned, no action taken."
;;
esac
else
sudo tee /sys/bus/pci/drivers/${DRIVER_GPU}/bind <<< ${PCI_ID} > /dev/null || echo " -- ERROR: Unable to assign ${PCI_ID} to ${DRIVER_GPU}."
echo " -- ${PCI_ID} successfully assigned driver (${DRIVER_GPU})."
fi
;;
'n' | 'N')
echo " -- Skipping GPU resaaignment to host"
;;
*)
echo " -- Response not understood, skipping GPU resaaignment to host"
;;
esac
Assign Device to HOST
#! /bin/bash
###############
# DEFINE VARS #
###############
PCI_ID='0000:4b:00.0'
VM_NAME='win11'
DRIVER_GPU='nvidia'
DRIVER_VFIO='vfio-pci'
######################
# ASSIGN GPU TO HOST #
######################
echo "Assigning ${PCI_ID} to Host:"
if [[ -e "/sys/bus/pci/devices/${PCI_ID}/driver" ]] ; then
CURRENT_DRIVER=`readlink "/sys/bus/pci/devices/${PCI_ID}/driver" | sed 's/.*\/drivers\///'`
case ${CURRENT_DRIVER} in
${DRIVER_GPU})
echo " -- ${PCI_ID} is already assigned driver (${DRIVER_GPU}). Nothing to do."
;;
${DRIVER_VFIO})
sudo tee /sys/bus/pci/drivers/${DRIVER_VFIO}/unbind <<< ${PCI_ID} > /dev/null || echo " -- ERROR: Unable to unassign ${PCI_ID} from ${DRIVER_VFIO}."
sudo tee /sys/bus/pci/drivers/${DRIVER_GPU}/bind <<< ${PCI_ID} > /dev/null || echo " -- ERROR: Unable to assign ${PCI_ID} to ${DRIVER_GPU}."
echo " -- ${PCI_ID} successfully assigned driver (${DRIVER_GPU})."
;;
*)
echo " -- ERROR: Unexpected driver (${CURRENT_DRIVER}) assigned, no action taken."
;;
esac
else
sudo tee /sys/bus/pci/drivers/${DRIVER_GPU}/bind <<< ${PCI_ID} > /dev/null || echo " -- ERROR: Unable to assign ${PCI_ID} to ${DRIVER_GPU}."
echo " -- ${PCI_ID} successfully assigned driver (${DRIVER_GPU})."
fi
Assign Device to VFIO
#! /bin/bash
###############
# DEFINE VARS #
###############
PCI_ID='0000:4b:00.0'
VM_NAME='win11'
DRIVER_GPU='nvidia'
DRIVER_VFIO='vfio-pci'
######################
# ASSIGN GPU TO VFIO #
######################
echo "Assigning ${PCI_ID} to VFIO:"
if [[ -e "/sys/bus/pci/devices/${PCI_ID}/driver" ]] ; then
CURRENT_DRIVER=`readlink "/sys/bus/pci/devices/${PCI_ID}/driver" | sed 's/.*\/drivers\///'`
case ${CURRENT_DRIVER} in
${DRIVER_VFIO})
echo " -- ${PCI_ID} is already assigned driver (${DRIVER_VFIO}). Nothing to do."
;;
${DRIVER_GPU})
sudo tee /sys/bus/pci/drivers/${DRIVER_GPU}/unbind <<< ${PCI_ID} > /dev/null || echo " -- ERROR: Unable to unassign ${PCI_ID} from ${DRIVER_GPU}."
sudo tee /sys/bus/pci/drivers/${DRIVER_VFIO}/bind <<< ${PCI_ID} > /dev/null || echo " -- ERROR: Unable to assign ${PCI_ID} to ${DRIVER_VFIO}."
echo " -- ${PCI_ID} successfully assigned driver (${DRIVER_VFIO})."
;;
*)
echo " -- ERROR: Unexpected driver (${CURRENT_DRIVER}) assigned, no action taken."
;;
esac
else
sudo tee /sys/bus/pci/drivers/${DRIVER_VFIO}/bind <<< ${PCI_ID} > /dev/null || echo " -- ERROR: Unable to assign ${PCI_ID} to ${DRIVER_VFIO}."
echo " -- ${PCI_ID} successfully assigned driver (${DRIVER_VFIO})."
fi
I hope this helps someone and let me know if there are any changes or improvements to the guide / scripts (i’m not a programmer by trade:))
Thanks,
Brian
