[SOLVED] Help with Dual Nvidia GPU and Looking Glass

wallacebw · October 21, 2022, 6:33pm

I wrote myself a little guide and a few scripts to manage GPU passthrough and thought I would share.

NOTE: This is for arch and seems to work for me, YMMV

Configuration for VMM PCI Passthrough with NVidia GPU for host and VM

Setup Host

mkinitpcio.conf

Early Load VFIO before Nvidia drivers

REF: https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF

MODULES=(vfio_pci vfio vfio_iommu_type1 vfio_virqfd nvidia nvidia_modeset nvidia_uvm nvidia_drm)

Boot Loader

Add Device ID(s) of passthrough device(s) to VFIO driver in options.

REF: https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF

options   vfio-pci.ids=10de:2684,10de:22ba

Enable Hugepages

REF: https://looking-glass.io/wiki/Performance_Optimization

create a sysctl entry (/etc/sysctl.d/10-hugepages.conf) to create the hugepages (see REF for sizing)

vm.nr_hugepages=33000
vm.hugetlb_shm_group=992
sys.kernel.mm.transparent_hugepage.defrag='never'
sys.kernel.mm.transparent_hugepage.enabled='never'

add Hugepages to /etc/fstab to mount with appropriate permissions

 # HUGEPAGES
 hugetlbfs           /dev/hugepages  hugetlbfs       mode=01770,gid=kvm        0 0

Create a file /etc/tmpfiles.d/10-looking-glass.conf

# Shared Memory
f /dev/shm/looking-glass 0660 brian kvm -

Install Looking Glass Client on arch

REF: https://aur.archlinux.org/packages/looking-glass

Setup VMM (libvirt) VM

Pre-Install

After creating the VM update it’s XML

EDITOR=vim virsh edit VM_Name

# Change Domain tag to include Namespace
# REF: https://www.reddit.com/r/VFIO/comments/qx4rg7/can_anyone_else_confirm_that_vfio_doesnt_work_w/
<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>

# Add qemu command line parameters to increase the mmio address space to 64MB -- ReBar Support (within Domain, after devices)
# REF: https://www.reddit.com/r/VFIO/comments/qx4rg7/can_anyone_else_confirm_that_vfio_doesnt_work_w/
<qemu:commandline>
 <qemu:arg value='-fw_cfg'/>
 <qemu:arg value='opt/ovmf/X-PciMmio64Mb,string=65536'/>
</qemu:commandline>

The following can also be done within VMM XML Editor

# Pin the VCPUs to PCPUs in the same CCD (within devices)
# REF: https://looking-glass.io/wiki/Performance_Optimization
  <cputune>
    <vcpupin vcpu='0' cpuset='8'/>
    <vcpupin vcpu='1' cpuset='9'/>
    <vcpupin vcpu='2' cpuset='10'/>
    <vcpupin vcpu='3' cpuset='11'/>
    <vcpupin vcpu='4' cpuset='12'/>
    <vcpupin vcpu='5' cpuset='13'/>
    <vcpupin vcpu='6' cpuset='14'/>
    <vcpupin vcpu='7' cpuset='15'/>
    <vcpupin vcpu='8' cpuset='40'/>
    <vcpupin vcpu='9' cpuset='41'/>
    <vcpupin vcpu='10' cpuset='42'/>
    <vcpupin vcpu='11' cpuset='43'/>
    <vcpupin vcpu='12' cpuset='44'/>
    <vcpupin vcpu='13' cpuset='45'/>
    <vcpupin vcpu='14' cpuset='46'/>
    <vcpupin vcpu='15' cpuset='47'/>
  </cputune>

# Create the shared memory space for looking glass (within devices)
# REF: https://looking-glass.io/docs/B5.0.1/install/
<shmem name='looking-glass'>
  <model type='ivshmem-plain'/>
  <size unit='M'>32</size>
</shmem>

# Enable HugePages memory backing (within devices)
# REF: https://looking-glass.io/wiki/Performance_Optimization
  <memoryBacking>
    <hugepages/>
  </memoryBacking>

# Enable shared clipboard (within devices)
# REF: https://looking-glass.io/docs/B5.0.1/install/#client-determining-memory
<channel type="spicevmc">
  <target type="virtio" name="com.redhat.spice.0"/>
  <address type="virtio-serial" controller="0" bus="0" port="1"/>
</channel>

# Disable membaloon (within devices)
# REF: https://looking-glass.io/docs/B5.0.1/install/
<memballoon model="none"/>

Install Guest

Pass through the necessary devices (Video / Storage / Network)
- Add Hardware → PCI Host Device
Install the Guest OS as usual
Install Looking Glass Host (Looking Glass - Download Looking Glass)
Install the IddSampleDriver if you don’t want to attach a physical display
(GitHub - ge9/IddSampleDriver: Add virtual monitors to your windows 10 device! Works with Oculus software, obs, and any desktop sharing software)

Post Install

Find your <video> device, and set <model type=‘none’/>
Remove the <input type=‘tablet’/> device, if you have one
Create an <input type=‘keyboard’ bus=‘virtio’/> device to improve keyboard usage
- This requires the vioinput driver from virtio-win to be installed in the guest

Automate GPU transition to/from host/guest

To simplify transitioning from GPU host usage to VM usage, a couple scripts may help. Should be easiy enough to modify for your use case. (review the VARS)

Assign Device to VFIO and Launch VM

#! /bin/bash
###############
# DEFINE VARS #
###############
PCI_ID='0000:4b:00.0'
VM_NAME='win11'
DRIVER_GPU='nvidia'
DRIVER_VFIO='vfio-pci'
REQUIRED_HUGEPAGES=32768

######################
# ASSIGN GPU TO VFIO #
######################
echo "Assigning ${PCI_ID} to VFIO:"
if [[ -e "/sys/bus/pci/devices/${PCI_ID}/driver" ]] ; then
    CURRENT_DRIVER=`readlink "/sys/bus/pci/devices/${PCI_ID}/driver" | sed 's/.*\/drivers\///'`
    case ${CURRENT_DRIVER} in
        ${DRIVER_VFIO})
            echo " -- ${PCI_ID} is already assigned driver (${DRIVER_VFIO}).  Nothing to do."
            ;;
        ${DRIVER_GPU})
            sudo tee /sys/bus/pci/drivers/${DRIVER_GPU}/unbind <<< ${PCI_ID} > /dev/null || echo " -- ERROR: Unable to unassign ${PCI_ID} from ${DRIVER_GPU}."
            sudo tee /sys/bus/pci/drivers/${DRIVER_VFIO}/bind <<< ${PCI_ID} > /dev/null  || echo " -- ERROR: Unable to assign ${PCI_ID} to ${DRIVER_VFIO}."
            echo " -- ${PCI_ID} successfully assigned driver (${DRIVER_VFIO})."
            ;;
        *)
            echo " -- ERROR: Unexpected driver (${CURRENT_DRIVER}) assigned, no action taken."
            ;;
    esac
else
    sudo tee /sys/bus/pci/drivers/${DRIVER_VFIO}/bind <<< ${PCI_ID} > /dev/null    || echo " -- ERROR: Unable to assign ${PCI_ID} to ${DRIVER_VFIO}."
    echo " -- ${PCI_ID} successfully assigned driver (${DRIVER_VFIO})."
fi

####################
# ASSIGN HUGEPAGES #
####################
echo "Assigning HugePages:"
FREE_HUGEPAGES=`cat /proc/meminfo | grep HugePages_Free: | grep --only-matching '[[:digit:]]\+'`
if [[ ${FREE_HUGEPAGES} -lt ${REQUIRED_HUGEPAGES} ]]; then
    NEEDED_HUGEPAGES=$((${REQUIRED_HUGEPAGES} - ${FREE_HUGEPAGES}))
    echo " -- Reserving an additional ${NEEDED_HUGEPAGES} HugePages ($((NEEDED_HUGEPAGES * 2))MB)"
    sudo tee /proc/sys/vm/nr_hugepages <<< ${NEEDED_HUGEPAGES} >>/dev/null || echo " -- ERROR: Unable to assign HugePages."
else
    echo " -- No additional HugePages required (${FREE_HUGEPAGES} available)"
fi


#####################################
# START VM AND LAUNCH LOOKING-GLASS #
#####################################
echo "Launching VM (${VM_NAME}):"
echo " -- Starting VM (${VM_NAME}):"
VM_STATUS=`sudo virsh list --all | grep 'win11' | colrm 1 16`
if [[ ${VM_STATUS} == 'shut off'  ]]; then
    sudo virsh start ${VM_NAME}  > /dev/null 2>&1 || echo " -- ERROR: Unable to start ${VM_NAME}"
else
    echo " -- ${VM_NAME} is already running."
fi

echo " -- Launching Looking Glass:"
looking-glass-client -F -k -Q -S  spice:captureOnStart > /dev/null 2>&1


######################
# VERIFY VM SHUTDOWN #
######################
echo "Waiting on VM (${VM_NAME}) to shutdown (MAX 60 Seconds)..."
i=0
VM_ONLINE=`sudo virsh list --all | grep 'win11' | colrm 1 16`
while [[ ${VM_ONLINE} != 'shut off' ]]; do
    if [[ $i == 60 ]]; then
        echo " -- ${VM_NAME} has not stopped, skipping remaining tasks"
        exit
    fi
    sleep 1
    VM_ONLINE=`sudo virsh list --all | grep 'win11' | colrm 1 16`
    ((i++))
done
echo " -- VM (${VM_NAME}) is now off."


##################
# FREE HUGEPAGES #
##################
echo -n 'Release Reserved HugePages? [Y,N]:'
#echo 'Releasing Reserved HugePages:'
read RELEASE_HUGEPAGES
case ${RELEASE_HUGEPAGES} in
    'y' | 'Y')
        TOTAL_HUGEPAGES=`cat /proc/meminfo | grep HugePages_Total: | grep --only-matching '[[:digit:]]\+'`
        FREE_HUGEPAGES=`cat /proc/meminfo | grep HugePages_Free: | grep --only-matching '[[:digit:]]\+'`
        REMAINING_HUGEPAGES=$((TOTAL_HUGEPAGES - FREE_HUGEPAGES))
        echo "  Releasing ${FREE_HUGEPAGES} HugePages ($((FREE_HUGEPAGES * 2))MB), ${REMAINING_HUGEPAGES} ($((REMAINING_HUGEPAGES *2 ))MB) HugePages remain in use. "
        sudo tee /proc/sys/vm/nr_hugepages <<< ${REMAINING_HUGEPAGES} >>/dev/null || echo " -- ERROR: Unable to Free HugePages."
        ;;
    'n' | 'N')
        echo " -- Skipping HugePage Release"
        ;;
    *)
        echo " -- Response not understood, skipping HugePage Release"
        ;;
esac


######################
# ASSIGN GPU TO HOST #
######################
#echo 'Reassigning GPU to host:'
echo -n 'Reassign GPU to host? [Y,N]:'
read ASSIGN_TO_HOST
case ${ASSIGN_TO_HOST} in
    'y' | 'Y')
        echo "Assigning ${PCI_ID} to Host"
        if [[ -e "/sys/bus/pci/devices/${PCI_ID}/driver" ]] ; then
            CURRENT_DRIVER=`readlink "/sys/bus/pci/devices/${PCI_ID}/driver" | sed 's/.*\/drivers\///'`
            case ${CURRENT_DRIVER} in
                ${DRIVER_GPU})
                    echo " -- ${PCI_ID} is already assigned driver (${DRIVER_GPU}).  Nothing to do."
                    ;;
                ${DRIVER_VFIO})
                    sudo tee /sys/bus/pci/drivers/${DRIVER_VFIO}/unbind <<< ${PCI_ID} > /dev/null || echo " -- ERROR: Unable to unassign ${PCI_ID} from ${DRIVER_VFIO}."
                    sudo tee /sys/bus/pci/drivers/${DRIVER_GPU}/bind <<< ${PCI_ID} > /dev/null    || echo " -- ERROR: Unable to assign ${PCI_ID} to ${DRIVER_GPU}."
                    echo " -- ${PCI_ID} successfully assigned driver (${DRIVER_GPU})."
                    ;;
                *)
                    echo " -- ERROR: Unexpected driver (${CURRENT_DRIVER}) assigned, no action taken."
                    ;;
            esac
        else
            sudo tee /sys/bus/pci/drivers/${DRIVER_GPU}/bind <<< ${PCI_ID} > /dev/null    || echo " -- ERROR: Unable to assign ${PCI_ID} to ${DRIVER_GPU}."
            echo " -- ${PCI_ID} successfully assigned driver (${DRIVER_GPU})."
        fi
        ;;
    'n' | 'N')
        echo " -- Skipping GPU resaaignment to host"
        ;;
    *)
        echo " -- Response not understood, skipping GPU resaaignment to host"
        ;;
esac

Assign Device to HOST

#! /bin/bash
###############
# DEFINE VARS #
###############
PCI_ID='0000:4b:00.0'
VM_NAME='win11'
DRIVER_GPU='nvidia'
DRIVER_VFIO='vfio-pci'

######################
# ASSIGN GPU TO HOST #
######################
echo "Assigning ${PCI_ID} to Host:"
if [[ -e "/sys/bus/pci/devices/${PCI_ID}/driver" ]] ; then
    CURRENT_DRIVER=`readlink "/sys/bus/pci/devices/${PCI_ID}/driver" | sed 's/.*\/drivers\///'`
    case ${CURRENT_DRIVER} in
        ${DRIVER_GPU})
            echo " -- ${PCI_ID} is already assigned driver (${DRIVER_GPU}).  Nothing to do."
            ;;
        ${DRIVER_VFIO})
            sudo tee /sys/bus/pci/drivers/${DRIVER_VFIO}/unbind <<< ${PCI_ID} > /dev/null || echo " -- ERROR: Unable to unassign ${PCI_ID} from ${DRIVER_VFIO}."
            sudo tee /sys/bus/pci/drivers/${DRIVER_GPU}/bind <<< ${PCI_ID} > /dev/null    || echo " -- ERROR: Unable to assign ${PCI_ID} to ${DRIVER_GPU}."
            echo " -- ${PCI_ID} successfully assigned driver (${DRIVER_GPU})."
            ;;
        *)
            echo " -- ERROR: Unexpected driver (${CURRENT_DRIVER}) assigned, no action taken."
            ;;
    esac
else
    sudo tee /sys/bus/pci/drivers/${DRIVER_GPU}/bind <<< ${PCI_ID} > /dev/null    || echo " -- ERROR: Unable to assign ${PCI_ID} to ${DRIVER_GPU}."
    echo " -- ${PCI_ID} successfully assigned driver (${DRIVER_GPU})."
fi

Assign Device to VFIO

#! /bin/bash
###############
# DEFINE VARS #
###############
PCI_ID='0000:4b:00.0'
VM_NAME='win11'
DRIVER_GPU='nvidia'
DRIVER_VFIO='vfio-pci'

######################
# ASSIGN GPU TO VFIO #
######################
echo "Assigning ${PCI_ID} to VFIO:"
if [[ -e "/sys/bus/pci/devices/${PCI_ID}/driver" ]] ; then
    CURRENT_DRIVER=`readlink "/sys/bus/pci/devices/${PCI_ID}/driver" | sed 's/.*\/drivers\///'`
    case ${CURRENT_DRIVER} in
        ${DRIVER_VFIO})
            echo " -- ${PCI_ID} is already assigned driver (${DRIVER_VFIO}).  Nothing to do."
            ;;
        ${DRIVER_GPU})
            sudo tee /sys/bus/pci/drivers/${DRIVER_GPU}/unbind <<< ${PCI_ID} > /dev/null || echo " -- ERROR: Unable to unassign ${PCI_ID} from ${DRIVER_GPU}."
            sudo tee /sys/bus/pci/drivers/${DRIVER_VFIO}/bind <<< ${PCI_ID} > /dev/null    || echo " -- ERROR: Unable to assign ${PCI_ID} to ${DRIVER_VFIO}."
            echo " -- ${PCI_ID} successfully assigned driver (${DRIVER_VFIO})."
            ;;
        *)
            echo " -- ERROR: Unexpected driver (${CURRENT_DRIVER}) assigned, no action taken."
            ;;
    esac
else
    sudo tee /sys/bus/pci/drivers/${DRIVER_VFIO}/bind <<< ${PCI_ID} > /dev/null    || echo " -- ERROR: Unable to assign ${PCI_ID} to ${DRIVER_VFIO}."
    echo " -- ${PCI_ID} successfully assigned driver (${DRIVER_VFIO})."
fi

I hope this helps someone and let me know if there are any changes or improvements to the guide / scripts (i’m not a programmer by trade:))

Thanks,
Brian