Can't make vfio_pci claim specific gpu with uniform gpus (rtx 3000)

Software & Operating Systems VFIO
, ,
1

Hi,
I’m trying to set up pass through on a threadripper system with 3 rtx cards, however I can’t seem to make vfio-pci claim the two gpus intended for VMs.
All the guides I’ve found so far focus on either integrated graphics or two gpus from different manufacturers (or at least different series), so I’ve been trying to use the scripts provided by the arch wiki for uniform gpu systems. I did initially try to use the “normal” pci id method b/c the two gpus inteded for VMs are a different model than the 3rd, however they all have the same audio device id, causing the computer to hang while starting X11.

Here are the steps I’ve taken so far, I’d greatly appreciate some advice or a resource to guide me (I’m new to pcie passthrough).

  1. Blacklisted nouveau in /etc/modprobe.d/vfio.conf:
#blacklist snd_hda_intel
blacklist nouveau
#options vfio_pci ids=10de:2204,10de:1aef
  1. Added vfio_pci to /etc/modules-load.d/modules.conf:
# List of modules to load at boot
vfio_pci
  1. Created the /usr/local/bin/vfio-pci-override.sh script as specified in the arch wiki:
#!/bin/sh

DEVS="0000:49:00.0 0000:21:00.0"

if [ ! -z "$(ls -A /sys/class/iommu)" ]; then
    for DEV in $DEVS; do
        #echo "finding devices in $DEV's iommu group"
        for IOMMUDEV in $(ls /sys/bus/pci/devices/$DEV/iommu_group/devices) ; do
            #echo "setting $IOMMUDEV to vfio"
            echo "vfio-pci" > /sys/bus/pci/devices/$IOMMUDEV/driver_override
        done
    done
fi

modprobe -i vfio-pci
  1. Modified /etc/mkinitcpio.conf to include vfio-pci-override.sh in FILES and moved modconf to the end of the HOOKS array (comments removed for readability):
# vim:set ft=sh
# MODULES
MODULES=(vfio_pci vfio vfio_iommu_type1 vfio_virqfd)

# BINARIES
BINARIES=()

# FILES
FILES=(/usr/local/bin/vfio-pci-override.sh)

# HOOKS
HOOKS=(base udev autodetect block keyboard keymap plymouth filesystems modconf)
  1. Regenerated initramfs and updated grub:
sudo mkinitcpio -P
sudo update-grub
  1. I tried the video=efifb:off kernel parameter, with no change. Also tested the pci override script that selects all non-boot vga devices for vfio_pci, with improvement.

Hopefully I’ve missed an important step, as changing the cards in the system isn’t an option at the moment.

Also if I’m missing details from this point lmk and I’ll fix it.

Thanks

Edit: I’m running an up-to-date install of Manjaro

2

Well I found a solution browsing through seemingly unrelated virtualization posts here, and it turns out that I just failed to read a line in the arch wiki.

One must actually install the pci override script using the line install vfio-pci /usr/local/bin/vfio-pci-override.sh in the modprobe.d conf file.

Apologies for the extra post.

1 Like