VFIO in 2019 -- Pop!_OS How-To (General Guide though) [DRAFT]

@wendell Yeah, both the 970 and the audio. I resat it a slot below my host GPU, gave the boot script the new prefixes and things are different:

Error starting domain: internal error: Process exited prior to exec: libvirt: QEMU Driver error : Unable to stat /dev/dri/by-path/pci-0000:41:00.0-render: No such file or directory

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 75, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 111, in tmpcb
    callback(*args, **kwargs)
  File "/usr/share/virt-manager/virtManager/libvirtobject.py", line 66, in newfn
    ret = fn(self, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/domain.py", line 1400, in startup
    self._backend.create()
  File "/usr/lib/python3/dist-packages/libvirt.py", line 1080, in create
    if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
libvirt.libvirtError: internal error: Process exited prior to exec: libvirt: QEMU Driver error : Unable to stat /dev/dri/by-path/pci-0000:41:00.0-render: No such file or directory

Here’s my ls-iommu now

Maybe I’m writing my bind_vfio script incorrectly? Here’s mine:

#!/bin/sh
PREREQS=""
DEVS="0000:41:00.0 0000:41:00.1"
for DEV in $DEVS;
  do echo "vfio-pci" > /sys/bus/pci/devices/$DEV/driver_override
done

modprobe -i vfio-pci

Both the VGA and audio device are being used by vfio-pci too.

Can you let /proc and verify this exists? Says it can t find it so maybe it’s on a different bus now?

it doens’t exist in the folder, no. only pci-0000:42:00.0-card and pci-0000:42:00.0-render being the host gpu. :confused:

check other folders farther up it may be pci-0001:000:41 for example, or not via the dri path… wait does dri work if vfio is using the card?

use the /sys/bus/pci/devices path in your vm config?

unfortunately there’s no further folder in the dev/dri/by-path folder but those two entries, but i can find it in the /sys/bus/pci/devices path. For context I added the GPU via the virt-manager GUI, in the config file though, would I replace

<gl enable='no' rendernode='/dev/dri/by-path/pci-0000:41:00.0-render'/>

with

 <gl enable='no' rendernode='/sys/bus/pci/devices/0000:41:00.0'/>

?

ohhhhh

okay.

so I am not 100% surea bout your setup.

I think first for passthrough you cannot eanble gl … -s it part of like a spice config?

You want to pass through the entire device as-is, first, and make sure that’s working.

Something like:

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x0'/>
    </hostdev>

or you can use the gui to pick the devices to pass through?

maybe you can post your whole config.

So the device you have there with is meant to be like spice/not a pcie passthrough device.

That setup is just for the spice video device, which you really probably can remove?

that’s a big mistake :no_mouth: here’s my config, i’ve removed the spice display and QXL video adapter, and it’s unfortunately back to square 1 with booting and nothing happening: https://pastebin.com/raw/9LnMDzZs

does it still do 127 error when you turn off? nothing on the display physically attached to the 970?

you may need to externally reference/load your video bios for the 970.

still 127 error. I have a monitor plugged into the 970 via displayport but I’ve just found that my system is completely freezing when the display is turned on during boot… sheesh, i’ve got vfio, vfio_iommu_type1, vfio_virqfd, and vfio_pci in the initramfs-tools modules file and Nouveau is blacklisted but gdm still tries to load Nouveau

try loading the vbios for that card from a file?

I’m unsure how you’d do that, not Googling any well either :frowning: would you do that on host launch or guest launch?

This was success with a 970, can you make sure your setup is similar?

https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF

GPU rom dumping section. Though ultimately you may have to download one from e.g. techpowerup or somewhere like that.

Slim chance this is realted, but worth trying imho

1 Like

thanks for the support so far, I’ll try this later tonight and see if I can come with something tomorrow

So since last night, I’ve followed that guide with a new vm, I’ve managed to fix the blacklisting issue, eventually I started applying a vbios in the rom which started some apparmor complaints, that’s fixed, but now I’m still at square one: The VM boots, but the GPU does nothing. Here’s my current config

On the topic of the BIOS, I did try downgrading my BIOS to 1.50 and redoing it’s settings, but that’s also done nothing, is the latest BIOS any worth it?

whats the model/brand of card? I can try to replicate on 9 series cards. I noticed someone else posted yesterday with a similar issue.

GIGABYTE GTX 970 G1 Gaming

I don’t know what i’m doing wrong but i cant get my GPU to use the vfio drivers. I’m using Pop_Os 19.04

nano /etc/default/grub

If you change this file, run ‘update-grub’ afterwards to update

/boot/grub/grub.cfg.

For full documentation of the options in this file, see:

info -f grub -n ‘Simple configuration’

GRUB_DEFAULT=0
GRUB_TIMEOUT=0
GRUB_DISTRIBUTOR=lsb_release -i -s 2> /dev/null || echo Debian
GRUB_CMDLINE_LINUX_DEFAULT=“quiet splash intel_iommu=on”
GRUB_CMDLINE_LINUX=""

nano /etc/initramfs-tools/scripts/init-top/bind_vfio.sh

#!/bin/sh

PREREQS=""
DEVS=“0000:02:00.0 0000:02:00.1”
for DEV in $DEVS;
do echo “vfio-pci” > /sys/bus/pci/devices/$DEV/driver_override
done

modprobe -i vfio-pci

lspci -nnv

02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP106 [GeForce GTX 1060 6GB] [10de:1c03] (rev a1) (prog-if 00 [VGA controller])
Subsystem: Micro-Star International Co., Ltd. [MSI] GP106 [GeForce GTX 1 060 6GB] [1462:3283]
Flags: fast devsel, IRQ 10
Memory at f4000000 (32-bit, non-prefetchable) [disabled] [size=16M]
Memory at 2fe0000000 (64-bit, prefetchable) [disabled] [size=256M]
Memory at 2ff0000000 (64-bit, prefetchable) [disabled] [size=32M]
I/O ports at d000 [disabled] [size=128]
Expansion ROM at f5000000 [disabled] [size=512K]
Capabilities: [60] Power Management version 3
Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [78] Express Legacy Endpoint, MSI 00
Capabilities: [100] Virtual Channel
Capabilities: [250] Latency Tolerance Reporting
Capabilities: [128] Power Budgeting <?>
Capabilities: [420] Advanced Error Reporting
Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 < ?>
Capabilities: [900] #19
Kernel modules: nvidiafb, nouveau

02:00.1 Audio device [0403]: NVIDIA Corporation GP106 High Definition Audio Cont roller [10de:10f1] (rev a1)
Subsystem: Micro-Star International Co., Ltd. [MSI] GP106 High Definitio n Audio Controller [1462:3283]
Flags: bus master, fast devsel, latency 0, IRQ 15
Memory at f5080000 (32-bit, non-prefetchable) [size=16K]
Capabilities: [60] Power Management version 3
Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [78] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Kernel modules: snd_hda_intel

when I pass through my GPU (rx480) the VM instantly pauses and wont unpause.

I’m on the Asrock x399 Professional Gaming. BTW great tutorial, any word on when the TODOs will be done?

Greetings, I’m following your guide, and I must say that it’s very comprehensive. However, I’m having a problem: when I rebooted after configuring the bind_vfio.sh, I’m hit with a blank screen. Fortunately, switching to an earlier kernel lets me use my PC without any problem. My bind_vfio.sh looks like this:

PREREQS=""
DEVS=“0000:0b:00.0 0000:0b:00.1”
for DEV in $DEVS;
do echo “vfio-pci” > /sys/bus/pci/devices/$DEV/driver_override
done

modprobe -i vfio-pci

My pc has a ryzen 5 3600, a rx 480 (which I want to pass through to my vm) and a r7 260x. My motherboard is an asrock b450 steel legend. I’m quite sure I have followed every instruction up to that point.

@wendell I like to write comprehensive guides & tutorials. Would you have any problem with me using yours as a point of reference? I can credit you if you like.

I’m an active member over on the LTT Forums. I know >99% of their demographic won’t have a use for this but I’d like to spread the information none-the-less.