Passing through a RTX 2080 gpu with a USB-C port?

Operating Systems & Open Source Virtualization
1

Hey,

I have bit difficulties to passthrough a RTX2080 that has an USB-C port on it. I’m running PopOs as host and Windows 10 as guest. I can passthrough other GPUs that dont have USB-C ports. But with this RTX2080 I have difficulties. When I try to start up the VM in QEMU, I get the following error:

`(qemu) qemu-system-x86_64: -device vfio-pci,host=35:00.0,bus=port.1,multifunction=on: vfio 0000:35:00.0: group 45 is not viable`

So I noticed that eventhough I blacklisted everything related to the Nvidia card (vga controller, serial bus, audio, usb), still the host machine sees the USB port on the Nvidia card. Every other device related to the Nvidia card has ‘kernel driver in use: vfio-pci’ except for the USB. (Below the results from lspci -nnv):

35:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU104 [GeForce RTX 2080 Rev. A] [10de:1e87] (rev a1) (prog-if 00 [VGA controller])
Subsystem: Micro-Star International Co., Ltd. \[MSI\] TU104 \[GeForce RTX 2080 Rev. A\] \[1462:3728\]
Flags: fast devsel, IRQ 255
Memory at dc000000 (32-bit, non-prefetchable) \[size=16M\]
Memory at 90000000 (64-bit, prefetchable) \[size=256M\]
Memory at a0000000 (64-bit, prefetchable) \[size=32M\]
I/O ports at e000 \[size=128\]
Expansion ROM at dd000000 \[disabled\] \[size=512K\]
Capabilities: <access denied>
Kernel driver in use: vfio-pci
Kernel modules: nvidiafb, nouveau, nvidia\_drm, nvidia

35:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:10f8] (rev a1)
Subsystem: Micro-Star International Co., Ltd. \[MSI\] Device \[1462:3728\]
Flags: fast devsel, IRQ 255
Memory at dd080000 (32-bit, non-prefetchable) \[disabled\] \[size=16K\]
Capabilities: <access denied>
Kernel driver in use: vfio-pci
Kernel modules: snd\_hda\_intel

35:00.2 USB controller [0c03]: NVIDIA Corporation Device [10de:1ad8] (rev a1) (prog-if 30 [XHCI])
Subsystem: Micro-Star International Co., Ltd. \[MSI\] Device \[1462:3728\]
Flags: fast devsel, IRQ 80
Memory at a2000000 (64-bit, prefetchable) \[size=256K\]
Memory at a2040000 (64-bit, prefetchable) \[size=64K\]
Capabilities: <access denied>
Kernel driver in use: xhci\_hcd

35:00.3 Serial bus controller [0c80]: NVIDIA Corporation Device [10de:1ad9] (rev a1)
Subsystem: Micro-Star International Co., Ltd. \[MSI\] Device \[1462:3728\]
Flags: fast devsel, IRQ 255
Memory at dd084000 (32-bit, non-prefetchable) \[disabled\] \[size=4K\]
Capabilities: <access denied>
Kernel driver in use: vfio-pci
Kernel modules: i2c\_nvidia\_gpu

Then I tried to use a script to unbind the USB from github kholia OSX-KVM. This successfully changed the ‘kernel driver in use’ to ‘vfio-pci’ for the USB. But as a side effect it changed the VGA controller’s ‘kernel driver in use’ to ‘nvidia’. So now I cant passthrough the Nvidia card because the host now see the card, and doesnt allow to passthough it.

So I’m curious if anybody has successfully passthroughed a gpu with USB port? Or has encountered this same problem?

2

You don’t mention what cpu/mobo you’re using, but I’m on TRX40 and due to usb reset issues inherent to the chipset I read that the kernel boot option “pcie_aspm=off” should be used in addition to acs override. Couple of great threads here at L1 discuss it too.

Long story short I went the pcie usb card route since I had an open slot and still wasn’t able to get the VM to boot until I added “pcie_aspm=off” to my kernel options via kernelstub. I’m running Pop_OS 19.10 as host too.

Oh and just checked via lspci, my USB controller for the 2080ti shows xhci_hcd loaded by kernel.

3

Sorry I forgot to mention. I’m running Ryzen 3950X and ASRock X570 Creator (and I have separate USB pcie card that I will also passthrough for the guest).

Did you had to apply also acs override? Or just the boot option “pcie_aspm=off”?

Hmm are you btw starting your VM from Qemu command line or from the Virt manager? I’m starting from a Qemu command line with this script:

#!/bin/bash    
wmname="windows10vm"
    export QEMU_AUDIO_DRV=pa
    export QEMU_PA_SAMPLES=8192
    export QEMU_AUDIO_TIMER_PERIOD=99
    export QEMU_PA_SERVER=/run/user/1000/pulse/native

    cp /usr/share/OVMF/OVMF_VARS.fd /tmp/my_vars.fd

    qemu-system-x86_64 \
        -name $vmname,process=$vmname \
        -machine q35,accel=kvm \
        -cpu host,kvm=off \
        -smp 28,cores=14 \
        -m 48G \
        -rtc clock=host,base=localtime \
        -vga none \
        -nographic \
        -serial none \
        -parallel none \
        -device ich9-intel-hda -device hda-output \
        -usb -device usb-kbd -device usb-mouse \
        -drive if=pflash,format=raw,readonly,file=/usr/share/OVMF/OVMF_CODE.fd \
        -drive if=pflash,format=raw,file=/tmp/my_vars.fd \
        -boot order=dc \
        -drive id=disk0,if=virtio,cache=none,format=raw,file=WindowsVirtualMachine.img \
        -drive file=win10.iso,index=1,media=cdrom \
        -drive file=virtio.iso,index=2,media=cdrom \
        -netdev user,id=net0 \
        -device e1000-82545em,netdev=net0,id=net0,mac=52:54:00:c9:18:27 \
        -device pcie-root-port,bus=pcie.0,multifunction=on,port=1,chassis=1,id=port.1 \
        -device vfio-pci,host=35:00.0,bus=port.1,multifunction=on \
        -device vfio-pci,host=35:00.1,bus=port.1 \
        -device vfio-pci,host=35:00.2,bus=port.1 \
        -device vfio-pci,host=35:00.3,bus=port.1 \
        -device vfio-pci,host=03:00.0,bus=pcie.0 \

But if I try to run it, then I will encounter the error that was in my first message (and cant start the VM).

4

Oh, you’re using the old method.

Most people use libvirt now. libvirt will automatically attempt to unbind any PCI device you pass through, and rebind it when you shut down the VM. (assuming your modprobe config doesn’t restrict it)

My recommendation would be to use libvirt.

1 Like
5

Thanks! I now tried the “VFIO in 2020” guide just below, and got the GPU passthrough working! (I was following another guide before, so maybe got bit old info, but now working!)

1 Like
6

Glad to hear it’s working!

So I don’t really know about that other guide, but I haven’t really seen people using that method in a couple years. It is doable, but a lot of the tooling for non-exotic vm config is all built into libvirt now, so at that point it just seems like duplication of effort to me.

1 Like
7

It might just been that I was reading too many guides at same time. And also I was reading through some guides about how to set up Mac VM, and I think those were mostly written using the old method (kinda also interested to set up a Mac VM also). But I’m really glad that I get it now working. Thanks again for point me to the right direction!

8

I’d be tempted to write this guide, but the problem is that you need a magic string, and in order to get this, you need to either have a mac or get it from someone who has a mac. :confused: It’s also technically in violation of EULA to do a mac vm on non-apple hardware.

So I just don’t know if I want to open the forum to that liability.

9

Aa yeah, maybe then it can be bit dangerous to talk about
…

Btw have you or maybe anybody else encountered the error 43 with Nvidia cards? Or I moved my RTX card to the first pcie slot from the second slot, but now I’m encountering the error 43 on windows 10 guest. (I moved the RTX to the first pcie slot because of better cooling/placement)

I have already added the following to the VM’s xml:

<hyperv>
   ...
   <vendor_id state="on" value="abcdefgh"/>
   ...
</hyperv>
<kvm>      
     <hidden state='on'/>
</kvm>
<features>
   ...
   <ioapic driver="kvm"/>
</features>

But still I get the error 43. And I even tried to add the vBios rom-file (like in this instructions: https://passthroughpo.st/explaining-csm-efifboff-setting-boot-gpu-manually/ ). But then I encounter this following problem, when I try to launch the VM:

Error starting domain: internal error: connection closed due to keepalive timeout

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 75, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 111, in tmpcb
    callback(*args, **kwargs)
  File "/usr/share/virt-manager/virtManager/object/libvirtobject.py", line 66, in newfn
    ret = fn(self, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/object/domain.py", line 1279, in startup
    self._backend.create()
  File "/usr/lib/python3/dist-packages/libvirt.py", line 1080, in create
    if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
libvirt.libvirtError: internal error: connection closed due to keepalive timeout

Don’t really understand what is happening or what’s the problem.

(here’s the whole xml)

<domain type="kvm">
  <name>win10</name>
  <uuid>180397c6-e903-4618-9f94-7aa3e29cb345</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://microsoft.com/win/10"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit="KiB">33554432</memory>
  <currentMemory unit="KiB">33554432</currentMemory>
  <vcpu placement="static">24</vcpu>
  <iothreads>3</iothreads>
  <cputune>
    <vcpupin vcpu="0" cpuset="4"/>
    <vcpupin vcpu="1" cpuset="20"/>
    <vcpupin vcpu="2" cpuset="5"/>
    <vcpupin vcpu="3" cpuset="21"/>
    <vcpupin vcpu="4" cpuset="6"/>
    <vcpupin vcpu="5" cpuset="22"/>
    <vcpupin vcpu="6" cpuset="7"/>
    <vcpupin vcpu="7" cpuset="23"/>
    <vcpupin vcpu="8" cpuset="8"/>
    <vcpupin vcpu="9" cpuset="24"/>
    <vcpupin vcpu="10" cpuset="9"/>
    <vcpupin vcpu="11" cpuset="25"/>
    <vcpupin vcpu="12" cpuset="10"/>
    <vcpupin vcpu="13" cpuset="26"/>
    <vcpupin vcpu="14" cpuset="11"/>
    <vcpupin vcpu="15" cpuset="27"/>
    <vcpupin vcpu="16" cpuset="12"/>
    <vcpupin vcpu="17" cpuset="28"/>
    <vcpupin vcpu="18" cpuset="13"/>
    <vcpupin vcpu="19" cpuset="29"/>
    <vcpupin vcpu="20" cpuset="14"/>
    <vcpupin vcpu="21" cpuset="30"/>
    <vcpupin vcpu="22" cpuset="15"/>
    <vcpupin vcpu="23" cpuset="31"/>
    <emulatorpin cpuset="0-1"/>
    <iothreadpin iothread="1" cpuset="2-3"/>
    <iothreadpin iothread="2" cpuset="16-17"/>
    <iothreadpin iothread="3" cpuset="18-19"/>
  </cputune>
  <os>
    <type arch="x86_64" machine="pc-q35-4.0">hvm</type>
    <loader readonly="yes" type="pflash">/usr/share/OVMF/OVMF_CODE.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/win10_VARS.fd</nvram>
    <bootmenu enable="yes"/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state="on"/>
      <vapic state="on"/>
      <spinlocks state="on" retries="8191"/>
      <vpindex state="on"/>
      <synic state="on"/>
      <stimer state="on"/>
      <reset state="on"/>
      <vendor_id state="on" value="abcdefgh"/>
      <frequencies state="on"/>
    </hyperv>
    <kvm>
      <hidden state="on"/>
    </kvm>
    <vmport state="off"/>
    <ioapic driver="kvm"/>
  </features>
  <cpu mode="host-passthrough" check="none">
    <topology sockets="1" cores="12" threads="2"/>
    <cache mode="passthrough"/>
    <feature policy="require" name="topoext"/>
  </cpu>
  <clock offset="localtime">
    <timer name="rtc" tickpolicy="catchup"/>
    <timer name="pit" tickpolicy="delay"/>
    <timer name="hpet" present="no"/>
    <timer name="hypervclock" present="yes"/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled="no"/>
    <suspend-to-disk enabled="no"/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type="file" device="cdrom">
      <driver name="qemu" type="raw"/>
      <source file="/home/username/VMs/Media/Win10_1909_Finnish_x64.iso"/>
      <target dev="sdb" bus="sata"/>
      <readonly/>
      <boot order="1"/>
      <address type="drive" controller="0" bus="0" target="0" unit="1"/>
    </disk>
    <disk type="file" device="cdrom">
      <driver name="qemu" type="raw"/>
      <source file="/home/username/VMs/Media/virtio-win-0.1.171.iso"/>
      <target dev="sdc" bus="sata"/>
      <readonly/>
      <boot order="2"/>
      <address type="drive" controller="0" bus="0" target="0" unit="2"/>
    </disk>
    <disk type="file" device="disk">
      <driver name="qemu" type="raw"/>
      <source file="/home/username/VMs/win10.img"/>
      <target dev="vda" bus="virtio"/>
      <boot order="3"/>
      <address type="pci" domain="0x0000" bus="0x0a" slot="0x00" function="0x0"/>
    </disk>
    <controller type="usb" index="0" model="qemu-xhci" ports="15">
      <address type="pci" domain="0x0000" bus="0x02" slot="0x00" function="0x0"/>
    </controller>
    <controller type="sata" index="0">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x1f" function="0x2"/>
    </controller>
    <controller type="pci" index="0" model="pcie-root"/>
    <controller type="pci" index="1" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="1" port="0x10"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0" multifunction="on"/>
    </controller>
    <controller type="pci" index="2" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="2" port="0x11"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x1"/>
    </controller>
    <controller type="pci" index="3" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="3" port="0x12"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x2"/>
    </controller>
    <controller type="pci" index="4" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="4" port="0x13"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x3"/>
    </controller>
    <controller type="pci" index="5" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="5" port="0x14"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x4"/>
    </controller>
    <controller type="pci" index="6" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="6" port="0x15"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x5"/>
    </controller>
    <controller type="pci" index="7" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="7" port="0x16"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x6"/>
    </controller>
    <controller type="pci" index="8" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="8" port="0x17"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x7"/>
    </controller>
    <controller type="pci" index="9" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="9" port="0x18"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0" multifunction="on"/>
    </controller>
    <controller type="pci" index="10" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="10" port="0x19"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x1"/>
    </controller>
    <controller type="pci" index="11" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="11" port="0x1a"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x2"/>
    </controller>
    <controller type="pci" index="12" model="pcie-to-pci-bridge">
      <model name="pcie-pci-bridge"/>
      <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
    </controller>
    <controller type="pci" index="13" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="13" port="0x1b"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x3"/>
    </controller>
    <controller type="virtio-serial" index="0">
      <address type="pci" domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
    </controller>
    <interface type="network">
      <mac address="52:54:00:5c:38:24"/>
      <source network="default"/>
      <model type="rtl8139"/>
      <address type="pci" domain="0x0000" bus="0x0c" slot="0x01" function="0x0"/>
    </interface>
    <channel type="spicevmc">
      <target type="virtio" name="com.redhat.spice.0"/>
      <address type="virtio-serial" controller="0" bus="0" port="1"/>
    </channel>
    <input type="mouse" bus="ps2"/>
    <input type="keyboard" bus="ps2"/>
    <graphics type="spice" autoport="yes">
      <listen type="address"/>
      <image compression="off"/>
    </graphics>
    <sound model="ich9">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x1b" function="0x0"/>
    </sound>
    <video>
      <model type="qxl" ram="65536" vram="65536" vgamem="16384" heads="1" primary="yes"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x0"/>
    </video>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x08" slot="0x00" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x32" slot="0x00" function="0x0"/>
      </source>
      <rom file="/home/username/MSI.RTX2080.8192.181018_1.rom"/>
      <address type="pci" domain="0x0000" bus="0x04" slot="0x00" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x32" slot="0x00" function="0x1"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x32" slot="0x00" function="0x2"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x06" slot="0x00" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x32" slot="0x00" function="0x3"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x07" slot="0x00" function="0x0"/>
    </hostdev>
    <redirdev bus="usb" type="spicevmc">
      <address type="usb" bus="0" port="2"/>
    </redirdev>
    <redirdev bus="usb" type="spicevmc">
      <address type="usb" bus="0" port="3"/>
    </redirdev>
    <memballoon model="virtio">
      <address type="pci" domain="0x0000" bus="0x09" slot="0x00" function="0x0"/>
    </memballoon>
  </devices>
</domain>

(Although I dont know if I should make a new topic about this, as the problem isnt anymore related to the USB-C that I had previously)