Blank screen KVM PCI Passthrough win10

Software & Operating Systems VFIO
, ,
21

No problem! It’s what the community is for!

im not sure where to start :sweat_smile:

Arcwiki and passthroughpo.st has some steps detailing this (both posted in the thread) - You were already pretty close as you were able to boot into windows with the 2080 detected. (getting rid of code 43 was the next big step)

The journey into linux is a long and a winding one - But rewarding :slight_smile: And it gets easier every day… I remember the days of struggling with mouse acceleration and finally fixing it. (Archwiki is da bomb. Really)

22

So i managed to boot into windows 1 tim, and started downloading regular programs… However after 1 reboot im back to not booting into windows, and its yet again spitting code 43 at me. The only reason i know is because i yet again booted and nothing happened to the now 1080 screens. Here is the xml file.

<domain type="kvm">
  <name>win10</name>
  <uuid>d636b433-aca2-41f0-8960-cdacceb57520</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://microsoft.com/win/10"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit="KiB">16384000</memory>
  <currentMemory unit="KiB">16384000</currentMemory>
  <vcpu placement="static" current="31">32</vcpu>
  <os>
    <type arch="x86_64" machine="pc-q35-5.0">hvm</type>
    <loader readonly="yes" secure="yes" type="pflash">/usr/share/edk2-ovmf/x64/OVMF_CODE.secboot.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/win10_VARS.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state="on"/>
      <vapic state="on"/>
      <spinlocks state="on" retries="8191"/>
      <vendor_id state="on" value="whatever"/>
    </hyperv>
    <kvm>
      <hidden state="on"/>
    </kvm>
    <vmport state="off"/>
    <smm state="on"/>
  </features>
  <cpu mode="host-model" check="partial">
    <topology sockets="1" dies="1" cores="16" threads="2"/>
  </cpu>
  <clock offset="localtime">
    <timer name="rtc" tickpolicy="catchup"/>
    <timer name="pit" tickpolicy="delay"/>
    <timer name="hpet" present="no"/>
    <timer name="hypervclock" present="yes"/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled="no"/>
    <suspend-to-disk enabled="no"/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type="file" device="disk">
      <driver name="qemu" type="qcow2"/>
      <source file="/run/media/matrucious/VM/win10.qcow2"/>
      <target dev="vda" bus="virtio"/>
      <boot order="2"/>
      <address type="pci" domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
    </disk>
    <disk type="file" device="cdrom">
      <driver name="qemu" type="raw"/>
      <source file="/home/matrucious/Downloads/Win10_2004_English_x64.iso"/>
      <target dev="sdb" bus="sata"/>
      <readonly/>
      <boot order="1"/>
      <address type="drive" controller="0" bus="0" target="0" unit="1"/>
    </disk>
    <disk type="file" device="cdrom">
      <driver name="qemu" type="raw"/>
      <source file="/home/matrucious/Downloads/virtio-win-0.1.171.iso"/>
      <target dev="sdc" bus="sata"/>
      <readonly/>
      <address type="drive" controller="0" bus="0" target="0" unit="2"/>
    </disk>
    <controller type="usb" index="0" model="qemu-xhci" ports="15">
      <address type="pci" domain="0x0000" bus="0x02" slot="0x00" function="0x0"/>
    </controller>
    <controller type="sata" index="0">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x1f" function="0x2"/>
    </controller>
    <controller type="pci" index="0" model="pcie-root"/>
    <controller type="pci" index="1" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="1" port="0x8"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x0" multifunction="on"/>
    </controller>
    <controller type="pci" index="2" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="2" port="0x9"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x1"/>
    </controller>
    <controller type="pci" index="3" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="3" port="0xa"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x2"/>
    </controller>
    <controller type="pci" index="4" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="4" port="0xb"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x3"/>
    </controller>
    <controller type="pci" index="5" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="5" port="0xc"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x4"/>
    </controller>
    <controller type="pci" index="6" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="6" port="0xd"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x5"/>
    </controller>
    <controller type="pci" index="7" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="7" port="0xe"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x6"/>
    </controller>
    <controller type="pci" index="8" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="8" port="0xf"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x7"/>
    </controller>
    <controller type="pci" index="9" model="pcie-to-pci-bridge">
      <model name="pcie-pci-bridge"/>
      <address type="pci" domain="0x0000" bus="0x07" slot="0x00" function="0x0"/>
    </controller>
    <interface type="network">
      <mac address="52:54:00:9a:bf:ba"/>
      <source network="default"/>
      <model type="e1000e"/>
      <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
    </interface>
    <input type="tablet" bus="usb">
      <address type="usb" bus="0" port="4"/>
    </input>
    <input type="mouse" bus="ps2"/>
    <input type="keyboard" bus="ps2"/>
    <graphics type="spice" autoport="yes">
      <listen type="address"/>
      <image compression="off"/>
      <gl enable="no" rendernode="/dev/dri/by-path/pci-0000:0b:00.0-render"/>
    </graphics>
    <video>
      <model type="cirrus" vram="16384" heads="1" primary="yes"/>
      <address type="pci" domain="0x0000" bus="0x09" slot="0x01" function="0x0"/>
    </video>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x0c" slot="0x00" function="0x0"/>
      </source>
      <rom file="/home/matrucious/Documents/VM/NVIDIA.GTX1080.8192.160509.rom"/>
      <address type="pci" domain="0x0000" bus="0x04" slot="0x00" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x0c" slot="0x00" function="0x1"/>
      </source>
      <rom file="/home/matrucious/Documents/VM/NVIDIA.GTX1080.8192.160509.rom"/>
      <address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="usb" managed="yes">
      <source>
        <vendor id="0x1532"/>
        <product id="0x0221"/>
      </source>
      <address type="usb" bus="0" port="1"/>
    </hostdev>
    <hostdev mode="subsystem" type="usb" managed="yes">
      <source>
        <vendor id="0x046d"/>
        <product id="0xc539"/>
      </source>
      <address type="usb" bus="0" port="2"/>
    </hostdev>
    <hostdev mode="subsystem" type="usb" managed="yes">
      <source>
        <vendor id="0x1038"/>
        <product id="0x1294"/>
      </source>
      <address type="usb" bus="0" port="3"/>
    </hostdev>
    <redirdev bus="usb" type="spicevmc">
      <address type="usb" bus="0" port="5"/>
    </redirdev>
    <redirdev bus="usb" type="spicevmc">
      <address type="usb" bus="0" port="6"/>
    </redirdev>
    <memballoon model="virtio">
      <address type="pci" domain="0x0000" bus="0x06" slot="0x00" function="0x0"/>
    </memballoon>
  </devices>
</domain>

And here is

cat win10.log

LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/var/lib/snapd/snap/bin \
HOME=/var/lib/libvirt/qemu/domain-13-win10 \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-13-win10/.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-13-win10/.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-13-win10/.config \
QEMU_AUDIO_DRV=spice \
/usr/bin/qemu-system-x86_64 \
-name guest=win10,debug-threads=on \
-S \
-object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-13-win10/master-key.aes \
-blockdev '{"driver":"file","filename":"/usr/share/edk2-ovmf/x64/OVMF_CODE.secboot.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}' \
-blockdev '{"driver":"file","filename":"/var/lib/libvirt/qemu/nvram/win10_VARS.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"}' \
-machine pc-q35-5.0,accel=kvm,usb=off,vmport=off,smm=on,dump-guest-core=off,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format \
-cpu EPYC-IBPB,x2apic=on,tsc-deadline=on,hypervisor=on,tsc-adjust=on,clwb=on,umip=on,stibp=on,arch-capabilities=on,ssbd=on,cmp-legacy=on,perfctr-core=on,clzero=on,wbnoinvd=on,amd-ssbd=on,virt-ssbd=on,rdctl-no=on,skip-l1dfl-vmentry=on,mds-no=on,pschange-mc-no=on,monitor=off,hv-time,hv-relaxed,hv-vapic,hv-spinlocks=0x1fff,hv-vendor-id=whatever,kvm=off \
-global driver=cfi.pflash01,property=secure,value=on \
-m 16000 \
-overcommit mem-lock=off \
-smp 32,sockets=1,dies=1,cores=16,threads=2 \
-uuid d636b433-aca2-41f0-8960-cdacceb57520 \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=30,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=localtime,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-hpet \
-no-shutdown \
-global ICH9-LPC.disable_s3=1 \
-global ICH9-LPC.disable_s4=1 \
-boot strict=on \
-device pcie-root-port,port=0x8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1 \
-device pcie-root-port,port=0x9,chassis=2,id=pci.2,bus=pcie.0,addr=0x1.0x1 \
-device pcie-root-port,port=0xa,chassis=3,id=pci.3,bus=pcie.0,addr=0x1.0x2 \
-device pcie-root-port,port=0xb,chassis=4,id=pci.4,bus=pcie.0,addr=0x1.0x3 \
-device pcie-root-port,port=0xc,chassis=5,id=pci.5,bus=pcie.0,addr=0x1.0x4 \
-device pcie-root-port,port=0xd,chassis=6,id=pci.6,bus=pcie.0,addr=0x1.0x5 \
-device pcie-root-port,port=0xe,chassis=7,id=pci.7,bus=pcie.0,addr=0x1.0x6 \
-device pcie-root-port,port=0xf,chassis=8,id=pci.8,bus=pcie.0,addr=0x1.0x7 \
-device pcie-pci-bridge,id=pci.9,bus=pci.7,addr=0x0 \
-device qemu-xhci,p2=15,p3=15,id=usb,bus=pci.2,addr=0x0 \
-blockdev '{"driver":"file","filename":"/run/media/matrucious/VM/win10.qcow2","node-name":"libvirt-3-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-3-format","read-only":false,"driver":"qcow2","file":"libvirt-3-storage","backing":null}' \
-device virtio-blk-pci,bus=pci.3,addr=0x0,drive=libvirt-3-format,id=virtio-disk0,bootindex=2 \
-blockdev '{"driver":"file","filename":"/home/matrucious/Downloads/Win10_2004_English_x64.iso","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-2-format","read-only":true,"driver":"raw","file":"libvirt-2-storage"}' \
-device ide-cd,bus=ide.1,drive=libvirt-2-format,id=sata0-0-1,bootindex=1 \
-blockdev '{"driver":"file","filename":"/home/matrucious/Downloads/virtio-win-0.1.171.iso","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":true,"driver":"raw","file":"libvirt-1-storage"}' \
-device ide-cd,bus=ide.2,drive=libvirt-1-format,id=sata0-0-2 \
-netdev tap,fd=32,id=hostnet0 \
-device e1000e,netdev=hostnet0,id=net0,mac=52:54:00:9a:bf:ba,bus=pci.1,addr=0x0 \
-device usb-tablet,id=input0,bus=usb.0,port=4 \
-spice port=5900,addr=127.0.0.1,disable-ticketing,image-compression=off,seamless-migration=on \
-device cirrus-vga,id=video0,bus=pci.9,addr=0x1 \
-chardev spicevmc,id=charredir0,name=usbredir \
-device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=5 \
-chardev spicevmc,id=charredir1,name=usbredir \
-device usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=6 \
-device vfio-pci,host=0000:0c:00.0,id=hostdev0,bus=pci.4,addr=0x0,romfile=/home/matrucious/Documents/VM/NVIDIA.GTX1080.8192.160509.rom \
-device vfio-pci,host=0000:0c:00.1,id=hostdev1,bus=pci.5,addr=0x0 \
-device usb-host,hostbus=7,hostaddr=3,id=hostdev2,bus=usb.0,port=1 \
-device usb-host,hostbus=7,hostaddr=2,id=hostdev3,bus=usb.0,port=2 \
-device usb-host,hostbus=1,hostaddr=6,id=hostdev4,bus=usb.0,port=3 \
-device virtio-balloon-pci,id=balloon0,bus=pci.6,addr=0x0 \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on

First time i booted, it went without passing the vBIOS, i have now tried adding it, and adding it to both pci devices. I’m relly confused as to what changed during a reboot.

23

First time i booted, it went without passing the vBIOS, i have now tried adding it, and adding it to both pci devices. I’m relly confused as to what changed during a reboot.

I’d go without the vBios - as you indicated it worked w/o it. I’ve yet to find any references that I personally passed the vBios to the VM. (The memory is a bit fuzzy - It’s possible I manually flashed the (patched) vBios onto the card) But I do remember messing around with it…

Checking your xml file I found this:

source file="/home/matrucious/Downloads/Win10_2004_English_x64.iso"/
boot order = “1”

source file="/run/media/matrucious/VM/win10.qcow2"
boot order = “2”

It’s possible you’re being dropped to the UEFI menu - Referring to Archwiki again:

It is possible you will be dropped into the UEFI menu instead of starting the installation upon powering your VM for the first time. Sometimes the correct ISO file was not automatically detected and you will need to manually specify the drive to boot.

Just nuke the win10 installation medium from your xml or change the boot order :slight_smile: (You don’t need win10 installation iso mounted on your win10 installation tho :smiley:)

24

Yeah i forgot to change the boot order back, but it doesn’t seem to have made a difference. But i removed the vBIOS from the xml, and now when i boot this is all that comes up on the spice screen.

Screenshot_20200623_174807
Screenshot_20200623_1748071919×1079 69.6 KB

25

You’ve passed the 1080 in to the vm? Remove the spice screen and try to watch the screen output from your monitor connected to the 1080. It seems as close as it can get… That same Tianocore logo comes on when I boot…

26

Nothing happens when i remove spice and boot. The reason i added spice back again was to see what was going on, and that little what seems like hex code has never shown up before. It gets stuck like that… I am now going to try to remake the VM

27

Okay good luck. Make sure to refer to the archwiki guide: “Setting up an OVMF-based guest VM” I followed it to the letter and was able to get everything running correctly.

It sounds really weird - Because you have correctly bind the 1080 to the vfio-pci driver and passed it on to the vm - you should see the output of the vm from the 1080…

28

Yup. I though that was weird too. I’m in the middle of setup now. I am going to first try to install windows. and then install virt-io drivers. Then shut it down, add the 1080, and remove spice, and boot

29

Something is definitely wrong somewhere. You can see the cpu activity druing install… then you see the activity when i boot it up again with spice removed and the gpu added. I can try to flash the vBios using the steps in the Archwiki guide. to see if that helps

Screenshot_20200623_185815
Screenshot_20200623_1858152560×1395 83.3 KB

30

I have a problem with nvflash however… considering the fact that i’m using two nvidia gpus… it will not allow me to run

modprobe -r nvidia_modeset nvidia

or

rmmod nvidia

So i guess im stuck again?

31

It works after going in and installing the latest DCH driver, uninstalling the device, restarting the vm, then shutting it down… remove the vBIOS passthrough, and booting up again… now i will be terrified of updating anything :rofl:
It feels kinda like an old car, where you have to do a special routine just to get it to start up.
But hey it works now, dispite all the hiccups. Heres what the xml looks like now btw.

 <domain type="kvm">
      <name>win10</name>
      <uuid>95b97836-0676-47d8-b234-75fa569d8a1b</uuid>
      <metadata>
        <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
          <libosinfo:os id="http://microsoft.com/win/10"/>
        </libosinfo:libosinfo>
      </metadata>
      <memory unit="KiB">16384000</memory>
      <currentMemory unit="KiB">16384000</currentMemory>
      <vcpu placement="static" current="16">32</vcpu>
      <os>
        <type arch="x86_64" machine="pc-q35-5.0">hvm</type>
        <loader readonly="yes" type="pflash">/usr/share/edk2-ovmf/x64/OVMF_CODE.fd</loader>
        <nvram>/var/lib/libvirt/qemu/nvram/win10_VARS.fd</nvram>
      </os>
      <features>
        <acpi/>
        <apic/>
        <hyperv>
          <relaxed state="on"/>
          <vapic state="on"/>
          <spinlocks state="on" retries="8191"/>
          <vendor_id state="on" value="whatever"/>
        </hyperv>
        <kvm>
          <hidden state="on"/>
        </kvm>
        <vmport state="off"/>
        <ioapic driver="kvm"/>
      </features>
      <cpu mode="host-passthrough" check="partial">
        <topology sockets="1" dies="1" cores="16" threads="2"/>
      </cpu>
      <clock offset="localtime">
        <timer name="rtc" tickpolicy="catchup"/>
        <timer name="pit" tickpolicy="delay"/>
        <timer name="hpet" present="no"/>
        <timer name="hypervclock" present="yes"/>
      </clock>
      <on_poweroff>destroy</on_poweroff>
      <on_reboot>restart</on_reboot>
      <on_crash>destroy</on_crash>
      <pm>
        <suspend-to-mem enabled="no"/>
        <suspend-to-disk enabled="no"/>
      </pm>
      <devices>
        <emulator>/usr/bin/qemu-system-x86_64</emulator>
        <disk type="file" device="disk">
          <driver name="qemu" type="qcow2"/>
          <source file="/run/media/matrucious/VM/win10.qcow2"/>
          <target dev="vda" bus="virtio"/>
          <boot order="2"/>
          <address type="pci" domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
        </disk>
        <disk type="file" device="cdrom">
          <driver name="qemu" type="raw"/>
          <source file="/home/matrucious/Downloads/virtio-win-0.1.171.iso"/>
          <target dev="sdc" bus="sata"/>
          <readonly/>
          <address type="drive" controller="0" bus="0" target="0" unit="2"/>
        </disk>
        <controller type="usb" index="0" model="qemu-xhci" ports="15">
          <address type="pci" domain="0x0000" bus="0x02" slot="0x00" function="0x0"/>
        </controller>
        <controller type="sata" index="0">
          <address type="pci" domain="0x0000" bus="0x00" slot="0x1f" function="0x2"/>
        </controller>
        <controller type="pci" index="0" model="pcie-root"/>
        <controller type="pci" index="1" model="pcie-root-port">
          <model name="pcie-root-port"/>
          <target chassis="1" port="0x10"/>
          <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0" multifunction="on"/>
        </controller>
        <controller type="pci" index="2" model="pcie-root-port">
          <model name="pcie-root-port"/>
          <target chassis="2" port="0x11"/>
          <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x1"/>
        </controller>
        <controller type="pci" index="3" model="pcie-root-port">
          <model name="pcie-root-port"/>
          <target chassis="3" port="0x12"/>
          <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x2"/>
        </controller>
        <controller type="pci" index="4" model="pcie-root-port">
          <model name="pcie-root-port"/>
          <target chassis="4" port="0x13"/>
          <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x3"/>
        </controller>
        <controller type="pci" index="5" model="pcie-root-port">
          <model name="pcie-root-port"/>
          <target chassis="5" port="0x14"/>
          <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x4"/>
        </controller>
        <controller type="pci" index="6" model="pcie-root-port">
          <model name="pcie-root-port"/>
          <target chassis="6" port="0x8"/>
          <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x0" multifunction="on"/>
        </controller>
        <controller type="pci" index="7" model="pcie-root-port">
          <model name="pcie-root-port"/>
          <target chassis="7" port="0x9"/>
          <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x1"/>
        </controller>
        <controller type="pci" index="8" model="pcie-to-pci-bridge">
          <model name="pcie-pci-bridge"/>
          <address type="pci" domain="0x0000" bus="0x07" slot="0x00" function="0x0"/>
        </controller>
        <interface type="network">
          <mac address="52:54:00:2b:6e:20"/>
          <source network="default"/>
          <model type="e1000e"/>
          <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
        </interface>
        <input type="tablet" bus="usb">
          <address type="usb" bus="0" port="3"/>
        </input>
        <input type="mouse" bus="ps2"/>
        <input type="keyboard" bus="ps2"/>
        <graphics type="spice" autoport="yes">
          <listen type="address"/>
          <image compression="off"/>
          <gl enable="no" rendernode="/dev/dri/by-path/pci-0000:0b:00.0-render"/>
        </graphics>
        <video>
          <model type="cirrus" vram="16384" heads="1" primary="yes"/>
          <address type="pci" domain="0x0000" bus="0x08" slot="0x01" function="0x0"/>
        </video>
        <hostdev mode="subsystem" type="usb" managed="yes">
          <source>
            <vendor id="0x1532"/>
            <product id="0x0221"/>
          </source>
          <address type="usb" bus="0" port="1"/>
        </hostdev>
        <hostdev mode="subsystem" type="usb" managed="yes">
          <source>
            <vendor id="0x046d"/>
            <product id="0xc539"/>
          </source>
          <address type="usb" bus="0" port="2"/>
        </hostdev>
        <hostdev mode="subsystem" type="pci" managed="yes">
          <source>
            <address domain="0x0000" bus="0x0c" slot="0x00" function="0x0"/>
          </source>
          <address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0"/>
        </hostdev>
        <hostdev mode="subsystem" type="pci" managed="yes">
          <source>
            <address domain="0x0000" bus="0x0c" slot="0x00" function="0x1"/>
          </source>
          <address type="pci" domain="0x0000" bus="0x06" slot="0x00" function="0x0"/>
        </hostdev>
        <memballoon model="virtio">
          <address type="pci" domain="0x0000" bus="0x04" slot="0x00" function="0x0"/>
        </memballoon>
      </devices>
    </domain>

Thanks for sticky by stuff going wrong after the fact

32

I’m glad to hear it finally works! :smiley:

topology sockets=“1” dies=“1” cores=“16” threads=“2”

Jeesus… A bit jealous - I’d leave atleast 1 core to linux. You can do this by “core pinning” (Refer to archwiki) or simply change cores =“15” to drop 2 cores to linux :slight_smile:

33

I’m just passing through 16 out of 32 threads, but I will in the following days do a lot of optimizing. And pass through probably 31/32 threads, what you’re reading from there is just the topology, and not what i’m passing through. Thanks a lot AMD for the glorious 3950x.

34

I am glad you got it working.

Using vfio feels a bit like bending over backwards though, to achieve same thing as dual boot or other means. And to my surprise none of the games that I had working on wine work on the real m$$ installs )-: , but that’s entirely another exercise.

As an addendum to the post I’d like to share a discovery I did in regards to the issue of :

boot _vga

So I don’t know what they guides say about the boot gpu, which in some motherboards at least in last gen ones is the only pcie at x16 speeds. It’s normal for people wanting to use that one for m$$ passthrough and achieve max fps and what not even though many guides state it’s unnecessary as games won’t use max pcie bandwithd.

The problem that I faced in this particular mobo is that when changing PEG/PCI to PCI/PEG in the bios the first card and monitor would STILL initiate and keep an big UNDERSCORE in the upper left part.

At some point I managed to get it off by some ‘echo 0 >’ unbinding but at another reboot it didn’t work. And this is after going to the extreme and sacrifing the efi framebuffer module, that is, disabling altogether in the kernel and whatever else was touching the gpu cards, nouveau removed. This way I killed some flexibility in the linux box as I wont be able to use the nvidia gpu. But that’s momentarily in order to get vfio/gpu working.

Allright, so even after disabling all the modules touching the gpu card, even ‘snd-hda-intel’ (later I learned about the module order for vfio to take over nvidia snd-hda-intel) the pcie x16 would still get the big UNDERSCORE on the screen.

This was a big wtf. What, who from where is getting the underscore onto the screen?

Here come the important part where you could apply it to your 2080 and use it in frist pcie x16 slot for your VM.

DISCONNECT any hdmi/displayport/vga/etc cables from the 2080.

And this is how I got the ‘underscore’ DISAPEAR from the screen.

What does this mean? My hypothesis is that the BIOS doesn’t detect any monitor and switches the dreaded

boot_vga

Then after you’re on linux desktop, perform all the ‘echo 0 >’ unbding because the vtcon0 and vtcon1 stil seem to be used. I presume vtcon0 is the first gpu. Also unbind the efifb or simplefb. All this BEFORE plugging any cables into the first gpu.

Seems silly but plugging a cable into the gpu appears to initiate something on it.

Even weirdier, if after rebooting your VM and you lost output when at first was all good, simply TURN OFF and ON the monitor and m$$ will magically appear on the screen!!

I am sharing this so you could try using the first slot GPU without having to take apart the watercooling and put your 256bit bandwith GPU in an underperforming slot.

The new vfio seems fairly matured, I recall my first attempt at gpu passthrough had a similar situation and ended up switching slots for the gpu. After setting it all the uningine heaven demo was capped at 30fps, and all the games for that matter.

That’s why this time around I went on to try the first pcie x16 slot, to see whether I could get ~60fps.

Let me know if you try the first gpu slot and how it goes.

regards.

35

I will maybe make an attempt at this… however you’re definitely talking a bit above my head here with a few things.

You lost me in this general section… since I don’t even know what " ‘echo 0 >’ unbindings" are, and where to read up on them yet. So I will maybe attempt this when work clears up a bit so I have time to do some reading/tinkering.

Thanks for the tip though, I would of course love to get max performance out of my setup.

36

Sorry for the long winding post is it was more of a narrative of my discovery than instructions.

Using your 1 PCIE SLOT GPU is absolutely POSSIBLE, this is the premise. Although tricky at first it’s just a matter of following a procedure.

The bios attempts to use frist PCIE slot as its display, so somehow marks it as busy or the gpu turns into a busy state which makes it hard for linux to release it afterwards.

But it’s possible to trick the bios into ignoring the gpu in 1 pcie slot, so it looks elsewhere.

And here are the steps:

1- Shutdown the pc, turn off the PSU, long press the power button to release all the leftover juice.

2 - While the computer is OFF, unplug any monitor cable from the 1 slot pcie GPU. (this is very important because bios tries to probe display edid and whatnot initiating the gpu).

3- Turn on pc.

4- As a safe measure perform the following.

echo 0 > /sys/class/vtconsole/vtcon0/bind
echo 0 > /sys/class/vtconsole/vtcon1/bind
echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind

Sometimes simplefb is also there

find out if it’s using it:

find /sys/ -iname ‘simplefb’

5- Enjoy your GPU in 1 pcie slot (-: (now you can plug a cable into it)

Then check with GPUZ the link speed , also vfio in dmesg will throw messages about speed change.

From what I understand in linux there are three pieces that want to use the GPU.

1- plain console (old style output or bare metal)
2- framebuffer
3- GPU driver. (with its sound-hda-intel module)

regards.

2 Likes
37

Great summary and a lot more understandable. However im not sure if it will make any difference, since in the motherboard manual it says that whenever you run dual gpu both slot 1 and slot 2 will run in 8x mode. I will investigate this a bit further however