GPU passthough with Windows 7 guest - hangs on boot

[gotomech]

@ChuckH I’m thinking you’re running into a basic Windows 7 issue, it doesnt automatically install in UEFI. And the ISO’s from microsoft don’t do UEFI without a little trickery. And as far as I know, OVMF can’t do BIOS booting.

On top of that, Windows 7 doesnt like cpus Skylake and newer, due to Microsoft trying to push people to Win10. So, you may have better luck by telling the vm it is using an older architecture like haswell (you’ll lose some flags, but should still work well).

OR you can try turning your USB to 2 instead of 3, I saw a couple people talking about being able to use newer hardware on baremetal with that fix on newer hardware: https://www.sevenforums.com/installation-setup/398482-windows-7-x64-uefi-install-even-possible.html

But from what I’m seeing, you may have to put the iso on a USB to get uefi/ovmf to boot nicely.

Been a while since I’ve played with Win7 on a VM…maybe I’ll see if I can get mine to play nice and get back to you with any real answers I can find.

[ChuckH]

@Whizdumb I did try installing the driver for the gt710 while using spice/qxl and still didn’t have success. If I boot with both it shows the gt710 as code 12 (I think).

@gotomech Well it does boot with OVMF and the spice/qxl no problem so it should be working with efi right? I can try changing cpu type, usb is already set at 2.

I will try staring over using seabios later this week to see if I have any better luck.

[gotomech]

spice/qxl was working for me too with seabios, but I couldnt even get the iso to boot with ovmf/uefi. So, yea, if you can get it with ovmf with spice/qxl, then it’s a different issue.

You did the original install with spice/qxl with ovmf/uefi right?

[ChuckH]

Yeah I did the original install with spice/qxl and ovmf.

I can’t get the VM I set up to boot with seabios. The repair utility can’t find the drive. I’m going to have to start over with a new VM to try seabios.

[ChuckH]

Tried seabios real quick tonight and I cant get it to boot with the gpu passed through either. Different problem though, I don’t get any output on the display.

I will have to do more searching on that problem but it doesnt look like seabios was the silver bullet.

[gotomech]

As far as I know, seabios cant do GPU passthrough, I never thought that would work, but you should be able to use virtio video from spice and get 3d acceleration from your host card with all the virtio drivers installed.

As for ovmf, I couldnt even get the install to work for me, so it seems like the passthrough issue is something different.

[TheCakeIsNaOH]

@gotomech

seabios cant do GPU passthrough,

It can, but you have to deal with VGA arbitration.

[gotomech]

Actually, @ChuckH, code 12 is resource conflict. How did you lock down the gt710 for passthrough?

[ChuckH]

@gotomech bound the pci-ids by kernel parameters:

vfio-pci.ids=10de:128b,10de:0e0f

and it looks like it worked:

lspci -nnk -d 10de:128b
4c:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208B [GeForce GT 710] [10de:128b] (rev a1)
	Subsystem: ZOTAC International (MCO) Ltd. GK208B [GeForce GT 710] [19da:5360]
	Kernel driver in use: vfio-pci
	Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia

lspci -nnk -d 10de:0e0f
4c:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)
	Subsystem: ZOTAC International (MCO) Ltd. GK208 HDMI/DP Audio Controller [19da:5360]
	Kernel driver in use: vfio-pci
	Kernel modules: snd_hda_intel

The code 12 is only when I have the GT710 passed through and the spice/qxl added to the machine as well. From what I’ve read that is not uncommon with windows 7 guests.

@gotomech

but you should be able to use virtio video

I wasn’t able to get virtio to work. I couldn’t find drivers that worked with windows 7 so it was limited to 1024*768 with no acceleration. I tried vmvga too but couldn’t find drivers that worked for it either (VMware drivers are for newer emulated hardware).

@TheCakeIsNaOH I’ll look into vga arbitration with seabios. I didn’t try anything fancy with it yet.

[gotomech]

@ChuckH the bottom of this page: https://docs.fedoraproject.org/en-US/quick-docs/creating-windows-virtual-machines-using-virtio-drivers/ has the virtio drivers for all the different systems in an iso that you can mount in your VM.

If you run the virtio-win-gt-x64.exe (or x86 depending) at the root of the iso, it should install all virtio drivers for all possible virtio devices you can install, even if they arent currently installed.

[ChuckH]

I think I tried that one. I’ll try it again tonight to make sure, but I don’t think it has drivers for virtio video, just qxl.

[ChuckH]

I think it is something with my setup or hardware not windows 7 explicitly.

I tried a windows 10 vm and only got marginally better results.
With everything default, the I can get windows 10 to boot to the gpu, but its locked at low resolution and the card is Code 43.

I add:

    <hyperv>
      <vendor_id state="on" value="123456789ab"/>
    </hyperv>
    <kvm>
      <hidden state="on"/>
    </kvm>
    <ioapic driver="kvm"/>

and it hangs on boot.

I checked my GPU I am trying to pass, it does support efi:

./rom-parser vbios.img 
Valid ROM signature found @0h, PCIR offset 190h
	PCIR: type 0 (x86 PC-AT), vendor: 10de, device: 128b, class: 030000
	PCIR: revision 0, vendor revision: 1
Valid ROM signature found @f600h, PCIR offset 1ch
	PCIR: type 3 (EFI), vendor: 10de, device: 128b, class: 030000
	PCIR: revision 3, vendor revision: 0
		EFI: Signature Valid, Subsystem: Boot, Machine: X64
	Last image

I went back to just qxl/spice, uninstalled the nvidia drivers, and was able to get the VM to boot with all the above options. When I installed the driver I lost video. driver version 456.55.

[gotomech]

As weird as this will sound, I think your vendor_id value is too short (maybe), should be 12 characters. Always used “Something123”, with no issues, I had weird issues with mine for a while until I changed it to that from some other suggestions.

But try just adding an additional letter to your value and see if it helps. Or try mine and see if it helps.

Past that, I never tried <ioapic driver="kvm"/>

Here is my features section:

<features>
<acpi/>
<apic/>
<hyperv>
  <relaxed state="on"/>
  <vapic state="on"/>
  <spinlocks state="on" retries="8191"/>
  <vendor_id state="on" value="Something123"/>
</hyperv>
<kvm>
  <hidden state="on"/>
</kvm>
<vmport state="off"/>

[ChuckH]

Update:

I have gotten passthrough working with windows 10 guest and linux guest. It seems that manually setting your cpu model conflicts with the kvm hidden state configuration.

I could get Win10 and linux to boot and function with the card passed through using the qemu options from:

Nvidia GPU passthrough with Virtual Machine Manager – How to fix error 43?

NVidia does not want you to use their GTX cards with VM’s. You have too spoof it so that the driver does not detect the VM. Replace the top line in the XML with: <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> Then at the bottom, before the </domain> tag add: <qemu:commandline> <qemu:arg value='-cpu'/> <qemu:argvalue='host,hv_time,kvm=off,hv_vendor_id=null'/> </qemu:commandline> This should resolve the problem.

I didn’t even need all the qemu arguments called out either. Only -cpu host,kvm=off. The kvm=off should be taken care of in the virt-manager xml:

  <features>
    <kvm>
      <hidden state="on"/>
    </kvm>
  </features>

The only other difference was the host mode for the CPU (I had manual set EPYC).

I could remove the qemu lines from the xml and change my CPU section to:

  <cpu mode="host-passthrough" check="partial">
    <topology sockets="1" cores="8" threads="2"/>
  </cpu>

and it worked! (for windows 10 and linux guests).

Sadly, this doesn’t work on the windows 7 guest.

[ChuckH]

Well I’m back at working on this again.

I picked up a Quadro card to use for my VMs. Worked piece of cake with W10, don’t even need to hide the VM state or vendor id from the Quadro driver. Works so good I almost couldn’t believe it!

Windows 7, still no luck. I tried a few different things (combo that worked with Win10) but I get stuck at the “Staring Windows” logo. Back to searching I go… Maybe I’ll find something new.

EDIT: I setup RDP to try and connect to the VM to see if boot was successful. Looks like the “hang at boot” on the card passed through is not a hang. It boots and I can log in via RDP. Device manager shows the card as Code 12. No virtual video or spice server is setup on the VM. Kinda confused.

[ChuckH]

VICTORY!!! (I did not expect it to be this easy)

It is a bug that isn’t getting fixed because Windows 7 is EOL and the bug doesn’t affect newer versions of Windows.

Solution:
For virt-manger xml, change top line of xml to:
<domain xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0" type="kvm">

then add your graphics card devices at the end via QEMU commands:

 <qemu:commandline>
    <qemu:arg value="-device"/>
    <qemu:arg value="vfio-pci,host=4c:00.0,id=hostpci0.0,bus=pcie.0,addr=0x10.0,multifunction=on"/>
    <qemu:arg value="-device"/>
    <qemu:arg value="vfio-pci,host=4c:00.1,id=hostpci0.1,bus=pcie.0,addr=0x10.1"/>
    <qemu:arg value="-device"/>
    <qemu:arg value="vfio-pci,host=4c:00.2,id=hostpci0.2,bus=pcie.0,addr=0x10.2"/>
  </qemu:commandline>

On my card there were 4 devices that made up the graphics card on 0000:4c:00.
0 video
1 audio
2 usb 3.1 controller
3 serial bus controller for USB-C

I had to add one of the devices normally or it would give me an error and the VM would not start. I added the last device normally and all the others via qemu commands and it works perfectly!

Sources:

[SOLVED] - GPU passthrough Windows 7 - Error Code 12

Is anybody who succeeded GPU passthrough (NVIDIA) with Windows 7 ?

https://bugzilla.redhat.com/show_bug.cgi?format=multiple&id=1273172

[iotdev]

Can you share your xml config file , cos im trying to to do the same but not working im using qemu 6.

edit1 I was able to make it simply not using virt-manager and using only qemu it seems virt-manager generate some config that will not work with windows 7 efi , interesting is same config works with win 10 efi … as i read virt-manager generate some pci-e brige and that why video cards attached to this bridge not work . tryed to add manulay video card with this qemu parameters you give example but it give me access denied , that why i wana see your full xml config file if you can share it!

[ChuckH]

@iotdev sorry for the slow reply, I haven’t been on here much lately.

Below is my xml. I’m running qemu version 4.2.1 and virt manager 2.2.1 so YMMV with this.

<domain xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0" type="kvm">
  <name>Phillip-J</name>
  <uuid>------------------------------------</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://microsoft.com/win/7"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit="KiB">16777216</memory>
  <currentMemory unit="KiB">16777216</currentMemory>
  <vcpu placement="static">8</vcpu>
  <os>
    <type arch="x86_64" machine="pc-q35-4.2">hvm</type>
    <loader readonly="yes" type="pflash">/usr/share/OVMF/OVMF_CODE.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/Phillip-J_VARS.fd</nvram>
    <bootmenu enable="yes"/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state="on"/>
      <vapic state="on"/>
      <spinlocks state="on" retries="8191"/>
    </hyperv>
    <vmport state="off"/>
  </features>
  <cpu mode="host-passthrough" check="partial">
    <topology sockets="1" cores="4" threads="2"/>
    <feature policy="disable" name="hypervisor"/>
  </cpu>
  <clock offset="localtime">
    <timer name="rtc" tickpolicy="catchup"/>
    <timer name="pit" tickpolicy="delay"/>
    <timer name="hpet" present="no"/>
    <timer name="hypervclock" present="yes"/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled="no"/>
    <suspend-to-disk enabled="no"/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type="file" device="disk">
      <driver name="qemu" type="qcow2"/>
      <source file="/home/------/---------/Phillip-J.qcow2"/>
      <target dev="sda" bus="scsi"/>
      <boot order="1"/>
      <address type="drive" controller="0" bus="0" target="0" unit="0"/>
    </disk>
    <controller type="sata" index="0">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x1f" function="0x2"/>
    </controller>
    <controller type="pci" index="0" model="pcie-root"/>
    <controller type="pci" index="1" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="1" port="0x10"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0" multifunction="on"/>
    </controller>
    <controller type="pci" index="2" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="2" port="0x11"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x1"/>
    </controller>
    <controller type="pci" index="3" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="3" port="0x12"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x2"/>
    </controller>
    <controller type="pci" index="4" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="4" port="0x13"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x3"/>
    </controller>
    <controller type="pci" index="5" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="5" port="0x14"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x4"/>
    </controller>
    <controller type="pci" index="6" model="pcie-to-pci-bridge">
      <model name="pcie-pci-bridge"/>
      <address type="pci" domain="0x0000" bus="0x04" slot="0x00" function="0x0"/>
    </controller>
    <controller type="pci" index="7" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="7" port="0x15"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x5"/>
    </controller>
    <controller type="pci" index="8" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="8" port="0x16"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x6"/>
    </controller>
    <controller type="pci" index="9" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="9" port="0x17"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x7"/>
    </controller>
    <controller type="pci" index="10" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="10" port="0x18"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0" multifunction="on"/>
    </controller>
    <controller type="pci" index="11" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="11" port="0x8"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x0" multifunction="on"/>
    </controller>
    <controller type="pci" index="12" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="12" port="0x9"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x1"/>
    </controller>
    <controller type="pci" index="13" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="13" port="0xa"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x2"/>
    </controller>
    <controller type="pci" index="14" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="14" port="0xb"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x3"/>
    </controller>
    <controller type="pci" index="15" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="15" port="0xc"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x4"/>
    </controller>
    <controller type="pci" index="16" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="16" port="0xd"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x5"/>
    </controller>
    <controller type="pci" index="17" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="17" port="0xe"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x6"/>
    </controller>
    <controller type="pci" index="18" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="18" port="0xf"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x7"/>
    </controller>
    <controller type="pci" index="19" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="19" port="0x19"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x1"/>
    </controller>
    <controller type="pci" index="20" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="20" port="0x1a"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x2"/>
    </controller>
    <controller type="pci" index="21" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="21" port="0x1b"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x3"/>
    </controller>
    <controller type="pci" index="22" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="22" port="0x1c"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x4"/>
    </controller>
    <controller type="pci" index="23" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="23" port="0x1d"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x5"/>
    </controller>
    <controller type="pci" index="24" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="24" port="0x1e"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x6"/>
    </controller>
    <controller type="pci" index="25" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="25" port="0x1f"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x7"/>
    </controller>
    <controller type="pci" index="26" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="26" port="0x20"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x04" function="0x0" multifunction="on"/>
    </controller>
    <controller type="pci" index="27" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="27" port="0x21"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x04" function="0x1"/>
    </controller>
    <controller type="pci" index="28" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="28" port="0x22"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x04" function="0x2"/>
    </controller>
    <controller type="pci" index="29" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="29" port="0x23"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x04" function="0x3"/>
    </controller>
    <controller type="pci" index="30" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="30" port="0x24"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x04" function="0x4"/>
    </controller>
    <controller type="pci" index="31" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="31" port="0x25"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x04" function="0x5"/>
    </controller>
    <controller type="pci" index="32" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="32" port="0x26"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x04" function="0x6"/>
    </controller>
    <controller type="pci" index="33" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="33" port="0x27"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x04" function="0x7"/>
    </controller>
    <controller type="pci" index="34" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="34" port="0x28"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x05" function="0x0" multifunction="on"/>
    </controller>
    <controller type="pci" index="35" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="35" port="0x29"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x05" function="0x1"/>
    </controller>
    <controller type="pci" index="36" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="36" port="0x2a"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x05" function="0x2"/>
    </controller>
    <controller type="pci" index="37" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="37" port="0x2b"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x05" function="0x3"/>
    </controller>
    <controller type="pci" index="38" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="38" port="0x2c"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x05" function="0x4"/>
    </controller>
    <controller type="pci" index="39" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="39" port="0x2d"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x05" function="0x5"/>
    </controller>
    <controller type="pci" index="40" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="40" port="0x2e"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x05" function="0x6"/>
    </controller>
    <controller type="pci" index="41" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="41" port="0x2f"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x05" function="0x7"/>
    </controller>
    <controller type="pci" index="42" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="42" port="0x30"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x06" function="0x0" multifunction="on"/>
    </controller>
    <controller type="pci" index="43" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="43" port="0x31"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x06" function="0x1"/>
    </controller>
    <controller type="pci" index="44" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="44" port="0x32"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x06" function="0x2"/>
    </controller>
    <controller type="pci" index="45" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="45" port="0x33"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x06" function="0x3"/>
    </controller>
    <controller type="pci" index="46" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="46" port="0x34"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x06" function="0x4"/>
    </controller>
    <controller type="pci" index="47" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="47" port="0x35"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x06" function="0x5"/>
    </controller>
    <controller type="pci" index="48" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="48" port="0x36"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x06" function="0x6"/>
    </controller>
    <controller type="pci" index="49" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="49" port="0x37"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x06" function="0x7"/>
    </controller>
    <controller type="pci" index="50" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="50" port="0x38"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x07" function="0x0" multifunction="on"/>
    </controller>
    <controller type="pci" index="51" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="51" port="0x39"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x07" function="0x1"/>
    </controller>
    <controller type="pci" index="52" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="52" port="0x3a"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x07" function="0x2"/>
    </controller>
    <controller type="pci" index="53" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="53" port="0x3b"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x07" function="0x3"/>
    </controller>
    <controller type="pci" index="54" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="54" port="0x3c"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x07" function="0x4"/>
    </controller>
    <controller type="pci" index="55" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="55" port="0x3d"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x07" function="0x5"/>
    </controller>
    <controller type="pci" index="56" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="56" port="0x3e"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x07" function="0x6"/>
    </controller>
    <controller type="pci" index="57" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="57" port="0x3f"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x07" function="0x7"/>
    </controller>
    <controller type="pci" index="58" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="58" port="0x40"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x08" function="0x0" multifunction="on"/>
    </controller>
    <controller type="pci" index="59" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="59" port="0x41"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x08" function="0x1"/>
    </controller>
    <controller type="pci" index="60" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="60" port="0x42"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x08" function="0x2"/>
    </controller>
    <controller type="pci" index="61" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="61" port="0x43"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x08" function="0x3"/>
    </controller>
    <controller type="pci" index="62" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="62" port="0x44"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x08" function="0x4"/>
    </controller>
    <controller type="pci" index="63" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="63" port="0x45"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x08" function="0x5"/>
    </controller>
    <controller type="pci" index="64" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="64" port="0x46"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x08" function="0x6"/>
    </controller>
    <controller type="scsi" index="0" model="virtio-scsi">
      <address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0"/>
    </controller>
    <controller type="usb" index="0" model="ich9-ehci1">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x1d" function="0x7"/>
    </controller>
    <controller type="usb" index="0" model="ich9-uhci1">
      <master startport="0"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x1d" function="0x0" multifunction="on"/>
    </controller>
    <controller type="usb" index="0" model="ich9-uhci2">
      <master startport="2"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x1d" function="0x1"/>
    </controller>
    <controller type="usb" index="0" model="ich9-uhci3">
      <master startport="4"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x1d" function="0x2"/>
    </controller>
    <interface type="direct">
      <mac address="52:54:00:a5:03:9d"/>
      <source dev="enp69s0" mode="bridge"/>
      <model type="virtio"/>
      <link state="down"/>
      <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
    </interface>
    <serial type="pty">
      <target type="isa-serial" port="0">
        <model name="isa-serial"/>
      </target>
    </serial>
    <console type="pty">
      <target type="serial" port="0"/>
    </console>
    <input type="tablet" bus="usb">
      <address type="usb" bus="0" port="1"/>
    </input>
    <input type="mouse" bus="ps2"/>
    <input type="keyboard" bus="ps2"/>
    <sound model="ich9">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x1b" function="0x0"/>
    </sound>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x4d" slot="0x00" function="0x3"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x02" slot="0x00" function="0x0"/>
    </hostdev>
    <memballoon model="virtio">
      <address type="pci" domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
    </memballoon>
  </devices>
  <qemu:commandline>
    <qemu:arg value="-device"/>
    <qemu:arg value="vfio-pci,host=4d:00.0,id=hostpci0.0,bus=pcie.0,addr=0x10.0,multifunction=on"/>
    <qemu:arg value="-device"/>
    <qemu:arg value="vfio-pci,host=4d:00.1,id=hostpci0.1,bus=pcie.0,addr=0x10.1"/>
    <qemu:arg value="-device"/>
    <qemu:arg value="vfio-pci,host=4d:00.2,id=hostpci0.2,bus=pcie.0,addr=0x10.2"/>
  </qemu:commandline>
</domain>

I don’t know how I ended up with so many pcie devices, but I can confirm this still works for me.

Also note: I had to add one of the pci devices from my video card the way you normally would via virt-manager to get it to work. (At least i think that’s why i did it that way. Its been a while since I did this)

[BillFleming]

Iotdev,

Can you share your qemu command line for this?
Are yall able to get this to still work on Qemu 6.2?
Every combination I have tried changing pci buses in libvirt either got code 12 on AMD or no boot.
Couldn’t get any farther on command line qemu with 440fx as it seems they removed some of the ability to create a root pcie bus with ioh3420, but do have a new Q35 VM that is patched to boot with CSM off. So maybe I will have more luck with QEMU command line.

So currently either I have to use qemu from early 2020 or use an old AMD 2020 driver (only with 440FX VM) as everything newer gives me code 12.

[Mihkel_Kiil]

I’ve been trying the same thing as posted here, trying to get GPU passthrough to work with a Windows 7 guest. I’m using a GTX 970, Asus Q87M-E motherboard, and Ubuntu Server 20.04 (so no virt-manager). I encountered the code 12 error as before, and then tried the solution posted by @ChuckH . However, I now encounter an error: error: internal error: qemu unexpectedly closed the monitor: 2022-07-23T17:58:15.608209Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostpci0.0,bus=pcie.0,addr=0x10.0,multifunction=on,x-vga=on: vfio 0000:01:00.0: failed to open /dev/vfio/1: No such file or directory

Looking around, it seems to be because it can’t access the GPU from the host, and the posted solution is usually to isolate the GPU from the host. However, I have already done so, and under lspci -nnv I see vfio-pci as the driver that’s in use. Has anyone also encountered this error under these conditions?