VGA passthrough to Legacy BIOS VM doesn't work (error 35)

I’m trying to boot Windows XP in KVM with VGA passthrough however it doesn’t work. Since Windows XP afaik doesn’t support UEFI I need to go with standard BIOS VM (not UEFI).

After booting VM GPU does not start correctly. Nothing shows up. In order to troubleshoot this issue I decided to grab Arch Linux and see what’s inside VM.

So getting it to do anything on i440x chipset is fairly challenging since VNC display doesn’t initialize and as such i need to blindly go through GRUB and then it still sometimes doesn’t switch to VNC display and console only shows “Guest has not initialized display” message. It’s a bit easier on Q35 which detects VNC as “first” GPU and shows TTY and GRUB there. VM does boot up and nothing extraordinary shows up in logs. GPU does fire up display but then it looks like this:

When OS is shutting down it does bring display diown before poweroff so I believe driver does try to perform some form of gentle shutdown but GPU is trashed after that first initialization. Any subsequent boot throws up nouveau errors during boot and GPU stays dead. It requires physical power cycle.

It’s worth to note that on UEFI+OVMF VMs it works just fine every single time, no issues at all. I can boot up, reboot, shutdown, start again and GPU just works.

I tried toggling ROM BAR but it doesn’t seem to do anything. I’m not sure if it’s better to keep it enabled or disabled.

Just to provide more context - I tried to do that before with 3090 with the same result but I kinda figured that maybe 3090 is just too recent GPU to work with 32 bit OS in legacy BIOS mode so this time I decided to plug my old Fermi Quadro 4000 (2GB). It still has Windows XP support so it should be somewhat okay for this use case.

Also please note that host OS driver never claims this GPU when pci-e goes back to host OS because current NVidia drivers do not support Fermi anymore.

Host workstation of course runs in UEFI mode. It’s WRX80 platform.

For now I’m trying x86_64 VM arch to simplify this case by matching host arch as it already doesn’t work but probably in final version it’d be preferable to run in full 32 bit mode (as long as it’s possible at all to reinitialize GPU into 32 bit guest from 64 bit host)

Host OS has Arch Linux with proprietary NVidia drivers but like I said they no longer support Fermi (that said i did not perform any vfio blacklisting on host)

When system boots up NVidia is visible under lspci


When GPU becomes brick nouveau fails to bring it up:

upon further investigation I realized that /dev/fb1 kind of works and I’m able to use apps that use simple framebuffer access and write directly to /dev/fb1. Unfortunately Windows was not able to properly use this device despite having drivers installed :confused: Nouveau also fails to run.

Here’s my XML

<domain type="kvm">
  <name>WinXP_KVM_oldpc_i440x</name>
  <uuid>1d975a60-99e8-466c-879e-b9f519905d68</uuid>
  <memory unit="KiB">4194304</memory>
  <currentMemory unit="KiB">4194304</currentMemory>
  <vcpu placement="static">2</vcpu>
  <os>
    <type arch="x86_64" machine="pc-i440fx-9.0">hvm</type>
  </os>
  <features>
    <acpi/>
    <apic/>
    <vmport state="off"/>
  </features>
  <cpu mode="host-model" check="partial"/>
  <clock offset="utc">
    <timer name="rtc" tickpolicy="catchup"/>
    <timer name="pit" tickpolicy="delay"/>
    <timer name="hpet" present="no"/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled="no"/>
    <suspend-to-disk enabled="no"/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type="file" device="cdrom">
      <driver name="qemu" type="raw"/>
      <source file="/iso/ubuntu-18.04.1-desktop-amd64.iso"/>
      <target dev="hdc" bus="ide"/>
      <readonly/>
      <boot order="1"/>
      <address type="drive" controller="0" bus="1" target="0" unit="0"/>
    </disk>
    <controller type="usb" index="0" model="ich9-ehci1">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x7"/>
    </controller>
    <controller type="usb" index="0" model="ich9-uhci1">
      <master startport="0"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0" multifunction="on"/>
    </controller>
    <controller type="usb" index="0" model="ich9-uhci2">
      <master startport="2"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x1"/>
    </controller>
    <controller type="usb" index="0" model="ich9-uhci3">
      <master startport="4"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x2"/>
    </controller>
    <controller type="pci" index="0" model="pci-root"/>
    <controller type="ide" index="0">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x1"/>
    </controller>
    <serial type="pty">
      <target type="isa-serial" port="0">
        <model name="isa-serial"/>
      </target>
    </serial>
    <console type="pty">
      <target type="serial" port="0"/>
    </console>
    <input type="mouse" bus="ps2"/>
    <input type="keyboard" bus="ps2"/>
    <graphics type="vnc" port="-1" autoport="yes">
      <listen type="address"/>
    </graphics>
    <audio id="1" type="none"/>
    <video>
      <model type="vga" vram="16384" heads="1" primary="yes"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x06" function="0x0"/>
    </video>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x9a" slot="0x00" function="0x0"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x9a" slot="0x00" function="0x1"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x04" function="0x0"/>
    </hostdev>
    <memballoon model="virtio">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x05" function="0x0"/>
    </memballoon>
  </devices>
</domain>

Okay so I found this thread on Reddit that suggests it’s no-do if host is booted into UEFI mode

https://www.reddit.com/r/VFIO/comments/uew6an/is_it_possible_to_do_a_gpu_passthrough_with/

My motherboard doesn’t post in Legacy BIOS mode so i could call it a day… But i ain’t done yet. Since I have this buddy laying around:

And apparently it doesn’t support UEFI so maybe it won’t initialize at all when host boots up?..

EDIT:

nvm, Quadro 4000 also doesn’t support UEFI