Cannot boot VFIO VM using q35

Hello,
Cannot use Q35 for some reason, i440FX works fine.
Using ACS patch.

Some facts :

  1. With I440FX i can pass only GPU’s video and everything works, with q35 qemu log shows error about bound device, so to silence that error i have to pass GPU’s audio too.

  2. But VM doesnt work, it boots, but CPU\MEM stats are unresponsive.

  3. If i remove PCI GPU, everything works with Spice video.

XML
> <domain type="kvm">
>   <name>win10-2</name>
>   <uuid>0be82354-5180-4d2e-b711-bfb3f1a722ab</uuid>
>   <metadata>
>     <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
>       <libosinfo:os id="http://microsoft.com/win/10"/>
>     </libosinfo:libosinfo>
>   </metadata>
>   <memory unit="KiB">4194304</memory>
>   <currentMemory unit="KiB">4194304</currentMemory>
>   <vcpu placement="static">3</vcpu>
>   <os>
>     <type arch="x86_64" machine="pc-q35-4.0.1">hvm</type>
>     <loader readonly="yes" type="pflash">/run/libvirt/nix-ovmf/OVMF_CODE.fd</loader>
>     <nvram>/var/lib/libvirt/qemu/nvram/win10-2_VARS.fd</nvram>
>     <boot dev="hd"/>
>   </os>
>   <features>
>     <acpi/>
>     <apic/>
>     <hyperv>
>       <relaxed state="on"/>
>       <vapic state="on"/>
>       <spinlocks state="on" retries="8191"/>
>     </hyperv>
>     <vmport state="off"/>
>   </features>
>   <cpu mode="host-model" check="partial">
>     <model fallback="allow"/>
>   </cpu>
>   <clock offset="localtime">
>     <timer name="rtc" tickpolicy="catchup"/>
>     <timer name="pit" tickpolicy="delay"/>
>     <timer name="hpet" present="no"/>
>     <timer name="hypervclock" present="yes"/>
>   </clock>
>   <on_poweroff>destroy</on_poweroff>
>   <on_reboot>restart</on_reboot>
>   <on_crash>destroy</on_crash>
>   <pm>
>     <suspend-to-mem enabled="no"/>
>     <suspend-to-disk enabled="no"/>
>   </pm>
>   <devices>
>     <emulator>/run/libvirt/nix-emulators/qemu-system-x86_64</emulator>
>     <disk type="file" device="disk">
>       <driver name="qemu" type="raw" cache="none" io="native"/>
>       <source file="/mnt/WINDOWS-CAGE/win10.img"/>
>       <target dev="sda" bus="sata"/>
>       <address type="drive" controller="0" bus="0" target="0" unit="0"/>
>     </disk>
>     <controller type="sata" index="0">
>       <address type="pci" domain="0x0000" bus="0x00" slot="0x1f" function="0x2"/>
>     </controller>
>     <controller type="pci" index="0" model="pcie-root"/>
>     <controller type="pci" index="1" model="pcie-root-port">
>       <model name="pcie-root-port"/>
>       <target chassis="1" port="0x10"/>
>       <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0" multifunction="on"/>
>     </controller>
>     <controller type="pci" index="2" model="pcie-root-port">
>       <model name="pcie-root-port"/>
>       <target chassis="2" port="0x11"/>
>       <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x1"/>
>     </controller>
>     <controller type="pci" index="3" model="pcie-root-port">
>       <model name="pcie-root-port"/>
>       <target chassis="3" port="0x12"/>
>       <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x2"/>
>     </controller>
>     <controller type="pci" index="4" model="pcie-root-port">
>       <model name="pcie-root-port"/>
>       <target chassis="4" port="0x13"/>
>       <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x3"/>
>     </controller>
>     <controller type="pci" index="5" model="pcie-root-port">
>       <model name="pcie-root-port"/>
>       <target chassis="5" port="0x14"/>
>       <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x4"/>
>     </controller>
>     <controller type="pci" index="6" model="pcie-root-port">
>       <model name="pcie-root-port"/>
>       <target chassis="6" port="0x15"/>
>       <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x5"/>
>     </controller>
>     <controller type="virtio-serial" index="0">
>       <address type="pci" domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
>     </controller>
>     <controller type="usb" index="0" model="ich9-ehci1">
>       <address type="pci" domain="0x0000" bus="0x00" slot="0x1d" function="0x7"/>
>     </controller>
>     <controller type="usb" index="0" model="ich9-uhci1">
>       <master startport="0"/>
>       <address type="pci" domain="0x0000" bus="0x00" slot="0x1d" function="0x0" multifunction="on"/>
>     </controller>
>     <controller type="usb" index="0" model="ich9-uhci2">
>       <master startport="2"/>
>       <address type="pci" domain="0x0000" bus="0x00" slot="0x1d" function="0x1"/>
>     </controller>
>     <controller type="usb" index="0" model="ich9-uhci3">
>       <master startport="4"/>
>       <address type="pci" domain="0x0000" bus="0x00" slot="0x1d" function="0x2"/>
>     </controller>
>     <interface type="network">
>       <mac address="52:54:00:9e:36:8f"/>
>       <source network="default"/>
>       <model type="e1000e"/>
>       <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
>     </interface>
>     <serial type="pty">
>       <target type="isa-serial" port="0">
>         <model name="isa-serial"/>
>       </target>
>     </serial>
>     <console type="pty">
>       <target type="serial" port="0"/>
>     </console>
>     <channel type="spicevmc">
>       <target type="virtio" name="com.redhat.spice.0"/>
>       <address type="virtio-serial" controller="0" bus="0" port="1"/>
>     </channel>
>     <input type="tablet" bus="usb">
>       <address type="usb" bus="0" port="1"/>
>     </input>
>     <input type="mouse" bus="ps2"/>
>     <input type="keyboard" bus="ps2"/>
>     <sound model="ich9">
>       <address type="pci" domain="0x0000" bus="0x00" slot="0x1b" function="0x0"/>
>     </sound>
>     <hostdev mode="subsystem" type="pci" managed="yes">
>       <source>
>         <address domain="0x0000" bus="0x02" slot="0x00" function="0x0"/>
>       </source>
>       <address type="pci" domain="0x0000" bus="0x04" slot="0x00" function="0x0" multifunction="on"/>
>     </hostdev>
>     <hostdev mode="subsystem" type="pci" managed="yes">
>       <source>
>         <address domain="0x0000" bus="0x02" slot="0x00" function="0x1"/>
>       </source>
>       <address type="pci" domain="0x0000" bus="0x04" slot="0x00" function="0x1"/>
>     </hostdev>
>     <redirdev bus="usb" type="spicevmc">
>       <address type="usb" bus="0" port="2"/>
>     </redirdev>
>     <redirdev bus="usb" type="spicevmc">
>       <address type="usb" bus="0" port="3"/>
>     </redirdev>
>     <memballoon model="virtio">
>       <address type="pci" domain="0x0000" bus="0x06" slot="0x00" function="0x0"/>
>     </memballoon>
>   </devices>
> </domain>

NOTE: PCI multifunction is my tests, without it nothing works too

I use virt-manager.

Wait you’re testing Pentium iis and core 2 duos?
Didn’t most early C2D chipsets have broken virtualization, surprised Piis could ever run a VM

Oh, that’s a part of trying everything on the internet found.

With native parameters it’s the same.

What is the output of /run/libvirt/nix-emulators/qemu-system-x86_64 -machine help?

Output

Supported machines are:
pc Standard PC (i440FX + PIIX, 1996) (alias of pc-i440fx-4.0)
pc-i440fx-4.0 Standard PC (i440FX + PIIX, 1996) (default)
pc-i440fx-3.1 Standard PC (i440FX + PIIX, 1996)
pc-i440fx-3.0 Standard PC (i440FX + PIIX, 1996)
pc-i440fx-2.9 Standard PC (i440FX + PIIX, 1996)
pc-i440fx-2.8 Standard PC (i440FX + PIIX, 1996)
pc-i440fx-2.7 Standard PC (i440FX + PIIX, 1996)
pc-i440fx-2.6 Standard PC (i440FX + PIIX, 1996)
pc-i440fx-2.5 Standard PC (i440FX + PIIX, 1996)
pc-i440fx-2.4 Standard PC (i440FX + PIIX, 1996)
pc-i440fx-2.3 Standard PC (i440FX + PIIX, 1996)
pc-i440fx-2.2 Standard PC (i440FX + PIIX, 1996)
pc-i440fx-2.12 Standard PC (i440FX + PIIX, 1996)
pc-i440fx-2.11 Standard PC (i440FX + PIIX, 1996)
pc-i440fx-2.10 Standard PC (i440FX + PIIX, 1996)
pc-i440fx-2.1 Standard PC (i440FX + PIIX, 1996)
pc-i440fx-2.0 Standard PC (i440FX + PIIX, 1996)
pc-i440fx-1.7 Standard PC (i440FX + PIIX, 1996)
pc-i440fx-1.6 Standard PC (i440FX + PIIX, 1996)
pc-i440fx-1.5 Standard PC (i440FX + PIIX, 1996)
pc-i440fx-1.4 Standard PC (i440FX + PIIX, 1996)
pc-1.3 Standard PC (i440FX + PIIX, 1996)
pc-1.2 Standard PC (i440FX + PIIX, 1996)
pc-1.1 Standard PC (i440FX + PIIX, 1996)
pc-1.0 Standard PC (i440FX + PIIX, 1996)
pc-0.15 Standard PC (i440FX + PIIX, 1996) (deprecated)
pc-0.14 Standard PC (i440FX + PIIX, 1996) (deprecated)
pc-0.13 Standard PC (i440FX + PIIX, 1996) (deprecated)
pc-0.12 Standard PC (i440FX + PIIX, 1996) (deprecated)
q35 Standard PC (Q35 + ICH9, 2009) (alias of pc-q35-4.0.1)
pc-q35-4.0.1 Standard PC (Q35 + ICH9, 2009)
pc-q35-4.0 Standard PC (Q35 + ICH9, 2009)
pc-q35-3.1 Standard PC (Q35 + ICH9, 2009)
pc-q35-3.0 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.9 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.8 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.7 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.6 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.5 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.4 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.12 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.11 Standard PC (Q35 + ICH9, 2009)
pc-q35-2.10 Standard PC (Q35 + ICH9, 2009)
isapc ISA-only PC
none empty machine

If passing GPU in VMM, you have removed all the video outs, like spice etc, so only video device would be the passed through GPU?
You should be getting at least default drivers. Adding the <kvm=hidden> later once first hurdle jumped

Make+model of GPU, Mobo and cpu might help in case anyone else hints?

I’ve said already, that with i440fx everything works just fine, so it’s a q35 specific issue. I want to work it out.

Ofc i’ve tried without Spice and everything, as i said, it doesnt work anyway.

CPU : i5-3570
Motherboard: Gigabyte GA-Z77X-UD3H
GPU: AMD R5 230

1 Like

UPDATE:

dmesg says :

[12936.881486] vfio-pci 0000:02:00.0: enabling device (0000 -> 0003)
[12936.881640] vfio_ecap_init: 0000:02:00.0 hiding ecap 0x19@0x270
[12936.881644] vfio_ecap_init: 0000:02:00.0 hiding ecap 0x1b@0x2d0
[12936.883962] vfio-pci 0000:02:00.1: enabling device (0000 -> 0002)
[13225.317368] vfio_ecap_init: 0000:02:00.0 hiding ecap 0x19@0x270
[13225.317373] vfio_ecap_init: 0000:02:00.0 hiding ecap 0x1b@0x2d0

Could it be “amd reset bug”, if it is, why only Q35? Strange.

SUCCESS:

Fixed by adding GPU via qemu:commandline, with creating pci-slot there too.

if I am not mistaken in order to use more modern q35 you need to add this:

-M q35,kernel_irqchip=on

and

ioh3420 instead of pcie-root-port

Getting it to work with i440FX doesn’t really count as a solution as it’s mad old.