First of all,
I’ve been a lurker for a very long time and I have a pet project that I’ve been working on and off for many years, VFIO. I’m not new to Linux, but it’s been a while since I’ve dived back into the environment on a primary system as I work with Windows systems in my day job.
As I was on and off I’ve been wanting to have a fledged out VFIO build for many years and kept experimenting with the idea, however there are guides that only pertain to one single OS and distro and others are so old they are no longer valid. There’s a huge divide in the VFIO guide community that there is a single massive problem, finding where the starting line is.
It get’s too much, you ingest content only to be let down that it’s 5 or 7 years old or you finally found a guide that’s recent (~1 year) but it’s for an entirely different OS you want to use.
Hear’s my plea, how can I make a windows guest that has GPU passthrough, on a physical storage drive that I can use via looking-glass on a close enough Arch based system Endeavor OS (fresh install, defaults, systemd)
CPU: AMD Ryzen 3900X
Memory: 64GB
Motherboard: ASUS X570-F
Storage: 2xNVMe (host, guest) 1xSSD (spare)
GPU: 6800XT (host), 7900XTX (guest)
From what I’ve researched, I don’t need a KVM if I can use looking-glass as with the display, sound and peripherals can be used as with host via evdev or spice.
So far, I’ve started here: wiki.archlinux(dot)org/title/PCI_passthrough_via_OVMF as a baseline but honestly the documentation is very confusing of what order you should read it, the issue I found is that most of the prerequisites don’t satisfy the dependencies that SHOULD be installed in kernel 6.9.4 (missing a lot of pre-built configs)
Have I completely messed up by not using grub? Or was the game rigged from the start?
Tech_janitor:
Hear’s my plea, how can I make a windows guest that has GPU passthrough, on a physical storage drive that I can use via looking-glass on a close enough Arch based system Endeavor OS (fresh install, defaults, systemd)
This guide is probably going to work for you. I wrote this a couple years back and have tweaked a few things last year that needed to be updated. In fact, you’re using a nearly identical system configuration to me;
When I built my system, I was running a 3900x, rx480, 6900xt, with NVMe passthrough for storage.
You can use systemd boot, You just have to change where you edit your config. (it’s in /boot/loader/)
Give the above link a read before we get into a back-and-forth about possible solutions to see if a solution jumps out there first though.
1 Like
Thanks for writing up this guide, however I did encounter several errors but you did note that it’s been over a year since an updated write-up.
The differences so far is I used dracut, there are CPU pinning errors which I’m trying to figure out even though we have the same CPU, there seems to be a weird error for Hyper-V synthetic interrupt and lastly a tricky network error which nothing is successful even if I create a vlan for that interface bound to primary, if I use the bridge mode following instructions it takes over my host network when created, so far you have to activate any network configuration via command line since virt-manager (ip ).
I figured out you have to autostart the default network under connection you are using qemu/KVM and the NAT interface works. Still figuring out the Hyper-V error that’s preventing to start the VM.
Following your instructions I stopped at the NVIDIA section as I should be able to install windows where I was and test the installation.
retox
June 22, 2024, 4:47pm
4
Hyper-V… are you trying to run docker or VMs inside your windows vm?
It sounds like your virtual CPU config is funky. Can the VM start if you remove/comment-out the CPU pinning? Can you share your XML config?
1 Like
<domain type="kvm">
<name>win11</name>
<uuid>58fc68fc-95ff-4bf4-864c-065dfc743b54</uuid>
<metadata>
<libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
<libosinfo:os id="http://microsoft.com/win/11"/>
</libosinfo:libosinfo>
</metadata>
<memory unit="KiB">16777216</memory>
<currentMemory unit="KiB">16777216</currentMemory>
<memoryBacking>
<hugepages/>
</memoryBacking>
<vcpu placement="static">12</vcpu>
<iothreads>2</iothreads>
<cputune>
<vcpupin vcpu="0" cpuset="1"/>
<vcpupin vcpu="1" cpuset="9"/>
<vcpupin vcpu="2" cpuset="2"/>
<vcpupin vcpu="3" cpuset="10"/>
<vcpupin vcpu="4" cpuset="3"/>
<vcpupin vcpu="5" cpuset="11"/>
<vcpupin vcpu="6" cpuset="4"/>
<vcpupin vcpu="7" cpuset="12"/>
<vcpupin vcpu="8" cpuset="5"/>
<vcpupin vcpu="9" cpuset="13"/>
<vcpupin vcpu="10" cpuset="6"/>
<vcpupin vcpu="11" cpuset="14"/>
<emulatorpin cpuset="7,15"/>
<iothreadpin iothread="1" cpuset="0,8"/>
<iothreadpin iothread="2" cpuset="7,15"/>
</cputune>
<os firmware="efi">
<type arch="x86_64" machine="pc-q35-9.0">hvm</type>
<firmware>
<feature enabled="no" name="enrolled-keys"/>
<feature enabled="no" name="secure-boot"/>
</firmware>
<loader readonly="yes" type="pflash">/usr/share/edk2/x64/OVMF_CODE.fd</loader>
<nvram template="/usr/share/edk2/x64/OVMF_VARS.fd">/var/lib/libvirt/qemu/nvram/win11_VARS.fd</nvram>
<boot dev="hd"/>
</os>
<features>
<acpi/>
<apic/>
<hyperv mode="custom">
<relaxed state="on"/>
<vapic state="on"/>
<spinlocks state="on" retries="8191"/>
<synic state="on"/>
<stimer state="on"/>
<reset state="on"/>
<frequencies state="on"/>
</hyperv>
<vmport state="off"/>
</features>
<cpu mode="host-passthrough" check="none" migratable="on"/>
<clock offset="localtime">
<timer name="rtc" tickpolicy="catchup"/>
<timer name="pit" tickpolicy="delay"/>
<timer name="hpet" present="no"/>
<timer name="hypervclock" present="yes"/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled="no"/>
<suspend-to-disk enabled="no"/>
</pm>
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<disk type="file" device="disk">
<driver name="qemu" type="qcow2" discard="unmap"/>
<source file="/var/lib/libvirt/images/win11.qcow2"/>
<target dev="vda" bus="virtio"/>
<address type="pci" domain="0x0000" bus="0x04" slot="0x00" function="0x0"/>
</disk>
<disk type="file" device="cdrom">
<driver name="qemu" type="raw"/>
<source file="/home/joshua/Downloads/Win11_23H2_English_x64v2.iso"/>
<target dev="sdb" bus="sata"/>
<readonly/>
<address type="drive" controller="0" bus="0" target="0" unit="1"/>
</disk>
<controller type="usb" index="0" model="qemu-xhci" ports="15">
<address type="pci" domain="0x0000" bus="0x02" slot="0x00" function="0x0"/>
</controller>
<controller type="pci" index="0" model="pcie-root"/>
<controller type="pci" index="1" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="1" port="0x10"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0" multifunction="on"/>
</controller>
<controller type="pci" index="2" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="2" port="0x11"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x1"/>
</controller>
<controller type="pci" index="3" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="3" port="0x12"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x2"/>
</controller>
<controller type="pci" index="4" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="4" port="0x13"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x3"/>
</controller>
<controller type="pci" index="5" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="5" port="0x14"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x4"/>
</controller>
<controller type="pci" index="6" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="6" port="0x15"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x5"/>
</controller>
<controller type="pci" index="7" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="7" port="0x16"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x6"/>
</controller>
<controller type="pci" index="8" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="8" port="0x17"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x7"/>
</controller>
<controller type="pci" index="9" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="9" port="0x18"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0" multifunction="on"/>
</controller>
<controller type="pci" index="10" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="10" port="0x19"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x1"/>
</controller>
<controller type="pci" index="11" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="11" port="0x1a"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x2"/>
</controller>
<controller type="pci" index="12" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="12" port="0x1b"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x3"/>
</controller>
<controller type="pci" index="13" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="13" port="0x1c"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x4"/>
</controller>
<controller type="pci" index="14" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="14" port="0x1d"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x5"/>
</controller>
<controller type="pci" index="15" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="15" port="0x1e"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x6"/>
</controller>
<controller type="pci" index="16" model="pcie-to-pci-bridge">
<model name="pcie-pci-bridge"/>
<address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
</controller>
<controller type="sata" index="0">
<address type="pci" domain="0x0000" bus="0x00" slot="0x1f" function="0x2"/>
</controller>
<controller type="virtio-serial" index="0">
<address type="pci" domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
</controller>
<interface type="network">
<mac address="52:54:00:a6:53:31"/>
<source network="default"/>
<model type="virtio"/>
<address type="pci" domain="0x0000" bus="0x09" slot="0x00" function="0x0"/>
</interface>
<serial type="pty">
<target type="isa-serial" port="0">
<model name="isa-serial"/>
</target>
</serial>
<console type="pty">
<target type="serial" port="0"/>
</console>
<channel type="spicevmc">
<target type="virtio" name="com.redhat.spice.0"/>
<address type="virtio-serial" controller="0" bus="0" port="1"/>
</channel>
<input type="tablet" bus="usb">
<address type="usb" bus="0" port="1"/>
</input>
<input type="mouse" bus="ps2"/>
<input type="keyboard" bus="ps2"/>
<graphics type="spice" autoport="yes">
<listen type="address"/>
<image compression="off"/>
</graphics>
<sound model="ich9">
<address type="pci" domain="0x0000" bus="0x00" slot="0x1b" function="0x0"/>
</sound>
<audio id="1" type="spice"/>
<video>
<model type="qxl" ram="65536" vram="65536" vgamem="16384" heads="1" primary="yes"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x0"/>
</video>
<hostdev mode="subsystem" type="pci" managed="yes">
<source>
<address domain="0x0000" bus="0x0e" slot="0x00" function="0x0"/>
</source>
<address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0"/>
</hostdev>
<hostdev mode="subsystem" type="pci" managed="yes">
<source>
<address domain="0x0000" bus="0x0e" slot="0x00" function="0x1"/>
</source>
<address type="pci" domain="0x0000" bus="0x06" slot="0x00" function="0x0"/>
</hostdev>
<hostdev mode="subsystem" type="pci" managed="yes">
<source>
<address domain="0x0000" bus="0x04" slot="0x00" function="0x0"/>
</source>
<address type="pci" domain="0x0000" bus="0x08" slot="0x00" function="0x0"/>
</hostdev>
<redirdev bus="usb" type="spicevmc">
<address type="usb" bus="0" port="2"/>
</redirdev>
<redirdev bus="usb" type="spicevmc">
<address type="usb" bus="0" port="3"/>
</redirdev>
<watchdog model="itco" action="reset"/>
<memballoon model="virtio">
<address type="pci" domain="0x0000" bus="0x07" slot="0x00" function="0x0"/>
</memballoon>
</devices>
</domain>If I remove the CPU pinning it doesn’t show that error, however the Hyper-V error is still persistent.
retox
June 23, 2024, 6:42am
7
I think you’re missing the <toplogy withing the <cpu/> section.
…From @SgtAwesomesauce ’s guide
You’ll also need to configure your CPU emulation settings as well. You need to set the mode to host-passthrough and specify the topology.cpu directive should already be there, just search for it and edit:
<cpu mode='host-passthrough' check='partial' migratable='off'>
<topology sockets='1' dies='1' cores='6' threads='2'/>
</cpu>
1 Like
Yep; if your topology isn’t correctly configured along with CPU-pinning, you’ll be in for a bunch of fun errors.
1 Like
Hmm, even when I put the topology in its still the same, shows that the cluster cannot be the value of 1?
XML:
<domain type="kvm">
<name>win11</name>
<uuid>58fc68fc-95ff-4bf4-864c-065dfc743b54</uuid>
<metadata>
<libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
<libosinfo:os id="http://microsoft.com/win/11"/>
</libosinfo:libosinfo>
</metadata>
<memory unit="KiB">16777216</memory>
<currentMemory unit="KiB">16777216</currentMemory>
<memoryBacking>
<hugepages/>
</memoryBacking>
<vcpu placement="static">12</vcpu>
<iothreads>2</iothreads>
<cputune>
<vcpupin vcpu="0" cpuset="1"/>
<vcpupin vcpu="1" cpuset="9"/>
<vcpupin vcpu="2" cpuset="2"/>
<vcpupin vcpu="3" cpuset="10"/>
<vcpupin vcpu="4" cpuset="3"/>
<vcpupin vcpu="5" cpuset="11"/>
<vcpupin vcpu="6" cpuset="4"/>
<vcpupin vcpu="7" cpuset="12"/>
<vcpupin vcpu="8" cpuset="5"/>
<vcpupin vcpu="9" cpuset="13"/>
<vcpupin vcpu="10" cpuset="6"/>
<vcpupin vcpu="11" cpuset="14"/>
<emulatorpin cpuset="7,15"/>
<iothreadpin iothread="1" cpuset="0,8"/>
<iothreadpin iothread="2" cpuset="7,15"/>
</cputune>
<os firmware="efi">
<type arch="x86_64" machine="pc-q35-9.0">hvm</type>
<firmware>
<feature enabled="no" name="enrolled-keys"/>
<feature enabled="no" name="secure-boot"/>
</firmware>
<loader readonly="yes" type="pflash">/usr/share/edk2/x64/OVMF_CODE.fd</loader>
<nvram template="/usr/share/edk2/x64/OVMF_VARS.fd">/var/lib/libvirt/qemu/nvram/win11_VARS.fd</nvram>
<boot dev="hd"/>
</os>
<features>
<acpi/>
<apic/>
<hyperv mode="custom">
<relaxed state="on"/>
<vapic state="on"/>
<spinlocks state="on" retries="8191"/>
<synic state="on"/>
<stimer state="on"/>
<reset state="on"/>
<frequencies state="on"/>
</hyperv>
<vmport state="off"/>
</features>
<cpu mode="host-passthrough" check="partial" migratable="off">
<topology sockets="1" dies="1" clusters="1" cores="6" threads="2"/>
</cpu>
<clock offset="localtime">
<timer name="rtc" tickpolicy="catchup"/>
<timer name="pit" tickpolicy="delay"/>
<timer name="hpet" present="no"/>
<timer name="hypervclock" present="yes"/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled="no"/>
<suspend-to-disk enabled="no"/>
</pm>
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<disk type="file" device="disk">
<driver name="qemu" type="qcow2" discard="unmap"/>
<source file="/var/lib/libvirt/images/win11.qcow2"/>
<target dev="vda" bus="virtio"/>
<address type="pci" domain="0x0000" bus="0x04" slot="0x00" function="0x0"/>
</disk>
<disk type="file" device="cdrom">
<driver name="qemu" type="raw"/>
<source file="/home/sarah/Downloads/Win11_23H2_English_x64v2.iso"/>
<target dev="sdb" bus="sata"/>
<readonly/>
<address type="drive" controller="0" bus="0" target="0" unit="1"/>
</disk>
<controller type="usb" index="0" model="qemu-xhci" ports="15">
<address type="pci" domain="0x0000" bus="0x02" slot="0x00" function="0x0"/>
</controller>
<controller type="pci" index="0" model="pcie-root"/>
<controller type="pci" index="1" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="1" port="0x10"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0" multifunction="on"/>
</controller>
<controller type="pci" index="2" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="2" port="0x11"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x1"/>
</controller>
<controller type="pci" index="3" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="3" port="0x12"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x2"/>
</controller>
<controller type="pci" index="4" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="4" port="0x13"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x3"/>
</controller>
<controller type="pci" index="5" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="5" port="0x14"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x4"/>
</controller>
<controller type="pci" index="6" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="6" port="0x15"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x5"/>
</controller>
<controller type="pci" index="7" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="7" port="0x16"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x6"/>
</controller>
<controller type="pci" index="8" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="8" port="0x17"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x7"/>
</controller>
<controller type="pci" index="9" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="9" port="0x18"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0" multifunction="on"/>
</controller>
<controller type="pci" index="10" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="10" port="0x19"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x1"/>
</controller>
<controller type="pci" index="11" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="11" port="0x1a"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x2"/>
</controller>
<controller type="pci" index="12" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="12" port="0x1b"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x3"/>
</controller>
<controller type="pci" index="13" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="13" port="0x1c"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x4"/>
</controller>
<controller type="pci" index="14" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="14" port="0x1d"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x5"/>
</controller>
<controller type="pci" index="15" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="15" port="0x1e"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x6"/>
</controller>
<controller type="pci" index="16" model="pcie-to-pci-bridge">
<model name="pcie-pci-bridge"/>
<address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
</controller>
<controller type="sata" index="0">
<address type="pci" domain="0x0000" bus="0x00" slot="0x1f" function="0x2"/>
</controller>
<controller type="virtio-serial" index="0">
<address type="pci" domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
</controller>
<interface type="network">
<mac address="52:54:00:a6:53:31"/>
<source network="default"/>
<model type="virtio"/>
<address type="pci" domain="0x0000" bus="0x09" slot="0x00" function="0x0"/>
</interface>
<serial type="pty">
<target type="isa-serial" port="0">
<model name="isa-serial"/>
</target>
</serial>
<console type="pty">
<target type="serial" port="0"/>
</console>
<channel type="spicevmc">
<target type="virtio" name="com.redhat.spice.0"/>
<address type="virtio-serial" controller="0" bus="0" port="1"/>
</channel>
<input type="tablet" bus="usb">
<address type="usb" bus="0" port="1"/>
</input>
<input type="mouse" bus="ps2"/>
<input type="keyboard" bus="ps2"/>
<graphics type="spice" autoport="yes">
<listen type="address"/>
<image compression="off"/>
</graphics>
<sound model="ich9">
<address type="pci" domain="0x0000" bus="0x00" slot="0x1b" function="0x0"/>
</sound>
<audio id="1" type="spice"/>
<video>
<model type="qxl" ram="65536" vram="65536" vgamem="16384" heads="1" primary="yes"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x0"/>
</video>
<hostdev mode="subsystem" type="pci" managed="yes">
<source>
<address domain="0x0000" bus="0x0e" slot="0x00" function="0x0"/>
</source>
<address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0"/>
</hostdev>
<hostdev mode="subsystem" type="pci" managed="yes">
<source>
<address domain="0x0000" bus="0x0e" slot="0x00" function="0x1"/>
</source>
<address type="pci" domain="0x0000" bus="0x06" slot="0x00" function="0x0"/>
</hostdev>
<hostdev mode="subsystem" type="pci" managed="yes">
<source>
<address domain="0x0000" bus="0x04" slot="0x00" function="0x0"/>
</source>
<address type="pci" domain="0x0000" bus="0x08" slot="0x00" function="0x0"/>
</hostdev>
<redirdev bus="usb" type="spicevmc">
<address type="usb" bus="0" port="2"/>
</redirdev>
<redirdev bus="usb" type="spicevmc">
<address type="usb" bus="0" port="3"/>
</redirdev>
<watchdog model="itco" action="reset"/>
<memballoon model="virtio">
<address type="pci" domain="0x0000" bus="0x07" slot="0x00" function="0x0"/>
</memballoon>
</devices>
</domain>
Errors:
Error starting domain: internal error: QEMU unexpectedly closed the monitor (vm='win11'): 2024-06-23T07:37:42.542904Z qemu-system-x86_64: warning: Deprecated CPU topology (considered invalid): Unsupported clusters parameter mustn't be specified as 1
2024-06-23T07:37:43.652043Z qemu-system-x86_64: Hyper-V synthetic interrupt controller (hv-synic) requires Hyper-V VP_INDEX MSR (hv-vpindex)
Traceback (most recent call last):
File "/usr/share/virt-manager/virtManager/asyncjob.py", line 72, in cb_wrapper
callback(asyncjob, *args, **kwargs)
File "/usr/share/virt-manager/virtManager/asyncjob.py", line 108, in tmpcb
callback(*args, **kwargs)
File "/usr/share/virt-manager/virtManager/object/libvirtobject.py", line 57, in newfn
ret = fn(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/share/virt-manager/virtManager/object/domain.py", line 1402, in startup
self._backend.create()
File "/usr/lib/python3.12/site-packages/libvirt.py", line 1379, in create
raise libvirtError('virDomainCreate() failed')
libvirt.libvirtError: internal error: QEMU unexpectedly closed the monitor (vm='win11'): 2024-06-23T07:37:42.542904Z qemu-system-x86_64: warning: Deprecated CPU topology (considered invalid): Unsupported clusters parameter mustn't be specified as 1
2024-06-23T07:37:43.652043Z qemu-system-x86_64: Hyper-V synthetic interrupt controller (hv-synic) requires Hyper-V VP_INDEX MSR (hv-vpindex)
Does it matter where the tag is? I simply replaced it where it was located.
retox
June 23, 2024, 5:07pm
10
Hmm… I’m 90% sure that the “Deprecated CPU topology” is just a warning - see here .
Next thing that jumps out to me is your “custom” <hyperv/> section…
My, really old but stable, win10 passthrough VM has the following:
<hyperv mode='passthrough'>
<relaxed state='on'/>
<vapic state='on'/>
<spinlocks state='on' retries='8191'/>
<vendor_id state='on' value='012375679CD'/>
</hyperv>
The <vendor_id /> is to deal with NVIDIA, but the rest was pretty standard, iirc. Maybe try removing the synthetic-timer specific options and see what happens?
Otherwise, looking at the <hyperv /> docs and the options you have it looks like your missing the <time/> setting, which required by <stimer/>.
1 Like
I think this might be an issue with some version mismatch between libvirt and qemu? Kinda taking a shot in the dark here, as libvirt shouldn’t be passing a “clusters” parameter to the CPU topology, based on your configuration.
This, however:
2024-06-23T07:37:43.652043Z qemu-system-x86_64: Hyper-V synthetic interrupt controller (hv-synic) requires Hyper-V VP_INDEX MSR (hv-vpindex)
Is your issue.
You need to add <vpindex state="on"/> to the hyperv configuration block. (ref )
If you read the error; it states “Hyper-V synthetic interrupt controller (hv-synic) requires Hyper-V VP_INDEX MSR (hv-vpindex)” vpindex was missing from your configuration, so qemu did not load that MSR feature.
2 Likes
Thank you guys for helping me out, it’s starting to make sense to me now.
I got so far into the windows installation until it detects that the VM doesn’t meet the system requirements, I’ve added TPM however nothing works (says either 1.2 and 2.0 are not supported in emulation and passthrough) until passthrough with TIS is applied.
For troubleshooting purposes I’ve applied your code snippet replacing the original and removed the vendor_id parameter.
The VM does start after that hyper-v change mentioned by retox, I’ve since changed the XML adding in the vpindex which didn’t hurt, in response to qemu vs libvirt, the packages I have installed are extra/qemu-full 9.0.1-1 and extra/libvirt 1:10.4.0-1 where libvirt is under extra/virt-install 4.1.0-3.
XML:
<domain type="kvm">
<name>win11</name>
<uuid>58fc68fc-95ff-4bf4-864c-065dfc743b54</uuid>
<metadata>
<libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
<libosinfo:os id="http://microsoft.com/win/11"/>
</libosinfo:libosinfo>
</metadata>
<memory unit="KiB">16777216</memory>
<currentMemory unit="KiB">16777216</currentMemory>
<memoryBacking>
<hugepages/>
</memoryBacking>
<vcpu placement="static">12</vcpu>
<iothreads>2</iothreads>
<cputune>
<vcpupin vcpu="0" cpuset="1"/>
<vcpupin vcpu="1" cpuset="9"/>
<vcpupin vcpu="2" cpuset="2"/>
<vcpupin vcpu="3" cpuset="10"/>
<vcpupin vcpu="4" cpuset="3"/>
<vcpupin vcpu="5" cpuset="11"/>
<vcpupin vcpu="6" cpuset="4"/>
<vcpupin vcpu="7" cpuset="12"/>
<vcpupin vcpu="8" cpuset="5"/>
<vcpupin vcpu="9" cpuset="13"/>
<vcpupin vcpu="10" cpuset="6"/>
<vcpupin vcpu="11" cpuset="14"/>
<emulatorpin cpuset="7,15"/>
<iothreadpin iothread="1" cpuset="0,8"/>
<iothreadpin iothread="2" cpuset="7,15"/>
</cputune>
<os firmware="efi">
<type arch="x86_64" machine="pc-q35-9.0">hvm</type>
<firmware>
<feature enabled="no" name="enrolled-keys"/>
<feature enabled="no" name="secure-boot"/>
</firmware>
<loader readonly="yes" type="pflash">/usr/share/edk2/x64/OVMF_CODE.fd</loader>
<nvram template="/usr/share/edk2/x64/OVMF_VARS.fd">/var/lib/libvirt/qemu/nvram/win11_VARS.fd</nvram>
</os>
<features>
<acpi/>
<apic/>
<hyperv mode="passthrough">
<relaxed state="on"/>
<vapic state="on"/>
<spinlocks state="on" retries="8191"/>
<vpindex state="on"/>
</hyperv>
<vmport state="off"/>
</features>
<cpu mode="host-passthrough" check="partial" migratable="off">
<topology sockets="1" dies="1" clusters="1" cores="6" threads="2"/>
</cpu>
<clock offset="localtime">
<timer name="rtc" tickpolicy="catchup"/>
<timer name="pit" tickpolicy="delay"/>
<timer name="hpet" present="no"/>
<timer name="hypervclock" present="yes"/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled="no"/>
<suspend-to-disk enabled="no"/>
</pm>
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<disk type="file" device="disk">
<driver name="qemu" type="qcow2" discard="unmap"/>
<source file="/var/lib/libvirt/images/win11.qcow2"/>
<target dev="vda" bus="virtio"/>
<boot order="2"/>
<address type="pci" domain="0x0000" bus="0x04" slot="0x00" function="0x0"/>
</disk>
<disk type="file" device="cdrom">
<driver name="qemu" type="raw"/>
<source file="/home/luke/Downloads/Win11_23H2_English_x64v2.iso"/>
<target dev="sdb" bus="sata"/>
<readonly/>
<boot order="1"/>
<address type="drive" controller="0" bus="0" target="0" unit="1"/>
</disk>
<controller type="usb" index="0" model="qemu-xhci" ports="15">
<address type="pci" domain="0x0000" bus="0x02" slot="0x00" function="0x0"/>
</controller>
<controller type="pci" index="0" model="pcie-root"/>
<controller type="pci" index="1" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="1" port="0x10"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0" multifunction="on"/>
</controller>
<controller type="pci" index="2" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="2" port="0x11"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x1"/>
</controller>
<controller type="pci" index="3" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="3" port="0x12"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x2"/>
</controller>
<controller type="pci" index="4" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="4" port="0x13"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x3"/>
</controller>
<controller type="pci" index="5" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="5" port="0x14"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x4"/>
</controller>
<controller type="pci" index="6" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="6" port="0x15"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x5"/>
</controller>
<controller type="pci" index="7" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="7" port="0x16"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x6"/>
</controller>
<controller type="pci" index="8" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="8" port="0x17"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x7"/>
</controller>
<controller type="pci" index="9" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="9" port="0x18"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0" multifunction="on"/>
</controller>
<controller type="pci" index="10" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="10" port="0x19"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x1"/>
</controller>
<controller type="pci" index="11" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="11" port="0x1a"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x2"/>
</controller>
<controller type="pci" index="12" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="12" port="0x1b"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x3"/>
</controller>
<controller type="pci" index="13" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="13" port="0x1c"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x4"/>
</controller>
<controller type="pci" index="14" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="14" port="0x1d"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x5"/>
</controller>
<controller type="pci" index="15" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="15" port="0x1e"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x6"/>
</controller>
<controller type="pci" index="16" model="pcie-to-pci-bridge">
<model name="pcie-pci-bridge"/>
<address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
</controller>
<controller type="sata" index="0">
<address type="pci" domain="0x0000" bus="0x00" slot="0x1f" function="0x2"/>
</controller>
<controller type="virtio-serial" index="0">
<address type="pci" domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
</controller>
<interface type="network">
<mac address="52:54:00:a6:53:31"/>
<source network="default"/>
<model type="virtio"/>
<address type="pci" domain="0x0000" bus="0x09" slot="0x00" function="0x0"/>
</interface>
<serial type="pty">
<target type="isa-serial" port="0">
<model name="isa-serial"/>
</target>
</serial>
<console type="pty">
<target type="serial" port="0"/>
</console>
<channel type="spicevmc">
<target type="virtio" name="com.redhat.spice.0"/>
<address type="virtio-serial" controller="0" bus="0" port="1"/>
</channel>
<input type="tablet" bus="usb">
<address type="usb" bus="0" port="1"/>
</input>
<input type="mouse" bus="ps2"/>
<input type="keyboard" bus="ps2"/>
<tpm model="tpm-tis">
<backend type="passthrough">
<device path="/dev/tpm0"/>
</backend>
</tpm>
<graphics type="spice" autoport="yes">
<listen type="address"/>
<image compression="off"/>
</graphics>
<sound model="ich9">
<address type="pci" domain="0x0000" bus="0x00" slot="0x1b" function="0x0"/>
</sound>
<audio id="1" type="spice"/>
<video>
<model type="qxl" ram="65536" vram="65536" vgamem="16384" heads="1" primary="yes"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x0"/>
</video>
<hostdev mode="subsystem" type="pci" managed="yes">
<source>
<address domain="0x0000" bus="0x0e" slot="0x00" function="0x0"/>
</source>
<address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0"/>
</hostdev>
<hostdev mode="subsystem" type="pci" managed="yes">
<source>
<address domain="0x0000" bus="0x0e" slot="0x00" function="0x1"/>
</source>
<address type="pci" domain="0x0000" bus="0x06" slot="0x00" function="0x0"/>
</hostdev>
<hostdev mode="subsystem" type="pci" managed="yes">
<source>
<address domain="0x0000" bus="0x04" slot="0x00" function="0x0"/>
</source>
<address type="pci" domain="0x0000" bus="0x08" slot="0x00" function="0x0"/>
</hostdev>
<redirdev bus="usb" type="spicevmc">
<address type="usb" bus="0" port="2"/>
</redirdev>
<redirdev bus="usb" type="spicevmc">
<address type="usb" bus="0" port="3"/>
</redirdev>
<watchdog model="itco" action="reset"/>
<memballoon model="virtio">
<address type="pci" domain="0x0000" bus="0x07" slot="0x00" function="0x0"/>
</memballoon>
</devices>
</domain>Here’s what I’m now block by, windows
Please note once windows is installed and running nominally with looking glass, I’ll dig into the optimizations (or is it a different process in this regard?).
1 Like
This is the thing I’m not so sure about. I haven’t played with windows 11 on VMs yet, and probably won’t be any time soon; I’ve kinda moved away from the passthrough game this year.
My understanding is you’ll need a software TPM, which can be pulled from the swtpm package, and secure boot enabled for it; which means you’ll need not only to enable secure boot on the firmware, but also use the secure-boot variant of OVMF.
2 Likes
Hey no worries, I’m thankful that I’m getting help.
The swtpm is a piece of the puzzle to use software (emulated) TPM which is now working, but I’ve missed an important option, switch from OVMF.fd to OVMf.secureboot.fd which when I looked back at this topic confirmed my suspicion as I’ve tested a basic vm with windows 11 just for a baseline.
Now it’s installing from the current “so far so good” configuration:
I’ll continue to experiment with your guide to see how far I can go without more help, as I feel I have a good foothold now. Heck, after I’m done will for sure post everything I’ve done to make it work as intended for anyone else to have a look and learn from it.
Be back soon.
2 Likes
Awesome! That’s exactly what we hope for here. Also, if you do find any issues with the guide, I can use this as an opportunity to tweak it and increase the accuracy.
I’m really glad you got it working; the secureboot part is the one area that I’m not familiar with, so I’m glad it wasn’t too much of a struggle.
1 Like
