Is there a chance that a bug has been introduced…?
Today only I saw two similar issues…
I’m seeing if I can replicate the VM, only error in virsh editing it was the las argument missed a "/ at the end, before the >
Just changing details to match my hardware
Okay, stripped out a bunch of stuff, and the VM runs fine with VGA passthrough.
Can’t get a console output at the same time though.
I would say start fresh, then remove all the extraneous junk.
Only things needed should be
<vendor_id state='on' value='0123456789ab'/>
in the hyperv section, and
<kvm> <hidden state='on'/> </kvm>
In the bit after hyperv, but before vmport.
I don’t have hugepages, so did not enable it on the new VM.
Also did not pass through a rom file.
I used an existing storage file, and am running an older version of libvirt.
If it helps, this is the vm I just spun up:
I added notes with #notes but this would have to be stripped out; virsh would throw a fit
my just made xml file
<domain type="kvm">
<name>testyVM</name>
<uuid>838f2b74-a1f5-49c3-af96-a760fe844db1</uuid>
<metadata>
<libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
<libosinfo:os id="http://microsoft.com/win/2k19"/>
</libosinfo:libosinfo>
</metadata>
<memory unit="KiB">4194304</memory>
<currentMemory unit="KiB">4194304</currentMemory>
<vcpu placement="static">12</vcpu>
<os>
<type arch="x86_64" machine="pc-q35-4.2">hvm</type>
<loader readonly="yes" type="pflash">/usr/share/OVMF/OVMF_CODE.ms.fd</loader>
<nvram>/var/lib/libvirt/qemu/nvram/testyVM_VARS.fd</nvram>
<boot dev="hd"/>
</os>######all the above was automagically configured by VMM. note older pc-q35-4.2
<features>
<acpi/>
<apic/>
<hyperv>
<relaxed state="on"/>
<vapic state="on"/>
<spinlocks state="on" retries="8191"/>
<vendor_id state="on" value="0123456789ab"/> #this bit you have in yours, and works fine for me
</hyperv>
<kvm>
<hidden state="on"/> #this bit works fine too
</kvm>
<vmport state="off"/>
</features>
<cpu mode="host-model" check="partial"/>
<clock offset="localtime">
<timer name="rtc" tickpolicy="catchup"/>
<timer name="pit" tickpolicy="delay"/>
<timer name="hpet" present="no"/>
<timer name="hypervclock" present="yes"/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled="no"/>
<suspend-to-disk enabled="no"/>
</pm>
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<disk type="file" device="disk">
<driver name="qemu" type="qcow2"/>
<source file="/home/trooper_ish/ImagesFlash/WinRoot/WinRoot1909v2.qcow2"/>
<target dev="sda" bus="sata"/>
<address type="drive" controller="0" bus="0" target="0" unit="0"/>
</disk>
<disk type="file" device="disk">
<driver name="qemu" type="qcow2"/>
<source file="/home/trooper_ish/13Vmm/VMDump/AAWinFiles.qcow2"/> #obviously not your drive
<target dev="sdb" bus="sata"/>
<address type="drive" controller="0" bus="0" target="0" unit="1"/>
</disk>
<controller type="usb" index="0" model="qemu-xhci" ports="15">
<address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
</controller>
<controller type="sata" index="0">
<address type="pci" domain="0x0000" bus="0x00" slot="0x1f" function="0x2"/>
</controller>
<controller type="pci" index="0" model="pcie-root"/>
<controller type="pci" index="1" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="1" port="0x8"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x0" multifunction="on"/>
</controller>
<controller type="pci" index="2" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="2" port="0x9"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x1"/>
</controller>
<controller type="pci" index="3" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="3" port="0xa"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x2"/>
</controller>
<controller type="pci" index="4" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="4" port="0xb"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x3"/>
</controller>
<controller type="pci" index="5" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="5" port="0xc"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x4"/>
</controller>
<controller type="pci" index="6" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="6" port="0xd"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x5"/>
</controller>
<controller type="pci" index="7" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="7" port="0xe"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x6"/>
</controller>
<controller type="pci" index="8" model="pcie-to-pci-bridge">
<model name="pcie-pci-bridge"/>
<address type="pci" domain="0x0000" bus="0x06" slot="0x00" function="0x0"/>
</controller>
<input type="mouse" bus="ps2"/>
<input type="keyboard" bus="ps2"/>
<graphics type="spice" autoport="yes">
<listen type="address"/>
<image compression="off"/>
<gl enable="no" rendernode="/dev/dri/by-path/pci-0000:43:00.0-render"/>
</graphics>
<sound model="ich9">
<address type="pci" domain="0x0000" bus="0x00" slot="0x1b" function="0x0"/>
</sound>
<video>
<model type="cirrus" vram="16384" heads="1" primary="yes"/>
<address type="pci" domain="0x0000" bus="0x08" slot="0x01" function="0x0"/>
</video>
<hostdev mode="subsystem" type="pci" managed="yes">
<source>
<address domain="0x0000" bus="0x09" slot="0x00" function="0x0"/>
</source>
<address type="pci" domain="0x0000" bus="0x02" slot="0x00" function="0x0"/>
</hostdev>
<hostdev mode="subsystem" type="pci" managed="yes">
<source>
<address domain="0x0000" bus="0x09" slot="0x00" function="0x1"/>
</source>
<address type="pci" domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
</hostdev>
<hostdev mode="subsystem" type="pci" managed="yes">
<source>
<address domain="0x0000" bus="0x0a" slot="0x00" function="0x3"/>
</source>
<address type="pci" domain="0x0000" bus="0x04" slot="0x00" function="0x0"/>
</hostdev>
<memballoon model="virtio">
<address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0"/>
</memballoon>
</devices>
</domain>
I tried to cut out as much as possible, to reduce conflicts, but vmm still adds a bunch of junk like pci devices
Once I created the machine, addded storage and removed what I could in VMM, I went to virsh to edit in the hidden state, and the vendor id.
The great thing about virsh, is it is brutal at pointing out any missing letters.
I was able to add video devices and boot, with the GPU working, but I got no output on the console after the tiannocore splash screen (the four lines you were talking about? they stayed on the console, but the passthrough worked okay)
So I would really reccomend starting fresh, then adding hugepages, etc.
I did not have to add any of the qemucommandline stuff, but I perhaps already have some stuff enabled at the kernel level?
I would say, if it works without, naybe add back later.
Also you later might want to tweak to get USB3 or stuff, but baby steps.
You might try this new xml yourself?
<domain type="kvm">
<name>win10-gamesToo</name>
<uuid>f5287a54-b804-4227-b248-b60f9155784c</uuid>
<metadata>
<libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
<libosinfo:os id="http://microsoft.com/win/10"/>
</libosinfo:libosinfo>
</metadata>
<memory unit="KiB">33554432</memory>
<currentMemory unit="KiB">33554432</currentMemory>
<memoryBacking>
<hugepages/>
</memoryBacking>
<vcpu placement="static">18</vcpu>
<os>
<type arch="x86_64" machine="pc-q35-5.0">hvm</type>
<loader readonly="yes" type="pflash">/usr/share/edk2-ovmf/x64/OVMF_CODE.fd</loader>
<nvram>/var/lib/libvirt/qemu/nvram/win10-gamesToo_VARS.fd</nvram>
</os>
<features>
<acpi/>
<apic/>
<hyperv>
<relaxed state="off"/>
<vapic state="on"/>
<spinlocks state="on" retries="8191"/>
<vpindex state="on"/>
<synic state="on"/>
<stimer state="on"/>
<vendor_id state="on" value="0123456789ab"/>
<frequencies state="on"/>
</hyperv>
<kvm>
<hidden state="on"/>
</kvm>
<vmport state="off"/>
<ioapic driver="kvm"/>
</features>
<cpu mode="host-passthrough" check="none">
<topology sockets="1" cores="18" threads="1"/>
</cpu>
<clock offset="localtime">
<timer name="rtc" tickpolicy="catchup"/>
<timer name="pit" tickpolicy="delay"/>
<timer name="hpet" present="no"/>
<timer name="hypervclock" present="yes"/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled="no"/>
<suspend-to-disk enabled="no"/>
</pm>
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<disk type="file" device="cdrom">
<driver name="qemu" type="raw"/>
<source file="/home/wizard/Data/Linux/Manjaro/virtio-win-0.1.160.iso"/>
<target dev="sda" bus="sata"/>
<readonly/>
<address type="drive" controller="0" bus="0" target="0" unit="0"/>
</disk>
<disk type="block" device="disk">
<driver name="qemu" type="raw" cache="none" io="native"/>
<source dev="/dev/sdc"/>
<target dev="sdc" bus="sata"/>
<boot order="1"/>
<address type="drive" controller="0" bus="0" target="0" unit="1"/>
</disk>
<disk type="block" device="disk">
<driver name="qemu" type="raw" cache="none" io="native"/>
<source dev="/dev/sdd"/>
<target dev="sdd" bus="sata"/>
<address type="drive" controller="1" bus="0" target="0" unit="1"/>
</disk>
<controller type="usb" index="0" model="qemu-xhci">
<address type="pci" domain="0x0000" bus="0x02" slot="0x00" function="0x0"/>
</controller>
<controller type="sata" index="0">
<address type="pci" domain="0x0000" bus="0x00" slot="0x1f" function="0x2"/>
</controller>
<controller type="pci" index="0" model="pcie-root"/>
<controller type="pci" index="1" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="1" port="0x10"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0" multifunction="on"/>
</controller>
<controller type="pci" index="2" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="2" port="0x11"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x1"/>
</controller>
<controller type="pci" index="3" model="pcie-to-pci-bridge">
<model name="pcie-pci-bridge"/>
<address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
</controller>
<controller type="pci" index="4" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="4" port="0x12"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x2"/>
</controller>
<controller type="pci" index="5" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="5" port="0x13"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x3"/>
</controller>
<controller type="pci" index="6" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="6" port="0x14"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x4"/>
</controller>
<controller type="pci" index="7" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="7" port="0x15"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x5"/>
</controller>
<controller type="pci" index="8" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="8" port="0x16"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x6"/>
</controller>
<controller type="pci" index="9" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="9" port="0x17"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x7"/>
</controller>
<controller type="virtio-serial" index="0">
<address type="pci" domain="0x0000" bus="0x04" slot="0x00" function="0x0"/>
</controller>
<serial type="pty">
<target type="isa-serial" port="0">
<model name="isa-serial"/>
</target>
</serial>
<input type="mouse" bus="ps2"/>
<input type="keyboard" bus="ps2"/>
<sound model="ich9">
<address type="pci" domain="0x0000" bus="0x00" slot="0x1b" function="0x0"/>
</sound>
<hostdev mode="subsystem" type="pci" managed="yes">
<source>
<address domain="0x0000" bus="0x06" slot="0x00" function="0x0"/>
</source>
<address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0"/>
</hostdev>
<hostdev mode="subsystem" type="pci" managed="yes">
<driver name="vfio"/>
<source>
<address domain="0x0000" bus="0x0b" slot="0x00" function="0x0"/>
</source>
<address type="pci" domain="0x0000" bus="0x06" slot="0x00" function="0x0"/>
</hostdev>
<hostdev mode="subsystem" type="pci" managed="yes">
<source>
<address domain="0x0000" bus="0x0b" slot="0x00" function="0x1"/>
</source>
<address type="pci" domain="0x0000" bus="0x07" slot="0x00" function="0x0"/>
</hostdev>
<hostdev mode="subsystem" type="usb" managed="yes">
<source>
<vendor id="0x1b1c"/>
<product id="0x1b81"/>
</source>
<address type="usb" bus="0" port="1"/>
</hostdev>
<redirdev bus="usb" type="spicevmc">
<address type="usb" bus="0" port="3"/>
</redirdev>
<redirdev bus="usb" type="spicevmc">
<address type="usb" bus="0" port="4"/>
</redirdev>
<memballoon model="virtio">
<address type="pci" domain="0x0000" bus="0x08" slot="0x00" function="0x0"/>
</memballoon>
</devices>
</domain>
or with rom file:
other xml
<domain type="kvm">
<name>win10-gamesToo</name>
<uuid>f5287a54-b804-4227-b248-b60f9155784c</uuid>
<metadata>
<libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
<libosinfo:os id="http://microsoft.com/win/10"/>
</libosinfo:libosinfo>
</metadata>
<memory unit="KiB">33554432</memory>
<currentMemory unit="KiB">33554432</currentMemory>
<memoryBacking>
<hugepages/>
</memoryBacking>
<vcpu placement="static">18</vcpu>
<os>
<type arch="x86_64" machine="pc-q35-5.0">hvm</type>
<loader readonly="yes" type="pflash">/usr/share/edk2-ovmf/x64/OVMF_CODE.fd</loader>
<nvram>/var/lib/libvirt/qemu/nvram/win10-gamesToo_VARS.fd</nvram>
</os>
<features>
<acpi/>
<apic/>
<hyperv>
<relaxed state="off"/>
<vapic state="on"/>
<spinlocks state="on" retries="8191"/>
<vpindex state="on"/>
<synic state="on"/>
<stimer state="on"/>
<vendor_id state="on" value="0123456789ab"/>
<frequencies state="on"/>
</hyperv>
<kvm>
<hidden state="on"/>
</kvm>
<vmport state="off"/>
<ioapic driver="kvm"/>
</features>
<cpu mode="host-passthrough" check="none">
<topology sockets="1" cores="18" threads="1"/>
</cpu>
<clock offset="localtime">
<timer name="rtc" tickpolicy="catchup"/>
<timer name="pit" tickpolicy="delay"/>
<timer name="hpet" present="no"/>
<timer name="hypervclock" present="yes"/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled="no"/>
<suspend-to-disk enabled="no"/>
</pm>
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<disk type="file" device="cdrom">
<driver name="qemu" type="raw"/>
<source file="/home/wizard/Data/Linux/Manjaro/virtio-win-0.1.160.iso"/>
<target dev="sda" bus="sata"/>
<readonly/>
<address type="drive" controller="0" bus="0" target="0" unit="0"/>
</disk>
<disk type="block" device="disk">
<driver name="qemu" type="raw" cache="none" io="native"/>
<source dev="/dev/sdc"/>
<target dev="sdc" bus="sata"/>
<boot order="1"/>
<address type="drive" controller="0" bus="0" target="0" unit="1"/>
</disk>
<disk type="block" device="disk">
<driver name="qemu" type="raw" cache="none" io="native"/>
<source dev="/dev/sdd"/>
<target dev="sdd" bus="sata"/>
<address type="drive" controller="1" bus="0" target="0" unit="1"/>
</disk>
<controller type="usb" index="0" model="qemu-xhci">
<address type="pci" domain="0x0000" bus="0x02" slot="0x00" function="0x0"/>
</controller>
<controller type="sata" index="0">
<address type="pci" domain="0x0000" bus="0x00" slot="0x1f" function="0x2"/>
</controller>
<controller type="pci" index="0" model="pcie-root"/>
<controller type="pci" index="1" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="1" port="0x10"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0" multifunction="on"/>
</controller>
<controller type="pci" index="2" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="2" port="0x11"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x1"/>
</controller>
<controller type="pci" index="3" model="pcie-to-pci-bridge">
<model name="pcie-pci-bridge"/>
<address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
</controller>
<controller type="pci" index="4" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="4" port="0x12"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x2"/>
</controller>
<controller type="pci" index="5" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="5" port="0x13"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x3"/>
</controller>
<controller type="pci" index="6" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="6" port="0x14"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x4"/>
</controller>
<controller type="pci" index="7" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="7" port="0x15"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x5"/>
</controller>
<controller type="pci" index="8" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="8" port="0x16"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x6"/>
</controller>
<controller type="pci" index="9" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="9" port="0x17"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x7"/>
</controller>
<controller type="virtio-serial" index="0">
<address type="pci" domain="0x0000" bus="0x04" slot="0x00" function="0x0"/>
</controller>
<serial type="pty">
<target type="isa-serial" port="0">
<model name="isa-serial"/>
</target>
</serial>
<input type="mouse" bus="ps2"/>
<input type="keyboard" bus="ps2"/>
<sound model="ich9">
<address type="pci" domain="0x0000" bus="0x00" slot="0x1b" function="0x0"/>
</sound>
<hostdev mode="subsystem" type="pci" managed="yes">
<source>
<address domain="0x0000" bus="0x06" slot="0x00" function="0x0"/>
</source>
<address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0"/>
</hostdev>
<hostdev mode="subsystem" type="pci" managed="yes">
<driver name="vfio"/>
<source>
<address domain="0x0000" bus="0x0b" slot="0x00" function="0x0"/>
</source>
<rom bar="on" file="/home/wizard/Downloads/Linux/EVGA.GTX1080Ti.11264.170418.rom"/>
<address type="pci" domain="0x0000" bus="0x06" slot="0x00" function="0x0"/>
</hostdev>
<hostdev mode="subsystem" type="pci" managed="yes">
<source>
<address domain="0x0000" bus="0x0b" slot="0x00" function="0x1"/>
</source>
<address type="pci" domain="0x0000" bus="0x07" slot="0x00" function="0x0"/>
</hostdev>
<hostdev mode="subsystem" type="usb" managed="yes">
<source>
<vendor id="0x1b1c"/>
<product id="0x1b81"/>
</source>
<address type="usb" bus="0" port="1"/>
</hostdev>
<redirdev bus="usb" type="spicevmc">
<address type="usb" bus="0" port="3"/>
</redirdev>
<redirdev bus="usb" type="spicevmc">
<address type="usb" bus="0" port="4"/>
</redirdev>
<memballoon model="virtio">
<address type="pci" domain="0x0000" bus="0x08" slot="0x00" function="0x0"/>
</memballoon>
</devices>
</domain>
Either way would require creating a VM with the name “win10-gamesToo” but I would again suggest starting fresh, and adding PCI drivers for GPU and sound
I have been very busy and I didn’t have time to test your suggestions, but I noticed a difference between our XMLs… I am using the “/usr/share/edk2-ovmf/x64/OVMF_CODE.fd” while you are using the older version.
Can this be the cause?
When you create a vm in vmm, the metadata or os bit will have the relevant file Location. Also, the pc-q35.versionnumber will be different.
I don’t see this as a major difference, and Virsh will not complain, but vmm will fail, and give an error for it
Something is wrong here…
I removed all parameters from kernel, vfio.conf and mkinitcpio.conf and I still don’t see my 1080 as available in Manjaro… Normaly I would have a 3rd monitor available, the one connected on it.
“sudo lspci -nnk -d 10de:1b06” returns
0b:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP102 [GeForce GTX 1080 Ti] [10de:1b06] (rev a1)
Subsystem: eVga.com. Corp. Device [3842:6696]
Kernel modules: nouveau
Is there a chance that something is wrong with the card…?
Huh, I get this:
trooper_ish@UbuTween:~$ lspci -nnk -d 10de:17c8
09:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM200 [GeForce GTX 980 Ti] [10de:17c8] (rev a1)
Kernel driver in use: vfio-pci
Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia
obviously I did not remove my device, but on Ubuntu, it lists the module in use.
I would grep dmesg for the card by pci address and device id
[ 2.290743] pci 0000:0b:00.0: [10de:1b06] type 00 class 0x030000
[ 2.290760] pci 0000:0b:00.0: reg 0x10: [mem 0xfb000000-0xfbffffff]
[ 2.290768] pci 0000:0b:00.0: reg 0x14: [mem 0xb0000000-0xbfffffff 64bit pref]
[ 2.290775] pci 0000:0b:00.0: reg 0x1c: [mem 0xc0000000-0xc1ffffff 64bit pref]
[ 2.290780] pci 0000:0b:00.0: reg 0x24: [io 0xf000-0xf07f]
[ 2.290784] pci 0000:0b:00.0: reg 0x30: [mem 0xfc000000-0xfc07ffff pref]
[ 2.290877] pci 0000:0b:00.0: 63.008 Gb/s available PCIe bandwidth, limited by 8.0 GT/s PCIe x8 link at 0000:00:03.1 (capable of 126.016 Gb/s with 8.0 GT/s PCIe x16 link)
[ 2.290902] pci 0000:0b:00.1: [10de:10ef] type 00 class 0x040300
[ 2.290916] pci 0000:0b:00.1: reg 0x10: [mem 0xfc080000-0xfc083fff]
[ 2.294070] pci 0000:0b:00.0: vgaarb: VGA device added: decodes=io+mem,owns=none,locks=none
[ 2.294071] pci 0000:0b:00.0: vgaarb: bridge control possible
[ 2.315753] pci_bus 0000:0b: resource 0 [io 0xf000-0xffff]
[ 2.315753] pci_bus 0000:0b: resource 1 [mem 0xfb000000-0xfc0fffff]
[ 2.315754] pci_bus 0000:0b: resource 2 [mem 0xb0000000-0xc1ffffff 64bit pref]
[ 2.316148] pci 0000:0b:00.1: D0 power state depends on 0000:0b:00.0
[ 2.392562] pci 0000:0b:00.0: Adding to iommu group 28
[ 2.392577] pci 0000:0b:00.1: Adding to iommu group 28
[ 4.224893] snd_hda_intel 0000:0b:00.1: Disabling MSI
[ 4.224901] snd_hda_intel 0000:0b:00.1: Handle vga_switcheroo audio client
[ 4.241550] input: HDA NVidia HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:03.1/0000:0b:00.1/sound/card0/input8
[ 4.241620] input: HDA NVidia HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:03.1/0000:0b:00.1/sound/card0/input9
[ 4.241643] input: HDA NVidia HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:03.1/0000:0b:00.1/sound/card0/input10
[ 4.241663] input: HDA NVidia HDMI/DP,pcm=9 as /devices/pci0000:00/0000:00:03.1/0000:0b:00.1/sound/card0/input11
[ 4.241684] input: HDA NVidia HDMI/DP,pcm=10 as /devices/pci0000:00/0000:00:03.1/0000:0b:00.1/sound/card0/input12
[ 4.241706] input: HDA NVidia HDMI/DP,pcm=11 as /devices/pci0000:00/0000:00:03.1/0000:0b:00.1/sound/card0/input13
[ 4.241727] input: HDA NVidia HDMI/DP,pcm=12 as /devices/pci0000:00/0000:00:03.1/0000:0b:00.1/sound/card0/input14
[wizard@wiz-amd-manjaro ~]$ ^C
Okay, I don’t this this helps, but I get:
Host
trooper_ish@UbuTween:~$ dmesg | grep "10de:13c0" && dmesg | grep "43:00.0"
[ 0.244317] pci 0000:43:00.0: [10de:13c0] type 00 class 0x030000
[ 0.244317] pci 0000:43:00.0: [10de:13c0] type 00 class 0x030000
[ 0.244340] pci 0000:43:00.0: reg 0x10: [mem 0x9e000000-0x9effffff]
[ 0.244352] pci 0000:43:00.0: reg 0x14: [mem 0x80000000-0x8fffffff 64bit pref]
[ 0.244364] pci 0000:43:00.0: reg 0x1c: [mem 0x90000000-0x91ffffff 64bit pref]
[ 0.244373] pci 0000:43:00.0: reg 0x24: [io 0x5000-0x507f]
[ 0.244381] pci 0000:43:00.0: reg 0x30: [mem 0x9f000000-0x9f07ffff pref]
[ 0.244396] pci 0000:43:00.0: BAR 3: assigned to efifb
[ 0.244484] pci 0000:43:00.0: 32.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s x16 link at 0000:40:03.1 (capable of 126.016 Gb/s with 8 GT/s x16 link)
[ 0.258094] pci 0000:43:00.0: vgaarb: setting as boot VGA device
[ 0.258094] pci 0000:43:00.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none
[ 0.258094] pci 0000:43:00.0: vgaarb: bridge control possible
[ 0.290945] pci 0000:43:00.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]
[ 0.290960] pci 0000:43:00.1: D0 power state depends on 0000:43:00.0
[ 0.413362] pci 0000:43:00.0: Adding to iommu group 36
[ 13.145666] nvidia 0000:43:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=io+mem
[ 13.302283] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:43:00.0 on minor 0
and guest:
guest
trooper_ish@UbuTween:~$ dmesg | grep "10de:17c8" && dmesg | grep "09:00.0"
[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.4.0-42-generic root=UUID=e1262712-a1a2-4aef-98c7-7e9b21903938 ro iommu=on amd_iommu=on vfio_pci.ids=10de:17c8,10de:0fb0
[ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.4.0-42-generic root=UUID=e1262712-a1a2-4aef-98c7-7e9b21903938 ro iommu=on amd_iommu=on vfio_pci.ids=10de:17c8,10de:0fb0
[ 0.237087] pci 0000:09:00.0: [10de:17c8] type 00 class 0x030000
[ 0.469655] vfio_pci: add [10de:17c8[ffffffff:ffffffff]] class 0x000000/00000000
[ 0.237087] pci 0000:09:00.0: [10de:17c8] type 00 class 0x030000
[ 0.237102] pci 0000:09:00.0: reg 0x10: [mem 0xd8000000-0xd8ffffff]
[ 0.237109] pci 0000:09:00.0: reg 0x14: [mem 0xc0000000-0xcfffffff 64bit pref]
[ 0.237116] pci 0000:09:00.0: reg 0x1c: [mem 0xd0000000-0xd1ffffff 64bit pref]
[ 0.237121] pci 0000:09:00.0: reg 0x24: [io 0x3000-0x307f]
[ 0.237125] pci 0000:09:00.0: reg 0x30: [mem 0xd9000000-0xd907ffff pref]
[ 0.258094] pci 0000:09:00.0: vgaarb: VGA device added: decodes=io+mem,owns=none,locks=none
[ 0.258094] pci 0000:09:00.0: vgaarb: bridge control possible
[ 0.290802] pci 0000:09:00.1: D0 power state depends on 0000:09:00.0
[ 0.401848] pci 0000:09:00.0: Adding to iommu group 16
[ 0.450545] vfio-pci 0000:09:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
[ 13.145664] vfio-pci 0000:09:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
I presume the issue might be a bit deeper.
Could the card be isolated any other way?
I’m literally spitballing now, I don’t think I have any ideas to help, apart from re-doing the cmdline to re-introduce the hardware id just in case, but it doesn’t look like it’s being either isolated by the normal vfio way, or used by the system…
Apologies, maybe I didn’t do myself clear… As you suggested, I started again from scratch. So, I removed all changes from kernel, vfio.conf and mkinitcpio.conf, ran mkinitcpio -p linux58 and my system should be “clean”. I was expecting though the card to be available to host system.
Now, I am doing all the changes, step by step with a new VM. So far:
sudo dmesg | grep -i vfio
[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.8-x86_64 root=UUID=2c8e84da-b359-4d2a-8eb3-3c53ee14eca3 rw quiet apparmor=1 security=apparmor resume=UUID=855cc007-54fd-4ea0-8dcd-8db736df2099 udev.log_priority=3 amd_iommu=on iommu=pt vfio-pci.ids=10de:1b06,10de:10ef,8086:1539
[ 0.097034] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.8-x86_64 root=UUID=2c8e84da-b359-4d2a-8eb3-3c53ee14eca3 rw quiet apparmor=1 security=apparmor resume=UUID=855cc007-54fd-4ea0-8dcd-8db736df2099 udev.log_priority=3 amd_iommu=on iommu=pt vfio-pci.ids=10de:1b06,10de:10ef,8086:1539
[ 3.385286] VFIO - User Level meta-driver version: 0.3
[ 3.387831] vfio-pci 0000:0b:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
[ 3.404055] vfio_pci: add [10de:1b06[ffffffff:ffffffff]] class 0x000000/00000000
[ 3.420827] vfio_pci: add [10de:10ef[ffffffff:ffffffff]] class 0x000000/00000000
[ 3.437485] vfio_pci: add [8086:1539[ffffffff:ffffffff]] class 0x000000/00000000
[ 7.523984] vfio-pci 0000:0b:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
sudo lspci -nnk -d 10de:1b06
0b:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP102 [GeForce GTX 1080 Ti] [10de:1b06] (rev a1)
Subsystem: eVga.com. Corp. Device [3842:6696]
Kernel driver in use: vfio-pci
Kernel modules: nouveau
Looks ok to me…
2 Likes
Cool, that does look good, And good idea to start fresh from the beginning, in case something came unstuck from some update breaking the previously working config.
And glad to see the driver in use is the vfio.
I was thrown, thinking the cards was not working for guest or host…
I am really fed up with this crap!!!
Did everything by the book. Each and every step following Arch Wiki and even another walk through specifically for my setup (Aorus X570, Ruzen9, Nvidia guest, AMD, host).
It is NOT WORKING!!!
Finally got it working (yeah, I am not known for quitting…)
So after examining all the details, and since it was working before, I turned my attention to main changes between Intel and AMD.
When I was installing Windows, right after the boot, I was getting a “KERNEL_SECURITY_CHECK_FAILURE” and the only way I found to bypass this was adding below lines right before the
<qemu:commandline>
<qemu:arg value="-cpu"/>
<qemu:arg value="host,topoext=on,invtsc=on,hv-time,hv-relaxed,hv-vapic,hv-spinlocks=0x1fff,hv-vpindex,hv-synic,hv-stimer,hv-reset,hv-frequencies,host-cache-info=on,l3-cache=off,-amd-stibp"/>
</qemu:commandline>
This worked and I was able to do the installation, however it seems that interferes with Nvidia VM status check.
After I removed it, it works as it should be.
Notice: some people, in certain games, have issues with the same BSoD. They suggested that you have to make this change permanent:
$ sudo nano /etc/modprobe.d/kvm.conf
Add this in:
options kvm ignore_msrs=Y
options kvm report_ignored_msrs=N
3 Likes