Hi everyone,
Ok, while that title is a bit of a doozey, I figured this is the best place to ask about a particular problem I’ve been having with a virtual machine setup I’ve been trying to create. The goal of this VM isn’t to be a powerhouse but rather act as an aide my work (I need to develop on Windows for some proprietary libraries that only target Windows, but daily drive Linux and prefer to do my development work there.) Since I deal with software targeted to other developers, a VM would provide me with the ability to create reproducible environments that recreate a clean system instead of my bloated windows install on another drive like I have currently.
First, the groundwork of what I have and what I’ve done so far. I have a host system running a partially-custom atomic operating system based on Fedora OSTree (Silverblue) using a tool called BlueBuild that runs a simple sway tiling window manager and some other QOL packages installed. The specs are as follows:
OS: Fedora Linux 40.20240711.0 (Forty) x86_64
MB: MPG X570 Gaming Plus
Linux Kernel: 6.9.7-200.fc40.x86_64
WM: sway
CPU: AMD Ryzen 7 5800X (16) @ 3.800GHz
GPU: AMD ATI Radeon RX 6650 XT / 6700S / 6800S
GPU: NVIDIA GeForce GTX 1060 3GB
Memory: 64215MiB
The system is relatively plain, but my aim was to simply continue to use my RX 6650 XT on my host system and make use of a rather unremarkable GTX 1060 3GB that was otherwise sitting unused in my old server chassis. Obviously, I’ve also gone through the process of setting up IOMMU and passthrough virtualization support on my motherboards’ bios. I’ve also verified that my GPU is isolated in the IOMMU group, which looks like so:
... ... ...
IOMMU Group 21:
21:0a.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]
2c:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 22:
23:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP106 [GeForce GTX 1060 3GB] [10de:1c02] (rev a1)
23:00.1 Audio device [0403]: NVIDIA Corporation GP106 High Definition Audio Controller [10de:10f1] (rev a1)
IOMMU Group 23:
26:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1061/ASM1062 Serial ATA Controller [1b21:0612] (rev 02)
... ... ...
The atomic operating system part of the equation added a new set of hurdles, but thankfully these were documented thanks to another user who wrote up a pretty thorough readme on how to do it on silverblue https://github.com/FilBot3/fedora-silverblue-gpu-passthrough that seems to be mostly right and up to date enough that I have successfully gotten my graphics card to be reserved for use by vfio drivers and use the appropriate kargs. They are as follows:
[snip ostree stuff here] ... amd_iommu=on iommu=pt vfio-pci.ids=10de:1c02,10de:10f1 rd.driver.pre=vfio_pci
Additionally, I had my gpu detected within a Windows VM as a valid graphics card, and achieved a correct readout and what seemed to be activity logged in GPU-Z, so I believed that I reached the holy grail of passthrough. I continued by installing the correct drivers in the VM which succeeded as normal and rebooted the machine.
However, my celebration ended after the realization that I was seemingly still using spice/built-in graphics functionality and it wasn’t actually using my GPU at all. While windows shows the monitor plugged into my graphics card as a valid target, changing my monitor over to that input signal resulted in only a no-signal screen. Trying to run a simple game like Counter Strike 1.6 seemed to result in 4fps which, let’s face it, is pretty sad. So my work wasn’t over, I figured more needed to be done.
I guessed at the time that I needed to completely remove spice from the equation and followed a rough procedure for this on the arch wiki https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF#Attaching_the_PCI_devices which seemed to be leading me down the right path. However, I was bummed to find out that I not only still had no-signal readout on my monitor’s other input, but I now no longer have any ability to access my windows system despite sacrificing spice. I thought that, perhaps, if I let the VM idle a while I could see if Windows suddenly decides to use the PCI device, as I saw some forum posts online claim that such a thing had happened before. Alas, it was fruitless.
After reading countless posts online with similar issues, I noticed a few that mentioned the same source of a similar problem: Resizable Bar. “Ok, well… Perhaps I should try disable resizable bar on my system to make sure there’s not some funky issue with that.” The short story is that, no, that changed nothing.
So then… I figured that perhaps I would have to provide a rom file for my GPU that I could supply to the virt manager configuration. However, that also didn’t work. (I didn’t make this rom file myself, for the record, I found it on the internet @ https://www.techpowerup.com/vgabios/ so I suppose that could be a problem. It did seem to have my specific card, but I’m not sure if there’s some kind of model revision.)
Oh yeah, and here’s my Virt Manager XML configuration as-is right now. Perhaps someone can highlight an error in there and solve this problem with a simple correction.
Unfold for Virt Manager Coniguration
<domain type='kvm'>
<name>win10</name>
<uuid>229093b2-a487-442f-b9aa-7f0e9445d521</uuid>
<metadata>
<libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
<libosinfo:os id="http://microsoft.com/win/10"/>
</libosinfo:libosinfo>
</metadata>
<memory unit='KiB'>30720000</memory>
<currentMemory unit='KiB'>30720000</currentMemory>
<memoryBacking>
<source type='memfd'/>
<access mode='shared'/>
</memoryBacking>
<vcpu placement='static'>14</vcpu>
<os firmware='efi'>
<type arch='x86_64' machine='pc-q35-8.2'>hvm</type>
<firmware>
<feature enabled='yes' name='enrolled-keys'/>
<feature enabled='yes' name='secure-boot'/>
</firmware>
<loader readonly='yes' secure='yes' type='pflash' format='qcow2'>/usr/share/edk2/ovmf/OVMF_CODE_4M.secboot.qcow2</loader>
<nvram template='/usr/share/edk2/ovmf/OVMF_VARS_4M.secboot.qcow2' format='qcow2'>/var/lib/libvirt/qemu/nvram/win10_VARS.qcow2</nvram>
<bootmenu enable='yes'/>
</os>
<features>
<acpi/>
<apic/>
<hyperv mode='custom'>
<relaxed state='on'/>
<vapic state='on'/>
<spinlocks state='on' retries='8191'/>
<vendor_id state='on' value='whatever'/>
</hyperv>
<kvm>
<hidden state='on'/>
</kvm>
<vmport state='off'/>
<smm state='on'/>
</features>
<cpu mode='host-passthrough' check='none' migratable='on'>
<topology sockets='1' dies='1' clusters='1' cores='7' threads='2'/>
</cpu>
<clock offset='localtime'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
<timer name='hypervclock' present='yes'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled='no'/>
<suspend-to-disk enabled='no'/>
</pm>
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/home/eoin/.var/app/org.gnome.Boxes/data/gnome-boxes/images/virt-convert/win11-2'/>
<target dev='sda' bus='sata'/>
<boot order='2'/>
<address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>
<disk type='file' device='cdrom'>
<driver name='qemu' type='raw'/>
<target dev='sdb' bus='sata'/>
<readonly/>
<boot order='1'/>
<address type='drive' controller='0' bus='0' target='0' unit='1'/>
</disk>
<controller type='usb' index='0' model='qemu-xhci' ports='15'>
<address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
</controller>
<controller type='pci' index='0' model='pcie-root'/>
<controller type='pci' index='1' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='1' port='0x10'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
</controller>
<controller type='pci' index='2' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='2' port='0x11'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
</controller>
<controller type='pci' index='3' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='3' port='0x12'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
</controller>
<controller type='pci' index='4' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='4' port='0x13'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
</controller>
<controller type='pci' index='5' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='5' port='0x14'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
</controller>
<controller type='pci' index='6' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='6' port='0x15'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>
</controller>
<controller type='pci' index='7' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='7' port='0x16'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x6'/>
</controller>
<controller type='pci' index='8' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='8' port='0x17'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x7'/>
</controller>
<controller type='pci' index='9' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='9' port='0x18'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0' multifunction='on'/>
</controller>
<controller type='pci' index='10' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='10' port='0x19'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x1'/>
</controller>
<controller type='pci' index='11' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='11' port='0x1a'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x2'/>
</controller>
<controller type='pci' index='12' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='12' port='0x1b'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x3'/>
</controller>
<controller type='pci' index='13' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='13' port='0x1c'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x4'/>
</controller>
<controller type='pci' index='14' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='14' port='0x1d'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x5'/>
</controller>
<controller type='sata' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
</controller>
<controller type='virtio-serial' index='0'>
<address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
</controller>
<interface type='network'>
<mac address='52:54:00:35:0c:23'/>
<source network='default'/>
<model type='e1000e'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</interface>
<serial type='pty'>
<target type='isa-serial' port='0'>
<model name='isa-serial'/>
</target>
</serial>
<console type='pty'>
<target type='serial' port='0'/>
</console>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<tpm model='tpm-crb'>
<backend type='emulator' version='2.0'/>
</tpm>
<sound model='ich9'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1b' function='0x0'/>
</sound>
<audio id='1' type='none'/>
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x23' slot='0x00' function='0x0'/>
</source>
<rom bar='on' file='/var/home/eoin/Downloads/NVIDIA.GTX10603GB.3072.181212.rom'/>
<address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x23' slot='0x00' function='0x1'/>
</source>
<address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
</hostdev>
<watchdog model='itco' action='reset'/>
<memballoon model='virtio'>
<address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
</memballoon>
</devices>
</domain>
So now I’m at an impasse. Any advice for someone like me rocking a relatively old card that wants to have a taste of the sweet GPU passthrough but struggling to get an image to display from the graphics card? Additionally, am I correct about how this should work where I can no longer use spice to view my VM and have to pipe all graphics through the graphics card (without looking-glass, of course, but I figure I should walk before I run.) Is this graphics card simply too old to achieve passthrough? Is there something else I’m doing wrong?
If you’ve made it to the end of this, thanks, and hopefully there’s a simple reason that this isn’t working. And sorry for the links in code blocks, I’m not allowed to link as I’m too new to the forum I suppose so I instead found a work around. 
I should have tried a direct connection earlier. But I’m getting good performance now on my system after all.