Well, here I am again. This time with way more details. I made some progress but still not there.
I’ll copy what I posted on reddit, hopefully someone has some insights
Specs:
AMD Ryzen 3800X
AMD Red Dragon 5700XT
Fedora 34 with kernel 5.11.20-300.fc34.x86_64
Guest: Win 10 Pro VM
Libvirt directory tree:
/etc/libvirt/hooks tree .
.
├── qemu
└── qemu.d
└── win10
├── prepare
│ └── begin
│ └── start.sh
└── release
└── end
└── stop.sh
6 directories, 3 files
/etc/libvirt/hooks/qemu:
#!/bin/bash
GUEST_NAME="$1"
HOOK_NAME="$2"
STATE_NAME="$3"
MISC="${@:4}"
BASEDIR="$(dirname $0)"
HOOKPATH="$BASEDIR/qemu.d/$GUEST_NAME/$HOOK_NAME/$STATE_NAME"
set -e # If a script exits with an error, we should as well.
# check if it's a non-empty executable file
if [ -f "$HOOKPATH" ] && [ -s "$HOOKPATH"] && [ -x "$HOOKPATH" ]; then
eval \"$HOOKPATH\" "$@"
elif [ -d "$HOOKPATH" ]; then
while read file; do
# check for null string
if [ ! -z "$file" ]; then
eval \"$file\" "$@"
fi
done <<< "$(find -L "$HOOKPATH" -maxdepth 1 -type f -executable -print;)"
fi
Start.sh hook:
#!/bin/bash
# Helpful to read output when debugging
set -x
# Stop your display manager. If youre on kde it ll be sddm.service. Gnome users should use killall gdm-x-session instead
systemctl stop display-manager.service
pipewire_pid=$(pgrep -u me pipewire)
gdm_pid=$(pgrep -u me gdm)
kill $pipewire_pid
kill $gdm_pid
# Unbind VTconsoles
echo 0 > /sys/class/vtconsole/vtcon0/bind
echo 0 > /sys/class/vtconsole/vtcon1/bind
# Avoid a race condition by waiting a couple of seconds. This can be calibrated to be shorter or longer if required for your system
sleep 4
# Unload all Radeon drivers
modprobe -r amdgpu
# Unbind the GPU from display driver
virsh nodedev-detach pci_0000_2b_00_0 < gpu
virsh nodedev-detach pci_0000_2b_00_1 < gpu audio
# Load VFIO kernel module
modprobe vfio
modprobe vfio_pci
modprobe vfio_iommu_type1
Stop.sh hook:
#!/bin/bash
# Helpful to read output when debugging
set -x
# Unload all the vfio modules
modprobe -r vfio_pci
modprobe -r vfio_iommu_type1
modprobe -r vfio
# Reattach the gpu
virsh nodedev-reattach pci_0000_2b_00_0 < gpu
virsh nodedev-reattach pci_0000_2b_00_1 < gpu audio
# Load all Radeon drivers
modprobe amdgpu
modprobe snd_hda_intel
modprobe ttm
modprobe drm_kms_helper
modprobe i2c_algo_bit
modprobe drm
# Start you display manager
systemctl start display-manager.service
Kernel params:
GRUB_CMDLINE_LINUX="rhgb quiet iommu=1 amd_iommu=on rd.driver.pre=vfio-pci"
After doing this and creating a Win10 VM in virt-manager, passing the GPU as a device and changing to host-passthrough I actually managed to install Windows 10… Unfortunately when I shut down the machine everything crashed and had to reboot my PC manually.
After rebooting I was logged into emergency/rescue mode due to something with initramfs and my btrfs partition being REALLY fucked. I tried to solve the issue but no no avail, I ended up formatting my PC which was really annoying.
Now… this morning, fresh install again. I did exactly the same changes to my setup and on top of that I added this:
cat vfio.conf
add_drivers+=" vfio-pci "
⚡ root@fedora /etc/dracut.conf.d pwd
/etc/dracut.conf.d
Then ran ‘dracut -f’ to apply the changes to my current kernel version. Re-deployed a Win 10 VM and it boots fine but after the windows 10 installation is finished and the guest is about to restart, the VM (and my host) crash and I’m forced to a hard reboot. Thankfully this time the host is not entirely broken so I can continue testing.
Any ideas what’s happening? I feel I’m SO close to getting this to work but I’m failing somewhere. I’m not really knowledgeable with dracut and I’m keen on removing the vfio-pci from being automatically loaded at boot time, but I’m afraid this will get me through the installation (as it did last night) but then break my system again, forcing me to re-install and spend 2 more hours.
Win 10 VM XML config example (ignore the amount of RAM and vCPUs) :
<domain type="kvm">
<name>win10</name>
<uuid>64fac0d7-32d7-424b-84ad-7ed5e9177a78</uuid>
<metadata>
<libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
<libosinfo:os id="http://microsoft.com/win/10"/>
</libosinfo:libosinfo>
</metadata>
<memory>12288000</memory>
<currentMemory>12288000</currentMemory>
<vcpu>8</vcpu>
<os>
<type arch="x86_64" machine="q35">hvm</type>
<loader readonly="yes" type="pflash">/usr/share/edk2/ovmf/OVMF_CODE.fd</loader>
<bootmenu enable="yes"/>
</os>
<features>
<acpi/>
<apic/>
<hyperv>
<relaxed state="on"/>
<vapic state="on"/>
<spinlocks state="on" retries="8191"/>
</hyperv>
<vmport state="off"/>
</features>
<cpu mode="host-passthrough"/>
<clock offset="localtime">
<timer name="rtc" tickpolicy="catchup"/>
<timer name="pit" tickpolicy="delay"/>
<timer name="hpet" present="no"/>
<timer name="hypervclock" present="yes"/>
</clock>
<pm>
<suspend-to-mem enabled="no"/>
<suspend-to-disk enabled="no"/>
</pm>
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<disk type="file" device="disk">
<driver name="qemu" type="qcow2"/>
<source file="/var/lib/libvirt/images/win10.qcow2"/>
<target dev="sda" bus="sata"/>
<boot order="4"/>
</disk>
<disk type="file" device="cdrom">
<driver name="qemu" type="raw"/>
<source file="/home/me/Downloads/Win10_21H1_EnglishInternational_x64.iso"/>
<target dev="sdb" bus="sata"/>
<readonly/>
<boot order="3"/>
</disk>
<controller type="usb" model="qemu-xhci" ports="15"/>
<interface type="network">
<source network="default"/>
<mac address="xx:xx:xx:xx"/>
<model type="e1000e"/>
</interface>
<sound model="ich9"/>
<hostdev mode="subsystem" type="pci" managed="yes">
<source>
<address domain="0" bus="43" slot="0" function="0"/>
</source>
</hostdev>
<hostdev mode="subsystem" type="pci" managed="yes">
<source>
<address domain="0" bus="43" slot="0" function="1"/>
</source>
</hostdev>
<hostdev mode="subsystem" type="usb" managed="yes">
<source>
<vendor id="0x05ac"/>
<product id="0x024f"/>
</source>
<boot order="1"/>
</hostdev>
<hostdev mode="subsystem" type="usb" managed="yes">
<source>
<vendor id="0x1532"/>
<product id="0x007b"/>
</source>
<boot order="2"/>
</hostdev>
</devices>
</domain>
Thanks in advance!
