QEMU VM being paused on startup

Hi

Im having some issues with PCIe Passthorugh with QEMU/IOMMU

System Specs
Ryzen 7 2700x
32GB DRR4@3000mhz
480GB NVME Host
500GB SSD Guest
RTX2080 Host
RX580 Guest
OS Ubuntu 18.04, Tried Fedora+Manjaro

Unable to get the XML file right now

Creating the VM works ok i can add the gpu but once the gpu is added i start the VM it gets paused straight away

Receiving an cont error, Virtual Machine needs resetting

Chipset is Q35 with OVMF UEFI

Tried with i440FX OVMF UEFI and it worked but froze on startup and hasn’t worked again

Unsure if anyone has got this before and knows of a way to fix it

Thanks


Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 89, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 125, in tmpcb
    callback(*args, **kwargs)
  File "/usr/share/virt-manager/virtManager/libvirtobject.py", line 82, in newfn
    ret = fn(self, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/domain.py", line 1543, in resume
    self._backend.resume()
  File "/usr/lib/python2.7/dist-packages/libvirt.py", line 1933, in resume
    if ret == -1: raise libvirtError ('virDomainResume() failed', dom=self)
libvirtError: internal error: unable to execute QEMU command 'cont': Resetting the Virtual Machine is required

I suspect you are running into the reset bug on the RX580.

I too suffered the symptons you described passing through my RX580 to the guest.

I got it working by applying the following:
Isolating the GPU as described in: https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#Isolating_the_GPU

Applying a patch to the kernel I found on the forums here:
https://forum.level1techs.com/t/attention-amd-vfio-users-do-not-update-your-bios/142685

https://clbin.com/VCiYJ

I hope this saves you from a nasty headache :stuck_out_tongue_winking_eye:

Tried to apply the patch and couldnt get it to work.

Do you know of a guide or anything to install the patch?

Thanks

Hi,

(Assuming Ubuntu 18.04 is the host OS)
I don’t run Ubuntu, thus I can’t test this guide myself.

https://askubuntu.com/questions/1048332/recompile-linux-kernel-of-ubuntu-18-04-with-patch-for-intel-display-drivers

Just replace any mention of the intel patch with the patch I linked to you.

Kernal Patched, no more errors on QEMU

Just getting the annoying and weird issue with booting with the VirtIO driver for the hard drive passthrough

INACCESSABLE_BOOT_DEVICE

don’t use virtual drives. SSDs are extremely cheap. I’m seeing a hell of a lot of 1 TB NVME PCI-E drives all over the place for $100. It’s mind blowing, actually.

I have an ssd passed though to the VM

using the IDE Driver it will boot, If i change it over to VirtIO it wont boot

you should be selecting “SATA” if it’s a real drive and including the custom path (/dev/sdX). Virtio is for fake virtual drives. Or, if it’s an nvme drive: you add it among the “PCI Devices”

Same issue with SATA drives. Only works on IDE

Possibly missing the sata driver but i will have to see.

One other thing. Any idea to why i get no video output to the gpu after passing it through, Spice is still added to the QEMU xml. i will post my XML later

probably because you haven’t set up the passthrough properly. Here’s the guide I used for Ubuntu (it’s been taken down along with the rest of the website it came from for some reason)

By Scott Miller | 2016/08/28 13 Comments

I have had success with PCI Passthrough and KVM under Debian Linux to pass a dedicated GPU to a Windows VM. Here are some notes!

Required:
Debian Stretch or later. Debian Stretch has the latest qemu (2.6) and libvirt (2.1.0) at time of writing. I was unsuccessful with Jessie due to fairly old qemu and libvirt, and no backports of qemu or libvirt are available at this time.
Recent motherboard and CPU which supports VT-d or AMD-Vi and IOMMU
Two video cards (GPUs). One will be for the Linux desktop and one will be dedicated for the Windows VM.
GPU brand: Get a recent AMD GPU. Nvidia disables their device in Windows with ‘Code 43‘ when virtualization is detected though there are some workarounds with recent versions of qemu. Intel passthrough is experimental at this point for Haswell or later and could be worth checking out if so inspired. AMD is the most straightforward for passthrough. Buy card, stick it in, and it works. One can even do some crazy stuff with AMD.
Second monitor connected to the passthrough GPU
Second USB keyboard and USB mouse to dedicate to the VM to make life easier. One could also use a KVM switch (the other KVM).
Windows 8.1 or later. Windows 8.1 has better UEFI support than Windows 7 and will work more care free overall with GPU devices and passthrough.
 01-gpupassthru-edit

GPU Passthrough Windows 8.1 VM on the left, Debian desktop on the right!

Reference for more info
Good general reference for PCI Passthrough:

ArchWiki
Performance
How is the performance?

With the aid of Hyper-V Enligtenments in KVM, performance is as native as bare metal.
OK Let’s Do This Bro!
Step 1: Install kvm and associated packages
sudo apt install virt-manager qemu-kvm ovmf bridge-utils uml-utilities libvirt-bin
It is a good idea to add your username to these groups for general kvm usage as a desktop user:
sudo gpasswd -a $(whoami) kvm
sudo gpasswd -a $(whoami) libvirt
sudo gpasswd -a $(whoami) libvirt-qemu
Step 2: Enable IOMMU
Edit /etc/default/grub and add the following to the GRUB_CMDLINE_LINUX_DEFAULT entry:

For Intel cpus, add:

intel_iommu=on
For AMD cpus, add:

amd_iommu=on
Step 3: Create new grub.cfg with that change
sudo grub-mkconfig -o /boot/grub/grub.cfg
Step 4: Blacklist the Linux driver of the passthrough device
Create / edit a file named, say, /etc/modprobe.d/blacklist.conf and add the following contents:
blacklist nouveau
blacklist radeon
blacklist amdgpu
Step 5: Reboot
sudo reboot
Step 6: Locate the PCI IDs of the passthrough GPU device
The following command will locate the iommu groups on your rig:
for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d); do echo "IOMMU group $(basename "$iommu_group")"; for device in $(ls -1 "$iommu_group"/devices/); do echo -n $'\t'; lspci -nns "$device"; done; done
In my case, the example output shows my AMD device to passthrough:
IOMMU group 11

        05:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Venus LE [Radeon HD 8830M] [1002:682b] (rev 87)
        05:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] [1002:aab0]
Note the PCI IDs of the GPU and also the HDMI audio device. The IDs are the last indications in brackets at the end of each device 1002:682b and 1002:aab0 in my case.
Step 7: Add PCI IDs to /etc/modprobe.d/vfio.conf
Create the file /etc/modprobe.d/vfio.conf and add both PCI IDs of the device to passthrough.

In my example, the file looks like:
# pass thru AMD gpu
options vfio-pci ids=1002:682b,1002:aab0
Step 8: Add vifo moudles to initrd
Edit the file /etc/initramfs-tools/modules

Add the following:
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
Save and then create a new initrd image via:
sudo update-initramfs -u
Step 9: Enable passthrough in qemu’s config
Edit /etc/libvirt/qemu.conf

Uncomment:
nvram = [    "/usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd"
]
Step 10: Reboot
sudo reboot
Windows VM Setup
From this point, create a VM in virt-manager or select an existing Windows VM one wishes to pass through the PCI device.

For best performance, enable Hyper-V Enlightenments for the Windows VM. A how-to guide can be found here.

To select the passthrough device, from virt-manager click on Add Hardware > PCI Host Device and select the passthrough GPU and then also its HDMI audio device. It is recommended to pass all components of a GPU device, from what I have gathered.```

Also for reference: https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF

Done everything that you have listed and everything on the Arch wiki

Worked on it last night and it seems to be the i440FX Chipset that causes the issue, If i run SATA on Q35 boots with no issues and i get it displaying on my monitor, Got it working on looking glass as well, but that SPICE keyboard and mouse capture bug is annoying.

Hopefully that gets fixed soon

Spice keyboard capture works perfectly for me. The mice capture is no good, though. I just use another mouse that I have passed through

Hows the latency with the keyboard capture?

not noticeable