VFIO in 2019 -- Pop!_OS How-To (General Guide though) [DRAFT]

It’s interesting to find out that the VFIO is actually working for you Tchuyev, it’s good that this seems to be only a reporting issue.

Thank you for linking me to the issue btblueskies, I will share my experience as well on the ticket and hopefully we can get this resolved soon.

Hey Guys,
I‘d like to ask if there‘s an update regarding the POP_OS 20.04 issues with the passthrough?

It still isn‘t working for me.

Same here! Whenever I isolate the Nvidia Card and reboot, I just get a black screen. I’m positive I’m only isolating the Nvidia and it’s in it’s own IOMMU group.

Weirder still, if boot with both cards enabled and then disable the Nvidia screen or unplug the cable, the system becomes very sluggish. Plug it the monitor back in or enable it in settings and the system runs fine.

An interesting new problem has developed on my otherwise working setup. Windows updated to 1909 on it’s own and now seemingly randomly if I walk away for a period of time the VM will go to sleep (pause) and if I try to resume it it just spits out an error message:

Error unpausing domain: internal error: unable to execute QEMU command 'cont': Resetting the Virtual Machine is required

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 75, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 111, in tmpcb
    callback(*args, **kwargs)
  File "/usr/share/virt-manager/virtManager/libvirtobject.py", line 66, in newfn
    ret = fn(self, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/domain.py", line 1435, in resume
    self._backend.resume()
  File "/usr/lib/python3/dist-packages/libvirt.py", line 2012, in resume
    if ret == -1: raise libvirtError ('virDomainResume() failed', dom=self)
libvirt.libvirtError: internal error: unable to execute QEMU command 'cont': Resetting the Virtual Machine is required

I tried disabling everything I can find that pertains to sleep in Windows but the VM keeps pausing. Anybody else run into this?

1 Like

having issues with huge pages right now its causing my ram to be 19gb without even starting the vm. how would i fix this? im using pop os 20.04

Any chance we can get the OP updated to include info on how to work with kernel 5.4 since vfio is built in?

Information on how to add options to systemd-boot would be helpful as well.

I did find some of that information here. That helped me get amd_iommu=on added correctly, and I think my gpu is isolated properly.

But passing through my pci-e USB card is not working. I get a “Failed to set iommu for container: Operation not permitted.” error. It mentions the iommu group for my usb card, so I know that is the issue.

I updated AppArmor per this post. But it didn’t seem to help.

syslog does show this error:

Jul 29 17:00:55 monolith kernel: [  169.269934] vfio_iommu_type1_attach_group: No interrupt remapping support.  Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform

So I tried to enable allow_unsafe_interrupts=1 the same way I added amd_iommu=on, but syslog is still logging the same error message after a reboot.

Edit:

Per this Proxmox forum post I was able to add the vfio_iommu_type1.allow_unsafe_interrupts=1 option. I just needed the module name in front, then add it the same way you do amd_iommu=on.

Now I think my usb card is being passed through, but when I switch monitor inputs to the Windows gpu, I don’t get any output from it. Syslog does show the external hdd I have connected to the usb card being added and removed when I turn on the vm, and/or turn it off.

Hmm…

Edit 2:
So, it seems that part of the no display issue may have just been Windows not showing anything without some kind of input from a keyboard or mouse. I added a QXL display, and after waiting a good while to be sure it booted, I hit a key on my keyboard, and the login screen shows up in the QXL window. If I switch to the GPU input on my monitor, and can see Windows as well.

Now if only my USB card was working again…

Can one pass through USB sound card dynamically? For example, I’d prefer not to have two sound cards - but when switching between the VM & host, I’d like to be able to re-assign the sound card. Otherwise, to have audio sometimes working in either, I’d have to have two sound cards and a mixer - unless someone has another idea :slight_smile:

I have a Gigabyte Z390 AORUS Master. My secondary 1080Ti and NVMe controller both have their own IOMMU group, but there is only one entry for USB.

IOMMU Group 0 00:00.0 Host bridge [0600]: Intel Corporation 8th Gen Core 8-core Desktop Processor Host Bridge/DRAM Registers [Coffee Lake S] [8086:3e30] (rev 0a)
IOMMU Group 10 00:1c.5 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #6 [8086:a33d] (rev f0)
IOMMU Group 11 00:1d.0 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #9 [8086:a330] (rev f0)
IOMMU Group 12 00:1f.0 ISA bridge [0601]: Intel Corporation Z390 Chipset LPC/eSPI Controller [8086:a305] (rev 10)
IOMMU Group 12 00:1f.3 Audio device [0403]: Intel Corporation Cannon Lake PCH cAVS [8086:a348] (rev 10)
IOMMU Group 12 00:1f.4 SMBus [0c05]: Intel Corporation Cannon Lake PCH SMBus Controller [8086:a323] (rev 10)
IOMMU Group 12 00:1f.5 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH SPI Controller [8086:a324] (rev 10)
IOMMU Group 12 00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (7) I219-V [8086:15bc] (rev 10)
IOMMU Group 13 04:00.0 Non-Volatile memory controller [0108]: Phison Electronics Corporation E12 NVMe Controller [1987:5012] (rev 01)
IOMMU Group 14 05:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP102 [GeForce GTX 1080 Ti] [10de:1b06] (rev a1)
IOMMU Group 14 05:00.1 Audio device [0403]: NVIDIA Corporation GP102 HDMI Audio Controller [10de:10ef] (rev a1)
IOMMU Group 15 07:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 01)
IOMMU Group 16 08:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 [144d:a808]
IOMMU Group 1 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 0a)
IOMMU Group 1 00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x8) [8086:1905] (rev 0a)
IOMMU Group 1 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP102 [GeForce GTX 1080 Ti] [10de:1b06] (rev a1)
IOMMU Group 1 01:00.1 Audio device [0403]: NVIDIA Corporation GP102 HDMI Audio Controller [10de:10ef] (rev a1)
IOMMU Group 1 02:00.0 Serial Attached SCSI controller [0107]: Broadcom / LSI SAS2116 PCI-Express Fusion-MPT SAS-2 [Meteor] [1000:0064] (rev 02)
IOMMU Group 2 00:12.0 Signal processing controller [1180]: Intel Corporation Cannon Lake PCH Thermal Controller [8086:a379] (rev 10)
IOMMU Group 3 00:14.0 USB controller [0c03]: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller [8086:a36d] (rev 10)
IOMMU Group 3 00:14.2 RAM memory [0500]: Intel Corporation Cannon Lake PCH Shared SRAM [8086:a36f] (rev 10)
IOMMU Group 4 00:16.0 Communication controller [0780]: Intel Corporation Cannon Lake PCH HECI Controller [8086:a360] (rev 10)
IOMMU Group 5 00:17.0 SATA controller [0106]: Intel Corporation Cannon Lake PCH SATA AHCI Controller [8086:a352] (rev 10)
IOMMU Group 6 00:1b.0 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #17 [8086:a340] (rev f0)
IOMMU Group 7 00:1b.4 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #21 [8086:a32c] (rev f0)
IOMMU Group 8 00:1b.6 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #23 [8086:a32e] (rev f0)
IOMMU Group 9 00:1c.0 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #1 [8086:a338] (rev f0)

This means I’ll need a separate USB controller to pass through keyboard and mouse, right? I have one PCIe x1 slot available. Suggestions?

edit Some possibly useful links for people using Ubuntu 20.04 or Mint 20
https://mathiashueber.com/pci-passthrough-ubuntu-2004-virtual-machine/ (posted above by @jerrac)
https://heiko-sieger.info/running-windows-10-on-linux-using-kvm-with-vga-passthrough/

edit2 One of the guides above indicates that it’s not necessary to add the NVMe id to the VFIO script. Is there no performance benefit to doing so?

Well… I got as far as seeing ‘Press any key to boot from CD or DVD…’ from the Windows 10 2004 ISO, but it doesn’t respond to keypresses. Tried passing through the USB keyboard and mouse individually as well as passing through the entire xHCI USB controller. I have a second USB controller on the way. It’s the one recommended here.

edit Got the Windows installation running. I’m guessing that adding VirtIO Keyboard is what did the trick. I had just proceeded to the Looking Glass instructions since I knew I was going to want it eventually.

Trying to get Looking Glass working, but Windows 10 complains about the signature for the IVSHMEM driver.

edit Disabled Secure Boot after finding this post and I’m up and running.

1 Like

Passing a separate USB controller to the guest and using a USB switch to swap the USB DAC between the two seems like the easiest solution.

1 Like

The link to the ‘How To’ for adding a systemd service to disable transparent hugepages is broken. Here’s a working link.

A physical USB switch? Or can the connect/disconnect of USB pass-through for an audio device (between guest & host) be done entirely in software?

Might be possible in software, but I pass one whole USB controller through, and have two extension cords to my desk; one for host, on for guest.
If you pass the controller through, USB hot plug just works.
If you pass a device through, and it disconnects from host (for whatever reason), it won’t go back to guest upon reconnection.
Unlike pci, which you isolate from boot, and don’t hot plug

1 Like

Yes, a physical switch. It can be done in software, but it’s so much less of a headache to manage and troubleshoot if you can just give the guest its own USB controller.

Cool, as long as it doesn’t add latency. The whole point for me was having a full quality sound card passed through for proaudio to a Windows guest - without a latency increase. Some of these USB3 audio interfaces can do <5ms now.

I dont have Pop!_OS, but Ubuntu 20.10.
Allready had qemu with omvf and passthrough running, but when I added a second r9 280x to the system I was unable to figure out how to pass only one of them to a VM. By using your initramfs script, I now have that working perfectly. There is only one issue I am hoping someone could help with.

I have 2x r9 280x and one RX550. One 280x is handed to amdgpu and used for host desktop. One RX550 is bound to vfio-pci and is passed to a windows vm. The last 280x is added to the override script, and is then passed to a VM.
This works perfectly, untill I reboot the VM with the 280X. When the VM shuts down, for some reason the gpu gets handed to the host, wich immediatly initializes and binds it to amdgpu driver. I can then start the VM again, and it will actually get unbound from amdgpu driver automatically, and the VM boots fine. However, if I reboot a second time, I end up with a reset bug, and the GPU will be unusable untill I reboot the host.
The RX550 bound by vfio-pci, handles unlimited reboots, and never gets initialized by host.

So, to my question. Is there some way to keep the host from seizing the gpu when vm gets shutdown? I belive thats what causes the reset bug in this case.

Is VFIO possible on a Razer blade 2019 base w/gtx1660ti? using Ubuntu 20.04LTS as host.

It appears that no Iommu groups are showing up on my system. I have not found any other posts with a similar issue. The output of the IOMMU script on my ROG STRIX Z390-H mobo (with VT-d enabled) is as follows:

IOMMU Group * 00:00.0 Host bridge [0600]: Intel Corporation 8th Gen Core 8-core Desktop Processor Host Bridge/DRAM Registers [Coffee Lake S] [8086:3e30] (rev 0a)
00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 0a)
00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x8) [8086:1905] (rev 0a)
00:14.0 USB controller [0c03]: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller [8086:a36d] (rev 10)
00:14.2 RAM memory [0500]: Intel Corporation Cannon Lake PCH Shared SRAM [8086:a36f] (rev 10)
00:16.0 Communication controller [0780]: Intel Corporation Cannon Lake PCH HECI Controller [8086:a360] (rev 10)
00:17.0 SATA controller [0106]: Intel Corporation Cannon Lake PCH SATA AHCI Controller [8086:a352] (rev 10)
00:1b.0 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #17 [8086:a340] (rev f0)
00:1c.0 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #1 [8086:a338] (rev f0)
00:1c.5 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #6 [8086:a33d] (rev f0)
00:1d.0 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #9 [8086:a330] (rev f0)
00:1f.0 ISA bridge [0601]: Intel Corporation Z390 Chipset LPC/eSPI Controller [8086:a305] (rev 10)
00:1f.3 Audio device [0403]: Intel Corporation Cannon Lake PCH cAVS [8086:a348] (rev 10)
00:1f.4 SMBus [0c05]: Intel Corporation Cannon Lake PCH SMBus Controller [8086:a323] (rev 10)
00:1f.5 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH SPI Controller [8086:a324] (rev 10)
00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (7) I219-V [8086:15bc] (rev 10)
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP102 [GeForce GTX 1080 Ti] [10de:1b06] (rev a1)
01:00.1 Audio device [0403]: NVIDIA Corporation GP102 HDMI Audio Controller [10de:10ef] (rev a1)
02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP102 [GeForce GTX 1080 Ti] [10de:1b06] (rev a1)
02:00.1 Audio device [0403]: NVIDIA Corporation GP102 HDMI Audio Controller [10de:10ef] (rev a1)
05:00.0 SATA controller [0106]: Marvell Technology Group Ltd. Device [1b4b:9215] (rev 11)

Any thoughts?

@wendell you could add https://github.com/duncanthrax/scream to Audio section as option.

I use Scream over year now and I didn’t had any problem, there is no CPU usage, quality is good and there is no latency, it uses same technology as Looking Glass, it uses IVSHMEM for sharing audio data between guest and host.

And for VM input (like KVM switch) I use evdev --> https://passthroughpo.st/using-evdev-passthrough-seamless-vm-input/ it is built into kernel, so it is efficient and no latency here.