GPU passthrough in Ubuntu 18.04 using libvirtd/VMM/Virt-Manager?

Hardware
1

Hey,

I’ve been wanting to try KVM for passthrough because it seems like a lot of people like it, but I haven’t run across a guide yet for using libvirtd/VMM/virt-manager, they’re all for virt-install from the command line.

I’ve used VMM to a pretty large extent (not sure which people are calling it these days) but mostly on servers and never for GPU passthrough. So I thought I’d just follow along to the part where I set up the vfio and then do the rest in VMM. Right? Wrong!

When I try to start a Windows VM that I made before passing through the GPU+HDMI+USB, it just hangs. Then VMM takes a shit and says it can’t connect to qemu-kvm.

I’ve tried using virt-install, too, but it doesn’t give me any output, it just hangs and I have to kill the PID manually from another terminal (can’t even CTRL-C out of it).

I probably should have installed OVMF first before making the VM I tried, maybe it’s because the VM is using BIOS boot/ BX440 firmware (?). Is that a dealbreaker?

Here’s what I’m working with:

Supermicro X10SRL-F 2011-3 w/ E5-2650 v4 QS QHZH - C612 Chipset
NVIDIA 740 GT SC + IPMI BMC (onboard)
64GB DDR4, 860 EVO 500GB (spare drive for experiments like this)

Not sure what else in HW spec I need to put in because I have this:

IOMMU group 17
	00:03.0 PCI bridge [0604]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 3 [8086:6f08]
IOMMU group 35
	0a:00.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [1000:0072] (rev 03)
IOMMU group 7
	ff:14.0 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 0 Thermal Control [8086:6fb0]
	ff:14.1 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 1 Thermal Control [8086:6fb1]
	ff:14.2 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 0 Error [8086:6fb2]
	ff:14.3 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 1 Error [8086:6fb3]
	ff:14.4 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Interface [8086:6fbc]
	ff:14.5 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Interface [8086:6fbd]
	ff:14.6 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Interface [8086:6fbe]
	ff:14.7 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Interface [8086:6fbf]
IOMMU group 25
	00:1c.0 PCI bridge [0604]: Intel Corporation C610/X99 series chipset PCI Express Root Port #1 [8086:8d10] (rev d5)
IOMMU group 15
	00:02.0 PCI bridge [0604]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 2 [8086:6f04]
IOMMU group 33
	03:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8609 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA [10b5:8609] (rev ba)
	03:00.1 System peripheral [0880]: PLX Technology, Inc. PEX 8609 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA [10b5:8609] (rev ba)
	04:01.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8609 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA [10b5:8609] (rev ba)
	04:05.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8609 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA [10b5:8609] (rev ba)
	04:07.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8609 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA [10b5:8609] (rev ba)
	04:09.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8609 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA [10b5:8609] (rev ba)
	05:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller [1b21:1142]
	06:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller [1b21:1142]
	07:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller [1b21:1142]
	08:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller [1b21:1142]
IOMMU group 5
	ff:12.0 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Home Agent 0 [8086:6fa0]
	ff:12.1 Performance counters [1101]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Home Agent 0 [8086:6f30]
	ff:12.4 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Home Agent 1 [8086:6f60]
	ff:12.5 Performance counters [1101]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Home Agent 1 [8086:6f38]
IOMMU group 23
	00:16.0 Communication controller [0780]: Intel Corporation C610/X99 series chipset MEI Controller #1 [8086:8d3a] (rev 05)
	00:16.1 Communication controller [0780]: Intel Corporation C610/X99 series chipset MEI Controller #2 [8086:8d3b] (rev 05)
IOMMU group 13
	00:01.0 PCI bridge [0604]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 1 [8086:6f02]
IOMMU group 41
	10:00.0 PCI bridge [0604]: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge [1a03:1150] (rev 03)
	11:00.0 VGA compatible controller [0300]: ASPEED Technology, Inc. ASPEED Graphics Family [1a03:2000] (rev 30)
IOMMU group 31
	01:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 [8086:1528] (rev 01)
IOMMU group 3
	ff:0f.0 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent [8086:6ff8]
	ff:0f.1 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent [8086:6ff9]
	ff:0f.2 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent [8086:6ffa]
	ff:0f.3 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent [8086:6ffb]
	ff:0f.4 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent [8086:6ffc]
	ff:0f.5 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent [8086:6ffd]
	ff:0f.6 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent [8086:6ffe]
IOMMU group 21
	00:11.0 Unassigned class [ff00]: Intel Corporation C610/X99 series chipset SPSR [8086:8d7c] (rev 05)
	00:11.4 SATA controller [0106]: Intel Corporation C610/X99 series chipset sSATA Controller [AHCI mode] [8086:8d62] (rev 05)
IOMMU group 11
	ff:1f.0 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit [8086:6f88]
	ff:1f.2 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit [8086:6f8a]
IOMMU group 1
	ff:0c.0 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent [8086:6fe0]
	ff:0c.1 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent [8086:6fe1]
	ff:0c.2 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent [8086:6fe2]
	ff:0c.3 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent [8086:6fe3]
	ff:0c.4 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent [8086:6fe4]
	ff:0c.5 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent [8086:6fe5]
	ff:0c.6 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent [8086:6fe6]
	ff:0c.7 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent [8086:6fe7]
IOMMU group 38
	0d:00.0 USB controller [0c03]: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller [1912:0015] (rev 02)
IOMMU group 28
	00:1c.6 PCI bridge [0604]: Intel Corporation C610/X99 series chipset PCI Express Root Port #7 [8086:8d1c] (rev d5)
IOMMU group 18
	00:03.2 PCI bridge [0604]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 3 [8086:6f0a]
IOMMU group 36
	0b:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK107 [GeForce GT 740] [10de:0fc8] (rev a1)
	0b:00.1 Audio device [0403]: NVIDIA Corporation GK107 HDMI Audio Controller [10de:0e1b] (rev a1)
IOMMU group 8
	ff:16.0 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Target Address/Thermal/RAS [8086:6f68]
	ff:16.1 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Target Address/Thermal/RAS [8086:6f79]
	ff:16.2 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Channel Target Address Decoder [8086:6f6a]
	ff:16.3 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Channel Target Address Decoder [8086:6f6b]
	ff:16.6 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 2/3 Broadcast [8086:6f6e]
	ff:16.7 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Global Broadcast [8086:6f6f]
IOMMU group 26
	00:1c.4 PCI bridge [0604]: Intel Corporation C610/X99 series chipset PCI Express Root Port #5 [8086:8d18] (rev d5)
IOMMU group 16
	00:02.2 PCI bridge [0604]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 2 [8086:6f06]
IOMMU group 34
	09:00.0 Non-Volatile memory controller [0108]: Device [1c5c:1285]
IOMMU group 6
	ff:13.0 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Target Address/Thermal/RAS [8086:6fa8]
	ff:13.1 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Target Address/Thermal/RAS [8086:6f71]
	ff:13.2 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address Decoder [8086:6faa]
	ff:13.3 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address Decoder [8086:6fab]
	ff:13.6 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Broadcast [8086:6fae]
	ff:13.7 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Global Broadcast [8086:6faf]
IOMMU group 24
	00:1a.0 USB controller [0c03]: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #2 [8086:8d2d] (rev 05)
IOMMU group 14
	00:01.1 PCI bridge [0604]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 1 [8086:6f03]
IOMMU group 32
	01:00.1 Ethernet controller [0200]: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 [8086:1528] (rev 01)
IOMMU group 4
	ff:10.0 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R2PCIe Agent [8086:6f1d]
	ff:10.1 Performance counters [1101]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R2PCIe Agent [8086:6f34]
	ff:10.5 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Ubox [8086:6f1e]
	ff:10.6 Performance counters [1101]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Ubox [8086:6f7d]
	ff:10.7 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Ubox [8086:6f1f]
IOMMU group 22
	00:14.0 USB controller [0c03]: Intel Corporation C610/X99 series chipset USB xHCI Host Controller [8086:8d31] (rev 05)
IOMMU group 12
	00:00.0 Host bridge [0600]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DMI2 [8086:6f00]
IOMMU group 40
	0f:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)
IOMMU group 30
	00:1f.0 ISA bridge [0601]: Intel Corporation C610/X99 series chipset LPC Controller [8086:8d44] (rev 05)
	00:1f.3 SMBus [0c05]: Intel Corporation C610/X99 series chipset SMBus Controller [8086:8d22] (rev 05)
IOMMU group 2
	ff:0d.0 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent [8086:6fe8]
	ff:0d.1 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent [8086:6fe9]
	ff:0d.2 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent [8086:6fea]
	ff:0d.3 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent [8086:6feb]
IOMMU group 20
	00:05.0 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Map/VTd_Misc/System Management [8086:6f28]
	00:05.1 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D IIO Hot Plug [8086:6f29]
	00:05.2 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D IIO RAS/Control Status/Global Errors [8086:6f2a]
	00:05.4 PIC [0800]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D I/O APIC [8086:6f2c]
IOMMU group 10
	ff:1e.0 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit [8086:6f98]
	ff:1e.1 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit [8086:6f99]
	ff:1e.2 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit [8086:6f9a]
	ff:1e.3 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit [8086:6fc0]
	ff:1e.4 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit [8086:6f9c]
IOMMU group 39
	0e:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)
IOMMU group 29
	00:1d.0 USB controller [0c03]: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #1 [8086:8d26] (rev 05)
IOMMU group 0
	ff:0b.0 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R3 QPI Link 0/1 [8086:6f81]
	ff:0b.1 Performance counters [1101]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R3 QPI Link 0/1 [8086:6f36]
	ff:0b.2 Performance counters [1101]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R3 QPI Link 0/1 [8086:6f37]
	ff:0b.3 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R3 QPI Link Debug [8086:6f76]
IOMMU group 19
	00:04.0 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 0 [8086:6f20]
	00:04.1 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 1 [8086:6f21]
	00:04.2 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 2 [8086:6f22]
	00:04.3 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 3 [8086:6f23]
	00:04.4 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 4 [8086:6f24]
	00:04.5 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 5 [8086:6f25]
	00:04.6 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 6 [8086:6f26]
	00:04.7 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 7 [8086:6f27]
IOMMU group 37
	0c:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM951/PM951 [144d:a802] (rev 01)
IOMMU group 9
	ff:17.0 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 1 - Channel 0 Thermal Control [8086:6fd0]
	ff:17.1 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 1 - Channel 1 Thermal Control [8086:6fd1]
	ff:17.2 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 1 - Channel 0 Error [8086:6fd2]
	ff:17.3 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 1 - Channel 1 Error [8086:6fd3]
	ff:17.4 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 2/3 Interface [8086:6fb8]
	ff:17.5 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 2/3 Interface [8086:6fb9]
	ff:17.6 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 2/3 Interface [8086:6fba]
	ff:17.7 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 2/3 Interface [8086:6fbb]
IOMMU group 27
	00:1c.5 PCI bridge [0604]: Intel Corporation C610/X99 series chipset PCI Express Root Port #6 [8086:8d1a] (rev d5)

You can see I have a lot of PCIe cards in the system because I’ve been using it for GPU passthrough with ESXi.

But anyway, the grouping looks pretty good from what I can tell, the GPU is on its own group with the HDMI audio, the USB I was going to pass through is on its own card with its own audio.

My /etc/modprobe.d/vfio.conf : or /etc/modules-load.d/vfio.conf

options vfio-pci ids=1912:0015,10de:0fc8,10de:0e1b

and etc/modules-load.d/vfio.conf:

vfio
vfio-pci

ran update-grub after adding intel_iommu=on on grub default line, update-initramfs -u, etc.

I’m guessing the next thing I should try is blacklisting the nouveau driver / nvidia card, but I’ve run lsmod and there’s no nvidia or nouveau drivers loaded

Is there anything obvious anyone can see that I am doing wrong?

Also, does anyone use VMM from a desktop installation for their GPU passthrough VMs, or is that kinda like something people have already figured out doesn’t work? Does everyone use the command line because it won’t work the desktop way?

Thanks!

2

Hi, there are a couple of tutorials on this very forum for setting up Windows KVM with GPU passthrough in Virt Manager (VMM).

First things first. You probably need to use OVMF, since setting it up with BIOS can be tricky/require some extra stuff, where OVMF normally works out of the box on most GPUs that support UEFI.

If you are unsure if your graphics device is bound to vfio, run lspci -k in terminal to list your PCI devices and what driver they are bound to.

IOMMU grouping looks stellar, GPU has it’s own group so there is nothing there than can cause any issues.

Furthermore, check out the following link, it goes to a tutorial on setting up KVM with Virt Manager. Did the same setup a few weeks ago on Ubuntu 18.04 and can confirm it works.

https://forum.level1techs.com/t/play-games-in-windows-on-linux-pci-passthrough-quick-guide/108981

3

If you wanna try fedora 28 I posted a ansible playbook that does all the set up for you.

4

Funny thing, about half an hour after writing the OP I was editing it to say that I had gotten it to boot using OVMF and blacklisting nouveau - but I forgot to hit “send”.

Was a LOT easier than the first time I tried with a i7+Q67 board, ACS and groupings really help a lot!

lspci -k flag is new, thanks for that. I think I always typed -nnn or -vvv or something.

My only issue when I first booted was the nVidia error 43, which thankfully my version of KVM is new enough I could just set hidden=on with virsh. One time I actually recompiled the drivers in Windows in order to try and get rid of it on an ESXi host, I guess before I had learned of hypervisor.cpu0=FALSE.

One thing that is troubling me is in Virt-Manager I can’t seem to allocate more than two cores for Windows 10. I’ve seen a bunch of different things about why that may be - one thread says licensing restrictions for multisocket boards.

According to an official RedHat tuning guide I read, KVM recommends increasing # of sockets and leaving cores and threads at 1 for best performance, but that would obviously be quite limiting if W10 is restricted to two sockets. Although, you would think I’d be able to adjust it with the manual topology settings, but no - that didn’t make any difference.

Different processor profiles, copying profile from host, doesn’t matter. Two cores only. I got tired of rebooting the VM over and over and finally gave up. I am running W10 Pro.

Any idea what that is about? Can you allocate more than two cores for your Windows VM?

5

Honestly, if anything I’d go back to OpenSUSE, its disaster recovery profile w/ snapper+zypper+BTRFS is phenomenal, and there were a LOT more options for things like OVMF profiles.

But I’d be curious to see it - is it on a developer repo board?

6
7

I have a total of 6 cores on the VM leaving 2 to my main system to avoid performance issues related to latency and VM and main system competing for processor resources.

My topology is set to 1 socket, 3 cores, 2 threads. I had some strange results when adding additional sockets as well, and for me adding more sockets did not necessarily equate better performance.

8

Yep! I think I figured it out, I set ‘threads’ to be a multiplier after reading the literature from RedHat a little more carefully. With 1 socket, 2 cores and 2 threads I got 4 cores in Windows. I thought I had done this previously but I guess I was wrong…