Really wish Debian had a simple “dnf @virtualization” command to take care of all that for you.
I followed both your guide and @wendell’s guide and with a 770 and a 1060, I was able to get success:
Now I just need to do this with a 1080 and a 1060…
BTW, fellow furry here. I’m a raccoon with a Sony PMW-EX1R XDCAM EX camera.
PSA (kinda):
If you are using hugepages, and the VM won’t work anymore, check this https://bugzilla.redhat.com/show_bug.cgi?id=1514538
This guy reported it on F27, but I have this issue on both F27 and Tumbleweed.
More weird hardware compatibility questions: I actually managed to forget the mobo in my current build only has one pcie 16x port 
but it does have a pcie 1x port available.
There’s no pcie 1x graphics cards available where I live, so… would this work with a pcie 1x to 16x riser? Specifically model PCE164P-N03 VER006C (only one available here).
Additionally, I don’t mean to impose, but I could really use a list/point of reference/similar on gpus that’d work for the host OS while the VM uses a R9 380. Outside Nvidia GTX 7xx or newer, I have no idea what the 2nd gpu requirements are.
If for example a HD 5450 works, that’d be fantastic.
Cheers!
I’ve followed first post, and everything worked as expected, to the point of adding “vfio-pci.ids=10de:13c0,10de:0fbb” and “pcie_acs_override=downstream” to “/etc/default/grub” obviously using my own ids. The problem is that after rebooting linux, it doesn’t boot totally anymore. Just shows the following error message:
[0.174037] ACPI Error: [^^^PEG0.PEGP.EASP] Namespace lookup failure, AE_NOT_FOUND (20160831/psargs-359)
[0.174044] ACPI Error: Method parse/execution failed [\SB.PCI0.LPCB.EC._REG] (Node ffff970cee0e8668), AE_NOT_FOUND (20160831/psparse-543)
/dev/sda2: clean, 438811/14557184 files, 8646422/58203271 blocks
Here’s the dedicated thread to my MSI GT73VR:
lspci -k
00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor DRAM Controller (rev 09)
Subsystem: Gigabyte Technology Co., Ltd Xeon E3-1200 v2/3rd Gen Core processor DRAM Controller
Kernel driver in use: ivb_uncore
Kernel modules: ie31200_edac
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)
Kernel driver in use: pcieport
Kernel modules: shpchp
00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller (rev 09)
Subsystem: Gigabyte Technology Co., Ltd Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller
Kernel driver in use: i915
Kernel modules: i915
00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller (rev 04)
Subsystem: Gigabyte Technology Co., Ltd 7 Series/C210 Series Chipset Family USB xHCI Host Controller
Kernel driver in use: xhci_hcd
Kernel modules: xhci_pci
00:16.0 Communication controller: Intel Corporation 7 Series/C216 Chipset Family MEI Controller #1 (rev 04)
Subsystem: Gigabyte Technology Co., Ltd 7 Series/C216 Chipset Family MEI Controller
Kernel driver in use: mei_me
Kernel modules: mei_me
00:1a.0 USB controller: Intel Corporation 7 Series/C216 Chipset Family USB Enhanced Host Controller #2 (rev 04)
Subsystem: Gigabyte Technology Co., Ltd 7 Series/C216 Chipset Family USB Enhanced Host Controller
Kernel driver in use: ehci-pci
Kernel modules: ehci_pci
00:1b.0 Audio device: Intel Corporation 7 Series/C216 Chipset Family High Definition Audio Controller (rev 04)
Subsystem: Gigabyte Technology Co., Ltd 7 Series/C216 Chipset Family High Definition Audio Controller
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
00:1c.0 PCI bridge: Intel Corporation 7 Series/C216 Chipset Family PCI Express Root Port 1 (rev c4)
Kernel driver in use: pcieport
Kernel modules: shpchp
00:1c.4 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 5 (rev c4)
Kernel driver in use: pcieport
Kernel modules: shpchp
00:1c.6 PCI bridge: Intel Corporation 82801 PCI Bridge (rev c4)
00:1d.0 USB controller: Intel Corporation 7 Series/C216 Chipset Family USB Enhanced Host Controller #1 (rev 04)
Subsystem: Gigabyte Technology Co., Ltd 7 Series/C216 Chipset Family USB Enhanced Host Controller
Kernel driver in use: ehci-pci
Kernel modules: ehci_pci
00:1f.0 ISA bridge: Intel Corporation Z77 Express Chipset LPC Controller (rev 04)
Subsystem: Gigabyte Technology Co., Ltd Z77 Express Chipset LPC Controller
Kernel driver in use: lpc_ich
Kernel modules: lpc_ich
00:1f.2 IDE interface: Intel Corporation 7 Series/C210 Series Chipset Family 4-port SATA Controller [IDE mode] (rev 04)
Subsystem: Gigabyte Technology Co., Ltd 7 Series/C210 Series Chipset Family 4-port SATA Controller [IDE mode]
Kernel driver in use: ata_piix
Kernel modules: ata_piix, ata_generic
00:1f.3 SMBus: Intel Corporation 7 Series/C216 Chipset Family SMBus Controller (rev 04)
Subsystem: Gigabyte Technology Co., Ltd 7 Series/C216 Chipset Family SMBus Controller
Kernel driver in use: i801_smbus
Kernel modules: i2c_i801
00:1f.5 IDE interface: Intel Corporation 7 Series/C210 Series Chipset Family 2-port SATA Controller [IDE mode] (rev 04)
Subsystem: Gigabyte Technology Co., Ltd 7 Series/C210 Series Chipset Family 2-port SATA Controller [IDE mode]
Kernel driver in use: ata_piix
Kernel modules: ata_piix, ata_generic
01:00.0 VGA compatible controller: NVIDIA Corporation GM107 [GeForce GTX 750 Ti] (rev a2)
Subsystem: ASUSTeK Computer Inc. GM107 [GeForce GTX 750 Ti]
Kernel driver in use: vfio-pci
Kernel modules: nouveau
01:00.1 Audio device: NVIDIA Corporation Device 0fbc (rev a1)
Subsystem: ASUSTeK Computer Inc. Device 84bb
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel
02:00.0 VGA compatible controller: NVIDIA Corporation GM107 [GeForce GTX 750] (rev a2)
Subsystem: eVga.com. Corp. GM107 [GeForce GTX 750]
Kernel driver in use: vfio-pci
Kernel modules: nouveau
02:00.1 Audio device: NVIDIA Corporation Device 0fbc (rev a1)
Subsystem: eVga.com. Corp. Device 2758
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel
03:00.0 Ethernet controller: Qualcomm Atheros AR8161 Gigabit Ethernet (rev 10)
Subsystem: Gigabyte Technology Co., Ltd AR8161 Gigabit Ethernet
Kernel driver in use: alx
Kernel modules: alx
04:00.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 41)
lspci -k
https://gist.github.com/Lego007/3c8d227bb2424099083bb3c5d33f7d03
[1_GTX750TI] Working.
https://gist.github.com/Lego007/4d0b30ef0539c0eb404bb0a5d80e4ee5
[2_GTX750TI] No Gpu in DM. Blackscreen when installing nvidia drivers.
https://gist.github.com/Lego007/dc477a34517fbbf7296272c875e95158
Just a heads up, The team at www.passthroughpo.st and I are developing tools to make the setup process easier:
Can you guys see if you can make an OBS plugin to grab the output from Looking Glass? I need it for a multi-GPU OBS power user tutorial I’m making.
if looking glass renders to a window you should be able to use the ‘window’ source in the meantime. We can ask gnif about something more integrated though.
He did say it was possible. but that “someone had to dedicate their time to make it.”
It’s a bit out of our scope for now, we need patrons if we’re going to put people on this stuff full time. We’ll look into it as a back-burner type thing, though.
Sorry for my english 
May be anybody can help me ? I have a 1 monitor witn 2 in (D-SUB+DVI). Host system:
CPU: Intel® Core™ i7-7700K CPU @ 4.20GHz (Kabylake)
MoBo: ASUS PRIME Z270-P
Video0 - Integrated Intel HD Graphics 630
Video1 - Asus GTX960 (UEFI VGA BIOS)
Video2 - Gigbyte GT640 (Non UEFI VGA BIOS)
I dont use integrated graphics. Main host system connected to physiacal monitor over DVI (Video1) and i try passthrough guest with GT640 (Video2) over D-SUB.
OS: Ubuntu 17.10
#> uname -a
Linux land33 4.15.0-041500rc7-generic #201801072330 SMP Sun Jan 7 23:31:29 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Check IOMMU:
#> dmesg | grep -e DMAR -e IOMMU
[ 0.000000] ACPI: DMAR 0x000000008407CF50 0000A8 (v01 INTEL KBL 00000001 INTL 00000001)
[ 0.000000] DMAR: IOMMU enabled
[ 0.004000] DMAR: Host address width 39
[ 0.004000] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[ 0.004000] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap 1c0000c40660462 ecap 19e2ff0505e
[ 0.004000] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[ 0.004000] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da
[ 0.004000] DMAR: RMRR base: 0x00000091884000 end: 0x000000918a3fff
[ 0.004000] DMAR: RMRR base: 0x00000093800000 end: 0x00000097ffffff
[ 0.004000] DMAR-IR: IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1
[ 0.004000] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[ 0.004000] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[ 0.004000] DMAR-IR: Enabled IRQ remapping in x2apic mode
[ 0.629101] DMAR: No ATSR found
[ 0.629126] DMAR: dmar0: Using Queued invalidation
[ 0.629209] DMAR: dmar1: Using Queued invalidation
[ 0.629415] DMAR: Setting RMRR:
[ 0.629470] DMAR: Setting identity map for device 0000:00:02.0 [0x93800000 - 0x97ffffff]
[ 0.629498] DMAR: Setting identity map for device 0000:00:14.0 [0x91884000 - 0x918a3fff]
[ 0.629502] DMAR: Prepare 0-16MiB unity mapping for LPC
[ 0.629524] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[ 0.629664] DMAR: Intel® Virtualization Technology for Directed I/O
GRUB default cmd line (GT640 video id and audio id + my pci-e wi-fi ethernet card):
GRUB_CMDLINE_LINUX_DEFAULT=“quiet splash iommu=1 intel_iommu=on iommu=pt vfio-pci.ids=10de:0fc1,10de:0e1b,168c:002e video=efifb:off”
file: /etc/modules
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
kvm
kvm_intel
file /etc/modprobe.d/kvm.conf
options kvm ignore_msrs=1
file /etc/modprobe.d/blacklist.conf (not all, default file + add 1 line)
blacklist nouveau
file /etc/modprobe.d/nvidia.conf
softdep nvidia pre: vfio vfio_pci
file /etc/modprobe.d/vfio.conf
options vfio-pci ids=10de:0fc1,10de:0e1b
file /etc/initramfs-tools
softdep nvidia pre: vfio vfio_pci
vfio
vfio_iommu_type1
vfio_virqfd
options vfio_pci ids=10de:0fc1,10de:0e1b
vfio_pci ids=10de:0fc1,10de:0e1b
vfio_pci
nvidia
kvm
kvm_intel
Script (now i run this manually) vfio-pci-override.sh
DEVS="0000:03:00.0 0000:03:00.1"
for DEV in $DEVS; do
echo “vfio-pci” > /sys/bus/pci/devices/$DEV/driver_override
done
modprobe -i vfio-pci
I use GUI and vm-manager for create VMs and try GT640 passthrough to Win10 VM. I create VM as SeaBIOS (BIOS), chipset emulated Q35 and attch host PCI-device over gui. Install guest OS. Remove all Video SPICE. Run guest but i dont see nothing on my phisical monitor over D-SUB (black screen) but i see and hear what VM working.
Next i back add Vido SPICE. Run VM an few past secon see on init screen:
Guest has not initialized the display (yet)
When guest loaded i add old Nvidia driver and see this:
Driver for GT640 in Proerties show “Error code 43”
I googling and special setup oldiest drivers, add few options to VM XML config:
<vendor_id state=‘on’ value=‘whatever’/> and hidden state=‘on’ but have same result. Then guest running i check:
#> dmesg | grep -i vgaarb
[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.15.0-041500rc7-generic root=UUID=cc5bcfe3-7f3a-45fa-90f2-042fca56e34d ro quiet splash iommu=1 intel_iommu=on vfio-pci.ids=10de:0fc1,10de:0e1b,168c:002e video=efifb:off i915.enable_hd_vgaarb=1 vt.handoff=7
[ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.15.0-041500rc7-generic root=UUID=cc5bcfe3-7f3a-45fa-90f2-042fca56e34d ro quiet splash iommu=1 intel_iommu=on vfio-pci.ids=10de:0fc1,10de:0e1b,168c:002e video=efifb:off i915.enable_hd_vgaarb=1 vt.handoff=7
[ 0.085988] pci 0000:01:00.0: vgaarb: setting as boot VGA device
[ 0.085988] pci 0000:01:00.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none
[ 0.085988] pci 0000:03:00.0: vgaarb: VGA device added: decodes=io+mem,owns=none,locks=none
[ 0.085988] pci 0000:03:00.0: vgaarb: bridge control possible
[ 0.085988] pci 0000:01:00.0: vgaarb: bridge control possible
[ 0.085988] vgaarb: loaded
[ 1.766735] vfio-pci 0000:03:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
[ 7.051265] nvidia 0000:01:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=io+mem
Check:
#> dmesg | grep -i vfio
[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.15.0-041500rc7-generic root=UUID=cc5bcfe3-7f3a-45fa-90f2-042fca56e34d ro quiet splash iommu=1 intel_iommu=on vfio-pci.ids=10de:0fc1,10de:0e1b,168c:002e video=efifb:off i915.enable_hd_vgaarb=1 vt.handoff=7
[ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.15.0-041500rc7-generic root=UUID=cc5bcfe3-7f3a-45fa-90f2-042fca56e34d ro quiet splash iommu=1 intel_iommu=on vfio-pci.ids=10de:0fc1,10de:0e1b,168c:002e video=efifb:off i915.enable_hd_vgaarb=1 vt.handoff=7
[ 1.762384] VFIO - User Level meta-driver version: 0.3
[ 1.766735] vfio-pci 0000:03:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
[ 1.784148] vfio_pci: add [10de:0fc1[ffff:ffff]] class 0x000000/00000000
[ 1.804155] vfio_pci: add [10de:0e1b[ffff:ffff]] class 0x000000/00000000
[ 1303.689601] vfio-pci 0000:03:00.0: enabling device (0000 -> 0003)
[ 1303.689815] vfio_ecap_init: 0000:03:00.0 hiding ecap 0x19@0x900
All look as fine but dont work corrctly and i don see nothing on monitor over D-SUB. I try with SPECI/VNC+passthrough GT640 but see only Error code 43 on guest OS. Without SPECI/VNC (only passthrough card GT640) i see black screen over D-SUB but on main scrin host system in GUI in vm-manager i see that VM wrking. After host system boot i run nvidia-smi and it show then i have only one card on host - GTX960 and check:
# lspci -nnk
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM206 [GeForce GTX 960] [10de:1401] (rev a1)
Subsystem: ASUSTeK Computer Inc. GM206 [GeForce GTX 960] [1043:8520]
Kernel driver in use: nvidia
Kernel modules: nvidiafb, nouveau, nvidia_384_drm, nvidia_384
01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fba] (rev a1)
Subsystem: ASUSTeK Computer Inc. Device [1043:8520]
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
03:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK107 [GeForce GT 640] [10de:0fc1] (rev a1)
Subsystem: Gigabyte Technology Co., Ltd GK107 [GeForce GT 640] [1458:353e]
Kernel driver in use: vfio-pci
Kernel modules: nvidiafb, nouveau, nvidia_384_drm, nvidia_384
03:00.1 Audio device [0403]: NVIDIA Corporation GK107 HDMI Audio Controller [10de:0e1b] (rev a1)
Subsystem: Gigabyte Technology Co., Ltd GK107 HDMI Audio Controller [1458:353e]
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel
OUT find /sys/kernel/iommu_groups/ -type l
/sys/kernel/iommu_groups/7/devices/0000:00:1b.4
/sys/kernel/iommu_groups/15/devices/0000:06:00.0
/sys/kernel/iommu_groups/5/devices/0000:00:17.0
/sys/kernel/iommu_groups/13/devices/0000:03:00.1
/sys/kernel/iommu_groups/13/devices/0000:03:00.0
/sys/kernel/iommu_groups/3/devices/0000:00:14.0
/sys/kernel/iommu_groups/11/devices/0000:00:1d.0
/sys/kernel/iommu_groups/1/devices/0000:01:00.1
/sys/kernel/iommu_groups/1/devices/0000:00:01.0
/sys/kernel/iommu_groups/1/devices/0000:01:00.0
/sys/kernel/iommu_groups/8/devices/0000:00:1c.0
/sys/kernel/iommu_groups/6/devices/0000:00:1b.0
/sys/kernel/iommu_groups/14/devices/0000:05:00.0
/sys/kernel/iommu_groups/4/devices/0000:00:16.0
/sys/kernel/iommu_groups/12/devices/0000:00:1f.3
/sys/kernel/iommu_groups/12/devices/0000:00:1f.4
/sys/kernel/iommu_groups/12/devices/0000:00:1f.2
/sys/kernel/iommu_groups/12/devices/0000:00:1f.0
/sys/kernel/iommu_groups/2/devices/0000:00:02.0
/sys/kernel/iommu_groups/10/devices/0000:00:1c.7
/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/9/devices/0000:00:1c.3
WHAT I DO WRONG? May be i need reflash my GT640 with UEFI bios and try run VM witn OVMF what?
I read that:
Passthrough seems to work but no output is displayed
Make sure if you are using virt-manager that UEFI firmware is selected for your virtual machine. Also, make sure you have passed the correct device to the VM.
I try use UEFI firmware (not BIOS) but not have result ;(
I need GT960 use on host! Any body can help me? Sorry for long text. Thanks.
I found solution.
and to the end XML config add (after end devices section and final line domain section)
<qemu:commandline>
<qemu:arg value=’-set’/>
<qemu:arg value=‘device.hostdev0.x-vga=on’/>
<qemu:env name=‘QEMU_AUDIO_DRV’ value=‘pa’/>
<qemu:env name=‘QEMU_PA_SERVER’ value=’/run/user/1000/pulse/native’/>
<qemu:env name=‘QEMU_PA_SAMPLES’ value=‘256’/>
</qemu:commandline>
find #user = "root" line and add afer
user = "<you_user_name>"
group = “<user_group_name>”
In vm-manager need set MICROCODE as UEFI (with OVMF) and Q35 chipset emulation and set RIGHT numer for hostdev in XML config
run VM from vm-manager, swithc monitor to D-SUB in and see profit !
When my monitor don’t have HDMI in i get sound from guest over pulseaudio (see configs).
When win installation process i use drivers from https://fedoraproject.org/wiki/Windows_Virtio_Drivers
May be its to be useful for somebody.
Good luck for all.
Is there a solution for this yet?
Maybe the methods you mentioned have gotten a little more reliable now?
It’s all dependent on the Device ID and which one vfio-pci can hook onto. If vfio-pci could hook by the subsystem ID, you could theoretically hook identical cards, but different vendors.
For those interested in how the recent patches have impacted virtualization, here is some benchmarks VM vs. Bare-metal. BIOS running latest verion (P1.70 w/Intel microcode 80), kernel 4.14.13-300.fc27.x86_64.
MB: X299 (ASRock Professional Gaming i9)
CPU: i9 7960X
RAM: 64GiB 3600Mhz
Host disk: Samsung 960 Pro
Guest disk: Samsung 960 evo (virtio scsi)
Other: 8C/16T cpupinning (isolcpu grub etc.), 1GiB hugepages x 32
For anyone using Fedora, OpenSUSE, and any other distro that may have an issue with their current version of OVMF in regards to Windows not booting under any EFI setting, download the RPM for edk OVMF from here, and then add
/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd:/usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd
to /etc/libvirt/qemu.conf
Thanks, this is a nice PSA. I’ve heard some users were having trouble with that.
Thanks, would appreciate it if there’s a guide for it since I have no idea how to do that.
Still sucks that I can’t build a machine with 4 identical graphics cards and pass them though to different VMs. Someone should seriously work on fixing this.
The band-aid solution is to use a Kernel patch called ACS, but it’s very not ideal.
