@chaos4u - sounds like you got a lot closer than I did …
How did you map your pci card into kvm devices on the MSI X399?
Also - have you looked into disabling the HET timer in the kvm which I have read seems to be a problem with windows / threadripper in other threads?
Thanks
TLDR; I think I can work around the problem I am having with a configuration change…
With my X399 MSI / 1950X / Fedora 26 setup … gtx 1080ti in pcie1 and gtx1050 in pci3 …
bios MS-7B09 1.40 08/21/2017
My secondary card, gtx1050, is in its own mmio group / bound to virtio … but when using virt-manager gui to add the device results in the following error during kvm start:
…
could not access /sys/bus/pci/devices/0000:11:00.0/config
So looks like libvirt is looking for a simple path based on the bus / slot / function of matching device I selected in the gui but is in the wrong place under my devices tree in fedora 26…
from my virsh edit …
...
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x11' slot='0x00' function='0x0'/>
</source>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
</hostdev>
Instead my device is showing up under … /sys/bus/pci/devices/0000:00:03.1/0000:0b:00.0/ … a lot deeper down … so my idea is a) find a way to make a symlink to point to the device (or remove and re-add the device somehow to make it appear where libvirt is looking for it)
or b) find a different way to describe the device …
The following idea of using a qemu arg almost works for me in virsh edit …
<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
<name>win10c2</name>
...
<qemu:commandline>
<qemu:arg value='-cpu'/>
<qemu:arg value='host,hv_time,kvm=off,hv_vendor_id=null'/>
<qemu:arg value='-device'/>
<qemu:arg value='vfio-pci,id=vgpu1,sysfsdev=/sys/bus/pci/devices/0000:00:03.1/0000:0b:00.0'/>
</qemu:commandline>
</domain>
But then I get stuck with qemu not being configured right to access /dev/vfio … no access … looks like I need to add /dev/vfio to the list of allowed devices for qemu on f26 to make this work… Stuck here for now unless there is another way to do it.
FYI…
ls-iommu …
ls-iommu | grep Group\ 2
…
IOMMU Group 2 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 2 00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
IOMMU Group 2 0b:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP107 [GeForce GTX 1050] [10de:1c81] (rev a1)
IOMMU Group 2 0b:00.1 Audio device [0403]: NVIDIA Corporation GP107GL High Definition Audio Controller [10de:0fb9] (rev a1)
lspci -tv
…
-+-[0000:40]-+-00.0 Advanced Micro Devices, Inc. [AMD] Device 1450
| +-00.2 Advanced Micro Devices, Inc. [AMD] Device 1451
| +-01.0 Advanced Micro Devices, Inc. [AMD] Device 1452
| +-01.3-[41]----00.0 Intel Corporation PCIe Data Center SSD
| +-02.0 Advanced Micro Devices, Inc. [AMD] Device 1452
| +-03.0 Advanced Micro Devices, Inc. [AMD] Device 1452
| +-03.1-[42]--+-00.0 NVIDIA Corporation GP102 [GeForce GTX 1080 Ti]
| | \-00.1 NVIDIA Corporation GP102 HDMI Audio Controller
| +-04.0 Advanced Micro Devices, Inc. [AMD] Device 1452
| +-07.0 Advanced Micro Devices, Inc. [AMD] Device 1452
| +-07.1-[43]--+-00.0 Advanced Micro Devices, Inc. [AMD] Device 145a
| | +-00.2 Advanced Micro Devices, Inc. [AMD] Device 1456
| | \-00.3 Advanced Micro Devices, Inc. [AMD] USB3 Host Controller
| +-08.0 Advanced Micro Devices, Inc. [AMD] Device 1452
| \-08.1-[44]--+-00.0 Advanced Micro Devices, Inc. [AMD] Device 1455
| \-00.2 Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode]
\-[0000:00]-+-00.0 Advanced Micro Devices, Inc. [AMD] Device 1450
+-00.2 Advanced Micro Devices, Inc. [AMD] Device 1451
+-01.0 Advanced Micro Devices, Inc. [AMD] Device 1452
+-01.1-[01-09]--+-00.0 Advanced Micro Devices, Inc. [AMD] Device 43ba
| +-00.1 Advanced Micro Devices, Inc. [AMD] Device 43b6
| \-00.2-[02-09]--+-00.0-[03]----00.0 ASMedia Technology Inc. Device 2142
| +-02.0-[04]--
| +-03.0-[05]--
| +-04.0-[06]--
| +-05.0-[07]--
| +-06.0-[08]----00.0 Intel Corporation I211 Gigabit Network Connection
| \-07.0-[09]--
+-01.3-[0a]----00.0 Mellanox Technologies MT27520 Family [ConnectX-3 Pro]
+-02.0 Advanced Micro Devices, Inc. [AMD] Device 1452
+-03.0 Advanced Micro Devices, Inc. [AMD] Device 1452
+-03.1-[0b]--+-00.0 NVIDIA Corporation GP107 [GeForce GTX 1050]
| \-00.1 NVIDIA Corporation GP107GL High Definition Audio Controller
…
virsh nodedev-list --tree
…
± pci_0000_00_03_1
| |
| ± pci_0000_0b_00_0
| ± pci_0000_0b_00_1
|
virsh nodedev-dumpxml pci_0000_0b_00_0
<device>
<name>pci_0000_0b_00_0</name>
<path>/sys/devices/pci0000:00/0000:00:03.1/0000:0b:00.0</path>
<parent>pci_0000_00_03_1</parent>
<driver>
<name>vfio-pci</name>
</driver>
<capability type='pci'>
<domain>0</domain>
<bus>11</bus>
<slot>0</slot>
<function>0</function>
<product id='0x1c81'>GP107 [GeForce GTX 1050]</product>
<vendor id='0x10de'>NVIDIA Corporation</vendor>
<iommuGroup number='2'>
<address domain='0x0000' bus='0x0b' slot='0x00' function='0x0'/>
<address domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
<address domain='0x0000' bus='0x0b' slot='0x00' function='0x1'/>
<address domain='0x0000' bus='0x00' slot='0x03' function='0x1'/>
</iommuGroup>
<numa node='0'/>
<pci-express>
<link validity='cap' port='0' speed='8' width='16'/>
<link validity='sta' speed='8' width='16'/>
</pci-express>
</capability>
</device>
ll /sys/bus/pci/devices/0000:00:03.1/0000:0b:00.0/iommu_group/devices/
total 0
lrwxrwxrwx. 1 root root 0 Sep 10 00:16 0000:00:03.0 -> ../../../../devices/pci0000:00/0000:00:03.0
lrwxrwxrwx. 1 root root 0 Sep 10 00:16 0000:00:03.1 -> ../../../../devices/pci0000:00/0000:00:03.1
lrwxrwxrwx. 1 root root 0 Sep 10 00:16 0000:0b:00.0 -> ../../../../devices/pci0000:00/0000:00:03.1/0000:0b:00.0
lrwxrwxrwx. 1 root root 0 Sep 10 00:16 0000:0b:00.1 -> ../../../../devices/pci0000:00/0000:00:03.1/0000:0b:00.1
Looks like Wendel got this working on the Gigabyte board … Level1 Linux: Livestream (Setting up PCIe Passthrough on Fedora on X299 and Threadripper systems) | Level One Techs
And GreyBoltWolf makes it look easy … Play games in Windows on Linux! PCI passthrough quick guide
See also https://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM
and https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/chap-Guest_virtual_machine_device_configuration.html
So I have hope it can be done if I can just finish jumping through the hoops…
Thanks