Some updates.
2 discrete GPUs in the system
I added a second discrete GPU only for testing. I am generally happy with using IGD for host (in fact, this was the reason I chose Coffee Lake over other platforms), but I wanted to investigate the prospect of using two discrete GPUs with this hardware.
Since some time I got a working USB 3.0 card, an ICY box IB-AC614a (VIA VL805). This is connected to one of the upstream ports of my KVM switch (the other upstream port is connected to the motherboard USB ports). I also tried putting a Radeon 7790 in the remaining empty PCIe x4 slot, which sits on the chipset and thus has its own IOMMU group. See:
IOMMU Group 0 00:00.0 Host bridge [0600]: Intel Corporation Device [8086:3ec2] (rev 07)
IOMMU Group 10 00:1c.0 PCI bridge [0604]: Intel Corporation 200 Series PCH PCI Express Root Port #1 [8086:a290] (rev f0)
IOMMU Group 11 00:1c.4 PCI bridge [0604]: Intel Corporation 200 Series PCH PCI Express Root Port #5 [8086:a294] (rev f0)
IOMMU Group 12 00:1f.0 ISA bridge [0601]: Intel Corporation Device [8086:a2c9]
IOMMU Group 12 00:1f.2 Memory controller [0580]: Intel Corporation 200 Series PCH PMC [8086:a2a1]
IOMMU Group 12 00:1f.3 Audio device [0403]: Intel Corporation 200 Series PCH HD Audio [8086:a2f0]
IOMMU Group 12 00:1f.4 SMBus [0c05]: Intel Corporation 200 Series PCH SMBus Controller [8086:a2a3]
IOMMU Group 13 00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (2) I219-V [8086:15b8]
IOMMU Group 14 04:00.0 USB controller [0c03]: VIA Technologies, Inc. VL805 USB 3.0 Host Controller [1106:3483] (rev 01)
IOMMU Group 15 05:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961 [144d:a804]
IOMMU Group 16 07:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Bonaire XT [Radeon HD 7790/8770 / R7 360 / R9 260/360 OEM] [1002:665c]
IOMMU Group 16 07:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:0002]
IOMMU Group 1 00:01.0 PCI bridge [0604]: Intel Corporation Skylake PCIe Controller (x16) [8086:1901] (rev 07)
IOMMU Group 1 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)
IOMMU Group 1 01:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)
IOMMU Group 2 00:02.0 VGA compatible controller [0300]: Intel Corporation Device [8086:3e92]
IOMMU Group 3 00:14.0 USB controller [0c03]: Intel Corporation 200 Series PCH USB 3.0 xHCI Controller [8086:a2af]
IOMMU Group 3 00:14.2 Signal processing controller [1180]: Intel Corporation 200 Series PCH Thermal Subsystem [8086:a2b1]
IOMMU Group 4 00:16.0 Communication controller [0780]: Intel Corporation 200 Series PCH CSME HECI #1 [8086:a2ba]
IOMMU Group 5 00:17.0 SATA controller [0106]: Intel Corporation 200 Series PCH SATA controller [AHCI mode] [8086:a282]
IOMMU Group 6 00:1b.0 PCI bridge [0604]: Intel Corporation 200 Series PCH PCI Express Root Port #17 [8086:a2e7] (rev f0)
IOMMU Group 7 00:1b.2 PCI bridge [0604]: Intel Corporation 200 Series PCH PCI Express Root Port #19 [8086:a2e9] (rev f0)
IOMMU Group 8 00:1b.3 PCI bridge [0604]: Intel Corporation 200 Series PCH PCI Express Root Port #20 [8086:a2ea] (rev f0)
IOMMU Group 9 00:1b.4 PCI bridge [0604]: Intel Corporation 200 Series PCH PCI Express Root Port #21 [8086:a2eb] (rev f0)We find the GTX 970 in Group 1 and the Radeon 7790 in Group 16. The USB controller is in group 14.
Passthrough of 2 GPUs (while using IGD for host graphics)
I tried assigning both the GTX 970 and the Radeon 7790 to different VMs and run them in parallel.
Win8.1: GTX 970 + USB controller
Win7: Radeon 7790
Radeon 7790 is passed through to the Windows 7 installation as secondary GPU - this means that Seabios is used and the system always boots from the qemu emulated video. It is easy to make Windows use the secondary GPU as if it was primary, however, by turning off other displays in Window’s native display settings. Should the discrete GPU suddenly not work for some reason, the emulated video should be automatically enabled (I did not test this recently, but according to past experience at least). In my case the emulated GPU turned off automatically as soon as the 7790 came online.
Installing radeon drivers in the VM: Autodetect of the 7790 worked! There has been problems in the past with AMD’s autodetect procedure, but now there were no problems detecting the card. This may be related to the absence of reset issues (see below).
GPU-Z reports the card running at PCIe 3.0 x4 when under load, as the specification suggests. So the PCIe speeds should not be a bottleneck for this card, perhaps still for some of the more extreme GPUs of today… So here are some benchmarks:
7790 benchmark from Unigine heaven 4.0 with Ultra settings, 8x AA & Extreme Tesselation, 1080p (PCIe 3.0 x4):
391pts, 15.5 [7.8-36.9] FPS
This can be compared to my performance using the same card and settings on a K10 platform in the past (PCIe 2.0 x16, K10 Magny-Cours with 4 x 2.6GHz assigned to VM):
389 points, 15.5 [6.8-36.4] FPS
So no discernible difference compared to an older and slower machine.
I also run the same benchmark on the 7790 and the GTX 970 in parallel in different VMs, with no notable performance impact on either. This is expected as they do not compete about any substantial resources.
What if we put some competition over the chipset bandwidth, while running Heaven 4?
dd if=/dev/nvme0n1 of=/dev/null bs=1M ; dd if=/dev/nvme0n1 of=/dev/null bs=1M ; dd if=/dev/nvme0n1 of=/dev/null bs=1M & [<-- samsung SM961 @nvme x4, 3 iterations to make sure it lasts through the GPU benchmark]
dd if=/dev/sda of=/dev/null bs=1M [<-- OCZ vertex4 @sata3, run in parallel with the other dd process](make sure you understand what you are doing, especially the if and of options to dd, in case you plan to try the above.)
So here I read data as fast as possible from both the nvme and the sata ssd:s, in parallel. dd reported reading 2.8Gb/s for nvme0 and 510GB/s from sda. Heaven 4.0 results for 7790:
306pts, 12.1 [2.9-36.6] FPS (substantially lower than before)
So reading about 3.2-3.3Gb/s from chipset devices at the same time does have a measurable impact, but running another GPU on the CPU- connected PCIe lines does not. This means that given the speeds of today’s nvme ssd:s it is not advicable to run a GPU in a chipset-controlled slot for anything serious together with nvme-based storage.
I expected to encounter the device reset bug with this card, as this as been the case when using it in the past, but not this time. Rebooting the VM without rebooting the host worked fine. I believe that some workarounds has been added to the vfio_pci module since last time I tried, some quick google results suggested that.
Note that I have not tested the 7790 with OVMF, only as secondary passthrough with Seabios (the same way I always used Radeon cards with passthrough in the past). Secondary passthrough is an alternative if you have an AMD card and do not want to fiddle with OVMF. However, there are reports that this does not work on Threadripper, and I don’t know whether it works with all Radeon cards.
Second GPU as host graphics
I finally tried to run the 7790 in the host under linux, while passing through the GTX970 to windows. I did not manage to initialise the 7790 in the chipset slot as primary GPU by UEFI, something I don’t think is supported by current UEFI on the ASrock z370m pro4. Then I tried to let radeon/AMDGPU bind to it after booting to IGD, for some reason this did not work. However, a bare-metal boot of Windows found both discrete GPUs and could use both, so I don’t think that my problems in Linux had to do with hardware limitations. It is likely that it will work after some troubleshooting in Linux, but don’t take my word as a promise.