The Vega 56 is isolated to a Windows guest and the RX560D to an OSX guest. Manjaro acting a host with onboard gfx. Running the host with two guests simultaneously and everything works fine for the most part.
However, OSX ONLY boots fully if the RX560D is in PCIE 2 (main 16x -> CPU). If the RX560D is in PCIE 4 (4x -> PCH), it will not boot (screen initializes and usual boot process but black screen when the desktop would usually appear). Windows doesn’t seem to care boots fine with any card in either slot. The 370F has two 16x size slots in 2 and 4, but slot 4 is 4x electric.
So instead I have the Vega in the 4x slot for Windows and the RX560 in the 16x for OSX. This isn’t optimal for my setup, as OSX is mainly a work platform where GPU performance is not such an issue, whist Windows is used primarily for gaming.
The GPU slot is making a difference in OSX somehow, presumably because it’s going via the PCH rather than CPU. But I’m a bit stuck as to how to diagnose further…?
That’s interesting you’re able to pass anything through from a PCH slot. What do your IOMMU groups look like? I would suspect the PCH slot to be grouped in with all the other “stuff” connected to the PCH.
How did you determine the x4 lanes are bottle-necking your Windows VM?
Yea I should probably try quantify the actual performance difference. Had a few hiccups but not certain they’ve been related, as I’ve been messing with a few other things also recently. But i’m curious in any case why this might not work
The Vega gfx is in group 17 here:
IOMMU Group 0 00:00.0 Host bridge [0600]: Intel Corporation 8th Gen Core 8-core Desktop Processor Host Bridge/DRAM Registers [Coffee Lake S] [8086:3e30] (rev 0d)
IOMMU Group 1 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 0d)
IOMMU Group 1 01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Baffin [Radeon RX 460/560D / Pro 450/455/460/555/555X/560/560X] [1002:67ef] (rev e5)
IOMMU Group 1 01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Baffin HDMI/DP Audio [Radeon RX 550 640SP / RX 560/560X] [1002:aae0]
IOMMU Group 2 00:02.0 VGA compatible controller [0300]: Intel Corporation UHD Graphics 630 (Desktop 9 Series) [8086:3e98] (rev 02)
IOMMU Group 3 00:14.0 USB controller [0c03]: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller [8086:a36d] (rev 10)
IOMMU Group 3 00:14.2 RAM memory [0500]: Intel Corporation Cannon Lake PCH Shared SRAM [8086:a36f] (rev 10)
IOMMU Group 4 00:16.0 Communication controller [0780]: Intel Corporation Cannon Lake PCH HECI Controller [8086:a360] (rev 10)
IOMMU Group 5 00:17.0 SATA controller [0106]: Intel Corporation Cannon Lake PCH SATA AHCI Controller [8086:a352] (rev 10)
IOMMU Group 6 00:1b.0 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #21 [8086:a32c] (rev f0)
IOMMU Group 7 00:1c.0 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #3 [8086:a33a] (rev f0)
IOMMU Group 8 00:1c.3 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #4 [8086:a33b] (rev f0)
IOMMU Group 9 00:1c.4 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #5 [8086:a33c] (rev f0)
IOMMU Group 10 00:1d.0 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #9 [8086:a330] (rev f0)
IOMMU Group 11 00:1f.0 ISA bridge [0601]: Intel Corporation H370 Chipset LPC/eSPI Controller [8086:a304] (rev 10)
IOMMU Group 11 00:1f.3 Audio device [0403]: Intel Corporation Cannon Lake PCH cAVS [8086:a348] (rev 10)
IOMMU Group 11 00:1f.4 SMBus [0c05]: Intel Corporation Cannon Lake PCH SMBus Controller [8086:a323] (rev 10)
IOMMU Group 11 00:1f.5 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH SPI Controller [8086:a324] (rev 10)
IOMMU Group 11 00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (7) I219-V [8086:15bc] (rev 10)
IOMMU Group 12 02:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 [144d:a808]
IOMMU Group 13 03:00.0 USB controller [0c03]: Fresco Logic FL1100 USB 3.0 Host Controller [1b73:1100] (rev 10)
IOMMU Group 14 04:00.0 Network controller [0280]: Broadcom Inc. and subsidiaries BCM4360 802.11ac Wireless Network Adapter [14e4:43a0] (rev 03)
IOMMU Group 15 05:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1470] (rev c3)
IOMMU Group 16 06:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1471]
IOMMU Group 17 07:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Vega 10 XL/XT [Radeon RX Vega 56/64] [1002:687f] (rev c3)
IOMMU Group 18 07:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Vega 10 HDMI Audio [Radeon Vega 56/64] [1002:aaf8]
IOMMU Group 19 08:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 [144d:a808]
Are you using the ACS kernel patch? Seeing the audio and video portions of the card in separate groups just makes me think that.
I don’t have direct experience with your chipset, the last Intel i’ve laid hands on for VFIO was x79… that’s the one that had the shared port in a group with a bunch of other stuff, and even using ACS to break up the group, I could not get passthrough working for anything in that shared slot. Similar experience on my AMD x399, the shared slot is very “busy”, so I parked a SAS controller there that never needs pass-through.