Hello,
I may be running in circles regarding this issue but I figured I’d ask some of you fine folks for any input, this has been driving me nuts.
Setup:
- 3900x OCed to 4.1GHz 1.2V (VID), 1.11V (SOC)
- AsRock Taichi X570
- 4x 16GB GSkill Trident Z CL17 @3600MHz via XMP2
- Quadro P400 used for the Host OS ( Arch Linux ), plugged on PCIe port 3 ( bottom )
- 2080Ti Asus Strix plugged in PCIe port 1 ( top, closest to the CPU )
- Host SSD: ADATA_SX6000PNP on m.2 closest to the CPU
- VM SSDs: ADATA_SP550 ( 2nd m.2 ) & Samsung_SSD_860_QVO_1TB ( SATA )
- virtualization, IOMMU, AER Cap ( Enabled ), ACS ( Auto ) set in BIOS
- using LVM + Luks for my Arch Linux install
The VM boots, works fine, GPU-Z as well as Nvidia Control Panel show the link speed at Gen3 x16 but I’m noticing considerably lower performance in the VM compared to on-Metal.
I’ve seen two potential solutions for this:
- ACS Patch ( But I don’t have anything else grouped up with the 2080Ti so I don’t believe it’s required )
- Set:
<qemu:commandline>
<qemu:arg value="-global"/>
<qemu:arg value="pcie-root-port.speed=8"/>
<qemu:arg value="-global"/>
<qemu:arg value="pcie-root-port.width=16"/>
</qemu:commandline>
With the qemu:arg try, it doesn’t work, I just get this error:
Error starting domain: internal error: qemu unexpectedly closed the monitor: 2020-05-21T15:09:54.815431Z qemu-system-x86_64: warning: This family of AMD CPU doesn't support hyperthreading(2)
Please configure -smp options properly or try enabling topoext feature.
2020-05-21T15:09:54.848928Z qemu-system-x86_64: -device pcie-root-port,port=0x8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1: can't apply global pcie-root-port.speed=8: Property '.speed' not found
Not even sure what causes this issue, thoughts?
I’ve had the q35 line set to 3.1, changed it to ‘pc-q35-5.0’ after checking /usr/bin/qemu-system-x86_64 -machine help
but it doesn’t seem to help with anything.
ls-iommu.sh output:
OMMU Group 0:
00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 1:
00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 10:
00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 11:
00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 12:
00:08.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 13:
00:08.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 14:
00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61)
00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 15:
00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 0 [1022:1440]
00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 1 [1022:1441]
00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 2 [1022:1442]
00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 3 [1022:1443]
00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 4 [1022:1444]
00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 5 [1022:1445]
00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 6 [1022:1446]
00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 7 [1022:1447]
IOMMU Group 16:
01:00.0 Non-Volatile memory controller [0108]: Realtek Semiconductor Co., Ltd. Device [10ec:5763] (rev 01)
IOMMU Group 17:
02:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse Switch Upstream [1022:57ad]
IOMMU Group 18:
03:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
IOMMU Group 19:
03:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
IOMMU Group 2:
00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 20:
03:03.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
IOMMU Group 21:
03:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]
0c:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
0c:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
0c:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU Group 22:
03:09.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]
0d:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 23:
03:0a.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]
0e:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 24:
04:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP107GL [Quadro P400] [10de:1cb3] (rev a1)
04:00.1 Audio device [0403]: NVIDIA Corporation GP107GL High Definition Audio Controller [10de:0fb9] (rev a1)
IOMMU Group 25:
05:00.0 Non-Volatile memory controller [0108]: Intel Corporation SSD 600P Series [8086:f1a5] (rev 03)
IOMMU Group 26:
06:00.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1184e PCIe Switch Port [1b21:1184]
IOMMU Group 27:
07:01.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1184e PCIe Switch Port [1b21:1184]
08:00.0 Network controller [0280]: Intel Corporation Wi-Fi 6 AX200 [8086:2723] (rev 1a)
IOMMU Group 28:
07:03.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1184e PCIe Switch Port [1b21:1184]
IOMMU Group 29:
07:05.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1184e PCIe Switch Port [1b21:1184]
0a:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
IOMMU Group 3:
00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 30:
07:07.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1184e PCIe Switch Port [1b21:1184]
IOMMU Group 31:
0f:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU102 [GeForce RTX 2080 Ti Rev. A] [10de:1e07] (rev a1)
0f:00.1 Audio device [0403]: NVIDIA Corporation TU102 High Definition Audio Controller [10de:10f7] (rev a1)
0f:00.2 USB controller [0c03]: NVIDIA Corporation TU102 USB 3.1 Host Controller [10de:1ad6] (rev a1)
0f:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU102 USB Type-C UCSI Controller [10de:1ad7] (rev a1)
IOMMU Group 32:
10:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
IOMMU Group 33:
11:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
IOMMU Group 34:
11:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP [1022:1486]
IOMMU Group 35:
11:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU Group 36:
11:00.4 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller [1022:1487]
IOMMU Group 37:
12:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 38:
13:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 4:
00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 5:
00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 6:
00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 7:
00:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 8:
00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 9:
00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
Looking at this I don’t think I need to apply the ACS patch
$cat /etc/sysctl.d/40-hugepage.conf
vm.nr_hugepages=8192
vm.hugetlb_shm_group=48
$cat /etc/modprobe.d/vfio.conf
options vfio-pci ids=10de:1e07,10de:10f7,10de:1ad6,10de:1ad7
$cat /etc/modprobe.d/blacklist-nouveau.conf
blacklist nouveau
options nouveau modeset=0
$cat /etc/modprobe.d/kvm.conf
options kvm ignore_msrs=1
Checking $nvidia-smi only shows the Quadro P400.
Sorted CSM ( believe it’s set to Disabled ) so now the Quadro ( on PCI-e slot 3 ) displays POST/BIOS, the BOOT menu and LUKS decrypt prompt and so on.
Boot loader:
title Arch
linux /vmlinuz-linux
initrd /initramfs-linux.img
options cryptdevice=UUID=<DiskUUID>:volume root=/dev/mapper/volume-root quiet amd_iommu=on iommu=pt pci=noaer pci=nommconf hugepages=8192 rw vfio-pci.ids=10de:1e07,10de:10f7,10de:1ad6,10de:1ad7
$lscpu -e
CPU NODE SOCKET CORE L1d:L1i:L2:L3 ON MAXMHz MINMHz
0 0 0 0 0:0:0:0 yes 4100.0000 2200.0000
1 0 0 1 1:1:1:0 yes 4100.0000 2200.0000
2 0 0 2 2:2:2:0 yes 4100.0000 2200.0000
3 0 0 3 3:3:3:1 yes 4100.0000 2200.0000
4 0 0 4 4:4:4:1 yes 4100.0000 2200.0000
5 0 0 5 5:5:5:1 yes 4100.0000 2200.0000
6 0 0 6 6:6:6:2 yes 4100.0000 2200.0000
7 0 0 7 7:7:7:2 yes 4100.0000 2200.0000
8 0 0 8 8:8:8:2 yes 4100.0000 2200.0000
9 0 0 9 9:9:9:3 yes 4100.0000 2200.0000
10 0 0 10 10:10:10:3 yes 4100.0000 2200.0000
11 0 0 11 11:11:11:3 yes 4100.0000 2200.0000
12 0 0 0 0:0:0:0 yes 4100.0000 2200.0000
13 0 0 1 1:1:1:0 yes 4100.0000 2200.0000
14 0 0 2 2:2:2:0 yes 4100.0000 2200.0000
15 0 0 3 3:3:3:1 yes 4100.0000 2200.0000
16 0 0 4 4:4:4:1 yes 4100.0000 2200.0000
17 0 0 5 5:5:5:1 yes 4100.0000 2200.0000
18 0 0 6 6:6:6:2 yes 4100.0000 2200.0000
19 0 0 7 7:7:7:2 yes 4100.0000 2200.0000
20 0 0 8 8:8:8:2 yes 4100.0000 2200.0000
21 0 0 9 9:9:9:3 yes 4100.0000 2200.0000
22 0 0 10 10:10:10:3 yes 4100.0000 2200.0000
23 0 0 11 11:11:11:3 yes 4100.0000 2200.0000
$sudo virsh vcpupin win10
VCPU CPU Affinity
----------------------
0 8
1 9
2 10
3 11
4 12
5 13
6 14
7 15
8 16
9 17
10 18
11 19
12 20
13 21
14 22
15 23
$lstopo
$lspci -nnv
Shows all four devices on the 2080Ti with "Kernel driver in use: vfio-pci"
Last but not least, here's the XML file:
<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
<name>win10</name>
<uuid>fc87d214-1499-4d4e-8a6d-bf5ac46f1c47</uuid>
<memory unit='KiB'>16777216</memory>
<currentMemory unit='KiB'>16777216</currentMemory>
<vcpu placement='static' cpuset='0-23'>16</vcpu>
<iothreads>2</iothreads>
<cputune>
<vcpupin vcpu='0' cpuset='8'/>
<vcpupin vcpu='1' cpuset='9'/>
<vcpupin vcpu='2' cpuset='10'/>
<vcpupin vcpu='3' cpuset='11'/>
<vcpupin vcpu='4' cpuset='12'/>
<vcpupin vcpu='5' cpuset='13'/>
<vcpupin vcpu='6' cpuset='14'/>
<vcpupin vcpu='7' cpuset='15'/>
<vcpupin vcpu='8' cpuset='16'/>
<vcpupin vcpu='9' cpuset='17'/>
<vcpupin vcpu='10' cpuset='18'/>
<vcpupin vcpu='11' cpuset='19'/>
<vcpupin vcpu='12' cpuset='20'/>
<vcpupin vcpu='13' cpuset='21'/>
<vcpupin vcpu='14' cpuset='22'/>
<vcpupin vcpu='15' cpuset='23'/>
<emulatorpin cpuset='0-15'/>
<iothreadpin iothread='1' cpuset='0-7'/>
<iothreadpin iothread='2' cpuset='8-15'/>
</cputune>
<os>
<type arch='x86_64' machine='pc-q35-5.0'>hvm</type>
<loader readonly='yes' type='pflash'>/usr/share/ovmf/x64/ovmf_x64.bin</loader>
<nvram>/var/lib/libvirt/qemu/nvram/win10_VARS.fd</nvram>
<boot dev='hd'/>
<bootmenu enable='yes'/>
</os>
<features>
<acpi/>
<apic/>
<hyperv>
<relaxed state='on'/>
<vapic state='on'/>
<spinlocks state='on' retries='8191'/>
</hyperv>
<kvm>
<hidden state='on'/>
</kvm>
<vmport state='off'/>
</features>
<cpu mode='host-passthrough' check='partial'>
<topology sockets='1' cores='8' threads='2'/>
<feature policy='require' name='topoext'/>
<feature policy='disable' name='monitor'/>
<feature policy='require' name='hypervisor'/>
<feature policy='disable' name='svm'/>
<feature policy='disable' name='x2apic'/>
</cpu>
<clock offset='localtime'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
<timer name='hypervclock' present='yes'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled='no'/>
<suspend-to-disk enabled='no'/>
</pm>
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='none' io='native'/>
<source dev='/dev/disk/by-id/ata-ADATA_SP550_2G3820048000'/>
<target dev='vda' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0c' function='0x0'/>
</disk>
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='none' io='native'/>
<source dev='/dev/disk/by-id/ata-Samsung_SSD_860_QVO_1TB_S4CZNF0M287506M'/>
<target dev='vdb' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0d' function='0x0'/>
</disk>
<controller type='usb' index='0' model='qemu-xhci' ports='15'>
<address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
</controller>
<controller type='pci' index='0' model='pcie-root'/>
<controller type='pci' index='1' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='1' port='0x8'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
</controller>
<controller type='pci' index='2' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='2' port='0x9'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
</controller>
<controller type='pci' index='3' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='3' port='0xa'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
</controller>
<controller type='pci' index='4' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='4' port='0xb'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/>
</controller>
<controller type='pci' index='5' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='5' port='0xc'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/>
</controller>
<controller type='pci' index='6' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='6' port='0xd'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/>
</controller>
<controller type='pci' index='7' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='7' port='0xe'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x6'/>
</controller>
<controller type='pci' index='8' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='8' port='0x1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1c' function='0x0' multifunction='on'/>
</controller>
<controller type='sata' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
</controller>
<interface type='direct'>
<mac address='52:54:00:60:5f:d8'/>
<source dev='enp10s0' mode='bridge'/>
<model type='e1000e'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</interface>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<sound model='ich9'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1b' function='0x0'/>
</sound>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x0f' slot='0x00' function='0x0'/>
</source>
<rom file='/usr/share/qemu/TU102.rom'/>
<address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x0f' slot='0x00' function='0x1'/>
</source>
<address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x0f' slot='0x00' function='0x2'/>
</source>
<address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x0f' slot='0x00' function='0x3'/>
</source>
<address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
</hostdev>
<redirdev bus='usb' type='spicevmc'>
<address type='usb' bus='0' port='2'/>
</redirdev>
<redirdev bus='usb' type='spicevmc'>
<address type='usb' bus='0' port='3'/>
</redirdev>
<memballoon model='virtio'>
<address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
</memballoon>
</devices>
<qemu:commandline>
<qemu:arg value='-cpu'/>
<qemu:arg value='host,hv_time,kvm=off,hv_vendor_id=null,-hypervisor'/>
<!-- if I uncomment the below 4 lines I get the "... this family of AMD CPU doesn't support hyperthreading..." error -->
<!-- <qemu:arg value='-global'/> -->
<!-- <qemu:arg value='pcie-root-port.speed=8'/> -->
<!-- <qemu:arg value='-global'/> -->
<!-- <qemu:arg value='pcie-root-port.width=16'/> -->
</qemu:commandline>
</domain>
Sorry for the wall of text but I figured it best to offer as much info as ( I think ) required and what I tried so far.
Curious what causes the performance drop in the VM. Seen several cases where people had almost 1:1 performance as on-metal. Any idea how I can get those qemu arguments to work?
And is the way I set my vcpu pinning correct?
I’m continuing to fiddle around with it and will come back with an update should I figure it out, maybe it’ll help anyone else having this issue.
Thanks!
Edit#1:
Noticed that after setting q35 from “pc-q35-5.0” back to 3.1, Nvidia’s Control Panel shows the PCI-e link at x1, GPU-Z still shows it at x16, didn’t notice any difference in performance though.