Ryzen 7950X / X670E Taichi VFIO Success (With Notes)

BuzzBumbleBee · October 3, 2022, 8:15am

Hello All,

Just writing up my findings of moving my QEMU / Libvirt virtual machine with GPU and NVME pass-through from my X570 setup to a new X670E setup.

Firstly here are some limitations i have found with the new platform :

The initial set of BIOS’ seem to lack a selector for the primary GPU selection mode, this means you cannot force the integrated GPU and currently will always select any dGPU as the BOOT GPU (this may differ on other board manufacturers but seeing as the option should be in the PBS menu its probably an AGESA issue)
The newer BIOS versions 1.07 and 1.08 seem top mess up the IOMMU groupings (i could not find any way to get this working again( regardless of having iommu set to on or auto in bios). Again this may well be an AGESA thing and not specific to ASRock
Even on the initial bios the IOMMU groupings have some limitations (will go over these below)

My System Spec :

CPU : Ryzen 7950X
Motherboard : ASrock X670E Taichi
RAM : 32GB DDR5-6000
Host GPU : Integrated GFX
Guest GPU : RX6800XT (PCIE1 / Top Slot / CPU Attached)
Host Storage : Crucial P2 (Second NVME Slot / Chipset Attached)
Guest Storage : Crucial P4 (First NVME Slot / CPU Attached)
OS : Arch Rolling

Resulting IOMMU Groups

IOMMU Group 0:
	00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
IOMMU Group 1:
	00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14db]
IOMMU Group 2:
	00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14db]
IOMMU Group 3:
	00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
IOMMU Group 4:
	00:02.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14db]
IOMMU Group 5:
	00:02.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14db]
IOMMU Group 6:
	00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
IOMMU Group 7:
	00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
IOMMU Group 8:
	00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
IOMMU Group 9:
	00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14dd]
IOMMU Group 10:
	00:08.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14dd]
IOMMU Group 11:
	00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 71)
	00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 12:
	00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e0]
	00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e1]
	00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e2]
	00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e3]
	00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e4]
	00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e5]
	00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e6]
	00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e7]
IOMMU Group 13:
	01:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch [1002:1478] (rev c1)
IOMMU Group 14:
	02:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch [1002:1479]
IOMMU Group 15:
	03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 [Radeon RX 6800/6800 XT / 6900 XT] [1002:73bf] (rev c1)
IOMMU Group 16:
	03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21/23 HDMI/DP Audio Controller [1002:ab28]
IOMMU Group 17:
	03:00.2 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:73a6]
IOMMU Group 18:
	03:00.3 Serial bus controller [0c80]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 USB [1002:73a4]
IOMMU Group 19:
	04:00.0 PCI bridge [0604]: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] [8086:1136] (rev 02)
IOMMU Group 20:
	05:00.0 PCI bridge [0604]: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] [8086:1136] (rev 02)
IOMMU Group 21:
	05:01.0 PCI bridge [0604]: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] [8086:1136] (rev 02)
IOMMU Group 22:
	05:02.0 PCI bridge [0604]: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] [8086:1136] (rev 02)
IOMMU Group 23:
	05:03.0 PCI bridge [0604]: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] [8086:1136] (rev 02)
IOMMU Group 24:
	06:00.0 USB controller [0c03]: Intel Corporation Thunderbolt 4 NHI [Maple Ridge 4C 2020] [8086:1137]
IOMMU Group 25:
	08:00.0 USB controller [0c03]: Intel Corporation Thunderbolt 4 USB Controller [Maple Ridge 4C 2020] [8086:1138]
IOMMU Group 26:
	45:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f4] (rev 01)
IOMMU Group 27:
	46:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
IOMMU Group 28:
	46:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
	48:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 02)
IOMMU Group 29:
	46:05.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
	49:00.0 Network controller [0280]: Intel Corporation Wi-Fi 6 AX210/AX211/AX411 160MHz [8086:2725] (rev 1a)
IOMMU Group 30:
	46:06.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
	4a:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. Killer E3000 2.5GbE Controller [10ec:3000] (rev 06)
IOMMU Group 31:
	46:07.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
	4b:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 02)
IOMMU Group 32:
	46:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
	4c:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f4] (rev 01)
	4d:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
	4d:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
	4d:05.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
	4d:06.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
	4d:07.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
	4d:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
	4d:0c.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
	4d:0d.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
	53:00.0 Non-Volatile memory controller [0108]: Micron/Crucial Technology P2 NVMe PCIe SSD [c0a9:540a] (rev 01)
	54:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f7] (rev 01)
	55:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f6] (rev 01)
IOMMU Group 33:
	46:0c.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
	56:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f7] (rev 01)
IOMMU Group 34:
	46:0d.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
	57:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f6] (rev 01)
IOMMU Group 35:
	58:00.0 Non-Volatile memory controller [0108]: Micron/Crucial Technology Device [c0a9:5407]
IOMMU Group 36:
	59:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Raphael [1002:164e] (rev c1)
IOMMU Group 37:
	59:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Rembrandt Radeon High Definition Audio Controller [1002:1640]
IOMMU Group 38:
	59:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] VanGogh PSP/CCP [1022:1649]
IOMMU Group 39:
	59:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15b6]
IOMMU Group 40:
	59:00.4 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15b7]
IOMMU Group 41:
	59:00.6 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Family 17h/19h HD Audio Controller [1022:15e3]
IOMMU Group 42:
	5a:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15b8]

As you can see from above although all CPU attached devices and some chipset attached devices are in their own groups the 2nd nvme drive that is attached to the chipset is bundled together with a USB and SATA controller. This does limit using the platform out of box for creating a system that can run 2 simultaneous guests with dedicated nvme and gpus passed through. Also of note is that for me passing through the last USB controller 5a:00.0 caused a system reboot (still investigating)

With all that said and taking those limitations into account i was able to setup the virtual machine with pass-through with no real alterations to what I had done on X570, I dont make use of binding the gpu to the vfio kernel module on boot seeing as RDNA2 and amdgpu can reset and unbind fine.

Mutter / Gnome On non boot gpu

One thing i added was a udev rule to force mutter (gnome and gdm to run on the IGP / 2nd display adapter), for this i created /usr/lib/udev/rules.d/61-mutter-primary-gpu.rules with the following rule

ENV{DEVNAME}=="/dev/dri/card1", TAG+="mutter-device-preferred-primary"

XML
For completeness here is my libvirt XML definition (i know the CPU pinning is bit odd in this version)

<domain type="kvm">
  <name>win10</name>
  <uuid>***-***-***-***</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://microsoft.com/win/10"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit="KiB">10485760</memory>
  <currentMemory unit="KiB">13312</currentMemory>
  <vcpu placement="static">12</vcpu>
  <iothreads>1</iothreads>
  <cputune>
    <vcpupin vcpu="0" cpuset="9"/>
    <vcpupin vcpu="1" cpuset="10"/>
    <vcpupin vcpu="2" cpuset="11"/>
    <vcpupin vcpu="3" cpuset="12"/>
    <vcpupin vcpu="4" cpuset="13"/>
    <vcpupin vcpu="5" cpuset="14"/>
    <vcpupin vcpu="6" cpuset="25"/>
    <vcpupin vcpu="7" cpuset="26"/>
    <vcpupin vcpu="8" cpuset="27"/>
    <vcpupin vcpu="9" cpuset="28"/>
    <vcpupin vcpu="10" cpuset="29"/>
    <vcpupin vcpu="11" cpuset="30"/>
    <emulatorpin cpuset="0,16"/>
    <iothreadpin iothread="1" cpuset="0,6"/>
  </cputune>
  <os>
    <type arch="x86_64" machine="pc-q35-7.0">hvm</type>
    <loader readonly="yes" secure="yes" type="pflash">/usr/share/edk2-ovmf/x64/OVMF_CODE.secboot.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/win10_VARS.fd</nvram>
    <boot dev="hd"/>
    <bootmenu enable="yes"/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv mode="custom">
      <relaxed state="on"/>
      <vapic state="on"/>
      <spinlocks state="on" retries="8191"/>
      <vendor_id state="on" value="myv3nd0r1d"/>
    </hyperv>
    <vmport state="off"/>
    <smm state="on"/>
  </features>
  <cpu mode="host-passthrough" check="none" migratable="on">
    <topology sockets="1" dies="1" cores="6" threads="2"/>
    <feature policy="require" name="topoext"/>
  </cpu>
  <clock offset="localtime">
    <timer name="rtc" tickpolicy="catchup"/>
    <timer name="pit" tickpolicy="delay"/>
    <timer name="hpet" present="no"/>
    <timer name="hypervclock" present="yes"/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled="no"/>
    <suspend-to-disk enabled="no"/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <controller type="usb" index="0" model="qemu-xhci" ports="15">
      <address type="pci" domain="0x0000" bus="0x02" slot="0x00" function="0x0"/>
    </controller>
    <controller type="pci" index="0" model="pcie-root"/>
    <controller type="pci" index="1" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="1" port="0x10"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0" multifunction="on"/>
    </controller>
    <controller type="pci" index="2" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="2" port="0x11"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x1"/>
    </controller>
    <controller type="pci" index="3" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="3" port="0x12"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x2"/>
    </controller>
    <controller type="pci" index="4" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="4" port="0x13"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x3"/>
    </controller>
    <controller type="pci" index="5" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="5" port="0x14"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x4"/>
    </controller>
    <controller type="pci" index="6" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="6" port="0x15"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x5"/>
    </controller>
    <controller type="pci" index="7" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="7" port="0x16"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x6"/>
    </controller>
    <controller type="pci" index="8" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="8" port="0x17"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x7"/>
    </controller>
    <controller type="pci" index="9" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="9" port="0x18"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0" multifunction="on"/>
    </controller>
    <controller type="pci" index="10" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="10" port="0x19"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x1"/>
    </controller>
    <controller type="pci" index="11" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="11" port="0x1a"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x2"/>
    </controller>
    <controller type="pci" index="12" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="12" port="0x1b"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x3"/>
    </controller>
    <controller type="pci" index="13" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="13" port="0x1c"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x4"/>
    </controller>
    <controller type="pci" index="14" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="14" port="0x1d"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x5"/>
    </controller>
    <controller type="pci" index="15" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="15" port="0x8"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x0"/>
    </controller>
    <controller type="pci" index="16" model="pcie-to-pci-bridge">
      <model name="pcie-pci-bridge"/>
      <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
    </controller>
    <controller type="sata" index="0">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x1f" function="0x2"/>
    </controller>
    <controller type="virtio-serial" index="0">
      <address type="pci" domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
    </controller>
    <interface type="network">
      <mac address="52:54:00:3e:fa:26"/>
      <source network="network"/>
      <model type="e1000e"/>
      <address type="pci" domain="0x0000" bus="0x0b" slot="0x00" function="0x0"/>
    </interface>
    <input type="mouse" bus="ps2"/>
    <input type="keyboard" bus="ps2"/>
    <tpm model="tpm-crb">
      <backend type="emulator" version="2.0"/>
    </tpm>
    <audio id="1" type="none"/>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x58" slot="0x00" function="0x0"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x08" slot="0x00" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x59" slot="0x00" function="0x3"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x09" slot="0x00" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x59" slot="0x00" function="0x4"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x0a" slot="0x00" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x04" slot="0x00" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x03" slot="0x00" function="0x1"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x03" slot="0x00" function="0x2"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x06" slot="0x00" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x03" slot="0x00" function="0x3"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x07" slot="0x00" function="0x0"/>
    </hostdev>
    <memballoon model="none"/>
  </devices>
</domain>

Note : cross posted to reddit r/vfio for viability

dstegos · October 3, 2022, 3:52pm

Thanks for this. I have been looking to upgrade my box, but the IOMMU groupings are kinda scaring me off. The constant grouping regression with AGESA upgrades is problematic.

felixrichter · December 8, 2022, 12:24am

Hey guys I just upgraded my desktop to an AMD 7950x it is running great with my linux install (except cpu temps … but that is another theread)

I was wondering is there a good resource on kernel cmdline parameters that can be used to effect (amd) cpu behavior?

Janos · January 20, 2023, 11:38pm

Hi BuzzBumbleBee,

can you please share a bit more of your configuration in regards to your guest GPU?
Without VFIO works for me with NVMEs and USB controllers, but the last time I tried it, not with the GPU, at least not reliably.
I mean you have to tell gnome and gdm somehow that there is a second GPU available, as soon as the VM is powered off.

For example with a Libvirt hook script.

> #!/bin/bash
> set -x
>   
> # Re-Bind GPU to Nvidia Driver
> virsh nodedev-reattach pci_0000_0c_00_1
> virsh nodedev-reattach pci_0000_0c_00_0
> 
> # Reload nvidia modules
> modprobe nvidia
> modprobe nvidia_modeset
> modprobe nvidia_uvm
> modprobe nvidia_drm
> 
> # Rebind VT consoles
> echo 1 > /sys/class/vtconsole/vtcon0/bind
> # Some machines might have more than 1 virtual console. Add a line for each corresponding VTConsole
> #echo 1 > /sys/class/vtconsole/vtcon1/bind
> 
> nvidia-xconfig --query-gpu-info > /dev/null 2>&1
> echo "efi-framebuffer.0" > /sys/bus/platform/drivers/efi-framebuffer/bind
> 
> # Restart Display Manager
> systemctl start display-manager.service

I use Majaro with KDE and also have a 6800XT.

Regards

DS_DV · January 31, 2023, 8:36am

Hi (:
thanks for your great write up (:
I am on an x670 Aorus Master myself and waiting for my 7900xtx to arrive.
Can you elaborate on how you pass though the GPU without binding it to vfio on boot?

That would potentially enable me to also use my setup for linux gaming

with kind regards

PS: my old system (x370 with a 2nd GPU (WX3200) did not manage to do that

softwaretools · February 8, 2023, 2:30pm

Have you tried turning on ACS in the BIOS? I read it can give better IOMMU groupings and may free up that NVME/USB/SATA bundling.

If you do decide to turn it on, could I see your IOMMU groupings after? Thanks!

DS_DV · February 8, 2023, 10:45pm

sadly not all bioses support this.

F8 on X670 Aorus Master as ARI AER but no ACS

softwaretools · February 8, 2023, 11:30pm

Darn, thanks for checking

DS_DV · February 11, 2023, 4:59pm

to add a late reply.

ACS as far as i understand is not a hardware thing. So as the AM5 Platform matures there is a chance of it getting added

nome_ficticio · March 2, 2023, 12:51pm

so, any of you using freesync and multimonitor?
if so, could you confirm if it is possible to have freesync working on the monitor used for the VM and gaming?

quilt · March 2, 2023, 1:07pm

You should give some more details about the setup you have in mind.

If you mean a monitor connected directly to a passed-through gpu, yes it will work the same as it would natively in Windows.

nome_ficticio · March 2, 2023, 1:29pm

i have one monitor with freesync (used with DP, 1440p 144hz) and 2 tvs i use as 2ndary screens (both are 1080p, via hdmi, no VRR).
i have 2 dgpus, a 7900xt and a vega 56.

i just want to game on linux, native or through a VM, retaining both freesync on the gaming screen and multimonitor, but it seems natively, this is impossible. so if a VM with the 7900xt passthrough lets me solve this, i think the cpu resource loss from the virtualization layer should be negligible enough to make it a more than viable solution.

i don’t have a feasible testing ability yet cus my current cpu is old and wouldnt be enough to play a reasonably demanding game in a VM without major fps loss.

quilt · March 2, 2023, 1:48pm

In any case if you passthrough the 7900xt, it will have the exact same capabilities as it would on Windows.

Natively on linux is pretty off topic, probably better to start your own thread. Have you tried both X11/Wayland?

nome_ficticio · March 2, 2023, 3:47pm

ill try to spin qemu once i get a cpu upgrade and see if everything works as it should under the VM. will need a new psu as well, to support both gpus without the system getting angry at me

nome_ficticio · March 2, 2023, 8:00pm

can anyone tell me if the gpu that stays with linux native being on a x4 pcie 4 slot has any relative negative impact given it wont be used for gaming?

Birthdayboy · March 5, 2023, 10:11am

Hi, have you figured out what caused the USB controller reboot? Also, I am interested in building an AM5 system for VFIO, would you suggest the X670E Taichi or the B650 variant? I plan on using an R9 7900 CPU and an RX6800 (non XT) GPU for lower TDP but I don’t think it will have any different impact, will it?

Also, how did you manage to use keyboard and mouse in the end?

nome_ficticio · March 6, 2023, 9:27pm

i recommend you check buildzoid’s vids on the topic (youtube “actually hardcore overclocking”), he has a vid for all asrock mbs, one for gigabyte’s and one for msi’s… “hardware unboxed” on yt also have a few vids going over and comparing all am5 boards, segmented by their brackets, which can have some useful info as well.

we each have our own needs and preferences, so it’s impossible to give you a recommendation that is precise and valid for you specifically…

im getting an MSI pro x670-p wifi. i want to spend as little as possible, but get 1 pcie 5 m.2 slot, i want 2 pcie slots for 2 gpus, so 1 x16 and another at least x4 that doesnt compromise the x16, plus one or 2 other slots, at least 1 for pcie expansion cards, like a 10gig lan… asus is just about the worst bet u can make rn on am5, and while gigabyte has the fastest boot times and out of the box memory timings for ram, i plan to tweak ram timings manually and boot time doesnt rly matter much for me. msi has been giving us the best overclocking and bios settings in the last few gens so, that’s the path im walking down.

jxdking · March 7, 2023, 9:17pm

ASUS ProArt B650 is the one with the most PCIE slots connectivity current available. More than any x670e board.

nome_ficticio · March 13, 2023, 10:22pm

tysm for telling me something i already knew and cared nothing about. next time, i recommend reading the actual posts so as to not randomly interject to mention water is wet to people.

the am5 asus boards are still the most terrible boards for gaming by far btw. worse than brands mainstream has never even heard about…

quilt · March 13, 2023, 11:42pm

Honestly reading your comment also gives me the impression you are looking for pcie connectivity over everything else. So it’s not a bad suggestion.

If you don’t like asus I doubt anyone cares. But if you don’t give any reason except “they’re really the worst this gen” that doesn’t add much to the discussion.