MSI MEG X570 Unify vs Gigabyte X570 Aorus Master for VFIO

nalox21506 · April 22, 2020, 1:33am

Hi all

I’m looking at purchasing an X570 motherboard for VFIO setup.

I have got the CPU (AMD Ryzen 3700x) and GPUs (Nvidia GTX680 and AMD 5700xt). I’m trying to pick the motherboard.

Gigabyte X570 Aorus Master

Pros:

Good IOMMU group support reported by forum members especially by @wendell.
Can select the boot GPU in the BIOS. So I can arrange the GPUs in my preferred orders. For example, host GPU in the last PCIe slot and the passthrough GPU in the first slot for best performance.

Cons:

Known problems with audio out support in Linux (probably due to the ESS SABRE 9118 DAC). See https:// forum.level1techs .com/t/x570-aorus-master-audio-issue-with-linux/148784
The IO controller chip (SuperIO) ITE IT8688E has no driver support in Linux, so no temperature and fans speed readings. See https://www.reddit.com/r/Amd/comments/cy34hc/x570_linux_users/

MSI MEG X570 Unify

Pros:

The IO controller is NUVOTON NCT6797, which is reportedly supported by Linux
The audio chip is the “vanilla” Realtek ALC1220, so should have no problems.

Cons:

No boot GPU options in BIOS
NIC is the Realtek 2.5G only. Not sure whether it’s well supported by Linux.

Regarding the boot GPU options

I read that there are ways to tweak the CSM setting in BIOS to enable the GPU in the chipset PCIe slot (i.e. the 3rd slot) as boot up device. And also the boot up screen will show on all GPUs/displays. I’m not sure whether this is sufficient to achieve booting into the host OS desktop using the stock mainline kernel without any patches/tweaks.
Maybe I can make a compromise and use the first/second PCIe slots (i.e. both connected directly to the CPU) for host and VM GPUs. So the PCIe split will be x8 and x8, which shouldn’t have too much impact on the performance. But again I’m not sure whether the two GPUs will be in the same IOMMU group and causing further passthrough problems. I prefer not to use the ACS kernel patch.
See Gigabyte motherboard manual: https://download.gigabyte.com/FileList/Manual/mb_manual_x570-aorus-master_1002_190708_e.pdf

Initial Display Output
Specifies the first initiation of the monitor display from the the installed PCI Express graphics card.
PCIe 1 Slot Sets the graphics card on the PCIEX16 slot as the first display. (Default)
PCIe 2 Slot Sets the graphics card on the PCIEX8 slot as the first display.
PCIe 3 Slot (Note) Sets the graphics card on the PCIEX4 slot as the first display.

Regarding the “ACS Enable” BIOS option

I read somewhere that If ACS is enabled in the BIOS (note not the kernel patch), the two GPUs on CPU PCIe lanes (i.e. the first/second PCIe slots), will be in separate IOMMU groups, which should avoid the need of the ACS override kernel patch. Is this correct?
Post reported positive result: https://forum.level1techs.com/t/gigabyte-x570-aorus-elite-iommu-groups/144937/39
Post reported problems: https://forum.level1techs.com/t/x570-taichi-iommu-groups/145762/6

Thanks! Looking forward to your insight and/or suggestions of other motherboard models.

(sorry for the broken urls)

*Edited with proper URLs

Odemia · April 22, 2020, 4:53am

I just got LookingGlassworking with my X570 Unify. Just proof of concept right now; 5700XT (host) in 1st x16 and R7 370X (guest) in 2nd x16. So it is x8/x8 and seeing no degradation in performance on the host, actually try to do most of my gaming in Linux these days anyway, so might just try to pick up a used 1660 or something for the few windows only titles.

FYI the wifi6 and 2.5G both just worked in Fedora31.

I didn’t even think to boot the host on the chipset’s x16. I will take a look at the Bios tomorrow and see what I can confirm regarding options to enable booting with video from 3rd PCIe.

nalox21506 · April 22, 2020, 10:59am

Ah! That’s a good idea! I will look into Looking Glass as well. It would be super convenient if works.

BTW, how is the chipset fan working on your MEG X570 Unify? Is it noisy and does it spin up often? Is the fan speed curve controllable in BIOS? I read that MSI BIOS used to have problems and chipset fan wouldn’t spin down. Not sure whether it’s fixed now.

Thanks for help!

Odemia · April 22, 2020, 9:39pm

Looking in BIOS reminded me. There are IOMMU and SVM options you need to enable to get kvm and pass through working. Those options do not appear to be reachable through the normal menus. But the search feature will show you everything if you know what you are looking for. I had to search (F9) for “IOMMU” and “SVM”, to enable them.

I do not see any options to control which PCIe lane to use for boot graphics. Just an option to initiate PEG (PCIe Graphics) and IGD (Integrated Graphics). I know when I was building this it seemed to default to PCIe_1 but would switch to PCIe_3. Not sure about PCIe_5.

Chipset fan has not been a problem, N=1.

Only ever noticed the chipset fan on the bench at boot when it would spin to full and wasn’t in a case. If it spins to full it would be annoying. Pretty sure it has not been spinning, if it has it is well managed.

That said in my system boot drive (NVME), both GPUs, keyboard, mouse and controller are connected directly to the CPU. Attached to the chipset are: USB card (PCIe 3.0 x1 in the bottom x16 slot), 1 SATA SSD (win10), 1 SATA HDD, USB 3.1 external drive dock. So my chipset also has no reason to be getting hot.

nalox21506 · April 23, 2020, 12:01am

Hi Odemia

Thank you so much for checking the BIOS settings.

Yep, I have done extensive research (based on the manuals) and Gigabyte is the only vendor supports that. I contacted MSI support and the answer was also no. I will try to convince them to implement the support.

Would you mind explain what “switch to PCIe_3” means?

If not too much trouble, do you mind checking whether lm-sensor reports the cpu/chipset/case fan speed and temperatures? Just want to make sure the IO controller is supported.

So overall I’m leaning toward MSI.

Thanks again

Odemia · April 23, 2020, 2:12am

When there was no GPU in PCIe_1 it auto switched to outputting on through PCIe_3.

Nothing from the mobo is reported currently.

# sensors
k10temp-pci-00c3
Adapter: PCI adapter
Tdie:         +54.8°C  (high = +70.0°C)
Tctl:         +54.8°C  

nvme-pci-0100
Adapter: PCI adapter
Composite:    +37.9°C  (low  =  -0.1°C, high = +76.8°C)
                       (crit = +79.8°C)

...

nalox21506 · April 23, 2020, 1:19pm

Odemia:

Nothing from the mobo is reported currently.

# sensors
k10temp-pci-00c3
Adapter: PCI adapter
Tdie:         +54.8°C  (high = +70.0°C)
Tctl:         +54.8°C  

nvme-pci-0100
Adapter: PCI adapter
Composite:    +37.9°C  (low  =  -0.1°C, high = +76.8°C)
                       (crit = +79.8°C)

...

I guess the IO Controller kernel module needs to be loaded to get the readings.

sensor-detect may be useful to see whether it can be detected.

Related discussions:
https://github.com/lm-sensors/lm-sensors/issues/134

Example of working output (not sure whether he/she has done anything to get it working)
https://github.com/ocerman/zenpower/issues/12#issuecomment-578497669

nct6797-isa-0a20
Adapter: ISA adapter
in0:                    +0.71 V  (min =  +0.00 V, max =  +1.74 V)
in1:                    +1.01 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in2:                    +3.33 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in3:                    +3.31 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in4:                    +1.01 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in5:                    +0.14 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in6:                    +0.91 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in7:                    +3.33 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in8:                    +3.28 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in9:                    +1.81 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in10:                   +0.67 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in11:                   +0.57 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in12:                   +1.10 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in13:                   +0.67 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in14:                   +1.52 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
fan1:                   545 RPM  (min =    0 RPM)
fan2:                   457 RPM  (min =    0 RPM)
fan3:                   586 RPM  (min =    0 RPM)
fan4:                   518 RPM  (min =    0 RPM)
fan5:                   499 RPM  (min =    0 RPM)
fan6:                     0 RPM  (min =    0 RPM)
fan7:                     0 RPM  (min =    0 RPM)
SYSTIN:                 +30.0°C  (high = +80.0°C, hyst = +75.0°C)  sensor = CPU diode
CPUTIN:                 +29.0°C  (high = +112.0°C, hyst = +85.0°C)  sensor = thermistor
AUXTIN0:                +31.0°C  (high = +112.0°C, hyst = +85.0°C)  sensor = thermistor
AUXTIN1:                +45.0°C    sensor = thermistor
AUXTIN2:                +52.0°C    sensor = thermistor
AUXTIN3:                 -2.0°C    sensor = thermistor
SMBUSMASTER 0:          +32.0°C  
PCH_CHIP_CPU_MAX_TEMP:   +0.0°C  
PCH_CHIP_TEMP:           +0.0°C  
PCH_CPU_TEMP:            +0.0°C  
intrusion0:            ALARM
intrusion1:            ALARM
beep_enable:           disabled

Odemia · April 23, 2020, 1:52pm

Yeah, running sensors-detect helps. Sorry not never used lm_sensors before.

That found nct6797-isa which has fan1-7:

$ sensors
k10temp-pci-00c3
Adapter: PCI adapter
Tdie:         +44.9°C  (high = +70.0°C)
Tctl:         +44.9°C  

nvme-pci-0100
Adapter: PCI adapter
Composite:    +34.9°C  (low  =  -0.1°C, high = +76.8°C)
                       (crit = +79.8°C)

nct6797-isa-0a20
Adapter: ISA adapter
in0:           +0.97 V  (min =  +0.00 V, max =  +1.74 V)
in1:           +1.01 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in2:           +3.34 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in3:           +3.34 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in4:           +1.01 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in5:           +0.14 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in6:           +0.73 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in7:           +3.34 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in8:           +3.23 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in9:           +1.83 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in10:          +0.00 V  (min =  +0.00 V, max =  +0.00 V)
in11:          +0.60 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in12:          +1.10 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in13:          +0.68 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in14:          +1.46 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
fan1:            0 RPM  (min =    0 RPM)
fan2:         1094 RPM  (min =    0 RPM)
fan3:          769 RPM  (min =    0 RPM)
fan4:          639 RPM  (min =    0 RPM)
fan5:            0 RPM  (min =    0 RPM)
fan6:          635 RPM  (min =    0 RPM)
fan7:            0 RPM  (min =    0 RPM)
SYSTIN:        +34.0°C  (high = +80.0°C, hyst = +75.0°C)  sensor = CPU diode
CPUTIN:        +36.0°C  (high = +108.0°C, hyst = +90.0°C)  sensor = thermistor
AUXTIN0:       +41.5°C  (high = +108.0°C, hyst = +90.0°C)  sensor = thermistor
AUXTIN1:      -128.0°C    sensor = thermistor
AUXTIN2:       +49.0°C    sensor = thermistor
AUXTIN3:        +1.0°C    sensor = thermistor
Virtual_TEMP:  +45.0°C  
Virtual_TEMP:  +45.0°C  
Virtual_TEMP:  +45.0°C  
Virtual_TEMP:  +34.0°C  
intrusion0:   ALARM
intrusion1:   ALARM
beep_enable:  disabled

amdgpu-pci-2f00
Adapter: PCI adapter
vddgfx:       +0.78 V  
fan1:         515 RPM  (min =    0 RPM, max = 3200 RPM)
edge:         +53.0°C  (crit = +118.0°C, hyst = -273.1°C)
                       (emerg = +99.0°C)
junction:     +53.0°C  (crit = +99.0°C, hyst = -273.1°C)
                       (emerg = +99.0°C)
mem:          +58.0°C  (crit = +99.0°C, hyst = -273.1°C)
                       (emerg = +99.0°C)
power1:       31.00 W  (cap = 165.00 W)

Note my system has CPU_FAN1, SYS_FAN{1,2,4}. Based on that and the reported RPMs I think the mapping is:

sensors  mobo       RPM
fan1     PUMP_FAN1  NC
fan2     CPU_FAN1   1094
fan3     SYS_FAN1   769 (exhaust)
fan4     SYS_FAN2   639 (Intake)
fan5     SYS_FAN3   NC
fan6     SYS_FAN4   635 (Intake)
fan7     SYS_FAN5   NC

So still no PCH fan, that is too bad.

nalox21506 · April 24, 2020, 11:29pm

This is very helpful. Thank you so much Odemia!

nalox21506 · May 4, 2020, 12:38am

@Odemia

I noticed that MSI BIOS doesn’t seem to have options to tweak ACS, AER, nor Alternative Routing settings. They are probably default to some values that MSI see fit.

Do you mind posting your full IOMMU group details here?

I’m curious to see how it compares to the Gigabyte X570 Aorus Master (like in this post https://forum.level1techs.com/t/am-i-screwed-if-both-my-gpus-are-in-the-same-iommu-group-aorus-x570-master/156654/8?u=nalox21506)

I suspect it’s going to be the same between MSI/Gigabyte because the CPU and the PCH are all AMD’s work at chip level and the motherboard vendors wouldn’t be able to change anything.

Just for my curiosity, especially around the USB controllers IOMMU arrangements.

The shell script to print the IOMMU group details can be found here:

https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#Ensuring_that_the_groups_are_valid

Thanks

Odemia · May 6, 2020, 7:40pm

Yeah I already did this as part of getting VFIO/Looking glass setup, even sorted the groups:

IOMMU Groups

$ ./iommu_groups.sh 
IOMMU Group 0:
	00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 1:
	00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 2:
	00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 3:
	00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 4:
	00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 5:
	00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 6:
	00:03.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 7:
	00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 8:
	00:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 9:
	00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 10:
	00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 11:
	00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 12:
	00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 13:
	00:08.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 14:
	00:08.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 15:
	00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61)
	00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 16:
	00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 0 [1022:1440]
	00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 1 [1022:1441]
	00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 2 [1022:1442]
	00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 3 [1022:1443]
	00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 4 [1022:1444]
	00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 5 [1022:1445]
	00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 6 [1022:1446]
	00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 7 [1022:1447]
IOMMU Group 17:
	01:00.0 Non-Volatile memory controller [0108]: Intel Corporation SSD 660P Series [8086:f1a8] (rev 03)
IOMMU Group 18:
	20:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse Switch Upstream [1022:57ad]
IOMMU Group 19:
	21:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
IOMMU Group 20:
	21:05.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
IOMMU Group 21:
	21:06.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
IOMMU Group 22:
	21:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]
	2a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
	2a:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
	2a:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU Group 23:
	21:09.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]
	2b:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 24:
	21:0a.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]
	2c:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 25:
	24:00.0 USB controller [0c03]: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller [1912:0014] (rev 03)
IOMMU Group 26:
	27:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller [10ec:8125]
IOMMU Group 27:
	28:00.0 Network controller [0280]: Intel Corporation Wi-Fi 6 AX200 [8086:2723] (rev 1a)
IOMMU Group 28:
	2d:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch [1002:1478] (rev c4)
IOMMU Group 29:
	2e:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch [1002:1479]
IOMMU Group 30:
	2f:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 [Radeon RX 5600 OEM/5600 XT / 5700/5700 XT] [1002:731f] (rev c4)
IOMMU Group 31:
	2f:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 HDMI Audio [1002:ab38]
IOMMU Group 32:
	30:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Curacao PRO [Radeon R7 370 / R9 270/370 OEM] [1002:6811] (rev 81)
	30:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Oland/Hainan/Cape Verde/Pitcairn HDMI Audio [Radeon HD 7000 Series] [1002:aab0]
IOMMU Group 33:
	31:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
IOMMU Group 34:
	32:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
IOMMU Group 35:
	32:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU Group 36:
	32:00.4 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller [1022:1487]
IOMMU Group 37:
	33:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 38:
	34:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)

I haven’t taken the time to figure out which USB ports are in IOMMU Group 21 vs 22. I just added the Renasas card in the bottom PCIe x16 slot (Group 24) and have my kvm grab that. I have a PCIex1 extension coming so I don’t have to use up the x16 slot.

My Navi card is in the top PCIe x16 slot (Group 28 and 29).

Devices 30:00.0 and 30:00.1 (Group 30) are being passed through to KVM.

Odemia · May 6, 2020, 7:49pm

When I looked I did not see any ACS settings. When I next reboot I will try to remember to check for AER or Routing settings.

Is this just out of curiosity or do you have a particular setup in mind?

nalox21506 · May 11, 2020, 12:56am

Thanks. I’m curious about the IOMMU group layout difference because MSI BIOS doesn’t have these setting

ACS Enable
Enable AER Cap
PCIe ARI Support

The Gigabyte one can be found here
https://forum.level1techs.com/t/am-i-screwed-if-both-my-gpus-are-in-the-same-iommu-group-aorus-x570-master/156654/8?u=nalox21506

So they are similar though MSI doesn’t seem to have this device

11:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP [1022:1486]

Also the WIFI device is missing (did you disable it?)

06:00.0 Network controller [0280]: Intel Corporation Wi-Fi 6 AX200 [8086:2723] (rev 1a)

The MSI doesn’t have Intel GbE network card so there is also that.

Odemia · May 11, 2020, 1:48am

yeah, it is disabled. Have no use for wifi on desktops.

FYI wifi worked with default kernel in Fedora 32, ethernet did not. After updating the kernel ethernet worked. Seems whatever kernel is in the installer is just out of date. Promptly disabled wifi and have used ethernet only since.

Correct, reran and searched for the device id “1486” and it is not there.

Odemia · May 11, 2020, 2:18am

Rebooted with Wifi enabled, updated IOMMU groups above. Still no Cryptographic Coprocessor.

Took another look for, but didn’t find any of these:

Access Control Services (ACS)
Advanced Error Reporting (AER)
Alternate Routing-ID Interpretation (AIR)

Only PCI lane settings I could find was the ability to force lane speed or bifurcate PCI_E1 and PCI_E3:

Still have no idea what you are trying to do so not sure if that will help.

localhost · May 15, 2020, 10:52pm

Hey, I know it hasn’t been long since this conversation but I’m wondering if you made a decision and have proceeded with your build yet?

I was hoping to build on an Aorus since they seem like the easiest to build on but they’re sold out everywhere I look. The MSI Unify looks solid but I haven’t seen many people set up vfio on it outside of this thread.

Tagging @Odemia too. I’d love to hear how seamless or difficult your experience was in setting that up. I’m also planning on Fedora as my host os

nalox21506 · May 17, 2020, 1:04am

This is extremely helpful Thanks! Nothing particular, just want to make sure there is no other (hidden) problems due to the lack of ACS and other options in MSI BIOS.

Also it seems like many motherboard vendors build their BIOSes based on the AMI one. AMI has utilities for tweaking the configs, which the BIOS modding community have been using to enable locked features. I’m curious to see whether the AMI utility can reveal the ACS options in MSI BIOS.

nalox21506 · May 17, 2020, 1:24am

I have been holding off to see what B550 may bring to the table, and also because the availability and pricing are not favorable at the moment.

https://www.techpowerup.com/266760/assortment-of-five-amd-b550-premium-motherboards-pictured

The Gigabyte B550 Aorus Master looks nice. Though the second PCI slot is probably PCIe3.0x4.

https://www.tomshardware.com/news/gigabyte-b550-aorus-master-motherboard-pcie-4

FurryJackman · May 17, 2020, 10:42am

Be careful. B550 is made by ASMedia, so anything past the chipset will return to being lumped into the same IOMMU group. The only AMD chipset that doesn’t have this is X570.

nalox21506 · May 18, 2020, 12:59am

Oh… thanks this is very important info