MSI MEG X570 Unify vs Gigabyte X570 Aorus Master for VFIO

Yeah, running sensors-detect helps. Sorry not never used lm_sensors before.

That found nct6797-isa which has fan1-7:

$ sensors
k10temp-pci-00c3
Adapter: PCI adapter
Tdie:         +44.9°C  (high = +70.0°C)
Tctl:         +44.9°C  

nvme-pci-0100
Adapter: PCI adapter
Composite:    +34.9°C  (low  =  -0.1°C, high = +76.8°C)
                       (crit = +79.8°C)

nct6797-isa-0a20
Adapter: ISA adapter
in0:           +0.97 V  (min =  +0.00 V, max =  +1.74 V)
in1:           +1.01 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in2:           +3.34 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in3:           +3.34 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in4:           +1.01 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in5:           +0.14 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in6:           +0.73 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in7:           +3.34 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in8:           +3.23 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in9:           +1.83 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in10:          +0.00 V  (min =  +0.00 V, max =  +0.00 V)
in11:          +0.60 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in12:          +1.10 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in13:          +0.68 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in14:          +1.46 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
fan1:            0 RPM  (min =    0 RPM)
fan2:         1094 RPM  (min =    0 RPM)
fan3:          769 RPM  (min =    0 RPM)
fan4:          639 RPM  (min =    0 RPM)
fan5:            0 RPM  (min =    0 RPM)
fan6:          635 RPM  (min =    0 RPM)
fan7:            0 RPM  (min =    0 RPM)
SYSTIN:        +34.0°C  (high = +80.0°C, hyst = +75.0°C)  sensor = CPU diode
CPUTIN:        +36.0°C  (high = +108.0°C, hyst = +90.0°C)  sensor = thermistor
AUXTIN0:       +41.5°C  (high = +108.0°C, hyst = +90.0°C)  sensor = thermistor
AUXTIN1:      -128.0°C    sensor = thermistor
AUXTIN2:       +49.0°C    sensor = thermistor
AUXTIN3:        +1.0°C    sensor = thermistor
Virtual_TEMP:  +45.0°C  
Virtual_TEMP:  +45.0°C  
Virtual_TEMP:  +45.0°C  
Virtual_TEMP:  +34.0°C  
intrusion0:   ALARM
intrusion1:   ALARM
beep_enable:  disabled

amdgpu-pci-2f00
Adapter: PCI adapter
vddgfx:       +0.78 V  
fan1:         515 RPM  (min =    0 RPM, max = 3200 RPM)
edge:         +53.0°C  (crit = +118.0°C, hyst = -273.1°C)
                       (emerg = +99.0°C)
junction:     +53.0°C  (crit = +99.0°C, hyst = -273.1°C)
                       (emerg = +99.0°C)
mem:          +58.0°C  (crit = +99.0°C, hyst = -273.1°C)
                       (emerg = +99.0°C)
power1:       31.00 W  (cap = 165.00 W)

Note my system has CPU_FAN1, SYS_FAN{1,2,4}. Based on that and the reported RPMs I think the mapping is:

sensors  mobo       RPM
fan1     PUMP_FAN1  NC
fan2     CPU_FAN1   1094
fan3     SYS_FAN1   769 (exhaust)
fan4     SYS_FAN2   639 (Intake)
fan5     SYS_FAN3   NC
fan6     SYS_FAN4   635 (Intake)
fan7     SYS_FAN5   NC

So still no PCH fan, that is too bad.

1 Like

This is very helpful. Thank you so much Odemia!

@Odemia

I noticed that MSI BIOS doesn’t seem to have options to tweak ACS, AER, nor Alternative Routing settings. They are probably default to some values that MSI see fit.

Do you mind posting your full IOMMU group details here?

I’m curious to see how it compares to the Gigabyte X570 Aorus Master (like in this post https://forum.level1techs.com/t/am-i-screwed-if-both-my-gpus-are-in-the-same-iommu-group-aorus-x570-master/156654/8?u=nalox21506)

I suspect it’s going to be the same between MSI/Gigabyte because the CPU and the PCH are all AMD’s work at chip level and the motherboard vendors wouldn’t be able to change anything.

Just for my curiosity, especially around the USB controllers IOMMU arrangements.

The shell script to print the IOMMU group details can be found here:

https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#Ensuring_that_the_groups_are_valid

Thanks

Yeah I already did this as part of getting VFIO/Looking glass setup, even sorted the groups:

IOMMU Groups
$ ./iommu_groups.sh 
IOMMU Group 0:
	00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 1:
	00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 2:
	00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 3:
	00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 4:
	00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 5:
	00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 6:
	00:03.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 7:
	00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 8:
	00:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 9:
	00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 10:
	00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 11:
	00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 12:
	00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 13:
	00:08.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 14:
	00:08.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 15:
	00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61)
	00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 16:
	00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 0 [1022:1440]
	00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 1 [1022:1441]
	00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 2 [1022:1442]
	00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 3 [1022:1443]
	00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 4 [1022:1444]
	00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 5 [1022:1445]
	00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 6 [1022:1446]
	00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 7 [1022:1447]
IOMMU Group 17:
	01:00.0 Non-Volatile memory controller [0108]: Intel Corporation SSD 660P Series [8086:f1a8] (rev 03)
IOMMU Group 18:
	20:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse Switch Upstream [1022:57ad]
IOMMU Group 19:
	21:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
IOMMU Group 20:
	21:05.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
IOMMU Group 21:
	21:06.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
IOMMU Group 22:
	21:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]
	2a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
	2a:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
	2a:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU Group 23:
	21:09.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]
	2b:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 24:
	21:0a.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]
	2c:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 25:
	24:00.0 USB controller [0c03]: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller [1912:0014] (rev 03)
IOMMU Group 26:
	27:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller [10ec:8125]
IOMMU Group 27:
	28:00.0 Network controller [0280]: Intel Corporation Wi-Fi 6 AX200 [8086:2723] (rev 1a)
IOMMU Group 28:
	2d:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch [1002:1478] (rev c4)
IOMMU Group 29:
	2e:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch [1002:1479]
IOMMU Group 30:
	2f:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 [Radeon RX 5600 OEM/5600 XT / 5700/5700 XT] [1002:731f] (rev c4)
IOMMU Group 31:
	2f:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 HDMI Audio [1002:ab38]
IOMMU Group 32:
	30:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Curacao PRO [Radeon R7 370 / R9 270/370 OEM] [1002:6811] (rev 81)
	30:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Oland/Hainan/Cape Verde/Pitcairn HDMI Audio [Radeon HD 7000 Series] [1002:aab0]
IOMMU Group 33:
	31:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
IOMMU Group 34:
	32:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
IOMMU Group 35:
	32:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU Group 36:
	32:00.4 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller [1022:1487]
IOMMU Group 37:
	33:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 38:
	34:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)

I haven’t taken the time to figure out which USB ports are in IOMMU Group 21 vs 22. I just added the Renasas card in the bottom PCIe x16 slot (Group 24) and have my kvm grab that. I have a PCIex1 extension coming so I don’t have to use up the x16 slot.

My Navi card is in the top PCIe x16 slot (Group 28 and 29).

Devices 30:00.0 and 30:00.1 (Group 30) are being passed through to KVM.

1 Like

When I looked I did not see any ACS settings. When I next reboot I will try to remember to check for AER or Routing settings.

Is this just out of curiosity or do you have a particular setup in mind?

1 Like

Thanks. I’m curious about the IOMMU group layout difference because MSI BIOS doesn’t have these setting

ACS Enable
Enable AER Cap
PCIe ARI Support

The Gigabyte one can be found here
https://forum.level1techs.com/t/am-i-screwed-if-both-my-gpus-are-in-the-same-iommu-group-aorus-x570-master/156654/8?u=nalox21506

So they are similar though MSI doesn’t seem to have this device

11:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP [1022:1486]

Also the WIFI device is missing (did you disable it?)

06:00.0 Network controller [0280]: Intel Corporation Wi-Fi 6 AX200 [8086:2723] (rev 1a)

The MSI doesn’t have Intel GbE network card so there is also that.

yeah, it is disabled. Have no use for wifi on desktops.

FYI wifi worked with default kernel in Fedora 32, ethernet did not. After updating the kernel ethernet worked. Seems whatever kernel is in the installer is just out of date. Promptly disabled wifi and have used ethernet only since.

Correct, reran and searched for the device id “1486” and it is not there.

Rebooted with Wifi enabled, updated IOMMU groups above. Still no Cryptographic Coprocessor.

Took another look for, but didn’t find any of these:

  • Access Control Services (ACS)
  • Advanced Error Reporting (AER)
  • Alternate Routing-ID Interpretation (AIR)

Only PCI lane settings I could find was the ability to force lane speed or bifurcate PCI_E1 and PCI_E3:

Still have no idea what you are trying to do so not sure if that will help.

1 Like

Hey, I know it hasn’t been long since this conversation but I’m wondering if you made a decision and have proceeded with your build yet?

I was hoping to build on an Aorus since they seem like the easiest to build on but they’re sold out everywhere I look. The MSI Unify looks solid but I haven’t seen many people set up vfio on it outside of this thread.

Tagging @Odemia too. I’d love to hear how seamless or difficult your experience was in setting that up. I’m also planning on Fedora as my host os

This is extremely helpful Thanks! Nothing particular, just want to make sure there is no other (hidden) problems due to the lack of ACS and other options in MSI BIOS.

Also it seems like many motherboard vendors build their BIOSes based on the AMI one. AMI has utilities for tweaking the configs, which the BIOS modding community have been using to enable locked features. I’m curious to see whether the AMI utility can reveal the ACS options in MSI BIOS.

I have been holding off to see what B550 may bring to the table, and also because the availability and pricing are not favorable at the moment.

https://www.techpowerup.com/266760/assortment-of-five-amd-b550-premium-motherboards-pictured

The Gigabyte B550 Aorus Master looks nice. Though the second PCI slot is probably PCIe3.0x4.

https://www.tomshardware.com/news/gigabyte-b550-aorus-master-motherboard-pcie-4

Be careful. B550 is made by ASMedia, so anything past the chipset will return to being lumped into the same IOMMU group. The only AMD chipset that doesn’t have this is X570.

Oh… thanks this is very important info

Sorry have been MIA for a while.

Have not done anything with trying to get the bottom PCIe slot to work for host/bios. But it is intriguing. Running 4.0x8/3.0x8 has been good enough as I play most games in Linux anyway, so 4.0x8 and 5700 for host. 3.0x8 is currently running R7 370 as proof of concept. Waiting on RTX 3xxx and RX6700 for long term second card.

Also currently have this in an old Fractal R4, so there is only room for single slot coolers in the bottom slot. Hoping for a short 4u case but everything has been out of stock. Once I get a new case for this I will look into using the bottom slot for BIOS/host and top slot in x16 for VM. But given the human malware and supply chain issues that may be a while.

Side note I actually found the SVM and IOMMU settings in the overclocking settings the other day. So they are reachable without using the find function. Just not at all where I would have placed them.

Other than finding the SVM and IOMMU settings I have not had any hardware issues getting VFIO/LG working. Had one minor config issue where windows VM switched itself to 4k output which caused LG server on windows to crash instantly on launch. But after resetting to 1440 It has all been flawless. Having a PCIe 4.0 card or reducing to 1080 would reduce lag a bit but that is a problem with me waiting for next gen GPU.

As for getting things setup in Fedora I basically just followed this thread. When that was missing something or not doing what I wanted I would read up on archwiki.

I kept a text file with all of my steps end to end encase I ever need to redo or adjust anything. Because I was converting a second drive with windows already installed my steps were a bit out of order from the vfio-in-2019 thread. Have been meaning to share my modified steps. Will see if i can format and post my steps as a new thread in the next couple days.

I have posted what is hopefully, all the steps I took in converting my F31+Win10 UEFI dual boot setup to VFIO-Passthrough/LG, here.

Thanks I’ll be sure to give this a proper read later today!

Regarding the bottom pcie slot, I was reading the MSI Unify documentation the other day, and the page on the PCIe slots seemed to suggest that the bottom slot wasn’t compatable with graphics cards, so I would guess that this isn’t possible. I would be interested to know if it’s possible to run the host/bios card in the second slot though, as opposed to your current setup

Just to update here

In the end I have gone for Gigabyte Aorus Master for these reasons

Outstanding problem

2 Likes

This topic was automatically closed 273 days after the last reply. New replies are no longer allowed.