[SOLVED] Unable to isolate GPU for VFIO (workaround)

Operating Systems & Open Source Virtualization
1

I had my VM up and running with my AM4 setup. Today, I finished my new AM5 system and been trying to setup my gaming VM, but no luck.

First I am setting into BIOS my iGPU as primary.
Even then, Manjaro KDE displays the login screen on the screen that is connected to my PCIe GPU (6900XT)

After I login, and since I have disable that screen, desktop loads on the correct one.

So, my first step is to exclude 6900’s HW-IDs from GRUB command line:

GRUB_CMDLINE_LINUX_DEFAULT="quiet apparmor=1 security=apparmor udev.log_priority=3 amd_iommu=on iommu=pt hugepages=16384 vfio-pci.ids=1002:73bf,1002:ab28,1002:73a6,1002:73a4

After updating GRUB and rebooting, login screen is still on 6900XT screen.

Then I am updating MODULES in mkinitcpio.conf:
original:

MODULES="crc32c"

modified:

MODULES="crc32c vfio vfio_iommu_type1 vfio_pci"

also tried this:

MODULES="crc32c vfio vfio_iommu_type1 vfio_pci vfio_virqfd irqbypass kvm kvm_amd xhci_hcd"

No changes needed here:

HOOKS="base udev autodetect modconf block keyboard keymap consolefont plymouth resume filesystems"

Then execute mkinitcpio -P and reboot.

At this point I have both screens black (not blank), I cannot login (I even tried blindly type my pass) and I can only access the console (ctrl+alt+F2) where I revert the changes to mkinitcpio.conf in order to access my desktop.

Any ideas why this is happening…?

Gigabyte Aorus Master X670E F10a
AMD Ryzen 7950X3D
Kernel: 6.1.23-1-MANJARO x86_64
1 Like
2

try without “quiet” parameter. maybe you’re suppressing an error that could be helpful here?

3

This is the log without “quiet”

IMG_2464
IMG_24641920×592 127 KB

4

Try boot parameter kvm.ignore_msrs=1

Also uncomment GRUB_TERMINAL=console in the grub file.

5

You are aware that after you changed the kernel parameters you need to run grub-mkconfig -o /boot/grub/grub.cfg as per Arch wiki for the changes to take effect?

6

Didn’t help either.
Slightly different error now

IMG_2469
IMG_24691920×885 209 KB

I also tried with Windows USB stick and they are displayed in my 6900XT screen, not the iGPU.

So I assume this is a BIOS bug, or some BIOS setting that prevents it for using iGPU as default

1 Like
7

Did your motherboard claim Linux support on it’s store page, manual, box, or anywhere else? If so, I would report the BIOS bug to the manufacturer and/or RMA the board.

If not, I welcome you to the buggy motherboard community. We find workarounds for incompetently written firmwares. Can you post dmidecode outputs for types motherboard system and bios?

I would be curious to see the output of “lspci -nnk”.

8

I am not sure, however the same problem exists in Windows, so it is generic.

dmidecode:

# dmidecode 3.5
Getting SMBIOS data from sysfs.
SMBIOS 3.5.0 present.
Table at 0x796E9000.

Handle 0x0000, DMI type 0, 26 bytes
BIOS Information
        Vendor: American Megatrends International, LLC.
        Version: F10a
        Release Date: 03/22/2023
        Address: 0xF0000
        Runtime Size: 64 kB
        ROM Size: 32 MB
        Characteristics:
                PCI is supported
                BIOS is upgradeable
                BIOS shadowing is allowed
                Boot from CD is supported
                Selectable boot is supported
                BIOS ROM is socketed
                EDD is supported
                Japanese floppy for NEC 9800 1.2 MB is supported (int 13h)
                Japanese floppy for Toshiba 1.2 MB is supported (int 13h)
                5.25"/360 kB floppy services are supported (int 13h)
                5.25"/1.2 MB floppy services are supported (int 13h)
                3.5"/720 kB floppy services are supported (int 13h)
                3.5"/2.88 MB floppy services are supported (int 13h)
                Print screen service is supported (int 5h)
                Serial services are supported (int 14h)
                Printer services are supported (int 17h)
                CGA/mono video services are supported (int 10h)
                ACPI is supported
                USB legacy is supported
                BIOS boot specification is supported
                Targeted content distribution is supported
                UEFI is supported
        BIOS Revision: 5.26

Handle 0x0001, DMI type 1, 27 bytes
System Information
        Manufacturer: Gigabyte Technology Co., Ltd.
        Product Name: X670E AORUS MASTER
        Version: -CF
        Serial Number: Default string
        UUID: 03560274-043c-054b-7406-7e0700080009
        Wake-up Type: Power Switch
        SKU Number: Default string
        Family: X670 MB

Handle 0x0002, DMI type 2, 15 bytes
Base Board Information
        Manufacturer: Gigabyte Technology Co., Ltd.
        Product Name: X670E AORUS MASTER
        Version: x.x
        Serial Number: Default string
        Asset Tag: Default string
        Features:
                Board is a hosting board
                Board is replaceable
        Location In Chassis: Default string
        Chassis Handle: 0x0003
        Type: Motherboard
        Contained Object Handles: 0

Handle 0x0003, DMI type 3, 22 bytes
Chassis Information
        Manufacturer: Default string
        Type: Desktop
        Lock: Not Present
        Version: Default string
        Serial Number: Default string
        Asset Tag: Default string
        Boot-up State: Safe
        Power Supply State: Safe
        Thermal State: Safe
        Security Status: None
        OEM Information: 0x00000000
        Height: Unspecified
        Number Of Power Cords: 1
        Contained Elements: 0
        SKU Number: Default string

Handle 0x0004, DMI type 10, 6 bytes
On Board Device Information
        Type: Video
        Status: Enabled
        Description:    To Be Filled By O.E.M.

Handle 0x0005, DMI type 11, 5 bytes
OEM Strings
        String 1: Default string

Handle 0x0006, DMI type 12, 5 bytes
System Configuration Options
        Option 1: Default string

Handle 0x0007, DMI type 32, 20 bytes
System Boot Information
        Status: No errors detected

Handle 0x0008, DMI type 44, 9 bytes
Unknown Type
        Header and Data:
                2C 09 08 00 FF FF 01 01 00

Handle 0x0009, DMI type 7, 27 bytes
Cache Information
        Socket Designation: L1 - Cache
        Configuration: Enabled, Not Socketed, Level 1
        Operational Mode: Write Back
        Location: Internal
        Installed Size: 1 MB
        Maximum Size: 1 MB
        Supported SRAM Types:
                Pipeline Burst
        Installed SRAM Type: Pipeline Burst
        Speed: 1 ns
        Error Correction Type: Multi-bit ECC
        System Type: Unified
        Associativity: 8-way Set-associative

Handle 0x000A, DMI type 7, 27 bytes
Cache Information
        Socket Designation: L2 - Cache
        Configuration: Enabled, Not Socketed, Level 2
        Operational Mode: Write Back
        Location: Internal
        Installed Size: 16 MB
        Maximum Size: 16 MB
        Supported SRAM Types:
                Pipeline Burst
        Installed SRAM Type: Pipeline Burst
        Speed: 1 ns
        Error Correction Type: Multi-bit ECC
        System Type: Unified
        Associativity: 8-way Set-associative

Handle 0x000B, DMI type 7, 27 bytes
Cache Information
        Socket Designation: L3 - Cache
        Configuration: Enabled, Not Socketed, Level 3
        Operational Mode: Write Back
        Location: Internal
        Installed Size: 128 MB
        Maximum Size: 128 MB
        Supported SRAM Types:
                Pipeline Burst
        Installed SRAM Type: Pipeline Burst
        Speed: 1 ns
        Error Correction Type: Multi-bit ECC
        System Type: Unified
        Associativity: 16-way Set-associative

Handle 0x000C, DMI type 4, 48 bytes
Processor Information
        Socket Designation: AM5
        Type: Central Processor
        Family: Zen
        Manufacturer: Advanced Micro Devices, Inc.
        ID: 12 0F A6 00 FF FB 8B 17
        Signature: Family 25, Model 97, Stepping 2
        Flags:
                FPU (Floating-point unit on-chip)
                VME (Virtual mode extension)
                DE (Debugging extension)
                PSE (Page size extension)
                TSC (Time stamp counter)
                MSR (Model specific registers)
                PAE (Physical address extension)
                MCE (Machine check exception)
                CX8 (CMPXCHG8 instruction supported)
                APIC (On-chip APIC hardware supported)
                SEP (Fast system call)
                MTRR (Memory type range registers)
                PGE (Page global enable)
                MCA (Machine check architecture)
                CMOV (Conditional move instruction supported)
                PAT (Page attribute table)
                PSE-36 (36-bit page size extension)
                CLFSH (CLFLUSH instruction supported)
                MMX (MMX technology supported)
                FXSR (FXSAVE and FXSTOR instructions supported)
                SSE (Streaming SIMD extensions)
                SSE2 (Streaming SIMD extensions 2)
                HTT (Multi-threading)
        Version: AMD Ryzen 9 7950X3D 16-Core Processor          
        Voltage: 1.3 V
        External Clock: 100 MHz
        Max Speed: 5750 MHz
        Current Speed: 4200 MHz
        Status: Populated, Enabled
        Upgrade: Other
        L1 Cache Handle: 0x0009
        L2 Cache Handle: 0x000A
        L3 Cache Handle: 0x000B
        Serial Number: Unknown
        Asset Tag: Unknown
        Part Number: Unknown
        Core Count: 16
        Core Enabled: 16
        Thread Count: 32
        Characteristics:
                64-bit capable
                Multi-Core
                Hardware Thread
                Execute Protection
                Enhanced Virtualization
                Power/Performance Control

Handle 0x000D, DMI type 18, 23 bytes
32-bit Memory Error Information
        Type: OK
        Granularity: Unknown
        Operation: Unknown
        Vendor Syndrome: Unknown
        Memory Array Address: Unknown
        Device Address: Unknown
        Resolution: Unknown

Handle 0x000E, DMI type 16, 23 bytes
Physical Memory Array
        Location: System Board Or Motherboard
        Use: System Memory
        Error Correction Type: None
        Maximum Capacity: 128 GB
        Error Information Handle: 0x000D
        Number Of Devices: 4

Handle 0x000F, DMI type 19, 31 bytes
Memory Array Mapped Address
        Starting Address: 0x00000000000
        Ending Address: 0x00FFFFFFFFF
        Range Size: 64 GB
        Physical Array Handle: 0x000E
        Partition Width: 2

Handle 0x0010, DMI type 18, 23 bytes
32-bit Memory Error Information
        Type: OK
        Granularity: Unknown
        Operation: Unknown
        Vendor Syndrome: Unknown
        Memory Array Address: Unknown
        Device Address: Unknown
        Resolution: Unknown

Handle 0x0011, DMI type 17, 92 bytes
Memory Device
        Array Handle: 0x000E
        Error Information Handle: 0x0010
        Total Width: Unknown
        Data Width: Unknown
        Size: No Module Installed
        Form Factor: Unknown
        Set: None
        Locator: DIMM 0
        Bank Locator: P0 CHANNEL A
        Type: Unknown
        Type Detail: Unknown

Handle 0x0012, DMI type 18, 23 bytes
32-bit Memory Error Information
        Type: OK
        Granularity: Unknown
        Operation: Unknown
        Vendor Syndrome: Unknown
        Memory Array Address: Unknown
        Device Address: Unknown
        Resolution: Unknown

Handle 0x0013, DMI type 17, 92 bytes
Memory Device
        Array Handle: 0x000E
        Error Information Handle: 0x0012
        Total Width: 64 bits
        Data Width: 64 bits
        Size: 32 GB
        Form Factor: DIMM
        Set: None
        Locator: DIMM 1
        Bank Locator: P0 CHANNEL A
        Type: DDR5
        Type Detail: Synchronous Unbuffered (Unregistered)
        Speed: 4800 MT/s
        Manufacturer: Unknown
        Serial Number: 00000000
        Asset Tag: Not Specified
        Part Number: CMK64GX5M2B6000Z40
        Rank: 2
        Configured Memory Speed: 4800 MT/s
        Minimum Voltage: 1.1 V
        Maximum Voltage: 1.1 V
        Configured Voltage: 1.1 V
        Memory Technology: DRAM
        Memory Operating Mode Capability: Volatile memory
        Firmware Version: Unknown
        Module Manufacturer ID: Bank 3, Hex 0x9E
        Module Product ID: Unknown
        Memory Subsystem Controller Manufacturer ID: Unknown
        Memory Subsystem Controller Product ID: Unknown
        Non-Volatile Size: None
        Volatile Size: 32 GB
        Cache Size: None
        Logical Size: None

Handle 0x0014, DMI type 20, 35 bytes
Memory Device Mapped Address
        Starting Address: 0x00000000000
        Ending Address: 0x007FFFFFFFF
        Range Size: 32 GB
        Physical Device Handle: 0x0013
        Memory Array Mapped Address Handle: 0x000F
        Partition Row Position: Unknown
        Interleave Position: Unknown
        Interleaved Data Depth: Unknown

Handle 0x0015, DMI type 18, 23 bytes
32-bit Memory Error Information
        Type: OK
        Granularity: Unknown
        Operation: Unknown
        Vendor Syndrome: Unknown
        Memory Array Address: Unknown
        Device Address: Unknown
        Resolution: Unknown

Handle 0x0016, DMI type 17, 92 bytes
Memory Device
        Array Handle: 0x000E
        Error Information Handle: 0x0015
        Total Width: Unknown
        Data Width: Unknown
        Size: No Module Installed
        Form Factor: Unknown
        Set: None
        Locator: DIMM 0
        Bank Locator: P0 CHANNEL B
        Type: Unknown
        Type Detail: Unknown

Handle 0x0017, DMI type 18, 23 bytes
32-bit Memory Error Information
        Type: OK
        Granularity: Unknown
        Operation: Unknown
        Vendor Syndrome: Unknown
        Memory Array Address: Unknown
        Device Address: Unknown
        Resolution: Unknown

Handle 0x0018, DMI type 17, 92 bytes
Memory Device
        Array Handle: 0x000E
        Error Information Handle: 0x0017
        Total Width: 64 bits
        Data Width: 64 bits
        Size: 32 GB
        Form Factor: DIMM
        Set: None
        Locator: DIMM 1
        Bank Locator: P0 CHANNEL B
        Type: DDR5
        Type Detail: Synchronous Unbuffered (Unregistered)
        Speed: 4800 MT/s
        Manufacturer: Unknown
        Serial Number: 00000000
        Asset Tag: Not Specified
        Part Number: CMK64GX5M2B6000Z40
        Rank: 2
        Configured Memory Speed: 4800 MT/s
        Minimum Voltage: 1.1 V
        Maximum Voltage: 1.1 V
        Configured Voltage: 1.1 V
        Memory Technology: DRAM
        Memory Operating Mode Capability: Volatile memory
        Firmware Version: Unknown
        Module Manufacturer ID: Bank 3, Hex 0x9E
        Module Product ID: Unknown
        Memory Subsystem Controller Manufacturer ID: Unknown
        Memory Subsystem Controller Product ID: Unknown
        Non-Volatile Size: None
        Volatile Size: 32 GB
        Cache Size: None
        Logical Size: None

Handle 0x0019, DMI type 20, 35 bytes
Memory Device Mapped Address
        Starting Address: 0x00800000000
        Ending Address: 0x00FFFFFFFFF
        Range Size: 32 GB
        Physical Device Handle: 0x0018
        Memory Array Mapped Address Handle: 0x000F
        Partition Row Position: Unknown
        Interleave Position: Unknown
        Interleaved Data Depth: Unknown

Handle 0x001A, DMI type 40, 14 bytes
Additional Information 1
        Referenced Handle: 0x0000
        Referenced Offset: 0x00
        String: AGESA!V9
        Value: 0x00000000

Handle 0x001B, DMI type 8, 9 bytes
Port Connector Information
        Internal Reference Designator: J1500
        Internal Connector Type: None
        External Reference Designator: USB-C
        External Connector Type: Access Bus (USB)
        Port Type: USB

Handle 0x001C, DMI type 8, 9 bytes
Port Connector Information
        Internal Reference Designator: J1501
        Internal Connector Type: None
        External Reference Designator: USB-C
        External Connector Type: Access Bus (USB)
        Port Type: USB

Handle 0x001D, DMI type 8, 9 bytes
Port Connector Information
        Internal Reference Designator: J1502
        Internal Connector Type: None
        External Reference Designator: USB-C
        External Connector Type: Access Bus (USB)
        Port Type: USB

Handle 0x001E, DMI type 8, 9 bytes
Port Connector Information
        Internal Reference Designator: J1503
        Internal Connector Type: None
        External Reference Designator: USB 2.0
        External Connector Type: Access Bus (USB)
        Port Type: USB

Handle 0x001F, DMI type 8, 9 bytes
Port Connector Information
        Internal Reference Designator: J1504
        Internal Connector Type: None
        External Reference Designator: USB 3.2
        External Connector Type: Access Bus (USB)
        Port Type: USB

Handle 0x0020, DMI type 8, 9 bytes
Port Connector Information
        Internal Reference Designator: J1100
        Internal Connector Type: None
        External Reference Designator: Nova
        External Connector Type: None
        Port Type: Video Port

Handle 0x0021, DMI type 8, 9 bytes
Port Connector Information
        Internal Reference Designator: J2100
        Internal Connector Type: None
        External Reference Designator: Front Audio
        External Connector Type: Mini Jack (headphones)
        Port Type: Audio Port

Handle 0x0022, DMI type 8, 9 bytes
Port Connector Information
        Internal Reference Designator: J2101
        Internal Connector Type: None
        External Reference Designator: Audio Jack
        External Connector Type: Mini Jack (headphones)
        Port Type: Audio Port

Handle 0x0023, DMI type 8, 9 bytes
Port Connector Information
        Internal Reference Designator: J2102
        Internal Connector Type: None
        External Reference Designator: HD Audio HDR
        External Connector Type: Mini Jack (headphones)
        Port Type: Audio Port

Handle 0x0024, DMI type 9, 17 bytes
System Slot Information
        Designation: PCIE1
        Type: x8 PCI Express x8
        Current Usage: In Use
        Length: Short
        ID: 1
        Characteristics:
                3.3 V is provided
                PME signal is supported
        Bus Address: 0000:00:01.1

Handle 0x0025, DMI type 9, 17 bytes
System Slot Information
        Designation: J3502
        Type: x4 PCI Express x4
        Current Usage: In Use
        Length: Short
        ID: 0
        Characteristics:
                3.3 V is provided
                PME signal is supported
        Bus Address: 0000:00:01.2

Handle 0x0026, DMI type 9, 17 bytes
System Slot Information
        Designation: PCIE4
        Type: x4 PCI Express x4
        Current Usage: In Use
        Length: Short
        ID: 6
        Characteristics:
                3.3 V is provided
                PME signal is supported
        Bus Address: 0000:00:02.1

Handle 0x0027, DMI type 13, 22 bytes
BIOS Language Information
        Language Description Format: Long
        Installable Languages: 15
                en|US|iso8859-1
                zh|TW|unicode
                zh|CN|unicode
                ru|RU|iso8859-5
                de|DE|iso8859-1
                ja|JP|unicode
                ko|KR|unicode
                es|ES|iso8859-1
                fr|FR|iso8859-1
                it|IT|iso8859-1
                pt|PT|iso8859-1
                vi|VI|iso8859-1
                id|ID|iso8859-1
                tr|TR|iso8859-1
                pl|PL|iso8859-1
        Currently Installed Language: en|US|iso8859-1

Handle 0x0028, DMI type 127, 4 bytes
End Of Table

lspci -nnk

        Subsystem: Advanced Micro Devices, Inc. [AMD] Device [1022:14d8]
00:00.2 IOMMU [0806]: Advanced Micro Devices, Inc. [AMD] Device [1022:14d9]
        Subsystem: Gigabyte Technology Co., Ltd Device [1458:d000]
00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14db]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
        Kernel driver in use: pcieport
00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14db]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
        Kernel driver in use: pcieport
00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
00:02.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14db]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
        Kernel driver in use: pcieport
00:02.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14db]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
        Kernel driver in use: pcieport
00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14dd]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Device [1022:14dd]
        Kernel driver in use: pcieport
00:08.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14dd]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Device [1022:14dd]
        Kernel driver in use: pcieport
00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 71)
        Subsystem: Gigabyte Technology Co., Ltd Device [1458:5001]
        Kernel modules: i2c_piix4, sp5100_tco
00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
        Subsystem: Gigabyte Technology Co., Ltd Device [1458:5001]
00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e0]
00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e1]
00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e2]
00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e3]
        Kernel driver in use: k10temp
        Kernel modules: k10temp
00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e4]
00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e5]
00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e6]
00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e7]
01:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch [1002:1478] (rev c0)
        Kernel driver in use: pcieport
02:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch [1002:1479]
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch [1002:1479]
        Kernel driver in use: pcieport
03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 [Radeon RX 6800/6800 XT / 6900 XT] [1002:73bf] (rev c0)
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Radeon RX 6900 XT [1002:0e3a]
        Kernel driver in use: amdgpu
        Kernel modules: amdgpu
03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21/23 HDMI/DP Audio Controller [1002:ab28]
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21/23 HDMI/DP Audio Controller [1002:ab28]
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel
03:00.2 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:73a6]
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:73a6]
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci
03:00.3 Serial bus controller [0c80]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 USB [1002:73a4]
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:0408]
        Kernel driver in use: i2c-designware-pci
04:00.0 Non-Volatile memory controller [0108]: ADATA Technology Co., Ltd. XPG GAMMIX S50 NVMe SSD [1cc1:5350] (rev 03)
        Subsystem: ADATA Technology Co., Ltd. XPG GAMMIX S50 NVMe SSD [1cc1:5350]
        Kernel driver in use: nvme
        Kernel modules: nvme
05:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f4] (rev 01)
        Subsystem: ASMedia Technology Inc. Device [1b21:3328]
        Kernel driver in use: pcieport
06:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        Subsystem: Device [3328:3328]
        Kernel driver in use: pcieport
06:0c.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        Subsystem: ASMedia Technology Inc. Device [1b21:3328]
        Kernel driver in use: pcieport
06:0d.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        Subsystem: ASMedia Technology Inc. Device [1b21:3328]
        Kernel driver in use: pcieport
0b:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f4] (rev 01)
        Subsystem: ASMedia Technology Inc. Device [1b21:3328]
        Kernel driver in use: pcieport
0c:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        Subsystem: Device [3328:3328]
        Kernel driver in use: pcieport
0c:06.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        Subsystem: Device [3328:3328]
        Kernel driver in use: pcieport
0c:07.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        Subsystem: Device [3328:3328]
        Kernel driver in use: pcieport
0c:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        Subsystem: Device [3328:3328]
        Kernel driver in use: pcieport
0c:0c.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        Subsystem: ASMedia Technology Inc. Device [1b21:3328]
        Kernel driver in use: pcieport
0c:0d.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        Subsystem: ASMedia Technology Inc. Device [1b21:3328]
        Kernel driver in use: pcieport
0d:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8161] (rev 15)
        Subsystem: Realtek Semiconductor Co., Ltd. TP-Link TG-3468 v4.0 Gigabit PCI Express Network Adapter [10ec:8168]
        Kernel driver in use: r8169
        Kernel modules: r8169
0f:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I225-V [8086:15f3] (rev 01)
        Subsystem: Gigabyte Technology Co., Ltd Device [1458:e000]
        Kernel driver in use: igc
        Kernel modules: igc
10:00.0 Network controller [0280]: Intel Corporation Wi-Fi 6 AX210/AX211/AX411 160MHz [8086:2725] (rev 1a)
        Subsystem: Intel Corporation Wi-Fi 6 AX210 160MHz [8086:0024]
        Kernel driver in use: iwlwifi
        Kernel modules: iwlwifi
11:00.0 Non-Volatile memory controller [0108]: Kingston Technology Company, Inc. Device [2646:5013] (rev 01)
        Subsystem: Kingston Technology Company, Inc. Device [2646:5013]
        Kernel driver in use: nvme
        Kernel modules: nvme
12:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f7] (rev 01)
        Subsystem: ASMedia Technology Inc. Device [1b21:1142]
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci
13:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f6] (rev 01)
        Subsystem: ASMedia Technology Inc. Device [1b21:1062]
        Kernel driver in use: ahci
14:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f7] (rev 01)
        Subsystem: ASMedia Technology Inc. Device [1b21:1142]
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci
15:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f6] (rev 01)
        Subsystem: ASMedia Technology Inc. Device [1b21:1062]
        Kernel driver in use: ahci
16:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 [144d:a808]
        Subsystem: Samsung Electronics Co Ltd SSD 970 EVO [144d:a801]
        Kernel driver in use: nvme
        Kernel modules: nvme
17:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Raphael [1002:164e] (rev c9)
        Subsystem: Gigabyte Technology Co., Ltd Device [1458:d000]
        Kernel driver in use: amdgpu
        Kernel modules: amdgpu
17:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Rembrandt Radeon High Definition Audio Controller [1002:1640]
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Rembrandt Radeon High Definition Audio Controller [1002:1640]
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel
17:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] VanGogh PSP/CCP [1022:1649]
        Subsystem: Advanced Micro Devices, Inc. [AMD] VanGogh PSP/CCP [1022:1649]
        Kernel driver in use: ccp
        Kernel modules: ccp
17:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15b6]
        Subsystem: Gigabyte Technology Co., Ltd Device [1458:5007]
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci
17:00.4 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15b7]
        Subsystem: Gigabyte Technology Co., Ltd Device [1458:5007]
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci
18:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15b8]
        Subsystem: Gigabyte Technology Co., Ltd Device [1458:5007]
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci
1 Like
9 
GRUB_CMDLINE_LINUX_DEFAULT="quiet apparmor=1 security=apparmor udev.log_priority=3 amd_iommu=on iommu=pt hugepages=16384 vfio-pci.ids=1002:73bf,1002:ab28,1002:73a6,1002:73a4

I think you masked the wrong device

10

That is strange… I was masking the same IDs in my AM4 setup…
Do I need to mask the “Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Radeon RX 6900 XT [1002:0e3a]” too ?

11

I see the Kernel driver in use is still “amdgpu”. It should be “vfio-pci”

12

I know, it won’t unless I manage to update “mkinitcpio.conf” but as soon as I am doing so, I don’t have desktop, just console.

The issue is even before GRUB, as just after POST, it reverts to 6900XT, instead of iGPU. That is why I believe it is either BIOS setting or bug

13

Even after grub, you should be able to forcibly specify the efifb display device in the kernel parameters.

14

I tried

video=efifb:off

and also (as I had it before on AM4)

video=efifb:off gfxpayload=console gfxpayload=auto

Same behavior

15

See:
https://wiki.archlinux.org/title/kernel_mode_setting

Section 4.1.
You should be able use that to try and force your kernel to use the integrated card as it’s video device. Might break X, but use text mode and see if the VM behaves in that state. We’ll worry about X11 if/when we can get that far.

Another approach would be to just never connect the VM GPU to a physical display.

16

Tried that too, same behavior.
I can type my password blindly, and then it loads X11 and desktop loads on the iGPU screen

1 Like
17

No output until X11 login? Even without the quiet parameter?

That’s very odd. With only one available display output, You should see the kernel messages until the dm or Getty login screen. Something else must be abnormal with your boot configuration. Unfortunately I am at a loss as to what could be causing this.

18

Does disabling the CSM cause any change in your boot behavior?

19

It is disabled. If I enabled it, I am getting one long beep, two short and no POST

Settings I tried so far in BIOS:

Settings > IO Ports > Initial Display Output > IGD Video
Settings > IO Ports > Integrated Grpahics > Forced
Settings > IO Ports > UMA Modes > Auto/UMA Specified/UMA Auto (tried all)
Settings > AMD CBS > NBIO Common Options > GFX Configuration > IGPU Configuration > Auto/UMA_AUTO/UMA_SPECIFIED (tried all)
Settings > AMD CBS > NBIO Common Options > GFX Configuration > UMA Version > Auto/Legacy/Non-Legacy (tried all)
1 Like
20

Until the cause of this can be determined and addressed, I’m afraid there’s not much that can be done. Something’s seriously screwing with your video output configuration at bootup.