[SOLVED] Unable to isolate GPU for VFIO (workaround)

With the update from kernel 5.14 to 5.15 my system had the problem that KDE crashed, you don’t get a login, softdep amdgpu fixed this issue.

KDE.
But as I said before, the same behavior exists with Windows.
I also tried Kubuntu (KDE) and Endeavour (KDE) and wasn’t even able to start installation.

I would need to know if you use Wayland or X11 still.

The intention of this question is that Linux wants to use the graphics device it initilized first to output video. However at the point where your operating system is loaded, if it still loads, it would be a possibility to add a service that uses a program like xrandr (X11) to change the video output to a different connection. Maybe you will loose video out during the boot process but the moment your desktop environment loads you would switch the output device. I do not know however if Linux continues the boot process the moment it looses its primary output. This would be the only other workaround that comes to mind.

The only option to make sure is to place the 6900 in one of the lower PCIe slots and a GPU for your host above it. With the instruction you followed so far I am pretty sure this would give you are working setup.

By the way I remember, with my X570 Aorus Pro and X570 Aorus Master, I was not allowed to have a monitor connected to the passthrough GPU in the upper most slot for this to work. I needed to boot and then connect a monitor to it.

with Majaro and KDE he is using wayland

No, I am using X11. Tried Wayland in the past but had issues, don’t remember exactly, and didn’t try again since

No problems with Aorus Master X570. I just had to set VGA Init PCIe3 in BIOS and below settings in kernel params:

video=efifb:off gfxpayload=console gfxpayload=auto

well Wayland works fine for me, let’s try this, boot your system and login via strg+alt F4 if you don’t get the login screen, then use this command to see which GPU is used

lspci -vnnn | perl -lne 'print if /^\d+\:.+(\[\S+\:\S+\])/' | grep VGA

We could give my suggestion a try. Place the following file in /etc/systemd/system/xrandr.service. Obviously you need to change the output to the port you want to have video on.

[Unit]
Description=Change primary video device after boot.

[Service]
Type=oneshot
ExecStart=/bin/bash -c "xrandr --output HDMI-1 --mode 1920x1080"

[Install]
WantedBy=graphical.target

After that you run the following command.

sudo systemctl enable xrandr.service

P.S. I am not sure how to let this service start after KDE at the moment. This would be a requirement tough.

This is the last idea I have to solve this, because from what I am seeing it seems the option in the UEFI to select the primary video output does not work for you.

My AM4 was working with 6.2 with no issues, however I gave it a try.

cat /etc/mkinitcpio.conf

MODULES="crc32c vfio_pci vfio vfio_iommu_type1"

BINARIES=()

FILES="/usr/local/bin/vfio-pci-override.sh"

HOOKS="base udev autodetect modconf block keyboard keymap consolefont plymouth resume filesystems"
...

cat /etc/modprobe.d/vfio.conf

install vfio-pci /usr/local/bin/vfio-pci-override.sh
softdep amdgpu pre: vfio vfio_pci vfio-pci
softdep xhci_pci pre: vfio vfio_pci vfio-pci
softdep pcieport pre: vfio vfio_pci vfio-pci

cat /usr/local/bin/vfio-pci-override.sh

#!/bin/sh

DEVS="0000:03:00.0 0000:03:00.1"

if [ ! -z "$(ls -A /sys/class/iommu)" ]; then
    for DEV in $DEVS; do
        echo "vfio-pci" > /sys/bus/pci/devices/$DEV/driver_override
    done
fi

modprobe -i vfio-pci

ls -l /usr/local/bin/vfio-pci-override.sh

-rwxr-xr-x 1 root root 220 Apr 28 13:10 /usr/local/bin/vfio-pci-override.sh

Same issue.

For now, I will implement the workaround of not connecting my 27" on 6900@DP and will connect it only when I need to run the VM.

Workaround, but it works.

I am still waiting for GB to respond to my ticket though, which was logged as a bug and nobody ever replied.

why only 2 devices, my GPU has four

1 Like

What about the error 43 into Windows?
My “old” settings don’t work…

As I mention in previous posts, Linux kernel 6.x is broken.
Try to run modprobe -i vfio-pci on your console directly. For me, it freezes my monitor every single time.
If I boot it with linux 5.19, no issue at all.

1 Like

sorry i only have bad experiences with MSI
I ts why i only use ASUS and Gigabyte.
MSI UEFIs are horrible even would go rather with ASROCK XD
hence Gigabyte and MSI is not comparable.

I advice you to use the Aorus Master with Ubtuntu or Fedora (Nobara).

The Nobara W38 release apparently is around the corner :slight_smile:
It even has several Gamingperformance Kernel Patches build in

ill take a look if i can dump my uefi settings somewhere and get you a copy :slight_smile:

EDIT1:
maybe this successsory helps in the meantime:

EDIT2: I found the save profile function but in contrast to asus the file is not in plain text i have no clue how to open / read it :confused:

1 Like

I am trying to copy his XML, at least most parts, but it doesn’t work for me.
Either I am getting the infamous error 43, it is stuck during boot at “display not initialized (yet)” message

[SOLVED]
I had to disable "Resizable BAR support " in BIOS to work.

Thank you all for your time and efforts, especially @H-i-v-e , @DS_DV and @Janos !

2 Likes

Oh fudge, I could have told you that. I did not cross my mind unfortunately. Glad it works now!

2 Likes

Doesn’t disabling rebar come with performance degradation?

Once I do my benchmarks, I will be able to answer that.
However, Arch Wiki (10.4 & 10.24) has some settings to define BAR size, but they didn’t work for me.

glad to see that you got it working now :smiley:
new platforms are always tricky. i wonder why you had to disable rebar.
That one i only had to disable on AM4 o.o

1 Like

Maybe it is a bug of RX 6900 XT
I can passthrough RTX 4070 Ti with resizable bar enabled on my MSI x670e Carbon just fine.