[SOLVED] Unable to isolate GPU for VFIO (workaround)

I read up on it and it seems from 6.1 onward you can use ReBar, but you don’t set it in the UEFI. There is a Reddit thread explaining the new feature. The article mentions support from UEFI is preferred, but given your setup outright refuses to work with it, I would let it disabled. For example I ran the checks from the article on my computer and my host already uses the biggest available bar sizes. So no need to touch anything for me. Everyone would need to check their own setup though.

P.S. I just checked and my graphics cards driver reports ReBar working, without me having done anything. ReBar is not activated in the UEFI and I have not touched the sysfs configuration. I figure you need to leave it disabled in the UEFI if the setting hinders the boot process, but you might check if it is already working and if not you can try to set the option manually as explained in the article via sysfs.

2 Likes

Just a heads up:
Gigabyte BIOS F10d, not only it didn’t fix the issue, but destroyed my bootloader, and made my boot partition act like read-only, so I have to re-install everything.

Finally, after many delays I got a reply from GB to the ticked I opened.

The reason why the screen display can be output through the graphics card is that during the boot menu loading, the display device is determined by the software program, not controlled by the BIOS.

It can be seen from the video that after the dual monitors are connected, the BIOS screen can be displayed on the iGPU normally, which means that the BIOS settings are normal.

The F12 boot menu will also be displayed on the iGPU normally, but if the external program is loaded, it has nothing to do with the BIOS.

This happens to Windows too, but they seem to blame Manjaro or OS in general, as I sent them a video of the problem.

So, if it is Manjaro to blame, they wont make any BIOS changes, therefore, how can I permanent fix it?

i had sooo many problems with manjaro that i now switched to nobara (a fedora fork from Glorious Eggroll). I did not yet get my VFIO up and running again due to time restrains.

but i can say that i upgraded my bios from F8>F9>F10a>F10b>F10c>F10d>F10 and now to F11a without any of the problems you describe.

Only thing i noticed is that after upgrade the UEFI defaults to windows regardless of the prioritys of the SSDs so i always have to make sure it does not boot the SSD with the WindowsVM.

Other than that setting Primary Display Out works fine for GRUB UEFI etc.

Fedora does seam to init all gpus and if dGPU is connected to a display it will be preferred. But there is a gihtub for a udev rule which you can use once its done to set primary display out in wayland.

PS: Nobara came with its own set of setup challenges but other than that it is much more stable and bug free than manjaro (which was the worst os i have ever used including windows vista )

Thank you for taking time to reply.
Although GB says it is a software issue, I am not buying that, they just think it is not a priority, therefore they won’t spend time to fix this.

Now, as I wrote before, I did test it with Windows too. As soon as I boot from the USB stick, Windows setup uses my dGPU to display setup process. Also I tried Kubuntu (Debian, not Arch based) and did the same with even more challenges, like failing to begin the installation, etc.).
Fedora too. I didn’t install it, but started the installation process, and goes straight to dGPU.

What baffles me is that I am excluding the dGPU IDs in the Kernel boot parameters, and still loads them… If that worked, as it did before with my 3900X and 5900X, it would be ok.

3900X and 5900x dont have IDGs … so they have to work differently.

try booting with displays only connected to the IDG via Motherboard and no display connected to the dGPU.

that way it should only initialize the IDG.

If you are lucky you can connect Displays to the dGPU after Boot and they work well :smiley:

With 3900X & 5900X I was using a XT5500 in PCIe-3, which was set into BIOS as initial GPU, like I do now with iGPU, and again, it worked perfect.

Also, I did try to boot without the cables on, and as soon as I connect them, VM doesn’t work.
I also tried to setup Manjaro, without 6900XT connected, set the IDs in kernel parameters and then connect it to PCIe. Didn’t work either.

The only way it works for me, is to boot up with the cable disconnected from the dGPU (tried to disconnect it from the monitor side too, didn’t work), start the VM, and once it on desktop, connect the cable.

Even if I try to reboot the VM, it fails the next time. If I want to reboot the VM, I have to never connect the cable, or if I do, restart my host without the cable too.

yep that is exactly what i ment ^ ^
the dGPU only inits with some thing connected.
when it does not init it will boot with the dGPU.

That it worked with a XT5500 is logical since both are dGPUs

somewhere in Linux is a thing that prioritizes dGPUs over IDGs

when the udev rule for wayland will land we may have a chance to counter that :smiley:

Just a small update.
Few people suggested that the problem is Manjaro. I wasn’t supporting that, but yesterday I had time to do some testing, so I tried:
OpenSUSE Tumbeweed, Kernel 6.5
Nobara KDE, Kernel 6.4

Both have the same behaviour and they don’t load X11/Wayland if the cable is connected to my VGA.

I’ve got some thoughts, after skimming this thread. You’re on an AM5 system with a 6900xt, right?

Have a look at this section of my guide, where it goes over setting driver preload priorities. Since you’re rocking AMD on both GPUs, you might be running into a problem on that front.

Thank you for your reply. I am not on 6900XT anymore, I replaced it with 4080 just to make sure I can blacklist the driver, without affecting the iGPU driver but I am having the same behaviour.

I will take a look on your guide though!

1 Like

Ahh, didn’t realize that. Glad you got it sorted!

Re-surfacing this one, as I am once again with an issue…

So, got a Dell monitor with 2 DP ports. Once was connected to my iGPU DP out and the other one on my dGPU (RTX 4080) on one of the DP outs.

With this setup, the trick I had to do is to keep the Dell monitor off until I see the boot menu from Manjaro. Then, I could switch on the monitor, and everything worked smoothly. Do removing cables, no lying under the desk, and most important, I could reboot my VMs as many times as I wanted, without the need to remove the cable.

However, Dell monitor proven to be unreliable, and I had 2 replacements in 9 months.
So, I requested a refund and got myself an Asus XG349C.
Unfortunately is does not have 2 DPs, but HDMI, DP and USB-C.

And now, I am having just the same issue, where I need to remove the cable in order to boot into Manjaro desktop with my iGPU.

Any ideas what is the actual issue here?
Does the DP Link stay on, when the monitor is off, so the POST process thinks the monitor is connected?
I tried removing the power plug from the monitor, and worked, therefore something is working even when the monitor is switched off…

Possible solution for Arch/Manjaro users, here: Trying to GPU-Passthrough on demand - #9 by lI_Simo_Hayha_Il

1 Like