[SOLVED] Unable to isolate GPU for VFIO (workaround)

This is my goal. But I cannot unbind my 6900XT from Linux…
lspci -nnv displays it is using the amdgpu driver, not the vfio-pci and as I wrote before, if I add vfio_pci in mkinitcpio.conf under MODULES, I boot into a black screen.

1 Like

On boot menu, is there any option to boot with Linux Kernel 5.XX (such as 5.19)?
I do find some VFIO bugs regarding Kernel 6.x. Have a try if there is an option.

No, it is a fresh install and I would have to manually install any 5.x Kernel. However, there is no point to, as they do not have optimizations for 7xx0X3D CPUs.

So, what I did is, I added “vfio_pci” in my kminitcpio.conf MODULES session and booted up.

This is what I see:


It is stuck there and doesn’t load Desktop environment.
I see the same thing, on both iGPU and 6900XT.

Pressing ctrl+alt+F2, I can open a terminal
If I type: lspci --nnv I am getting below:


This is 6900XT and it is using vfio_pci driver, as it should.

Same for HDMI sound and the USB controller.

However, for the Serial Controller (what is that???) id doesn’t load vfio_pci drivers, although I am setting the ID in vfiopci.conf

Last but not least, running dmesg, I noticed few thing in the log:


Also, device 17:00.x is the iGPU, and I am not sure it is loaded as it should.

1 Like

On Linux kernel 6.2, my 7950x igpu is stuck with “colour dummy device 80x25” as soon as I “virsh start”. Also, lspci -nnk returns empty driver for my dgpu (RTX 4070 Ti). I remember it should be vfio-pci, but it is empty.
I haven’t dug the issue deep. I don’t have this issue with Kernel 5.19.
One guess, vfio_pci module is not loaded for some reason. I just did quick search online. You may try add “vfio_pci” in /etc/modules-load.d/modules.conf. Don’t forget to “update-initramfs -u” afterwards.
I will try this once I get back from work.

1 Like

Tried to “startx” while in terminal (after enabling vfio_pci) and I am getting this:

Maybe this helps (not me, someone with better knowledge…)

if it does not show vfio-pci as kernel module in use than you are missing a part.

take a look at the first guide i posted from heikos blog.
It is a step by step and really in depth. If you follow it it will work :slight_smile:

Maybe also watch a video tutorial from folks like SomeOrdenaryGamer on vfio setups.

I think you are missing some basic steps here :wink:

1 Like

Check out this thread.

https://forum.level1techs.com/t/linux-kernel-6-seems-to-be-incompatible-with-the-vfio-pci-module-needed-for-pci-passthrough/190039

cannot confirm :slight_smile:
(you can also see that the os is running on the iGPU all with latest Manjaro)

1 Like

Could you try “linux 6.1.25-1”?
I don’t know “linux 6.2.12-1”.
I know “linux 6.2.0-20” doesn’t work for me. I am on Ubuntu, though.

This is the guide I always used, as my first try to setup a VM was on an identical setup with his.

Any idea why

sudo hwinfo --framebuffer

returns nothing?

as you can see in the top right window i don’t have that kernel as an option.
only newer one is an RC which i never use use for stability reasons

maybe when i switch to nobara - but i hope not ^^

sudo hwinfo --framebuffer

mine also doesn’t :frowning: but i am no linux expert :confused:

I am this close on giving up and return to Windows after 6 years… I’m really fed up with this. Already replaced a mobo and CPU, cannot all 4 be faulty.

Been following every advice, tutorial, guide I found, ending up in the same wall. I have re-install my system 3 times, and nothing has changed.

Now, after following another guide, I ended up here, and although I am reverting the changes I am not able to boot :face_with_symbols_over_mouth::face_with_symbols_over_mouth::face_with_symbols_over_mouth:

Yep, most guides from internet are obsolete, and somewhat over complicated.
The most important settings are these two boot parameters: amd_iommu=on and vfio-pci.ids=xxxxxxx
Just installing qemu daemon (e.g. qemu-kvm libvirt-daemon-system) and virt-manager package, nothing more, is enough to make it work.
For me, it is the Kernel issue. I don’t have issue on kernel 5.19 from the boot menu. Probably, they have already fixed that in the latest kernel.

2 Likes

When I am using this, I am getting error on the log, that ‘on’ is not a valid option…
The correct one is ‘force_enable’ or ‘on’ is by default

1 Like

You have enabled iommu and amd SVM in the bios, right?

What you need is that bit from other posters command line for Linux boot:

rd.driver.pre=vfio-pci

Like so

GRUB_CMDLINE_LINUX_DEFAULT="apparmor=1 security=apparmor udev.log_priority=3 amd_iommu=on iommu=pt hugepages=16384 rd.driver.pre=vfio-pci

(adding video=efifb:off is not a bad idea tbh)

Then you just toss that vfio-options line into modprobe.d:

/etc/modprobe.d/vfio.sh or something:

options vfio-pci ids=1002:73bf,1002:ab28,1002:0408,1002:73a6

Edit: I’ve got that also on /etc/modules (with loading of vfio-pci and other stuff) but I’m not sure which one makes it tick :smiley:
Edit 2: ^^^ reading the net indicated options shouldn’t work but one should load modules there, stuff like the vfio_pci and others.

Appreciate it. Will try that.

Yeah, just remember to do the mkinitcpio, update-grub and/or update-initramfs dance depending on your distro, so easy to forget

Yes.

Tried all these, don’t help.