Having trouble placing second GPU in it's own IOMMU group

Using Pop_OS 19.04 with kernel 5.0.0-13-generic with acs override enabled (or at least I think it is). I’m setting up a Windows 10 VM with gpu passthrough basically just so I can play Destiny 2. I have used this script to compile and patch my kernel. I have also added the options

intel_iommu=on
pcie_acs_override=downstream

to my /boot/efi/loader/entries/Pop_OS.conf file. I’ve created a /etc/modeprobe.d/local.conf file with the modalias entries for my gpu/audio and added options

vfio-pci ids=10xx:xxxx,10xx:xxxx
vfio-pci disable_vga=1
echo “options kvm ignore_msrs=1” >> /etc/modprobe.d/kvm.conf

and added vfio modules to my initramfs-tool/modules file

vfio vfio_iommu_type1 vfio_pci vfio_virqfd vhost-net

and then updated my initramfs.

All of this was pulled from 2 or more different tutorials as I tried to work my way through this gpu passthrough. However my second GPU is being put in the same IOMMU group as my first even after acs override is supposed to be enabled. Is there a different way to enable acs override in systemd-boot compared to grub? I was under the impression that all the options worked the same.

Sorry for the jumbled topic and thanks to any who have taken time to read this.

you sure you’re using the patched kernel? run uname -a

also make sure modconf is in the hooks section of the initramfs config

there’s also a ppa to automate that first bit

https://launchpad.net/~queuecumber/+archive/ubuntu/acso

1 Like

uname -a outputs
Linux pop-os 5.0.0-13-generic #14-Ubuntu SMP Mon Apr 15 14:59:14 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Sorry I’m a bit new where exactly is the hooks section of the initramfs config?

After creating the patched kernel, you need to install the new patched kernel and use it. Then update your initramfs (initial ram file system, the first thing that runs on the system with info of what kernel and drivers modules to initialize the system with).

what @kriss120 said, that or use the PPA to automate the install process.

The ppa seems to be failing for me {404 not found}. I actually manually downloaded and installed from queuecumber’s page following this person’s instructions. That, however, did not seem to help. Which is why I tried that script I listed above. Is there something I’m missing?

that’s the same guy that (apparently used to) run the ppa

still have to install the kernel after compilation, I think that script just compiles it for you

If you compile the patched kernel and install it over the top of the existing system it should fix your issue. You may not be using the patched kernel

I downloaded and installed the 5.1 mainline header and image from here using sudo dpkg -i <file>.deb and then re-added intel_iommu=on and pcie_acs_override=downstream to /boot/efi/loader/entries/Pop_OS-current.conf and rebooted. Whenever initramfs updates it wipes my options from the pop_os config file. I don’t know if that is normal or not. Any who uname -a still gives me Linux pop-os 5.0.0-13-generic #14-Ubuntu SMP Mon Apr 15 14:59:14 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux. My 2 GPUs are still in IOMMU group 1.

Is the ACS patch not taking, or am I putting the wrong option in my config file to enable the override?

Boot, hit e and see what the listed vars are to make sure they’re present from the edit you made to the loader as well.

so your system isn’t booting with the newly installed kernel despite it being installed, either due to a header mismatch or some other misconfiguration. I think the most popular route for kernel management on ubuntu based distros is ukuu, so maybe try that.

don’t use xbuntu’s often enough to tell you specifically what it entails on that platform but there’s a gui tool that apparently works (ukuu)

Grabbed ukuu and it seems that the patched kernel is installed but the pc is using the old kernel for some reason. How do I got about changing this? I think it wouldn’t be so complicated if I weren’t using systemd-boot…

EDIT: KERNELSTUB IS THE POPOS SANCTIONED WAY TO MANAGE KERNELS

does ukuu not let you choose a kernel from the list? maybe it was only designed for use with grub.

paste the contents of your /boot/loader/entries/popOS.conf file, you might not be pointing to the right kernel and initramfs in the linux and initrd lines. should look like (for example)

linux   /ubuntu-whatever-acso-patched
initrd  /intel-ucode.img
initrd  /initramfs-linux-acso-patched.img

instead of pointing to the default ones

these are placeholder names as I have no actual idea what names the package you used installs them with. check your ESP for kernel EFI stubs

you can also delete the timeout line and try to manually select the kernel from the list on boot in the unlikely event that the package or ukuu generated a loader for it.

also you may need to generate an efi stub if the .deb didn’t install one for the new kernel. this apparently can be achieved using kernelstub on popOS

note i pulled this out of systemd-boot documentation and dont use it personally

gotta love not-invented-here syndrome huh? custom tools for a bootloader no other buntu distro uses for a system that doesn’t work with the automatic tools made for ubuntu because their kernel compilation and installation process is more annoying than other distros.

Layers of solutions to problems the solvers caused in the first place.

I finally got it. I found reddit post where someone had to manually make the efi entries and redirect everything to the new kernel. After a little copy and paste I got it rolling. Now on to new problems with no gpu output lol. Thanks for all of your help! I truly do appreciate it.