Enabling ACS override in Pop!_OS 20.04

Operating Systems & Open Source Virtualization
1

Hello all,

I have been using a trial of UnRAID 6.8.3 to test out my hardware and its ability to run VMs for my purposes. I have things running to my satisfaction, I now want to “roll my own” with Pop!_OS as the host. I was following this tutorial and have become stuck at the point of enabling ACS override (I know from my UnRAID setup that I need the ACS override to get the two graphics cards into their own IOMMU groups - I am using Xeon E-2286G with its consumer-grade ACS :anguished:).

I saw in the /boot directory of my PopOS system that it is using linux 5.4.0-7634-generic, so I went here to download what I believe are the corresponding .deb packages with ACS override. After installing all 3 packages, I used sudo kernelstub with the -k and -i options to specify the path to the kernel and initramfs image, respectively and added the pcie_acs_override=downstream,multifunction option (which is what I needed in the syslinux.cfg file of my UnRAID setup).

I rebooted…and now my PopOS won’t start at all. It hangs after the UEFI boot logo with a blank screen and keyboard does not respond (shift lock does not toggle the LED). The machine would not respond to ping either so I thought I was hosed. But then I remembered I could boot back into UnRAID and then poke around my PopOS drive. Going to the /boot directory, I have:

-rw------- 1 root root  4737705 May 28 16:13 System.map-5.4.0-7634-generic
-rw-r--r-- 1 root root  4481025 Nov 25  2019 System.map-5.4.0-acso
-rw-r--r-- 1 root root   237782 May 28 16:13 config-5.4.0-7634-generic
-rw-r--r-- 1 root root   236234 Nov 25  2019 config-5.4.0-acso
drwxr-xr-x 2 root root     4096 Jun 27 19:23 efi/
lrwxrwxrwx 1 root root       21 Jun 27 21:31 initrd.img -> initrd.img-5.4.0-acso
-rw-r--r-- 1 root root 84438483 Jun 27 19:32 initrd.img-5.4.0-7634-generic
-rw-r--r-- 1 root root 82894159 Jun 28 08:45 initrd.img-5.4.0-acso
lrwxrwxrwx 1 root root       29 Jun  1 07:40 initrd.img.old -> initrd.img-5.4.0-7634-generic
lrwxrwxrwx 1 root root       18 Jun 27 21:32 vmlinuz -> vmlinuz-5.4.0-acso
-rw------- 1 root root 11660160 May 28 16:13 vmlinuz-5.4.0-7634-generic
-rw-r--r-- 1 root root 11286912 Nov 25  2019 vmlinuz-5.4.0-acso
lrwxrwxrwx 1 root root       26 Jun  1 07:40 vmlinuz.old -> vmlinuz-5.4.0-7634-generic

I then mounted the ESP and went into the EFI/Pop_OS folder and found the following:

-rwxrwxrwx 1 root root      225 Jun 28 13:54 cmdline*
-rwxrwxrwx 1 root root 82894159 Jun 28 13:54 initrd.img*
-rwxrwxrwx 1 root root 82894159 Jun 28 13:54 initrd.img-previous*
-rwxrwxrwx 1 root root 11286912 Jun 28 13:54 vmlinuz-previous.efi*
-rwxrwxrwx 1 root root 11660160 Jun 28 13:54 vmlinuz.efi*

So this puzzles me because for both initrd.img and initrd.img-previous, it appears to be using the ACSO version (given the file size), but for the vmlinuz.efi, it appears to be using the original version, whereas vmlinuz-previous.efi is the ACSO version.

I am totally clueless regarding all this bootloader stuff (that’s why I started by trying it all out on UnRAID), but did I do something wrong in trying to enable the kernel with the ACS override patch? I am not even sure if switching kernels is as plug-and-play as I am assuming above. I know others recommend GRUB2, but I would need some guidance on switching over to that as well. Thanks in advance for any advice.

2

Have you updated the initramfs? sudo update-initramfs -u

Also, this might help-

3

Thanks for the reply, sorry I did not get back sooner, as it took time to do some research and poke around the ESP further to understand update-initramfs as you had suggested. I am not totally sure how I ended up with this configuration (I still don’t understand how kernelstub and update-initramfs work in conjunction to setup the ESP), but now the entries in the bootloader are such that “Pop_OS-current” points to the generic kernel, and “Pop_OS-oldkern” points to the ACSO. I added a “timeout 3” line to my loader.conf file so that I can have the opportunity choose the “oldkern” option before it defaults to “current”.

I know this is not ideal as I would like the default boot choice to be the ACSO kernel, but I just wanted to move forward to the next step in enabling GPU passthrough. I have the ACS override enabled now and all my devices are in separate IOMMU groups, so I continued on, but ran into other issues, for which I will start a separate thread.

4

I think you change the default parameter in esp/loader/loader.conf Or sudo bootctl set-default "boot-entry-name-here"

5

Hi Footfang,

I was able to get the ACS override patch working in Pop!_OS 20.04 using a script I found on GitHub, but it seems I am not allowed to post links. Should I just zip it and upload it?

With that said, I am new to Linux and still have not been able to get video output from the GPU I am passing through, so I can’t say my setup is 100% working yet.

6

It looks like I can post links now. This is the script I used.

7

I came across this thread looking to see if anyone noticed if the ACS flag is enabled in Pop OS kernel by default.

I installed their repos in Ubuntu just to get some extra packages initially, then I noticed their kernel works better than Ubuntu’s generic on my Dell Precision 7730 laptop, so I did an “apt full-upgrade” and did away with Ubuntu’s snapd gnome-shell

cat /etc/lsb-release && uname -a
DISTRIB_ID=Pop
DISTRIB_RELEASE=20.10
DISTRIB_CODENAME=groovy
DISTRIB_DESCRIPTION="Pop!_OS 20.10"
Linux ChubbyChipmunk 5.8.0-7625-generic #26~1603126178~20.10~210fe73 SMP Mon Oct 19 18:43:05 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

As a fluke, I just noticed PCIe passthrough has gotten a lot easier. My Dell has 3 devices in IOMMU group 6:

IOMMU6d:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. RTS5260 PCI Express Card Reader [10ec:5260] (rev 01)
IOMMU6e:00.0 Network controller [0280]: Intel Corporation Wireless-AC 9260 [8086:2526] (rev 29)
IOMMU6f:00.0 Non-Volatile memory controller [0108]: SK hynix PC300 NVMe Solid State Drive 1TB [1c5c:1285]

I would think having 3 completely unrelated devices in group 6 would make it hard to pass one device through, but no!

I added vfio_pci to grub + sd boot a while ago but I haven’t even really used it.

Opened up virt-manager and a VM, configured it to use PCI 000:6e:00.0. Threw my WIFI into a CentOS 8 Stream VM like it was nothing.

I use both bootloaders since it’s a zfs install, which can be perilous. Here’s my kernel flags:

(systemd-boot kernel mgmt, this script is in /etc/kernel/postinst.d/zz-update-systemd-boot and also an ln -s to /etc/initramfs/post-update.d/zz-update-systemd-boot)

ROOTFLAGS="init_on_alloc=0 intel_iommu=on i915.enable_fbc=1 i915.enable_guc_submission=1 i915.enable_guc=2"

Note: You can get the full version of that script from GitHub here - https://gist.githubusercontent.com/rflbianco/9f6d0f24af4d5dd2f69cd99d879be5f3/raw/cce6cff45681773c7e93a179bf6816963b3980d2/zz-update-systemd-boot.template

Grub (/etc/default/grub):

GRUB_CMDLINE_LINUX_DEFAULT="init_on_alloc=0 intel_iommu=on i915.enable_fbc=1 i915.enable_guc_submission=1 i915.enable_guc=2"

You can see I literally only have one flag related to kvm and it’s not even vfio.

Only one thing KVM related in /etc/modprobe.d, too:

cat /etc/modprobe.d/kvm.conf

options kvm_intel nested=1

I thought I’d share this other .conf file in case it has anything to do with how easy this was (it probably doesn’t):

cat /etc/modprobe.d/iwlwifi.conf 

# /etc/modprobe.d/iwlwifi.conf
# iwlwifi will dyamically load either iwldvm or iwlmvm depending on the
# microcode file installed on the system.  When removing iwlwifi, first
# remove the iwl?vm module and then iwlwifi.
remove iwlwifi \
(/sbin/lsmod | grep -o -e ^iwlmvm -e ^iwldvm -e ^iwlwifi | xargs /sbin/rmmod) \
&& /sbin/modprobe -r mac80211

Anywho, I have no idea why this was so easy, but leads me to believe that Pop OS kernel might have ACS flag enabled by default. @TheCakeIsNaOH if you can shed any light on the subject I’d love to know!

I mean, I don’t know any reason why it SHOULDN’T be enabled by default, it doesn’t hurt anything, right?