How to apply ACS Override patch (Kubuntu 18.10 & Kernel 4.18.16)

Mirad · October 22, 2018, 10:52pm

Hey all, sorry if this is a stupid question.

I am extremely new to Linux (working in it for about a month) and am moving over from Windows 10. Ever since I switched, I’m falling more and more in love with it. I still have my windows partition as a back up, but I’m trying very hard to switch solely to Linux. The latest challenge I’m facing is being able to game. I’ve done the research and what I’m going to do a mix of native gaming, WINE gaming, and a VM with a GPU passthrough using VFIO. After a week and a half of work I’ve managed to figure out all problems with VFIO but now that I can create the VM I’m having troubles with my IOMMU groupings. My GPU and companion sound controller are in a group with 4 other devices. I’ve read about a ACS override patch (I’m running on a Xeon E3-1200 controller which I’ve read doesn’t support ACS) and it looks like its extremely hardware specific to me and the one Wendell made looks looks like its only for radeon cards? If I’m wrong about this, I would really appreciate it if someone could help me by providing me a link to a good ACS patch and how to apply it to my kernel

EDIT: changing to kernel 4.19 did not affect the problem at all. Could an earlier kernel help me?

Here are my IOMMU groups so that you can see my issue

PC Specs:

Motherboard: ASRock Z370 Killer SLI/ac

GPUs: GeForce GTX 780, GeForce GTX 1070

CPU: Intel Core i7-8700k @ 3.70GHz

RAM: 16 GB DDR4

OS: Kubuntu 18.10

Kernel: 4.18.16

TheCakeIsNaOH · October 23, 2018, 3:09am

With most distros, you have to download the kernel source code, insert the ACS patch, and compile+install the patched kernel.

Since you are running Ubuntu(or spin thereof), there are pre-compiled kernels with the acs patch available here-
https://queuecumber.gitlab.io/linux-acs-override/

Download the kernel and headers, and install them with sudo dpkg -i <file>.deb, obviously substituting the correct path for it. Put pcie_acs_override=downstream in /etc/default/grub in the line GRUB_CMDLINE_LINUX_DEFAULT=" ". Then run sudo update-gruband reboot.

Hopefully, then they will be separated.

If you want to learn to compile your own custom kernel, you can download the patch at the page above. Download the patch for the kernel version you will be compiling, the patch versions are very similar but with minor tweaks.

Mirad · October 23, 2018, 11:54pm

Thank you so much for the response! I got the kernel and my IOMMU groupings are now much more separated to the point where each GPU is in its own group. I had a small problem with the install but I looked at the error and it pointed me to a log where I just had to download a dependency (not sure why they had to print that to a log instead of just on the terminal, but whatever). However now I’m getting very low FPS after reinstalling my proprietary Nvidia driver (390.87) to the point where the entire system will gradually slow down to a halt if given a large work load (like actually creating the VM) and I have to reboot the system. Do you or anyone else have a fix for this?

EDIT: I tried it with the old kernel (that worked fine) and it looks like that is having the same problem now. My windows partition works great however, so it is not a hardware based issue and it is contained to linux

elixir77 · October 24, 2018, 2:04pm

I am having issues with the per-compiled acso kernels. They won’t install the proprietary nvidia drivers properly using my RTX 2080ti for the host.

When I try with my Vega 64 as the host, it seems to work but the RTX 2080ti does not get separated into the IOMMU groups properly. The VGA, Audio, USD, and Serial components of the RTX 2080ti all remain in the same IOMMU group. Every other device is split correctly in separate groups.

Is the ACS patch generic from the kernel dev team or are they written by the people applying them?

I would prefer to use elementary OS/Ubuntu for that stability (no rolling updates that could break things) and not having to deal with root permissions every time I want to do something like mounting drives (though installing gnome disks helps with that). However, I may try acso 4.19 or try to compile the kernel with the ACS patch file. If that fails, I’ll probably go back to ubuntu 16.04 or try some version of Arch.

TheCakeIsNaOH · October 24, 2018, 2:13pm

I would recomend starting a new thread with all the info and troubleshooting you have done for the nvidia driver.

You also could try the nvidia-410 driver from this PPA-
https://launchpad.net/~graphics-drivers/+archive/ubuntu/ppa

TheCakeIsNaOH · October 24, 2018, 2:25pm

It is a generic patch, not hardware specific.

Stuff like not installing properly is why proprietary drivers ideal on linux.

You did enable the patch in grub and reboot before checking the IOMMU groups? The reason for the patch is if you have thing(s) you do not want to passthrough in the same group as your GPU. So if it is the audio and stuff of the GPU, then just pass everything in the group through.

elixir77 · October 24, 2018, 3:04pm

Thanks for the quick reply, I will do that re the thread.

I am actually using the ppa. I’ve had so many issues over the year with nvidia’s *.run drivers that all I use now are community tested and provided drivers.

elixir77 · October 24, 2018, 3:20pm

You did enable the patch in grub and reboot before checking the IOMMU groups? The reason for the patch is if you have thing(s) you do not want to passthrough in the same group as your GPU. So if it is the audio and stuff of the GPU, then just pass everything in the group through.

Yes, I did check the IOMMU groups after rebooting and Vega 64 was split up into 4 groups but the nvidia card had all it’s components under the same group (iommu group 17). I tried running my VM but it kept saying 0000:04.00.00 in group 17 was not valid and I should check to make sure I had separated the groups.
I need to remove the Vega 64 and reinstall kvm and ovmf so the vfio-pci model takes hold of the card. As things stand right now the OS is putting priority on the AMD card even though Nvidia card is in the primary pcie slot. I had to remove it to last night to force the OS to use the nvidia card.

EDIT::
I seem to have broken my Asus Z87-Pro motherboard or at least got it stuck in the A2 IDE Detect error code so probably going to have to buy another motherboard if I can’t fix the A2 issue

Mirad · October 25, 2018, 1:38am

My system continues to slow down to an absolute freeze, however I think I may have found the cause. I found this trying to fix my packagekit daemon which has been crashing a lot lately. Look at the pastebin below for my terminal output (its too long to screenshot in 1 shot). If i can fix this I think I may be able to fix the freezing issue (which happens after about 10-20 minutes of basic use) since to me that seems like a graphics problem because cpu load is 2-10% and ram is at 1.5-2 gb at time of freezing.

https://pastebin.com/PpkHTpUa

Is there a way to be able to grab these dependencies? I have no important files on this partition (as it is only a week old) so would it be simpler to just reinstall kubuntu and start over?

Mirad · October 25, 2018, 1:43am

Ah! You were replying to me, not Elixir. I’ll start a new thread with the new info if the new graphics driver doesn’t help, thanks for the link to the ppa btw.

TheCakeIsNaOH · October 25, 2018, 2:49am

I would recomend removing everything nvidia so it switches back to nouveau after a reboot, then rebooting, update+upgrade, then reboot again and try to install the nvidia driver.

The stable nvidia driver and new kernel versions often do not get along. It is mostly on nvidia for not open-sourcing their driver, and partly on the kernel developers for constantly changing the kernel interfaces.

Mirad · October 26, 2018, 6:05am

I’ve done that and the graphics problem isnt fixed.

However, I’ve started a new thread for the new problem, thank you for your help with the vfio issue.

Here’s the new thread if you wanna take a look

elixir77 · November 13, 2018, 3:28am

Update…got my replacement motherboard. Asrock Z97X Killer. Hoped no ACS patching needed but was wrong.
Anyway, I download the acso patch then manually recompiled the kernel following GaryBoltWolf’s old post here

Play games in Windows on Linux! PCI passthrough quick guide Linux

Hello all! I am Gray from GrayWolfTech. I make videos on Linux and other interesting technology. Wendell has given me permission to post my written guides here for my videos. This is great for two reasons: First it helps grow this community and second my videos reach a wider audience to share what I have learned. I look forward to posting all my guides and videos here, and I am more than happy to answer any questions about the guide you may have. – Onward! – Now it’s time for one of the coolest and more advanced capabilities of Linux. KVM, or kernel virtual machines. We are going to go through step by step, how to setup PCI pass-through to a Windows virtual machine on a Debian host to play games! Thanks to Redhat, KVM can run virtual machines with almost bare metal performance and supports a wide variety of other neat features. The one we are going to focus on in this video is of course, PCI pass-through. The process of allowing a virtual machine full access to a PCI express graphics card for gaming, CAD, or 3D rendering. With this neat capability, you can run Linux as your host OS, and then pass your GPU (or one of your GPUs if you have multiple) to a virtual machine to play games. Let’s go over the requirements for this project: First and most important, you should backup any data you have on your PC. If you know what you are doing then you will not lose anything, but accidentally selecting the wrong drive when installing can lead you to be very sad when you figure out the drive you wiped had all your really good porn on it. Second, this process will take time. For someone who is an experienced Linux user it should take about 20 minutes (not counting time to install Debian or recompile the kernel). For someone who is new it might take longer. Make sure you allow yourself an entire afternoon or a good chunk of 4-5 hours where you could potentially not have a working PC. Third, follow the instructions I lay out in the video. I will answer questions down belo…

I found a script on github someone had written to apply the patch and update the kernel it worked but I forgot to set the downstream so didn’t think it worked even though it did. You can find the script here

gist.github.com

https://gist.github.com/mdPlusPlus/031ec2dac2295c9aaf1fc0b0e808e21a

compile_newest_stable_kernel_with_acso_patch.sh

#!/bin/bash

function install_dependencies() {
	echo "Installing dependencies..."
	sudo apt -qq update
	sudo apt -qq install curl git wget
	
	## Apparently this does not satisfy all package requirements
	#sudo apt -qq build-dep linux-image-generic
	sudo apt -qq install build-essential kernel-package fakeroot libncurses5-dev libssl-dev ccache

This file has been truncated. show original

Anyway I manually applied recompiled the kernel on 4.15.18 because I was having one of those forgetful moments. From comments on the github, it seems the issue I was having is a common one with the acso pre-compiled kernels. Using the script or manually compiling yourself fixes this driver issue.

My other issue with the AMD gpu always being made to primary GPU seems to be both motherboard and kernel settings related. I think I had passed the nvida card to vfio initially but didn’t change all the settings back. Then of course the kernel would boot the AMDGPU driver before the vfio

Anyway, figured that out when after getting the new motherboard and reinstalling everything. I had to force amdgpu to stop loading before vfio following wendell’s post here

Ubuntu 17.04 -- VFIO PCIe Passthrough & Kernel Update (4.14-rc1) Linux

DRAFT of the article for ubuntu passthrough setup from the live stream. Images are missing for right now. Any comments? A video is being edited that goes through the how-to step by step. Step 1 install ubuntu step 2 install ukuu sudo apt-add-repository -y ppa:teejee2008/ppa sudo apt-get update sudo apt-get install ukuu Use ukuu to install the latest kernel. In our case we want Kernel 4.14-rc1 because that is the latest kernel as of the time of this writing and this kernel is known to work on this hardware configuration, even though it is a pre-release kernel. This kernel, or later, is also important for AMD Ryzen or Threadripper users as it contains a number of updates for those platforms. The update utility will often fail the first time due to the cache folders not being created automatically. Just run it again and it will work the second time. Notice that as it updates the kernel, it also automatically runs update-grub and update-initramfs and that it complains about missing microcode. It isn’t clear if the microcode is required, but the microcode is non-free. It can be found on the internet, however. We must also add some kernel parameters and re-run the update commands above, but that can be done later. step 3 (optional, skip) find microcode and copy to /var/lib/firmware step 4 (doing the vfio stuff) We are ready to setup a virtual machine and pass through real graphics hardware to the virtual machine. To start, we need to make sure the host platform has been configured correctly: That hardware virtualization extensions are enabled, and that the IOMMU groupings are suitable. First, make sure that vt-d (or SVM on the amd platform) is enabled in the UEFI. This varies from platform to platform but typically Level1 Techs motherboard review cover where these options are and what the IOMMU groupings are for particular motherboards. Most Linux distros do not enable IOMMU by default. You will need to update your grub bootloader config to support IOMMU. F…

Now to figure out how to get the Vega 64 to stop the issue it has shutting down when you rebooting or power off the VM, so I don’t have to keep rebooting the entire system or leave the VM permanently on once booted