Patch NPT on Ryzen for Better Performance | Level One Techs

[wendell]

Are you also the one that did the thing with “laptop gsync panels?” if so I hexedited the driver and was able to confirm that. They shut that down so fast I’m still reeling over it.

PM issue fixed.

[gnif]

Thanks :). And no, that one wasn’t me. PM sent.

[SgtAwesomesauce]

No, this is the first time I have been involved to this level with something of this nature. I’d appreciate any advice available on this.

I’m not sure how this works, Wendell might know better.

The fist time I looked into PCI passthrough was about 10 years ago on Xen and very broken, I gave up on it quickly when I found that my new motherboard had a broken IOMMU implementation.

I had the same experience when I realized my 3770k didn’t support IOMMU. Very sad.

That would be great, people have already been generous with donations for the NPT fix already, it really took me by surprise how much interest there is in this.

Yeah, everyone in my department is running Linux with a windows VM and a 480 or a 580 passed through for proprietary windows stuff and games. When the NPT patch hit patchwork, one of my underlings came running into my office with a huge smile on. You’re a hero in our office.

[Marf]

Wendel! You are my Hero again!

I compiled a patched Linux kernel just yet! Thanks alot for the guide! I never did it before (because there was no reason )

Tested yet 2 Benchmarks: Unigine Vally: DX11 (high preset, AA off) and Resident Evil 6 Bench Tool: DX9c (all High, AA off)
both on a passthroughed Sapphire RX 560 oc to 1434 MHz CPU and 2GHz VRAM.

Before the Patch:
Unigine ~52 FPS; 2170 Points (min FPS 18,5 / max FPS 103,8)
RE6 BT ~3700 Points

After the Patch:
Unigine ~ 58,4; 2443 Points (min FPS 28,0 / max FPS 107,2)
RE6 BT ~9600 Points

DirectX9c Games got a huuuuugh boost! DX11 Games where always playable but slightly choppy.

Now its veeeery smooth on my System.
Btw. I’m rocking the following System:

Ryzen 5 1600x OC to 3.925Ghz
RAM: Corsair Vengeance 3200 @ 3200MHz of course
Board: Asus Crosshair 6 Hero BIOS 17.01
Host Disk: Crucial M500 m2 SSD
Guest Disk: QCow2 Image
GPU: Sapphire RX 560 Pulse 4GB + Arctic Accelero Xtreme IV 280x Custom Cooler

Im quite happy with the results

Mit freundlichen Grüßen
Marfi

Update 2017-11-02:
I have to add: in my case a crucial package was not installed on my Ubuntu to do the Kernel compile. Please take care the package libssl-dev is installed via apt or synpatic package-manager.

[Marf]

Hi all,

I started yesterday a larger benchmark session with a wide varity of DX11, DX9c, DX12 and on Vulkan API Games. I want to compair NPT=1 vs NPT=0 vs Native Windows 10 performance and post my resultes (probably in a new Thread…). I hope some one find it usefull or “entertaining”
Here the list of Games/Benchmarks that i wanne to use (i.g. started to use):

Benchmarks

• Unigine Vally (DX11)
• Unigine Heaven (DX11)
• Unigine Superposition (DX11)
• Resident Evil 6 Bechmark Tool (DX9c)
• Resident Evil 5 Benchmark 1 (DX9c)
• Resident Evil 5 Benchmark 2 (DX9c)
• Tomb Raider
• Rise of Tomb Raider (DX12)
• Steam VR Performance Test
• Ashes of the Singularity: Escalation (DX12 and Vulkan)

Now I will not use my RX 560 card for the Passthrough, instead I pass my XFX R9 280x thru my VM. On earlier tests I noticed around 30% performance loss compaired to native use. My RX 560 “only” lost around 10% Performance with the buggy Nested Page Tables (what was quite intresting).

I think I can post the NPT=1 numbers this evening (German Time)…

Viele Grüße
Marf

Edit 2017-11-03:
As promised, I created just yet a new Thread where I will post my benchmark numbers. Please understand… WIP …
You will find the thread here: GPU Passthrough Performance Numbers: Ryzen NPT Patch vs Buggy NPT vs Native Windows

[mrjakesk8]

An update for anyone watching: I made 2 concurrent mistakes while testing the patch, causing me much confusion for an couple of hours. I both failed to apply the patch correctly (first time using a custom kernel) and failed to properly re-enable NPT. This explains why I was experiencing less-than-ideal performance. After remedying these 2 errors performance improved hugely, and I am (more or less ) officially done tinkering. This passthrough thing is now final the solution to all my woes! Benchmarks in this thread: GPU Passthrough Performance Numbers: Ryzen NPT Patch vs Buggy NPT vs Native Windows

[dailan]

Having trouble applying the patch on fedora 27 (trying kernel 4.13.11-301), and I’m out of ideas for on how to fix it. I’m able to compile the kernel on its own, or even just with a working Aur-acs patch. However adding the NTP patch on top of that causes rpmbuild to fail before I even start compiling.

Here are the commands I use to reproduce this

fedpkg clone -a kernel
git checkout -b my_kernel origin/f27
sudo dnf builddep kernel.spec
./scripts/newpatch.sh Aur-acs.patch
./scripts/newpatch.sh ryzen.patch
make release
sudo fedpkg local

pulled the patch from patchwork kernel, and I applied the aur-acs fixes (which compiled just fine on its own). I don’t think its required, but I installed my matching kernel-devel package just in case. (also util-linux)
I’m going to assume I’m missing something obvious, but if you need any more information I can provide it. Sorry if this is an inappropriate thread for this, at this point.

[mrjakesk8]

I’m sorry I can’t be of more help - I found this stage challenging too. In the end I downloaded the kernel from kernel.org, applied the patch and built it using variations on Wendell’s commands with help from this page: https://fedoraproject.org/wiki/Building_a_custom_kernel

Perhaps it’s a missing dependency issue? I fear any advice of mine beyond here is likely to cause more harm than good. I recommend reading that Fedora wiki thoroughly though

[dailan]

OK, I think I got it working. I had to use another method to apply the patch (We can pretend I used the “cat | patch” method as described in the fedora wiki and not just gedit like the filthy casual I am). I’ll run some tests tomorrow to see if I really got it working. Maybe I’ll throw together a few benchmarks against the kernel without the ntp patch if I have the time. Thanks for your help.

[gnif]

@dailan post the build log (use pastebin or something similar), it’s very unlikely that anyone here is going to replicate your steps to figure out the error you were getting.

[dailan]

Unless I’m missing something, the build-log doesn’t seem to have any useful information. Here it is. It does, however, point to this code. The warning about unexpanded macros seems to be harmless.
AND just in case, my current ryzen.patch file. If anyone wants any more information, I would be happy to provide it.

[dailan]

Alright here are some quick and dirty benchmarks, the difference that this patch made was night and day.

In total war warhammer II’s campaign benchmark at 1080p
Before: Min 10, Max 25, Average 16.9 FPS
After: Min 43, Max 86, Average 61.6 FPS

In total war warhammer II’s battle benchmark at 1080p
Before: Min 6, Max 17, Average 10.8 FPS
After: Min 43, Max 61, Average 54.3 FPS

In Civ 6’s graphics benchmark at 34440x1440
Before: 99th 65.95, Average 47.66 frame times in MS
After: 99th 24.16, Average 18.68 frame times in MS

Civ 6’s AI benchmark at 3440x1440
Before: Average 33.2 second turns
After: Average 26.5 second turns

Skyrim (just eyeballing it) outside
Before: Min 1, Max 35, Average 20 FPS
After: Min 30, Max 50, Average 40 FPS

Skyrim (just eyeballing it) inside
Before: Min 30, Max 55, Average 40 FPS
After: Min 60, Max 60, Average 60 FPS

[gelmi]

I am using my old card GTS 450 for passing through to my Ubuntu VM, but this card suffers from a reset bug. I have tried detaching the card via command line and then shutdown VM, tried rom dump and feed it in the config - nothing helps. Every time I shut down the VM or reboot, I get black screen and I need to reboot host PC.
Is there any way to power cycle PCIe from host command line in order to initialize it more than once?
BTW, I am on Ryzen 1600 and Asus X370-pro with the latest BIOS (1001).

[SgtAwesomesauce]

This is a hardware bug that can’t be fixed. On some GPU that have dual-bios (my Fury Nitro for example) can be reset by switching to the other BIOS.

[gelmi]

OK and what about that patch that Wendell is talking about in the stream starting @11:35? Kernel 4.15 patch with power cycle to the PCIe?

[mihawk90]

Basically this (as I understand at least):

Linux Host, Windows Guest, GPU passthrough reinitialization fix

I noticed in Wendell’s recent GPU pass-through live stream, that he mentioned that there isn’t a reliable solution for the AMD GPU reinitialization problem. He mentioned that once a Windows guest VM on linux is shut down, the AMD GPU will refuse to reinitialize, and that this requires a reboot of the Host machine to fix. In my personal experience, this actually seemed to be rather random. Sometimes the VM would be able to boot back up, sometimes not. I found that it was not actually the shut down of the VM, but if the host machine enters a sleep state, while the VM was shut down, after having run at least once. The root issue I suspect has less to do with how Linux is treating the GPU, but the behaviour of the Windows OS while it is a guest. Mainly the Windows guest OS is not sending a signal to the GPU to shut-down or start up while it is running in a VM. I assume this is because when on bare-metal, when a Windows OS shuts down, power is actually killed to the GPU from the motherboard/PSU but when in a guest VM, this does not happen, leaving the GPU in an initialized state. Then if/when the host machine enters a low power state, and power is significantly reduced to the GPU, it enters a non-responsive state after the host machine wakes up. Anyway, I found a fix/work-around. I’ve borrowed bits from around the web for this. I don’t write how-to’s often, so bare with me. (note, all these steps take place inside the Windows Guest OS) Step 1 We are going to need the Windows Device Console (DevCon) It is included bundled with the Windows Driver Kit, which you can download and install like a chump, but you don’t require all the added bloat. So, instead download and install Chocolatey, so you can install it from terminal like a bad-ass. Follow the instructions on the Chocolatey website to install. Once Chocolatey is installed open a Windows command prompt and run this command to install DevCon: choco install devcon.portable DevCon is rather straight fo…

[SgtAwesomesauce]

I’m not super up to date on it, because I’m not using Vega or any other GPUs that suffer from this bug (at least, not for passthrough)

I’m going to defer to @mihawk90 and Wendell since they’re clearly following it more closely.

[mihawk90]

actually not really, just reading a bit

[SgtAwesomesauce]

Still following it more closely than I am. I just stay away from GPU with the issue.

[gelmi]

So, for my issue it would be either to somehow reboot or shutdown Ubuntu VM with a some kind of script rather than sudo shutdown or to power cycle on the host system.
I have tried to run this from host OS:

virsh detach-device Ubuntu /mnt/user/system/gpudev.xml
virsh detach-device Ubuntu /mnt/user/system/audiodev.xml
virsh destroy Ubuntu

where xmls are:

GPU
gpudev.xml

  <hostdev mode='subsystem' type='pci' managed='yes'>
    <source>
      <address domain='0x000' bus='0x29' slot='0x00' function='0x0'/>
    </source>
  </hostdev>

HDMI audio
audiodev.xml

  <hostdev mode='subsystem' type='pci' managed='yes'>
    <source>
      <address domain='0x000' bus='0x15' slot='0x00' function='0x1'/>
    </source>
  </hostdev>

After that it says that device was successfully detached and vm was destroyed, but still I cannot use GPU for the second time without host OS reboot.
Any ideas?