Have you tried updating to the latest UEFI for your motherboard? A lot of ryzen boards came with dysfunctional passthrough support on the software side, of which most have had patches released by now.
Also, are you running on windows 10 v1607 (anniversary edition)? M$ introduced this new feature called 'GameDVR' which is part of the xbox app. It basically allows you to record your games and take pictures through the xbox app. In doing so, it enforces vsync on your games (capping them to 30fps in some cases) and causing input lag from the overlay. On top of that, it takes up an insane amount of resources just to run. Here's the catch; the option to disable it officially is in the xbox app, but you can't get in to the xbox app without having a M$ account... and even removing the xbox app through powershell, GameDVR persists. It must be integrated in to the entire OS... the only solution I found is to bypass all of this by going in to the registry and manually disabling it. There's a video demonstrating how to do it here
Hmm... Disabling Game DVR helped a little bit but there is still a very noticeable lockup when switching between menus or respawning in the game.
I have htop open on my Linux host on another monitor and it seems that all 8 of the VM's virtual cores shoot up to 100% whenever the lockups occur. This shouldn't really be the case especially since Rocket League is a single threaded game. Perhaps it's an issue with the Nvidia drivers?
For now I've given up. If anyone else has encountered poor performance on Ryzen, let me know. My i7-4790K with half the cores performs better in KVM w/ GPU passthrough with the exact same GPU, so I'm not sure what the issue is. Hopefully it will be resolved soon.
I read some where that there seems to be incompatibilities between Ryzen and the most common used hypervisor. The post which I read said to try Xen instead of the commonly used hypervisor. Of course their wasn't any instructions in the post on how to set it up. I have posted a question on the official Xen forum asking how is the performance with Xen and how to set it up. If I get an answer back I will post it here.
I was using Linux's built-in KVM hypervisor, as it has the best support for GPU passthrough. Xen may be an option, but since I have an Nvidia card, it must support hyperv vendor id spoofing to trick the drivers into working.
Update: I was able to partially resolve it by disabling Nested Page Tables. I thought I had done this previously, but it turns out there was an error in the Level1Techs Ryzen GPU Passthrough Setup Guide
Specifically, in /etc/modprobe.d/kvm.conf, the guide says:
Make sure the kvm_amd and kvm_intel nested=1 lines are commented out to disable AVIC and nested page tables since they have performance/stability issues currently.
However, the nested=1 parameter refers to nested virtualization, not nested page tables. To properly disable nested page tables, you have to add the following to the file:
options kvm_amd npt=0
Maybe @wendell could update the guide to reflect this change?
After adding this, all stuttering in Rocket League was gone, but performance in GTA V was still clearly suffering from a CPU bottleneck. This is most likely due to the increased overhead of emulating page tables in the hypervisor as opposed to having the hardware do it for you. Hopefully a solution is presented in the future, but for now at least games are playable. For those of you wanting to buy a CPU for GPU Passthrough, it seems the best option is still Intel, since their CPUs don't seem to suffer from this bug.
At 4K High/Very High settings it was dropping down to 41FPS while driving. If you lowered the settings it'd probably be playable, but it's nowhere near native performance.
I was curious since I have a future build plan based around a 16 core Threadripper, to be used as a development workstation (compiling-ish stuff) + gaming machine alongside. And naturally, same as you probably, I don't really care if it's slower than native as long as I don't have to reboot to game - your mention of "41fps while driving" helps put that into perspective.
Most people claim that having npt enabled helps the general virtualization case with CPU workloads virtualized, where as when passing through the GPU - the performance is better with npt disabled (which hurts the CPU workloads).
I wonder if it would make more sense to run windows as native with linux inside hyper-v in my case until npt is fixed.
I'm in a very similar situation (AMD multi-core beast for development/compiling but gaming on the side). In my case, I don't use Windows enough to justify using it as my main OS with Linux as a guest, but if you were going to game frequently and wanted better performance it seems like a good solution.
It seems the latest 4.13-rc has some improvements to the amd_iommu driver, with the patch notes even claiming to reduce reduce flush rates to 1% in a certain part of the driver. Unfortunately, in my preliminary testing this doesn't seem to have fixed the npt performance woes.
I'll update this thread if I make any breakthroughs.
All I can say is to keep checking for UEFI updates to your motherboard. Watching Wendell's GPU-passthrough videos, he always notes later in the video that passthrough bugs were fixed by updating the UEFI, or at least the vendor stated that they were working on a software patch. Also need to be careful which motherboard you get, because some vendors might not care to patch it or it can turn out to be a hardware limitation.
Yeah - I'll definitely keep an eye out for future UEFI updates. The NPT issue seems to affect all ryzen systems though, so AMD themselves will most likely need to push out a patch (either to Linux's KVM code or to systems in the form of an updated AGESA blob).
Oh I forgot to mention, Wendell made a video recently on Qubes OS. This distro's entire firewall is built around using everything with VM's, and it uses the Xen hypervisor by default. If you're struggling on figuring out how to install Xen, this would get you straight there so you can test to see if these bugs are fixed.
Funnily enough, I have actually tried Qubes on this rig, but it seems to use an old Linux kernel without support for either one of my GPUs or even my Ryzen CPU. Guess I'll have to wait for an updated version.
Yeah I'm actually running a custom compiled kernel on my Fedora install right now. The only issue is that I can't even install Qubes with its outdated kernel. The website had some mentions of a remote install over SSH if your graphics are unsupported, so I may end up trying that then swapping the kernel, but honestly it seems like a lot of work and I'm a bit too lazy. I don't think I would be able to use Xen to passthrough my Nvidia GTX card anyways, since the drivers are gimped to shut off if they detect that they're being run in a VM. As far as I know, Linux's KVM is the only hypervisor that supports spoofing this to allow the drivers to work.
Apparently there is a bug with nested page tables decreasing the GPU performance by ~25-50% as one of you guys stated earlier.
A guy from AMD joined the conversation in that thread and seems to be on the case.
For those people who did not invest in a Ryzen system for GPU passthrough until now:
Either wait for a fix for the NPT bug or go the Intel route.
Another option would be to utilize Xen as it does not seem to be impacted as hard as KVM.