AMD R9 390 passthrough on KVM/Kubuntu

Operating Systems & Open Source Linux
,
1

Hi folks,

I was inspired by Wendell's GPU passthrough video and decided to attempt this once I saw what was possible. Unfortunately I have been attempting this off and on for months now with no luck.

I have been following two guides mainly when attempting this which are https://wiki.debian.org/VGAPassthrough and http://www.bnote.us/2015/04/kvm-vga-passthrough-in-ubuntu-1504-vivid.html.

This is my setup:
AMD8350
Asus M5A97 R2.0
AMD R9 390 (for passthrough)
Nvidia 610 (for Linux Host)
HP Monitor with dual inputs, DVI to the R9 390 and VGA to the 610
Switching between Kubuntu & Ubuntu 16.04

Rough Summary:

Every time I attempt to passthrough the GPU via pci stub, only the audio is passed through and the video part of the card is not. It keeps saying in use by radeon even though I have blacklisted radeon in /etc/modprobe.d/blacklist.conf. At this point the machine boots into a blank screen. Switching to a virtual console allows me to see the login screen, however the screen blinks every couple of seconds and does not respond to input. Also at this point when checking the monitor, the only signal I get is from the DVI (R9 390) and no signal at all from the VGA(610).

I had also tried switching to a virtual console, stopping the display manager and attempting "rmod radeon" but this doesn't work as the GPU still remains "in use by radeon".

Any ideas or suggestions on where I might be going wrong with this?

2

is the nvidia card in the first pcie slot? its easier if the card for your host is also the card your bios initialized with.

Also I would recommend trying vfio-pci instead of pci-stub, as it seems to be the recommended tool for 4.1+ kernels.

3

The Nvidia card is actually in the second slot which is unfortunately a 4X slot. I was hoping I could keep the 390 on the 1st slot which is the 16x slot.

I don't really see any options on the bios to set the primary graphics adapter. There is an option for PCI/PEG or PEG/PCI but that doesn't seem to do anything. I was fearing that the primary graphics adapter was based on the physical slot.

I guess what I can probably try to do as a quick test is try to passthrough the Nvidia card since it is currently in the second slot.

4

Vfio-pci is a much better method.

@op we need more info. What kernel, what do your configuration look like, what command are you using for qemu?

Also, the arch wiki has a good article on it, doesn't quite compare to Ubuntu, but it's pretty damn close and it's helpful because it explains how it all works as well. I'll update the post when the plane WiFi is working well enough.

Edit: as promised, https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF

5

A 390 won't notice much hiccup with amdgpu and a 4x slot. Swap it and give it a try.

Nvidia cards are funny with passthrough, but feel free to give it a try.

6

uefi ready cards tend to handle passthrough better in my experience. I never could get an nvidia 6xx card to passthrough properly, though i only had an hour or so to try it. (on loan)

my nvidia 770 handles passthrough pretty well, as long as i remembered to do the workarounds to get the driver working. though i had some terrible microstutter in bl2 i could never fix.

experiment, see if you can get vfio-pci to properly bind to the card.

1 Like
7

Have you tried virt-manager? You can just select any hardware to passthrough from the virtual machine settings.

8

Thanks for all the advice guys. I actually just swapped the GPU locations and rebooted. The 390 is now in the 4x slot and the 610 in the 16x. And....

marlon@MBLPC:~$ dmesg | grep pci-stub
[ 5.427954] pci-stub: add 1002:67B1 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[ 5.427967] pci-stub 0000:06:00.0: claimed by stub
[ 5.427973] pci-stub: add 1002:AAC8 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[ 5.427979] pci-stub 0000:06:00.1: claimed by stub

Success! 1002:67B1 is the video and 1002:AAC8 is the audio.

I do recall trying to perform this with the 390 in the 4x slot in the past and having no success but I guess I had probably done something wrong back then. I also did attempt to use vfio-pci sometime back but it didn't work, I guess due to the physical card location.

I will have to try this out with virt-manager as well. I wasn't aware you can passthrough the hardware from there.

Now that step 1 is completed let me see if I can finally get this thing working.

9

great, virt-manager (and libvirt for that matter) added support a while back, it should be under like pci devices or something in the gui.

10

I have been following Alex Williamson's "how to series" in trying to set up the GPU passthrough via virt manager. I have had a little success as the VM display is being displayed on the HDMI input of my monitor via the AMD graphics card, so the GPU passthrough seems to be working. Unfortunately I am not getting the windows OS to boot on the VM at all. The VM just remains on the UEFI interactive shell. Typing 'fs0:' results in "fs0 is not a valid mapping".

I realized that when I attempt to use an Ubuntu OS instead, the VM actually boots straight into the Ubuntu OS installer but does not display on the AMD graphics card??? If I try to continue with the Ubuntu installation it just crashes.

I have tried doing some research on it online but so far I have had no luck getting it past the EFI shell to start the windows installation. I stumbled across this : https://lime-technology.com/forum/index.php?topic=39493.0 but it didn't make a difference. Any suggestions on where I might be going wrong with this?

Here is my VM config:


Win8.1
f023c0fd-8499-4664-af62-6c8deae83fc7
8294400
8294400
4

hvm
/usr/share/OVMF/OVMF_CODE.fd
/var/lib/libvirt/qemu/nvram/Win8.1_VARS.fd












destroy
restart
restart





/usr/bin/kvm-spice
























































































11

Hmmm, that copy and paste didn't work out so well....

12

After a week of attempting this I decided to post a summary of my experience thus far. This is gonna be long, please bare with it.

I actually started over from scratch, installing Ubuntu 16.04 (stock kernel) instead of Kubuntu and had some success.

I detached the 390 through pci_stub and had to blacklist the radeon drivers in /etc/modprobe.d/blacklist.conf. Note that the blacklist did not work until the physical machine was shutdown completely, restarting after applying the blacklist did not work. Is this normal?

I then installed qemu-kvm, virt-manager and the latest unstable debian OVMF. Using the ovmf version in the Ubuntu repository causes the VM to take very long to start for me. Sorry did not document the versions....

I created the VM following both Alex Williamson's how to series (http://vfio.blogspot.com/2015/05/vfio-gpu-how-to-series-part-4-our-first.html) and TPC 010 videos (https://www.youtube.com/watch?v=JKtYRFPhmYE).

I was not able to get it to work with windows 7 or 8.1. The VM would just sit on the EFI screen without FS0 being mounted. Trying an Ubuntu OS instead actually starts up the boot process but when trying to install Ubuntu, it crashes out, still not sure what caused this.

I instead downloaded a copy of Windows 10 and that actually worked. It was able to boot from OS CD and install successfully. I had to use the virtIO drivers for win10 to see the HDD and then it was smooth sailing from there...until restart. The Win10 PC kept crashing on boot.
I realized I had to install the Windows 10 OS and all required virtIO drivers before attaching the PCI card to it. Once it was done in this sequence it worked and the display now came out through the DVI connection of the Monitor....Success!!!

I installed the AMD drivers and the Win10 VM then saw the R9 390 in the device manager. From here I had to switch from VNC to Spice to get audio.

With everything looking like it was working, I installed steam and tested Dead Island riptide. When I tried to switch the game from windowed to full-screen is when the new problem began. I ended up with a blank screen after switching to full-screen.
After some troubleshooting I noticed the VM was running in extended mode with 2 monitors, one as the console screen output from virt manager and the other as the physical monitor with DVI connection. Again is this normal?
After messing around with it, windows crashed and I had to force off the VM,

After this windows 10 crashed on the "your PC/ Device needs to be repaired screen" where none of the provided options works. Not sure if this is an actually windows crash or something related to the VM.

And that's basically where I am at with this setup at the moment.

I noticed some strange thing about my machine when attempting this. Hopefully someone with experience can chime in:

I cannot install Ubuntu or any Linux OS for that matter with the R9 390 installed in the 4x PCI slot. The OS crashes right after selecting the Install or Try Ubuntu options. It seems to crash right after attempting to load the USB devices.
I then tried to switch IOMMU off, on the board and re-attempt the installation. At this point it actually loads up properly and I am presented with the first screen of the GUI to start the installation. Everything looks normal now, except no USB devices work so no mouse or keyboard.
I actually had to remove the 390, install Ubuntu, then reinstall the 390. This seems very strange, problem with the motherboard maybe?
Also with the 390 in the 4x slot now, after Ubuntu has been installed I see a AMD VI completion loop timeout" message being repeated until Ubuntu boots.