Return to Level1Techs.com

Ubuntu 17.04 -- VFIO PCIe Passthrough & Kernel Update (4.14-rc1)

vfio
passthrough

#83

Going dark for the rest of today…my HTPC just hit me with a

“failed to isolate default target freezing”

Have absolutely no clue that this is but it happened after I had to fsck my boot device. So I may not answer questions today


#84

Oh, good points - thanks!

It looks like i can’t use it for gaming, if i understand correctly? :frowning: Sad news.

Do you agree?


#85

See this issue, it has solution that has worked for a couple people.


#87

OK, so I tried this with both VGA devices (08:00 and 09:00) and neither worked. Both times the display just froze. So I don’t think it is differentiating between the two different cards, even though they’re in separate IOMMU groups.

Here is my lspci output:

08:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480] [1002:67df] (rev e7) (prog-if 00 [VGA controller])
        Subsystem: Sapphire Technology Limited Ellesmere [Radeon RX 470/480/570/580] (Nitro+ Radeon RX 580 4GB) [1da2:e366]
        Flags: bus master, fast devsel, latency 0, IRQ 75
        Memory at e0000000 (64-bit, prefetchable) [size=256M]
        Memory at f0000000 (64-bit, prefetchable) [size=2M]
        I/O ports at d000 [size=256]
        Memory at fe900000 (32-bit, non-prefetchable) [size=256K]
        Expansion ROM at 000c0000 [disabled] [size=128K]
        Capabilities: <access denied>
        Kernel driver in use: amdgpu
        Kernel modules: amdgpu

08:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 580] [1002:aaf0]
        Subsystem: Sapphire Technology Limited Ellesmere [Radeon RX 580] [1da2:aaf0]
        Flags: bus master, fast devsel, latency 0, IRQ 82
        Memory at fe960000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: <access denied>
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel
09:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480] [1002:67df] (rev e7) (prog-if 00 [VGA controller])
        Subsystem: Sapphire Technology Limited Ellesmere [Radeon RX 470/480/570/580] (Nitro+ Radeon RX 580 4GB) [1da2:e366]
        Flags: bus master, fast devsel, latency 0, IRQ 77
        Memory at c0000000 (64-bit, prefetchable) [size=256M]
        Memory at d0000000 (64-bit, prefetchable) [size=2M]
        I/O ports at c000 [size=256]
        Memory at fe800000 (32-bit, non-prefetchable) [size=256K]
        Expansion ROM at fe840000 [disabled] [size=128K]
        Capabilities: <access denied>
        Kernel driver in use: amdgpu
        Kernel modules: amdgpu

09:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 580] [1002:aaf0]
        Subsystem: Sapphire Technology Limited Ellesmere [Radeon RX 580] [1da2:aaf0]
        Flags: bus master, fast devsel, latency 0, IRQ 84
        Memory at fe860000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: <access denied>
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel

And I tried running both sudo bash vfio-pci-bind.sh 0000:08:00.0 0000:08:00.1 and sudo bash vfio-pci-bind.sh 0000:09:00.0 0000:09:00.1, and both times the screen froze (but audio continued to work).

EDIT: Running it with sh instead gave me this output:

/usr/local/sbin/vfio-pci-bind.sh: 53: /usr/local/sbin/vfio-pci-bind.sh: [[: not found
/usr/local/sbin/vfio-pci-bind.sh: 58: /usr/local/sbin/vfio-pci-bind.sh: [[: not found
/usr/local/sbin/vfio-pci-bind.sh: 60: /usr/local/sbin/vfio-pci-bind.sh: [[: not found
Please supply Domain:Bus:Device.Function of PCI device in form: dddd:bb:dd.f

I did notice, while rebooting my PC, that I’m getting an error message when Linux starts saying:

sh: /usr/bin/vfio-pci-override.sh: not found

Well, /usr/bin/vfio-pci-override.sh does exist, and has these contents:

 #!/bin/sh    
for i in /sys/bus/pci/devices/*/boot_vga; do
            if [ $(cat "$i") -eq 0 ]; then
                    GPU="${i%/boot_vga}"
                    AUDIO="$(echo "$GPU" | sed -e "s/0$/1/")"
                    echo "vfio-pci" > "$GPU/driver_override"
                    if [ -d "$AUDIO" ]; then
                            echo "vfio-pci" > "$AUDIO/driver_override"
                    fi
            fi
    done 

I checked the permissions on the file: -rwxr-xr-x 1 root root 286

So now I’m getting pretty confused. I found this Arch thread from yesterday that seems to confirm I don’t need to bother with the initcpio hook:

https://bbs.archlinux.org/viewtopic.php?id=242847

So I don’t think it’s that.

I may just grab the Vega 64 Newegg has on sale right now, so I don’t have to deal with this same-gpu nonsense. I really just want to get Star Citizen running in Linux somehow…


#88

Hi, Happy New Year.

Sorry for the late reply. Been dark for Christmas. Yeah, the identical card stuff is a real pain in the behind. Did you get the Vega?

While you may be waiting for the Vega, you can try installing your preferred Windows OS and Star Citizen, Alpha 3.4.1 in a VirtualBox VM or even a standard Virt-manager windows VM without GPU passthrough.

Be sure you pass an external hard drive to the VM or share a folder location on your Linux home with the VM.

Once you have installed Alpha 3.4.1 copy the “Star Citizen” folder to the external hard drive or shared folder

Now go to the Lutris website download and install Lutris.

Next go to the Star Citizen page on the Lutris website

https://lutris.net/games/star-citizen/

Click install on the line with the Live Version

This download and install the required Version of Wine and all dependencies of the game. It will then install the launcher.

Once the launcher is installed, you can open the game but don’t log in just yet. Instead go to the folder you copied the Star Citizen folder from the VM into and copy the entire folder into the "~/program files/robert space industries/ folder Lutris created.
Mine looks like this
30
You have to do the Windows install and copy till we can get Wine to emulate the Windows admin account privileges. It is getting there but not completely. Before the launcher would only download 13GB of the Data.p4k file, now it is downloading all but 10MB or so of the ~49GB. Hopefully, Wine 4.20 will fix this completely.

Anyway, once you’ve copied the folder over to your Lutris star-citizen prefix, you can log in and allow the launcher to install the game. If everything works as it should, the launcher should just flash from saying “Install” to saying “Launch Game”
Assuming you have Vulkan installed,

$ sudo add-apt-repository ppa:oibaf/graphics-drivers
$ sudo apt update
$ sudo apt upgrade
$ sudo apt install libvulkan1 mesa-vulkan-drivers vulkan-utils

The game will launch and you can play the game. FOIP will not work and will crash the game if you try to enable it last I checked in 3.3.7.
My main issue with 3.3.7 was that it would through a .Net error and crash after about 10mins. I have not had this issue in 3.4.1 though I have only tested it 2 times and not in serious game play, more just opening the PU stepping outside the room and leaving the game alone.
I will do more testing later this week maybe even during the day later today to verify. But my guesses are that the .Net error bug has been fixed by CIG though there are some memory leakage bugs still in the game that Linux/Wine is identifying more so than Windows.

See this thread on the LUX Spectrum forum.

https://robertsspaceindustries.com/spectrum/community/LUG/forum/149/thread/its-working-open-mouth-full-persistent-universe/1842813

My guesses are that once we get to March and Alpha 3.5, there may be a switch from DX11 to Vulkan which may really improve Linux compatibility. All that said, the main issues are in the persistent universe, both Arena Commander and Star Marine have been working fine since Alpha 3.2 according to the posts on Spectrum.


#89

I really want to get this working. My hopeful setup:

  • Ubuntu 18.10 on Intel 6700k Integrated Graphics
  • GTX 980ti passthrough to Windows 10 VM

lspci output:

00:02.0 VGA compatible controller [0300]: Intel Corporation HD Graphics 530 [8086:1912] (rev 06) (prog-if 00 [VGA controller])
	Subsystem: Micro-Star International Co., Ltd. [MSI] HD Graphics 530 [1462:7977]
	Flags: bus master, fast devsel, latency 0, IRQ 140
	Memory at dd000000 (64-bit, non-prefetchable) [size=16M]
	Memory at b0000000 (64-bit, prefetchable) [size=256M]
	I/O ports at f000 [size=64]
	[virtual] Expansion ROM at 000c0000 [disabled] [size=128K]
	Capabilities: <access denied>
	Kernel driver in use: i915
	Kernel modules: i915

02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM200 [GeForce GTX 980 Ti] [10de:17c8] (rev a1) (prog-if 00 [VGA controller])
	Subsystem: eVga.com. Corp. GM200 [GeForce GTX 980 Ti] [3842:4996]
	Flags: bus master, fast devsel, latency 0, IRQ 141
	Memory at de000000 (32-bit, non-prefetchable) [size=16M]
	Memory at c0000000 (64-bit, prefetchable) [size=256M]
	Memory at d0000000 (64-bit, prefetchable) [size=32M]
	I/O ports at e000 [size=128]
	Expansion ROM at df000000 [disabled] [size=512K]
	Capabilities: <access denied>
	Kernel driver in use: nouveau
	Kernel modules: nvidiafb, nouveau

Right now I’m stuck at the “getting vfio working” step. Here’s what I’ve done:

initramfs-tools/modules

softdep nouveau pre: vfio vfio_pci
softdep nvidiafb pre: vfio vfio_pci
softdep i915 pre: vfio vfio_pci # tried with this enabled and commented out

vfio
vfio_iommu_type1
vfio_virqfd
options vfio_pci ids=8086:1912,10de:17c8
vfio_pci # doesn't seem to like the ids here
i915 # tried with this enabled and commented out
nouveau

/etc/modules

vfio
vfio_iommu_type1
vfio_pci # doesn't seem to like the ids here

modprobe.d/nouveau.conf

softdep nouveau pre: vfio vfio_pci

modprobe.d/vfio_pci.conf

options vfio_pci ids=8086:1912,10de:17c8

I’m using the 4.16.18 kernel, because it has the Nvidia and Intel firmware.


I also edited the grub file to enable IOMMU, and ensured it’s enabled in the BIOS settings. If I edit the boot arguments within grub on startup and remove the IOMMU enables, the system will boot up. But, if I don’t disable IOMMU the system will hang either at a blank black screen, a blank “ubuntu orange” screen, or a blank black screen with a single solid cursor at the top right. It depends on what configuration the different stuff is in.

Here’s some abbreviated journalctl logs that might be helpful:

Jan 08 00:03:51 desktop kernel: vfio-pci 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=io+mem
Jan 08 00:03:51 desktop kernel: vfio_pci: add [8086:1912[ffff:ffff]] class 0x000000/00000000
Jan 08 00:03:51 desktop kernel: vfio-pci 0000:02:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
Jan 08 00:03:51 desktop kernel: vfio_pci: add [10de:17c8[ffff:ffff]] class 0x000000/00000000

Jan 08 00:04:05 desktop gnome-shell[1474]: Failed to create backend: Could not find a primary drm kms device
Jan 08 00:04:05 desktop gnome-session[1460]: gnome-session-binary[1460]: WARNING: App 'org.gnome.Shell.desktop' exited with code 1
Jan 08 00:04:05 desktop gnome-session-binary[1460]: WARNING: App 'org.gnome.Shell.desktop' exited with code 1
Jan 08 00:04:05 desktop gnome-session-binary[1460]: Unrecoverable failure in required component org.gnome.Shell.desktop
Jan 08 00:04:05 desktop gdm-launch-environment][1383]: pam_unix(gdm-launch-environment:session): session closed for user gdm
Jan 08 00:04:05 desktop gdm3[1372]: GdmDisplay: display lasted 0.430835 seconds
Jan 08 00:04:05 desktop systemd-logind[915]: Session c1 logged out. Waiting for processes to exit.
Jan 08 00:04:05 desktop systemd-logind[915]: Removed session c1.
Jan 08 00:04:05 desktop gdm3[1372]: Child process -1451 was already dead.
Jan 08 00:04:05 desktop systemd[1]: Received SIGRTMIN+21 from PID 403 (plymouthd).
Jan 08 00:04:05 desktop gdm3[1372]: Child process -1451 was already dead.
Jan 08 00:04:05 desktop systemd[1]: Started Hold until boot process finishes up.

Jan 08 00:04:05 desktop kernel: vfio-pci 0000:02:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
Jan 08 00:04:05 desktop kernel: vfio-pci 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=io+mem

It appears that gnome is failing to load and this results in the system shutting down, but the computer remaining powered on.

In the case where IOMMU is disabled, I get the following:

Jan 08 00:05:36 desktop kernel: vfio-pci: probe of 0000:02:00.0 failed with error -22
Jan 08 00:05:36 desktop kernel: vfio_pci: add [10de:17c8[ffff:ffff]] class 0x000000/00000000

Which is the error code you seem to get when IOMMU is unavailable.


I am running sudo update-initramfs -u -k 4.16.18-041618-generic and sudo update-grub after making any changes.

I hope someone can help me. Thanks.

Edit:

Finally figured out my problem. I thought you were supposed to use the addresses for both GPUs. You should have the addresses for the GPU and Audio controller only of the device you wish to passthrough.