Ubuntu 17.04 -- VFIO PCIe Passthrough & Kernel Update (4.14-rc1)

Software & Operating Systems Linux
,
72

Does anyone know of a way to do this with identical cards? My two 580 show up in separate IOMMU groups (14 and 15) but have the same ID:

IOMMU Group 14 08:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480] [1002:67df] (rev e7)
IOMMU Group 14 08:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 580] [1002:aaf0]
IOMMU Group 15 09:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480] [1002:67df] (rev e7)
IOMMU Group 15 09:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 580] [1002:aaf0]

I’m assuming that if I tell it to bind the device ID for vfio, it’ll bind both and I will have no display card?

EDIT: I found this script

Not sure if it’ll work though, or if it can be integrated with this method.

73

The concept should work, but that example may need a bit of tweaking for a current distro.

Here are a couple more examples-
https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#Using_identical_guest_and_host_GPUs

2 Likes
74

OK, this may be a wait for the weekend project. Thanks for the link.

75

Taking from

Arch PCI passthrough via OVMF

Using identical guest and host GPUs

Reason: A number of users have been having issues with this, it should probably be adressed by the article.

Due to how vfio-pci uses your vendor and device id pair to identify which device they need to bind to at boot, if you have two GPUs sharing such an ID pair you will not be able to get your passthough driver to bind with just one of them. This sort of setup makes it necessary to use a script, so that whichever driver you are using is instead assigned by pci bus address using the driver_override mechanism.

Script variants
Passthrough all GPUs but the boot GPU
Here, we will make a script to bind vfio-pci to all GPUs but the boot gpu. Create the script /usr/bin/vfio-pci-override.sh:

#!/bin/sh

for i in /sys/bus/pci/devices/*/boot_vga; do
	if [ $(cat "$i") -eq 0 ]; then
		GPU="${i%/boot_vga}"
		AUDIO="$(echo "$GPU" | sed -e "s/0$/1/")"
		echo "vfio-pci" > "$GPU/driver_override"
		if [ -d "$AUDIO" ]; then
			echo "vfio-pci" > "$AUDIO/driver_override"
		fi
	fi
done

modprobe -i vfio-pci

Passthrough selected GPU
In this case we manually specify the GPU to bind.

#!/bin/sh

GROUP="0000:00:03.0"
DEVS="0000:03:00.0 0000:03:00.1 ."

if [ ! -z "$(ls -A /sys/class/iommu)" ]; then
	for DEV in $DEVS; do
		echo "vfio-pci" > /sys/bus/pci/devices/$GROUP/$DEV/driver_override
	done
fi

modprobe -i vfio-pci
76

Oops I duplicated your reply

77

Should I avoid modifying the initramfs-tools/modules file, if I go this route?

EDIT: So when I run

sudo modprobe -i vfio-pci

I get:

libkmod: ERROR …/libkmod/libkmod-config.c:656 kmod_config_parse: /etc/modprobe.d/vfio_pci.conf line 1: ignoring bad line starting with ‘options’

I’m assuming something is wrong with my /etc/modprobe.d/vfio_pci.conf file. Was there supposed to be a preexisting configuration there? Mine was blank so I just pasted what Wendell had minus the ids part (so the contents of the file are just “options vfio_pci”)

2nd EDIT:

So I got that error message to go away by changing the file to read:

install vfio-pci /usr/bin/vfio-pci-override.sh

but it’s still not binding the graphics card. Both are saying amdgpu as the kernel driver and kernel modules for both graphics cards.

I have grub set up to use iommu:

GRUB_CMDLINE_LINUX_DEFAULT=“quiet splash iommu=1 amd_iommu=on”

Here’s the contents of my /etc/initramfs-tools/modules file:

softdep amdgpu pre: vfio vfio_pci
vfio
vfio_iommu_type1
vfio_virqfd
vfio_pci
amdgpu

here’s my /etc/modules file:

8812au
8812au
vfio
vfio_iommu_type1
vfio_pci

Could it possibly be that I need to include/call the script in initramfs? I’m not sure how to go about doing that.

78

Sorry, just saw your replay and realized you are using 2 AMD GPUs for some reason I was thinking 2 nvidia cards even after reading RX580. I think part of the issue may be the loading time of the AMDGPU/radeon driver. When I was initially passing through an R9 390 I had to literally blacklist the radeon and AMDGPU drivers for it to work.

That said your /etc/initramfs-tools/modules file should also have the lines

options vfio_pci ids=10de:1b80,10de:10f0
vfio_pci ids=10de:1b80,10de:10f0

in it where the ids are those of you passthrough GPU. You may want to split the options vfio_pci into 2 lines, one for the video and one for the audio parts of the GPU. It may be overkill, but literally have it 3 times in my file

options vfio_pci ids=10de:1b80
options vfio_pci ids=10de:10f0
options vfio_pci ids=10de:1b80,10de:10f0

Add these lines before the last 2 lines of your current file.

Your /etc/modules file should also have in it the line

vfio_pci ids=10de:1b80,10de:10f0

You almost need for force vfio to take hold of the GPU

If you are on a Kernel from 4.18.16 forward you may also have to try this: I am copying and pasting from the Arch Wiki

Create /etc/modprobe.d/vfio.conf with the following:

install vfio-pci /usr/bin/vfio-pci-override.sh

Edit /etc/mkinitcpio.conf

Remove any video drivers from MODULES, and add vfio-pci, and vfio_iommu_type1

MODULES=(ext4 vfat vfio-pci vfio_iommu_type1)

Add/etc/modprobe.d/vfio.conf and /usr/bin/vfio-pci-override.sh to FILES:

FILES=(/etc/modprobe.d/vfio.conf /usr/bin/vfio-pci-override.sh)

regenerate initramfs and reboot

update-initramfs -u
reboot

I have avoided identical GPUs just because it seemed a complicated process and have not been confident in using vfio but forcing myself to learn what I can right now. Hopeful these work for you.

79

I’ve actually hoped that i could do this trick on my laptop. It’s a Dell 9570, so i have both a dedicated and a integrated GPU. My daily drivers is the integrated Intel, so my wish was to use the 1050 Ti for Looking Glass.

After multiple hours in Linux Mint, i’m almost giving up. I tried to type down every step, so that i can post it here, if i make it till the very end, but at the moment, i find it very hard.

Here is a few details about what is under control:

  • Virt-manager is installed, including a lot of different packages

  • iOOMU is under control (guess so?) and 10de:1c8c is now shared:

softdep nvidia pre: vfio vfio_pci

vfio
vfio_iommu_type1
vfio_virqfd
options vfio_pci ids=10de:1c8c
vfio_pci ids=10de:1c8c
vfio_pci
nvidia

It’s also listed correctly with lspci -nnv |less:

01:00.0 3D controller [0302]: NVIDIA Corporation GP107M [GeForce GTX 1050 Ti Mobile][10de:1c8c] (rev a1)
        Subsystem: Dell GP107M [GeForce GTX 1050 Ti Mobile][1028:087c]
        Flags: bus master, fast devsel, latency 0, IRQ 11
        Memory at ec000000 (32-bit, non-prefetchable) [size=16M]
        Memory at c0000000 (64-bit, prefetchable) [size=256M]
        Memory at d0000000 (64-bit, prefetchable) [size=32M]
        I/O ports at 3000 [disabled][size=128]
        Expansion ROM at ed000000 [disabled][size=512K]
        Capabilities: <access denied>
        Kernel driver in use: vfio-pci
        Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia

  • Windows 10 is installed in a VM, and the gfx PCI is added. I’ve keeped the default video output to debug and install windows the first time.

    image
    image.png823x523 42.4 KB

  • <kvm><hidden state='on'/></kvm> is added after </hyperv> (with virsh edit <nameofvm>), and a few other things, mentioned here

  • In Device Manager i’m able to find a few devices

Currenly my main problem/question is:

  1. Nvidia Driver can’t be installed. Is failing. I saw a post here about patching the drivers. But is that actually my problem?

    image
    image.jpg793x597 96.6 KB

  2. Will i be able to use Looking Glass for my Dell XPS 9570, or have i just waisted a damn long amount of time :-)?

80

See this github readme.
https://gist.github.com/Misairu-G/616f7b2756c488148b7309addc940b28

  1. You do not need to patch the drivers as in that post. Adding hidden to KVM in VM virsh edit and whatnot takes care of that Issue without the need to patched windows drivers.

  2. Your lspci shows that the 1050 ti is a 3d controller and not a VGA controller. That means that it is mux-less graphics setup, see link above for explanation.

  3. I actually do not know if looking-glass would work, but it might. You need to first get the Nvidia driver successfully installed, with the card not reporting problems.

  4. To have any hope of it working, you will need to give the VM a copy of the 1050ti’s vROM(otherwise known as a vBIOS). With normal video cards, this entails dumping it to a .rom file, then add the path to it in your VM XML. In your case, the 1050ti vROM probably is embedded into your motherboard UEFI, which complicates things.

In the case that the vROM is baked into your laptop’s UEFI, you will need to first get a complete copy of your motherboard’s UEFI, updates are often delta only and not complete, so a complete copy might be hard to get. Next, you will need to rip it apart to get the GPU vROM, then you can take the vROM and compile it with a couple of not really tested patches into custom OVMF ACPI tables. Then you have a hope of getting passthrough working.

81

Thanks so much for the help. Question though, here’s my lspci -nnv output:

08:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480] [1002:67df] (rev e7) (prog-if 00 [VGA controller])
Subsystem: Sapphire Technology Limited Ellesmere [Radeon RX 470/480/570/580] (Nitro+ Radeon RX 580 4GB) [1da2:e366]

and

09:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480] [1002:67df] (rev e7) (prog-if 00 [VGA controller])
Subsystem: Sapphire Technology Limited Ellesmere [Radeon RX 470/480/570/580] (Nitro+ Radeon RX 580 4GB) [1da2:e366]

What is the ID number? Is it the first one (1002:67df) or the second one (1da2:e366)?

EDIT: Tried both and no luck, they’re still both using amdgpu. And I don’t have an /etc/mkinitcpio.conf file at all, but reading the Arch Wiki makes it seem like this problem was corrected in 4.19, which I’m running, so I shouldn’t need to do that step, regardless.

82

Sorry, went to take a short nap earlier, woke up at almost 2am, guess I needed to catch up on some sleep.

Yeah, I think you are facing the same issue I was having with my R9 390, and possibly Vega 64 too, related to the timing of the GPU driver binding that meant I simply had no choice but to blacklist the AMDGPU and Radeon drivers.

Obviously, blacklisting is not an option for you so try manually binding the GPU. You can download the vfio-pci-bind script from this Andre Richter github make sure it is executable and copy it to /usr/local/sbin/

sudo cp -i /home/####/Downloads/vfio-pci-bind.sh /usr/local/sbin/

then run the following in a terminal to see if it works (changing the 4 to the number of the GPU you want to passthorugh from lspci -nnv |less )

sudo vfio-pci-bind 0000:04:00.0 0000:04:00.1

I remember I used to have this line at the top of my old and very first gpu passthrough KVM/Qemu execution script. But I think when I switched from Arch to elementaryOS/Ubuntu I no longer need it and lost my version of the script.

Hopefully this will work and force the GPU and everything in it’s IOMMU group to bind to vfio. I am not sure if you can use it in the virt-manager XML. So you may have to run it in terminal each time before you start your VM. When I first did this, I wasn’t using virt-manger because it kept breaking my VM on almost every other Arch update.

1 Like
83

Going dark for the rest of today…my HTPC just hit me with a

“failed to isolate default target freezing”

Have absolutely no clue that this is but it happened after I had to fsck my boot device. So I may not answer questions today

84

Oh, good points - thanks!

It looks like i can’t use it for gaming, if i understand correctly? :frowning: Sad news.

Do you agree?

85

See this issue, it has solution that has worked for a couple people.

1 Like
87

OK, so I tried this with both VGA devices (08:00 and 09:00) and neither worked. Both times the display just froze. So I don’t think it is differentiating between the two different cards, even though they’re in separate IOMMU groups.

Here is my lspci output:

08:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480] [1002:67df] (rev e7) (prog-if 00 [VGA controller])
        Subsystem: Sapphire Technology Limited Ellesmere [Radeon RX 470/480/570/580] (Nitro+ Radeon RX 580 4GB) [1da2:e366]
        Flags: bus master, fast devsel, latency 0, IRQ 75
        Memory at e0000000 (64-bit, prefetchable) [size=256M]
        Memory at f0000000 (64-bit, prefetchable) [size=2M]
        I/O ports at d000 [size=256]
        Memory at fe900000 (32-bit, non-prefetchable) [size=256K]
        Expansion ROM at 000c0000 [disabled] [size=128K]
        Capabilities: <access denied>
        Kernel driver in use: amdgpu
        Kernel modules: amdgpu

08:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 580] [1002:aaf0]
        Subsystem: Sapphire Technology Limited Ellesmere [Radeon RX 580] [1da2:aaf0]
        Flags: bus master, fast devsel, latency 0, IRQ 82
        Memory at fe960000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: <access denied>
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel
09:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480] [1002:67df] (rev e7) (prog-if 00 [VGA controller])
        Subsystem: Sapphire Technology Limited Ellesmere [Radeon RX 470/480/570/580] (Nitro+ Radeon RX 580 4GB) [1da2:e366]
        Flags: bus master, fast devsel, latency 0, IRQ 77
        Memory at c0000000 (64-bit, prefetchable) [size=256M]
        Memory at d0000000 (64-bit, prefetchable) [size=2M]
        I/O ports at c000 [size=256]
        Memory at fe800000 (32-bit, non-prefetchable) [size=256K]
        Expansion ROM at fe840000 [disabled] [size=128K]
        Capabilities: <access denied>
        Kernel driver in use: amdgpu
        Kernel modules: amdgpu

09:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 580] [1002:aaf0]
        Subsystem: Sapphire Technology Limited Ellesmere [Radeon RX 580] [1da2:aaf0]
        Flags: bus master, fast devsel, latency 0, IRQ 84
        Memory at fe860000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: <access denied>
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel

And I tried running both sudo bash vfio-pci-bind.sh 0000:08:00.0 0000:08:00.1 and sudo bash vfio-pci-bind.sh 0000:09:00.0 0000:09:00.1, and both times the screen froze (but audio continued to work).

EDIT: Running it with sh instead gave me this output:

/usr/local/sbin/vfio-pci-bind.sh: 53: /usr/local/sbin/vfio-pci-bind.sh: [[: not found
/usr/local/sbin/vfio-pci-bind.sh: 58: /usr/local/sbin/vfio-pci-bind.sh: [[: not found
/usr/local/sbin/vfio-pci-bind.sh: 60: /usr/local/sbin/vfio-pci-bind.sh: [[: not found
Please supply Domain:Bus:Device.Function of PCI device in form: dddd:bb:dd.f

I did notice, while rebooting my PC, that I’m getting an error message when Linux starts saying:

sh: /usr/bin/vfio-pci-override.sh: not found

Well, /usr/bin/vfio-pci-override.sh does exist, and has these contents:

 #!/bin/sh    
for i in /sys/bus/pci/devices/*/boot_vga; do
            if [ $(cat "$i") -eq 0 ]; then
                    GPU="${i%/boot_vga}"
                    AUDIO="$(echo "$GPU" | sed -e "s/0$/1/")"
                    echo "vfio-pci" > "$GPU/driver_override"
                    if [ -d "$AUDIO" ]; then
                            echo "vfio-pci" > "$AUDIO/driver_override"
                    fi
            fi
    done

I checked the permissions on the file: -rwxr-xr-x 1 root root 286

So now I’m getting pretty confused. I found this Arch thread from yesterday that seems to confirm I don’t need to bother with the initcpio hook:

https://bbs.archlinux.org/viewtopic.php?id=242847

So I don’t think it’s that.

I may just grab the Vega 64 Newegg has on sale right now, so I don’t have to deal with this same-gpu nonsense. I really just want to get Star Citizen running in Linux somehow…

88

Hi, Happy New Year.

Sorry for the late reply. Been dark for Christmas. Yeah, the identical card stuff is a real pain in the behind. Did you get the Vega?

While you may be waiting for the Vega, you can try installing your preferred Windows OS and Star Citizen, Alpha 3.4.1 in a VirtualBox VM or even a standard Virt-manager windows VM without GPU passthrough.

Be sure you pass an external hard drive to the VM or share a folder location on your Linux home with the VM.

Once you have installed Alpha 3.4.1 copy the “Star Citizen” folder to the external hard drive or shared folder

Now go to the Lutris website download and install Lutris.

Next go to the Star Citizen page on the Lutris website

https://lutris.net/games/star-citizen/

Click install on the line with the Live Version

34
Screenshot from 2019-01-02 00.16.34.png1178x755 129 KB

This download and install the required Version of Wine and all dependencies of the game. It will then install the launcher.

Once the launcher is installed, you can open the game but don’t log in just yet. Instead go to the folder you copied the Star Citizen folder from the VM into and copy the entire folder into the "~/program files/robert space industries/ folder Lutris created.
Mine looks like this
30
You have to do the Windows install and copy till we can get Wine to emulate the Windows admin account privileges. It is getting there but not completely. Before the launcher would only download 13GB of the Data.p4k file, now it is downloading all but 10MB or so of the ~49GB. Hopefully, Wine 4.20 will fix this completely.

Anyway, once you’ve copied the folder over to your Lutris star-citizen prefix, you can log in and allow the launcher to install the game. If everything works as it should, the launcher should just flash from saying “Install” to saying “Launch Game”
Assuming you have Vulkan installed,

$ sudo add-apt-repository ppa:oibaf/graphics-drivers
$ sudo apt update
$ sudo apt upgrade
$ sudo apt install libvulkan1 mesa-vulkan-drivers vulkan-utils

The game will launch and you can play the game. FOIP will not work and will crash the game if you try to enable it last I checked in 3.3.7.
My main issue with 3.3.7 was that it would through a .Net error and crash after about 10mins. I have not had this issue in 3.4.1 though I have only tested it 2 times and not in serious game play, more just opening the PU stepping outside the room and leaving the game alone.
I will do more testing later this week maybe even during the day later today to verify. But my guesses are that the .Net error bug has been fixed by CIG though there are some memory leakage bugs still in the game that Linux/Wine is identifying more so than Windows.

See this thread on the LUX Spectrum forum.

https://robertsspaceindustries.com/spectrum/community/LUG/forum/149/thread/its-working-open-mouth-full-persistent-universe/1842813

My guesses are that once we get to March and Alpha 3.5, there may be a switch from DX11 to Vulkan which may really improve Linux compatibility. All that said, the main issues are in the persistent universe, both Arena Commander and Star Marine have been working fine since Alpha 3.2 according to the posts on Spectrum.

1 Like
89

I really want to get this working. My hopeful setup:

  • Ubuntu 18.10 on Intel 6700k Integrated Graphics
  • GTX 980ti passthrough to Windows 10 VM

lspci output:

00:02.0 VGA compatible controller [0300]: Intel Corporation HD Graphics 530 [8086:1912] (rev 06) (prog-if 00 [VGA controller])
	Subsystem: Micro-Star International Co., Ltd. [MSI] HD Graphics 530 [1462:7977]
	Flags: bus master, fast devsel, latency 0, IRQ 140
	Memory at dd000000 (64-bit, non-prefetchable) [size=16M]
	Memory at b0000000 (64-bit, prefetchable) [size=256M]
	I/O ports at f000 [size=64]
	[virtual] Expansion ROM at 000c0000 [disabled] [size=128K]
	Capabilities: <access denied>
	Kernel driver in use: i915
	Kernel modules: i915

02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM200 [GeForce GTX 980 Ti] [10de:17c8] (rev a1) (prog-if 00 [VGA controller])
	Subsystem: eVga.com. Corp. GM200 [GeForce GTX 980 Ti] [3842:4996]
	Flags: bus master, fast devsel, latency 0, IRQ 141
	Memory at de000000 (32-bit, non-prefetchable) [size=16M]
	Memory at c0000000 (64-bit, prefetchable) [size=256M]
	Memory at d0000000 (64-bit, prefetchable) [size=32M]
	I/O ports at e000 [size=128]
	Expansion ROM at df000000 [disabled] [size=512K]
	Capabilities: <access denied>
	Kernel driver in use: nouveau
	Kernel modules: nvidiafb, nouveau

Right now I’m stuck at the “getting vfio working” step. Here’s what I’ve done:

initramfs-tools/modules

softdep nouveau pre: vfio vfio_pci
softdep nvidiafb pre: vfio vfio_pci
softdep i915 pre: vfio vfio_pci # tried with this enabled and commented out

vfio
vfio_iommu_type1
vfio_virqfd
options vfio_pci ids=8086:1912,10de:17c8
vfio_pci # doesn't seem to like the ids here
i915 # tried with this enabled and commented out
nouveau

/etc/modules

vfio
vfio_iommu_type1
vfio_pci # doesn't seem to like the ids here

modprobe.d/nouveau.conf

softdep nouveau pre: vfio vfio_pci

modprobe.d/vfio_pci.conf

options vfio_pci ids=8086:1912,10de:17c8

I’m using the 4.16.18 kernel, because it has the Nvidia and Intel firmware.

I also edited the grub file to enable IOMMU, and ensured it’s enabled in the BIOS settings. If I edit the boot arguments within grub on startup and remove the IOMMU enables, the system will boot up. But, if I don’t disable IOMMU the system will hang either at a blank black screen, a blank “ubuntu orange” screen, or a blank black screen with a single solid cursor at the top right. It depends on what configuration the different stuff is in.

Here’s some abbreviated journalctl logs that might be helpful:

Jan 08 00:03:51 desktop kernel: vfio-pci 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=io+mem
Jan 08 00:03:51 desktop kernel: vfio_pci: add [8086:1912[ffff:ffff]] class 0x000000/00000000
Jan 08 00:03:51 desktop kernel: vfio-pci 0000:02:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
Jan 08 00:03:51 desktop kernel: vfio_pci: add [10de:17c8[ffff:ffff]] class 0x000000/00000000

Jan 08 00:04:05 desktop gnome-shell[1474]: Failed to create backend: Could not find a primary drm kms device
Jan 08 00:04:05 desktop gnome-session[1460]: gnome-session-binary[1460]: WARNING: App 'org.gnome.Shell.desktop' exited with code 1
Jan 08 00:04:05 desktop gnome-session-binary[1460]: WARNING: App 'org.gnome.Shell.desktop' exited with code 1
Jan 08 00:04:05 desktop gnome-session-binary[1460]: Unrecoverable failure in required component org.gnome.Shell.desktop
Jan 08 00:04:05 desktop gdm-launch-environment][1383]: pam_unix(gdm-launch-environment:session): session closed for user gdm
Jan 08 00:04:05 desktop gdm3[1372]: GdmDisplay: display lasted 0.430835 seconds
Jan 08 00:04:05 desktop systemd-logind[915]: Session c1 logged out. Waiting for processes to exit.
Jan 08 00:04:05 desktop systemd-logind[915]: Removed session c1.
Jan 08 00:04:05 desktop gdm3[1372]: Child process -1451 was already dead.
Jan 08 00:04:05 desktop systemd[1]: Received SIGRTMIN+21 from PID 403 (plymouthd).
Jan 08 00:04:05 desktop gdm3[1372]: Child process -1451 was already dead.
Jan 08 00:04:05 desktop systemd[1]: Started Hold until boot process finishes up.

Jan 08 00:04:05 desktop kernel: vfio-pci 0000:02:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
Jan 08 00:04:05 desktop kernel: vfio-pci 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=io+mem

It appears that gnome is failing to load and this results in the system shutting down, but the computer remaining powered on.

In the case where IOMMU is disabled, I get the following:

Jan 08 00:05:36 desktop kernel: vfio-pci: probe of 0000:02:00.0 failed with error -22
Jan 08 00:05:36 desktop kernel: vfio_pci: add [10de:17c8[ffff:ffff]] class 0x000000/00000000

Which is the error code you seem to get when IOMMU is unavailable.

I am running sudo update-initramfs -u -k 4.16.18-041618-generic and sudo update-grub after making any changes.

I hope someone can help me. Thanks.

Edit:

Finally figured out my problem. I thought you were supposed to use the addresses for both GPUs. You should have the addresses for the GPU and Audio controller only of the device you wish to passthrough.

90

Hi @wendell and #everyone , I’ve just finished setting up a VM - launch it and get into a EFI grub menu rather than windows, I’m assuming I’m doing something incorrectly with the drive:

This is the output:

UEFI Interactive Shell v2.2
EDK II
UEFI v2.70 (EDK II, 0x00010000)
Mapping table
     BLK0: Alias(s):
          PciRoot(0x0)/Pci(0x1,0x0)/Floppy(0x0)
     BLK1: Alias(s):
          PciRoot(0x0)/Pci(0x1,0x0)/Floppy(0x1)
     BLK2: Alias(s):
          PciRoot(0x0)/Pci(0x9,0x0)/Sata(0x0,0xFFFF,0x0)
     BLK3: Alias(s):
          PciRoot(0x0)/Pci(0x9,0x0)/Sata(0x0,0xFFFF,0x0)/HD(1,MBR,0x073C329B,0x800,0xFA000)
     BLK4: Alias(s):
          PciRoot(0x0)/Pci(0x9,0x0)/Sata(0x0,0xFFFF,0x0)/HD(2,MBR,0x073C329B,0xFA800,0x19000000)
     BLK5: Alias(s):
          PciRoot(0x0)/Pci(0x9,0x0)/Sata(0x0,0xFFFF,0x0)/HD(3,MBR,0x073C329B,0x190FAFC1,0x2119A83F)
     BLK7: Alias(s):
          PciRoot(0x0)/Pci(0x9,0x0)/Sata(0x0,0xFFFF,0x0)/HD(4,MBR,0x073C329B,0x3A296000,0xEF000)
     BLK6: Alias(s):
          PciRoot(0x0)/Pci(0x9,0x0)/Sata(0x0,0xFFFF,0x0)/HD(3,MBR,0x073C329B,0x190FAFC1,0x2119A83F)/HD(1,MBR,0x00000000,0x190FB000,0x2119A800)
Shell> SC in 1 seconds to skip startup.nsh or any other key to continue.
Shell>

I’m currently trying to set the VM to boot from a SSD I’ve got plugged into sata port x on the mb. Linux sees it as /dev/sdd and it detects there is a Windows install on that device. My VM config for the drive looks like this:

<disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/sdd'/>
      <target dev='hda' bus='sata'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>


Device info 

Device     Boot     Start       End   Sectors   Size Id Type
/dev/sdd1  *         2048   1026047   1024000   500M  7 HPFS/NTFS/exFAT
/dev/sdd2         1026048 420456447 419430400   200G  7 HPFS/NTFS/exFAT
/dev/sdd3       420458433 975788031 555329599 264.8G  f W95 Ext'd (LBA)
/dev/sdd4       975790080 976769023    978944   478M 27 Hidden NTFS WinRE
/dev/sdd5       420458496 975788031 555329536 264.8G  7 HPFS/NTFS/exFAT

I would like to passthrough a nvme but the problem is it’s hard to determine what are my NVMe’s IOMMU/PCI tags as they are WD Black and don’t appear correctly or rather there is no mention of them in the lspci list/script list when grepping.

Here is the output from the IOMMU list script:

IOMMU Group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 10 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
IOMMU Group 11 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 59)
IOMMU Group 11 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 12 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0 [1022:1460]
IOMMU Group 12 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1 [1022:1461]
IOMMU Group 12 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2 [1022:1462]
IOMMU Group 12 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3 [1022:1463]
IOMMU Group 12 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4 [1022:1464]
IOMMU Group 12 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5 [1022:1465]
IOMMU Group 12 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric Device 18h Function 6 [1022:1466]
IOMMU Group 12 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7 [1022:1467]
IOMMU Group 13 00:19.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0 [1022:1460]
IOMMU Group 13 00:19.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1 [1022:1461]
IOMMU Group 13 00:19.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2 [1022:1462]
IOMMU Group 13 00:19.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3 [1022:1463]
IOMMU Group 13 00:19.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4 [1022:1464]
IOMMU Group 13 00:19.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5 [1022:1465]
IOMMU Group 13 00:19.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric Device 18h Function 6 [1022:1466]
IOMMU Group 13 00:19.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7 [1022:1467]
IOMMU Group 14 01:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ba] (rev 02)
IOMMU Group 14 01:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b6] (rev 02)
IOMMU Group 14 01:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b1] (rev 02)
IOMMU Group 14 02:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU Group 14 02:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU Group 14 02:05.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU Group 14 02:06.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU Group 14 02:07.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU Group 14 04:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
IOMMU Group 14 05:00.0 Network controller [0280]: Intel Corporation Device [8086:24fb] (rev 10)
IOMMU Group 14 06:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
IOMMU Group 15 08:00.0 Non-Volatile memory controller [0108]: Sandisk Corp Device [15b7:5002]
IOMMU Group 16 09:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104 [GeForce GTX 1070] [10de:1b81] (rev a1)
IOMMU Group 16 09:00.1 Audio device [0403]: NVIDIA Corporation GP104 High Definition Audio Controller [10de:10f0] (rev a1)
IOMMU Group 17 0a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:145a]
IOMMU Group 18 0a:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor [1022:1456]
IOMMU Group 19 0a:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) USB 3.0 Host Controller [1022:145c]
IOMMU Group 1 00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
IOMMU Group 20 0b:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:1455]
IOMMU Group 21 0b:00.2 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 22 0b:00.3 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) HD Audio Controller [1022:1457]
IOMMU Group 23 40:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 24 40:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
IOMMU Group 25 40:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 26 40:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 27 40:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
IOMMU Group 28 40:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 29 40:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 2 00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
IOMMU Group 30 40:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
IOMMU Group 31 40:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 32 40:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
IOMMU Group 33 41:00.0 Non-Volatile memory controller [0108]: Sandisk Corp Device [15b7:5002]
IOMMU Group 34 42:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480] [1002:67df] (rev c7)
IOMMU Group 34 42:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 580] [1002:aaf0]
IOMMU Group 35 43:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:145a]
IOMMU Group 36 43:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor [1022:1456]
IOMMU Group 37 43:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) USB 3.0 Host Controller [1022:145c]
IOMMU Group 38 44:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:1455]
IOMMU Group 39 44:00.2 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 3 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 4 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 5 00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
IOMMU Group 6 00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 7 00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 8 00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
IOMMU Group 9 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge [1022:1452]

I’m getting successfull output from the GPU on a secondary monitor when launching the vm - which leads me to believe that that is fine.

Specs listed below:

Ryzen 1920x - Asrock Taichi x399 (full size)
8 GB Ram (I’m getting more just using this to setup)
RX 480 8 GB (host)
GTX 1070 (passhtrough) - *I’ve done the code 43 workaround
2 x WD Black 500 GB NVMe drives running linux off of one want the other for windows
2 x 4 TB Drives (Raid 1) - irrelevant
PSU - irrelevant

I have no other devices connected to the MB.

Apologies for the long post - first time poster here !

If you need config files etc. please ask.

92

Hi all,

I was wondering if anyone could offer some advice - I’m currently up to “We downloaded the Windows ISO from Microsoft and set it up using this method.”

The problem is that the virtual machine has a black screen. I’m not sure where I’ve gone wrong as I can clearly see that Kernel driver in use: vfio-pciis set correctly for both devices. I’m not sure what logs I should be looking at in /var/lognor what I should grep.

Any advice on how to troubleshoot this would be appreciated.

  • Ubuntu 18.04, Dell XPS 13 with Intel iGPU
  • Radeon RX 580 passthrough to Windows 10 VM
93

I am trying to setup a raw disk as my boot device. I used these settings in the xml file.

<disk type='block' device='disk'>
  <driver name='qemu' type='raw' cache='none'/>
  <source dev='/dev/disk/by-id/nvme-Samsung_SSD_970_EVO_1TB_S467NX0KC27998E'/>
  <target dev='vda' bus='sata'/>
  <boot order='1'/>
  <address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>

The raw disk is an nvme drive. It boots to grub.

And ideas how to make this work?

EDIT
I think I’ve discovered what is wrong. My mbr is on the nvme drive and grub is loaded on a ssd drive so when I pass through the nvme grub is no where to be found. I am not sure the best way to fix this so I’ll post a question in the linux software forum.