Return to Level1Techs.com

Stay Glassy with PCI Passthrough + Looking Glass -- Guide

guide
passthrough

#1

Tested using Fedora (tips (red)hat)

Here’s what I do, starting with a fresh install with compatible hardware and bios settings.

I’m laying out this document as I have personally prepared it for my own personal use if I ever needed to quickly reset the PC to a fully functional PCI-passthrough machine with a very functional Linux host that I use for daily computing + office work. I took pieces here and there (from the interwebs, this forum, etc). For me, it works, 100% of the time. Thanks for all of those involved in making this possible!


Set Repositories/Install RPM Fusion + Installing Software + Installing Sound/Video


su -c 'dnf install https://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm https://download1.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-$(rpm -E %fedora).noarch.rpm'

sudo dnf install vlc audacity pdfshuffler evince firefox gimp libreoffice openshot obs-studio filezilla p7zip-gui.x86_64 mediawriter.x86_64 simple-scan unrar vim htop nethogs screenfetch lm_sensors.x86_64 hddtemp gnome-terminal-nautilus wine @virtualization

*also check for visual studio code online

sudo dnf install gstreamer1-libav gstreamer1-plugins-good gstreamer1-plugins-ugly gstreamer1-plugins-bad-free gstreamer-ffmpeg gstreamer-plugins-good gstreamer-plugins-ugly gstreamer-plugins-bad gstreamer-plugins-bad-free gstreamer-plugins-bad-nonfree

sudo dnf install clalsadrv.x86_64 alsa-firmware.noarch alsa-lib.x86_64 alsa-utils.x86_64 alsamixergui.x86_64

sudo modprobe snd-oxygen
sudo modprobe snd-pcm-oss
sudo modprobe snd-mixer-oss
sudo modprobe snd-seq-oss
alsamixer

For DGX Sound Card
***Set multichannel
***Analog Surround 4.0

For Thinkpad Laptops
sudo dnf install tlp
systemctl enable tlp.service
systemctl enable tlp-sleep.service
sudo tlp start



*** Verification of IOMMU groupings if needed

  1. gedit ls-iommu.sh

    #!/bin/bash
    for d in /sys/kernel/iommu_groups//devices/; do
    n=${d#/iommu_groups/}; n=${n%%/}
    printf 'IOMMU Group %s ’ "$n"
    lspci -nns "${d##
    /}"
    done

  2. chmod +x ./ls-iommu.sh
    ./ls-iommu.sh



Choose your method for passthrough: pci-stub or vfio


  • Passthrough with pci-stub

  1. Check Hardware IDs

lspci -nn | grep AMD

  1. Add the following to Kernel Line in /etc/grub2-efi.cfg or /etc/default/grub - This Gives the VM (in my case) Graphics Card Video/Audio, Disables HDMI audio of Main Card and Passthrough Motherboard Audio. You passthrough whatever hardware you desire, just make sure to match your hardware IDs.

amd_iommu=on pci-stub.ids=1002:6899,1002:aa50,1002:4383

3.sudo grub2-mkconfig -o /boot/grub2/grub.cfg

  1. Reboot

  2. To Check if pci-stub Worked

dmesg | grep pci-stub


  • Passthrough with VFIO

  1. Add to Kernel Line in /etc/default/grub

iommu=1 amd_iommu=on rd.driver.pre=vfio-pci

  1. To Get IDs

lspci -nn

  1. Add in /etc/modprobe.d/kvm.conf and comment out kvm stuff

options vfio-pci ids=1002:6899,1002:aa50,1002:4383

  1. Create a vfio.conf in /etc/dracut.conf.d with

add_drivers+=“vfio vfio_iommu_type1 vfio_pci vfio_virqfd”

  1. dracut -f --kver `uname -r`

  2. grub2-mkconfig > /etc/grub2-efi.cfg

  3. Reboot



Setup for Virtmanager



CPU allocation: 8 out of 8
CPU model: Hypervisor Default
Don’t copy host configuration
Manually set CPU topology : 1/8/1
RAM: 16000 MB
LAN: macvtap enp7s0 VEPA rtl8139
Disks: /dev/sdd and /dev/sde
UEFI

  • I’m using an AMD 8320 and 2 dedicated disks for the Windows VM. You’ll have to play around with this as you see fit with your own hardware. At least it’s a GUI, amirite. Here, you tell the software which devices you want to assign to it. If you’re curious, the GPUs I have tested this with are R9 280 and an old HD 5850.


Setup for LookingGlass



  1. sudo dnf install git SDL2-devel SDL2_ttf-devel openssl-devel spice-protocol fontconfig-devel libX11-devel gnu-free-mono-fonts ivshmem-tools libgle-devel.x86_64

  2. sudo dnf groupinstall "Development Tools" "Development Libraries"

  3. git clone https://github.com/gnif/LookingGlass.git

  4. cd LookingGlass/client

make

  1. sudo usermod -aG qemu yourusername

  2. Edit /etc/libvirt/qemu.conf and set

user = "qemu"
group = "qemu"
dynamic_ownership = 1

  1. Edit /etc/selinux/config and set to permissive

  2. virsh edit WindowsVMname

add the following under kvm (next to it add xmlns:qemu=‘http://libvirt.org/schemas/domain/qemu/1.0’)

qemu:commandline
<qemu:arg value=’-device’/>
<qemu:arg value=‘ivshmem-doorbell,chardev=ivshmem,vectors=1’/>
<qemu:arg value=’-chardev’/>
<qemu:arg value=‘socket,path=/home/yourusername/ivshmem_socket,id=ivshmem’/>
</qemu:commandline>

  1. sudo -u qemu ivshmem-server -p /tmp/ivshmem.pid -S /tmp/ivshmem_socket -l 16M -n 8

If you need to reset server at some point
sudo rm /tmp/ivshmem_socket

  1. Give read/write access to /tmp/ivshmem_socket

sudo chmod 600 /tmp/ivshmem_socket

  1. Run VM on Linux

  2. Install drivers and exe on Windows

  3. Run exe on Windows

  4. Run
    ./client/bin/looking-glass-client



Have fun!

I’ll update as recommendations come in.


#2
<qemu:arg value=‘socket,path=/home/yourusername/ivshmem_socket,id=ivshmem’/>

sudo -u qemu ivshmem-server -p /tmp/ivshmem.pid -S /tmp/ivshmem_socket -l 16M -n 8

You’re telling virt-manager to look for ivshmem_socket in your home directory but creating it in /tmp


#3

whenever you have code you can use this symbol - `
This is a short tutorial


#4

Fixing this thanks


#5

Thanks, looks better


#6

Currently trying to get this running with arch linux. Managed to build client. Running ivshmem-server with: sudo ivshmem-server -F -v -p /tmp/ivshmem.pid -S /tmp/ivshmem_socket -l 32M -n 8

After that chmod 777/770 /tmp/ivshmem_socket etc…

Somethings seems to be connecting. Output when starting virtual machine terminal gets following:
accept()=5
new peer id = 0
peer->sock_fd=5
peer->sock_fd=5
free peer 0

But the problem is that when i try to start host program in windows 10 it errors for me:

Unable to enumerate the device, is it attached? and fails to start service on windows side.

http://pastebin.com/4aC6VsN0

Using latest alpha6 windows binary build.

Attached GPU: Gigabyte gtx 750 10de:1381 with error 43 fixed in xml.
Monitor output: 37" fullhd tv via hdmi and spice virtual "screen"
OS: Windows 10 home 64bit
MB: x370 prime pro
CPU: Ryzen 7 1700
Linux host GPU: AMD rx 480


Edit:

Solved the problem for windows host. Forget that needed to install the patched IVSHMEM Driver on windows side from https://github.com/virtio-win/kvm-guest-drivers-windows/issues/217


#7

770 = read, write & execute for owner and group members… do you really want to let people try to run the shared memory mapping as an executable?

660 = read, write for owner & group members, or better yet, 600 = read for owner.

Please learn how linux filesystem security works before you post such guides.


#8

That should be fine because most people are running the system with just one user account and most of the time the default user group of that user is also the user name.

I don’t think supplementary groups would still have access? Unless they were running with privileges from that user account?


#9

Remember that this shared memory is a weak point that could allow malicious software to break out of the VM into the host. I would suggest ensuring it is as secure as possible and not adopt the idea that it’s only one user.


#10

Ahh yes. Maybe this should be taken care of by a separate user/group. As I’m sure by default the primary user will belong to the sudo group.


#11

Could we use a chroot jail for /tmp/ivshmem (or something along those lines?).

Based on my limited knowledge of Docker containers (as a user, not a developer), I understand this is based on lxc, which “essentially” locks processes/permissions into its own closed space (cgroups).

Similarly, thoughts on a Docker looking glass image?

You may have to do something like this though to allow passing through container > host > framebuffer comms (link: https://github.com/NVIDIA/nvidia-docker).

CC @sgtawesomesauce it would be cool do be able to do docker run gnif/looking-glass (or docker-compose up with the same configured, and have this as a truly turn-key solution.

Of course, this is after all the UEFI/grub configs are in place; the Docker image would have pre-baked -

  • Fetching the latest source from Github
  • Compiling the looking-glass binary
  • Automating virsh setup (or this may have to be done on the host side)
  • Keep the ‘sealed’ contents inside the booted container and only allow access to the host where needed (?)

^^ What I’m on about here isn’t typical Docker use, but referring to it as an example; in place of lxc containers.

There’s a quick example here to play with a very basic lxc container https://linuxcontainers.org/lxc/documentation/


#12

Yeah, might as well also install gentoo so I can facebook better.


#14

when i type make, it says:
make: *** No targets specified and no makefile found. Stop.
running fedora 29, i installed fedora because i heard that qemu 3.0 included a patch for amd smt bug that allowed amd processors with smt to use it in a vm, im running amd threadripper 1950x and gtx 1080 ti. i saw a 30 fps gain coming from ubuntu which is running qemu 2.16, and yes ive tried to install qemu 3.0.0(on ubuntu), but like using Looking glass on fedora there are very few articles i could find to install it, and when i did install it it just made everything break. So im sticking with fedora, but id like to use looking glass.


#15

https://looking-glass.hostfission.com/quickstart/linux/build


#16

i will try this, thank you
and thank you for your work its amazing