Benchmark or Advice on Virtio-GPU / Virlgl?

picocreator · May 29, 2021, 2:28pm

IMO: Being able to run multiple virtual GPU should be a consumer security feature - however it ain’t here yet, due to enterprise locks on the GPU providers (fingers crossed on intel XE)

Virtio-GPU / Virlgl - works around these limitations by “paravirtualizing” the drivers, running them on the host as an OpenGL program. As far as the host machine is concern, its just running multiple OpenGL “applications”.

Status / Links of project?

Apparently still active as of latest blog update. Things have understandbly slowed down however this err past year.

Summary homepage
Project Git Repos: virgl
QEMU Support on linux is considered mature

Of course this will have limitations

It is limited to OpenGL calls, so directX and other drivers is ???, the focus of the project seems to be on linux desktops
Windows support is effectively a POC and needs more development
There is probably a significant overhead, how significant it is however is an unknown (anyone got a benchmark?), hopefully even at 2:1 ratio, this is useful enough to accelerate desktop work, or even older games (probably painful on newer games)
Does not seem to have much control knobs to neatly partition the GPU, this might not be an issue for a trusted environment. But can be an issue if one user decides to run a crisis, while another user is trying to watch youtube.
Security? Without the hardware level sandboxing? Who knows - really a huge unknown right now, probably a deal breaker even to many - Sadly its for this same reason that Microsoft is killing off its equivalent project in Hyper-V

What can be done?

Got a linux host, running a good graphic card: Benchmark !
Already got this setup?: Whats your experience / advice on it, what potholes should be avoided
Team / Project really needs more love and support

Running full desktop machines on VM’s on docker containers, qubes, etc - have been my obsession lately, however like most folks I do not have access to the enterpisy cards.

Planning to further explore this on a 6700XT once it arrives (backorder, about a month). Meanwhile installing Ubuntu on a Thrashcan 2015 MacPro with an AMD GPU while waiting for the 6700XT to arrive to try this - but that got its own err… Apple problems.

Update on the 6700XT

WorBlux · May 30, 2021, 7:40pm

There is also initial/experimental support for vulkan, and recently the Linux DRM/Mesa have been looking to integrate explicit fencing, that makes the memory model look a bit more like DX12, and I suspect it will allow more performant para-virtualization.

It’s certainly not as secure as SR-IOV, but I suspect it is more secure than a software jail like Docker. It’s not an enterprise solution, but I wouldn’t hesitate to use it in a pinch.

picocreator · May 31, 2021, 9:17am

@WorBlux - Definitely, considering that such virtio GPU usage is currently not widespread in the wild - current security concerns is really more theoretical then practical - and frankly anyone who is that concern of security at this level : probably can afford the enterprise cards

Got any links regarding the explicit fencing? Mesa and Vulkan are sadly still really less then ideal google keywords.

WorBlux · May 31, 2021, 2:07pm

What? Are you telling me that you don’t want to go to a Treck convention in Arizona?

And Article and .pd from when it was introduces to mainline a few years ago.

http://events17.linuxfoundation.org/sites/events/files/slides/Explicit-Fencing_Talk_0.pdf

And the more recent news of AMD shifting it’s driver model to use it, and that brought my attention to it.

https://www.phoronix.com/scan.php?page=news_item&px=AMD-Explicit-Fences-Everywhere

picocreator · June 1, 2021, 7:50am

Or maybe, google knows too well - that i would want to go

Hmm, i’m actually surprised this is motivated by performance improvements - i would thought fencing would cause a penalty.

Seems like a long haul though (3 years +?) but its a step in the right direction, given the number of non-graphical computation workload that’s happening these days.

picocreator · June 18, 2021, 1:41am

Update - the 6700XT is finally here - woohoo (after a whole month backorder)

So i probably will get testing this weekend.

Also apparently for windows, a similar project for GPUP is already working on hyper-v for windows

and

picocreator · June 20, 2021, 10:11am

Update Summary - It works great for office dekstop environments !!! (but not games)

8 VM’s running HD youtube videos off a single shared 6700XT.

It seems like the latest virtio-gpu-pci (pcie) does work, at about ~50% ish efficiency. With one huge gotcha

Each VM seems to be capped at approximately an “integrated graphics card” level of performance.

While this is a deal breaker if you are trying to play modern games. Its already an incredible huge uplift from CPU based GPU in terms of being able to move desktop browsers around and watch full HD video’s at 60fps.

Another catch, is the options are not supported by the libvirt UI, and I can’t seem to get the libvirt XML working. So you would need to elbow grease the QEMU command line.

In overall, this does seem like a promising compromise for a VM heavy mixed system with vGPU unlock for nvidia on heavy GPU/Gaming workloads - while using virtio for non-gaming VM’s

Video playback of individual full screens.

PS: If anyone knows how to tune the “vitio-gpu-pci” which seems to lack any control knobs, to use as much of the GPU as possible - do let me know - so i can retest this.

picocreator · June 20, 2021, 10:11am

Update In Details - What I done in chronological order (this is not a how to guide)

To start with, when my new prebuilt 6700 XT came in, I did a fresh Ubuntu 20 LTS install and updated it accordingly, installed the required drivers (it did not install out of the box) and gave it a phoronix benchmark to form a baseline, before doing the VM based tests. For consistency between host and VM, i am limiting the test to only 1920 x 1080 benchmarks.

PS: if you want to try replicate, I would suggest to use Ubuntu 18 instead, as there was a fair bit of backporting needed to get some drivers to work. For install steps you can refer at the end of the post.

1) Host Specs Benchmark

GLmark2 score: 16,902
Openbenchmark upload: Qemu-glmark2 Benchmarks - OpenBenchmarking.org

There was also a test that managed to hit 18,206 - but i was testing around and didnt save the result online.

2) VM with “virtio 3D accleration” option in Virtual Machine Manager

Now that we got the baseline, the next thing was to setup a VM quickly via the virtual machine manager, with a raw disk and install ubuntu on it once again.

Under the virtual machine manager, enable virtio, and 3d acceleration (screenshot stolen from here - as i forget to take a screenshot for this step)

Disappointingly the phoronix test got us a potato score of : 195

Openbenchmark Upload: Qemu-glmark2 Benchmarks - OpenBenchmarking.org

From what I understand (I need someone with better knowledge to confirm) this seems to be using the virtio-vga driver, which provides more backwards compatibility at the cost of performance.

Because I could not get the virtio-gpu-pci option to work in the UI, this is where the elbow greasing on the XML (failed) and the command line begun (worked)

3) Attempting to modify libvirt-server xml files

Im just gonna skip this part … as nothing worked properly, if someone knows how to configure this properly do let me know.

4) Switching to QEMU command line

Taking the same raw image file used in 2, manually craft the QEMU command line with the same settings (or as close to it as I could)

Somehow in the process the potatoe got worse even if I manage to get it to switch from llvmpipe, to virgl driver in the process.

Openbenchmark Upload: Qemu-glmark2 Benchmarks - OpenBenchmarking.org

5) Tweaking around at the command line

After testing multiple time, and going back and forth the settings inside the VM (listing pcie devices, etc) I realized that “by default” the VGA GPU is always added. Degrading the performance accordingly, so once i got that removed.

Well VNC stopped working (i guessed it depended on the VGA?) besides it was suggested to use the virtio-gpu-pci with gtk in various examples, so i made the switch as well to use it with spice.

And success ! The score is now brought up to : 800

virtio-800

This is comparable to older integrated graphics, and is generally “good enough” for HD video playback and office work. The result however was disappointing from a gaming perspective.

However what i noticed was that, when putting the VM through the paces of testing multiple times - at no point was the GPU ever fully loaded. So time for the next part …

6) Run 8 VM with glmark2 at the same time !

Note that I suspect I might be able to push it slightly higher, as the GPU clock has not been saturated to 100% most of the time, but at this point I’m slowly running out of RAM

Here is the confusing part, it actually performed better - on average the score for all 8 instances were between 900 ~ 1050.

I have validated this increase, by rerunning twice all 8 tests, and the single test separately.

one-of-eight

Openbencmark Upload: Qemu-glmark2-1-of-8 Benchmarks - OpenBenchmarking.org

I suspect what is happening here, is that by pushing the GPU closer to the limit, it exists power saving mode, or something similar. But who knows =/

This however gives the VM’s about a total combined score of 8k (1k * 8), which is half of the host 16k - so at about 50% efficiency (napkin math) - if anything at this point, im potentially RAM bound rather then GPU bound in my current setup.

So not bad for a start I guess?

7) Run other workloads, like youtube and stuff

One thing to note - by default Firefox does not enable GPU acceleration for linux install, so go into the config and enable. Once you have that enabled.

Things worked smoothly

Fun aside, i misconfigured the original ram limits - and when opening the browsers, they happily went to take up large chunks of ram, and started crashing one VM after another. - Meaning starting the browser takes up WAY more ram then running GPU benchmarks (hahaha)

picocreator · June 20, 2021, 10:12am

Install Commands / Running Commands

Reminder: This is not a HOW-TO, its more of notes for other users who is familiar with QEMU/KVM to replicate the setup. I will not be updating / maintaining the links to newer package versions

First update and upgrade your system

sudo apt-get update
sudo apt-get upgrade
sudo apt-get dist-upgrade

Check if the graphics card is properly installed

Run the following

sudo lshw -C display

If the GPU shows up as display UNCLAIMED - this could be due to an installation bug with the driver. You will need to install it manually. For ubuntu 20 refer to:

https://www.amd.com/en/support/kb/release-notes/rn-amdgpu-unified-linux-21-10

Install vulkan SDK

Refer to : LunarXchange

(Note if you are doing this after june 2021, you may want to check for a newer version)

wget -qO - http://packages.lunarg.com/lunarg-signing-key-pub.asc | sudo apt-key add -
sudo wget -qO /etc/apt/sources.list.d/lunarg-vulkan-focal.list http://packages.lunarg.com/vulkan/lunarg-vulkan-focal.list
sudo apt update
sudo apt install vulkan-sdk

PS: You probably should refer to the original source, instead of trusting a random person on the internet with this copy and paste.

Install mesa, and glmark2

sudo apt install mesa-utils glmark2

Install phoronix test suite

sudo apt install gdebi-core

Then download the package from the phoronix website : Phoronix Test Suite - Linux Testing & Benchmarking Platform, Automated Testing, Open-Source Benchmarking

Or alternatively

wget http://phoronix-test-suite.com/releases/repo/pts.debian/files/phoronix-test-suite_10.4.0_all.deb

Then install the package

sudo gdebi phoronix-test-*.deb

Running the test

phoronix-test-suite benchmark glmark2

Install QEMU

sudo apt install qemu-kvm libvirt-clients libvirt-daemon-system bridge-utils virt-manager

Extra: Install radeontop to see GPU Usage

For other GPU: software recommendation - How to measure GPU usage? - Ask Ubuntu

sudo apt install radeontop

How to run the QEMU VM

Create a bash script (easier to edit and manage) with the following

#!/usr/bin/env bash

# Qemu arguments, this is intentionlly an array
# so that i can easily add in comments between args
args=(
        #
        # Basic CPU and ram allocation
        #
        -enable-kvm
        -cpu host
        
        # Number of CPU threads
        -smp 12

        # 16 GB ram allocation
        # -m "16384" 
        # 2 GB ram allocation
        -m "2048"

        #
        # USB mouse & keyboard
        #
        -usb -device usb-kbd -device usb-tablet
        -device usb-ehci,id=ehci

        #
        # RNG Device passthrough (it does boost the test score)
        #
        -object rng-random,id=rng0,filename=/dev/urandom -device virtio-rng-pci,rng=rng0

        #
        # Graphics card
        #

        # VGA based virtio implementation (poorer performance)
        # -vga virtio
        # -display gtk,gl=on

        # PCIE Emulated VGA GPU (we loose VNC)
        -vga none
        -device virtio-gpu-pci,virgl=on
        -display gtk,gl=on

        #
        # VNC terminal
        #
        # -vnc 0.0.0.0:0

        #
        # Sata Driver
        #
        -device ich9-ahci,id=sata

        #
        # Snapshot mode (readonly the disk image)
        # !!! Use this if you want to run multiple instances in different terminals
        #     Alternatively, obviously comment this out if your modifying the image
        #
        -snapshot

        #
        # The main OS disk - Replace the path with YOUR image path
        #
        -drive id=MainHDD,if=none,format=raw,file="/home/qemu-test/Desktop/qemu-virtio-test/ubuntu-disk.raw"
        -device ide-hd,bus=sata.1,drive=MainHDD
)

# Run the qemu system
sudo qemu-system-x86_64-spice "${args[@]}"

You will need to enable WebGL for browsers

Grendel · July 8, 2021, 6:25pm

I’m actually interested/surprised you say video playback is improved. I don’t know much about virgl but I was under the impression it just sped up openGL tasks. I thought video playback would be hardware decoders which I did not think would be exposed by this.

picocreator · July 9, 2021, 2:10am

Ahh your right about that, forgot to explain that - it depends on the video being played.

Some specific video formats, and players (like vlc?) does a decent job of detecting CPU hardware acceleration - and applying them - but frankly in linux this is a huge hit and miss : Especially for 4K video playback.

Because of that pretty much everything else seems to fallbacks to openGL acceleration (or webGL) or worse CPU rendering - as its the most “compatible” out there.

Firefox for example - ships by default in linux with webGL disabled. Most youtube videos somehow (might be firefox issue) does not use CPU hardware acceleration. Opting for CPU rendering

Grendel · July 9, 2021, 5:13pm

Thanks for that. I think I have a misunderstanding about the different methods of supporting hardware accelerated decoding.