GPU Passthrough? Why? Split Everything with Containers! Yes, Gaming!

Operating Systems & Open Source Linux
1

As an avid watcher of the L1 techs YouTube channel, I love ingesting information on edge use cases of enterprise or HEDT hardware and applying some of that to my homelab situation. Starting around 2020, I began using VFIO for passing through my GPUs to a Windows VM for remote gaming and image processing.

As Wendell mentioned, with every major kernel revision and new GPU generation, the guide keeps changing, and the bugs are aplenty. I’ve been able to get it to work each time, but essentially have to leave it there, “frozen in time” to ensure reliability.

I’ve decided to go a different route. Linux is now great for gaming and AI processing. Why can’t we leverage the power of having an AMD GPU that follows the DRM specification and pass /dev/dri through to individual containers, including a linux gaming one?

The basics:

Proxmox 8
Xeon E5 2690 v4
128GB DDR4 2133 in quad channel
1TB nVME SSD
1TB SATA SSD
5x 4TB Seagate Enterprise SATA HDDs
AMD RX 5700 XT
10GB NICS

I’ve followed a hundred different resources, but I’ve managed to undo my passthrough setup that blocks most of the GPU drivers and instead use the following lines in the LXC config to pass through the 5700 XT using the host’s built-in amdgpu driver:

lxc.cgroup2.devices.allow: c 226:0 rwm
lxc.cgroup2.devices.allow: c 226:128 rwm
lxc.mount.entry: /dev/dri dev/dri none bind,optional,create=dir
lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file

I then install xrdp, add an appropriately permissioned user to access /dev/dri/*, then install a desktop environment of choice.

I’ve found that Gnome, KDE, and lxqt are all no-go since they do all kinds of goofy things inside Proxmox’s containers. I’ve settled on Cinnamon, which I’ve modded to look a lot like Gnome, my DE of choice.

Lots of other little things need tweaking, like certain permissions, installing additional packages that aren’t included in the meta package for Cinnamon, installing Flatpak manually and using mostly Flatpak apps to avoid messing with the base system too much and getting a lot of Gnome apps without actually installing Gnome and screwing up the DE.

Lastly, I installed Sunshine from Flatpak, which allows me to GAME! It uses /dev/dri for hardware acceleration, and is extremely low latency and seemingly reliable. I can’t say the same about the DEB or AppImage packages, which both have issues using anything but the software renderer, but the Flatpak does seem to do the trick.

Steam is installed and running games!

At the same time I’m running my gaming DE container, I have AgentDVR encoding/decoding camera footage, Jellyfin playing/encoding movies, CodeProject.ai running AI things/stuff, and whatever else I want for the future. I’m even running a KVM with PopOS! as my primary desktop, but utilizing the VirGL GPU display type, for OpenGL desktop acceleration.

I’m getting a guide together on getting all of this done. There’s nothing out there that I’ve seen in a single place, unless I’m missing something. There seem to be guides on xrdp and various desktops, but little to nothing on getting them hardware accelerated, low latency, and gaming, while sharing with the host and other containers.

Let me know what you think. Have you tried this? Is there another resource out there that I’m missing? Why aren’t we doing this?

Thanks!

7 Likes
2

Yeah we probably can’t have too many guides for this kind of stuff. Not sure i fully understand what you’re proposing here or if it’d work in my own case (Dell PowerEdge R720 running Proxmox 8 with an older Quadro).

On my latest attempts playing around with this sort of thing, the xrdp part seemed to be the weak link. GPU performance was apparently fine, but CPU performance went to crap due to the RDP. This was at 2560x1440 though - lower resolutions seemed to work better I guess.

1 Like
3

When I first started VFIO/passthrough in 2019, I had very little idea what it’s and what it’ll be like. It’s purely out of curiosity. After many many moons, now I know what it’s and how it works. I probably won’t want to use VFIO/passthrough. But since I’m so invested emotionally and time-wise on VFIO/passthrough, it’ll be a waste to throw it away. That’s probably one of the main reasons I continue to do VFIO/passthrough at home. If I can offer one advice to other people, it’ll be a way better experience to have a couple of smaller/less power hungry bare metals including e.g. SBCs than a big machine with a couple of VMs including VFIO/passthrough.

By the same token, since I already understand what “container” is and the fundamentals of how it works, personally I won’t drill down this rabbit hole. I can see many use cases in a multi-user environment. Also uses in a hostile compute environment. And other people here commonly quote in the name of laboratory and complimentary to their day jobs. So I fail to see what are the benefits of containerisation on a single-user environment like at home. I would suggest before you go into the HOW-TO, perhaps you should write a couple of paragraphs on the WHY-TO to arouse the motivation in your audience.

4

As a hobbyist game dev and as someone developing Linux based embedded products, one big advantage in a home space I could see is to bake in a game into a virtual machine and then just run said VM instead of the game in question.

Advantages? How about dedicated cores and RAM for the game, per game optimizations in the VM instead of in the drivers and easier distribution of the game itself?

So there are use cases even on a home desktop. :slightly_smiling_face:

1 Like
5

Legitimately I want to use this in order to go from thin clients for kids into gaming thin clients for kids, home-baked cloud services, in a sense.

1 Like
6

I can see the advantage if you want

a) to get rid of Windows entirely for gaming
b) if you want to run a really slimmed down container for the game streaming without additional overhead of a VM
c) if you want a environment which is build around containerisation and virtualisiation and not bend a Desktop distro for this

Locally, you could accomplish the same running the games on a Linux Desktop distro.

7

So let’s start from square one. This point is exactly a strong reason against running the same thing inside LXC (or more generically let’s call it “containerisation”).

In a single-user environment, what are the merits/real benefits of running in LXC rather than without LXC? That was my original question in my previous post.

8

You gain isolation and the ability to run steam/games in a different runtime than your host OS. E.g. run games in ubuntu runtime on an arch system. Not have shady drm wine apps running on the host OS with possibly access to your whole home directory.

1 Like
9

You gain isolation

Isolation and quota on resource are the main points of containerisation. The question is why do you need isolation and impose quota

ability to run steam/games in a different runtime than your host OS. E.g. run games in ubuntu runtime on an arch system.

I’ve been running a Debian environment inside my Arch Linux for more than a decade. LXC was not born back then. I don’t mean LXC has no merit in this case. Without it, same thing can be accomplished. Just saying.

Not have shady drm wine apps running on the host OS with possibly access to your whole home directory.

Create a new user account. While the machine has one user that’s you. You can have multiple accounts. One for steam, one for stream, one of scream, another one for s…what not

1 Like
10

So you mean you cannot see the advantage of doing it something like this…?

linux-to-vm

11

Well…first thanks for your previous response to my question. But your answer did not address my question, actually seems far from it. I didn’t answer in the benefit of not derailing this thread.

Now with this funny picture. Would you elaborate what you mean by this briefly?

12

Quite simple, really. You make a small Linux distro VM or container that contains The Game™, in the picture above, that is csgo and starfield as two examples. Thus, runtime is known beforehand → no need to worry about binary compatibility. At all. If the VM can run it, the game can run it.

These are then passthrough’d to a vGPU that acts as a MUX - sends all GPU render calls to /dev/null but returns OK unless it is the active connection (could probably happen through a pipewire interface even and only lose 50 µs latency or so).

The small Linux kernel or Linux layer in the vm can be altered to fix any game-breaking bugs and do on-the-fly translations from borked OpenGL / Vulkan calls to unborked calls.

Same architecture works for controllers, network and Audio too of course. The above image only shows GPUs right now. Advantages? How about per game dedicated cores, dedicated RAM, dedicated VRAM, network packets…?

You could do it in Linux only, true, but only in a janky, non-standard way that is difficult to replicate on a different distro. The above image is an example architecture where you would not have to care what host OS you run, it’s all in there.

13

The benefit of LXC is lightweight i.e. light very light not much weight. One essential feature of containerisation. One key element being light not much weight is sharing the same kernel with the host.

I’ve seen some people trying to run a VM inside a LXC. yeah. cool. fun. What’s the point? Now they run a kernel on top of the host kernel. Tech demo for sure. Lose the light not much weight of LXC.

Can I summarise what you said with the description above?

14

Most embedded Linux distros are between 50-200 MB in size.

If you can make a VM with 50-200 MB overhead on a 20 GB game, and that will lose you 5% performance but will cut down on your tech support by 50%… Should you?

To be fair I want it to be either container or VM, both is a bit excessive. That said, a pure QEmu VM running virtualized is awesome, and even better is if the OS has a hypervisor that allows you to compartmentalize into a game part and host part.

We are still only starting to realise what can be done with virtualisation on the desktop, now that we have a ton of cores on every desktop. Let’s see how this plays out. I would not be too quick in judgement right now.

1 Like
15

did anyone manage to pass gamepads via moonlight?

16

I realize this is a few months old thread but wanted to chime in and ask for help. I was able to get everything working, video and audio streaming through sunshine, except for the virtual controls. This is the last issue that I can’t seem to overcome. Here is the error I cannot figure out how to get past.

From sunshine:
uinput: Failed to write ‘change’ to ‘/sys/devices/virtual/misc/uinput/uevent’: Read-only file system

I’ve tried chmod and chown but neither worked since it’s read only. I’m a newb though so set me straight if I’m way off.

Using Proxmox 8.2.4 with ubuntu 22.04 lxc running in priveleged mode.

Here’s my lxc.conf file:

arch: amd64
cores: 8
features: fuse=1,mount=nfs;cifs,nesting=1
hostname: Desktop-LXC
memory: 12288
net0: name=eth0,bridge=vmbr0,firewall=1,gw=192.168.1.1,hwaddr=BC:24:11:D9:1A:C2,ip=192.168.1.221/24>
ostype: ubuntu
rootfs: local-lvm:vm-205-disk-0,size=256G
swap: 2048
lxc.cgroup2.devices.allow: c 226:2 rwm
lxc.cgroup2.devices.allow: c 226:129 rwm
lxc.cgroup2.devices.allow: c 13:* rwm
lxc.cgroup2.devices.allow: c 10:223 rwm
lxc.mount.entry: /dev/input dev/input none bind,optional,create=dir
lxc.mount.entry: /dev/uinput dev/uinput none bind,optional,create=file
lxc.mount.entry: /dev/dri/card2 dev/dri/card2 none bind,optional,create=file
lxc.mount.entry: /dev/dri/renderD129 dev/dri/renderD129 none bind,optional,create=file

From what I can find on the web, that uinput/uevent path that I listed above is a module that is hard baked into the proxmox kernel which is 6.8.8-1.

CPU: AMD 5900x
GPU: AMD 6700XT

I also have an Nvidia RTX 2060 but the 6700XT is the one I want for this LXC container.

Anyone know how to get around this?

17

LucyCharms at the risk of make mods mad and restarting this thread. It is my experince that amd gpu are giant pain. To get work though passthough. Best I have ever gotten my gpu 5700xt to due is to only kind work. That was actually in esxi but would likely have the same issue with proxmox. The issue I have always run into card reset. Nvidia card reset via power though pcie bus. Amd card to correct reset require command. So when I have played vifo on 5700xt usually envoles the make so the card does not get shutdown command when os powers down. I usually turn off the pcie lane before the card get command.
The danger in this longer card is up more likely you are to have issue. If you don’t turn the card off your not actually clearing the card. Last time I tired also issue if though linux guest os and then windows guest os. As long as card getting same os general fine. Never had that make past 90 day online. Have machine nvidia evga 2070 super. That will have online for 274 days in row. Machine done this couple times. With out much of issue.

18

LuckyChrams the issue a suspect that you having right now is that it the primary gpu of proxmox. I suspect you would have to blacklist gpu to get work. On my promox test box. I have cheap nvidia 710 it runs proxmox from. Then run passthough intel gpu. You can do one gpu but I prefer be able to mess with the host os for troubleshooting. Also I use exsi more then proxmox. Current verison of exsi really easier to have 2 gpu. But you don’t need much of gpu. I use to always buy pcie 1x gpu or if can’t find put the host gpu in bottom slot in mainboard. Most boards it means the gpu comes off the mainboard. Not cpu.

19

Thank you for replying. I think you’re referring to the vendor reset issue, which according to some members from the proxmox group does not exist with the 6000 series AMD GPU card and it isn’t the issue for me. I have blacklisted the nouveau driver for my nvidia card and that’s working fine for my plex container with transcoding. Nouveau and Radeon are the two that I have in blacklist.conf. AMDGPU is not blackisted as that is the native amd driver and the one that is being used for my LXC. Really, the issue I’m having is with Sunshine’s need to write to /sys/devices/virtual/misc/uinput/uevent for virtual input. I cannot seem to do this however since it is a read only file system. I can’t seem to change permissions on it to allow the LXC to see it as writable.

20

Well I hope yours is different I have xfx merc 318 6900xt and xfx merc 318 7900xt both have a reset issue. Both that would be testing though esxi. So on that os I don’t have to play with that ever. Just have to make sure it does not attach as primary card.