A journey to the "2 Players 1 PC (+homelab)" setup with Proxmox and vfio

Hello world!
First post here so pls don’t be too hard on me for any mistakes :slight_smile:

Why I’m writing this post? I wanted to share my experiences with the community, I hope at least a single person will find this useful.

Why I did this setup? I’ve always wanted a setup that serves as both a homelab host and a dual gaming rig to share with my significant other. So I finally pulled the trigger. Here is the hardware:

  • Base: Used Dell Precision 7920 Tower

  • CPU: 2x Xeon Gold 6150

  • RAM: 128GB

  • GPUs: PowerColor RX 9070 XT Reaper & PowerColor RX 9060 XT Reaper

  • USB: Sonnet Allegro USB-C 4-Port PCIe Card

(yes, I know Xeons and PCIe Gen 3 aren’t the absolute peak for gaming)

So, here is how the build went and the hurdles I had to overcome.

Hurdle 1: The AMD Reset Bug

Right out of the gate, both cards suffered from the AMD reset bug. Neither the 9070 XT nor the 9060 XT supports Function Level Reset (FLR). Shutting down a VM meant the card wouldn’t reset, requiring a full host reboot. Which is suboptimal to say the least.

After weeks of troubleshooting, I adapted a script from the Proxmox forums, which I attached to each VM as a hookscript using any of those two cards to handle this issue on the VM post-stop phase:

#!/bin/bash

phase="$2"
gpu_pci_address="0000:75:00.0" # Offending RX 90_0 XT address

GetCurrentGpuDriverName()
{
        basename "$(readlink /sys/bus/pci/devices/$gpu_pci_address/driver)"
}

echo "Phase is $phase"
echo "Driver currently in use: $(GetCurrentGpuDriverName)"

if [ "$phase" == "post-stop" ]; # ONLY act when the VM shuts down!
then
        # Manually unbind the card from the vfio driver
        echo "Unbinding from vfio-pci"
        echo "$gpu_pci_address" > /sys/bus/pci/drivers/vfio-pci/unbind 2>/dev/null
        sleep 7 # Give the hardware time to settle

        # Manually bind the card to amdgpu to re-initialize it for the next VM boot
        echo "Binding to amdgpu"
        echo "$gpu_pci_address" > /sys/bus/pci/drivers/amdgpu/bind 2>/dev/null
        sleep 7 
fi

The Catch: It still occasionally bricks! I seams to only happen when one VM shuts down and another tries to grab the same GPU too quickly (like for example during backups in proxmox), the amdgpu driver or the card itself just panics with error messages like those:

vfio-pci 0000:75:00.1: Refused to change power state from D0 to D3hot

…

pcieport 0000:74:00.0: broken device, retraining non-functional downstream link at 2.5GT/s

…

amdgpu 0000:75:00.0: trn=2 ACK should not assert! wait again !

the message above like 10 times

xgpu_nv_mailbox_trans_msg: 2425 callbacks suppressed

This “ACK should not assert” loops and once this loop starts, it repeats forever. A reboot will not fix it because the amdgpu driver will prevent the kernel from shutting down itself! Imagine that! I literally have to unplug the poor PC from the wall when this happens. Luckily it is rare.

I reached out to AMD regarding all of this mess, and their response was essentially to buy the R9700, which costs literally twice as much for the exact same performance. They did log my use case, at least.

Hurdle 2: USB Passthrough and KVM Quirks

Since I had video out at this point I now needed a way to pass peripherals to those VMs reliably. The Sonnet Allegro USB-C card is perfect on paper: it has two ASMedia 3142 controllers, allowing me to pass one controller to each VM independently.

Here is the USB layout for my side of things:

And it worked flawlessly! Well… Until I started switching PCs on my Level1Techs KVM. The controller inside the VM would freeze the entire VM for like 3 seconds, then timeout, and then the kernel would kill it permanently with those error messages:

xhci_hcd 0000:01:00.0: Abort failed to stop command ring: -110
xhci_hcd 0000:01:00.0: xHCI host controller not responding, assume dead
xhci_hcd 0000:01:00.0: HC died; cleaning up
xhci_hcd 0000:01:00.0: Timeout while waiting for setup device command
xhci_hcd 0000:01:00.0: Timeout while waiting for configure endpoint command

After a lot of troubleshooting to avoid constantly rebooting the VMs, I finally managed to band-aid this and with the help of AI I wrote a watchdog script wired to a systemd service to catch the failure and perform a hardware-level reset:

#!/usr/bin/env bash

set -u
PCI="0000:01:00.0" # PCI address of the offending ASMedia controller

journalctl -kf -n0 -o cat --grep "xhci_hcd ${PCI}: xHCI host controller not responding, assume dead" |
while IFS= read -r _; do
    # This loop only iterates when the controller actually dies,
    # the call to journalctl above ensures this

    logger -t asm3142-watchdog "HC ${PCI} died; recovering"

    echo "${PCI}" > /sys/bus/pci/drivers/xhci_hcd/unbind; sleep 3  # clear the software => unbind from the kernel
    echo 1 > "/sys/bus/pci/devices/${PCI}/reset"; sleep 3          # clear the hardware => FLR (essentially reboots the device)
    echo "${PCI}" > /sys/bus/pci/drivers/xhci_hcd/bind             # bind the fresh device back to the kernel

    logger -t asm3142-watchdog "Rebind of HC ${PCI} completed"
done

It is a disgusting hack, but it kinda works. I still have to wait about 30 seconds for the timeout, reset, and device reconnection, but it beats a hard reboot. Interestingly, this seems to be a Linux-specific issue as it does happen on the proxmox host itself too but windows VMs handle the KVM switching without dropping the controller. I suspect a flaky Icy Dock USB hub might be contributing, as the KVM’s emulated HID ports don’t trigger this on their own but on the other hand maybe the KVM does something to the USB 3 connection that the ASMedia controller does not like, I cant tell at this point.

The Verdict

If you want to build a “2 Players 1 PC” setup yourself, it is possible. Me and my girl daily drive this configuration on both Windows and Linux VMs with acceptable performance while the same PC runs all of my homelab stuff in the background.

However, if you want to go for this be prepared to fight your way through this. Even in the year of our Lord 2026, this is still not plug-and-play. Expect edge cases, scripts and above all => hardware bugs.

Hope somebody finds this useful :slight_smile:

Questions? Suggestions? Hit me up

Cheers,
~Adrian

3 Likes

Hey there,
I wanted to let you know that for the reset on AMD cards you can compile the vendor-reset kernel module by our forum resident gnif: GitHub - gnif/vendor-reset: Linux kernel vendor specific hardware reset module for sequences that are too complex/complicated to land in pci_quirks.c · GitHub

It works for a quite big range of cards. I used and use it myself.

1 Like

Hello! Thanks for this suggestion :smiley:
I have seen this project before, however I did not even try this because in their readme they state that this supports up to Navi 14 - my cards are Navi 48 (9070 XT) and Navi 44 (9060 XT), but if you say that it works for you then it might be worth it to try after all. What card(s) do you have?

Currently I use it with Vega era cards, but when I had a 6900XT it still worked perfectly well and I read about people using it with 7000 and 9000 series cards as well.

1 Like