Proxmox 9.0 + Intel B50 SR-IOV -- finally its almost here! Early Adopter's Guide

Have the same board… Tyan is lying, or at least its not FULLY enabled by default on the S8030. For what is working (sorta, still testing) for me: https://www.reddit.com/r/Proxmox/comments/1nm3mpw/comment/nmt0keo/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button

Check out this link for how I got it working… still testing but initial results are promising:

https://www.reddit.com/r/Proxmox/comments/1nm3mpw/comment/nmt0keo/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button

Making some more progress. I had to move the VM out of the qemu/kvm user session to the root qemu/kvm session for more permissions. After doing that, the VM started. I’m working on installing the graphics driver from inside the VM. Pretty excited to see this much progress with the iGPU on Meteor Lake.


Update

More success!!

has anyone tried the B60 yet for this same usecase?

I think the limited availability will make it a little less common at the moment

I can’t imagine it’s much different, it’s literally the same chip, just the B50 is cut down.

At the moment in Europe there is availability of the asrock model but it’s going at 700+ where the B50 is 350-400, so not great value at the moment.

Do you have any details on how you got this working? I’m not using proxmox, just Debian on the host. I have Windows VMs working great, but my Linux VMs don’t have DRI3 available and no acceleration. That is on a basic Debian VM with gnome installed.

I have installed 6.17 kernel on the guest and latest mesa drivers and still no joy. I can’t run sunshine or even vkcube on the on the VF.

1 Like

Same here though haven’t tried sunshine yet. No vkcube, no glxgears (or only on the qemu gpu), this is on either arch/fedora 43 with 6.17 kernel.

can you post your lspci -vvv from one of you clients?

I tried, no luck so far.

I am unsure if skill issue on my side but I have noticed that for B50 it is necessary to update firmware. I wasn’t able to get my B60 to update its firmware in windows and then dug deeper in to the release notes of the 6979 driver and noticed it only contains updated firmware for B50 and A series.

Thanks for pointing us in this direction, Wendell. I just saw your video and picked up two B50 cards for some testing at work. I’m looking forward to trying this out this week as time permits. We may have loss of use of our facility in about 6 months for a week or two while new generators get installed, and if this is stable enough by then, we might virtualize a bunch of staff’s desktops temporarily. The licensing should be fun to sort out, but much better than alternatives we have investigated in the past (VMWare/Nvidia VDI circa 2019 pricing).

For those having trouble getting the B50 to work on Linux, I’m posting my Proxmox/Arch guide so you can follow my exact steps.

This guide might not be perfectly clear, so please ask questions if you find any errors or confusing parts. The main hurdle, which took me a moment to solve, was the non-existent display output.

The B50 is a pure rendering card, which means you must create a virtual monitor in Linux. This gives the B50 a target to render to, which tools like Sunshine can then capture. In Sway, this is pretty easy to accomplish. However, I’m still having trouble getting this to work reliably in GNOME and KDE at this moment.

This seems to be a Wayland-specific issue. In traditional X11, you can typically just provide an EDID file to create a virtual monitor, but I still need to test that configuration as well.

I’m experimenting with this setup quite a lot in my spare time, so I apologize if the guide isn’t perfectly polished.

Here is the lspci -vvv output from my client, LNX-CLIENT-04, which is running Arch/Sway:

[builder@LNX-CLIENT-04 ~]$ sudo lspci -vvv
00:00.0 Host bridge: Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
        Subsystem: Red Hat, Inc. QEMU Virtual Machine
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Kernel modules: intel_agp

00:1a.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #4 (rev 03) (prog-if 00 [UHCI])
        Subsystem: Red Hat, Inc. QEMU Virtual Machine
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 16
        Region 4: I/O ports at 8100 [size=32]
        Kernel driver in use: uhci_hcd

00:1a.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #5 (rev 03) (prog-if 00 [UHCI])
        Subsystem: Red Hat, Inc. QEMU Virtual Machine
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin B routed to IRQ 17
        Region 4: I/O ports at 80e0 [size=32]
        Kernel driver in use: uhci_hcd

00:1a.2 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #6 (rev 03) (prog-if 00 [UHCI])
        Subsystem: Red Hat, Inc. QEMU Virtual Machine
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin C routed to IRQ 18
        Region 4: I/O ports at 80c0 [size=32]
        Kernel driver in use: uhci_hcd

00:1a.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #2 (rev 03) (prog-if 20 [EHCI])
        Subsystem: Red Hat, Inc. QEMU Virtual Machine
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin D routed to IRQ 19
        Region 0: Memory at 8124a000 (32-bit, non-prefetchable) [size=4K]
        Kernel driver in use: ehci-pci

00:1b.0 Audio device: Intel Corporation 82801I (ICH9 Family) HD Audio Controller (rev 03)
        Subsystem: Red Hat, Inc. QEMU Virtual Machine
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 43
        Region 0: Memory at 81240000 (32-bit, non-prefetchable) [size=16K]
        Capabilities: [60] MSI: Enable+ Count=1/1 Maskable- 64bit+
                Address: 00000000fee00000  Data: 0026
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel

00:1c.0 PCI bridge: Red Hat, Inc. QEMU PCIe Root port (prog-if 00 [Normal decode])
        Subsystem: Red Hat, Inc. Device 0000
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 16
        Region 0: Memory at 81249000 (32-bit, non-prefetchable) [size=4K]
        Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
        I/O behind bridge: 1000-1fff [size=4K] [16-bit]
        Memory behind bridge: 81000000-811fffff [size=2M] [32-bit]
        Prefetchable memory behind bridge: c000000000-c080ffffff [size=2064M] [32-bit]
        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
        BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [54] Express (v2) Root Port (Slot+), IntMsgNum 0
                DevCap: MaxPayload 128 bytes, PhantFunc 0
                        ExtTag+ RBE+ TEE-IO-
                DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 128 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
                LnkCap: Port #1, Speed 16GT/s, Width x32, ASPM L0s, Exit Latency L0s <64ns
                        ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- FltModeDis-
                LnkSta: Speed 2.5GT/s, Width x1
                        TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
                SltCap: AttnBtn+ PwrCtrl+ MRL- AttnInd+ PwrInd+ HotPlug+ Surprise+
                        Slot #0, PowerLimit 0W; Interlock+ NoCompl-
                SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
                        Control: AttnInd Off, PwrInd Off, Power- Interlock-
                SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
                        Changed: MRL- PresDet- LinkState-
                RootCap: CRSVisible-
                RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
                RootSta: PME ReqID 0000, PMEStatus- PMEPending-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis- NROPrPrP- LTR-
                         10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt+ EETLPPrefix+, MaxEETLPPrefixes 4
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- LN System CLS Not Supported, TPHComp- ExtTPHComp- ARIFwd+
                         AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- ARIFwd-
                         AtomicOpsCtl: ReqEn- EgressBlck-
                         IDOReq- IDOCompl- LTR- EmergencyPowerReductionReq-
                         10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
                LnkCap2: Supported Link Speeds: 2.5-16GT/s, Crosslink- Retimer- 2Retimers- DRS-
                LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported, FltMode-
        Capabilities: [48] MSI-X: Enable+ Count=1 Masked-
                Vector table: BAR=0 offset=00000000
                PBA: BAR=0 offset=00000800
        Capabilities: [40] Subsystem: Red Hat, Inc. Device 0000
        Capabilities: [100 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
                        ECRC- UnsupReq- ACSViol- UncorrIntErr- BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
                        PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
                        ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr+
                        PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+
                        ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
                        PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr- CorrIntErr- HeaderOF-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ CorrIntErr+ HeaderOF+
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap+ MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
                RootCmd: CERptEn+ NFERptEn+ FERptEn+
                RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
                         FirstFatal- NonFatalMsg- FatalMsg- IntMsgNum 0
                ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
        Capabilities: [148 v1] Access Control Services
                ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans+
                ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
        Kernel driver in use: pcieport

00:1c.1 PCI bridge: Red Hat, Inc. QEMU PCIe Root port (prog-if 00 [Normal decode])
        Subsystem: Red Hat, Inc. Device 0000
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 16
        Region 0: Memory at 81248000 (32-bit, non-prefetchable) [size=4K]
        Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
        I/O behind bridge: 2000-2fff [size=4K] [16-bit]
        Memory behind bridge: 80e00000-80ffffff [size=2M] [32-bit]
        Prefetchable memory behind bridge: c0a0000000-c0bfffffff [size=512M] [32-bit]
        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
        BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [54] Express (v2) Root Port (Slot+), IntMsgNum 0
                DevCap: MaxPayload 128 bytes, PhantFunc 0
                        ExtTag+ RBE+ TEE-IO-
                DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 128 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
                LnkCap: Port #2, Speed 16GT/s, Width x32, ASPM L0s, Exit Latency L0s <64ns
                        ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- FltModeDis-
                LnkSta: Speed 16GT/s, Width x32
                        TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
                SltCap: AttnBtn+ PwrCtrl+ MRL- AttnInd+ PwrInd+ HotPlug+ Surprise+
                        Slot #0, PowerLimit 0W; Interlock+ NoCompl-
                SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
                        Control: AttnInd Off, PwrInd Off, Power+ Interlock-
                SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
                        Changed: MRL- PresDet- LinkState-
                RootCap: CRSVisible-
                RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
                RootSta: PME ReqID 0000, PMEStatus- PMEPending-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis- NROPrPrP- LTR-
                         10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt+ EETLPPrefix+, MaxEETLPPrefixes 4
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- LN System CLS Not Supported, TPHComp- ExtTPHComp- ARIFwd+
                         AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- ARIFwd-
                         AtomicOpsCtl: ReqEn- EgressBlck-
                         IDOReq- IDOCompl- LTR- EmergencyPowerReductionReq-
                         10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
                LnkCap2: Supported Link Speeds: 2.5-16GT/s, Crosslink- Retimer- 2Retimers- DRS-
                LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported, FltMode-
        Capabilities: [48] MSI-X: Enable+ Count=1 Masked-
                Vector table: BAR=0 offset=00000000
                PBA: BAR=0 offset=00000800
        Capabilities: [40] Subsystem: Red Hat, Inc. Device 0000
        Capabilities: [100 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
                        ECRC- UnsupReq- ACSViol- UncorrIntErr- BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
                        PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
                        ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr+
                        PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+
                        ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
                        PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr- CorrIntErr- HeaderOF-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ CorrIntErr+ HeaderOF+
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap+ MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
                RootCmd: CERptEn+ NFERptEn+ FERptEn+
                RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
                         FirstFatal- NonFatalMsg- FatalMsg- IntMsgNum 0
                ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
        Capabilities: [148 v1] Access Control Services
                ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans+
                ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
        Kernel driver in use: pcieport

00:1c.2 PCI bridge: Red Hat, Inc. QEMU PCIe Root port (prog-if 00 [Normal decode])
        Subsystem: Red Hat, Inc. Device 0000
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 16
        Region 0: Memory at 81247000 (32-bit, non-prefetchable) [size=4K]
        Bus: primary=00, secondary=03, subordinate=03, sec-latency=0
        I/O behind bridge: 3000-3fff [size=4K] [16-bit]
        Memory behind bridge: 80c00000-80dfffff [size=2M] [32-bit]
        Prefetchable memory behind bridge: c0c0000000-c0dfffffff [size=512M] [32-bit]
        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
        BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [54] Express (v2) Root Port (Slot+), IntMsgNum 0
                DevCap: MaxPayload 128 bytes, PhantFunc 0
                        ExtTag+ RBE+ TEE-IO-
                DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 128 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
                LnkCap: Port #3, Speed 16GT/s, Width x32, ASPM L0s, Exit Latency L0s <64ns
                        ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- FltModeDis-
                LnkSta: Speed 16GT/s, Width x32
                        TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
                SltCap: AttnBtn+ PwrCtrl+ MRL- AttnInd+ PwrInd+ HotPlug+ Surprise+
                        Slot #0, PowerLimit 0W; Interlock+ NoCompl-
                SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
                        Control: AttnInd Off, PwrInd Off, Power+ Interlock-
                SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
                        Changed: MRL- PresDet- LinkState-
                RootCap: CRSVisible-
                RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
                RootSta: PME ReqID 0000, PMEStatus- PMEPending-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis- NROPrPrP- LTR-
                         10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt+ EETLPPrefix+, MaxEETLPPrefixes 4
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- LN System CLS Not Supported, TPHComp- ExtTPHComp- ARIFwd+
                         AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- ARIFwd-
                         AtomicOpsCtl: ReqEn- EgressBlck-
                         IDOReq- IDOCompl- LTR- EmergencyPowerReductionReq-
                         10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
                LnkCap2: Supported Link Speeds: 2.5-16GT/s, Crosslink- Retimer- 2Retimers- DRS-
                LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported, FltMode-
        Capabilities: [48] MSI-X: Enable+ Count=1 Masked-
                Vector table: BAR=0 offset=00000000
                PBA: BAR=0 offset=00000800
        Capabilities: [40] Subsystem: Red Hat, Inc. Device 0000
        Capabilities: [100 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
                        ECRC- UnsupReq- ACSViol- UncorrIntErr- BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
                        PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
                        ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr+
                        PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+
                        ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
                        PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr- CorrIntErr- HeaderOF-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ CorrIntErr+ HeaderOF+
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap+ MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
                RootCmd: CERptEn+ NFERptEn+ FERptEn+
                RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
                         FirstFatal- NonFatalMsg- FatalMsg- IntMsgNum 0
                ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
        Capabilities: [148 v1] Access Control Services
                ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans+
                ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
        Kernel driver in use: pcieport

00:1c.3 PCI bridge: Red Hat, Inc. QEMU PCIe Root port (prog-if 00 [Normal decode])
        Subsystem: Red Hat, Inc. Device 0000
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 16
        Region 0: Memory at 81246000 (32-bit, non-prefetchable) [size=4K]
        Bus: primary=00, secondary=04, subordinate=04, sec-latency=0
        I/O behind bridge: 4000-4fff [size=4K] [16-bit]
        Memory behind bridge: 80a00000-80bfffff [size=2M] [32-bit]
        Prefetchable memory behind bridge: c0e0000000-c0ffffffff [size=512M] [32-bit]
        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
        BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [54] Express (v2) Root Port (Slot+), IntMsgNum 0
                DevCap: MaxPayload 128 bytes, PhantFunc 0
                        ExtTag+ RBE+ TEE-IO-
                DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 128 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
                LnkCap: Port #4, Speed 16GT/s, Width x32, ASPM L0s, Exit Latency L0s <64ns
                        ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- FltModeDis-
                LnkSta: Speed 16GT/s, Width x32
                        TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
                SltCap: AttnBtn+ PwrCtrl+ MRL- AttnInd+ PwrInd+ HotPlug+ Surprise+
                        Slot #0, PowerLimit 0W; Interlock+ NoCompl-
                SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
                        Control: AttnInd Off, PwrInd Off, Power+ Interlock-
                SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
                        Changed: MRL- PresDet- LinkState-
                RootCap: CRSVisible-
                RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
                RootSta: PME ReqID 0000, PMEStatus- PMEPending-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis- NROPrPrP- LTR-
                         10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt+ EETLPPrefix+, MaxEETLPPrefixes 4
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- LN System CLS Not Supported, TPHComp- ExtTPHComp- ARIFwd+
                         AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- ARIFwd-
                         AtomicOpsCtl: ReqEn- EgressBlck-
                         IDOReq- IDOCompl- LTR- EmergencyPowerReductionReq-
                         10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
                LnkCap2: Supported Link Speeds: 2.5-16GT/s, Crosslink- Retimer- 2Retimers- DRS-
                LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported, FltMode-
        Capabilities: [48] MSI-X: Enable+ Count=1 Masked-
                Vector table: BAR=0 offset=00000000
                PBA: BAR=0 offset=00000800
        Capabilities: [40] Subsystem: Red Hat, Inc. Device 0000
        Capabilities: [100 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
                        ECRC- UnsupReq- ACSViol- UncorrIntErr- BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
                        PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
                        ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr+
                        PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+
                        ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
                        PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr- CorrIntErr- HeaderOF-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ CorrIntErr+ HeaderOF+
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap+ MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
                RootCmd: CERptEn+ NFERptEn+ FERptEn+
                RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
                         FirstFatal- NonFatalMsg- FatalMsg- IntMsgNum 0
                ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
        Capabilities: [148 v1] Access Control Services
                ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans+
                ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
        Kernel driver in use: pcieport

00:1d.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 (rev 03) (prog-if 00 [UHCI])
        Subsystem: Red Hat, Inc. QEMU Virtual Machine
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 16
        Region 4: I/O ports at 80a0 [size=32]
        Kernel driver in use: uhci_hcd

00:1d.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 (rev 03) (prog-if 00 [UHCI])
        Subsystem: Red Hat, Inc. QEMU Virtual Machine
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin B routed to IRQ 17
        Region 4: I/O ports at 8080 [size=32]
        Kernel driver in use: uhci_hcd

00:1d.2 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 (rev 03) (prog-if 00 [UHCI])
        Subsystem: Red Hat, Inc. QEMU Virtual Machine
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin C routed to IRQ 18
        Region 4: I/O ports at 8060 [size=32]
        Kernel driver in use: uhci_hcd

00:1d.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 (rev 03) (prog-if 20 [EHCI])
        Subsystem: Red Hat, Inc. QEMU Virtual Machine
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin D routed to IRQ 19
        Region 0: Memory at 81245000 (32-bit, non-prefetchable) [size=4K]
        Kernel driver in use: ehci-pci

00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92) (prog-if 01 [Subtractive decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Bus: primary=00, secondary=05, subordinate=09, sec-latency=0
        I/O behind bridge: 6000-7fff [size=8K] [16-bit]
        Memory behind bridge: 80000000-808fffff [size=9M] [32-bit]
        Prefetchable memory behind bridge: c100000000-c17fffffff [size=2G] [32-bit]
        Secondary status: 66MHz+ FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
        BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B+
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [50] Subsystem: Device 0000:0000

00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface Controller (rev 02)
        Subsystem: Red Hat, Inc. QEMU Virtual Machine
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Kernel driver in use: lpc_ich
        Kernel modules: lpc_ich

00:1f.2 SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] (rev 02) (prog-if 01 [AHCI 1.0])
        Subsystem: Red Hat, Inc. QEMU Virtual Machine
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 30
        Region 4: I/O ports at 8040 [size=32]
        Region 5: Memory at 81244000 (32-bit, non-prefetchable) [size=4K]
        Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit+
                Address: 00000000fee00000  Data: 0021
        Capabilities: [a8] SATA HBA v1.0 BAR4 Offset=00000004
        Kernel driver in use: ahci

00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev 02)
        Subsystem: Red Hat, Inc. QEMU Virtual Machine
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 16
        Region 4: I/O ports at 8000 [size=64]
        Kernel driver in use: i801_smbus
        Kernel modules: i2c_i801

01:00.0 VGA compatible controller: Intel Corporation Battlemage G21 [Intel Graphics] (prog-if 00 [VGA controller])
        Subsystem: Intel Corporation Device 1114
        Physical Slot: 0
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin ? routed to IRQ 44
        Region 0: Memory at c080000000 (64-bit, prefetchable) [size=16M]
        Region 2: Memory at c000000000 (64-bit, prefetchable) [size=2G]
        Capabilities: [70] Express (v2) Endpoint, IntMsgNum 0
                DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W TEE-IO-
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- FLReset-
                        MaxPayload 128 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- FltModeDis-
                LnkSta: Speed unknown (downgraded), Width x0 (downgraded)
                        TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range B, TimeoutDis+ NROPrPrP- LTR+
                         10BitTagComp+ 10BitTagReq+ OBFF Not Supported, ExtFmt+ EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
                         AtomicOpsCtl: ReqEn-
                         IDOReq- IDOCompl- LTR- EmergencyPowerReductionReq-
                         10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
                LnkCap2: Supported Link Speeds: 2.5GT/s, Crosslink- Retimer- 2Retimers- DRS-
                LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported, FltMode-
        Capabilities: [ac] MSI: Enable+ Count=1/1 Maskable+ 64bit+
                Address: 00000000fee01000  Data: 0027
                Masking: 00000000  Pending: 00000000
        Capabilities: [100 v0] Null
        Capabilities: [200 v1] Address Translation Service (ATS)
                ATSCap: Invalidate Queue Depth: 00
                ATSCtl: Enable+, Smallest Translation Unit: 00
        Kernel driver in use: xe
        Kernel modules: xe

05:01.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge (prog-if 00 [Normal decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 21
        Region 0: Memory at 80800000 (64-bit, non-prefetchable) [size=256]
        Bus: primary=05, secondary=06, subordinate=06, sec-latency=0
        I/O behind bridge: 7000-7fff [size=4K] [16-bit]
        Memory behind bridge: 80600000-807fffff [size=2M] [32-bit]
        Prefetchable memory behind bridge: c160000000-c17fffffff [size=512M] [32-bit]
        Secondary status: 66MHz+ FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
        BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [4c] MSI: Enable- Count=1/1 Maskable+ 64bit+
                Address: 0000000000000000  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [48] Slot ID: 0 slots, First+, chassis 01
        Capabilities: [40] Hot-plug capable

05:02.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge (prog-if 00 [Normal decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 22
        Region 0: Memory at 80801000 (64-bit, non-prefetchable) [size=256]
        Bus: primary=05, secondary=07, subordinate=07, sec-latency=0
        I/O behind bridge: f000-0fff [disabled] [16-bit]
        Memory behind bridge: 80400000-805fffff [size=2M] [32-bit]
        Prefetchable memory behind bridge: c140000000-c15fffffff [size=512M] [32-bit]
        Secondary status: 66MHz+ FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
        BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [4c] MSI: Enable- Count=1/1 Maskable+ 64bit+
                Address: 0000000000000000  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [48] Slot ID: 0 slots, First+, chassis 02
        Capabilities: [40] Hot-plug capable

05:03.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge (prog-if 00 [Normal decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 23
        Region 0: Memory at 80802000 (64-bit, non-prefetchable) [size=256]
        Bus: primary=05, secondary=08, subordinate=08, sec-latency=0
        I/O behind bridge: f000-0fff [disabled] [16-bit]
        Memory behind bridge: 80200000-803fffff [size=2M] [32-bit]
        Prefetchable memory behind bridge: c120000000-c13fffffff [size=512M] [32-bit]
        Secondary status: 66MHz+ FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
        BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [4c] MSI: Enable- Count=1/1 Maskable+ 64bit+
                Address: 0000000000000000  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [48] Slot ID: 0 slots, First+, chassis 03
        Capabilities: [40] Hot-plug capable

05:04.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge (prog-if 00 [Normal decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 20
        Region 0: Memory at 80803000 (64-bit, non-prefetchable) [size=256]
        Bus: primary=05, secondary=09, subordinate=09, sec-latency=0
        I/O behind bridge: 6000-6fff [size=4K] [16-bit]
        Memory behind bridge: 80000000-801fffff [size=2M] [32-bit]
        Prefetchable memory behind bridge: c100000000-c11fffffff [size=512M] [32-bit]
        Secondary status: 66MHz+ FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
        BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [4c] MSI: Enable- Count=1/1 Maskable+ 64bit+
                Address: 0000000000000000  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [48] Slot ID: 0 slots, First+, chassis 04
        Capabilities: [40] Hot-plug capable

06:03.0 Unclassified device [00ff]: Red Hat, Inc. Virtio memory balloon
        Subsystem: Red Hat, Inc. Device 0005
        Physical Slot: 3
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 20
        Region 0: I/O ports at 7080 [size=64]
        Region 1: Memory at 80602000 (32-bit, non-prefetchable) [size=4K]
        Region 4: Memory at c160008000 (64-bit, prefetchable) [size=16K]
        Capabilities: [98] MSI-X: Enable+ Count=2 Masked-
                Vector table: BAR=1 offset=00000000
                PBA: BAR=1 offset=00000800
        Capabilities: [84] Vendor Specific Information: VirtIO: <unknown>
                BAR=0 offset=00000000 size=00000000
        Capabilities: [70] Vendor Specific Information: VirtIO: Notify
                BAR=4 offset=00003000 size=00001000 multiplier=00000004
        Capabilities: [60] Vendor Specific Information: VirtIO: DeviceCfg
                BAR=4 offset=00002000 size=00001000
        Capabilities: [50] Vendor Specific Information: VirtIO: ISR
                BAR=4 offset=00001000 size=00001000
        Capabilities: [40] Vendor Specific Information: VirtIO: CommonCfg
                BAR=4 offset=00000000 size=00001000
        Kernel driver in use: virtio-pci

06:08.0 Communication controller: Red Hat, Inc. Virtio console
        Subsystem: Red Hat, Inc. Device 0003
        Physical Slot: 8-2
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 21
        Region 0: I/O ports at 7040 [size=64]
        Region 1: Memory at 80601000 (32-bit, non-prefetchable) [size=4K]
        Region 4: Memory at c160004000 (64-bit, prefetchable) [size=16K]
        Capabilities: [98] MSI-X: Enable+ Count=2 Masked-
                Vector table: BAR=1 offset=00000000
                PBA: BAR=1 offset=00000800
        Capabilities: [84] Vendor Specific Information: VirtIO: <unknown>
                BAR=0 offset=00000000 size=00000000
        Capabilities: [70] Vendor Specific Information: VirtIO: Notify
                BAR=4 offset=00003000 size=00001000 multiplier=00000004
        Capabilities: [60] Vendor Specific Information: VirtIO: DeviceCfg
                BAR=4 offset=00002000 size=00001000
        Capabilities: [50] Vendor Specific Information: VirtIO: ISR
                BAR=4 offset=00001000 size=00001000
        Capabilities: [40] Vendor Specific Information: VirtIO: CommonCfg
                BAR=4 offset=00000000 size=00001000
        Kernel driver in use: virtio-pci

06:12.0 Ethernet controller: Red Hat, Inc. Virtio network device
        Subsystem: Red Hat, Inc. Device 0001
        Physical Slot: 18-2
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 23
        Region 0: I/O ports at 7000 [size=64]
        Region 1: Memory at 80600000 (32-bit, non-prefetchable) [size=4K]
        Region 4: Memory at c160000000 (64-bit, prefetchable) [size=16K]
        Expansion ROM at 80640000 [disabled] [size=256K]
        Capabilities: [98] MSI-X: Enable+ Count=4 Masked-
                Vector table: BAR=1 offset=00000000
                PBA: BAR=1 offset=00000800
        Capabilities: [84] Vendor Specific Information: VirtIO: <unknown>
                BAR=0 offset=00000000 size=00000000
        Capabilities: [70] Vendor Specific Information: VirtIO: Notify
                BAR=4 offset=00003000 size=00001000 multiplier=00000004
        Capabilities: [60] Vendor Specific Information: VirtIO: DeviceCfg
                BAR=4 offset=00002000 size=00001000
        Capabilities: [50] Vendor Specific Information: VirtIO: ISR
                BAR=4 offset=00001000 size=00001000
        Capabilities: [40] Vendor Specific Information: VirtIO: CommonCfg
                BAR=4 offset=00000000 size=00001000
        Kernel driver in use: virtio-pci

09:01.0 SCSI storage controller: Red Hat, Inc. Virtio SCSI
        Subsystem: Red Hat, Inc. Device 0008
        Physical Slot: 1-4
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 21
        Region 0: I/O ports at 6000 [size=64]
        Region 1: Memory at 80000000 (32-bit, non-prefetchable) [size=4K]
        Region 4: Memory at c100000000 (64-bit, prefetchable) [size=16K]
        Capabilities: [98] MSI-X: Enable+ Count=7 Masked-
                Vector table: BAR=1 offset=00000000
                PBA: BAR=1 offset=00000800
        Capabilities: [84] Vendor Specific Information: VirtIO: <unknown>
                BAR=0 offset=00000000 size=00000000
        Capabilities: [70] Vendor Specific Information: VirtIO: Notify
                BAR=4 offset=00003000 size=00001000 multiplier=00000004
        Capabilities: [60] Vendor Specific Information: VirtIO: DeviceCfg
                BAR=4 offset=00002000 size=00001000
        Capabilities: [50] Vendor Specific Information: VirtIO: ISR
                BAR=4 offset=00001000 size=00001000
        Capabilities: [40] Vendor Specific Information: VirtIO: CommonCfg
                BAR=4 offset=00000000 size=00001000
        Kernel driver in use: virtio-pci

[builder@LNX-CLIENT-04 ~]$ 

Here are the steps I take to create the virtual machine (and, essentially, play games on it):

### Part 1: VM Creation & "Production" Hardware Swap

1.  **Create the VM in Proxmox:**
      * **OS:** Select your Arch Linux ISO.
      * **System:** `Machine: q35`, `BIOS: OVMF (UEFI)`.
      * **Display:** **`VirtIO-GPU`** (This is *only* for the installation).
      * **Network:** `VirtIO (paravirtualized)`.
2.  **Install Base System:**
      * Start the VM and use the Proxmox `Console`.
      * Perform your standard Arch install.
      * **Critically,** install `openssh` and `nano` and create your user (e.g., `builder`).
      * After installation, `chroot` into your new system, set a password for `builder`, and enable the SSH daemon: `systemctl enable sshd.service`.
3.  **Shut down the VM.**
4.  **Perform the Hardware Swap (In Proxmox):**
      * Select your new VM and go to the **Hardware** tab.
      * **Remove Display:** Select the `VirtIO-GPU` and click `Remove`.
      * **Add Headless Display:** Click `Add` -\> `Display`. Set `Graphic card: None`.
      * **Add GPU Passthrough:** Click `Add` -\> `PCI Device`.
          * **Device:** Select your B50 Virtual Function (e.g., `03:00.1`).
          * Check `PCI-Express`. Click `Add`.
      * **Add Stable Audio:** Click `Add` -\> `Audio Device`.
          * **Device:** `ich9-hda`
          * **Backend Driver:** `none` (This is critical).
5.  **Start the VM.** It is now fully headless.

-----

### Part 2: Headless System Configuration (via SSH)

1.  **Connect via SSH** from your client machine:
    ```bash
    ssh builder@<VM-IP-ADDRESS>
    ```
2.  **Enable Multilib:**
    ```bash
    sudo nano /etc/pacman.conf
    ```
      * Uncomment the `[multilib]` section and its `Include` line. Save and exit.
      * Sync the repos: `sudo pacman -Syu`
3.  **Install `paru` (AUR Helper):**
      * *Why:* We need this to install `sunshine-git`.
    <!-- end list -->
    ```bash
    sudo pacman -S --needed base-devel git
    git clone https://aur.archlinux.org/paru.git /tmp/paru
    cd /tmp/paru
    makepkg -si --noconfirm
    cd ~ # Go back home
    ```
4.  **Install All Software (The One-Liner):**
      * *Why:* We use `paru` to install everything from both the official repos and the AUR in a single command.
    <!-- end list -->
    ```bash
    paru -S \
      vulkan-intel \
      intel-media-driver \
      sway \
      xorg-xwayland \
      foot \
      wmenu \
      pipewire \
      wireplumber \
      pipewire-pulse \
      pipewire-alsa \
      pipewire-audio \
      pavucontrol \
      seatd \
      sunshine-git
    ```
    *When/if it asks for a `jack` provider, choose `pipewire-jack`.*

-----

### Part 3: The Headless Service Chain

1.  **Grant User Permissions:**

      * *Why:* We must give your user permission to access all hardware (`render`, `video`, `input`, `seat`).

    <!-- end list -->

    ```bash
    sudo usermod -aG render,video,input,seat builder
    ```

2.  **Enable `seatd` and `linger`:**

      * *Why:* We enable the `seatd` daemon, then enable `linger` for our user. `Linger` allows your user's services to start at boot *without* a login.

    <!-- end list -->

    ```bash
    sudo systemctl enable --now seatd
    sudo loginctl enable-linger builder
    ```

3.  **Create the `sway.service`:**

      * *Why:* This defines Sway as a boot-time service. `WLR_BACKENDS=headless` is the magic that creates our virtual display.

    <!-- end list -->

    ```bash
    mkdir -p ~/.config/systemd/user/
    nano ~/.config/systemd/user/sway.service
    ```

      * Paste this into the file:
        ```ini
        [Unit]
        Description=Sway Wayland Compositor (Headless)

        [Service]
        Type=simple
        Environment="LIBVA_DRIVER_NAME=iHD"
        Environment="WLR_BACKENDS=headless,libinput"
        ExecStart=/usr/bin/sway
        Restart=on-failure
        RestartSec=5

        [Install]
        WantedBy=default.target
        ```

4.  **Create the `sunshine.service` Override:**

      * *Why:* This forces Sunshine to use the correct Intel `iHD` driver for hardware encoding.

    <!-- end list -->

    ```bash
    systemctl --user edit sunshine.service
    ```

      * Paste this into the blank file and save:
        ```ini
        [Service]
        Environment="LIBVA_DRIVER_NAME=iHD"
        ```

5.  **Create the Sway Config:**

      * *Why:* We copy the default config and modify it for our headless setup. `sudo` is required, as the source is in `/etc`.

    <!-- end list -->

    ```bash
    mkdir -p ~/.config/sway
    sudo cp /etc/sway/config ~/.config/sway/config
    sudo chown -R builder:builder /home/builder/.config
    nano ~/.config/sway/config
    ```

      * **Add/Modify** these blocks in the file. (The `wmenu-run` line should already be correct, just verify it).
        ```sway
        # Verify this line is correct
        set $menu wmenu-run

        ### Headless Output Configuration
        # This configures the virtual display created by WLR_BACKENDS=headless
        output HEADLESS-1 {
            mode 1920x1080@60Hz
            position 0,0
        }

        ### Input Configuration
        input * {
            xkb_layout "de"
        }

        ### Autostart Services
        # We let Sway start our other services to ensure they
        # inherit the WAYLAND_DISPLAY environment correctly.
        # The 3-second delay fixes a race condition, giving Sway
        # and PipeWire time to initialize before Sunshine starts.
        exec systemctl --user start wireplumber.service
        exec --no-startup-id /bin/sh -c "sleep 3 && systemctl --user start sunshine.service"

        ### Aesthetics
        # Remove all window borders
        default_border pixel 0
        ```
      * **Comment out** the `bar { ... }` section at the bottom to hide the status bar.

6.  **Enable the Sway Service:**

      * *Why:* This is the only service we need to enable. It will start at boot (thanks to `linger`) and then use its config file to start everything else.

    <!-- end list -->

    ```bash
    systemctl --user daemon-reload
    systemctl --user enable --now sway.service
    ```

-----

### Part 4: Final Setup & Connection

1.  **Reboot the VM:**
    ```bash
    sudo reboot
    ```
2.  **Wait 60 seconds.** Do *not* log in. The `linger` service is starting Sway, which is starting PipeWire and Sunshine.
3.  **SSH Port Forward:**
      * From your client machine, run this to securely access the web UI.
    <!-- end list -->
    ```bash
    ssh -L 47990:localhost:47990 builder@<VM-IP-ADDRESS>
    ```
4.  **Configure Sunshine:**
      * Open `https://localhost:47990` in your browser.
      * Complete the first-time setup (create a username/password).
      * Go to the **Applications** tab, click `Add New`, and create your "Sway Desktop":
          * **App Name:** `Sway Desktop`
          * **Command (do):** `/usr/bin/foot`
      * Go to the **PIN** tab to pair your Moonlight client.
5.  **Connect with Moonlight:**
      * You are done. You can now launch your "Sway Desktop" from Moonlight. You will be greeted with a terminal in a clean, headless environment. Press `Mod+d` (Windows Key + D) to launch `wmenu` and run any other application.

If there are any mistakes, feel free to point them out!

Also, add these kernel parameters: pcie_acs_override=downstream,multifunction to break up iommu groups, and xe.enable_guc=3 to enable all hardware functions.

3 Likes

Is this on the host or on the guest?

The details about the virtual display got me further. If I disable the qemu GPU, I can enable a virtual display with mutter -

# mutter --wayland --headless --virtual-monitor 5120x2160
libmutter-Message: 03:00:38.528: Running Mutter (using mutter 48.4) as a Wayland display server
libmutter-Message: 03:00:38.572: Added device '/dev/dri/renderD128' (xe) using no mode setting.
libmutter-Message: 03:00:38.575: Created gbm renderer for '/dev/dri/renderD128'
libmutter-Message: 03:00:38.575: Boot VGA GPU /dev/dri/renderD128 selected as primary
** Message: 03:00:38.578: Obtained a high priority EGL context
libmutter-Message: 03:00:38.586: Added virtual monitor Meta-0
libmutter-Message: 03:00:38.587: Using public X11 display :0, (using :1 for managed services)
libmutter-Message: 03:00:38.587: Using Wayland display name 'wayland-0'

Now when I ssh and start sunshine I can see it actually grabbing the display. When I try to connect, it looks like it is going to work, but I end up with a black screen for about 5-10 seconds and sunshine segfaults.

The output from sunshine just keeps looping over the display detection with the error message about not getting an image reply -

[2025-11-05 02:57:49.185]: Info: Detecting displays
[2025-11-05 02:57:49.187]: Info: Detected display: HDMI-A-0 (id: 0)HDMI-A-0 connected: false
[2025-11-05 02:57:49.187]: Info: Detected display: DisplayPort-0 (id: 1)DisplayPort-0 connected: true
[2025-11-05 02:57:49.187]: Info: Detected display: DisplayPort-1 (id: 2)DisplayPort-1 connected: false
[2025-11-05 02:57:49.187]: Info: Screencasting with X11
[2025-11-05 02:57:49.189]: Info: Configuring selected display (1) to stream
[2025-11-05 02:57:49.191]: Info: Streaming display: DisplayPort-0 with res 5120x2160 offset by 0x0
[2025-11-05 02:57:49.193]: Error: Could not get image reply
[2025-11-05 02:57:49.193]: Info: Detecting displays
[2025-11-05 02:57:49.195]: Info: Detected display: HDMI-A-0 (id: 0)HDMI-A-0 connected: false
[2025-11-05 02:57:49.195]: Info: Detected display: DisplayPort-0 (id: 1)DisplayPort-0 connected: true
[2025-11-05 02:57:49.196]: Info: Detected display: DisplayPort-1 (id: 2)DisplayPort-1 connected: false
[2025-11-05 02:57:49.196]: Info: Screencasting with X11
[2025-11-05 02:57:49.198]: Info: Configuring selected display (1) to stream
[2025-11-05 02:57:49.199]: Info: Streaming display: DisplayPort-0 with res 5120x2160 offset by 0x0
[2025-11-05 02:57:49.201]: Info: Creating encoder [hevc_vaapi]
[2025-11-05 02:57:49.201]: Info: Color coding: SDR (Rec. 601)
[2025-11-05 02:57:49.201]: Info: Color depth: 8-bit
[2025-11-05 02:57:49.201]: Info: Color range: MPEG
[2025-11-05 02:57:49.201]: Error: Could not get image reply
libva info: VA-API version 1.22.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_22
libva info: va_openDriver() returns 0
[2025-11-05 02:57:49.221]: Info: vaapi vendor: Intel iHD driver for Intel(R) Gen Graphics - 25.2.3 ()
[2025-11-05 02:57:49.222]: Info: Streaming bitrate is 14988000
[2025-11-05 02:57:49.222]: Info: Using normal encoding mode
[2025-11-05 02:57:49.222]: Info: Using VBR with single frame VBV size
[2025-11-05 02:57:49.297]: Info: Minimum FPS target set to ~30fps (33.3333ms)
zsh: segmentation fault  sunshine

This is more progress than before when it would try to connect, flash a white screen, and crash immediately.

It looks like I was wrong about the above output. The sunshine logs are the same with the mutter display or not unless I set the WAYLAND_DISPLAY env var to wayland-0 from the mutter output.

At that point I see sunshine grabbing the virtual monitor but it is unable to find an encoder when using wayland -

[2025-11-05 16:29:14.109]: Info: Sunshine version: 2025.1027.181930 commit: e657df0c476bbaaa3493f1db7a6714e14abaf0b3
[2025-11-05 16:29:14.109]: Info: Package Publisher: LizardByte
[2025-11-05 16:29:14.109]: Info: Publisher Website: https://app.lizardbyte.dev
[2025-11-05 16:29:14.109]: Info: Get support: https://app.lizardbyte.dev/support
Cannot load libcuda.so.1
[2025-11-05 16:29:14.110]: Error: Couldn't load cuda: -1
[2025-11-05 16:29:14.110]: Info: Found display [wayland-0]
[2025-11-05 16:29:14.110]: Info: Found interface: wl_output(3) version 4
[2025-11-05 16:29:14.110]: Info: Found interface: zxdg_output_manager_v1(4) version 3
[2025-11-05 16:29:14.110]: Info: Found interface: zwp_linux_dmabuf_v1(22) version 5
[2025-11-05 16:29:14.110]: Warning: Missing Wayland wire for wlr-export-dmabuf
[2025-11-05 16:29:14.110]: Info: /dev/dri/card0 -> xe
[2025-11-05 16:29:14.110]: Error: GPU driver doesn't support universal planes: /dev/dri/card0
[2025-11-05 16:29:14.110]: Info: Found display [wayland-0]
[2025-11-05 16:29:14.110]: Info: Found display [wayland-0]
[2025-11-05 16:29:14.111]: Info: Found interface: wl_output(3) version 4
[2025-11-05 16:29:14.111]: Info: Found interface: zxdg_output_manager_v1(4) version 3
[2025-11-05 16:29:14.111]: Info: Found interface: zwp_linux_dmabuf_v1(22) version 5
[2025-11-05 16:29:14.111]: Info: Resolution: 5120x2160
[2025-11-05 16:29:14.111]: Info: Offset: 0x0
[2025-11-05 16:29:14.111]: Info: Logical size: 5120x2160
[2025-11-05 16:29:14.111]: Info: Name: Meta-0
[2025-11-05 16:29:14.111]: Info: Found monitor: MetaVendor
[2025-11-05 16:29:14.111]: Info: -------- Start of KMS monitor list --------
[2025-11-05 16:29:14.111]: Error: Unknown Monitor connector type [Meta]: Please report this to the GitHub issue tracker
[2025-11-05 16:29:14.111]: Info: --------- End of KMS monitor list ---------
[2025-11-05 16:29:14.111]: Error: Unable to initialize capture method
[2025-11-05 16:29:14.111]: Error: Platform failed to initialize
[2025-11-05 16:29:14.207]: Info: // Testing for available encoders, this may generate errors. You can safely ignore those errors. //
[2025-11-05 16:29:14.207]: Info: Trying encoder [nvenc]
[2025-11-05 16:29:14.607]: Info: Encoder [nvenc] failed
[2025-11-05 16:29:14.607]: Info: Trying encoder [vaapi]
[2025-11-05 16:29:15.008]: Info: Encoder [vaapi] failed
[2025-11-05 16:29:15.008]: Info: Trying encoder [software]
[2025-11-05 16:29:15.408]: Info: Encoder [software] failed
[2025-11-05 16:29:15.408]: Fatal: Unable to find display or encoder during startup.
[2025-11-05 16:29:15.408]: Fatal: Please check that a display is connected and powered on.
[2025-11-05 16:29:15.408]: Error: Video failed to find working encoder
[2025-11-05 16:29:15.408]: Info: Starting system tray
[2025-11-05 16:29:15.410]: Info: Configuration UI available at [https://localhost:47990]
[2025-11-05 16:29:15.410]: Info: Adding avahi service desktop

(sunshine:74139): libayatana-appindicator-WARNING **: 16:29:15.428: libayatana-appindicator is deprecated. Please use libayatana-appindicator-glib in newly written code.
[2025-11-05 16:29:15.429]: Info: System tray created
[2025-11-05 16:29:15.429]: Info: Starting main loop

(sunshine:74139): Gtk-CRITICAL **: 16:29:15.632: gtk_widget_get_scale_factor: assertion 'GTK_IS_WIDGET (widget)' failed
[2025-11-05 16:29:16.327]: Info: Avahi service desktop successfully established.

So the only way the encoder is currently found is if I ssh with X11 forwarding on and the DISPLAY env var is set. If I ssh without X11 forwarding, it has no DISPLAY var and therefore no encoder is found by sunshine.

Hey all!

I’ve got a B50 Pro up and running in Proxmox 9, 6.17 Proxmox kernel (not the 6.17 from OP), B50 is using xe drivers (both before and after SR-IOV - ran into issues with using VFIO drivers where I couldn’t set the number of instances for SR-IOV) , SR-IOV setup for 6 instances… All of that seems to be working fine, and I did do the firmware upgrade using the Sept 3 drivers from the OP (latest Pro driver) via Windows prior to using in Proxmox.

The issue I’m running into is that every single VM that I setup to use the B50 GPU crashes with VIDEO_TDR_FAILURE as the cause within igdkmdnd64.sys. This happens whether or not I have a virtual display attached via Proxmox, and happens both via RDP (where I might expect it), and via noVNC console in Proxmox where the B50 isn’t even being used.

I saw one other instance in THIS thread of the same crashing error, but I don’t see a specific resolution. My machine is a Minisforum MS-A2, 9955HX CPU, with 96GB of RAM, latest BIOS (1.02 from June 2025), resizable BAR is enabled, and the PCIe slot is set to Gen 5.

Looking for any ideas on what to try next…

This is a complete tangent, but… How did you get the b50 pro to fit in the MS-A2?

It’s running topless until the n3rdware heatsink comes out. :slight_smile:

1 Like

try the pcie_acs downstream,override options, and their various permutations. might also be helpful to post the output of your ls_iommu with all the VFs enabled.

also the output of dmesg once the tdr error(s) happen.

IOMMU is enabled:

root@prox2:~# dmesg | grep -e DMAR -e IOMMU
[    0.532284] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[    0.662465] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).

lspci of the B50 with all of the SR-IOV devices.

root@prox2:~# lspci -vvv -s 0a:00
0a:00.0 VGA compatible controller: Intel Corporation Battlemage G21 [Intel Graphics] (prog-if 00 [VGA controller])
        Subsystem: Intel Corporation Device 1114
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin ? routed to IRQ 172
        IOMMU group: 17
        Region 0: Memory at fa0c000000 (64-bit, prefetchable) [size=16M]
        Region 2: Memory at f000000000 (64-bit, prefetchable) [size=16G]
        Expansion ROM at fc000000 [disabled] [size=2M]
        Capabilities: [40] Vendor Specific Information: Len=0c <?>
        Capabilities: [70] Express (v2) Endpoint, IntMsgNum 0
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W TEE-IO-
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x1
                        TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range B, TimeoutDis+ NROPrPrP- LTR+
                         10BitTagComp+ 10BitTagReq+ OBFF Not Supported, ExtFmt+ EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
                         AtomicOpsCtl: ReqEn-
                         IDOReq- IDOCompl- LTR+ EmergencyPowerReductionReq-
                         10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
                LnkCap2: Supported Link Speeds: 2.5GT/s, Crosslink- Retimer- 2Retimers- DRS-
                LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [ac] MSI: Enable+ Count=1/1 Maskable+ 64bit+
                Address: 00000000fee00000  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [d0] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI)
                ARICap: MFVC- ACS-, Next Function: 0
                ARICtl: MFVC- ACS-, Function Group: 0
        Capabilities: [110 v1] Null
        Capabilities: [200 v1] Address Translation Service (ATS)
                ATSCap: Invalidate Queue Depth: 00
                ATSCtl: Enable+, Smallest Translation Unit: 00
        Capabilities: [420 v1] Physical Resizable BAR
                BAR 2: current size: 16GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB
        Capabilities: [220 v1] Virtual Resizable BAR
                BAR 2: current size: 4GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB
        Capabilities: [320 v1] Single Root I/O Virtualization (SR-IOV)
                IOVCap: Migration- 10BitTagReq+ IntMsgNum 0
                IOVCtl: Enable+ Migration- Interrupt- MSE+ ARIHierarchy+ 10BitTagReq-
                IOVSta: Migration-
                Initial VFs: 12, Total VFs: 12, Number of VFs: 6, Function Dependency Link: 00
                VF offset: 1, stride: 1, Device ID: e212
                Supported Page Size: 00000553, System Page Size: 00000001
                Region 0: Memory at 000000fa00000000 (64-bit, prefetchable)
                Region 2: Memory at 000000f400000000 (64-bit, prefetchable)
                VF Migration: offset: 00000000, BIR: 0
        Capabilities: [400 v1] Latency Tolerance Reporting
                Max snoop latency: 1048576ns
                Max no snoop latency: 1048576ns
        Kernel driver in use: xe
        Kernel modules: xe

0a:00.1 VGA compatible controller: Intel Corporation Battlemage G21 [Intel Graphics] (prog-if 00 [VGA controller])
        Subsystem: Intel Corporation Device 1114
        Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin ? routed to IRQ 176
        IOMMU group: 31
        Region 0: Memory at fa00000000 (64-bit, prefetchable) [disabled] [size=16M]
        Region 2: Memory at f400000000 (64-bit, prefetchable) [virtual] [size=4G]
        Capabilities: [70] Express (v2) Endpoint, IntMsgNum 0
                DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W TEE-IO-
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- FLReset-
                        MaxPayload 128 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed unknown (downgraded), Width x0 (downgraded)
                        TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range B, TimeoutDis+ NROPrPrP- LTR+
                         10BitTagComp+ 10BitTagReq+ OBFF Not Supported, ExtFmt+ EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
                         AtomicOpsCtl: ReqEn-
                         IDOReq- IDOCompl- LTR- EmergencyPowerReductionReq-
                         10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [ac] MSI: Enable+ Count=1/1 Maskable+ 64bit+
                Address: 00000000fee00000  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI)
                ARICap: MFVC- ACS-, Next Function: 2
                ARICtl: MFVC- ACS-, Function Group: 0
        Capabilities: [200 v1] Address Translation Service (ATS)
                ATSCap: Invalidate Queue Depth: 00
                ATSCtl: Enable+, Smallest Translation Unit: 00
        Kernel driver in use: xe
        Kernel modules: xe

0a:00.2 VGA compatible controller: Intel Corporation Battlemage G21 [Intel Graphics] (prog-if 00 [VGA controller])
        Subsystem: Intel Corporation Device 1114
        Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin ? routed to IRQ 177
        IOMMU group: 32
        Region 0: Memory at fa01000000 (64-bit, prefetchable) [disabled] [size=16M]
        Region 2: Memory at f500000000 (64-bit, prefetchable) [virtual] [size=4G]
        Capabilities: [70] Express (v2) Endpoint, IntMsgNum 0
                DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W TEE-IO-
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- FLReset-
                        MaxPayload 128 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed unknown (downgraded), Width x0 (downgraded)
                        TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range B, TimeoutDis+ NROPrPrP- LTR+
                         10BitTagComp+ 10BitTagReq+ OBFF Not Supported, ExtFmt+ EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
                         AtomicOpsCtl: ReqEn-
                         IDOReq- IDOCompl- LTR- EmergencyPowerReductionReq-
                         10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [ac] MSI: Enable+ Count=1/1 Maskable+ 64bit+
                Address: 00000000fee00000  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI)
                ARICap: MFVC- ACS-, Next Function: 3
                ARICtl: MFVC- ACS-, Function Group: 0
        Capabilities: [200 v1] Address Translation Service (ATS)
                ATSCap: Invalidate Queue Depth: 00
                ATSCtl: Enable+, Smallest Translation Unit: 00
        Kernel driver in use: xe
        Kernel modules: xe

0a:00.3 VGA compatible controller: Intel Corporation Battlemage G21 [Intel Graphics] (prog-if 00 [VGA controller])
        Subsystem: Intel Corporation Device 1114
        Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin ? routed to IRQ 178
        IOMMU group: 33
        Region 0: Memory at fa02000000 (64-bit, prefetchable) [disabled] [size=16M]
        Region 2: Memory at f600000000 (64-bit, prefetchable) [virtual] [size=4G]
        Capabilities: [70] Express (v2) Endpoint, IntMsgNum 0
                DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W TEE-IO-
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- FLReset-
                        MaxPayload 128 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed unknown (downgraded), Width x0 (downgraded)
                        TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range B, TimeoutDis+ NROPrPrP- LTR+
                         10BitTagComp+ 10BitTagReq+ OBFF Not Supported, ExtFmt+ EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
                         AtomicOpsCtl: ReqEn-
                         IDOReq- IDOCompl- LTR- EmergencyPowerReductionReq-
                         10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [ac] MSI: Enable+ Count=1/1 Maskable+ 64bit+
                Address: 00000000fee00000  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI)
                ARICap: MFVC- ACS-, Next Function: 4
                ARICtl: MFVC- ACS-, Function Group: 0
        Capabilities: [200 v1] Address Translation Service (ATS)
                ATSCap: Invalidate Queue Depth: 00
                ATSCtl: Enable+, Smallest Translation Unit: 00
        Kernel driver in use: xe
        Kernel modules: xe

0a:00.4 VGA compatible controller: Intel Corporation Battlemage G21 [Intel Graphics] (prog-if 00 [VGA controller])
        Subsystem: Intel Corporation Device 1114
        Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin ? routed to IRQ 179
        IOMMU group: 34
        Region 0: Memory at fa03000000 (64-bit, prefetchable) [disabled] [size=16M]
        Region 2: Memory at f700000000 (64-bit, prefetchable) [virtual] [size=4G]
        Capabilities: [70] Express (v2) Endpoint, IntMsgNum 0
                DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W TEE-IO-
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- FLReset-
                        MaxPayload 128 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed unknown (downgraded), Width x0 (downgraded)
                        TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range B, TimeoutDis+ NROPrPrP- LTR+
                         10BitTagComp+ 10BitTagReq+ OBFF Not Supported, ExtFmt+ EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
                         AtomicOpsCtl: ReqEn-
                         IDOReq- IDOCompl- LTR- EmergencyPowerReductionReq-
                         10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [ac] MSI: Enable+ Count=1/1 Maskable+ 64bit+
                Address: 00000000fee00000  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI)
                ARICap: MFVC- ACS-, Next Function: 5
                ARICtl: MFVC- ACS-, Function Group: 0
        Capabilities: [200 v1] Address Translation Service (ATS)
                ATSCap: Invalidate Queue Depth: 00
                ATSCtl: Enable+, Smallest Translation Unit: 00
        Kernel driver in use: xe
        Kernel modules: xe

0a:00.5 VGA compatible controller: Intel Corporation Battlemage G21 [Intel Graphics] (prog-if 00 [VGA controller])
        Subsystem: Intel Corporation Device 1114
        Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin ? routed to IRQ 180
        IOMMU group: 35
        Region 0: Memory at fa04000000 (64-bit, prefetchable) [disabled] [size=16M]
        Region 2: Memory at f800000000 (64-bit, prefetchable) [virtual] [size=4G]
        Capabilities: [70] Express (v2) Endpoint, IntMsgNum 0
                DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W TEE-IO-
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- FLReset-
                        MaxPayload 128 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed unknown (downgraded), Width x0 (downgraded)
                        TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range B, TimeoutDis+ NROPrPrP- LTR+
                         10BitTagComp+ 10BitTagReq+ OBFF Not Supported, ExtFmt+ EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
                         AtomicOpsCtl: ReqEn-
                         IDOReq- IDOCompl- LTR- EmergencyPowerReductionReq-
                         10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [ac] MSI: Enable+ Count=1/1 Maskable+ 64bit+
                Address: 00000000fee00000  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI)
                ARICap: MFVC- ACS-, Next Function: 6
                ARICtl: MFVC- ACS-, Function Group: 0
        Capabilities: [200 v1] Address Translation Service (ATS)
                ATSCap: Invalidate Queue Depth: 00
                ATSCtl: Enable+, Smallest Translation Unit: 00
        Kernel driver in use: xe
        Kernel modules: xe

0a:00.6 VGA compatible controller: Intel Corporation Battlemage G21 [Intel Graphics] (prog-if 00 [VGA controller])
        Subsystem: Intel Corporation Device 1114
        Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin ? routed to IRQ 181
        IOMMU group: 36
        Region 0: Memory at fa05000000 (64-bit, prefetchable) [disabled] [size=16M]
        Region 2: Memory at f900000000 (64-bit, prefetchable) [virtual] [size=4G]
        Capabilities: [70] Express (v2) Endpoint, IntMsgNum 0
                DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W TEE-IO-
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- FLReset-
                        MaxPayload 128 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed unknown (downgraded), Width x0 (downgraded)
                        TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range B, TimeoutDis+ NROPrPrP- LTR+
                         10BitTagComp+ 10BitTagReq+ OBFF Not Supported, ExtFmt+ EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
                         AtomicOpsCtl: ReqEn-
                         IDOReq- IDOCompl- LTR- EmergencyPowerReductionReq-
                         10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [ac] MSI: Enable+ Count=1/1 Maskable+ 64bit+
                Address: 00000000fee00000  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI)
                ARICap: MFVC- ACS-, Next Function: 7
                ARICtl: MFVC- ACS-, Function Group: 0
        Capabilities: [200 v1] Address Translation Service (ATS)
                ATSCap: Invalidate Queue Depth: 00
                ATSCtl: Enable+, Smallest Translation Unit: 00
        Kernel driver in use: xe
        Kernel modules: xe

All of the IOMMU groups…

root@prox2:~# find /sys/kernel/iommu_groups/ -type l
/sys/kernel/iommu_groups/17/devices/0000:0a:00.0
/sys/kernel/iommu_groups/35/devices/0000:0a:00.5
/sys/kernel/iommu_groups/7/devices/0000:00:03.2
/sys/kernel/iommu_groups/25/devices/0000:01:00.1
/sys/kernel/iommu_groups/15/devices/0000:08:01.0
/sys/kernel/iommu_groups/33/devices/0000:0a:00.3
/sys/kernel/iommu_groups/5/devices/0000:00:03.0
/sys/kernel/iommu_groups/23/devices/0000:04:00.0
/sys/kernel/iommu_groups/13/devices/0000:00:18.3
/sys/kernel/iommu_groups/13/devices/0000:00:18.1
/sys/kernel/iommu_groups/13/devices/0000:00:18.6
/sys/kernel/iommu_groups/13/devices/0000:00:18.4
/sys/kernel/iommu_groups/13/devices/0000:00:18.2
/sys/kernel/iommu_groups/13/devices/0000:00:18.0
/sys/kernel/iommu_groups/13/devices/0000:00:18.7
/sys/kernel/iommu_groups/13/devices/0000:00:18.5
/sys/kernel/iommu_groups/31/devices/0000:0a:00.1
/sys/kernel/iommu_groups/3/devices/0000:00:02.0
/sys/kernel/iommu_groups/21/devices/0000:05:00.1
/sys/kernel/iommu_groups/11/devices/0000:00:08.3
/sys/kernel/iommu_groups/1/devices/0000:00:01.1
/sys/kernel/iommu_groups/28/devices/0000:01:00.4
/sys/kernel/iommu_groups/18/devices/0000:09:00.0
/sys/kernel/iommu_groups/36/devices/0000:0a:00.6
/sys/kernel/iommu_groups/8/devices/0000:00:04.0
/sys/kernel/iommu_groups/26/devices/0000:01:00.2
/sys/kernel/iommu_groups/16/devices/0000:08:02.0
/sys/kernel/iommu_groups/34/devices/0000:0a:00.4
/sys/kernel/iommu_groups/6/devices/0000:00:03.1
/sys/kernel/iommu_groups/24/devices/0000:01:00.0
/sys/kernel/iommu_groups/14/devices/0000:07:00.0
/sys/kernel/iommu_groups/32/devices/0000:0a:00.2
/sys/kernel/iommu_groups/4/devices/0000:00:02.1
/sys/kernel/iommu_groups/22/devices/0000:03:00.0
/sys/kernel/iommu_groups/12/devices/0000:00:14.3
/sys/kernel/iommu_groups/12/devices/0000:00:14.0
/sys/kernel/iommu_groups/30/devices/0000:02:00.0
/sys/kernel/iommu_groups/2/devices/0000:00:01.2
/sys/kernel/iommu_groups/20/devices/0000:05:00.0
/sys/kernel/iommu_groups/10/devices/0000:00:08.1
/sys/kernel/iommu_groups/29/devices/0000:01:00.6
/sys/kernel/iommu_groups/0/devices/0000:00:01.0
/sys/kernel/iommu_groups/19/devices/0000:0b:00.0
/sys/kernel/iommu_groups/9/devices/0000:00:08.0
/sys/kernel/iommu_groups/27/devices/0000:01:00.3

DMESG output when the crash happens (and this crash was after tweaking the acs downstreams). Note the vfio-pci resetting (the rest is recurring errors that fill up dmesg).

[  427.916302] vfio-pci 0000:0a:00.5: resetting
[  427.917230] xe 0000:0a:00.0: [drm] GT1: PF: VF5 FLR
[  427.917470] xe 0000:0a:00.0: [drm] GT0: PF: VF5 FLR
[  428.023032] vfio-pci 0000:0a:00.5: reset done
[  430.907794] aer_ratelimit: 52954 callbacks suppressed
[  430.907878] pcieport 0000:00:01.1: AER: Multiple Correctable error message received from 0000:07:00.0
[  430.907884] pcieport 0000:07:00.0: PCIe Bus Error: severity=Correctable, type=Data Link Layer, (Transmitter ID)
[  430.907885] pcieport 0000:07:00.0:   device [8086:e2ff] error status/mask=00001000/0000e000
[  430.907887] pcieport 0000:07:00.0:    [12] Timeout
[  430.907922] pcieport 0000:00:01.1: AER: Multiple Correctable error message received from 0000:07:00.0
[  430.907924] pcieport 0000:07:00.0: PCIe Bus Error: severity=Correctable, type=Physical Layer, (Transmitter ID)
[  430.907925] pcieport 0000:07:00.0:   device [8086:e2ff] error status/mask=00001001/0000e000
[  430.907926] pcieport 0000:07:00.0:    [ 0] RxErr                  (First)
[  430.907927] pcieport 0000:07:00.0:    [12] Timeout
[  430.908113] pcieport 0000:00:01.1: AER: Multiple Correctable error message received from 0000:07:00.0
[  430.908115] pcieport 0000:07:00.0: PCIe Bus Error: severity=Correctable, type=Physical Layer, (Transmitter ID)
[  430.908116] pcieport 0000:07:00.0:   device [8086:e2ff] error status/mask=00001101/0000e000
[  430.908116] pcieport 0000:07:00.0:    [ 0] RxErr                  (First)
[  430.908117] pcieport 0000:07:00.0:    [ 8] Rollover
[  430.908117] pcieport 0000:07:00.0:    [12] Timeout
[  430.908118] pcieport 0000:07:00.0: AER:   Error of this Agent is reported first
[  430.908255] pcieport 0000:00:01.1: AER: Multiple Correctable error message received from 0000:00:01.1
[  430.908287] pcieport 0000:07:00.0: PCIe Bus Error: severity=Correctable, type=Physical Layer, (Transmitter ID)
[  430.908288] pcieport 0000:07:00.0:   device [8086:e2ff] error status/mask=00001001/0000e000
[  430.908288] pcieport 0000:07:00.0:    [ 0] RxErr                  (First)
[  430.908289] pcieport 0000:07:00.0:    [12] Timeout
[  430.908462] pcieport 0000:00:01.1: AER: Multiple Correctable error message received from 0000:07:00.0
[  430.908465] pcieport 0000:07:00.0: PCIe Bus Error: severity=Correctable, type=Data Link Layer, (Transmitter ID)
[  430.908465] pcieport 0000:07:00.0:   device [8086:e2ff] error status/mask=00001000/0000e000
[  430.908466] pcieport 0000:07:00.0:    [12] Timeout
[  430.908466] pcieport 0000:07:00.0: AER:   Error of this Agent is reported first
[  430.908501] pcieport 0000:00:01.1: AER: Correctable error message received from 0000:07:00.0
[  430.908642] pcieport 0000:00:01.1: AER: Multiple Correctable error message received from 0000:07:00.0
[  430.908655] pcieport 0000:07:00.0: PCIe Bus Error: severity=Correctable, type=Data Link Layer, (Transmitter ID)
[  430.908655] pcieport 0000:07:00.0:   device [8086:e2ff] error status/mask=00001100/0000e000
[  430.908656] pcieport 0000:07:00.0:    [ 8] Rollover
[  430.908656] pcieport 0000:07:00.0:    [12] Timeout
[  430.908774] pcieport 0000:00:01.1: AER: Multiple Correctable error message received from 0000:07:00.0
[  430.908776] pcieport 0000:07:00.0: PCIe Bus Error: severity=Correctable, type=Data Link Layer, (Transmitter ID)
[  430.908777] pcieport 0000:07:00.0:   device [8086:e2ff] error status/mask=00001000/0000e000
[  430.908777] pcieport 0000:07:00.0:    [12] Timeout
[  430.908778] pcieport 0000:07:00.0: AER:   Error of this Agent is reported first
[  430.908783] pcieport 0000:00:01.1: AER: Correctable error message received from 0000:07:00.0
[  430.908943] pcieport 0000:00:01.1: AER: Multiple Correctable error message received from 0000:07:00.0
[  430.908959] pcieport 0000:07:00.0: PCIe Bus Error: severity=Correctable, type=Physical Layer, (Transmitter ID)
[  430.908960] pcieport 0000:07:00.0:   device [8086:e2ff] error status/mask=00001001/0000e000
[  430.908960] pcieport 0000:07:00.0:    [ 0] RxErr                  (First)
[  430.908961] pcieport 0000:07:00.0:    [12] Timeout
[  430.909084] aer_ratelimit: 23617 callbacks suppressed
[  430.909148] pcieport 0000:00:01.1: AER: Multiple Correctable error message received from 0000:00:01.1
[  430.909149] pcieport 0000:00:01.1: PCIe Bus Error: severity=Correctable, type=Data Link Layer, (Transmitter ID)
[  430.909149] pcieport 0000:00:01.1:   device [1022:14db] error status/mask=00001040/00006000
[  430.909150] pcieport 0000:00:01.1:    [ 6] BadTLP
[  430.909150] pcieport 0000:00:01.1:    [12] Timeout
[  430.909424] pcieport 0000:00:01.1: AER: Multiple Correctable error message received from 0000:00:01.1
[  430.909425] pcieport 0000:00:01.1: PCIe Bus Error: severity=Correctable, type=Data Link Layer, (Transmitter ID)
[  430.909426] pcieport 0000:00:01.1:   device [1022:14db] error status/mask=00001040/00006000
[  430.909426] pcieport 0000:00:01.1:    [ 6] BadTLP
[  430.909427] pcieport 0000:00:01.1:    [12] Timeout
[  430.909427] pcieport 0000:00:01.1: AER:   Error of this Agent is reported first
[  430.909494] pcieport 0000:00:01.1: AER: Multiple Correctable error message received from 0000:07:00.0
[  430.909496] pcieport 0000:00:01.1: PCIe Bus Error: severity=Correctable, type=Data Link Layer, (Transmitter ID)
[  430.909496] pcieport 0000:00:01.1:   device [1022:14db] error status/mask=00001000/00006000
[  430.909497] pcieport 0000:00:01.1:    [12] Timeout
[  430.909790] pcieport 0000:00:01.1: AER: Multiple Correctable error message received from 0000:00:01.1
[  430.909794] pcieport 0000:00:01.1: PCIe Bus Error: severity=Correctable, type=Data Link Layer, (Receiver ID)
[  430.909794] pcieport 0000:00:01.1:   device [1022:14db] error status/mask=00000040/00006000
[  430.909795] pcieport 0000:00:01.1:    [ 6] BadTLP
[  430.910038] pcieport 0000:00:01.1: AER: Multiple Correctable error message received from 0000:07:00.0
[  430.910040] pcieport 0000:00:01.1: PCIe Bus Error: severity=Correctable, type=Data Link Layer, (Transmitter ID)
[  430.910040] pcieport 0000:00:01.1:   device [1022:14db] error status/mask=00001040/00006000
[  430.910041] pcieport 0000:00:01.1:    [ 6] BadTLP
[  430.910041] pcieport 0000:00:01.1:    [12] Timeout
[  430.910189] pcieport 0000:00:01.1: AER: Multiple Correctable error message received from 0000:07:00.0
[  430.910191] pcieport 0000:00:01.1: PCIe Bus Error: severity=Correctable, type=Data Link Layer, (Transmitter ID)
[  430.910191] pcieport 0000:00:01.1:   device [1022:14db] error status/mask=00001040/00006000
[  430.910192] pcieport 0000:00:01.1:    [ 6] BadTLP
[  430.910192] pcieport 0000:00:01.1:    [12] Timeout
[  430.910465] pcieport 0000:00:01.1: AER: Multiple Correctable error message received from 0000:00:01.1
[  430.910466] pcieport 0000:00:01.1: PCIe Bus Error: severity=Correctable, type=Data Link Layer, (Transmitter ID)
[  430.910467] pcieport 0000:00:01.1:   device [1022:14db] error status/mask=00001040/00006000
[  430.910467] pcieport 0000:00:01.1:    [ 6] BadTLP
[  430.910468] pcieport 0000:00:01.1:    [12] Timeout
[  430.910741] pcieport 0000:00:01.1: AER: Multiple Correctable error message received from 0000:07:00.0
[  430.910743] pcieport 0000:00:01.1: PCIe Bus Error: severity=Correctable, type=Data Link Layer, (Transmitter ID)
[  430.910743] pcieport 0000:00:01.1:   device [1022:14db] error status/mask=00001000/00006000
[  430.910744] pcieport 0000:00:01.1:    [12] Timeout
[  430.910937] pcieport 0000:00:01.1: AER: Multiple Correctable error message received from 0000:00:01.1 (no details found
[  430.911012] pcieport 0000:00:01.1: AER: Multiple Correctable error message received from 0000:07:00.0
[  430.911013] pcieport 0000:00:01.1: PCIe Bus Error: severity=Correctable, type=Data Link Layer, (Transmitter ID)
[  430.911014] pcieport 0000:00:01.1:   device [1022:14db] error status/mask=00001000/00006000
[  430.911014] pcieport 0000:00:01.1:    [12] Timeout