Off topic w.r.t. B50, but I doubt Blackwell is supported on nouveau yet.
tried testing on mint today. found a guide here on how to load the ubuntu 6.17 kernel. according to the article this is supposed to have xe support for sriov. the virtual card does load the xe driver but still looks weird and glxgears does not use the gpu. as a satiny check i ran glxgears on my normal linux machine to confirm that it works and it does.
has anyone gotten this card to load the vfio drivers? i followed the pci passthrough guide here. and black listed xe and promox just does not load any driver.
43:00.0 VGA compatible controller: Intel Corporation Battlemage G21 [Intel Graphics] (prog-if 00 [VGA controller])
Subsystem: Intel Corporation Device 1114
Flags: fast devsel, IOMMU group 28
Memory at 2001c000000 (64-bit, prefetchable) [disabled] [size=16M]
Memory at 20000000000 (64-bit, prefetchable) [disabled] [size=256M]
Expansion ROM at b2000000 [disabled] [size=2M]
Capabilities: [40] Vendor Specific Information: Len=0c <?>
Capabilities: [70] Express Endpoint, IntMsgNum 0
Capabilities: [ac] MSI: Enable- Count=1/1 Maskable+ 64bit+
Capabilities: [d0] Power Management version 3
Capabilities: [100] Alternative Routing-ID Interpretation (ARI)
Capabilities: [110] Null
Capabilities: [200] Address Translation Service (ATS)
Capabilities: [420] Physical Resizable BAR
Capabilities: [220] Virtual Resizable BAR
Capabilities: [320] Single Root I/O Virtualization (SR-IOV)
Capabilities: [400] Latency Tolerance Reporting
Kernel modules: xe
you can try add vfio-pci.ids=8086:e212 at grub.
where do you get this id from?
and
root@hpws:~# lspci -nnk | grep 43
43:00.0 VGA compatible controller [0300]: Intel Corporation Battlemage G21 [Intel Graphics] [8086:e212]
thank you. this fixed it.
ok, now i have the issue where sriov_numvfs is not writeable, but it is there. any ideas on how to write it like this? vfio-pci is in use on the card.
root@pve:~# ls -la /sys/devices/pci0000:40/0000:40:03.1/0000:41:00.0/0000:42:01.0/0000:43:00.0/sriov_numvfs
-rw-r--r-- 1 root root 4096 Oct 24 19:00 /sys/devices/pci0000:40/0000:40:03.1/0000:41:00.0/0000:42:01.0/0000:43:00.0/sriov_numvfs
root@pve:~# echo 4 > /sys/devices/pci0000:40/0000:40:03.1/0000:41:00.0/0000:42:01.0/0000:43:00.0/sriov_numvfs
-bash: echo: write error: No such file or directory
43:00.0 VGA compatible controller: Intel Corporation Battlemage G21 [Intel Graphics] (prog-if 00 [VGA controller])
Subsystem: Intel Corporation Device 1114
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
IOMMU group: 28
Region 0: Memory at 2001c000000 (64-bit, prefetchable) [disabled] [size=16M]
Region 2: Memory at 20000000000 (64-bit, prefetchable) [disabled] [size=256M]
Expansion ROM at b2000000 [disabled] [size=2M]
Capabilities: [40] Vendor Specific Information: Len=0c <?>
Capabilities: [70] Express (v2) Endpoint, IntMsgNum 0
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W TEE-IO-
DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1
TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range B, TimeoutDis+ NROPrPrP- LTR+
10BitTagComp+ 10BitTagReq+ OBFF Not Supported, ExtFmt+ EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS- TPHComp- ExtTPHComp-
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
AtomicOpsCtl: ReqEn-
IDOReq- IDOCompl- LTR- EmergencyPowerReductionReq-
10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
LnkCap2: Supported Link Speeds: 2.5GT/s, Crosslink- Retimer- 2Retimers- DRS-
LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [ac] MSI: Enable- Count=1/1 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [d0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
Status: D3 NoSoftRst+ PME-Enable+ DSel=0 DScale=0 PME-
Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [110 v1] Null
Capabilities: [200 v1] Address Translation Service (ATS)
ATSCap: Invalidate Queue Depth: 00
ATSCtl: Enable+, Smallest Translation Unit: 00
Capabilities: [420 v1] Physical Resizable BAR
BAR 2: current size: 256MB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB
Capabilities: [220 v1] Virtual Resizable BAR
BAR 2: current size: 2GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB
Capabilities: [320 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration- 10BitTagReq+ IntMsgNum 0
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+ 10BitTagReq-
IOVSta: Migration-
Initial VFs: 12, Total VFs: 12, Number of VFs: 0, Function Dependency Link: 00
VF offset: 1, stride: 1, Device ID: e212
Supported Page Size: 00000553, System Page Size: 00000001
Region 0: Memory at 0000020010000000 (64-bit, prefetchable)
Region 2: Memory at 000001fa00000000 (64-bit, prefetchable)
VF Migration: offset: 00000000, BIR: 0
Capabilities: [400 v1] Latency Tolerance Reporting
Max snoop latency: 0ns
Max no snoop latency: 0ns
Kernel driver in use: vfio-pci
Kernel modules: xe
can you share, what you have in dmesg for this device?
I guess, I can’t just use vfio-pci for root device, you have to use xe driver initially
then you need write ‘echo 4 > sriov_numvfs’ and only after you should get
I have the driver manually set to vfio in etc default grub. if i unbind it it loads as xe, but if i set sriov_numvfs while using xe, its stays on the xe driver.
reading over this. its concerning that the memory is being split between the vfs. I though sriov was supposed to just share the full 16GB between each virtual function.
when i set 8 vfs
[ 0.886914] pci 0000:43:00.0: [8086:e212] type 00 class 0x030000 PCIe Endpoint
[ 0.886965] pci 0000:43:00.0: BAR 0 [mem 0x2001c000000-0x2001cffffff 64bit pref]
[ 0.886971] pci 0000:43:00.0: BAR 2 [mem 0x20000000000-0x2000fffffff 64bit pref]
[ 0.886978] pci 0000:43:00.0: ROM [mem 0xb2000000-0xb21fffff pref]
[ 0.887098] pci 0000:43:00.0: PME# supported from D0 D3hot
[ 0.887162] pci 0000:43:00.0: VF BAR 0 [mem 0x20010000000-0x20010ffffff 64bit pref]
[ 0.887166] pci 0000:43:00.0: VF BAR 0 [mem 0x20010000000-0x2001bffffff 64bit pref]: contains BAR 0 for 12 VFs
[ 0.887172] pci 0000:43:00.0: VF BAR 2 [mem 0x1fa00000000-0x1fa7fffffff 64bit pref]
[ 0.887175] pci 0000:43:00.0: VF BAR 2 [mem 0x1fa00000000-0x1ffffffffff 64bit pref]: contains BAR 2 for 12 VFs
[ 0.918388] pci 0000:43:00.0: vgaarb: setting as boot VGA device
[ 0.918388] pci 0000:43:00.0: vgaarb: bridge control possible
[ 0.918388] pci 0000:43:00.0: vgaarb: VGA device added: decodes=io+mem,owns=none,locks=none
[ 0.961656] pci_bus 0000:43: resource 1 [mem 0xb2000000-0xb21fffff]
[ 0.961659] pci_bus 0000:43: resource 2 [mem 0x1fa00000000-0x2001cffffff 64bit pref]
[ 0.965312] pci 0000:43:00.0: Adding to iommu group 28
[ 6.391969] xe 0000:43:00.0: enabling device (0000 -> 0002)
[ 6.392322] xe 0000:43:00.0: [drm] Running in SR-IOV PF mode
[ 6.392481] xe 0000:43:00.0: [drm] Found battlemage (device ID e212) discrete display version 14.01 stepping B0
[ 6.393895] xe 0000:43:00.0: [drm] Attempting to resize bar from 256MiB -> 16384MiB
[ 6.393904] xe 0000:43:00.0: BAR 2 [mem 0x20000000000-0x2000fffffff 64bit pref]: releasing
[ 6.393926] xe 0000:43:00.0: [drm] Failed to resize BAR2 to 16384M (-ENOENT). Consider enabling 'Resizable BAR' support in your BIOS
[ 6.393934] xe 0000:43:00.0: BAR 2 [mem 0x20000000000-0x2000fffffff 64bit pref]: assigned
[ 6.393962] xe 0000:43:00.0: [drm] VISIBLE VRAM: 0x0000020000000000, 0x0000000010000000
[ 6.393995] xe 0000:43:00.0: [drm] Small BAR device
[ 6.393997] xe 0000:43:00.0: [drm] VRAM[0, 0]: Actual physical size 0x0000000400000000, usable size exclude stolen 0x00000003fb000000, CPU accessible size 0x0000000010000000
[ 6.394001] xe 0000:43:00.0: [drm] VRAM[0, 0]: DPA range: [0x0000000000000000-400000000], io range: [0x0000020000000000-20010000000]
[ 6.394005] xe 0000:43:00.0: [drm] VRAM: 0x0000000400000000 is larger than resource 0x0000000010000000
[ 6.394008] xe 0000:43:00.0: [drm] Total VRAM: 0x0000020000000000, 0x0000000400000000
[ 6.394011] xe 0000:43:00.0: [drm] Available VRAM: 0x0000020000000000, 0x00000003fb000000
[ 6.515598] xe 0000:43:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=none:owns=none
[ 6.527524] xe 0000:43:00.0: [drm] Finished loading DMC firmware i915/bmg_dmc.bin (v2.6)
[ 6.531840] xe 0000:43:00.0: [drm] GT0: Using GuC firmware from xe/bmg_guc_70.bin version 70.45.2
[ 6.915213] xe 0000:43:00.0: [drm] GT0: ccs1 fused off
[ 6.915218] xe 0000:43:00.0: [drm] GT0: ccs2 fused off
[ 6.915221] xe 0000:43:00.0: [drm] GT0: ccs3 fused off
[ 6.940374] xe 0000:43:00.0: [drm] GT1: Using GuC firmware from xe/bmg_guc_70.bin version 70.45.2
[ 6.944615] xe 0000:43:00.0: [drm] GT1: Using HuC firmware from xe/bmg_huc.bin version 8.2.10
[ 6.955876] xe 0000:43:00.0: [drm] GT1: vcs1 fused off
[ 6.955883] xe 0000:43:00.0: [drm] GT1: vcs3 fused off
[ 6.955886] xe 0000:43:00.0: [drm] GT1: vcs4 fused off
[ 6.955889] xe 0000:43:00.0: [drm] GT1: vcs5 fused off
[ 6.955892] xe 0000:43:00.0: [drm] GT1: vcs6 fused off
[ 6.955894] xe 0000:43:00.0: [drm] GT1: vcs7 fused off
[ 6.955897] xe 0000:43:00.0: [drm] GT1: vecs2 fused off
[ 6.955900] xe 0000:43:00.0: [drm] GT1: vecs3 fused off
[ 6.989409] xe 0000:43:00.0: [drm] Registered 4 planes with drm panic
[ 6.989415] [drm] Initialized xe 1.1.0 for 0000:43:00.0 on minor 1
[ 7.029469] xe 0000:43:00.0: [drm] Cannot find any crtc or sizes
[ 7.041347] xe 0000:43:00.0: [drm] Using mailbox commands for power limits
[ 7.041802] xe 0000:43:00.0: [drm] PL2 is supported on channel 0
[ 7.110467] xe 0000:43:00.0: [drm] Cannot find any crtc or sizes
[ 7.150458] snd_hda_intel 0000:44:00.0: bound 0000:43:00.0 (ops intel_audio_component_bind_ops [xe])
[ 7.150597] xe 0000:43:00.0: [drm] Cannot find any crtc or sizes
[ 7.710868] xe 0000:43:00.0: [drm] GT0: PF: VF1..VF8 provisioned with 522584064 (498 MiB) GGTT
[ 7.714743] xe 0000:43:00.0: [drm] GT0: PF: VF1..VF8 provisioned with 2116026368 (1.97 GiB) LMEM
[ 7.715230] xe 0000:43:00.0: [drm] GT0: PF: VF1..VF8 provisioned with 8159 GuC context IDs
[ 7.715727] xe 0000:43:00.0: [drm] GT0: PF: VF1..VF8 provisioned with 32 GuC doorbell IDs
[ 7.716279] xe 0000:43:00.0: [drm] GT1: PF: VF1..VF8 provisioned with 8159 GuC context IDs
[ 7.716758] xe 0000:43:00.0: [drm] GT1: PF: VF1..VF8 provisioned with 32 GuC doorbell IDs
[ 7.818467] pci 0000:43:00.1: [8086:e212] type 00 class 0x030000 PCIe Endpoint
[ 7.818763] pci 0000:43:00.1: Adding to iommu group 77
[ 7.819004] pci 0000:43:00.1: vgaarb: no bridge control possible
[ 7.819007] pci 0000:43:00.1: vgaarb: VGA device added: decodes=io+mem,owns=none,locks=none
[ 7.819011] xe 0000:43:00.0: vgaarb: VGA decodes changed: olddecodes=none,decodes=none:owns=io
[ 7.819105] xe 0000:43:00.1: enabling device (0000 -> 0002)
[ 7.819451] xe 0000:43:00.1: [drm] Running in SR-IOV VF mode
[ 7.820890] xe 0000:43:00.1: [drm] VF: migration not supported by this module version
[ 7.824166] xe 0000:43:00.1: [drm] VISIBLE VRAM: 0x000001fa00000000, 0x0000000080000000
[ 7.824171] xe 0000:43:00.1: [drm] VRAM[0, 0]: Actual physical size 0x000000007e200000, usable size exclude stolen 0x000000007e200000, CPU accessible size 0x000000007e200000
[ 7.824176] xe 0000:43:00.1: [drm] VRAM[0, 0]: DPA range: [0x0000000000000000-7e200000], io range: [0x000001fa00000000-1fa7e200000]
[ 7.824180] xe 0000:43:00.1: [drm] Total VRAM: 0x000001fa00000000, 0x000000007e200000
[ 7.824183] xe 0000:43:00.1: [drm] Available VRAM: 0x000001fa00000000, 0x000000007e200000
[ 7.890150] xe 0000:43:00.1: [drm] GT0: ccs1 fused off
[ 7.890157] xe 0000:43:00.1: [drm] GT0: ccs2 fused off
[ 7.890160] xe 0000:43:00.1: [drm] GT0: ccs3 fused off
[ 7.901664] xe 0000:43:00.1: [drm] GT1: vcs1 fused off
[ 7.901669] xe 0000:43:00.1: [drm] GT1: vcs3 fused off
[ 7.901672] xe 0000:43:00.1: [drm] GT1: vcs4 fused off
[ 7.901675] xe 0000:43:00.1: [drm] GT1: vcs5 fused off
[ 7.901677] xe 0000:43:00.1: [drm] GT1: vcs6 fused off
[ 7.901680] xe 0000:43:00.1: [drm] GT1: vcs7 fused off
[ 7.901682] xe 0000:43:00.1: [drm] GT1: vecs2 fused off
[ 7.901685] xe 0000:43:00.1: [drm] GT1: vecs3 fused off
[ 7.908927] [drm] Initialized xe 1.1.0 for 0000:43:00.1 on minor 2
[ 7.909301] pci 0000:43:00.2: [8086:e212] type 00 class 0x030000 PCIe Endpoint
[ 7.909748] pci 0000:43:00.2: Adding to iommu group 78
[ 7.909991] pci 0000:43:00.2: vgaarb: no bridge control possible
[ 7.909996] pci 0000:43:00.2: vgaarb: VGA device added: decodes=io+mem,owns=none,locks=none
[ 7.910001] xe 0000:43:00.0: vgaarb: VGA decodes changed: olddecodes=none,decodes=none:owns=io
[ 7.910103] xe 0000:43:00.2: enabling device (0000 -> 0002)
[ 7.910393] xe 0000:43:00.2: [drm] Running in SR-IOV VF mode
[ 7.911706] xe 0000:43:00.2: [drm] VF: migration not supported by this module version
[ 7.912486] xe 0000:43:00.2: [drm] VISIBLE VRAM: 0x000001fa80000000, 0x0000000080000000
[ 7.912492] xe 0000:43:00.2: [drm] VRAM[0, 0]: Actual physical size 0x000000007e200000, usable size exclude stolen 0x000000007e200000, CPU accessible size 0x000000007e200000
[ 7.912497] xe 0000:43:00.2: [drm] VRAM[0, 0]: DPA range: [0x0000000000000000-7e200000], io range: [0x000001fa80000000-1fafe200000]
[ 7.912501] xe 0000:43:00.2: [drm] Total VRAM: 0x000001fa80000000, 0x000000007e200000
[ 7.912505] xe 0000:43:00.2: [drm] Available VRAM: 0x000001fa80000000, 0x000000007e200000
[ 7.988996] xe 0000:43:00.2: [drm] GT0: ccs1 fused off
[ 7.989004] xe 0000:43:00.2: [drm] GT0: ccs2 fused off
[ 7.989006] xe 0000:43:00.2: [drm] GT0: ccs3 fused off
[ 7.995815] xe 0000:43:00.2: [drm] GT1: vcs1 fused off
[ 7.995820] xe 0000:43:00.2: [drm] GT1: vcs3 fused off
[ 7.995823] xe 0000:43:00.2: [drm] GT1: vcs4 fused off
[ 7.995826] xe 0000:43:00.2: [drm] GT1: vcs5 fused off
[ 7.995829] xe 0000:43:00.2: [drm] GT1: vcs6 fused off
[ 7.995832] xe 0000:43:00.2: [drm] GT1: vcs7 fused off
[ 7.995835] xe 0000:43:00.2: [drm] GT1: vecs2 fused off
[ 7.995837] xe 0000:43:00.2: [drm] GT1: vecs3 fused off
[ 8.002381] [drm] Initialized xe 1.1.0 for 0000:43:00.2 on minor 3
[ 8.002732] pci 0000:43:00.3: [8086:e212] type 00 class 0x030000 PCIe Endpoint
[ 8.003124] pci 0000:43:00.3: Adding to iommu group 79
[ 8.003223] pci 0000:43:00.3: vgaarb: no bridge control possible
[ 8.003226] pci 0000:43:00.3: vgaarb: VGA device added: decodes=io+mem,owns=none,locks=none
[ 8.003231] xe 0000:43:00.0: vgaarb: VGA decodes changed: olddecodes=none,decodes=none:owns=io
[ 8.003317] xe 0000:43:00.3: enabling device (0000 -> 0002)
[ 8.003583] xe 0000:43:00.3: [drm] Running in SR-IOV VF mode
[ 8.004287] xe 0000:43:00.3: [drm] VF: migration not supported by this module version
[ 8.005058] xe 0000:43:00.3: [drm] VISIBLE VRAM: 0x000001fb00000000, 0x0000000080000000
[ 8.005063] xe 0000:43:00.3: [drm] VRAM[0, 0]: Actual physical size 0x000000007e200000, usable size exclude stolen 0x000000007e200000, CPU accessible size 0x000000007e200000
[ 8.005068] xe 0000:43:00.3: [drm] VRAM[0, 0]: DPA range: [0x0000000000000000-7e200000], io range: [0x000001fb00000000-1fb7e200000]
[ 8.005072] xe 0000:43:00.3: [drm] Total VRAM: 0x000001fb00000000, 0x000000007e200000
[ 8.005076] xe 0000:43:00.3: [drm] Available VRAM: 0x000001fb00000000, 0x000000007e200000
[ 8.075017] xe 0000:43:00.3: [drm] GT0: ccs1 fused off
[ 8.075025] xe 0000:43:00.3: [drm] GT0: ccs2 fused off
[ 8.075028] xe 0000:43:00.3: [drm] GT0: ccs3 fused off
[ 8.081458] xe 0000:43:00.3: [drm] GT1: vcs1 fused off
[ 8.081463] xe 0000:43:00.3: [drm] GT1: vcs3 fused off
[ 8.081466] xe 0000:43:00.3: [drm] GT1: vcs4 fused off
[ 8.081469] xe 0000:43:00.3: [drm] GT1: vcs5 fused off
[ 8.081471] xe 0000:43:00.3: [drm] GT1: vcs6 fused off
[ 8.081474] xe 0000:43:00.3: [drm] GT1: vcs7 fused off
[ 8.081477] xe 0000:43:00.3: [drm] GT1: vecs2 fused off
[ 8.081480] xe 0000:43:00.3: [drm] GT1: vecs3 fused off
[ 8.088023] [drm] Initialized xe 1.1.0 for 0000:43:00.3 on minor 4
[ 8.088345] pci 0000:43:00.4: [8086:e212] type 00 class 0x030000 PCIe Endpoint
[ 8.088640] pci 0000:43:00.4: Adding to iommu group 80
[ 8.088730] pci 0000:43:00.4: vgaarb: no bridge control possible
[ 8.088733] pci 0000:43:00.4: vgaarb: VGA device added: decodes=io+mem,owns=none,locks=none
[ 8.088737] xe 0000:43:00.0: vgaarb: VGA decodes changed: olddecodes=none,decodes=none:owns=io
[ 8.088814] xe 0000:43:00.4: enabling device (0000 -> 0002)
[ 8.089011] xe 0000:43:00.4: [drm] Running in SR-IOV VF mode
[ 8.089381] xe 0000:43:00.4: [drm] VF: migration not supported by this module version
[ 8.089984] xe 0000:43:00.4: [drm] VISIBLE VRAM: 0x000001fb80000000, 0x0000000080000000
[ 8.089989] xe 0000:43:00.4: [drm] VRAM[0, 0]: Actual physical size 0x000000007e200000, usable size exclude stolen 0x000000007e200000, CPU accessible size 0x000000007e200000
[ 8.089993] xe 0000:43:00.4: [drm] VRAM[0, 0]: DPA range: [0x0000000000000000-7e200000], io range: [0x000001fb80000000-1fbfe200000]
[ 8.089997] xe 0000:43:00.4: [drm] Total VRAM: 0x000001fb80000000, 0x000000007e200000
[ 8.090001] xe 0000:43:00.4: [drm] Available VRAM: 0x000001fb80000000, 0x000000007e200000
[ 8.158000] xe 0000:43:00.4: [drm] GT0: ccs1 fused off
[ 8.158008] xe 0000:43:00.4: [drm] GT0: ccs2 fused off
[ 8.158011] xe 0000:43:00.4: [drm] GT0: ccs3 fused off
[ 8.164538] xe 0000:43:00.4: [drm] GT1: vcs1 fused off
[ 8.164543] xe 0000:43:00.4: [drm] GT1: vcs3 fused off
[ 8.164546] xe 0000:43:00.4: [drm] GT1: vcs4 fused off
[ 8.164549] xe 0000:43:00.4: [drm] GT1: vcs5 fused off
[ 8.164552] xe 0000:43:00.4: [drm] GT1: vcs6 fused off
[ 8.164555] xe 0000:43:00.4: [drm] GT1: vcs7 fused off
[ 8.164557] xe 0000:43:00.4: [drm] GT1: vecs2 fused off
[ 8.164560] xe 0000:43:00.4: [drm] GT1: vecs3 fused off
[ 8.171000] [drm] Initialized xe 1.1.0 for 0000:43:00.4 on minor 5
[ 8.171372] pci 0000:43:00.5: [8086:e212] type 00 class 0x030000 PCIe Endpoint
[ 8.171980] pci 0000:43:00.5: Adding to iommu group 81
[ 8.172235] pci 0000:43:00.5: vgaarb: no bridge control possible
[ 8.172239] pci 0000:43:00.5: vgaarb: VGA device added: decodes=io+mem,owns=none,locks=none
[ 8.172244] xe 0000:43:00.0: vgaarb: VGA decodes changed: olddecodes=none,decodes=none:owns=io
[ 8.172445] xe 0000:43:00.5: enabling device (0000 -> 0002)
[ 8.172773] xe 0000:43:00.5: [drm] Running in SR-IOV VF mode
[ 8.173215] xe 0000:43:00.5: [drm] VF: migration not supported by this module version
[ 8.174357] xe 0000:43:00.5: [drm] VISIBLE VRAM: 0x000001fc00000000, 0x0000000080000000
[ 8.174363] xe 0000:43:00.5: [drm] VRAM[0, 0]: Actual physical size 0x000000007e200000, usable size exclude stolen 0x000000007e200000, CPU accessible size 0x000000007e200000
[ 8.174367] xe 0000:43:00.5: [drm] VRAM[0, 0]: DPA range: [0x0000000000000000-7e200000], io range: [0x000001fc00000000-1fc7e200000]
[ 8.174372] xe 0000:43:00.5: [drm] Total VRAM: 0x000001fc00000000, 0x000000007e200000
[ 8.174375] xe 0000:43:00.5: [drm] Available VRAM: 0x000001fc00000000, 0x000000007e200000
[ 8.238999] xe 0000:43:00.5: [drm] GT0: ccs1 fused off
[ 8.239007] xe 0000:43:00.5: [drm] GT0: ccs2 fused off
[ 8.239010] xe 0000:43:00.5: [drm] GT0: ccs3 fused off
[ 8.245590] xe 0000:43:00.5: [drm] GT1: vcs1 fused off
[ 8.245595] xe 0000:43:00.5: [drm] GT1: vcs3 fused off
[ 8.245598] xe 0000:43:00.5: [drm] GT1: vcs4 fused off
[ 8.245601] xe 0000:43:00.5: [drm] GT1: vcs5 fused off
[ 8.245604] xe 0000:43:00.5: [drm] GT1: vcs6 fused off
[ 8.245606] xe 0000:43:00.5: [drm] GT1: vcs7 fused off
[ 8.245609] xe 0000:43:00.5: [drm] GT1: vecs2 fused off
[ 8.245612] xe 0000:43:00.5: [drm] GT1: vecs3 fused off
[ 8.251780] [drm] Initialized xe 1.1.0 for 0000:43:00.5 on minor 6
[ 8.252142] pci 0000:43:00.6: [8086:e212] type 00 class 0x030000 PCIe Endpoint
[ 8.253493] pci 0000:43:00.6: Adding to iommu group 82
[ 8.253525] pci 0000:43:00.6: vgaarb: no bridge control possible
[ 8.253529] pci 0000:43:00.6: vgaarb: VGA device added: decodes=io+mem,owns=none,locks=none
[ 8.253534] xe 0000:43:00.0: vgaarb: VGA decodes changed: olddecodes=none,decodes=none:owns=io
[ 8.253642] xe 0000:43:00.6: enabling device (0000 -> 0002)
[ 8.253942] xe 0000:43:00.6: [drm] Running in SR-IOV VF mode
[ 8.254244] xe 0000:43:00.6: [drm] VF: migration not supported by this module version
[ 8.254923] xe 0000:43:00.6: [drm] VISIBLE VRAM: 0x000001fc80000000, 0x0000000080000000
[ 8.254930] xe 0000:43:00.6: [drm] VRAM[0, 0]: Actual physical size 0x000000007e200000, usable size exclude stolen 0x000000007e200000, CPU accessible size 0x000000007e200000
[ 8.254935] xe 0000:43:00.6: [drm] VRAM[0, 0]: DPA range: [0x0000000000000000-7e200000], io range: [0x000001fc80000000-1fcfe200000]
[ 8.254940] xe 0000:43:00.6: [drm] Total VRAM: 0x000001fc80000000, 0x000000007e200000
[ 8.254944] xe 0000:43:00.6: [drm] Available VRAM: 0x000001fc80000000, 0x000000007e200000
[ 8.321327] xe 0000:43:00.6: [drm] GT0: ccs1 fused off
[ 8.321335] xe 0000:43:00.6: [drm] GT0: ccs2 fused off
[ 8.321338] xe 0000:43:00.6: [drm] GT0: ccs3 fused off
[ 8.327654] xe 0000:43:00.6: [drm] GT1: vcs1 fused off
[ 8.327659] xe 0000:43:00.6: [drm] GT1: vcs3 fused off
[ 8.327662] xe 0000:43:00.6: [drm] GT1: vcs4 fused off
[ 8.327664] xe 0000:43:00.6: [drm] GT1: vcs5 fused off
[ 8.327667] xe 0000:43:00.6: [drm] GT1: vcs6 fused off
[ 8.327670] xe 0000:43:00.6: [drm] GT1: vcs7 fused off
[ 8.327673] xe 0000:43:00.6: [drm] GT1: vecs2 fused off
[ 8.327675] xe 0000:43:00.6: [drm] GT1: vecs3 fused off
[ 8.332204] [drm] Initialized xe 1.1.0 for 0000:43:00.6 on minor 7
[ 8.332602] pci 0000:43:00.7: [8086:e212] type 00 class 0x030000 PCIe Endpoint
[ 8.333068] pci 0000:43:00.7: Adding to iommu group 83
[ 8.333096] pci 0000:43:00.7: vgaarb: no bridge control possible
[ 8.333100] pci 0000:43:00.7: vgaarb: VGA device added: decodes=io+mem,owns=none,locks=none
[ 8.333105] xe 0000:43:00.0: vgaarb: VGA decodes changed: olddecodes=none,decodes=none:owns=io
[ 8.333217] xe 0000:43:00.7: enabling device (0000 -> 0002)
[ 8.333493] xe 0000:43:00.7: [drm] Running in SR-IOV VF mode
[ 8.333761] xe 0000:43:00.7: [drm] VF: migration not supported by this module version
[ 8.334429] xe 0000:43:00.7: [drm] VISIBLE VRAM: 0x000001fd00000000, 0x0000000080000000
[ 8.334436] xe 0000:43:00.7: [drm] VRAM[0, 0]: Actual physical size 0x000000007e200000, usable size exclude stolen 0x000000007e200000, CPU accessible size 0x000000007e200000
[ 8.334441] xe 0000:43:00.7: [drm] VRAM[0, 0]: DPA range: [0x0000000000000000-7e200000], io range: [0x000001fd00000000-1fd7e200000]
[ 8.334447] xe 0000:43:00.7: [drm] Total VRAM: 0x000001fd00000000, 0x000000007e200000
[ 8.334451] xe 0000:43:00.7: [drm] Available VRAM: 0x000001fd00000000, 0x000000007e200000
[ 8.400296] xe 0000:43:00.7: [drm] GT0: ccs1 fused off
[ 8.400304] xe 0000:43:00.7: [drm] GT0: ccs2 fused off
[ 8.400307] xe 0000:43:00.7: [drm] GT0: ccs3 fused off
[ 8.406450] xe 0000:43:00.7: [drm] GT1: vcs1 fused off
[ 8.406455] xe 0000:43:00.7: [drm] GT1: vcs3 fused off
[ 8.406458] xe 0000:43:00.7: [drm] GT1: vcs4 fused off
[ 8.406461] xe 0000:43:00.7: [drm] GT1: vcs5 fused off
[ 8.406464] xe 0000:43:00.7: [drm] GT1: vcs6 fused off
[ 8.406467] xe 0000:43:00.7: [drm] GT1: vcs7 fused off
[ 8.406469] xe 0000:43:00.7: [drm] GT1: vecs2 fused off
[ 8.406472] xe 0000:43:00.7: [drm] GT1: vecs3 fused off
[ 8.410975] [drm] Initialized xe 1.1.0 for 0000:43:00.7 on minor 8
[ 8.411317] pci 0000:43:01.0: [8086:e212] type 00 class 0x030000 PCIe Endpoint
[ 8.411704] pci 0000:43:01.0: Adding to iommu group 84
[ 8.411729] pci 0000:43:01.0: vgaarb: no bridge control possible
[ 8.411732] pci 0000:43:01.0: vgaarb: VGA device added: decodes=io+mem,owns=none,locks=none
[ 8.411736] xe 0000:43:00.0: vgaarb: VGA decodes changed: olddecodes=none,decodes=none:owns=io
[ 8.411858] xe 0000:43:01.0: enabling device (0000 -> 0002)
[ 8.412120] xe 0000:43:01.0: [drm] Running in SR-IOV VF mode
[ 8.412365] xe 0000:43:01.0: [drm] VF: migration not supported by this module version
[ 8.413031] xe 0000:43:01.0: [drm] VISIBLE VRAM: 0x000001fd80000000, 0x0000000080000000
[ 8.413036] xe 0000:43:01.0: [drm] VRAM[0, 0]: Actual physical size 0x000000007e200000, usable size exclude stolen 0x000000007e200000, CPU accessible size 0x000000007e200000
[ 8.413040] xe 0000:43:01.0: [drm] VRAM[0, 0]: DPA range: [0x0000000000000000-7e200000], io range: [0x000001fd80000000-1fdfe200000]
[ 8.413044] xe 0000:43:01.0: [drm] Total VRAM: 0x000001fd80000000, 0x000000007e200000
[ 8.413048] xe 0000:43:01.0: [drm] Available VRAM: 0x000001fd80000000, 0x000000007e200000
[ 8.481807] xe 0000:43:01.0: [drm] GT0: ccs1 fused off
[ 8.481814] xe 0000:43:01.0: [drm] GT0: ccs2 fused off
[ 8.481817] xe 0000:43:01.0: [drm] GT0: ccs3 fused off
[ 8.487976] xe 0000:43:01.0: [drm] GT1: vcs1 fused off
[ 8.487981] xe 0000:43:01.0: [drm] GT1: vcs3 fused off
[ 8.487984] xe 0000:43:01.0: [drm] GT1: vcs4 fused off
[ 8.487987] xe 0000:43:01.0: [drm] GT1: vcs5 fused off
[ 8.487989] xe 0000:43:01.0: [drm] GT1: vcs6 fused off
[ 8.487992] xe 0000:43:01.0: [drm] GT1: vcs7 fused off
[ 8.487995] xe 0000:43:01.0: [drm] GT1: vecs2 fused off
[ 8.487998] xe 0000:43:01.0: [drm] GT1: vecs3 fused off
[ 8.492626] [drm] Initialized xe 1.1.0 for 0000:43:01.0 on minor 9
[ 8.493811] xe 0000:43:00.0: [drm] PF: Enabled 8 of 12 VFs
[ 318.278188] xe 0000:43:00.0: 8 VFs already enabled. Disable before enabling 4 VFs
it seems you have way more better progress then me.
My issue with xe is
[ 4385.371721] xe 0000:43:00.0: [drm] *ERROR* failed to map registers
[ 4385.371724] xe 0000:43:00.0: probe with driver xe failed with error -5
and nothing more happened, can’t load xe driver.
your logs is
[ 6.393904] xe 0000:43:00.0: BAR 2 [mem 0x20000000000-0x2000fffffff 64bit pref]: releasing
[ 6.393926] xe 0000:43:00.0: [drm] Failed to resize BAR2 to 16384M (-ENOENT). Consider enabling 'Resizable BAR' support in your BIOS
check your Resizable BAR feature in the bios. For example each PCI slot of my device has feature Resizable BARs.
my bios does not have rebar support.
You can try checking this in Windows, but it may display completely different information.
I don’t know why, but previously in the Arc Software program, rebar was indicated in green.
If not, try to upgrade your mb BIOS
Hey, do you know, there is any possible issue with firmware of B50?
One I install install B50 into my workstation I was able to get xe driver in use.
Kernel driver in use: xe
Kernel modules: xe
But SR-IOV wasn’t available, after cycle to windows and driver installation, i got SR-IOV label, but now it can’t load the driver.
I think to try another motherboard with this device, hope it would work.
you will have xe on the .0 device and vfio-pci on the .1, .2 .2 .3 etc devices
work is being done to allow sriov control in a generic way from the vfio driver
dmesg should give you clues about why it stops working with sriov
I think it motherboard issue
truebest@hpws:~$ sudo dmesg | grep 43:00.0
[ 1.282899] pci 0000:43:00.0: [8086:e212] type 00 class 0x030000 PCIe Endpoint
[ 1.282929] pci 0000:43:00.0: BAR 0 [mem 0x11800000000-0x11800ffffff 64bit pref]
[ 1.282933] pci 0000:43:00.0: BAR 2 [mem 0x11400000000-0x117ffffffff 64bit pref]
[ 1.282936] pci 0000:43:00.0: ROM [mem 0xffe00000-0xffffffff pref]
[ 1.283005] pci 0000:43:00.0: PME# supported from D0 D3hot
[ 1.283040] pci 0000:43:00.0: VF BAR 0 [mem 0x00000000-0x00ffffff 64bit pref]
[ 1.283042] pci 0000:43:00.0: VF BAR 0 [mem 0x00000000-0x0bffffff 64bit pref]: contains BAR 0 for 12 VFs
[ 1.283045] pci 0000:43:00.0: VF BAR 2 [mem 0x00000000-0x7fffffff 64bit pref]
[ 1.283047] pci 0000:43:00.0: VF BAR 2 [mem 0x00000000-0x5ffffffff 64bit pref]: contains BAR 2 for 12 VFs
[ 1.303834] pci 0000:43:00.0: vgaarb: bridge control possible
[ 1.303835] pci 0000:43:00.0: vgaarb: VGA device added: decodes=io+mem,owns=none,locks=none
[ 1.306959] pnp 00:00: disabling [mem 0xe0000000-0xefffffff] because it overlaps 0000:43:00.0 BAR 9 [mem 0x00000000-0x5ffffffff 64bit pref]
[ 1.307323] pnp 00:03: disabling [mem 0xfedf1000-0xfedf1fff] because it overlaps 0000:43:00.0 BAR 9 [mem 0x00000000-0x5ffffffff 64bit pref]
[ 1.327631] pci 0000:43:00.0: ROM [mem 0xffe00000-0xffffffff pref]: can't claim; no compatible bridge window
[ 1.328031] pci 0000:43:00.0: VF BAR 2 [mem size 0x600000000 64bit pref]: can't assign; no space
[ 1.328033] pci 0000:43:00.0: VF BAR 2 [mem size 0x600000000 64bit pref]: failed to assign
[ 1.328035] pci 0000:43:00.0: VF BAR 0 [mem size 0x0c000000 64bit pref]: can't assign; no space
[ 1.328037] pci 0000:43:00.0: VF BAR 0 [mem size 0x0c000000 64bit pref]: failed to assign
[ 1.328038] pci 0000:43:00.0: ROM [mem size 0x00200000 pref]: can't assign; no space
[ 1.328040] pci 0000:43:00.0: ROM [mem size 0x00200000 pref]: failed to assign
[ 1.328530] pci 0000:43:00.0: BAR 2 [mem size 0x400000000 64bit pref]: can't assign; no space
[ 1.328531] pci 0000:43:00.0: BAR 2 [mem size 0x400000000 64bit pref]: failed to assign
[ 1.328533] pci 0000:43:00.0: VF BAR 2 [mem size 0x600000000 64bit pref]: can't assign; no space
[ 1.328535] pci 0000:43:00.0: VF BAR 2 [mem size 0x600000000 64bit pref]: failed to assign
[ 1.328536] pci 0000:43:00.0: BAR 0 [mem size 0x01000000 64bit pref]: can't assign; no space
[ 1.328538] pci 0000:43:00.0: BAR 0 [mem size 0x01000000 64bit pref]: failed to assign
[ 1.328539] pci 0000:43:00.0: VF BAR 0 [mem size 0x0c000000 64bit pref]: can't assign; no space
[ 1.328541] pci 0000:43:00.0: VF BAR 0 [mem size 0x0c000000 64bit pref]: failed to assign
[ 1.328543] pci 0000:43:00.0: ROM [mem size 0x00200000 pref]: can't assign; no space
[ 1.328544] pci 0000:43:00.0: ROM [mem size 0x00200000 pref]: failed to assign
[ 1.328546] pci 0000:43:00.0: BAR 2 [mem size 0x400000000 64bit pref]: can't assign; no space
[ 1.328547] pci 0000:43:00.0: BAR 2 [mem size 0x400000000 64bit pref]: failed to assign
[ 1.328549] pci 0000:43:00.0: BAR 0 [mem size 0x01000000 64bit pref]: can't assign; no space
[ 1.328551] pci 0000:43:00.0: BAR 0 [mem size 0x01000000 64bit pref]: failed to assign
[ 1.329163] pci 0000:43:00.0: BAR 2 [mem size 0x400000000 64bit pref]: can't assign; no space
[ 1.329165] pci 0000:43:00.0: BAR 2 [mem size 0x400000000 64bit pref]: failed to assign
[ 1.329167] pci 0000:43:00.0: VF BAR 2 [mem size 0x600000000 64bit pref]: can't assign; no space
[ 1.329168] pci 0000:43:00.0: VF BAR 2 [mem size 0x600000000 64bit pref]: failed to assign
[ 1.329170] pci 0000:43:00.0: BAR 0 [mem size 0x01000000 64bit pref]: can't assign; no space
[ 1.329171] pci 0000:43:00.0: BAR 0 [mem size 0x01000000 64bit pref]: failed to assign
[ 1.329173] pci 0000:43:00.0: VF BAR 0 [mem size 0x0c000000 64bit pref]: can't assign; no space
[ 1.329175] pci 0000:43:00.0: VF BAR 0 [mem size 0x0c000000 64bit pref]: failed to assign
[ 1.329176] pci 0000:43:00.0: ROM [mem size 0x00200000 pref]: can't assign; no space
[ 1.329178] pci 0000:43:00.0: ROM [mem size 0x00200000 pref]: failed to assign
[ 1.329179] pci 0000:43:00.0: BAR 2 [mem size 0x400000000 64bit pref]: can't assign; no space
[ 1.329181] pci 0000:43:00.0: BAR 2 [mem size 0x400000000 64bit pref]: failed to assign
[ 1.329183] pci 0000:43:00.0: BAR 0 [mem size 0x01000000 64bit pref]: can't assign; no space
[ 1.329184] pci 0000:43:00.0: BAR 0 [mem size 0x01000000 64bit pref]: failed to assign
[ 1.329519] pci 0000:43:00.0: BAR 2 [mem size 0x400000000 64bit pref]: can't assign; no space
[ 1.329521] pci 0000:43:00.0: BAR 2 [mem size 0x400000000 64bit pref]: failed to assign
[ 1.329522] pci 0000:43:00.0: VF BAR 2 [mem size 0x600000000 64bit pref]: can't assign; no space
[ 1.329524] pci 0000:43:00.0: VF BAR 2 [mem size 0x600000000 64bit pref]: failed to assign
[ 1.329525] pci 0000:43:00.0: BAR 0 [mem size 0x01000000 64bit pref]: can't assign; no space
[ 1.329527] pci 0000:43:00.0: BAR 0 [mem size 0x01000000 64bit pref]: failed to assign
[ 1.329529] pci 0000:43:00.0: VF BAR 0 [mem size 0x0c000000 64bit pref]: can't assign; no space
[ 1.329530] pci 0000:43:00.0: VF BAR 0 [mem size 0x0c000000 64bit pref]: failed to assign
[ 1.329532] pci 0000:43:00.0: ROM [mem 0x80000000-0x801fffff pref]: assigned
[ 1.329534] pci 0000:43:00.0: BAR 2 [mem size 0x400000000 64bit pref]: can't assign; no space
[ 1.329535] pci 0000:43:00.0: BAR 2 [mem size 0x400000000 64bit pref]: failed to assign
[ 1.329537] pci 0000:43:00.0: BAR 0 [mem size 0x01000000 64bit pref]: can't assign; no space
[ 1.329539] pci 0000:43:00.0: BAR 0 [mem size 0x01000000 64bit pref]: failed to assign
[ 1.329540] pci 0000:43:00.0: ROM [mem 0x80000000-0x801fffff pref]: assigned
[ 1.329542] pci 0000:43:00.0: VF BAR 2 [mem size 0x600000000 64bit pref]: can't assign; no space
[ 1.329544] pci 0000:43:00.0: VF BAR 2 [mem size 0x600000000 64bit pref]: failed to assign
[ 1.329545] pci 0000:43:00.0: VF BAR 0 [mem size 0x0c000000 64bit pref]: can't assign; no space
[ 1.329547] pci 0000:43:00.0: VF BAR 0 [mem size 0x0c000000 64bit pref]: failed to assign
[ 1.339505] pci 0000:43:00.0: Adding to iommu group 52
[ 22.356116] xe 0000:43:00.0: [drm] *ERROR* failed to map registers
[ 22.356150] xe 0000:43:00.0: probe with driver xe failed with error -5
truebest@hpws:~$
(Windows) side is fine, But Linux logs are the clincher: they show the PCI core can’t assign any of the GPU’s BARs (and even the VF BARs) because the upstream bridge windows are too small:
43:00.0: BAR 2 [size 0x400000000 64bit pref]: can’t assign; no space
43:00.0: BAR 0 [size 0x01000000 64bit pref]: can’t assign; no space
One card start support SR-IOV, it can't assign a space
Capabilities: [320 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration- 10BitTagReq+ IntMsgNum 0
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+ 10BitTagReq-
IOVSta: Migration-
Initial VFs: 12, Total VFs: 12, Number of VFs: 0, Function Dependency Link: 00
VF offset: 1, stride: 1, Device ID: e212
Supported Page Size: 00000553, System Page Size: 00000001
Region 0: Memory at 0000000000000000 (64-bit, prefetchable)
Region 2: Memory at 0000000000000000 (64-bit, prefetchable)
Can’t load the driver. I also tried ubuntu 25 with strongtz dkms driver, got exactly same result. I also tried manually load the driver, got io -5
It seems to be HP workstations (not servers) is luck of SR-IOV support in BIOS or any other mb/bios troubles. I assume something should be activated in chipset/bios/etc.
Overall i can confirm, that is my HP Z6 G5A motherboard issues.
I just take from storage my old NAS based on X570S I AORUS PRO AX with AMD 5700X, put my B50 into it.
I literly followed by instruction and finally got
root@nas:/# lspci -s 0c:00
0c:00.0 VGA compatible controller: Intel Corporation Battlemage G21 [Intel Graphics]
0c:00.1 VGA compatible controller: Intel Corporation Battlemage G21 [Intel Graphics]
0c:00.2 VGA compatible controller: Intel Corporation Battlemage G21 [Intel Graphics]
0c:00.3 VGA compatible controller: Intel Corporation Battlemage G21 [Intel Graphics]
0c:00.4 VGA compatible controller: Intel Corporation Battlemage G21 [Intel Graphics]
I’ll work with HP support on it.
