Hello everyone,
TL:DR: My board seem to not have enough MMIO resources for SR-IOV.
Recently I updated drivers on my Intel Arc Pro B60 ( Proxmox host, updated driver in Windows VM ) which enabled SR-IOV.
All works fine in that regard as shown by lspci -vvv:
03:00.0 VGA compatible controller: Intel Corporation Battlemage G21 [Intel Graphics] (prog-if 00 [VGA controller])
Subsystem: Device 172f:0300
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin ? routed to IRQ 183
IOMMU group: 18
Region 0: Memory at 4400000000 (64-bit, prefetchable) [size=16M]
Region 2: Memory at 4800000000 (64-bit, prefetchable) [size=32G]
Expansion ROM at 50800000 [disabled] [size=2M]
Capabilities: [40] Vendor Specific Information: Len=0c <?>
Capabilities: [70] Express (v2) Endpoint, IntMsgNum 0
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W TEE-IO-
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM L1 Enabled; RCB 64 bytes, LnkDisable- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1
TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range B, TimeoutDis+ NROPrPrP- LTR+
10BitTagComp+ 10BitTagReq+ OBFF Not Supported, ExtFmt+ EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS- TPHComp- ExtTPHComp-
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
AtomicOpsCtl: ReqEn-
IDOReq- IDOCompl- LTR+ EmergencyPowerReductionReq-
10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
LnkCap2: Supported Link Speeds: 2.5GT/s, Crosslink- Retimer- 2Retimers- DRS-
LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [ac] MSI: Enable+ Count=1/1 Maskable+ 64bit+
Address: 00000000fee00a18 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [d0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [110 v1] Null
Capabilities: [200 v1] Address Translation Service (ATS)
ATSCap: Invalidate Queue Depth: 00
ATSCtl: Enable-, Smallest Translation Unit: 00
Capabilities: [420 v1] Physical Resizable BAR
BAR 2: current size: 32GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB
Capabilities: [220 v1] Virtual Resizable BAR
BAR 2: current size: 8GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB
Capabilities: [320 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration- 10BitTagReq+ IntMsgNum 0
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+ 10BitTagReq-
IOVSta: Migration-
Initial VFs: 7, Total VFs: 7, Number of VFs: 0, Function Dependency Link: 00
VF offset: 1, stride: 1, Device ID: e211
Supported Page Size: 00000553, System Page Size: 00000001
Region 0: Memory at 0000004401000000 (64-bit, prefetchable)
Region 2: Memory at 0000000000000000 (64-bit, prefetchable)
VF Migration: offset: 00000000, BIR: 0
Capabilities: [400 v1] Latency Tolerance Reporting
Max snoop latency: 15728640ns
Max no snoop latency: 15728640ns
Kernel driver in use: xe
Kernel modules: xe
I already have a working SR-IOV setup with 7 VFās provided by iGPU in my i5 13500 ( using the GitHub - strongtz/i915-sriov-dkms: dkms module of Linux i915 driver with SR-IOV support ) so I just assumed it will work.
However for the past three days I have been fighting with the following errors:
dmesg | grep '03:00.0'
[ 0.444260] pci 0000:03:00.0: [8086:e211] type 00 class 0x030000 PCIe Endpoint
[ 0.444290] pci 0000:03:00.0: BAR 0 [mem 0x6800000000-0x6800ffffff 64bit pref]
[ 0.444292] pci 0000:03:00.0: BAR 2 [mem 0x6000000000-0x67ffffffff 64bit pref]
[ 0.444296] pci 0000:03:00.0: ROM [mem 0x50800000-0x509fffff pref]
[ 0.444363] pci 0000:03:00.0: PME# supported from D0 D3hot
[ 0.444398] pci 0000:03:00.0: VF BAR 0 [mem 0x00000000-0x00ffffff 64bit pref]
[ 0.444399] pci 0000:03:00.0: VF BAR 0 [mem 0x00000000-0x06ffffff 64bit pref]: contains BAR 0 for 7 VFs
[ 0.444402] pci 0000:03:00.0: VF BAR 2 [mem 0x00000000-0x1ffffffff 64bit pref]
[ 0.444403] pci 0000:03:00.0: VF BAR 2 [mem 0x00000000-0xdffffffff 64bit pref]: contains BAR 2 for 7 VFs
[ 0.467897] pci 0000:03:00.0: vgaarb: bridge control possible
[ 0.467897] pci 0000:03:00.0: vgaarb: VGA device added: decodes=io+mem,owns=none,locks=none
[ 0.472599] pnp 00:03: disabling [mem 0xfedc0000-0xfedc7fff] because it overlaps 0000:03:00.0 BAR 9 [mem 0x00000000-0xdffffffff 64bit pref]
[ 0.472601] pnp 00:03: disabling [mem 0xfeda0000-0xfeda0fff] because it overlaps 0000:03:00.0 BAR 9 [mem 0x00000000-0xdffffffff 64bit pref]
[ 0.472603] pnp 00:03: disabling [mem 0xfeda1000-0xfeda1fff] because it overlaps 0000:03:00.0 BAR 9 [mem 0x00000000-0xdffffffff 64bit pref]
[ 0.472604] pnp 00:03: disabling [mem 0xc0000000-0xcfffffff disabled] because it overlaps 0000:03:00.0 BAR 9 [mem 0x00000000-0xdffffffff 64bit pref]
[ 0.472606] pnp 00:03: disabling [mem 0xfed20000-0xfed7ffff] because it overlaps 0000:03:00.0 BAR 9 [mem 0x00000000-0xdffffffff 64bit pref]
[ 0.472607] pnp 00:03: disabling [mem 0xfed90000-0xfed93fff] because it overlaps 0000:03:00.0 BAR 9 [mem 0x00000000-0xdffffffff 64bit pref]
[ 0.472608] pnp 00:03: disabling [mem 0xfed45000-0xfed8ffff] because it overlaps 0000:03:00.0 BAR 9 [mem 0x00000000-0xdffffffff 64bit pref]
[ 0.472610] pnp 00:03: disabling [mem 0xfee00000-0xfeefffff] because it overlaps 0000:03:00.0 BAR 9 [mem 0x00000000-0xdffffffff 64bit pref]
[ 0.496500] pci 0000:03:00.0: VF BAR 2 [mem size 0xe00000000 64bit pref]: can't assign; no space
[ 0.496502] pci 0000:03:00.0: VF BAR 2 [mem size 0xe00000000 64bit pref]: failed to assign
[ 0.496504] pci 0000:03:00.0: VF BAR 0 [mem size 0x07000000 64bit pref]: can't assign; no space
[ 0.496505] pci 0000:03:00.0: VF BAR 0 [mem size 0x07000000 64bit pref]: failed to assign
[ 0.496844] pci 0000:03:00.0: BAR 2 [mem size 0x800000000 64bit pref]: can't assign; no space
[ 0.496845] pci 0000:03:00.0: BAR 2 [mem size 0x800000000 64bit pref]: failed to assign
[ 0.496846] pci 0000:03:00.0: VF BAR 2 [mem size 0xe00000000 64bit pref]: can't assign; no space
[ 0.496847] pci 0000:03:00.0: VF BAR 2 [mem size 0xe00000000 64bit pref]: failed to assign
[ 0.496848] pci 0000:03:00.0: BAR 0 [mem size 0x01000000 64bit pref]: can't assign; no space
[ 0.496849] pci 0000:03:00.0: BAR 0 [mem size 0x01000000 64bit pref]: failed to assign
[ 0.496850] pci 0000:03:00.0: VF BAR 0 [mem size 0x07000000 64bit pref]: can't assign; no space
[ 0.496851] pci 0000:03:00.0: VF BAR 0 [mem size 0x07000000 64bit pref]: failed to assign
[ 0.496853] pci 0000:03:00.0: BAR 2 [mem size 0x800000000 64bit pref]: can't assign; no space
[ 0.496854] pci 0000:03:00.0: BAR 2 [mem size 0x800000000 64bit pref]: failed to assign
[ 0.496855] pci 0000:03:00.0: BAR 0 [mem size 0x01000000 64bit pref]: can't assign; no space
[ 0.496856] pci 0000:03:00.0: BAR 0 [mem size 0x01000000 64bit pref]: failed to assign
[ 0.497203] pci 0000:03:00.0: BAR 2 [mem size 0x800000000 64bit pref]: can't assign; no space
[ 0.497204] pci 0000:03:00.0: BAR 2 [mem size 0x800000000 64bit pref]: failed to assign
[ 0.497206] pci 0000:03:00.0: VF BAR 2 [mem size 0xe00000000 64bit pref]: can't assign; no space
[ 0.497207] pci 0000:03:00.0: VF BAR 2 [mem size 0xe00000000 64bit pref]: failed to assign
[ 0.497208] pci 0000:03:00.0: BAR 0 [mem size 0x01000000 64bit pref]: can't assign; no space
[ 0.497209] pci 0000:03:00.0: BAR 0 [mem size 0x01000000 64bit pref]: failed to assign
[ 0.497210] pci 0000:03:00.0: VF BAR 0 [mem size 0x07000000 64bit pref]: can't assign; no space
[ 0.497211] pci 0000:03:00.0: VF BAR 0 [mem size 0x07000000 64bit pref]: failed to assign
[ 0.497212] pci 0000:03:00.0: ROM [mem 0x50800000-0x509fffff pref]: assigned
[ 0.497214] pci 0000:03:00.0: BAR 2 [mem size 0x800000000 64bit pref]: can't assign; no space
[ 0.497215] pci 0000:03:00.0: BAR 2 [mem size 0x800000000 64bit pref]: failed to assign
[ 0.497216] pci 0000:03:00.0: BAR 0 [mem size 0x01000000 64bit pref]: can't assign; no space
[ 0.497217] pci 0000:03:00.0: BAR 0 [mem size 0x01000000 64bit pref]: failed to assign
[ 0.497363] pci 0000:03:00.0: BAR 2 [mem 0x4800000000-0x4fffffffff 64bit pref]: assigned
[ 0.497371] pci 0000:03:00.0: VF BAR 2 [mem size 0xe00000000 64bit pref]: can't assign; no space
[ 0.497372] pci 0000:03:00.0: VF BAR 2 [mem size 0xe00000000 64bit pref]: failed to assign
[ 0.497373] pci 0000:03:00.0: BAR 0 [mem 0x4400000000-0x4400ffffff 64bit pref]: assigned
[ 0.497379] pci 0000:03:00.0: VF BAR 0 [mem 0x4401000000-0x4407ffffff 64bit pref]: assigned
[ 0.497383] pci 0000:03:00.0: ROM [mem 0x50800000-0x509fffff pref]: assigned
[ 0.497384] pci 0000:03:00.0: BAR 2 [mem 0x4800000000-0x4fffffffff 64bit pref]: assigned
[ 0.497391] pci 0000:03:00.0: BAR 0 [mem 0x4400000000-0x4400ffffff 64bit pref]: assigned
[ 0.497397] pci 0000:03:00.0: ROM [mem 0x50800000-0x509fffff pref]: assigned
[ 0.497399] pci 0000:03:00.0: VF BAR 2 [mem size 0xe00000000 64bit pref]: can't assign; no space
[ 0.497400] pci 0000:03:00.0: VF BAR 2 [mem size 0xe00000000 64bit pref]: failed to assign
[ 0.497402] pci 0000:03:00.0: VF BAR 0 [mem 0x4401000000-0x4407ffffff 64bit pref]: assigned
[ 0.499463] pci 0000:03:00.0: Adding to iommu group 18
[ 5.289475] xe 0000:03:00.0: [drm] Running in SR-IOV PF mode
[ 5.289772] xe 0000:03:00.0: [drm] Found battlemage (device ID e211) discrete display version 14.01 stepping B0
[ 5.291068] xe 0000:03:00.0: [drm] VISIBLE VRAM: 0x0000004800000000, 0x0000000800000000
[ 5.291085] xe 0000:03:00.0: [drm] VRAM[0]: Actual physical size 0x0000000600000000, usable size exclude stolen 0x00000005fa000000, CPU accessible size 0x00000005fa000000
[ 5.291086] xe 0000:03:00.0: [drm] VRAM[0]: DPA range: [0x0000000000000000-600000000], io range: [0x0000004800000000-4dfa000000]
[ 5.291087] xe 0000:03:00.0: [drm] VRAM[0]: Actual physical size 0x0000000600000000, usable size exclude stolen 0x00000005fa000000, CPU accessible size 0x00000005fa000000
[ 5.291088] xe 0000:03:00.0: [drm] VRAM[0]: DPA range: [0x0000000000000000-600000000], io range: [0x0000004800000000-4dfa000000]
[ 5.420013] xe 0000:03:00.0: [drm] Finished loading DMC firmware i915/bmg_dmc.bin (v2.6)
[ 5.422058] xe 0000:03:00.0: [drm] Tile0: GT0: Using GuC firmware from xe/bmg_guc_70.bin version 70.49.4
[ 5.510405] xe 0000:03:00.0: [drm] Tile0: GT0: ccs2 fused off
[ 5.510407] xe 0000:03:00.0: [drm] Tile0: GT0: ccs3 fused off
[ 5.535965] xe 0000:03:00.0: [drm] Tile0: GT1: Using GuC firmware from xe/bmg_guc_70.bin version 70.49.4
[ 5.538540] xe 0000:03:00.0: [drm] Tile0: GT1: Using HuC firmware from xe/bmg_huc.bin version 8.2.10
[ 5.550144] xe 0000:03:00.0: [drm] Tile0: GT1: vcs1 fused off
[ 5.550145] xe 0000:03:00.0: [drm] Tile0: GT1: vcs3 fused off
[ 5.550146] xe 0000:03:00.0: [drm] Tile0: GT1: vcs4 fused off
[ 5.550147] xe 0000:03:00.0: [drm] Tile0: GT1: vcs5 fused off
[ 5.550147] xe 0000:03:00.0: [drm] Tile0: GT1: vcs6 fused off
[ 5.550148] xe 0000:03:00.0: [drm] Tile0: GT1: vcs7 fused off
[ 5.550148] xe 0000:03:00.0: [drm] Tile0: GT1: vecs2 fused off
[ 5.550149] xe 0000:03:00.0: [drm] Tile0: GT1: vecs3 fused off
[ 5.578805] xe 0000:03:00.0: [drm] Can't init xe mei late bind missing mei component
[ 5.579619] xe 0000:03:00.0: [drm] Registered 4 planes with drm panic
[ 5.579621] [drm] Initialized xe 1.1.0 for 0000:03:00.0 on minor 1
[ 5.612087] xe 0000:03:00.0: [drm] Cannot find any crtc or sizes
[ 5.621220] xe 0000:03:00.0: [drm] Using mailbox commands for power limits
[ 5.621578] xe 0000:03:00.0: [drm] PL2 is supported on channel 0
[ 5.623302] xe 0000:03:00.0: [drm] PF: migration disabled: requires GuC version >= 70.54.0
[ 5.624312] xe 0000:03:00.0: [drm] PF: migration disabled: requires GuC version >= 70.54.0
[ 5.677317] xe 0000:03:00.0: [drm] Cannot find any crtc or sizes
[ 5.677320] snd_hda_intel 0000:04:00.0: bound 0000:03:00.0 (ops intel_audio_component_bind_ops [xe])
[ 5.709525] xe 0000:03:00.0: [drm] Cannot find any crtc or sizes
[ 6.027697] xe 0000:03:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=none:owns=io
[ 6.048745] xe 0000:03:00.0: vgaarb: VGA decodes changed: olddecodes=none,decodes=none:owns=io
[ 6.068513] xe 0000:03:00.0: vgaarb: VGA decodes changed: olddecodes=none,decodes=none:owns=io
[ 6.086581] xe 0000:03:00.0: vgaarb: VGA decodes changed: olddecodes=none,decodes=none:owns=io
[ 6.104948] xe 0000:03:00.0: vgaarb: VGA decodes changed: olddecodes=none,decodes=none:owns=io
[ 6.121129] xe 0000:03:00.0: vgaarb: VGA decodes changed: olddecodes=none,decodes=none:owns=io
[ 6.137274] xe 0000:03:00.0: vgaarb: VGA decodes changed: olddecodes=none,decodes=none:owns=io
[ 2622.497627] xe 0000:03:00.0: [drm] PF: Tile0: GT0: VF1 provisioned with 2147483648 (2.00 GiB) GGTT
[ 2622.498293] xe 0000:03:00.0: [drm] PF: Tile0: GT0: VF1 provisioned with 25492979712 (23.7 GiB) LMEM
[ 2622.498408] xe 0000:03:00.0: [drm] PF: Tile0: GT0: VF1 provisioned with 32768 GuC context IDs
[ 2622.498495] xe 0000:03:00.0: [drm] PF: Tile0: GT0: VF1 provisioned with 128 GuC doorbell IDs
[ 2622.498694] xe 0000:03:00.0: [drm] PF: Tile0: GT1: VF1 provisioned with 32768 GuC context IDs
[ 2622.499115] xe 0000:03:00.0: [drm] PF: Tile0: GT1: VF1 provisioned with 128 GuC doorbell IDs
[ 2622.499123] xe 0000:03:00.0: not enough MMIO resources for SR-IOV
[ 2622.544935] xe 0000:03:00.0: [drm] PF: Failed to enable 1 VF (-ENOMEM)
with the:
[ 2622.499123] xe 0000:03:00.0: not enough MMIO resources for SR-IOV
[ 2622.544935] xe 0000:03:00.0: [drm] PF: Failed to enable 1 VF (-ENOMEM)
being the main problem. No matter what I do I canāt enable even 1 VF on B60.
I have all the right BIOS switches enabled, above 4G decoding, VT-d, ReBAR ( called Clever Access Memory in BIOS ), SR-IOV support and CSM disabled, yet no matter what I do with GRUB and so on nothing seems to help, B60 does not create VFās while iGPU does, disabling iGPU does not make any difference, neither moving card to different PCIe slots.
The card occupies the top Gen 5 x16 slot, the slot work, just to be sure I tested it with Nvidia Tesla P100 and it is fine, I even removed CPU to inspect it and the socket, all in good working order ( I thought that board or CPU might be damaged because of LnkCap: Port #0, Speed 2.5GT/s, Width x1,, but it turns out to be bug in reporting ).
The card works without any problems with PCIe Passthrough, I even played some games on it, but SR-IOV just refuses to work.
Iām out of ideas and I suspect that my motherboard, the ASRock B760 Pro RS is just not up to the task, both:
[ 0.497399] pci 0000:03:00.0: VF BAR 2 [mem size 0xe00000000 64bit pref]: can't assign; no space
[ 0.497400] pci 0000:03:00.0: VF BAR 2 [mem size 0xe00000000 64bit pref]: failed to assign
and:
[ 2622.499123] xe 0000:03:00.0: not enough MMIO resources for SR-IOV
[ 2622.544935] xe 0000:03:00.0: [drm] PF: Failed to enable 1 VF (-ENOMEM)
seem to suggest that this is just board issue, strange considering that SR-IOV on iGPU works flawlessly but I have no other ideas at this point, I tried everyting, every BIOS switch and kernel parameter.
So I would like to ask if anybody here has similar experience with B760 or in general MMIO problems with B60 ? Perhaps upgrading to Z790 would help but that is just a guess at this point, I donāt have a a ton of experience with this stuff so any help is more than welcome.