Intel Arc Pro B60 SR-IOV not working with ASRock B760 Pro RS

Hello everyone,

TL:DR: My board seem to not have enough MMIO resources for SR-IOV.

Recently I updated drivers on my Intel Arc Pro B60 ( Proxmox host, updated driver in Windows VM ) which enabled SR-IOV.

All works fine in that regard as shown by lspci -vvv:

03:00.0 VGA compatible controller: Intel Corporation Battlemage G21 [Intel Graphics] (prog-if 00 [VGA controller])
        Subsystem: Device 172f:0300
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin ? routed to IRQ 183
        IOMMU group: 18
        Region 0: Memory at 4400000000 (64-bit, prefetchable) [size=16M]
        Region 2: Memory at 4800000000 (64-bit, prefetchable) [size=32G]
        Expansion ROM at 50800000 [disabled] [size=2M]
        Capabilities: [40] Vendor Specific Information: Len=0c <?>
        Capabilities: [70] Express (v2) Endpoint, IntMsgNum 0
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W TEE-IO-
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, LnkDisable- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x1
                        TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range B, TimeoutDis+ NROPrPrP- LTR+
                         10BitTagComp+ 10BitTagReq+ OBFF Not Supported, ExtFmt+ EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
                         AtomicOpsCtl: ReqEn-
                         IDOReq- IDOCompl- LTR+ EmergencyPowerReductionReq-
                         10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
                LnkCap2: Supported Link Speeds: 2.5GT/s, Crosslink- Retimer- 2Retimers- DRS-
                LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [ac] MSI: Enable+ Count=1/1 Maskable+ 64bit+
                Address: 00000000fee00a18  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [d0] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI)
                ARICap: MFVC- ACS-, Next Function: 0
                ARICtl: MFVC- ACS-, Function Group: 0
        Capabilities: [110 v1] Null
        Capabilities: [200 v1] Address Translation Service (ATS)
                ATSCap: Invalidate Queue Depth: 00
                ATSCtl: Enable-, Smallest Translation Unit: 00
        Capabilities: [420 v1] Physical Resizable BAR
                BAR 2: current size: 32GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB
        Capabilities: [220 v1] Virtual Resizable BAR
                BAR 2: current size: 8GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB
        Capabilities: [320 v1] Single Root I/O Virtualization (SR-IOV)
                IOVCap: Migration- 10BitTagReq+ IntMsgNum 0
                IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+ 10BitTagReq-
                IOVSta: Migration-
                Initial VFs: 7, Total VFs: 7, Number of VFs: 0, Function Dependency Link: 00
                VF offset: 1, stride: 1, Device ID: e211
                Supported Page Size: 00000553, System Page Size: 00000001
                Region 0: Memory at 0000004401000000 (64-bit, prefetchable)
                Region 2: Memory at 0000000000000000 (64-bit, prefetchable)
                VF Migration: offset: 00000000, BIR: 0
        Capabilities: [400 v1] Latency Tolerance Reporting
                Max snoop latency: 15728640ns
                Max no snoop latency: 15728640ns
        Kernel driver in use: xe
        Kernel modules: xe

I already have a working SR-IOV setup with 7 VF’s provided by iGPU in my i5 13500 ( using the GitHub - strongtz/i915-sriov-dkms: dkms module of Linux i915 driver with SR-IOV support ) so I just assumed it will work.
However for the past three days I have been fighting with the following errors:

dmesg | grep '03:00.0'
[    0.444260] pci 0000:03:00.0: [8086:e211] type 00 class 0x030000 PCIe Endpoint
[    0.444290] pci 0000:03:00.0: BAR 0 [mem 0x6800000000-0x6800ffffff 64bit pref]
[    0.444292] pci 0000:03:00.0: BAR 2 [mem 0x6000000000-0x67ffffffff 64bit pref]
[    0.444296] pci 0000:03:00.0: ROM [mem 0x50800000-0x509fffff pref]
[    0.444363] pci 0000:03:00.0: PME# supported from D0 D3hot
[    0.444398] pci 0000:03:00.0: VF BAR 0 [mem 0x00000000-0x00ffffff 64bit pref]
[    0.444399] pci 0000:03:00.0: VF BAR 0 [mem 0x00000000-0x06ffffff 64bit pref]: contains BAR 0 for 7 VFs
[    0.444402] pci 0000:03:00.0: VF BAR 2 [mem 0x00000000-0x1ffffffff 64bit pref]
[    0.444403] pci 0000:03:00.0: VF BAR 2 [mem 0x00000000-0xdffffffff 64bit pref]: contains BAR 2 for 7 VFs
[    0.467897] pci 0000:03:00.0: vgaarb: bridge control possible
[    0.467897] pci 0000:03:00.0: vgaarb: VGA device added: decodes=io+mem,owns=none,locks=none
[    0.472599] pnp 00:03: disabling [mem 0xfedc0000-0xfedc7fff] because it overlaps 0000:03:00.0 BAR 9 [mem 0x00000000-0xdffffffff 64bit pref]
[    0.472601] pnp 00:03: disabling [mem 0xfeda0000-0xfeda0fff] because it overlaps 0000:03:00.0 BAR 9 [mem 0x00000000-0xdffffffff 64bit pref]
[    0.472603] pnp 00:03: disabling [mem 0xfeda1000-0xfeda1fff] because it overlaps 0000:03:00.0 BAR 9 [mem 0x00000000-0xdffffffff 64bit pref]
[    0.472604] pnp 00:03: disabling [mem 0xc0000000-0xcfffffff disabled] because it overlaps 0000:03:00.0 BAR 9 [mem 0x00000000-0xdffffffff 64bit pref]
[    0.472606] pnp 00:03: disabling [mem 0xfed20000-0xfed7ffff] because it overlaps 0000:03:00.0 BAR 9 [mem 0x00000000-0xdffffffff 64bit pref]
[    0.472607] pnp 00:03: disabling [mem 0xfed90000-0xfed93fff] because it overlaps 0000:03:00.0 BAR 9 [mem 0x00000000-0xdffffffff 64bit pref]
[    0.472608] pnp 00:03: disabling [mem 0xfed45000-0xfed8ffff] because it overlaps 0000:03:00.0 BAR 9 [mem 0x00000000-0xdffffffff 64bit pref]
[    0.472610] pnp 00:03: disabling [mem 0xfee00000-0xfeefffff] because it overlaps 0000:03:00.0 BAR 9 [mem 0x00000000-0xdffffffff 64bit pref]
[    0.496500] pci 0000:03:00.0: VF BAR 2 [mem size 0xe00000000 64bit pref]: can't assign; no space
[    0.496502] pci 0000:03:00.0: VF BAR 2 [mem size 0xe00000000 64bit pref]: failed to assign
[    0.496504] pci 0000:03:00.0: VF BAR 0 [mem size 0x07000000 64bit pref]: can't assign; no space
[    0.496505] pci 0000:03:00.0: VF BAR 0 [mem size 0x07000000 64bit pref]: failed to assign
[    0.496844] pci 0000:03:00.0: BAR 2 [mem size 0x800000000 64bit pref]: can't assign; no space
[    0.496845] pci 0000:03:00.0: BAR 2 [mem size 0x800000000 64bit pref]: failed to assign
[    0.496846] pci 0000:03:00.0: VF BAR 2 [mem size 0xe00000000 64bit pref]: can't assign; no space
[    0.496847] pci 0000:03:00.0: VF BAR 2 [mem size 0xe00000000 64bit pref]: failed to assign
[    0.496848] pci 0000:03:00.0: BAR 0 [mem size 0x01000000 64bit pref]: can't assign; no space
[    0.496849] pci 0000:03:00.0: BAR 0 [mem size 0x01000000 64bit pref]: failed to assign
[    0.496850] pci 0000:03:00.0: VF BAR 0 [mem size 0x07000000 64bit pref]: can't assign; no space
[    0.496851] pci 0000:03:00.0: VF BAR 0 [mem size 0x07000000 64bit pref]: failed to assign
[    0.496853] pci 0000:03:00.0: BAR 2 [mem size 0x800000000 64bit pref]: can't assign; no space
[    0.496854] pci 0000:03:00.0: BAR 2 [mem size 0x800000000 64bit pref]: failed to assign
[    0.496855] pci 0000:03:00.0: BAR 0 [mem size 0x01000000 64bit pref]: can't assign; no space
[    0.496856] pci 0000:03:00.0: BAR 0 [mem size 0x01000000 64bit pref]: failed to assign
[    0.497203] pci 0000:03:00.0: BAR 2 [mem size 0x800000000 64bit pref]: can't assign; no space
[    0.497204] pci 0000:03:00.0: BAR 2 [mem size 0x800000000 64bit pref]: failed to assign
[    0.497206] pci 0000:03:00.0: VF BAR 2 [mem size 0xe00000000 64bit pref]: can't assign; no space
[    0.497207] pci 0000:03:00.0: VF BAR 2 [mem size 0xe00000000 64bit pref]: failed to assign
[    0.497208] pci 0000:03:00.0: BAR 0 [mem size 0x01000000 64bit pref]: can't assign; no space
[    0.497209] pci 0000:03:00.0: BAR 0 [mem size 0x01000000 64bit pref]: failed to assign
[    0.497210] pci 0000:03:00.0: VF BAR 0 [mem size 0x07000000 64bit pref]: can't assign; no space
[    0.497211] pci 0000:03:00.0: VF BAR 0 [mem size 0x07000000 64bit pref]: failed to assign
[    0.497212] pci 0000:03:00.0: ROM [mem 0x50800000-0x509fffff pref]: assigned
[    0.497214] pci 0000:03:00.0: BAR 2 [mem size 0x800000000 64bit pref]: can't assign; no space
[    0.497215] pci 0000:03:00.0: BAR 2 [mem size 0x800000000 64bit pref]: failed to assign
[    0.497216] pci 0000:03:00.0: BAR 0 [mem size 0x01000000 64bit pref]: can't assign; no space
[    0.497217] pci 0000:03:00.0: BAR 0 [mem size 0x01000000 64bit pref]: failed to assign
[    0.497363] pci 0000:03:00.0: BAR 2 [mem 0x4800000000-0x4fffffffff 64bit pref]: assigned
[    0.497371] pci 0000:03:00.0: VF BAR 2 [mem size 0xe00000000 64bit pref]: can't assign; no space
[    0.497372] pci 0000:03:00.0: VF BAR 2 [mem size 0xe00000000 64bit pref]: failed to assign
[    0.497373] pci 0000:03:00.0: BAR 0 [mem 0x4400000000-0x4400ffffff 64bit pref]: assigned
[    0.497379] pci 0000:03:00.0: VF BAR 0 [mem 0x4401000000-0x4407ffffff 64bit pref]: assigned
[    0.497383] pci 0000:03:00.0: ROM [mem 0x50800000-0x509fffff pref]: assigned
[    0.497384] pci 0000:03:00.0: BAR 2 [mem 0x4800000000-0x4fffffffff 64bit pref]: assigned
[    0.497391] pci 0000:03:00.0: BAR 0 [mem 0x4400000000-0x4400ffffff 64bit pref]: assigned
[    0.497397] pci 0000:03:00.0: ROM [mem 0x50800000-0x509fffff pref]: assigned
[    0.497399] pci 0000:03:00.0: VF BAR 2 [mem size 0xe00000000 64bit pref]: can't assign; no space
[    0.497400] pci 0000:03:00.0: VF BAR 2 [mem size 0xe00000000 64bit pref]: failed to assign
[    0.497402] pci 0000:03:00.0: VF BAR 0 [mem 0x4401000000-0x4407ffffff 64bit pref]: assigned
[    0.499463] pci 0000:03:00.0: Adding to iommu group 18
[    5.289475] xe 0000:03:00.0: [drm] Running in SR-IOV PF mode
[    5.289772] xe 0000:03:00.0: [drm] Found battlemage (device ID e211) discrete display version 14.01 stepping B0
[    5.291068] xe 0000:03:00.0: [drm] VISIBLE VRAM: 0x0000004800000000, 0x0000000800000000
[    5.291085] xe 0000:03:00.0: [drm] VRAM[0]: Actual physical size 0x0000000600000000, usable size exclude stolen 0x00000005fa000000, CPU accessible size 0x00000005fa000000
[    5.291086] xe 0000:03:00.0: [drm] VRAM[0]: DPA range: [0x0000000000000000-600000000], io range: [0x0000004800000000-4dfa000000]
[    5.291087] xe 0000:03:00.0: [drm] VRAM[0]: Actual physical size 0x0000000600000000, usable size exclude stolen 0x00000005fa000000, CPU accessible size 0x00000005fa000000
[    5.291088] xe 0000:03:00.0: [drm] VRAM[0]: DPA range: [0x0000000000000000-600000000], io range: [0x0000004800000000-4dfa000000]
[    5.420013] xe 0000:03:00.0: [drm] Finished loading DMC firmware i915/bmg_dmc.bin (v2.6)
[    5.422058] xe 0000:03:00.0: [drm] Tile0: GT0: Using GuC firmware from xe/bmg_guc_70.bin version 70.49.4
[    5.510405] xe 0000:03:00.0: [drm] Tile0: GT0: ccs2 fused off
[    5.510407] xe 0000:03:00.0: [drm] Tile0: GT0: ccs3 fused off
[    5.535965] xe 0000:03:00.0: [drm] Tile0: GT1: Using GuC firmware from xe/bmg_guc_70.bin version 70.49.4
[    5.538540] xe 0000:03:00.0: [drm] Tile0: GT1: Using HuC firmware from xe/bmg_huc.bin version 8.2.10
[    5.550144] xe 0000:03:00.0: [drm] Tile0: GT1: vcs1 fused off
[    5.550145] xe 0000:03:00.0: [drm] Tile0: GT1: vcs3 fused off
[    5.550146] xe 0000:03:00.0: [drm] Tile0: GT1: vcs4 fused off
[    5.550147] xe 0000:03:00.0: [drm] Tile0: GT1: vcs5 fused off
[    5.550147] xe 0000:03:00.0: [drm] Tile0: GT1: vcs6 fused off
[    5.550148] xe 0000:03:00.0: [drm] Tile0: GT1: vcs7 fused off
[    5.550148] xe 0000:03:00.0: [drm] Tile0: GT1: vecs2 fused off
[    5.550149] xe 0000:03:00.0: [drm] Tile0: GT1: vecs3 fused off
[    5.578805] xe 0000:03:00.0: [drm] Can't init xe mei late bind missing mei component
[    5.579619] xe 0000:03:00.0: [drm] Registered 4 planes with drm panic
[    5.579621] [drm] Initialized xe 1.1.0 for 0000:03:00.0 on minor 1
[    5.612087] xe 0000:03:00.0: [drm] Cannot find any crtc or sizes
[    5.621220] xe 0000:03:00.0: [drm] Using mailbox commands for power limits
[    5.621578] xe 0000:03:00.0: [drm] PL2 is supported on channel 0
[    5.623302] xe 0000:03:00.0: [drm] PF: migration disabled: requires GuC version >= 70.54.0
[    5.624312] xe 0000:03:00.0: [drm] PF: migration disabled: requires GuC version >= 70.54.0
[    5.677317] xe 0000:03:00.0: [drm] Cannot find any crtc or sizes
[    5.677320] snd_hda_intel 0000:04:00.0: bound 0000:03:00.0 (ops intel_audio_component_bind_ops [xe])
[    5.709525] xe 0000:03:00.0: [drm] Cannot find any crtc or sizes
[    6.027697] xe 0000:03:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=none:owns=io
[    6.048745] xe 0000:03:00.0: vgaarb: VGA decodes changed: olddecodes=none,decodes=none:owns=io
[    6.068513] xe 0000:03:00.0: vgaarb: VGA decodes changed: olddecodes=none,decodes=none:owns=io
[    6.086581] xe 0000:03:00.0: vgaarb: VGA decodes changed: olddecodes=none,decodes=none:owns=io
[    6.104948] xe 0000:03:00.0: vgaarb: VGA decodes changed: olddecodes=none,decodes=none:owns=io
[    6.121129] xe 0000:03:00.0: vgaarb: VGA decodes changed: olddecodes=none,decodes=none:owns=io
[    6.137274] xe 0000:03:00.0: vgaarb: VGA decodes changed: olddecodes=none,decodes=none:owns=io
[ 2622.497627] xe 0000:03:00.0: [drm] PF: Tile0: GT0: VF1 provisioned with 2147483648 (2.00 GiB) GGTT
[ 2622.498293] xe 0000:03:00.0: [drm] PF: Tile0: GT0: VF1 provisioned with 25492979712 (23.7 GiB) LMEM
[ 2622.498408] xe 0000:03:00.0: [drm] PF: Tile0: GT0: VF1 provisioned with 32768 GuC context IDs
[ 2622.498495] xe 0000:03:00.0: [drm] PF: Tile0: GT0: VF1 provisioned with 128 GuC doorbell IDs
[ 2622.498694] xe 0000:03:00.0: [drm] PF: Tile0: GT1: VF1 provisioned with 32768 GuC context IDs
[ 2622.499115] xe 0000:03:00.0: [drm] PF: Tile0: GT1: VF1 provisioned with 128 GuC doorbell IDs
[ 2622.499123] xe 0000:03:00.0: not enough MMIO resources for SR-IOV
[ 2622.544935] xe 0000:03:00.0: [drm] PF: Failed to enable 1 VF (-ENOMEM)

with the:

[ 2622.499123] xe 0000:03:00.0: not enough MMIO resources for SR-IOV
[ 2622.544935] xe 0000:03:00.0: [drm] PF: Failed to enable 1 VF (-ENOMEM)

being the main problem. No matter what I do I can’t enable even 1 VF on B60.

I have all the right BIOS switches enabled, above 4G decoding, VT-d, ReBAR ( called Clever Access Memory in BIOS ), SR-IOV support and CSM disabled, yet no matter what I do with GRUB and so on nothing seems to help, B60 does not create VF’s while iGPU does, disabling iGPU does not make any difference, neither moving card to different PCIe slots.

The card occupies the top Gen 5 x16 slot, the slot work, just to be sure I tested it with Nvidia Tesla P100 and it is fine, I even removed CPU to inspect it and the socket, all in good working order ( I thought that board or CPU might be damaged because of LnkCap: Port #0, Speed 2.5GT/s, Width x1,, but it turns out to be bug in reporting ).

The card works without any problems with PCIe Passthrough, I even played some games on it, but SR-IOV just refuses to work.

I’m out of ideas and I suspect that my motherboard, the ASRock B760 Pro RS is just not up to the task, both:

[    0.497399] pci 0000:03:00.0: VF BAR 2 [mem size 0xe00000000 64bit pref]: can't assign; no space
[    0.497400] pci 0000:03:00.0: VF BAR 2 [mem size 0xe00000000 64bit pref]: failed to assign

and:

[ 2622.499123] xe 0000:03:00.0: not enough MMIO resources for SR-IOV
[ 2622.544935] xe 0000:03:00.0: [drm] PF: Failed to enable 1 VF (-ENOMEM)

seem to suggest that this is just board issue, strange considering that SR-IOV on iGPU works flawlessly but I have no other ideas at this point, I tried everyting, every BIOS switch and kernel parameter.

So I would like to ask if anybody here has similar experience with B760 or in general MMIO problems with B60 ? Perhaps upgrading to Z790 would help but that is just a guess at this point, I don’t have a a ton of experience with this stuff so any help is more than welcome.

1 Like

G’day mate, same or similar issue here.

I sent the following to ASRock a few minutes ago:

Dear ASRock Support,

I am experiencing a PCIe MMIO resource issue with SR-IOV on my ASRock Z890 Pro-A WiFi motherboard when using an ASRock Intel Arc Pro B60 Passive GPU.

The B60 advertises SR-IOV with 7 VFs in PCIe capabilities. When I attempt to enable even a single VF via the standard Linux SR-IOV sysfs interface:

echo 1 > /sys/bus/pci/devices/0000:03:00.0/sriov_numvfs
[   48.673761] xe 0000:03:00.0: [drm] GT0: PF: VF1 provisioned with 4171833344 (3.88 GiB) GGTT
[   48.674082] xe 0000:03:00.0: [drm] GT0: PF: VF1 provisioned with 25501368320 (23.8 GiB) LMEM
[   48.674190] xe 0000:03:00.0: [drm] GT0: PF: VF1 provisioned with 65278 GuC context IDs
[   48.674478] xe 0000:03:00.0: [drm] GT0: PF: VF1 provisioned with 256 GuC doorbell IDs
[   48.674656] xe 0000:03:00.0: [drm] GT1: PF: VF1 provisioned with 65279 GuC context IDs
[   48.674719] xe 0000:03:00.0: [drm] GT1: PF: VF1 provisioned with 256 GuC doorbell IDs
[   48.674732] xe 0000:03:00.0: not enough MMIO resources for SR-IOV
[   48.849766] xe 0000:03:00.0: [drm] PF: Failed to enable 1 VF (-ENOMEM)

And the PCI configuration of the GPU:

03:00.0 VGA compatible controller: Intel Corporation Battlemage G21 [Arc Pro B60] (prog-if 00 [VGA controller])
  Subsystem: ASRock Incorporation Device 6024
  Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
  Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
  Latency: 0, Cache Line Size: 64 bytes
  Interrupt: pin ? routed to IRQ 323
  IOMMU group: 15
  Region 0: Memory at bc00000000 (64-bit, prefetchable) [size=16M]
  Region 2: Memory at c000000000 (64-bit, prefetchable) [size=32G]
  Expansion ROM at 8c000000 [disabled] [size=2M]
  Capabilities: [40] Vendor Specific Information: Len=0c <?>
  Capabilities: [70] Express (v2) Endpoint, IntMsgNum 0
    DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
      ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W TEE-IO-
    DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
      RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
      MaxPayload 256 bytes, MaxReadReq 512 bytes
    DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
    LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
      ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
    LnkCtl: ASPM L1 Enabled; RCB 64 bytes, LnkDisable- CommClk-
      ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
    LnkSta: Speed 2.5GT/s, Width x1
      TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
    DevCap2: Completion Timeout: Range B, TimeoutDis+ NROPrPrP- LTR+
       10BitTagComp+ 10BitTagReq+ OBFF Not Supported, ExtFmt+ EETLPPrefix-
       EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
       FRS- TPHComp- ExtTPHComp-
       AtomicOpsCap: 32bit- 64bit- 128bitCAS-
    DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
       AtomicOpsCtl: ReqEn-
       IDOReq- IDOCompl- LTR+ EmergencyPowerReductionReq-
       10BitTagReq+ OBFF Disabled, EETLPPrefixBlk-
    LnkCap2: Supported Link Speeds: 2.5GT/s, Crosslink- Retimer- 2Retimers- DRS-
    LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
       Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
       Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
    LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
       EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
       Retimer- 2Retimers- CrosslinkRes: unsupported
  Capabilities: [ac] MSI: Enable+ Count=1/1 Maskable+ 64bit+
    Address: 00000000fee01b98  Data: 0000
    Masking: 00000000  Pending: 00000000
  Capabilities: [d0] Power Management version 3
    Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
    Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
  Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI)
    ARICap: MFVC- ACS-, Next Function: 0
    ARICtl: MFVC- ACS-, Function Group: 0
  Capabilities: [110 v1] Null
  Capabilities: [200 v1] Address Translation Service (ATS)
    ATSCap: Invalidate Queue Depth: 00
    ATSCtl: Enable-, Smallest Translation Unit: 00
  Capabilities: [420 v1] Physical Resizable BAR
    BAR 2: current size: 32GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB
  Capabilities: [220 v1] Virtual Resizable BAR
    BAR 2: current size: 8GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB
  Capabilities: [320 v1] Single Root I/O Virtualization (SR-IOV)
    IOVCap: Migration- 10BitTagReq+ IntMsgNum 0
    IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+ 10BitTagReq-
    IOVSta: Migration-
    Initial VFs: 7, Total VFs: 7, Number of VFs: 0, Function Dependency Link: 00
    VF offset: 1, stride: 1, Device ID: e211
    Supported Page Size: 00000553, System Page Size: 00000001
    Region 0: Memory at 000000bc01000000 (64-bit, prefetchable)
    Region 2: Memory at 0000000000000000 (64-bit, prefetchable)
    VF Migration: offset: 00000000, BIR: 0
  Capabilities: [400 v1] Latency Tolerance Reporting
    Max snoop latency: 15728640ns
    Max no snoop latency: 15728640ns
  Kernel driver in use: xe
  Kernel modules: xe

Note that:

* The platform successfully can resize the PF BAR2 all the way down to 256MB (using the xe.vram_bar_size parameter).
* The Xe driver successfully provisions VF1 with 3.88 GiB GGTT and 23.8 GiB LMEM.
* The failure occurs when the PCI core attempts to assign MMIO space for the VF BAR2 region, at which point it reports ā€œnot enough MMIO resources for SR-IOVā€ and leaves VF BAR2 at 0.

This strongly suggests that the PCIe root complex / firmware MMIO aperture on the Z890 Pro-A WiFi is too small to accommodate the Arc Pro B60’s PF + VF BAR2 MMIO requirements, even for a single VF.

Prior troubleshooting:

* Confirmed 4G decoding, ReBAR, SR-IOV, VT-d all enabled.
* Tried kernel parameters: intel_iommu=on, iommu=pt, pci=realloc;
* Limited Xe driver to max_vfs=1 and forced small BAR2 sizes via xe.vram_bar_size=256 & 4096; and
* Despite reducing PF BAR2 and limiting to a single VF, MMIO allocation for the VF still fails with -ENOMEM.

I have found a very similar report on the Level1Techs forum for an ASRock B760 Pro RS with an Intel Arc Pro B60, where enabling VFs also fails with:
VF BAR 2 [mem size ... 64bit pref]: can't assign; no space
xe 0000:03:00.0: not enough MMIO resources for SR-IOV

See https://forum.level1techs.com/t/intel-arc-pro-b60-sr-iov-not-working-with-asrock-b760-pro-rs/246317

Could you please:

1. Confirm whether the current Z890 Pro-A WiFi BIOS reserves sufficient 64‑bit prefetchable MMIO space to support SR-IOV for large‑BAR GPUs like the Intel Arc Pro B60 (PF + up to 7 VFs); and
2. Provide a BIOS update (or beta BIOS) that increases the MMIO aperture / PCIe resource windows so that SR-IOV VF BAR2 regions for the Arc Pro B60 can be successfully mapped?

I am happy to:

* Test a beta BIOS that increases the 64‑bit MMIO window for the PCIe x16 slot used by the B60.
* Provide full dmesg and lspci -vvv output for the host bridge and GPU, if that helps engineering reproduce the issue.

At the moment, every other SR-IOV prerequisite is in place (driver, OS, GPU firmware), but the VFs cannot be enabled due to MMIO resource exhaustion on this board.

Thank you and best regards,

David

Will let you know if they manage to help, however suspect I will need to get a new motherboard.

2 Likes

also, will chuck my other b60 in my desktop which is an msi mag x670e tomahawk and see if it works there.

edit: same issue there

Linux ocp 6.17.0-14-generic #14-Ubuntu SMP PREEMPT_DYNAMIC Fri Jan  9 17:01:16 UTC 2026 x86_64 GNU/Linux

[  102.480336] xe 0000:14:00.0: [drm] GT0: PF: VF1 provisioned with 4180680704 (3.89 GiB) GGTT
[  102.481364] xe 0000:14:00.0: [drm] GT0: PF: VF1 provisioned with 25509756928 (23.8 GiB) LMEM
[  102.481483] xe 0000:14:00.0: [drm] GT0: PF: VF1 provisioned with 65278 GuC context IDs
[  102.481556] xe 0000:14:00.0: [drm] GT0: PF: VF1 provisioned with 256 GuC doorbell IDs
[  102.481758] xe 0000:14:00.0: [drm] GT1: PF: VF1 provisioned with 65279 GuC context IDs
[  102.482183] xe 0000:14:00.0: [drm] GT1: PF: VF1 provisioned with 256 GuC doorbell IDs
[  102.482193] xe 0000:14:00.0: not enough MMIO resources for SR-IOV
[  102.649780] xe 0000:14:00.0: [drm] PF: Failed to enable 1 VF (-ENOMEM)

i guess it relies on the total vfs for allocation even if i’m only trying to start one, so the huge VF BAR2 is a hindrance. would need nearly 400GB PCIE MMIO if this is correct. if i set xe.max_vfs, the total vf count still seems to remain at 7.

I have a different problem, but that part works for me with Asus TUF GAMING Z790-PLUS D4 + ASROCK Arc Pro B60 Creator

dmesg | grep "0000:09"
[    0.404251] pci 0000:09:00.0: [8086:e211] type 00 class 0x030000 PCIe Endpoint
[    0.404310] pci 0000:09:00.0: BAR 0 [mem 0x5607000000-0x5607ffffff 64bit pref]
[    0.404316] pci 0000:09:00.0: BAR 2 [mem 0x4000000000-0x47ffffffff 64bit pref]
[    0.404321] pci 0000:09:00.0: ROM [mem 0x88000000-0x881fffff pref]
[    0.404451] pci 0000:09:00.0: PME# supported from D0 D3hot
[    0.404520] pci 0000:09:00.0: VF BAR 0 [mem 0x5600000000-0x5600ffffff 64bit pref]
[    0.404521] pci 0000:09:00.0: VF BAR 0 [mem 0x5600000000-0x5606ffffff 64bit pref]: contains BAR 0 for 7 VFs
[    0.404524] pci 0000:09:00.0: VF BAR 2 [mem 0x4800000000-0x49ffffffff 64bit pref]
[    0.404525] pci 0000:09:00.0: VF BAR 2 [mem 0x4800000000-0x55ffffffff 64bit pref]: contains BAR 2 for 7 VFs
[    0.434905] pci 0000:09:00.0: vgaarb: bridge control possible
[    0.434905] pci 0000:09:00.0: vgaarb: VGA device added: decodes=io+mem,owns=none,locks=none
[    0.458997] pci_bus 0000:09: resource 1 [mem 0x88000000-0x881fffff]
[    0.458998] pci_bus 0000:09: resource 2 [mem 0x4000000000-0x5607ffffff 64bit pref]
[    0.460001] pci 0000:09:00.0: Adding to iommu group 24
[    6.536393] xe 0000:09:00.0: [drm] Running in SR-IOV PF mode
[    6.536476] xe 0000:09:00.0: [drm] Found battlemage (device ID e211) discrete display version 14.01 stepping B0
[    6.537795] xe 0000:09:00.0: [drm] VISIBLE VRAM: 0x0000004000000000, 0x0000000800000000
[    6.537827] xe 0000:09:00.0: [drm] VRAM[0]: Actual physical size 0x0000000600000000, usable size exclude stolen 0x00000005fa000000, CPU accessible size 0x00000005fa000000
[    6.537829] xe 0000:09:00.0: [drm] VRAM[0]: DPA range: [0x0000000000000000-600000000], io range: [0x0000004000000000-45fa000000]
[    6.537830] xe 0000:09:00.0: [drm] VRAM[0]: Actual physical size 0x0000000600000000, usable size exclude stolen 0x00000005fa000000, CPU accessible size 0x00000005fa000000
[    6.537831] xe 0000:09:00.0: [drm] VRAM[0]: DPA range: [0x0000000000000000-600000000], io range: [0x0000004000000000-45fa000000]
[    6.638253] xe 0000:09:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=none:owns=none
[    6.649494] xe 0000:09:00.0: [drm] Finished loading DMC firmware i915/bmg_dmc.bin (v2.6)
[    6.651287] xe 0000:09:00.0: [drm] GT0: Using GuC firmware from xe/bmg_guc_70.bin version 70.45.2
[    6.766257] xe 0000:09:00.0: [drm] GT0: ccs2 fused off
[    6.766260] xe 0000:09:00.0: [drm] GT0: ccs3 fused off
[    6.789648] xe 0000:09:00.0: [drm] GT1: Using GuC firmware from xe/bmg_guc_70.bin version 70.45.2
[    6.792046] xe 0000:09:00.0: [drm] GT1: Using HuC firmware from xe/bmg_huc.bin version 8.2.10
[    6.803617] xe 0000:09:00.0: [drm] GT1: vcs1 fused off
[    6.803618] xe 0000:09:00.0: [drm] GT1: vcs3 fused off
[    6.803619] xe 0000:09:00.0: [drm] GT1: vcs4 fused off
[    6.803619] xe 0000:09:00.0: [drm] GT1: vcs5 fused off
[    6.803619] xe 0000:09:00.0: [drm] GT1: vcs6 fused off
[    6.803620] xe 0000:09:00.0: [drm] GT1: vcs7 fused off
[    6.803620] xe 0000:09:00.0: [drm] GT1: vecs2 fused off
[    6.803620] xe 0000:09:00.0: [drm] GT1: vecs3 fused off
[    6.834405] xe 0000:09:00.0: [drm] Registered 4 planes with drm panic
[    6.834407] [drm] Initialized xe 1.1.0 for 0000:09:00.0 on minor 1
[    6.914305] xe 0000:09:00.0: [drm] Cannot find any crtc or sizes
[    6.915211] xe 0000:09:00.0: [drm] Using mailbox commands for power limits
[    6.915725] xe 0000:09:00.0: [drm] PL2 is supported on channel 0
[    6.954312] xe 0000:09:00.0: [drm] Cannot find any crtc or sizes
[    6.954315] snd_hda_intel 0000:0a:00.0: bound 0000:09:00.0 (ops intel_audio_component_bind_ops [xe])
[    8.306192] xe 0000:09:00.0: [drm] GT0: PF: VF1..VF2 provisioned with 2090340352 (1.95 GiB) GGTT
[    8.307054] xe 0000:09:00.0: [drm] GT0: PF: VF1..VF2 provisioned with 12754878464 (11.9 GiB) LMEM
[    8.307226] xe 0000:09:00.0: [drm] GT0: PF: VF1..VF2 provisioned with 32639 GuC context IDs
[    8.307422] xe 0000:09:00.0: [drm] GT0: PF: VF1..VF2 provisioned with 128 GuC doorbell IDs
[    8.307892] xe 0000:09:00.0: [drm] GT1: PF: VF1..VF2 provisioned with 32639 GuC context IDs
[    8.308613] xe 0000:09:00.0: [drm] GT1: PF: VF1..VF2 provisioned with 128 GuC doorbell IDs
[    8.413348] pci 0000:09:00.1: [8086:e211] type 00 class 0x030000 PCIe Endpoint
[    8.413603] pci 0000:09:00.1: Adding to iommu group 27
[    8.413618] pci 0000:09:00.1: vgaarb: no bridge control possible
[    8.413619] pci 0000:09:00.1: vgaarb: VGA device added: decodes=io+mem,owns=none,locks=none
[    8.413620] xe 0000:09:00.0: vgaarb: VGA decodes changed: olddecodes=none,decodes=none:owns=io
[    8.413688] xe 0000:09:00.1: enabling device (0000 -> 0002)
[    8.414071] xe 0000:09:00.1: [drm] Running in SR-IOV VF mode
[    8.415202] xe 0000:09:00.1: [drm] VF: migration not supported by this module version
[    8.416919] xe 0000:09:00.1: [drm] VISIBLE VRAM: 0x0000004800000000, 0x0000000400000000
[    8.416921] xe 0000:09:00.1: [drm] VRAM[0]: Actual physical size 0x00000002f8400000, usable size exclude stolen 0x00000002f8400000, CPU accessible size 0x00000002f8400000
[    8.416923] xe 0000:09:00.1: [drm] VRAM[0]: DPA range: [0x0000000000000000-2f8400000], io range: [0x0000004800000000-4af8400000]
[    8.416925] xe 0000:09:00.1: [drm] VRAM[0]: Actual physical size 0x00000002f8400000, usable size exclude stolen 0x00000002f8400000, CPU accessible size 0x00000002f8400000
[    8.416926] xe 0000:09:00.1: [drm] VRAM[0]: DPA range: [0x0000000000000000-2f8400000], io range: [0x0000004800000000-4af8400000]
[    8.501662] xe 0000:09:00.1: [drm] GT0: ccs2 fused off
[    8.501665] xe 0000:09:00.1: [drm] GT0: ccs3 fused off
[    8.509685] xe 0000:09:00.1: [drm] GT1: vcs1 fused off
[    8.509686] xe 0000:09:00.1: [drm] GT1: vcs3 fused off
[    8.509687] xe 0000:09:00.1: [drm] GT1: vcs4 fused off
[    8.509687] xe 0000:09:00.1: [drm] GT1: vcs5 fused off
[    8.509688] xe 0000:09:00.1: [drm] GT1: vcs6 fused off
[    8.509688] xe 0000:09:00.1: [drm] GT1: vcs7 fused off
[    8.509688] xe 0000:09:00.1: [drm] GT1: vecs2 fused off
[    8.509689] xe 0000:09:00.1: [drm] GT1: vecs3 fused off
[    8.515025] [drm] Initialized xe 1.1.0 for 0000:09:00.1 on minor 0
[    8.515197] pci 0000:09:00.2: [8086:e211] type 00 class 0x030000 PCIe Endpoint
[    8.515422] pci 0000:09:00.2: Adding to iommu group 28
[    8.515432] pci 0000:09:00.2: vgaarb: no bridge control possible
[    8.515433] pci 0000:09:00.2: vgaarb: VGA device added: decodes=io+mem,owns=none,locks=none
[    8.515434] xe 0000:09:00.0: vgaarb: VGA decodes changed: olddecodes=none,decodes=none:owns=io
[    8.515484] xe 0000:09:00.2: enabling device (0000 -> 0002)
[    8.516471] xe 0000:09:00.2: [drm] Running in SR-IOV VF mode
[    8.516868] xe 0000:09:00.2: [drm] VF: migration not supported by this module version
[    8.517717] xe 0000:09:00.2: [drm] VISIBLE VRAM: 0x0000004c00000000, 0x0000000400000000
[    8.517720] xe 0000:09:00.2: [drm] VRAM[0]: Actual physical size 0x00000002f8400000, usable size exclude stolen 0x00000002f8400000, CPU accessible size 0x00000002f8400000
[    8.517721] xe 0000:09:00.2: [drm] VRAM[0]: DPA range: [0x0000000000000000-2f8400000], io range: [0x0000004c00000000-4ef8400000]
[    8.517723] xe 0000:09:00.2: [drm] VRAM[0]: Actual physical size 0x00000002f8400000, usable size exclude stolen 0x00000002f8400000, CPU accessible size 0x00000002f8400000
[    8.517724] xe 0000:09:00.2: [drm] VRAM[0]: DPA range: [0x0000000000000000-2f8400000], io range: [0x0000004c00000000-4ef8400000]
[    8.602345] xe 0000:09:00.2: [drm] GT0: ccs2 fused off
[    8.602358] xe 0000:09:00.2: [drm] GT0: ccs3 fused off
[    8.608878] xe 0000:09:00.2: [drm] GT1: vcs1 fused off
[    8.608880] xe 0000:09:00.2: [drm] GT1: vcs3 fused off
[    8.608881] xe 0000:09:00.2: [drm] GT1: vcs4 fused off
[    8.608882] xe 0000:09:00.2: [drm] GT1: vcs5 fused off
[    8.608882] xe 0000:09:00.2: [drm] GT1: vcs6 fused off
[    8.608883] xe 0000:09:00.2: [drm] GT1: vcs7 fused off
[    8.608883] xe 0000:09:00.2: [drm] GT1: vecs2 fused off
[    8.608884] xe 0000:09:00.2: [drm] GT1: vecs3 fused off
[    8.614514] [drm] Initialized xe 1.1.0 for 0000:09:00.2 on minor 7
[    8.614706] xe 0000:09:00.0: [drm] PF: Enabled 2 of 7 VFs
 sudo lspci -vvv -s 09:00
09:00.0 VGA compatible controller: Intel Corporation Battlemage G21 [Intel Graphics] (prog-if 00 [VGA controller])
        Subsystem: ASRock Incorporation Device 6023
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin ? routed to IRQ 240
        IOMMU group: 24
        Region 0: Memory at 5607000000 (64-bit, prefetchable) [size=16M]
        Region 2: Memory at 4000000000 (64-bit, prefetchable) [size=32G]
        Expansion ROM at 88000000 [disabled] [size=2M]
        Capabilities: [40] Vendor Specific Information: Intel Capabilities v1
                CapA: Peg60Dis- Peg12Dis- Peg11Dis- Peg10Dis- PeLWUDis- DmiWidth=x4
                      EccDis- ForceEccEn- VTdDis- DmiG2Dis- PegG2Dis- DDRMaxSize=Unlimited
                      1NDis- CDDis- DDPCDis- X2APICEn- PDCDis- IGDis- CDID=0 CRID=0
                      DDROCCAP+ OCEn- DDRWrtVrefEn+ DDR3LEn+
                CapB: ImguDis- OCbySSKUCap- OCbySSKUEn- SMTCap- CacheSzCap 0x0
                      SoftBinCap- DDR3MaxFreqWithRef100=Disabled PegG3Dis-
                      PkgTyp- AddGfxEn- AddGfxCap- PegX16Dis- DmiG3Dis- GmmDis-
                      DDR3MaxFreq=2932MHz LPDDR3En-
        Capabilities: [70] Express (v2) Endpoint, IntMsgNum 0
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W TEE-IO-
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, LnkDisable- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- FltModeDis-
                LnkSta: Speed 2.5GT/s, Width x1
                        TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range B, TimeoutDis+ NROPrPrP- LTR+
                         10BitTagComp+ 10BitTagReq+ OBFF Not Supported, ExtFmt+ EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
                         AtomicOpsCtl: ReqEn-
                         IDOReq- IDOCompl- LTR+ EmergencyPowerReductionReq-
                         10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
                LnkCap2: Supported Link Speeds: 2.5GT/s, Crosslink- Retimer- 2Retimers- DRS-
                LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported, FltMode-
        Capabilities: [ac] MSI: Enable+ Count=1/1 Maskable+ 64bit+
                Address: 00000000fee011b8  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [d0] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
                Status: D3 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI)
                ARICap: MFVC- ACS-, Next Function: 0
                ARICtl: MFVC- ACS-, Function Group: 0
        Capabilities: [110 v1] Null
        Capabilities: [200 v1] Address Translation Service (ATS)
                ATSCap: Invalidate Queue Depth: 00
                ATSCtl: Enable-, Smallest Translation Unit: 00
        Capabilities: [420 v1] Physical Resizable BAR
                BAR 2: current size: 32GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB
        Capabilities: [220 v1] Virtual Resizable BAR
                BAR 2: current size: 16GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB
        Capabilities: [320 v1] Single Root I/O Virtualization (SR-IOV)
                IOVCap: Migration- 10BitTagReq+ IntMsgNum 0
                IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+ 10BitTagReq-
                IOVSta: Migration-
                Initial VFs: 7, Total VFs: 7, Number of VFs: 0, Function Dependency Link: 00
                VF offset: 1, stride: 1, Device ID: e211
                Supported Page Size: 00000553, System Page Size: 00000001
                Region 0: Memory at 0000005600000000 (64-bit, prefetchable)
                Region 2: Memory at 0000004800000000 (64-bit, prefetchable)
                VF Migration: offset: 00000000, BIR: 0
        Capabilities: [400 v1] Latency Tolerance Reporting
                Max snoop latency: 3145728ns
                Max no snoop latency: 3145728ns
        Kernel driver in use: xe
        Kernel modules: xe
uname -r
6.17.0-14-generic
1 Like

Thank you all, it’s good to know that I’m not the only one, at least that tells me that the issue is not caused by me damaging the board somehow :sweat_smile:

It will be great if ASRock can come up with some solution. I don’t have much hope that it will happen, but fortune might smile upon us.

It’s also very helpful to know that Asus Z790 board works, I wonder is there is something in spec that we can use as a reference point to say ā€œyup that one will probably workā€, I’m not aware of anything like that though, so for now my plan is to buy Asus PRIME Z790-P ( somewhere next month probably ) and see how it goes.

3 Likes

im tempted to try rebaruefi to increase the mmio address space but my partner will get very upset if media services go down for an extended period.

have already patched the firmware, but haven’t seen any reports of using it on a z890/arrow lake. too chicken to flash :smiley:

1 Like

I am considering that as well ( saw it mentioned in other thread ) but I never did it before so the time investment will be considerable and I’m not sure if it will help, more so considering the fact that the motherboard supports ReBar natively.

If it’s possible to only add the patches described in section UEFI Patching then the risk might be decreased because the native implementation of ReBar is intact ? but then will those patches have effect ?

Regardless of risk if that can fix the issue then I’m willing to try :grin:

i don’t think any of those patches apply to systems this modern so it may not even be worth the attempt.

i added the FFS module, and the patches said ā€œnothing to doā€ when attempted. i just need to flash then run ReBarState which may not even have any impact, assuming it boots.

I’m afraid that it might not have any effect, might try it regardless if I have some time, now that I read the guide it does not seem terribly complicated ( famous last words :laughing: )

When it comes to any kind of efi shimming, here be dragons. Definitley not something you should try without a revert plan or being ready for high downtime to make it work.

1 Like

absolutely. my current thinking is move a card to a testbed system i’m not using of similar recency and trying there (if i even have the same problem on that system, it’s an asus).

or buying a new motherboard, and just trying directly before replacing.

however i strongly suspect it will have no impact towards remediating the issue.

1 Like

just tried it on Asus Prime B560M-A motherboard, worked immediately.

	Capabilities: [320 v1] Single Root I/O Virtualization (SR-IOV)
		IOVCap:	Migration- 10BitTagReq+ Interrupt Message Number: 000
		IOVCtl:	Enable+ Migration- Interrupt- MSE+ ARIHierarchy+ 10BitTagReq-
		IOVSta:	Migration-
		Initial VFs: 7, Total VFs: 7, Number of VFs: 7, Function Dependency Link: 00
		VF offset: 1, stride: 1, Device ID: e211
		Supported Page Size: 00000553, System Page Size: 00000001
		Region 0: Memory at 0000007600000000 (64-bit, prefetchable)
		Region 2: Memory at 0000006800000000 (64-bit, prefetchable)
		VF Migration: offset: 00000000, BIR: 0
[  487.821871] xe 0000:03:00.0: [drm] PF: Enabled 7 of 7 VFs

asrock instant flash wouldn’t let me do rebaruefi, very rude.

think i’m going to buy as asus z890 prime, but will wait to see what asrock support says. nil response as yet.

It’s quite surprising that instant flash will stop user from flashing modded bios, combined with fact that we now have two confirmed cases of Asus providing working SR-IOV while ASRock fails to do so means that I will not buy ASRock again, or recommend it for anything else than basic PC usage.

Maybe I’m too harsh here but if Asus can do SR-IOV then ASRock should too, they even have SR-IOV Support toggle in bios.

Should have bought Asus to begin with, well at least now I have a spare board to do something with :sweat_smile: ( well I will have it when I buy the Asus one, budget for unexpected expenses for this month has reached it’s limit )

1 Like

yeah the asrock ā€œsecure flashā€ component of instant flash cracks the shits. i guess they have some kind of firmware integrity check.

ASRock responded:

Dear David,

Thank you for contacting us.

Regarding your inquiry, please refer to the following response.

1. Confirm whether the current Z890 Pro-A WiFi BIOS reserves sufficient 64‑bit prefetchable MMIO space to support SR-IOV for large‑BAR GPUs like the Intel Arc Pro B60 (PF + up to 7 VFs)

Generally, under some configuration, Z890 Pro-A WiFi should have enough resources for Intel Arc Pro B60 Passive.

However, other devices such as HDDs and PCIe cards also share these resources.
We recommend trying the following steps to verify.

Disable the ā€œIntegrated Thunderbolt Enableā€ option in the BIOS to free up resources for testing.
(BIOS > Advanced > Intel Thunderbolt > Integrated Thunderbolt Enable)

If the issue persists, please try removing PCIe devices, such as the LSI SAS3416 HBA card, to further isolate the problem.

2. Provide a BIOS update (or beta BIOS) that increases the MMIO aperture / PCIe resource windows so that SR-IOV VF BAR2 regions for the Arc Pro B60 can be successfully mapped?

The ā€˜MMIO Resource Increase’ feature is exclusively supported by Intel workstation-class CPUs.

Consequently, the Z890 Pro-A WiFi does not support this feature due to platform specification.

With appreciation,

ASRock TSD

I tried turning off thunderbolt enable, no change. I’m not going to pull out the nic and sas hba, but instead added a similar sas hba and nic to the B560M-A testbed machine i have a b60 in. sr-iov continued to enable fine.

i have ordered an asus z890 board, will likely do the swapout this weekend and report back. it may be that the sas hba and nic are a limiting factor in either board, but i will figure that out when i do the swap and have a z890 board leftover to experiment with.

Well as expected no solution provided and none will be provided :sweat_smile:

I tried disabling the Integrated Thunderbolt Enable on my board too when I was tinkering with it and it had no effect as well, so it’s safe to assume it’s not a fix.
I’m also not willing to remove PCIe cards and so on from the board, therefore Asus is the way then.

That BAR assignment failure definitely looks like a BIOS/MMIO resource bottleneck, which is such a grind when you just want a clean Proxmox setup. Hopefully, a Z790 board or a BIOS update clears up that ā€œno spaceā€ error, so your Battlemage can finally run as hot as a https://cs2.ad/items/mac-10-heat.

1 Like

asus z890 board in, all the same hardware as before, 7 vfs created no issues. i guess asrock just limit mmio address space for no real good reason.

2 Likes

Well my Asus board refused to boot with all 4 dims populated, 3 was the max I managed to get working, so I will return it ( known good CPU/RAM so on ) and try my luck with some other board, right now I reinstalled the ASRock board, works just fine ( except for the SR-IOV of course )