Arc Pro B60 SR-IOV or how my new homelab is driving me insane.

Hardware Hub GPU
, ,
1

Hello Boy and Girls

I am at my wits end suffering through the early adopters tax. Hoping someone here can help me solve my issues or at least tell me where to look.

First things first. Specs:

Ryzen 5700X

Gigabyte X570 UD v1.1

2X32GB memory

1 TB NVME

Asrock Arc Pro B60 Creator 24GB

Okay now that the specs our out of the way a few more things. I have a clean proxmox 9.1 with the 6.17.13-2-pve kernel and the B60 firmware has been updated to support SR-IOV. Assume Rebar/4g encoding/csm disabled/IOMMU.

When trying to split the GPU I get.

root@pve:~# echo 2 > /sys/devices/pci0000:00/0000:00:03.1/0000:08:00.0/0000:09:01.0/0000:0a:00.0/sriov_numvfs
-bash: echo: write error: Cannot allocate memory
root@pve:~#

I have tried a ton of things but at this point I am starting clean and trying every suggestion individually.

Just to speed things up a bit here is my LSPCI.

0a:00.0 VGA compatible controller: Intel Corporation Battlemage G21 [Intel Graphics] (prog-if 00 [VGA controller])
Subsystem: ASRock Incorporation Device 6023
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin ? routed to IRQ 83
IOMMU group: 24
Region 0: Memory at 1400000000 (64-bit, prefetchable) [size=16M]
Region 2: Memory at 1800000000 (64-bit, prefetchable) [size=32G]
Expansion ROM at fc200000 [disabled] [size=2M]
Capabilities: [40] Vendor Specific Information: Len=0c <?>
Capabilities: [70] Express (v2) Endpoint, IntMsgNum 0
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W TEE-IO-
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1
TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range B, TimeoutDis+ NROPrPrP- LTR+
10BitTagComp+ 10BitTagReq+ OBFF Not Supported, ExtFmt+ EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS- TPHComp- ExtTPHComp-
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
AtomicOpsCtl: ReqEn-
IDOReq- IDOCompl- LTR+ EmergencyPowerReductionReq-
10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
LnkCap2: Supported Link Speeds: 2.5GT/s, Crosslink- Retimer- 2Retimers- DRS-
LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [ac] MSI: Enable+ Count=1/1 Maskable+ 64bit+
Address: 00000000fee00000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [d0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [110 v1] Null
Capabilities: [200 v1] Address Translation Service (ATS)
ATSCap: Invalidate Queue Depth: 00
ATSCtl: Enable+, Smallest Translation Unit: 00
Capabilities: [420 v1] Physical Resizable BAR
BAR 2: current size: 32GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB
Capabilities: [220 v1] Virtual Resizable BAR
BAR 2: current size: 8GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB
Capabilities: [320 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration- 10BitTagReq+ IntMsgNum 0
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+ 10BitTagReq-
IOVSta: Migration-
Initial VFs: 7, Total VFs: 7, Number of VFs: 0, Function Dependency Link: 00
VF offset: 1, stride: 1, Device ID: e211
Supported Page Size: 00000553, System Page Size: 00000001
Region 0: Memory at 0000001401000000 (64-bit, prefetchable)
Region 2: Memory at 0000000000000000 (64-bit, prefetchable)
VF Migration: offset: 00000000, BIR: 0
Capabilities: [400 v1] Latency Tolerance Reporting
Max snoop latency: 1048576ns
Max no snoop latency: 1048576ns
Kernel driver in use: xe
Kernel modules: xe

And here is my dmesg | grep xe

[ 0.000000] NX (Execute Disable) protection: active
[ 0.000238] MTRR map: 8 entries (4 fixed + 4 variable; max 21), built from 9 variable MTRRs
[ 0.742762] … fixed-purpose events: 0
[ 0.878497] pci 0000:0a:00.0: VF BAR 2 [mem size 0xe00000000 64bit pref]: can’t assign; no space
[ 0.878499] pci 0000:0a:00.0: VF BAR 2 [mem size 0xe00000000 64bit pref]: failed to assign
[ 0.878614] pci 0000:09:01.0: bridge window [mem 0x00000000-0x800ffffff 64bit pref] to [bus 0a] requires relaxed alignment rules
[ 0.878741] pci 0000:0a:00.0: VF BAR 2 [mem size 0xe00000000 64bit pref]: can’t assign; no space
[ 0.878743] pci 0000:0a:00.0: VF BAR 2 [mem size 0xe00000000 64bit pref]: failed to assign
[ 0.878879] pci 0000:09:01.0: bridge window [mem 0x400000000-0x1a07ffffff 64bit pref] to [bus 0a] requires relaxed alignment rules
[ 0.878885] pci 0000:08:00.0: bridge window [mem 0x00000000-0x800ffffff 64bit pref] to [bus 09-0b] requires relaxed alignment rules
[ 0.879029] pci 0000:0a:00.0: VF BAR 2 [mem size 0xe00000000 64bit pref]: can’t assign; no space
[ 0.879031] pci 0000:0a:00.0: VF BAR 2 [mem size 0xe00000000 64bit pref]: failed to assign
[ 0.879341] pci 0000:0a:00.0: VF BAR 2 [mem size 0xe00000000 64bit pref]: can’t assign; no space
[ 0.879343] pci 0000:0a:00.0: VF BAR 2 [mem size 0xe00000000 64bit pref]: failed to assign
[ 0.879397] pci 0000:0a:00.0: VF BAR 2 [mem size 0xe00000000 64bit pref]: can’t assign; no space
[ 0.879398] pci 0000:0a:00.0: VF BAR 2 [mem size 0xe00000000 64bit pref]: failed to assign
[ 4.898743] systemd[1]: Set up automount proc-sys-fs-binfmt_misc.automount - Arbitrary Executable File Formats File System Automount Point.
[ 5.306257] RAPL PMU: API unit is 2^-32 Joules, 2 fixed counters, 163840 ms ovfl timer
[ 5.634720] xe 0000:0a:00.0: vgaarb: deactivate vga console
[ 5.634918] xe 0000:0a:00.0: [drm] Running in SR-IOV PF mode
[ 5.635001] xe 0000:0a:00.0: [drm] Found battlemage (device ID e211) discrete display version 14.01 stepping B0
[ 5.636244] xe 0000:0a:00.0: [drm] VISIBLE VRAM: 0x0000001800000000, 0x0000000800000000
[ 5.636275] xe 0000:0a:00.0: [drm] VRAM[0]: Actual physical size 0x0000000600000000, usable size exclude stolen 0x00000005fa000000, CPU accessible size 0x00000005fa000000

5.636277\] xe 0000:0a:00.0: \[drm\] VRAM\[0\]: DPA range: \[0x0000000000000000-600000000\], io range: \[0x0000001800000000-1dfa000000

[ 5.636279] xe 0000:0a:00.0: [drm] VRAM[0]: Actual physical size 0x0000000600000000, usable size exclude stolen 0x00000005fa000000, CPU accessible size 0x00000005fa000000

5.636281\] xe 0000:0a:00.0: \[drm\] VRAM\[0\]: DPA range: \[0x0000000000000000-600000000\], io range: \[0x0000001800000000-1dfa000000

[ 5.767540] xe 0000:0a:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=none:owns=none
[ 5.778802] xe 0000:0a:00.0: [drm] Finished loading DMC firmware i915/bmg_dmc.bin (v2.6)
[ 5.785391] xe 0000:0a:00.0: [drm] GT0: Using GuC firmware from xe/bmg_guc_70.bin version 70.58.0
[ 6.078456] xe 0000:0a:00.0: [drm] GT0: ccs2 fused off
[ 6.078460] xe 0000:0a:00.0: [drm] GT0: ccs3 fused off
[ 6.103564] xe 0000:0a:00.0: [drm] GT1: Using GuC firmware from xe/bmg_guc_70.bin version 70.58.0
[ 6.106620] xe 0000:0a:00.0: [drm] GT1: Using HuC firmware from xe/bmg_huc.bin version 8.2.10
[ 6.118498] xe 0000:0a:00.0: [drm] GT1: vcs1 fused off
[ 6.118502] xe 0000:0a:00.0: [drm] GT1: vcs3 fused off
[ 6.118504] xe 0000:0a:00.0: [drm] GT1: vcs4 fused off
[ 6.118505] xe 0000:0a:00.0: [drm] GT1: vcs5 fused off
[ 6.118507] xe 0000:0a:00.0: [drm] GT1: vcs6 fused off
[ 6.118509] xe 0000:0a:00.0: [drm] GT1: vcs7 fused off
[ 6.118511] xe 0000:0a:00.0: [drm] GT1: vecs2 fused off
[ 6.118513] xe 0000:0a:00.0: [drm] GT1: vecs3 fused off
[ 6.167480] xe 0000:0a:00.0: [drm] Registered 4 planes with drm panic
[ 6.167483] [drm] Initialized xe 1.1.0 for 0000:0a:00.0 on minor 0
[ 7.670297] fbcon: xedrmfb (fb0) is primary device
[ 7.722282] xe 0000:0a:00.0: [drm] fb0: xedrmfb frame buffer device
[ 7.726943] xe 0000:0a:00.0: [drm] Using mailbox commands for power limits
[ 7.727386] xe 0000:0a:00.0: [drm] PL2 is supported on channel 0
[ 7.740266] Creating 4 MTD partitions on “xe.nvm.2560”:
[ 7.740269] 0x000000000000-0x000000001000 : “xe.nvm.2560.DESCRIPTOR”
[ 7.741713] 0x000000001000-0x00000054e000 : “xe.nvm.2560.GSC”
[ 7.743207] 0x00000054e000-0x00000074e000 : “xe.nvm.2560.OptionROM”
[ 7.744701] 0x00000074e000-0x00000075e000 : “xe.nvm.2560.DAM”
[ 10.703517] snd_hda_intel 0000:0b:00.0: bound 0000:0a:00.0 (ops intel_audio_component_bind_ops [xe])
[ 765.937954] xe 0000:0a:00.0: [drm] GT0: PF: VF1..VF2 provisioned with 2086236160 (1.94 GiB) GGTT
[ 765.939376] xe 0000:0a:00.0: [drm] GT0: PF: VF1..VF2 provisioned with 12750684160 (11.9 GiB) LMEM
[ 765.939525] xe 0000:0a:00.0: [drm] GT0: PF: VF1..VF2 provisioned with 32639 GuC context IDs
[ 765.939643] xe 0000:0a:00.0: [drm] GT0: PF: VF1..VF2 provisioned with 128 GuC doorbell IDs
[ 765.940254] xe 0000:0a:00.0: [drm] GT1: PF: VF1..VF2 provisioned with 32639 GuC context IDs
[ 765.940368] xe 0000:0a:00.0: [drm] GT1: PF: VF1..VF2 provisioned with 128 GuC doorbell IDs
[ 765.940378] xe 0000:0a:00.0: not enough MMIO resources for SR-IOV
[ 766.218640] xe 0000:0a:00.0: [drm] PF: Failed to enable 2 VFs (-ENOMEM)

Any assistance would be welcome and thanks in advance.

2

some of us have had similar problems: Intel Arc Pro B60 SR-IOV not working with ASRock B760 Pro RS

some boards just have less mmio resources defined in the firmware it seems.

1 Like
3

Thanks. I think I will order new board and test.

Intel Arc Pro B60 SR-IOV not working with ASRock B760 Pro RS
4

You might try your luck at https://esupport.gigabyte.com/ asking for an updated BIOS. I was getting that exact error message trying to split my B50 on my X570 Aorus Master. I found someone else on reddit with the exact same board that had received a BIOS from Gigabyte that got it to work. I tried my luck and they sent an even newer version to me.

1 Like
5

Ooh! Thanks. I ordered a quick B550 for testing. That should arrive tomorrow but if I can get the X570 working I would be so happy.

6

@Walter_Ego B550 board immediately worked out of the box. Thanks

2 Likes
7

@GelatinousSlime It WORKED! They sent me a custom bios and after some issues getting it running it now works flawlessly.

I had to follow that reddit threads solution to the error 99 though.

Still I am happy. Thank you very much guys.

1 Like
8

Ey lol that’s me.

I see there’s an ‘official’ update now on the gigabyte site. Have you tested it?