Ascrock Creator Arc Pro B60 and SR-IOV

Hello!

this day I was experimenting with an Asrock Arc Pro B60 and I wanted to try SR-IOV GPU virtualization in proxmox (9.2.10)…

newly installed machine, nothing special. Of course sr-iov enabled in BIOS and above 4g decoding. disabled CSM. In the first place, it everything seemed as expected, I could enable 4 virtual functions and create a mapping in Proxmox, assiged the mapping to the vm an booted up. Unfornately I ended up with error 43 in the VM. After a bit trial and error I decided I would try with the whole GPU in the vm first. So I assigend the GPU to the VM and bootet, my virtual functions disappeared, but the card worked correctly in Windows guest.

I tried to enable SR IOV again, whoch failed. So i rebooted the host and afterwards tried again, but this time I got a differen error:

[ 288.451761] xe 0000:04:00.0: [drm] PF: Tile0: GT0: VF1..VF4 provisioned with 536870912 (512 MiB) GGTT
[ 288.453980] xe 0000:04:00.0: [drm] PF: Tile0: GT0: VF1..VF4 provisioned with 6371147776 (5.93 GiB) LMEM
[ 288.454215] xe 0000:04:00.0: [drm] PF: Tile0: GT0: VF1..VF4 provisioned with 8192 GuC context IDs
[ 288.454439] xe 0000:04:00.0: [drm] PF: Tile0: GT0: VF1..VF4 provisioned with 32 GuC doorbell IDs
[ 288.454774] xe 0000:04:00.0: [drm] PF: Tile0: GT1: VF1..VF4 provisioned with 8192 GuC context IDs
[ 288.454972] xe 0000:04:00.0: [drm] PF: Tile0: GT1: VF1..VF4 provisioned with 32 GuC doorbell IDs
[ 288.454979] xe 0000:04:00.0: not enough MMIO resources for SR-IOV
[ 288.509740] xe 0000:04:00.0: [drm] PF: Failed to enable 4 VFs (-ENOMEM)

I cannot enable sr-iov anymore at all, although I am pretty sure I did not persist any settings.

Furthermore the card now seems to report “only“ 7 virtual functions while I am pretty sure! it where more in the first place (12 or 24?)… so loading the intel driver suite in windows seems to have changed sth about the cards behaviour? Is this possible?

And finally, I seems I am not even able anymore to assign the whole card to the vm. I mean, I can assign it but it renders the VM defunct. It does start without error but never seems to boot up, just allocating all ram and never finishind to start up.

in the host log I see

[ 492.255988] vfio-pci 0000:04:00.0: resetting
[ 492.363512] vfio-pci 0000:04:00.0: reset done
[ 492.406398] vfio-pci 0000:04:00.0: resetting
[ 492.507438] vfio-pci 0000:04:00.0: reset done

when I boot the vm and that is it. I tried disabling sr-iov in the BIOS which did not help. I cannot use the card anymore for virtualization as it seems. How is that even possbile? Does the intel driver “silently“ update firmware or change any settings of the card?

fyi: this is a supermicro x10-sri-f (yes it is an old board, but in general it should support the relevant functions).

You probably need resize bar. Is it on? Do you have the option? If not you may need a hacked bios to enable it

Hello Wendell and thank you for your reply. I have not yet have the opportunity to enable Rebar on this exact board, but I tried with the card and a similar setup with a similar board from the same generation. I got the exact same problem “cannot assign memory“, when trying to enable sr-iov. It really seems that loading the intel driver suite did change sth about the cards behaviour. I did enable Rebar (after modifying and flashing bios) and verified it is on in Windows with gpu-z. Unfortunately it does not change anything. So it currently it seems I am out of luck and I’ll have to try fo find a more modern board for my test to continue…

interestingly I AM able to pass through an AMD GPU to the exakt same vm on the exact same board without problems, but for the Intel GPU (which DID WORK at a point) it seems I am out of luck now…