Intel 10Gb NIC SR-IOV not working on AM4 X370 AGESA Combo-AM4 1.0.0.1

Operating Systems & Open Source Linux
1

Im trying to activate SR-IOV on a Intel X540 T2 NIC on Fedora 32 and get the following dmesg:

[46381.219112] ixgbe 0000:0c:00.1 enp12s0f1: SR-IOV enabled with 1 VFs
[46381.219238] ixgbe 0000:0c:00.1: removed PHC on enp12s0f1
[46381.546374] ixgbe 0000:0c:00.1: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4 XDP Queue count = 0
[46381.566596] pps pps2: new PPS source ptp2
[46381.566623] ixgbe 0000:0c:00.1: registered PHC device on enp12s0f1
[46381.795282] pci 0000:0c:10.1: [8086:1515] type 7f class 0xffffff
[46381.795803] pci 0000:0c:10.1: unknown header type 7f, ignoring device
[46382.827254] ixgbe 0000:0c:00.1: Failed to enable PCI sriov: -5
[46384.838218] ixgbe 0000:0c:00.1 enp12s0f1: NIC Link is Up 1 Gbps, Flow Control: RX/TX

I try to activate it via:

[root@server ~]# echo 1 > /sys/class/net/enp12s0f1/device/sriov_numvfs 
-bash: echo: write error: Input/output error

In Bios IOMMU and SR-IOV are activated.

My grub defaults:

[root@server ~]# cat /etc/default/grub 
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="rd.lvm.lv=fedora/root rd.lvm.lv=fedora/swap rhgb quiet selinux=0 amd_iommu=on iommu=pt pci_pt_e820_access=on pci=assign-busses "
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true

My module settings:

[root@server ~]# cat /etc/modprobe.d/sriov.conf 
options ixgbe max_vfs=1

system Infos:

[root@server ~]# neofetch 
          /:-------------:\          root@server 
       :-------------------::        ----------- 
     :-----------/shhOHbmp---:\      OS: Fedora 32 (Server Edition) x86_64 
   /-----------omMMMNNNMMD  ---:     Kernel: 5.6.15-300.agesa.fc32.x86_64 
  :-----------sMMMMNMNMP.    ---:    Uptime: 12 hours, 57 mins 
 :-----------:MMMdP-------    ---\   Packages: 1363 (rpm) 
,------------:MMMd--------    ---:   Shell: bash 5.0.11 
:------------:MMMd-------    .---:   Terminal: /dev/pts/0 
:----    oNMMMMMMMMMNho     .----:   CPU: AMD Ryzen 7 1700 (16) @ 3.000GHz 
:--     .+shhhMMMmhhy++   .------/   GPU: NVIDIA GeForce GT 710 
:-    -------:MMMd--------------:    Memory: 443MiB / 64243MiB 
:-   --------/MMMd-------------;
:-    ------/hMMMy------------:                              
:-- :dMNdhhdNMMNo------------;                               
:---:sdNMMMMNds:------------:
:------:://:-------------::
:---------------------://

NIC:

0c:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)
0c:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)

I’ve also applied the AGESA Patch to my kernel to sort out issues on this side.

Does anybody experienced that error code (-5) yet?

Tnx, Stefan

2

Hmm the only time I saw that error was when I tried using the ACS patch kernel.

Here’s my efforts with the SFP+ version of that nic, the x520. Maybe something helpful in there?

3

Thanks for the quick answer.
I got an Asrock Board and there are options for ACS, IOMMU and SR-IOV. I enabled them all.
I tried the pure Fedora 32 kernel and the AGESA patched version - so ACS patch cant be the cause.
Both with same result.

4

After some permutation testing in BIOS i was able to sort it out.
The setting in question was:

Advanced → AMD CBS → NBIO Common Options → PCIe ARI Support → Enabled

It needs to be enabled “Auto” isn’t enough there.
A patched kernel is not needed.
Hope it helps somebody running into the same problem (with an ASRock board).

4 Likes
5

Another case where “auto” makes no sense … why do the devs even include it, rather than just the basic on/off or yes/no options.

Glad you found the solution!

6

This is a shameless necro bump, because I wanted to say, thank you @derstef ! This was the last in a series of problems with my Intel X550-T2 that needed solving before it just now started to work!

1 Like
7

Legit just made a L1T account to say thank you. This was the solution for me as well.