Any kernel gearheads? irq affinity question?

tl;dr: I’m having network performance issues on my KVM guests, possibly related to SRIOV / vfio / irq handling?

Seeing a lot of these in the logs, well, ever since I started using SRIOV for networking in my guests

[3587751.438487] irq 150: Affinity broken due to vector space exhaustion.
[3639410.959080] irq 150: Affinity broken due to vector space exhaustion.
[3639410.960207] irq 150: Affinity broken due to vector space exhaustion.
[3639783.045026] irq 150: Affinity broken due to vector space exhaustion.
[3639783.045639] irq 149: Affinity broken due to vector space exhaustion.
[3639783.046873] irq 149: Affinity broken due to vector space exhaustion.
[3639783.047425] irq 150: Affinity broken due to vector space exhaustion.
[3639830.916299] irq 150: Affinity broken due to vector space exhaustion.
[3639830.917404] irq 150: Affinity broken due to vector space exhaustion.
[3639865.884657] irq 160: Affinity broken due to vector space exhaustion.
[3639865.885711] irq 160: Affinity broken due to vector space exhaustion.

I’m not seeing much by way of a fix, or even explanation as to what’s going on. I tried to make sense of it using this very old (from 2.4) doc, but at this hour, getting lost pretty easy.

https://cs.uwaterloo.ca/~brecht/servers/apic/SMP-affinity.txt

The guests seem to “bog down” trying to do much over 20 megabytes/sec despite having a 10 gigabit connection. Heavy network traffic results in those log messages on the host. The IRQs referenced are assigned to the VFIO driver, with specific pci-id’s pointing at my network card’s virtual functions

Any thoughts?

How many virtual functions did you make?

It could also be unrelated to performance issues, is virtio networking an option?

Does irqbalance do anything for you?

(Background CPUs have hardware interrupts where literally they have a pin on the cpu reserved for such purpose, then they have software interrupts in total worth hardware and internal ones there can be max of 255 of them, it’s a limitation of the x86 architecture. With PCIe and modern apic there’s something called msi-x ; that makes interrupt processing slower, but each device gets up to 2048 possible interrupts values for itself. That means in theory a network card could have e.g. 1000 different queues to deliver packets from different ip/ports to different cores each with it’s own interrupt line).

Are you using q35 as the platform for the VM?

2 Likes

7 VF, the card supports up to 63. Card is an Intel x520, using the stock Linux ixgbe driver.

Vitrio seems like a kludge compared to SR-IOV. I was happy to move away from using virtio and a host bridge.

Yes, q35 for the guests. Guests are pinned to specific CPU cores. I wonder if that’s part of the problem. I pinned CPUs based on which numa nodes own the GPUs, but the network card is on a different note if I recall correctly.

The host is a 2950x, with numa nodes enabled (?).

Hmm, there were a bunch of bugs around interrupt handling when q35-4.0 first came out, but that was a few months back and should be fixed by now. Maybe moving back and forth between different versions here helps.

Can you check if you have msi or msi-x in the guest lspci -vv output?

I don’t understand why it’s even bothering allocating so many legacy interrupts, it seems crazy - it shouldn’t be doing that at all.

I’m assuming both host and guest are running recent 5.4/5.5 mostly vanilla kernels? (for host / guest drivers)

Host is Fedora 31, qemu 4.1.1, kernel 5.4.14-200.fc31.x86_64

/proc/interrupts 08:10 and 08:11 are network (this is on the host)

 146:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI 4468737-edge      vfio-msix[1](0000:08:10.6)
 147:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI 4468738-edge      vfio-msix[2](0000:08:10.6)
 148:          0       6430          0          0          0     177092          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI 4472832-edge      vfio-msix[0](0000:08:11.0)
 149:          0          0          0          0          0          0          0          0          0          0          0          0          0      50969          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI 35127296-edge      vfio-msi[0](0000:43:00.0)
 150:          0          0          0          0          0          0       7472          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0     186730          0          0          0          0  IR-PCI-MSI 4472833-edge      vfio-msix[1](0000:08:11.0)
 151:          0      56317         65          0          0    8228215        122          0     274446          0          0    2306287          0          0          0      39029          0          0   10484193          0    6098138        401        137          0      45943          0          0      18940          0    4884697          0          0  IR-PCI-MSI 4460545-edge      vfio-msix[1](0000:08:10.2)
 152:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0         48          0          0          0          0          0          0          0          0          0  IR-PCI-MSI 4460546-edge      vfio-msix[2](0000:08:10.2)
 155:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI 35655680-edge      vfio-msi[0](0000:44:00.2)
 156:          0          0          0          0          0          0       4513          0     176208     351079          0          0      26485          0          0          0          0          0        135          0          0          0          0     152332          0          0          0          0          0          0          0          0  IR-PCI-MSI 4464640-edge      vfio-msix[0](0000:08:10.4)
 157:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI 4464641-edge      vfio-msix[1](0000:08:10.4)
 158:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI 4464642-edge      vfio-msix[2](0000:08:10.4)
 159:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI 35657728-edge      vfio-msi[0](0000:44:00.3)
 160:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0         84          0          4          0          0          0          0          0          0  IR-PCI-MSI 4472834-edge      vfio-msix[2](0000:08:11.0)

From a guest running Ubuntu 18.04, kernel 5.3.0-40-generic

01:00.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
        Subsystem: Intel Corporation 82599 Ethernet Controller Virtual Function
        Physical Slot: 0
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Region 0: Memory at c1c00000 (64-bit, non-prefetchable) [size=16K]
        Region 3: Memory at c1c04000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [70] MSI-X: Enable+ Count=3 Masked-
                Vector table: BAR=3 offset=00000000
                PBA: BAR=3 offset=00002000
        Capabilities: [a0] Express (v1) Endpoint, MSI 00
                DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset- SlotPowerLimit 0.000W
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 128 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed unknown, Width x0, ASPM not supported, Exit Latency L0s <64ns, L1 <1us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed unknown, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
                AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
        Kernel driver in use: ixgbevf
        Kernel modules: ixgbevf

Upgraded the host to kernel 5.5, hasn’t helped these errors.