* Some of the technical info may be wrong as am not an expert which is why I try to include as much sources as I can.
Eariler in the week I posted a thread on reddit about IOMMU AVIC getting some fixes/improvements allowing for easier general usage.
After some discussion it would seem the performance difference I found was due to something else - My gut feeling after testing and reviewing documenation to the best of my understanding is that it’s due to SVM AVIC which still can provide speed up to interrupt performance.
Documentation am I referring to is AMD64 Architecture Programmer’s Manual Volume 2: System Programming. This line being the reason I think SVM AVIC is working.
AVIC Enable—Virtual Interrupt Control, Bit 31. The AVIC hardware support may be enabled on
a per virtual processor basis. This bit determines whether or not AVIC is enabled for a particular virtual
processor. Any guest configured to use AVIC must also enable RVI (nested paging). Enabling AVIC
implicitly disables the V_IRQ, V_INTR_PRIO, V_IGN_TPR, and V_INTR_VECTOR fields in the
VMCB Control Word.
For people that can understand things here is the AMD I/O Virtualization Technology (IOMMU) Specification also.
To enable AVIC keep the below in mind -
-
avic=1 npt=1
needs to be added as part of kvm_amd module options.options kvm-amd nested=0 avic=1 npt=1
. NPT is needed. -
If using with a Windows guest hyperv stimer + synic is incompatible. If you are worried about timer performance (don’t be
) just ensure you have hypervclock and invtsc exposed in your cpu features.
<cpu mode="host-passthrough" check="none"> <feature policy="require" name="invtsc"/> </cpu> <clock offset="utc"> <timer name="hypervclock" present="yes"/> </clock>
-
AVIC is deactivated when x2apic is enabled. This change is coming in Linux 5.7 so you will want to remove x2apic from your CPUID like so -
<cpu mode="host-passthrough" check="none"> <feature policy="disable" name="x2apic"/> </cpu>
-
AVIC does not work with nested virtualization
Either disabled nested via kvm_amd options or remove svm from your CPUID like so -<cpu mode="host-passthrough" check="none"> <feature policy="disable" name="svm"/> </cpu>
-
AVIC needs pit to be set as discard
<timer name='pit' tickpolicy='discard'/>
-
Some other hyper-v enlightenments can get in the way of AVIC working optimally. vapic helps provide paravirtualized EOI processing which is in conflict with what SVM AVIC provides.
In particular, this enlightenment allows paravirtualized
(exit-less) EOI processing.
hv-tlbflush/hv-ipi likely also would interfere but wasn’t tested as these are also things SVM AVIC helps to accelerate.
Nested related enlightenments wasn’t tested but don’t look like they should cause problems.
hv-reset/hv-vendor-id/hv-crash/hv-vpindex/hv-spinlocks/hv-relaxed
also look to be fine.
The patches to get things working (With some important fixes) were merged in Linux Kernel 5.6.
I made a patch for 5.5.13 tested applying against 5.5.13 arch/stable git/fedora sources (May work on older hasn’t older 5.5.x series but haven’t tested it. You will also want this patch if hasn’t being backported by Greg Kroah-Hartman) - https://pastebin.com/FmEc81zu
Patch was made using the merged changes from the kvm git tracking repo. Also included the GA Log tracepoint patch and these two fixes -
https://git.kernel.org/pub/scm/virt/kvm/kvm.git/commit/?h=for-linus&id=93fd9666c269877fffb74e14f52792d9c000c1f2
https://git.kernel.org/pub/scm/virt/kvm/kvm.git/commit/?h=for-linus&id=7943f4acea3caf0b6d5b6cdfce7d5a2b4a9aa608
Background info to this
AVIC (Advance Virtual Interrupt Controller) is AMD’s implementation of Advanced Programmable Interrupt Controller similar to Intel’s APICv. Main benefit for us causal/advanced users is it aims to improved interrupt performance. And unlike with Intel it’s not limited to only to HEDT/Server processors.
These patches added AVIC support some times ago
KVM: x86: Introduce SVM AVIC support
iommu/AMD: Introduce IOMMU AVIC support
However, until to now it hasn’t been easy to use as it had some limitations as best explained by Suravee Suthikulpanit from AMD who implemented the initial patch and follow ups.
kvm: x86: Support AMD SVM AVIC w/ in-kernel irqchip mode
The ‘commit 67034bb9dd5e (“KVM: SVM: Add irqchip_split() checks before enabling AVIC”)’ was introduced to fix miscellaneous boot-hang issues when enable AVIC. This is mainly due to AVIC hardware doest not #vmexit on write to LAPIC EOI register resulting in-kernel PIC and IOAPIC to wait and do not inject new interrupts (e.g. PIT, RTC). This limits AVIC to only work with kernel_irqchip=split mode, which is not currently enabled by default, and also required user-space to support split irqchip model, which might not be the case.
Please see the original reddit thread for full details in terms of things like performance differences measured in terms of latency in the original thread. If people want me however to copy the thread over here just let me know. Just remember it will make this post very long
@wendell @gnif - This might interest you
Hello @futurefade
Edit 1 -
After some further investigating it is likely IOMMU AVIC isn’t working as I though in my original post on reddit and as Aiberia on reddit suspected. The performance difference I saw was from SVM AVIC.
Below is details from my investigation -
Using perf kvm --host top -p `pidof qemu-system-x86_64`
here is what I found -
Linux -
0.12% [kvm_amd] [k] avic_vcpu_put.part.0
0.10% [kvm_amd] [k] avic_vcpu_load
0.02% [kvm_amd] [k] avic_incomplete_ipi_interception
0.01% [kvm_amd] [k] svm_deliver_avic_intr
2.83% [kernel] [k] iommu_completion_wait
0.87% [kernel] [k] __iommu_queue_command_sync
0.16% [kernel] [k] amd_iommu_update_ga
0.03% [kernel] [k] iommu_flush_irt
Windows -
0.61% [kvm_amd] [k] svm_deliver_avic_intr
0.05% [kvm_amd] [k] avic_vcpu_put.part.0
0.02% [kvm_amd] [k] avic_vcpu_load
0.14% [kvm] [k] kvm_emulate_wrmsr
amd_iommu_update_ga references to this function
svm_deliver_avic_intr references to this function.
Edit 2 -
Added some more info on requirements for AVIC
Edit 3 -
Update on WIndows AVIC IOMMU & vapic/enlightments.
Windows AVIC IOMMU is now working as of this patch but performance doesn’t appear to be completely stable atm.
Edit 4 -
Patch above has been merged in Linux 5.6.13/5.4.41. To continue to use SVM AVIC either revert the patch from edit 3 or don’t upgrade your kernel.
Another thing to note is with AVIC IOMMU there seems to be some problems with some PCIe devices causing the guest to not boot. In testing this was a Mellanox Connect X3 card and for Aiber from reddit it was his Samsung 970(Not sure on what model) personally my Samsung 970 Evo has worked so it appears to be YMMV kind of thing until we know the cause of the issues.
If you want more detail on testing and have discord see this post I made in the VFIO discord