CPU Pinning understanding

Hi
I have successfully created a KVM, and i’m trying to follow this guide on how to do cpu pinning.

In their example XML code i see that they are passing the vCPU, and then the next thing that is written a cpuset. I don’t quite understand what this is, and how it correlates in this way in their example.Could anyone please help me a bit here?
Here is my cpu layout, and I want to pass through everything but core 0. Which is then every vCPU except 0 and 16. But how should my xml look like then?

my: lscpu -e

CPU NODE SOCKET CORE L1d:L1i:L2:L3 ONLINE    MAXMHZ    MINMHZ
  0    0      0    0 0:0:0:0          yes 3500,0000 2200,0000
  1    0      0    1 1:1:1:0          yes 3500,0000 2200,0000
  2    0      0    2 2:2:2:0          yes 3500,0000 2200,0000
  3    0      0    3 3:3:3:0          yes 3500,0000 2200,0000
  4    0      0    4 4:4:4:1          yes 3500,0000 2200,0000
  5    0      0    5 5:5:5:1          yes 3500,0000 2200,0000
  6    0      0    6 6:6:6:1          yes 3500,0000 2200,0000
  7    0      0    7 7:7:7:1          yes 3500,0000 2200,0000
  8    0      0    8 8:8:8:2          yes 3500,0000 2200,0000
  9    0      0    9 9:9:9:2          yes 3500,0000 2200,0000
 10    0      0   10 10:10:10:2       yes 3500,0000 2200,0000
 11    0      0   11 11:11:11:2       yes 3500,0000 2200,0000
 12    0      0   12 12:12:12:3       yes 3500,0000 2200,0000
 13    0      0   13 13:13:13:3       yes 3500,0000 2200,0000
 14    0      0   14 14:14:14:3       yes 3500,0000 2200,0000
 15    0      0   15 15:15:15:3       yes 3500,0000 2200,0000
 16    0      0    0 0:0:0:0          yes 3500,0000 2200,0000
 17    0      0    1 1:1:1:0          yes 3500,0000 2200,0000
 18    0      0    2 2:2:2:0          yes 3500,0000 2200,0000
 19    0      0    3 3:3:3:0          yes 3500,0000 2200,0000
 20    0      0    4 4:4:4:1          yes 3500,0000 2200,0000
 21    0      0    5 5:5:5:1          yes 3500,0000 2200,0000
 22    0      0    6 6:6:6:1          yes 3500,0000 2200,0000
 23    0      0    7 7:7:7:1          yes 3500,0000 2200,0000
 24    0      0    8 8:8:8:2          yes 3500,0000 2200,0000
 25    0      0    9 9:9:9:2          yes 3500,0000 2200,0000
 26    0      0   10 10:10:10:2       yes 3500,0000 2200,0000
 27    0      0   11 11:11:11:2       yes 3500,0000 2200,0000
 28    0      0   12 12:12:12:3       yes 3500,0000 2200,0000
 29    0      0   13 13:13:13:3       yes 3500,0000 2200,0000
 30    0      0   14 14:14:14:3       yes 3500,0000 2200,0000
 31    0      0   15 15:15:15:3       yes 3500,0000 2200,0000

Here is the xml code that has me confused

...
<vcpu placement='static'>8</vcpu>
<iothreads>1</iothreads>
<cputune>
    <vcpupin vcpu='0' cpuset='2'/>
    <vcpupin vcpu='1' cpuset='8'/>
    <vcpupin vcpu='2' cpuset='3'/>
    <vcpupin vcpu='3' cpuset='9'/>
    <vcpupin vcpu='4' cpuset='4'/>
    <vcpupin vcpu='5' cpuset='10'/>
    <vcpupin vcpu='6' cpuset='5'/>
    <vcpupin vcpu='7' cpuset='11'/>
    <emulatorpin cpuset='0,6'/>
    <iothreadpin iothread='1' cpuset='0,6'/>
</cputune>
    ...
    <topology sockets='1' cores='4' threads='2'/>
    ...

I don’t understand why their “cpuset” seems to be so random. From their lscpu -e output

CPU NODE SOCKET CORE L1d:L1i:L2:L3 ONLINE MAXMHZ    MINMHZ
0   0    0      0    0:0:0:0       yes    4600.0000 800.0000
1   0    0      1    1:1:1:0       yes    4600.0000 800.0000
2   0    0      2    2:2:2:0       yes    4600.0000 800.0000
3   0    0      3    3:3:3:0       yes    4600.0000 800.0000
4   0    0      4    4:4:4:0       yes    4600.0000 800.0000
5   0    0      5    5:5:5:0       yes    4600.0000 800.0000
6   0    0      0    0:0:0:0       yes    4600.0000 800.0000
7   0    0      1    1:1:1:0       yes    4600.0000 800.0000
8   0    0      2    2:2:2:0       yes    4600.0000 800.0000
9   0    0      3    3:3:3:0       yes    4600.0000 800.0000
10  0    0      4    4:4:4:0       yes    4600.0000 800.0000
11  0    0      5    5:5:5:0       yes    4600.0000 800.0000

I have also tried using this command to get some sens as of what cpu set fits where, and tried the xml code below, with horrible results.

virsh capabilities | grep “cpu id”

            <cpu id='0' socket_id='0' die_id='0' core_id='0' siblings='0,16'/>
            <cpu id='1' socket_id='0' die_id='0' core_id='1' siblings='1,17'/>
            <cpu id='2' socket_id='0' die_id='0' core_id='2' siblings='2,18'/>
            <cpu id='3' socket_id='0' die_id='0' core_id='3' siblings='3,19'/>
            <cpu id='4' socket_id='0' die_id='0' core_id='4' siblings='4,20'/>
            <cpu id='5' socket_id='0' die_id='0' core_id='5' siblings='5,21'/>
            <cpu id='6' socket_id='0' die_id='0' core_id='6' siblings='6,22'/>
            <cpu id='7' socket_id='0' die_id='0' core_id='7' siblings='7,23'/>
            <cpu id='8' socket_id='0' die_id='0' core_id='8' siblings='8,24'/>
            <cpu id='9' socket_id='0' die_id='0' core_id='9' siblings='9,25'/>
            <cpu id='10' socket_id='0' die_id='0' core_id='10' siblings='10,26'/>
            <cpu id='11' socket_id='0' die_id='0' core_id='11' siblings='11,27'/>
            <cpu id='12' socket_id='0' die_id='0' core_id='12' siblings='12,28'/>
            <cpu id='13' socket_id='0' die_id='0' core_id='13' siblings='13,29'/>
            <cpu id='14' socket_id='0' die_id='0' core_id='14' siblings='14,30'/>
            <cpu id='15' socket_id='0' die_id='0' core_id='15' siblings='15,31'/>
            <cpu id='16' socket_id='0' die_id='0' core_id='0' siblings='0,16'/>
            <cpu id='17' socket_id='0' die_id='0' core_id='1' siblings='1,17'/>
            <cpu id='18' socket_id='0' die_id='0' core_id='2' siblings='2,18'/>
            <cpu id='19' socket_id='0' die_id='0' core_id='3' siblings='3,19'/>
            <cpu id='20' socket_id='0' die_id='0' core_id='4' siblings='4,20'/>
            <cpu id='21' socket_id='0' die_id='0' core_id='5' siblings='5,21'/>
            <cpu id='22' socket_id='0' die_id='0' core_id='6' siblings='6,22'/>
            <cpu id='23' socket_id='0' die_id='0' core_id='7' siblings='7,23'/>
            <cpu id='24' socket_id='0' die_id='0' core_id='8' siblings='8,24'/>
            <cpu id='25' socket_id='0' die_id='0' core_id='9' siblings='9,25'/>
            <cpu id='26' socket_id='0' die_id='0' core_id='10' siblings='10,26'/>
            <cpu id='27' socket_id='0' die_id='0' core_id='11' siblings='11,27'/>
            <cpu id='28' socket_id='0' die_id='0' core_id='12' siblings='12,28'/>
            <cpu id='29' socket_id='0' die_id='0' core_id='13' siblings='13,29'/>
            <cpu id='30' socket_id='0' die_id='0' core_id='14' siblings='14,30'/>
            <cpu id='31' socket_id='0' die_id='0' core_id='15' siblings='15,31'/>

XML code for implementing cpu pinning

<cputune>
    <vcpupin vcpu="0" cpuset="1"/>
    <vcpupin vcpu="1" cpuset="17"/>
    <vcpupin vcpu="2" cpuset="2"/>
    <vcpupin vcpu="3" cpuset="18"/>
    <vcpupin vcpu="4" cpuset="3"/>
    <vcpupin vcpu="5" cpuset="19"/>
    <vcpupin vcpu="6" cpuset="4"/>
    <vcpupin vcpu="7" cpuset="20"/>
    <vcpupin vcpu="8" cpuset="5"/>
    <vcpupin vcpu="9" cpuset="21"/>
    <vcpupin vcpu="10" cpuset="6"/>
    <vcpupin vcpu="12" cpuset="22"/>
    <vcpupin vcpu="13" cpuset="7"/>
    <vcpupin vcpu="14" cpuset="23"/>
    <vcpupin vcpu="15" cpuset="8"/>
    <vcpupin vcpu="16" cpuset="24"/>
    <vcpupin vcpu="17" cpuset="9"/>
    <vcpupin vcpu="18" cpuset="25"/>
    <vcpupin vcpu="19" cpuset="10"/>
    <vcpupin vcpu="20" cpuset="26"/>
    <vcpupin vcpu="21" cpuset="11"/>
    <vcpupin vcpu="22" cpuset="27"/>
    <vcpupin vcpu="23" cpuset="12"/>
    <vcpupin vcpu="24" cpuset="28"/>
    <vcpupin vcpu="25" cpuset="13"/>
    <vcpupin vcpu="26" cpuset="29"/>
    <vcpupin vcpu="27" cpuset="14"/>
    <vcpupin vcpu="28" cpuset="30"/>
    <vcpupin vcpu="29" cpuset="15"/>
    <vcpupin vcpu="30" cpuset="31"/>
    <emulatorpin cpuset="0,16"/>
    <iothreadpin iothread='1' cpuset='0,16'/>
</cputune>

Current running XML code for my entire VM.

<domain type="kvm">
  <name>win10</name>
  <uuid>95b97836-0676-47d8-b234-75fa569d8a1b</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://microsoft.com/win/10"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit="KiB">16384000</memory>
  <currentMemory unit="KiB">16384000</currentMemory>
  <vcpu placement="static" current="20">32</vcpu>
  <os>
    <type arch="x86_64" machine="pc-q35-5.0">hvm</type>
    <loader readonly="yes" type="pflash">/usr/share/edk2-ovmf/x64/OVMF_CODE.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/win10_VARS.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state="on"/>
      <vapic state="on"/>
      <spinlocks state="on" retries="8191"/>
      <vendor_id state="on" value="something"/>
    </hyperv>
    <kvm>
      <hidden state="on"/>
    </kvm>
    <vmport state="off"/>
    <ioapic driver="kvm"/>
  </features>
  <cpu mode="host-passthrough" check="partial">
    <topology sockets="1" dies="1" cores="16" threads="2"/>
  </cpu>
  <clock offset="localtime">
    <timer name="rtc" tickpolicy="catchup"/>
    <timer name="pit" tickpolicy="delay"/>
    <timer name="hpet" present="no"/>
    <timer name="hypervclock" present="yes"/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled="no"/>
    <suspend-to-disk enabled="no"/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type="block" device="disk">
      <driver name="qemu" type="raw"/>
      <source dev="/dev/nvme0n1"/>
      <target dev="vda" bus="virtio"/>
      <boot order="1"/>
      <address type="pci" domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
    </disk>
    <disk type="file" device="cdrom">
      <driver name="qemu" type="raw"/>
      <source file="/home/matrucious/Downloads/virtio-win-0.1.171.iso"/>
      <target dev="sda" bus="sata"/>
      <readonly/>
      <address type="drive" controller="0" bus="0" target="0" unit="0"/>
    </disk>
    <controller type="usb" index="0" model="qemu-xhci" ports="15">
      <address type="pci" domain="0x0000" bus="0x02" slot="0x00" function="0x0"/>
    </controller>
    <controller type="sata" index="0">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x1f" function="0x2"/>
    </controller>
    <controller type="pci" index="0" model="pcie-root"/>
    <controller type="pci" index="1" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="1" port="0x10"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0" multifunction="on"/>
    </controller>
    <controller type="pci" index="2" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="2" port="0x11"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x1"/>
    </controller>
    <controller type="pci" index="3" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="3" port="0x12"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x2"/>
    </controller>
    <controller type="pci" index="4" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="4" port="0x13"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x3"/>
    </controller>
    <controller type="pci" index="5" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="5" port="0x14"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x4"/>
    </controller>
    <controller type="pci" index="6" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="6" port="0x8"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x0" multifunction="on"/>
    </controller>
    <controller type="pci" index="7" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="7" port="0x9"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x1"/>
    </controller>
    <controller type="pci" index="8" model="pcie-to-pci-bridge">
      <model name="pcie-pci-bridge"/>
      <address type="pci" domain="0x0000" bus="0x07" slot="0x00" function="0x0"/>
    </controller>
    <controller type="pci" index="9" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="9" port="0xa"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x2"/>
    </controller>
    <interface type="network">
      <mac address="52:54:00:2b:6e:20"/>
      <source network="default"/>
      <model type="virtio"/>
      <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
    </interface>
    <input type="mouse" bus="ps2"/>
    <input type="keyboard" bus="ps2"/>
    <hostdev mode="subsystem" type="usb" managed="yes">
      <source>
        <vendor id="0x1038"/>
        <product id="0x1294"/>
      </source>
      <address type="usb" bus="0" port="4"/>
    </hostdev>
    <hostdev mode="subsystem" type="usb" managed="yes">
      <source>
        <vendor id="0x1038"/>
        <product id="0x1290"/>
      </source>
      <address type="usb" bus="0" port="5"/>
    </hostdev>
    <hostdev mode="subsystem" type="usb" managed="yes">
      <source>
        <vendor id="0x08bb"/>
        <product id="0x2902"/>
      </source>
      <address type="usb" bus="0" port="6"/>
    </hostdev>
    <hostdev mode="subsystem" type="usb" managed="yes">
      <source>
        <vendor id="0x1b1c"/>
        <product id="0x0c10"/>
      </source>
      <address type="usb" bus="0" port="3"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x0c" slot="0x00" function="0x0"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x0c" slot="0x00" function="0x1"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x06" slot="0x00" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="usb" managed="yes">
      <source>
        <vendor id="0x046d"/>
        <product id="0xc08d"/>
      </source>
      <address type="usb" bus="0" port="1"/>
    </hostdev>
    <hostdev mode="subsystem" type="usb" managed="yes">
      <source>
        <vendor id="0x1532"/>
        <product id="0x0221"/>
      </source>
      <address type="usb" bus="0" port="2"/>
    </hostdev>
    <hostdev mode="subsystem" type="usb" managed="yes">
      <source>
        <vendor id="0x046d"/>
        <product id="0xc539"/>
        <address bus="7" device="5"/>
      </source>
      <address type="usb" bus="0" port="7"/>
    </hostdev>
    <memballoon model="virtio">
      <address type="pci" domain="0x0000" bus="0x04" slot="0x00" function="0x0"/>
    </memballoon>
  </devices>
</domain>

Lastly if anyone finds it easier to help me using lstopo, here it is.

Any help is greatly appreciated! I already tried to post a similar question on the Manjaro forums, but with no answers.
My main goal is to pass through every CPU core, except Core 0.
I haven’t done isolcpus yet, because I can’t quite understand that either.

It looks to me as if they are pinning virtual CPUs to physical Hyperthread pairs. CPU 2 and 8 are both the same core, for example.

I don’t do mine that way. For my Windows VM on a Ryzen 3900X I gave the VM 12 cores and made them all independent.

But if you don’t trust the shared cores for security reasons then giving both HT/SMT logical cores to the VM makes sense. It doesn’t matter then if the VM is hostile and trying to extract side-channel data because the only code running on the HT pair is its own.

2 Likes

I have a 3950x, so do you mind helping me a bit out with how i should do mine, considering they are quite similar cpus?
I don’t want to give all cores to my VM, probably 14 15 cores.
However I thought that I had to pin the Hyperthreaded pins aswell, so Windows can actually recognize that it can use up to for example 28/30 threads?

If you want the VM to use 30 threads on a 16 core CPU then you must absolutely add the SMT logical cores.

Windows will recognize what you tell it to recognize. That’s in the CPU topology section. You can vary from claiming it is 16 physical CPUs in 16 sockets, to 1 CPU core with 16 SMT threads. It believes what the BIOS tells it.

This is exactly right.

I wrote a guide a while back on CPU pinning and numa node optimization, linked below. It’s focused on Threadripper, but I think you’ll find it very helpful:

1 Like

You can lose a lot of performance by pinning to a wrong cache level. You want to emulate the same groups. It is not random. For example in my 7700k I nave cores 2-4 and each according to lstopo.

I had it mismatched and lost a enough performance to have games unplayable even unity games like MTG Arena. (This is also why it is not recommended to pin in the first place - wrong CPU pinning will result in worse performance than automatic QEMU)

image

<cputune>
<vcpupin vcpu="0" cpuset="1"/>
<vcpupin vcpu="1" cpuset="2"/>
<vcpupin vcpu="2" cpuset="3"/>
<vcpupin vcpu="3" cpuset="5"/>
<vcpupin vcpu="4" cpuset="6"/>
<vcpupin vcpu="5" cpuset="7"/>
<emulatorpin cpuset="0"/>
</cputune>
1 Like