CPU optimization between two VMs

lI_Simo_Hayha_Il · July 6, 2021, 11:28am

I am trying to configure two VMs, that run at the same time for optimal CPU usage.

In short:
Host: Manjaro
Gaming VM: Windows 10 with VGA, SSD, NVME, LAN passthrough running on SSD
Coding VM: Windows 10 with webcam passthrough running on qcow2 file

So, my daily routine is that I am working on my “Coding VM” and when I have breaks, or long periods without work (waiting for answers, approvals), I am playing games on my “Gaming VM”.

The problem I am trying to solve is that on my “Gaming VM”, I have big framedrops (140 → 70), with no apparent reason, while the GPU usage is <70%. I have seen differences while changing settings in CPU pinning, therefore I assume it is CPU related, but haven’t been able to solve it. Most probably I am doing something wrong in CPU configuration, but I don’t know what…

Host:

Operating System: Manjaro Linux
KDE Plasma Version: 5.21.5
KDE Frameworks Version: 5.82.0
Qt Version: 5.15.2
Kernel Version: 5.12.9-1-MANJARO
OS Type: 64-bit
Graphics Platform: X11
Processors: 24 × AMD Ryzen 9 5900X 12-Core Processor
Memory: 62.7 GiB of RAM
Graphics Processor: Radeon RX 5500 XT
Passthrough Graphics Processor: Radeon RX 6900 XT

Gaming:

<domain type='kvm'>
  <name>GamingVM</name>
  <uuid>xxxxxx</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://microsoft.com/win/10"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit='KiB'>16777216</memory>
  <currentMemory unit='KiB'>16777216</currentMemory>
  <vcpu placement='static'>16</vcpu>
  <iothreads>2</iothreads>
  <cputune>
    <vcpupin vcpu='0' cpuset='4'/>
    <vcpupin vcpu='1' cpuset='16'/>
    <vcpupin vcpu='2' cpuset='5'/>
    <vcpupin vcpu='3' cpuset='17'/>
    <vcpupin vcpu='4' cpuset='6'/>
    <vcpupin vcpu='5' cpuset='18'/>
    <vcpupin vcpu='6' cpuset='7'/>
    <vcpupin vcpu='7' cpuset='19'/>
    <vcpupin vcpu='8' cpuset='8'/>
    <vcpupin vcpu='9' cpuset='20'/>
    <vcpupin vcpu='10' cpuset='9'/>
    <vcpupin vcpu='11' cpuset='21'/>
    <vcpupin vcpu='12' cpuset='10'/>
    <vcpupin vcpu='13' cpuset='22'/>
    <vcpupin vcpu='14' cpuset='11'/>
    <vcpupin vcpu='15' cpuset='23'/>
    <emulatorpin cpuset='0-3'/>
    <iothreadpin iothread='1' cpuset='0-1'/>
    <iothreadpin iothread='2' cpuset='2-3'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-q35-5.2'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/edk2-ovmf/x64/OVMF_CODE.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/win10-games_VARS.fd</nvram>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vpindex state='on'/>
      <runtime state='on'/>
      <synic state='on'/>
      <stimer state='on'/>
      <reset state='on'/>
      <vendor_id state='on' value='OriginalAMD'/>
      <frequencies state='on'/>
    </hyperv>
    <kvm>
      <hidden state='on'/>
    </kvm>
    <vmport state='off'/>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <topology sockets='1' dies='1' cores='8' threads='2'/>
    <cache mode='passthrough'/>
    <feature policy='require' name='topoext'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='yes'/>
 </clock>
..... (omitted non-related lines)
</domain>

Coding:

<domain type='kvm'>
  <name>CodingVM</name>
  <uuid>zzzzzzz</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://microsoft.com/win/10"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit='KiB'>16777216</memory>
  <currentMemory unit='KiB'>16777216</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <iothreads>2</iothreads>
  <cputune>
    <emulatorpin cpuset='0-3'/>
    <iothreadpin iothread='1' cpuset='0-1'/>
    <iothreadpin iothread='2' cpuset='2-3'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-q35-6.0'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/edk2-ovmf/x64/OVMF_CODE.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/win10-autoaddress_VARS.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>
    <vmport state='off'/>
  </features>
  <cpu mode='host-model' check='partial'>
    <topology sockets='1' dies='1' cores='2' threads='2'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='yes'/>
  </clock>
..... (omitted non-related lines)
</domain>

anon86748826 · July 6, 2021, 12:22pm

When I understand this correctly you use all of the available 12 Cores for either the emulation or computation of the gaming VM. Simultaneously your other VM as well as all the threads of you host operating system will have to be scheduled on some of these cores as well. That means both the host OS as well as any other VMs may interfere with your games while they compete for resources. Pinning only means that your gaming VM will run on the cores specified, but it is no isolation. Other threads can be scheduled on these cores as well.

What I would do is limit the gaming VM to 6 cores, plus an additional 1 or 2 cores for emulation and IO. Then install vfio-isolate to shield the gaming VM from the rest of the system. vfio-isolate can reserve the specified cores for the gaming VM and keep the host as well as any other VMs from accessing them.

You can initiate vfio-isolate via a qemu hook like the following example for a 16 Core processor with a VM called gaming on the last 8 cores ,in /etc/libvirt/hooks/qemu:

#!/bin/bash

if [ "$1" != "gaming" ]; then
	exit 0
fi

RSET=/

HSET=/host.slice
HNODE=C0-6,16-22

MNODE=C8-15,24-31

UNDOFILE=/var/run/libvirt/qemu/vfio-isolate-undo.bin

disable_isolation () {
	vfio-isolate \
		restore $UNDOFILE

	taskset -pc 0-31 2
}

enable_isolation () {
	vfio-isolate cpuset-delete $HSET

	vfio-isolate \
		-u $UNDOFILE \
		drop-caches \
		cpuset-create --cpus $HNODE $HSET \
		compact-memory \
		move-tasks $RSET $HSET \
		cpu-governor performance $MNODE \
		irq-affinity mask $MNODE

	taskset -pc 0-6,16-22 2
}



case "$2" in
"prepare")
	enable_isolation
	;;
"started")
	;;
"release")
	disable_isolation
	;;
esac

Edit: vfio-isolate is available in the AUR

Edit: If you need further help setting this up, ping me!

lI_Simo_Hayha_Il · July 6, 2021, 12:57pm

Thank you for your reply. Actually, I totally forgot that I have done some sort of isolation already…
Don’t remember installing vfio-isolate, but I am using below scripts:

/etc/libvirt/qemu/hooks/qemu.d/win10-games/started/begin/limit-host-cpus.sh

#!/bin/bash

systemctl set-property --runtime -- user.slice AllowedCPUs=0,1,2,3,12,13,14,15
systemctl set-property --runtime -- system.slice AllowedCPUs=0,1,2,3,12,13,14,15
systemctl set-property --runtime -- init.scope AllowedCPUs=0,1,2,3,12,13,14,15

/etc/libvirt/qemu/hooks/qemu.d/win10-games/release/end/

#!/bin/bash

systemctl set-property --runtime -- user.slice AllowedCPUs=0-23
systemctl set-property --runtime -- system.slice AllowedCPUs=0-23
systemctl set-property --runtime -- init.scope AllowedCPUs=0-23

Is this the same as vfio-isolate?
Should I do something similar for my ‘Coding VM’ too?

anon86748826 · July 6, 2021, 1:05pm

As far as I know vfio-isolate is doing a lot more things. When you run something like stress -c 24 on the host OS, is it affecting your gaming VMs performance? Maybe you can try if vfio-isolate helps anyways. Install vfio-isolate, comment out your scripts, edit the one I posted. I certainly know I can stress my host as much as I want the VM has no issues. Your issue could originate from a entirely different problem still.

lI_Simo_Hayha_Il · July 6, 2021, 1:14pm

Yes, it actually does… From 144 fps it dropped to 75-85, just like when I am using the 2nd VM

Can you please explain these parameters?

HNODE=C0-6,16-22

MNODE=C8-15,24-31

anon86748826 · July 6, 2021, 1:20pm

HNODE specifies which cores should be used by the host. MNODE specifies the cores used by the guest for computation. Core 7 and Core 23 is what I have used for my gaming VMs emulator and IO set. It should be left out of both.

lI_Simo_Hayha_Il · July 6, 2021, 1:53pm

What about the “Coding VM” ?
Should I create 2 sections in the script like:

if [ "$1" == "win10-gaming" ]; then
	---code here
fi
if [ "$1" == "win10-coding" ]; then
	---code here
fi

or just let it use the cores I assign in the HNODE ?

anon86748826 · July 6, 2021, 1:56pm

Let it use the cores from HNODE. As soon as you stop the gaming VM the coding VM will have access to all cores again. The script restores to original state when the gaming VM ist stopped and since you did not pin any cores in the coding VM it will be automatically scheduled to free cores.

if [ "$1" != "gaming" ]; then
	exit 0
fi

This part basically means ignore the rest of the script when it is not the gaming VM. The hook is invoked every time any VM is started or stopped.

lI_Simo_Hayha_Il · July 6, 2021, 2:01pm

Thank you again!

One last question (I hope…)
Even after installing ‘vfio-isolate’ from AUR, I don’t have this file:
/var/run/libvirt/qemu/vfio-isolate-undo.bin

Am I missing something?

anon86748826 · July 6, 2021, 2:05pm

vfio-isolate generates the UNDOFILE when it is run. This options is just there in case you have a special use case or different distribution and want to specify another place where it is temporarely stored.

Please test again is everything works. You now should be able to run stress -c 24 and it should use all your cores. You can check this in any system monitor you have. When you start your gaming VM core utilization should be restricted to the cores you defined in HNODE and what you specified in MNODE should only be utilized if your gaming VM is used.

lI_Simo_Hayha_Il · July 6, 2021, 2:22pm

I must be doing something wrong…
This is what I am getting when VM is OFF:

stress -c 24
stress: info: [98039] dispatching hogs: 24 cpu, 0 io, 0 vm, 0 hdd

This is what I am getting while VM is running:

stress -c 24
stress: info: [98998] dispatching hogs: 24 cpu, 0 io, 0 vm, 0 hdd

This is my qemu script located in ‘/etc/libvirt/qemu/hooks/’ :

#!/bin/bash

if [ "$1" != "win10-games" ]; then
	exit 0
fi

RSET=/

HSET=/host.slice
HNODE=C0-2,12-14

MNODE=C4-11,16-23

UNDOFILE=/var/run/libvirt/qemu/vfio-isolate-undo.bin

disable_isolation () {
	vfio-isolate \
		restore $UNDOFILE

	taskset -pc 0-23 2
}

enable_isolation () {
	vfio-isolate cpuset-delete $HSET

anon86748826 · July 6, 2021, 2:33pm

If you use the stress -c 24 it will always tell you 24 cpu because that is the setting you just made. You need to run this command in a terminal and check in a separate system monitor which cores are getting used.

Did you miss to paste part of the script because half of it is missing. In my first post you can’t see the entire script you have to scroll down. The small window with the script is scrollable!

lI_Simo_Hayha_Il · July 6, 2021, 2:59pm

Ooops…
But still it is not working…
As soon as I run the “stress -c 24” FPS drops to 40-50 and it is almost unresponsive.
This is the corrected script:

#!/bin/bash

if [ "$1" != "win10-games" ]; then
	exit 0
fi

RSET=/

HSET=/host.slice

HNODE=C0-2,12-14

MNODE=C4-11,16-23

UNDOFILE=/var/run/libvirt/qemu/vfio-isolate-undo.bin

disable_isolation () {
	vfio-isolate \
		restore $UNDOFILE

	taskset -pc 0-23 2
}

enable_isolation () {
	vfio-isolate cpuset-delete $HSET

	vfio-isolate \
		-u $UNDOFILE \
		drop-caches \
		cpuset-create --cpus $HNODE $HSET \
		compact-memory \
		move-tasks $RSET $HSET \
		cpu-governor performance $MNODE \
		irq-affinity mask $MNODE

	taskset -pc 0-2,12-14 2
}



case "$2" in
"prepare")
	enable_isolation
	;;
"started")
	;;
"release")
	disable_isolation
	;;
esac

anon86748826 · July 6, 2021, 3:04pm

Is the name correct? It has to be the name specified in the VMs XML <name></name> tag! Maybe do a reboot and try again!

anon86748826 · July 6, 2021, 3:07pm

This is wrong, the path is /etc/libvirt/hooks/qemu. The last part qemu being the name of the file.

anon86748826 · July 6, 2021, 3:17pm

Also you need to make the script executable with
sudo chmod +x /etc/libvirt/hooks/qemu
and then restart at least libvirt with
sudo systemctl restart libvirtd.service

lI_Simo_Hayha_Il · July 6, 2021, 3:50pm

Ok… Maybe that was the problem on the first place…
I had the scripts under the wrong folder all along…
There is a folder ‘/etc/libvirt/qemu/’, I created ‘hooks’ in it and put there my prepare, release, started scripts, and obviously they didn’t work…

Now I moved the qemu file you suggested under ‘/etc/libvirt/’ folder, made it executable (it should have been, as I edited my old one) and tried again, but still the same…

I will try a reboot a bit later and comeback.

edit: found this in my logs: libvirtd.service internal error: End of file from qemu monitor
Can it be related?

lI_Simo_Hayha_Il · July 6, 2021, 4:27pm

If I try to run in terminal:

sudo bash /etc/libvirt/hooks/qemu win10-games prepare

I am getting these errors:

  File "/usr/lib/python3.9/site-packages/vfio_isolate/cpuset.py", line 57, in get_cpus
    with open(self.__path("cpuset.cpus"), "r") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/sys/fs/cgroup/cpuset/host.slice/cpuset.cpus'
.........
  File "/usr/lib/python3.9/site-packages/vfio_isolate/cpuset.py", line 43, in create
    os.mkdir(self.__path())
FileNotFoundError: [Errno 2] No such file or directory: '/sys/fs/cgroup/cpuset/host.slice'

anon86748826 · July 6, 2021, 6:48pm

Is there a timestamp when this error occurred?

I do not think you can use it that way. What is happening when you start the VM, how do you see that is not working?

lI_Simo_Hayha_Il · July 6, 2021, 7:02pm

When I start the stress, fps drops from >140 to <50 and it is almost unresponsive.
As soon as I stop the stress, everything is back to normal.