High kernel process usage on CPU 0 isolcpus

Operating Systems & Open Source Virtualization
26

Ok, as promised below is hook for libvirt/qemu. BUT like I said before, in my case (tested on Manjaro 20.2) its pointless, because everything you need for pinning can be set from livirt’s XML VM file.
So to make it at least somewhat useful I implemented logging of CPU affinities. Someone may find it helpful for debugging settings in XML, or just as an example of hook for doing other things.

You need to put this script in /etc/libvirt/hooks/qemu and change VM1NAME to desired string. Location of log can be changed also.

#!/bin/sh
LOG="/var/log/libvirt_qemu_hook.log"
VM1NAME="w10test"

if [[ $1 == $VM1NAME ]] && [[ $2 == "started" ]]; then
  echo "VM $VM1NAME started" >> $LOG
  #Finds main PID of qemu
  for CPID in $(pidof qemu-system-x86_64);do
    MPID=$(pstree -pa $CPID | grep $VM1NAME | awk -F',' '{print $2}' | awk '{print $1}')
    if [ -n "$MPID" ];then echo "Found $VM1NAME pid $MPID" >> $LOG; break; fi
  done
  #Loops over all qemu threads
  for CPID in $(pstree -pa $MPID | cut -d',' -f2 | cut -d' ' -f1); do
    #log affinity for pids
     taskset -pc $CPID >> $LOG
  done

fi
if [[ $1 == $VM1NAME ]] && [[ $2 == "stopped" ]]; then
  echo "VM $VM1NAME stopped" >> $LOG
fi

Using vcpupin, emulatorpin and iothreadpin in XML you can change pinning as you like. For example, pinning 8 vcpus to cpu threads 8-15 like this:

<cputune>
    <vcpupin vcpu="0" cpuset="8"/>
   (... pins 1 through 6 ...)
    <vcpupin vcpu="7" cpuset="15"/>
    <emulatorpin cpuset="6-7"/>
    <iothreadpin iothread="1" cpuset="5"/>
  </cputune>

You should get following log:

VM w10test started
Found w10test pid 347930
pid 347930's current affinity list: 6,7
pid 347945's current affinity list: 6,7
pid 347946's current affinity list: 5
pid 347947's current affinity list: 6,7
pid 347949's current affinity list: 6,7
pid 347950's current affinity list: 8
pid 347951's current affinity list: 9
pid 347952's current affinity list: 10
pid 347953's current affinity list: 11
pid 347954's current affinity list: 12
pid 347955's current affinity list: 13
pid 347956's current affinity list: 14
pid 347957's current affinity list: 15
pid 347960's current affinity list: 6,7
VM w10test stopped

Of course instead logging taskset output you can pin them in other way, but like I said its pointless/redundant, and I made this script basically for practice.

Also I made another hook, that is way more useful, that replaces need for using “isolcpus”. Rebooting each time I need all cores for working on host, and rebooting again to run VM is just too much hassle. Just like FLR bug, thanks to @gnif and @belfrypossum hopefully thing of the past.

This script needs cpuset script from https://github.com/lpechacek/cpuset and allows for thread isolation for VM.
Again, you have to replace VM1NAME and VM1ISOL has to have list of threads that you want to reserve only for VM. Example is for VM above so 8-15:

#!/bin/sh
LOG="/var/log/libvirt_qemu_hook.log"
VM1NAME="w10test"
VM1ISOL="8-15"

if [[ $1 == $VM1NAME ]] && [[ $2 == "prepare" ]]; then
  echo "VM $VM1NAME preparing" >> $LOG
  cset shield -c $VM1ISOL >> $LOG
fi
if [[ $1 == $VM1NAME ]] && [[ $2 == "stopped" ]]; then
  echo "VM $VM1NAME stopped" >> $LOG
  cset shield --reset >> $LOG
fi

Like in first script there’s log that you can check if something goes wrong with cset.

1 Like