This is what you needed? I’m stressing 8 threads 9-16 through VM running cpu-z. And its tad faster than i7-6700K:
If yes then post your isolcpus line.
can you post the code you used for taskset?
I’m not using taskset. I need your isolcpus, to give you solution.
rcu_nocbs=0-7 nohz_full=0-7 isolcpus=0-7
Erm, if you are isolating 0-7 then everything works as it supposed to…
But If you want effect like mine (or from that guide on null-src), then you have to do at least change:
isolcpus=8-15
And change your cputune VM config to:
<cputune>
<vcpupin vcpu="0" cpuset="8"/>
<vcpupin vcpu="1" cpuset="9"/>
<vcpupin vcpu="2" cpuset="10"/>
<vcpupin vcpu="3" cpuset="11"/>
<vcpupin vcpu="4" cpuset="12"/>
<vcpupin vcpu="5" cpuset="13"/>
<vcpupin vcpu="6" cpuset="14"/>
<vcpupin vcpu="7" cpuset="15"/>
...
And libvirt is pinning everything correctly as it supposed to. No need for tasksel.
Also, this original script from null-src is broken. Because I ran VM with it from shell without libvirt, and it just does first spawn with tasksel. Then it waits 20 seconds and exits, because it doesn’t find any processes. So its smoke and mirrors. I tested that VM with cpuz and its bit slower than through libvirt on 8 cores and bit faster on single core.
Anyway, I have more about this but its after 7am here so I need to sleep 
Ok, as promised below is hook for libvirt/qemu. BUT like I said before, in my case (tested on Manjaro 20.2) its pointless, because everything you need for pinning can be set from livirt’s XML VM file.
So to make it at least somewhat useful I implemented logging of CPU affinities. Someone may find it helpful for debugging settings in XML, or just as an example of hook for doing other things.
You need to put this script in /etc/libvirt/hooks/qemu and change VM1NAME to desired string. Location of log can be changed also.
#!/bin/sh
LOG="/var/log/libvirt_qemu_hook.log"
VM1NAME="w10test"
if [[ $1 == $VM1NAME ]] && [[ $2 == "started" ]]; then
echo "VM $VM1NAME started" >> $LOG
#Finds main PID of qemu
for CPID in $(pidof qemu-system-x86_64);do
MPID=$(pstree -pa $CPID | grep $VM1NAME | awk -F',' '{print $2}' | awk '{print $1}')
if [ -n "$MPID" ];then echo "Found $VM1NAME pid $MPID" >> $LOG; break; fi
done
#Loops over all qemu threads
for CPID in $(pstree -pa $MPID | cut -d',' -f2 | cut -d' ' -f1); do
#log affinity for pids
taskset -pc $CPID >> $LOG
done
fi
if [[ $1 == $VM1NAME ]] && [[ $2 == "stopped" ]]; then
echo "VM $VM1NAME stopped" >> $LOG
fi
Using vcpupin, emulatorpin and iothreadpin in XML you can change pinning as you like. For example, pinning 8 vcpus to cpu threads 8-15 like this:
<cputune>
<vcpupin vcpu="0" cpuset="8"/>
(... pins 1 through 6 ...)
<vcpupin vcpu="7" cpuset="15"/>
<emulatorpin cpuset="6-7"/>
<iothreadpin iothread="1" cpuset="5"/>
</cputune>
You should get following log:
VM w10test started
Found w10test pid 347930
pid 347930's current affinity list: 6,7
pid 347945's current affinity list: 6,7
pid 347946's current affinity list: 5
pid 347947's current affinity list: 6,7
pid 347949's current affinity list: 6,7
pid 347950's current affinity list: 8
pid 347951's current affinity list: 9
pid 347952's current affinity list: 10
pid 347953's current affinity list: 11
pid 347954's current affinity list: 12
pid 347955's current affinity list: 13
pid 347956's current affinity list: 14
pid 347957's current affinity list: 15
pid 347960's current affinity list: 6,7
VM w10test stopped
Of course instead logging taskset output you can pin them in other way, but like I said its pointless/redundant, and I made this script basically for practice.
Also I made another hook, that is way more useful, that replaces need for using “isolcpus”. Rebooting each time I need all cores for working on host, and rebooting again to run VM is just too much hassle. Just like FLR bug, thanks to @gnif and @belfrypossum hopefully thing of the past.
This script needs cpuset script from https://github.com/lpechacek/cpuset and allows for thread isolation for VM.
Again, you have to replace VM1NAME and VM1ISOL has to have list of threads that you want to reserve only for VM. Example is for VM above so 8-15:
#!/bin/sh
LOG="/var/log/libvirt_qemu_hook.log"
VM1NAME="w10test"
VM1ISOL="8-15"
if [[ $1 == $VM1NAME ]] && [[ $2 == "prepare" ]]; then
echo "VM $VM1NAME preparing" >> $LOG
cset shield -c $VM1ISOL >> $LOG
fi
if [[ $1 == $VM1NAME ]] && [[ $2 == "stopped" ]]; then
echo "VM $VM1NAME stopped" >> $LOG
cset shield --reset >> $LOG
fi
Like in first script there’s log that you can check if something goes wrong with cset.
1 Like
csets are very different to isolcpus, there are benefits to using it over cset such as nohz and keeping interrupts off those cores also. It is not a direct replacement and does not give reliable low latency when devices get busy. If you are setting up cores that are dedicated to a VM, isolcpus are always preferred.
Yeah, i know, i was gonna mention that, just forgot. But restart needed with isolcpus rules it out for me. Especially that games work ok on my windows console even without any isolation. So cset if anything is a step up.
Thanks for pointing that out tho.
Edit: Also forgot to add, that the same thing supposed to be achieve directly from libvirt with cgroups. But couldn’t get it working, hence the hook.