Loading vfio again and rebooting now.
what are the benefits, isn’t it better to build the VM like the underlying hardware?
It depends what you are doing. For gaming, this method gives you higher frame rate. Also Windows sucks when you expose complicated underlying hardware.
hm, I have an TR 2920x, 12 Cores two NUMA nodes, and got my best results with a NUMA aware configuration.
There is a problem with CPUs that do not have all cores enabled, the 3 level is not correct. I haven’t tested this for a long time, maybe it works now without special config, but with my CPU I had to configure it that way
<memoryBacking>
<hugepages/>
</memoryBacking>
<vcpu placement='static' current='16'>24</vcpu>
<vcpus>
<vcpu id='0' enabled='yes' hotpluggable='no'/>
<vcpu id='1' enabled='yes' hotpluggable='yes'/>
<vcpu id='2' enabled='yes' hotpluggable='yes'/>
<vcpu id='3' enabled='yes' hotpluggable='yes'/>
<vcpu id='4' enabled='no' hotpluggable='yes'/>
<vcpu id='5' enabled='no' hotpluggable='yes'/>
<vcpu id='6' enabled='yes' hotpluggable='yes'/>
<vcpu id='7' enabled='yes' hotpluggable='yes'/>
<vcpu id='8' enabled='yes' hotpluggable='yes'/>
<vcpu id='9' enabled='yes' hotpluggable='yes'/>
<vcpu id='10' enabled='no' hotpluggable='yes'/>
<vcpu id='11' enabled='no' hotpluggable='yes'/>
<vcpu id='12' enabled='yes' hotpluggable='yes'/>
<vcpu id='13' enabled='yes' hotpluggable='yes'/>
<vcpu id='14' enabled='yes' hotpluggable='yes'/>
<vcpu id='15' enabled='yes' hotpluggable='yes'/>
<vcpu id='16' enabled='no' hotpluggable='yes'/>
<vcpu id='17' enabled='no' hotpluggable='yes'/>
<vcpu id='18' enabled='yes' hotpluggable='yes'/>
<vcpu id='19' enabled='yes' hotpluggable='yes'/>
<vcpu id='20' enabled='yes' hotpluggable='yes'/>
<vcpu id='21' enabled='yes' hotpluggable='yes'/>
<vcpu id='22' enabled='no' hotpluggable='yes'/>
<vcpu id='23' enabled='no' hotpluggable='yes'/>
</vcpus>
<iothreads>1</iothreads>
<cputune>
<vcpupin vcpu='0' cpuset='1'/>
<vcpupin vcpu='1' cpuset='13'/>
<vcpupin vcpu='2' cpuset='2'/>
<vcpupin vcpu='3' cpuset='14'/>
<vcpupin vcpu='4' cpuset='3'/>
<vcpupin vcpu='5' cpuset='15'/>
<vcpupin vcpu='6' cpuset='4'/>
<vcpupin vcpu='7' cpuset='16'/>
<vcpupin vcpu='8' cpuset='7'/>
<vcpupin vcpu='9' cpuset='19'/>
<vcpupin vcpu='10' cpuset='8'/>
<vcpupin vcpu='11' cpuset='20'/>
<vcpupin vcpu='12' cpuset='9'/>
<vcpupin vcpu='13' cpuset='21'/>
<vcpupin vcpu='14' cpuset='10'/>
<vcpupin vcpu='15' cpuset='22'/>
<emulatorpin cpuset='5,11,17,23'/>
<iothreadpin iothread='1' cpuset='0,6,12,18'/>
</cputune>
<numatune>
<memory mode='strict' nodeset='0-1'/>
<memnode cellid='0' mode='strict' nodeset='0'/>
<memnode cellid='1' mode='strict' nodeset='1'/>
</numatune>
<cpu mode='custom' match='exact' check='none'>
<model fallback='allow'>EPYC</model>
<topology sockets='1' dies='4' cores='3' threads='2'/>
<feature policy='require' name='topoext'/>
<numa>
<cell id='0' cpus='0-11' memory='12582912' unit='KiB'/>
<cell id='1' cpus='12-23' memory='12582912' unit='KiB'/>
</numa>
</cpu>
With this configuration I have about the same CPU performance as a 2700X sometimes even considerably better, in e.g. CPU PhotoWorxx is better than a 12900k, but the AES performance is bad.
But I will test your recommendation, let’s see if it works better with my CPU too.
2 Likes
Threadripper is another beast. NUMA aware configuration probably makes sense.
For 7900x/7950x, I don’t think there is much penalty to switch context between 2 CCDs.
1 Like
I just came across a thread that’s the most comprehensive I’ve seen on the subject that I think many would find helpful as it clarifies multiple aspects of a very complicated topic.
Note that it’s targeted toward proxmox. The biggest thing to know with proxmox is that it typically stores it’s vm/qemu options in a simplified format in a .conf file, rather than XML.
https://forum.proxmox.com/threads/hey-proxmox-community-lets-talk-about-resources-isolation.124256/
@wendell If you still use proxmox, you’ll likely appreciate this as well.
And another fantastic writeup is here. Restricting latency sensitive things to CCD’s can definitely be worthwhile: https://kb.blockbridge.com/technote/proxmox-tuning-low-latency-storage/
4 Likes
too bad, Proxmox didn’t like the post and deleted it, did someone make a copy?
Or more likely the clock in your guest drifts and gives you an illusion of achieving higher scores. 
1 Like
Edit: Unless you mean some additional posts he added recently, I don’t have those yet, we’ll have to wait a bit. The Proxmox forums are notorious for having overactive automod scripts. See 1 and 2
Original post:
Link seems to work fine for me still. If you still can’t see it, then here’s the internet archive page: [TUTORIAL] - Hey Proxmox & Community - Let's talk about resources isolation | Proxmox Support Forum
It can’t explain that it only drifts for cpu score not on gpu score.
I got 2 times something like page not found, now it works again, thanks!
Exactly what I observed in a broken VM before fixed.
Thus, how to fix it?