[Solved] VFIO VMM poor performance - Win10

Software & Operating Systems VFIO
, ,
1

My mouse is skipping when in the VM; locks up after 5m…GPU passthrough is working fine, windows driver is installed, display out is working…

https://textuploader.com/1krci - xml

I’ve tried some CPU xml settings suggested on VFIO reddit I’ve tried creating a different partition as NTFS and setting raw [apparently BTRFS has IO issues?(according to a guide)](I’ve also used qcow2 on btrfs partition)

I’ve tried some different distros I’m currently hosting on xubuntu 19.10 QEMU emulator version 4.0.0 (Debian 1:4.0+dfsg-0ubuntu9) I had previously tried proxmox/manjaro.

Hardare: Asus Z170A, I7 6700k; RTX2080; 2 x 16gb ddr4; xpg sx8200 NVME SSD

2

It may be worth trying hugepages?

3

You’ve got a 6700k, that means you have 8 threads, 4 cores. You should be doing the following:

<cputune>
    <vcpupin vcpu="0" cpuset="0"/>
    <vcpupin vcpu="1" cpuset="1"/>
    <vcpupin vcpu="2" cpuset="2"/>
    <vcpupin vcpu="3" cpuset="3"/>
    <vcpupin vcpu="4" cpuset="4"/>
    <vcpupin vcpu="5" cpuset="5"/>
    <emulatorpin cpuset="6,7"/>
    <iothreadpin iothread="1" cpuset="6,7"/>

What this does is match the cores and threads on the physical machine to what the VM will expect. The way you’ve got it, it does something like this:

VM      PHYSICAL
C1T1 -> C1T2
C1T2 -> C2T1
C2T1 -> C2T2
C2T2 -> C3T1

I’m not sure if KVM and the VM are smart enough to tell the difference, but on my system, it made a significant difference.

If you don’t want the VM to share core 0, try this:

<cputune>
    <vcpupin vcpu="0" cpuset="2"/>
    <vcpupin vcpu="1" cpuset="3"/>
    <vcpupin vcpu="2" cpuset="4"/>
    <vcpupin vcpu="3" cpuset="5"/>
    <vcpupin vcpu="4" cpuset="6"/>
    <vcpupin vcpu="5" cpuset="7"/>
    <emulatorpin cpuset="0,1"/>
    <iothreadpin iothread="1" cpuset="0,1"/>

Ideally, you want to leave one core for Linux, but that doesn’t seem to be what you’re going for.

Very much so. You should use a direct partition or XFS/EXT4 backed storage, unless you have a large array, on ZFS.

As far as the lockups go, try this, right after <vmport state='off'/>:

<ioapic driver='kvm'/>

I have a ton of little tweaks to the hyperv section, feel free to pick and choose:

<hyperv>
  <relaxed state='on'/>
  <vapic state='on'/>
  <spinlocks state='on' retries='8191'/>
  <vpindex state='on'/>
  <runtime state='on'/>
  <synic state='on'/>
  <stimer state='on'/>
  <reset state='on'/>
  <vendor_id state='on' value='1234567890ab'/>
  <frequencies state='on'/>
  <tlbflush state='on'/>
</hyperv>

None of this will make a lick of difference without hugepages though.

EDIT:

<video>
      <model type="virtio" heads="1" primary="yes">
        <acceleration accel3d="yes"/>
      </model>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x0"/>
    </video>

This is likely your problem. You’re trying to use virtio video. It’s far from stable, and you should really disable this and fallback to QXL.

4

The following enhancements that I stole from Aiber’s XML did wonders for me, though I have no idea what most of them do:

    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='16384'/>
      <vpindex state='on'/>
      <runtime state='on'/>
      <synic state='on'/>
      <stimer state='on'/>
      <reset state='on'/>
      <vendor_id state='on' value='other'/>
      <frequencies state='on'/>
      <reenlightenment state='on'/>
      <tlbflush state='on'/>
      <ipi state='off'/>
      <evmcs state='off'/>
    </hyperv>
    <kvm>
      <hidden state='on'/>
    </kvm>
    <vmport state='off'/>
    <ioapic driver='kvm'/>
  </features>
  <cpu mode='host-model' check='partial'>
    <model fallback='allow'/>
    <topology sockets='1' cores='6' threads='2'/>
    <feature policy='require' name='topoext'/>
    <feature policy='require' name='invtsc'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='yes'/>
    <timer name='tsc' present='yes' mode='native'/>
  </clock>
5

QXL no diff
Virt was recommended everywhere else.
I previously just set 8core, but on the VFIO subreddit this config was most upped on a thread relating to 6700 I’ve tried yours, no diff
hyper v - no diff

<vmport state="off"/>
<ioapic driver="kvm"/>

Fixed the constant hitching; there’s still a stutter though and mouse is no where near accurate but before it would freeze for up to a second then jump, etc.

no diff = no noticeable performance difference.

6

By who?

That config was not ideal. Not sure how they came to that decision.

Good. This is progress.

You’re using a physical display and directly connecting the keyboard and mouse, not just using the spice display, right?

Have you enabled hugepages?

7

have not tried huge pages yet, this is usb passthrough / display out

8

Okay, that shouldn’t be a problem.

Hugepages make a massive difference on dual-channel systems. I would highly recommend trying it.

Oh, last question: Did you install the memballoon driver? That is known to cause stuttering and errors. You don’t want memballoon.

9

Also, when spice is disabled, there’s still a virtual display. Do you know how to completely disable the vdisplay? Obviously I can disable the non-pnp mon/ms display adapter in device manager, but I mean how can I disable the virtual display on the host? https://ibb.co/cvc3ms7

10

Yeah, you need to disable it through the device manager.

You can remove the device “video qxl” and “display spice” from your vm config, and it’ll go away.

That said, you don’t have access to the vm that way anymore, so I like to keep it just in case.

11

Disabling spice still has the vdisplay render just can’t use it
If I remove video [qxl] there’s no display out whatsoever

12

Wait, no display output on your physical display after removing qxl?

that’s odd.

13

Yes, no display out.

14

Interesting. I’m not sure what’s going on there.

You’re giving it time to boot into windows, right? Some GPUs only provide display out on DVI before login screen, for some reason, even if they don’t have a DVI.

15

It’s via display port; I have tried several times, and have waited several minutes. You can see when boot is done, as the perf graph shows 0% CPU utilization. I have tried this before mentioning here…
Been messing around with KVM for over a month. [only 16hrs a week or something before i gave up] :stuck_out_tongue:

16

Hmm, I couldn’t say what’s going on with the display out.

It sounds like a windows issue, but I’m really not a windows pro, so I would defer to someone with more experience.

17

Well I feel like there’s more stuttering but bench vs native was pretty much margin of error. This is still without hugepages. Maybe the bench is too easy/ out of date? But seems hugepages are uneccesary and I have perfect virtualization according to this bench.
Unigine 4k High
unigine heaven FPS 69|1750|8min|161MAX - virtual display QXL
unigine heaven FPS 70|1764|22min|159MAX - virtual display and adapter disabled QXL
unigine heaven FPS 70|1767|14min|158MAX - virtual display and adapter disabled Virt
unigine heaven FPS 71|1788|12min|153MAX - native

18

I don’t mess with the memory pages, myself.

19

No display out was fixed by switching from seabios to ovmf