Return to Level1Techs.com

Any Tips For KVM Performance Improvements?


#1

I recently installed ubuntu as my daily driver. Even crazier, I’m proud to say I got KVM working with GPU passthrough.

However I’m still a KVM noob, and would like to squeeze out as much performance as possible. I doubt I’ll have true bare-metal speeds. But I’d still like to know what I can do to improve performance of the vm.

Any advice is appreciated.

  • Titanae

#2

https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#Performance_tuning


#3

post your current kvm config xml if you can;

and we can go from there. (also how familiar you are with nix?)


#5

Not very familiar. Though I’m willing to break anything if it means I get to learn what’s up.

I’ve also attached a pastebin of my current configuration. If you have time I would appreciate any advice on improving my system.

Two weird things I noticed:

  1. Running ‘wmic cpu get caption’ + cpuz I noticed my VM only recognizes 1 cpu core. I’m curious to know why that is.
  2. Since I managed to get GPU passthrough running, the display driver automatically defaults to GTX 1070. Not sure if it’s reading the device from the PCI bus or if this is a bug that I’m not aware of.

https://pastebin.com/60mjqX7i


#6

I would definitely recommend CPU pinning. @tkoham posted the Arch Wiki link to that. If you want an example or two just ask. This will help the VM a great deal.

You didn’t post specs, so I assume you aren’t having the dreaded Code 43 problem with Nvidia. If you were I’d suggest removing anything 'hyperv" related in the xml, and enabling the ‘hidden state.’ Again, just ask for an example. Probably wouldn’t help (or hurt) if you are passing through an AMD GPU, though.

Looks like you are using SATA as the bus for your VM image file. That’s fine, I found that to be perfectly adequate. I was told to have the VM image as an img file, instead of qcow or any other type. Having the file as raw or img will offer the best speed, as far as I can tell. May be too late to change that, though.

When you make the VM, make sure you use UEFI OVMF instead of BIOS, and use the Q35 chipset.

The last thing I know of is hugepages. I had that set up in my previous install. I couldn’t really tell any speed difference, to be honest. I probably won’t enable that this time, as I just recently reinstalled and set everything up again.

Have fun!


#7

So much to digest with this.

The one thing I’m confused about is the NVidia drivers. A few things I’m not 100% clear on:

  1. I’ve read that NVidia doesn’t play nice, hence needing to enable ‘hidden state’. That said, I haven’t seen a Code 43 yet.
  2. How vfio benefits me. I understand it passes the video card to the system, allowing it to act close to bare metal. But once the gpu is passed through… what then?

Also I would love to see an example if possible. I did run a few tests, and the system was fairly slow and sluggish. That could be because of the qcow format though. Oddly enough I’m reinstalling it under the Raw format. It’s definitely faster on install, though it feels oddly jumpy moving the mouse around.

Also (and I’m embarrassed to admit this) but I allowed for up to 8 cores… but only ever allocated 1. Oi vey.


#8

There’s a few things that need explaining, and I don’t want to overwhelm either you or me. We’ll start off with this:

In your VM, when you go to Device Manager, do you see the Nvidia card under Display Adapters? If you do, is there a little yellow icon next to it? If you don’t see the card, then either you didn’t add it in the VM settings, or your passthrough didn’t work.


#9

Little yellow tick next to the card.


#10

And if you double click on the card, you’ll probably see Code 43. Correct?


#11

Yes, there is a Code 43.


#12

Then this is the point where you will need to do the editing of the xml file to hide the fact that you are running Windows in a VM. We can also do the CPU pinning while we are in there if you want. What CPU is the host running on?

Shut down the VM, and on the host run sudo virsh edit [VM name]

I’ll get an example from my setup in a minute.


#13

:slight_smile:

Thank you


#14

Actually, before you run the virsh command, lets set and check a couple things in the VM settings in Virtual Machine Manager.

Under CPUs, allocate 4, then under Topology check the box and do 1 socket, 2 cores, 2 threads.

Under Overview does it say the Firmware is ‘UEFI…’, and the Chipset ‘Q35’?


#15

Firmware - Bios

Chipset i440FX


#16

Setting up a new VM with those settings you’ve recommended.


#17

You’re going to need those changed to UEFI and Q35. The only (easy) way that I know of is deleting the VM and making it again, but on the very last step check the box to make additional changes. You probably can do in in the xml, but I don’t really know if it will work properly.

You can add the VM image as your storage instead of making a new one.

Then just do the CPU stuff, and I’ll get the example pinning and hiding from my setup in a bit.


#18

One big one for Intel CPUs is adding nopti to your /etc/default/grub

It’s a tradeoff between security and performance, but it’s noticeable with NVMe SSDs on Intel when used in passthrough.


#19

Alright, so when you do virsh edit [vm name] you’ll see the xml config for you VM.

This is vi or vim, which I am definitely not an expert with. Basically you need to enter insert mode by pressing i. You press esc to exit insert mode. You press :wq to save and exit virsh.

Under the line <vcpu placement='static'>4</vcpu>, copy/paste this:

  <cputune>
    <vcpupin vcpu='0' cpuset='2'/>
    <vcpupin vcpu='1' cpuset='3'/>
    <vcpupin vcpu='2' cpuset='6'/>
    <vcpupin vcpu='3' cpuset='7'/>
  </cputune>

This may change based on what CPU you have. Mine is 4 core, 8 thread, so I give the VM two physical cores and their corresponding threads.

Then, for the hiding part, under features you should see a couple lines for acpi and apic. Under those add this:

    <kvm>
      <hidden state='on'/>
    </kvm>

You can also delete anything related to ‘hyperv’ and anything else that looks like it’s related to virtual machine stuff. The spot where you put in the above section has a hyperv section. Just delete that. There’s also a timer in the clock section that needs to be removed.

Also, back in the VM setup in Virtual Machine Manager under CPUs, make sure you have the model set to ‘host-passthrough’. You’ll probably have to type that in.

That should give you a good start. Any other questions just ask.


#20

I came across these ‘virtio’ driver for windows. Would you recommend using them or just leaving them alone?


#21

You can try it. You need to get the virtio driver image and add it to the VM as a CDROM. You’d use it for the main storage image that you install Windows to, so you’d have to set the bus as virtio before installing Windows and use the drivers at install time.

Speed is “supposed” to be better with virtio, but I just used SATA and left it. I didn’t notice any difference.