Protip: When creating new VMs using virt-manager, disable Hugepages

Software & Operating Systems Linux
1

In troubleshooting an issue with no video with the Q35 chipset, I found the solution and a big tip if you’re new to passthrough and are messing with Hugepages:

Hugepages pre-reserves memory. This gives the OS less user addressable memory, so when you make a new VM with virt-manager, it tries to query user memory, not Hugepages. If you set your hugepages to a high amount, and you try to make another VM that can’t fit inside the user addressable memory space, it will try to use swap, and that will time out the QEMU monitor.

So: ALWAYS DISENGAGE Hugepages before making a new VM with the virt-manager GUI.

Also, you can solve “no video” issues with Q35 by simply deleting the console and channel spice before installation.

I have regained some sanity after trying to figure out for hours why my kernel was freezing… CHECK YOUR HUGEPAGES.

1 Like
2

And what about using HugePages for VM?
https://wiki.archlinux.org/index.php/KVM#Enabling_huge_pages

3

I had it enabled, forgot about it, and was panicking about why my kernel was freezing making new VMs.

To be on the safe side, if it was on, turn it off via Grub when you’re about to create a new VM.

4

When I used to use HP on NUMA with dedicated PCIe 16x+die for VM, I realized 32GB RAM is not enough. :frowning:

5

Qemu and libvirt require hugepages to be explicitly defined for each machine with the startup arguments of qemu or inside the libvirt xml machine definition

<memoryBacking>
     <hugePages/>
</memoryBacking>

I’m not exactly sure how you managed to enable that by default nevermind in the GUI.
The virt-manager GUI doesn’t even have that option and it’s not there for a reason.

I can only imagine uncommenting this in libvirt/qemu.conf having an effect potentially.

image

But even then the guest has to explicitly request it. Which requires editing the xml or giving qemu the appropriate commands.

6

No, I wasn’t clear in the beginning. I had it enabled from previous VMs I was testing it with that I had properly setup. When going to make a new VM though, it ran out of user addressable RAM because Hugepages reserved it from my previous VM configs and I forgot to revert it.

1 Like
7

You can assign hugetblfs to multiple VM’s dynamically at runtime as well via qemu hooks (scripts that run when creating/destroying/changing VM’s).

https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#Dynamic_huge_pages

I’m not sure where the script is right now, but I know it exists.

1 Like
8

Yeah, virt-manager I don’t believe has this functionality. It’s as dumb as bricks when it comes to setting up hugepages. I had to directly edit the XML to enable it for a VM, and forgetting to undo the Grub entry meant virt-manager was trying to use standard unreserved memory.

9

So are you trying to say that a vm that was powered off had allocated all your HugePages?

Because that’s not how it works. I’ve never encountered this problem on any of the nodes I work on in OpenStack clusters, or at home.

10

No, I added it as a Grub argument to the host to pre-reserve the memory and forgot about it when going to make a new VM.

I had hugepages=8192 in my Grub kernel boot command line and forgot to take it off.

11

Ah, yeah. I allocate them after I boot. I just close down all applications and run a script that allocates 2m pages until I get the number I want.