VM cannot allocate memory anymore

Software & Operating Systems VFIO
1

Lately my VM has stopped running, when i try to start it says it cannot allocate memory (it was working before)

Error starting domain: internal error: qemu unexpectedly closed the monitor: 2019-07-18T11:51:10.814566Z qemu-system-x86_64: unable to map backing store for guest RAM: Cannot allocate memory

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 75, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 111, in tmpcb
    callback(*args, **kwargs)
  File "/usr/share/virt-manager/virtManager/libvirtobject.py", line 66, in newfn
    ret = fn(self, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/domain.py", line 1400, in startup
    self._backend.create()
  File "/usr/lib/python3/dist-packages/libvirt.py", line 1080, in create
    if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2019-07-18T11:51:10.814566Z qemu-system-x86_64: unable to map backing store for guest RAM: Cannot allocate memory

the VM is set tu use 16gb and i definetly have the free ram for it

free -mh
              total        used        free      shared  buff/cache   available
Mem:          125Gi        22Gi        95Gi       625Mi       7.3Gi       101Gi
Swap:         4.0Gi          0B       4.0Gi

this is my ram settings in the VM

  <memory unit='KiB'>16777216</memory>
  <currentMemory unit='KiB'>16777216</currentMemory>
  <memoryBacking>
    <hugepages/>
  </memoryBacking>

im running Pop!OS

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=19.04
DISTRIB_CODENAME=disco
DISTRIB_DESCRIPTION="Pop!_OS 19.04"

kernel 5.0.0-21-generic one thing is kind of weird is that if i lower the ram by a gb the VM does start.

2

definetly related to hugepages, not sure whats going on, it was also making the VM very unstable windows was blue screening a lot.

3

Have you allocated 16GB of static hugepages?

What’s your hardware setup?

4

im running a 2990wx with 128GB of ram 1 2080 for the host and a 2080ti for the VM.

this is my hugepage setup

vm.nr_hugepages=8192
vm.hugetlb_shm_group=48
5

output of cat /proc/meminfo | grep Huge?

Also, drop hugetlb_shm_group. You shouldn’t need it.

6 
cat /proc/meminfo | grep Huge
AnonHugePages:  16746496 kB
ShmemHugePages:        0 kB
HugePages_Total:    8192
HugePages_Free:     8185
HugePages_Rsvd:       64
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:        16777216 kB
7

Are you running any containers or Java applications?

They tend to snag a few hugepages.

HugePages_Total:    8400
HugePages_Free:      177
HugePages_Rsvd:      175

I set min up like above ^^ and it works better for me that way.

8

no java, tons of containers, probably 20 to 40.

will try that out thanks for the help!!

9

That’s 100% the cause.

Let me know if it doesn’t help!