How can I provide cgroups to a VM guest?

Operating Systems & Open Source Linux
1

I’m trying to get k3s working in VMs (host and guests f32 server).
k3s fails to start with:

warn: Failed to find cpuset cgroup, you may need to add \"cgroup_enable=cpuset\" to your linux cmdline (/boot/cmdline.txt on a Raspberry Pi)
warn and fatal: Failed to find memory cgroup, you may need to add \"cgroup_memory=1 cgroup_enable=memory\" to your linux cmdline (/boot/cmdline.txt on a Raspberry Pi)

I googled around, and have no idea on how to add the variables to the guest’s linux cmdline.

This could also be the host’s fault:

$ virt-host-validate 
  QEMU: Checking for hardware virtualization                                 : PASS
  QEMU: Checking if device /dev/kvm exists                                   : PASS
  QEMU: Checking if device /dev/kvm is accessible                            : PASS
  QEMU: Checking if device /dev/vhost-net exists                             : PASS
  QEMU: Checking if device /dev/net/tun exists                               : PASS
  QEMU: Checking for cgroup 'cpu' controller support                         : WARN (Enable 'cpu' in kernel Kconfig file or mount/enable cgroup controller in your system)
  QEMU: Checking for cgroup 'cpuacct' controller support                     : PASS
  QEMU: Checking for cgroup 'cpuset' controller support                      : WARN (Enable 'cpuset' in kernel Kconfig file or mount/enable cgroup controller in your system)
  QEMU: Checking for cgroup 'memory' controller support                      : PASS
  QEMU: Checking for cgroup 'devices' controller support                     : WARN (Enable 'devices' in kernel Kconfig file or mount/enable cgroup controller in your system)
  QEMU: Checking for cgroup 'blkio' controller support                       : WARN (Enable 'blkio' in kernel Kconfig file or mount/enable cgroup controller in your system)
  QEMU: Checking for device assignment IOMMU support                         : PASS
  QEMU: Checking if IOMMU is enabled by kernel                               : WARN (IOMMU appears to be disabled in kernel. Add intel_iommu=on to kernel cmdline arguments)
/-/

…but since the fatal part is the memory cgroup, and it passes that test, I’m not sure.

How can I provide the cgroups to the guest?

2

What is your guest OS in the VM?

3

Fedora 32 Server (Cloud, via virt-builder + virt-install).

4

Ah, missed that.

Cmdline arguments are normally passed by the bootloader. Fedora uses the GRUB bootloader, so you configure grub to boot the system with the required argument(s).

Edit the /etc/default/grub file in the VM.
Add cgroup_memory=1 cgroup_enable=cpuset to the line starting with GRUB_CMDLINE_LINUX
So something like GRUB_CMDLINE_LINUX="quiet cgroup_memory=1 cgroup_enable=cpuset".

Save the file, then run
UEFI: grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg
BIOS: grub2-mkconfig -o /boot/grub2/grub.cfg
to update the config. I think the VM would use UEFI, but it could be using BIOS.

See for more info: https://fedoraproject.org/wiki/GRUB_2

5

Modified on guest to:

GRUB_CMDLINE_LINUX="console=tty0 rd_NO_PLYMOUTH resume=UUID=f848575f-64bd-4651-984f-73d67f8768bc console=ttyS0,115200 cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory"

But to no success.

Is there any place for logs other than /var/log/boot.log to see if something is failing?

Maybe k3s is having a problem, how would one check if something from cgroups is enabled?

6

You also rebooted?

7

Is there any place for logs other than /var/log/boot.log to see if something is failing?

I’ve used journalctl. the -b flag gives logs from the most recent boot.

8

Multiple times with a slightly changed configuration, yes.

9

What is the output of cat /proc/cmdline

10

As expected:

BOOT_IMAGE=(hd0,gpt2)/vmlinuz-5.7.8-200.fc32.x86_64 root=UUID=a4e2e44b-d799-4ac8-8ba5-fd674f44da66 ro console=tty0 rd_NO_PLYMOUTH resume=UUID=f848575f-64bd-4651-984f-73d67f8768bc console=ttyS0,115200 group_enable=cpuset cgroup_memory=1 cgroup_enable=memory

journalctl -b indicates that the arguments got loaded, and I did not spot any errors. I did find one other relevant cgroup occurance, kernel: SELinux: policy capability cgroup_seclabel=1

11

I just remembered hearing something about fedora cgroups.

Here it is: https://www.redhat.com/sysadmin/fedora-31-control-group-v2

Looks like the correct thing to add to the cmdline is actually systemd.unified_cgroup_hierarchy=0

1 Like
12

I commend the change, but what even does use v2?

I now remember as well, have had the same problem with docker as well.