Proxmox 5.1 AppArmor updated, LXC containers now will not start

Software & Operating Systems Linux
1

Hi all,

Yesterday i updated the packages on my Proxmox 5.1 node. Everything appeared to run fine, but this morning i noticed my Owncloud client wouldn’t connect. So i logged in and found my Ubuntu 16.04 container that runs Owncloud was off. Looking through the logs it showed that the Owncloud container turned off and backed up to the NFS share normally, but could not restart after the backup. I tried to start it via the GUI and via lxc-start -n 103 -F but nada. So i rebooted the node. Now all of my containers cannot start because rebooting shut them all off (of course).

The output when trying to start container 103 is:

sudo lxc-start -n 103 -F
lxc-start: 103: lxccontainer.c: do_lxcapi_start: 984 Permission denied - Failed to make / rslave at start up
lxc-start: 103: tools/lxc_start.c: main: 371 container failed to start.
lxc-start: 103: tools/lxc_start.c: main: 375 Additional information can be obtained by setting the --logfile and --logpriority options

i ran “lxe-checkconfig 103” and the output of it came back all green, no apparent errors.

So next i used journalctl -xe and found this little nugget in the logs:

node1 audit[5540]: AVC apparmor="DENIED" operation="mount" info-"failed mntpnt match" error=-13 profile="/usr/bin/lxc-start" name="/" pid=5540 comm="lxc-start" flags="rw, rslave"

So it seems apparmor is not letting the container mount it’s filesystems because of permissions. So i pull the update logs using

awk '$1=="2018-03-15" && $3=="upgrade"' /var/log/dpkg.log

and i confirmed that yes AppArmor update to 2.11.0-3+deb9u2 when i did the package update
(i pulled that command from: https://superuser.com/questions/38717/how-can-i-undo-or-rollback-an-apt-get-upgrade-command-on-ubuntu )

So i try to pull the previous version of the AppArmor package from the local cache of packages on the node but it’s already gone.

So now is the point where i’m at a brickwall. I don’t know how to manipulate profiles or permissions regarding apparmor, and i can’t find any resources on how to roll back the service to a previous version.

Help?

2

Bump? Anyone?

3

Looks like this has happened with Proxmox before. You should be able to downgrade the culprit packages from the online repos if they’re no longer available in your cache. What output do you get with the following?

awk '$1=="2018-03-15" && $3=="upgrade" && $6=="2.11.0-3+deb9u2"' /var/log/dpkg.log

Edit: If you get no results, your dpkg.log may have rotated, so…

ls -al /var/log/dpkg*

…and try awk’ing against any older versions (e.g., /var/log/dpkg.log.1)

4

Output from:
ls -al /var/log/dpkg*
is:
/var/log/dpkg.log
/var/log/dpkg.log.1

using your awk command on either file produces no results.

I did find the article you linked but running the commands shown in that article give no result.

5

I must have mistyped the commands in that article the first go round. Now the output for

sudo apt install $(dpkg-query --show -f '${Package},${Version}\n' | grep '2.11.0-3+deb9u2$' | sed =e 's/,.*$/=2.11.0-3/' | tr '\n' ' ')

just gives

Reading Package lists... done
Building depency tree
Reading state information... done
E: Version '2.11.0-3' for 'apparmor' was not found
E: Version '2.11.0-3' for 'libapparmor-perl' was not found
E: Version '2.11.0-3' for 'libapparmor1' was not found

running dpkg-query --show apparmor* just gives

apparmor       2.11.0-3+deb9u2
apparmor -profiles
apparmor-profiles-extra
apparmor-utils
libapparmor-perl      2.11.0-3+deb9u2
libapparmor1:amd64      2.11.0-3+deb9u2

No other versions are shown.

6

I was having the same problem. Short version: I added the Proxmox “No Subscription” repository and updated Proxmox. (Instructions at https://pve.proxmox.com/wiki/Package_Repositories).
Long version: Edit “/etc/apt/sources.list”.
Add line “deb http://download.proxmox.com/debian/pve stretch pve-no-subscription”.
Run “apt update” followed by “apt upgrade”.
That updated a few things. I think the important package was “pve-kernel-4.13.13-2-pve”.
After that, my LXC containers would start and run again.

3 Likes
7

You are the f*****g Man! Wish i could buy you a beer mate. I completely forgot after i upgrade from 4.5 to 5.1 i never added that repo back to the sources list. Never even occurred to me to check. Containers are all starting up now.

@Eden This topic can be marked closed now