As possibly a lot of you, I’ve been very much interested in the TrueNAS migration to linux with/due to docker/kubernetes and KVM. As soon as TrueNAS Scale hit the public beta stage, I’ve had it installed. Everything seems kind of usable but there lies a not-so-often-raised issue of “ix-applications” dataset snapshots. Over less than a year, with very modest/limited use of TrueNAS Scale for homelab/home-media-server purposes TrueNAS/TrueCharts/docker/kubernetes has accumulated 47 thousand snapshots You read that right - 47502 to be exact as of yesterday. A colleague of mine has over 300 snapshots created since the installation about a week ago. These snapshots are not scheduled or managed by the users. No, these are coming from applications and their PVs as managed/arranged by TrueNAS containers/applications, as all of these are for “ix-applications” dataset (and it’s subdatasets) that is being created upon storage pool selection for Kubernetes/Docker storage backing pool. A lot of these cannot even be removed in an easy way from the UI - often these are referenced a lot or have clones blocking removals. There’s no automated housekeeping option to clear it all up, to prune them or squash them, and to maintain clear state.
I think this is also why there are users who report lowered performance over time - in my case I can see that nearly all storage-related actions take long. I mean pool importing on boot takes hours, nearly days. Accessing/checking ZFS storage details, let alone snapshots checks, takes dozens of minutes.
This issue gets even more exacerbated when your storage pool consists of spinning rust.
So yeah, I think that this lack of housekeeping/cleanup functionality for ix-applications dataset, may impact more and more users as the product reached Release stage over a month ago.
I bet we have quite a few TrueNAS Scale users here, so perhaps someone has already come up with a tested and working solution. Or maybe someone knows how to deal with these in a safe manner…
I was considering 2 projects:
but since I haven’t used either and the first one was devised for older TrueNAS Core (due to its missing working data datasets’ snapshots management at that time) while the other is a rather general ZFS, not TrueNAS Scale ix-applications specific, support solution, I wanted to ask here for your opinions/solutions etc. to the problem.
I’ve asked about it in TrueNAS Scale forum, but I might not get a lot of help there. Apparently, the snapshots creation is simply part of how TrueNAS Scale’s kubernetes/docker deals with deploying and keeping apps working. There is no extra logic to eg. clean old snapshots there before creating new ones, no pruning or squashing, or limiting. Sure, overall they’re not taking a lot space on the storage, but they’re impacting storage subsystem performance. It’s like with inodes and bytes used, once you get very high with either your storage performance will suffer regardless whichever it is. ZFS by default reports datasets should not exceed 512 snapshots per dataset for optimum performance, but it seems that TrueNAS Scale implementation simply creates new ones, references old ones or clones them for space optimisation using ZFS underlying tech, and easily breaches this lofty snapshots count within well under a fortnight.
auto-date, all of these are exactly two weeks old and within the timeframe for usual snapshot lifetime
Around the same amount of Snapshots with a non-dated ID, which are all older than two weeks and seemingly the main clog.
For now this is just some observations, I’ll try to look through logs a bit tomorrow.
I’d also like to hear whether you get a reply over at iXSystems, especially if its from someone official.
“auto-” ones are the ones that are created by the Data Protection automatas - you must have pool-wide Snapshots config. These are managed/created/deleted by the automata, and they are easy to remove manually. Or modify. By default there are 14-days daily snapshots added if you enable data protection for the whole pool.
The problem lies with the others, in docker and those other subpaths/subdatasets - once you start to use the device the amount of these grows drastically. And any OS updates seem to make automatic snapshots of all snapshots/datasets with “backup-/update-” naming pattern
Regardless, I am yet to get any solution/answer from IX or TrueNAS Scale team.
I have only found some of the descriptions in some older discussions where “it just works like that” comments from TrueCharts teams were offered with a bit more thorough explanation.
I bet it’d be better if the solution was offered for them to test and implement/add fast/soon. though what I have found are two scripts as mentioned above, neither written for the Scale situation, nor vetted/tested by TrueNAS Scale team.
Yeah, that’s where after some time you might be getting drastically reduced performance and stability. Once any dataset gets over 512 snaps or the total amount gets really high, the storage layer actions start to be hit. Once you reach tens of thousands on a typical NAS or small server hardware, you’re going to feel the reactivity/snappiness loss. It seems noone speaks of the issue anywhere… So yeah, TrueNAS Scale is good, having the option to play/work with docker containers and kubernetes pods and KVM on a storage oriented solution fits a lot of bills, but if you risk getting the machine become sluggish just by having those apps installed it becomes a negative. Especially now since the product is in release version where such “quirks” should have been ironed out and fixed.
Another potentially related issue with the snapshots created automatically in TrueNAS Scale is of apps not starting due to snapshots inaccessible after reboot reported by others.
I’ve had my issue with snapshots described in this TrueNAS forum thread I’ve created where you can already see other users chiming in, but without proper vetted/safe/easy solution.
I’ve also raised this under another thread of mine to discuss this issue and where the impacts start to be felt first before they become very apparent and impactful. I mean, I could understand multi-hour startup/boot procedure if the machine was not stopped safely and there were filesystem issues to be automatically scanned and fixed or the pool needed resilvering on multiterabyte (as in used space) filesystems/pools, but that’s what I got upon system update when everything should have clean stop/shutdown/reboot.
Perhaps there are many more users impacted by this but they’re simply taking this as normal behaviour of older/weaker “storage” oriented home/homelab systems running ZFS and kubernetes and with clustering abilities. But if people don’t speak out, TrueNAS team might not even be aware how impactful this “feature” they have is after the first few days/weeks after install when most media outlets or users report on issues/behaviour.
On the other hand, I guess Wendell since you’ve done a few things with TrueNAS Scale already, perhaps you can find a solution for them and all the communities Or at least perhaps you can make it more public knowledge as part of the Level1Linux or Level1Techs channels? Cause you know, Milan-X is great but not many people will be able to afford such platforms for eg. homelab/SMB solutions, and the storage component is often delegated to platforms such as TrueNAS - the easy to use, great WebUI, tried and tested, free or cheap, open source based solutions are prevalent in those markets.
On yet another hand, maybe we have users here who have already tested/fixed this issue on their deployments and they could share their solution for all the communities to know and maybe TrueNAS even consider adding/implementing similar within the UI.
I do, but these are mostly empty Snapshots, my tasks are configured not to take those.
Which works well for every other directory except for the two snapshot locations I listed.
It seems like most other people with the issue are having it much worse than me (I’m steadily at around 500 Snapshots over the span of two pools with two week snapshot lifetime and at least 200 of those are legitimate).
Which makes me think that the fact I run my apps manually via the “Launch Docker” Dialogue and only one official Chart might be the reason for this.
Let’s just have a lot of folks voting for the JIRA ticket and maybe the devs provide a solution soon.
Otherwise, maybe someone with a test server can spare some free time and, in case some commands brick TrueNAS Scale, test and report here. I don’t have, unfortunately, anymore a “running” test server, as I had to finalise move to production and send it out to its new home… The testing is important cause I think, as in that’s my impression, some ix-applications snaps and related filesystems are used in conjunction with TrueNAS Scale app rollback features (that’s more or less obvious, but since the UI and middleware may also be using some containers under the hood they may also be impacted by force removal of snaps from CLI) and more importantly boot environments to have the working state of the applications datasets from respective installs/upgrades.
And as a side topic - I wonder if QNAP’s ZFS-based solution also suffers from similar snapshotting issue, or maybe have they devised some solution to manage/keep it in check? I don’t have QTS-hero or QTS-enterprise devices on my hands to check, nor do I even know where to look for any housekeeping scripts possibly deployed in those platforms…
Just FYI, while the sheer number of snapshots issue is not fixed in another not-mine Scale thread, the reporting person suggested that, for obvious reasons, moving of ix-applications to an SSD pool allows far better performance of the storage layer for Kubernetes - that’s obvious, but that was at the Alpha stages, possibly with far fewer snapshots existing on the machine. Possibly once these hit huge numbers the impact will be felt, only on spinners it’s far more exacerbated.
While it’s not an official solution, given how some SCALE users eg. suggested going for docker image prune --all --force --filter "until=24h" once in a while to “manually” clean the system of any non-needed images, and some reported some success in improving the situation just from running docker container prune --force --filter "until=24h" once in a while (though if you want to combine both you’d need to first prune containers and then images, and then run docker volume prune --force cause volumes do not know the until filter), I’ve went through with docker specific commands…
I’ve used docker system prune --all --force --volumes (unfortunately if I want to clean volumes as well I cannot use until filter) in a daily cron job (though I think I’ll move it to something like a weekly or monthly job) - sure it’s a workaround, it’s rough around the edges, it’s not allowing a combined volumes and date filters, and most importantly it somewhat “breaks” the application’s instant rollbacking capability (seems like TrueNAS connects “previous entries” for the app with their on filesystem sets of snapshots/clones that are removed upon docker prune), but I can kind of live with that as I’m testing app (re)deployments usually between 16:00 and 24:00 and have the daily prune run set for 03:00. Sure I could combine all those domain specific docker prunes as well but I decided to go with a complete clean-up instead.
In effect:
all the stopped/finished containers upon run have been removed/deleted and now the app restarts for the remaining active containers/pods have greatly improved in snappiness (back to how it was) - overall reported container count dropped from 2k+ to nearly 100…
thousands upon thousands of snaps, clones and filesystems/volumes have been removed along with the containers - i’m from 46.6k down to 0.6k of snaps, and in storage terms that’s nearly 100GB freed…
hundreds of hanging/older versions of images have been deleted - i’m down from way over 2k to less than 20 now…
network routing to cluster and through, to and from respective containers has also improved…
docker build caches have dramatically reduced the storage aspect - down from over 3GB used to less than 1GB…
my CPU now reports under 10% on idle (consider idle as no active compute from running containers) with temps then quickly dropping at idle times to 40C - previously even on idle CPU in my server was hovering around 70% usage with temps at similar number though in degs C…
Overall, I think I’m going to become a bit happier user with this primitive workaround. But since docker support’s approach is this is by design behaviour to enable docker volumes/layers analysis on failing/broken containers (which I honestly did and still do use a lot upon building/testing my own docker images with my own docker repository) and any maintenance is to be organised elsewhere, and TrueNAS team’s approach is this is docker dogmatic issue that it’s not properly cleaning after itself in the long run, I think this will do for now until better solution is devised/offered in some future version of TrueNAS (as in periodic housekeeping of any trash docker/kubernetes leftovers).
I’ve also replaced all the aliases for docker run with docker run --rm for any stable docker image/container for my users (to reduce stopped/finished but hanging in docker lists containers counts), and left the regular not clean after self on fail docker run command for the small subset of build/test deployments for debug purposes. Hopefully my set of workarounds will help others.
Please bear in mind that this workaround “clears” anything docker created for non-currently running containers, so if you have some stopped app that you start only now and then you need to have it running when the prune command analyses the containers, otherwise the container/image/snaps/volumes/networks created by docker will get purged. I currently have only 2 small docker images that I build from scratch/source in my CI/CD pipeline for a one time action docker app that are stopped, other apps are constantly running/in use, so this solution works for me. But it might not for you…
Just FYI, this is not a complete solution, the result is still about 500 snaps remaining, some of which are the the ones from the Data protection feature - so the automated daily snaps that are automatically cleared. There are still droves of snaps taken seemingly upon reboot/kubernetes restart that are non-removable due to snapshot has dependent clones returned message on attempting of their zfs destroy runs by hand (these are on <pool>/ix-applications/<docker>/<dockerimagelayerfilesystem>), and multiples are even removable (these are on <pool>/ix-applications/release/<appname>/<numberedversion> subdataset), both of which are snaps taken by docker/kubernetes for apps in the environment. The latter of which with their contents refering to some specific versions of the app deployments (historically) in use on the machine/server (I’ve had more than 50 release snaps for some of the apps deployed) - these are not in use per se, but are “recipes/snaps” for specific application deployment rollbacks as used in Installed Applications → App → Rollback functionality, if I recall/understand correctly. I’m not sure which docker/kubernetes mechanism manages this, but over time even only a handful of running apps will grow the number of these 2 types of snaps to over a hundred or easily over 500. Sure it’s still a reduction thanks to the daily docker prune runs compared to multiples of thousands before, but these are hundreds of snaps not taken by user managed/deployed automatas but instead these are snaps taken for the Kubernetes/docker environment by those tools.
Perhaps in future TrueNAS Scale dev team will offer some intelligent cleaning utility to take care of these in a smart manner automatically or give an option to trim creation of these in app deployment forms. Sure, they’re not taking a lot space on the storage per se thanks to the snapshotting nature, but these are still hundreds of snapshots that should be better managed by the system.
