I've finally moved away from TrueNAS to pure Proxmox (sort of)

Hardware Hub Storage
, , ,
1

Over the last few months, I’ve become concerned about TrueNAS as a product. While it does have some benefits, I can’t help but notice that it has started to feel more like a closed ecosystem rather than something community oriented. What really bothers me most is that the TrueNAS build process has become opaque and cannot easily be audited by the community. Maybe I’m just paranoid, but I would prefer not to risk having something unexpected slip into my storage VM unnoticed.

Regarding my original setup, TrueNAS ran inside a VM with a PCIe SATA card passed through. While I don’t have that much data, I like using it as shared storage between my virtual machines via NFS or SSHFS. My services run in Podman containers (via quadlets) locally, and all data is then backed up to TrueNAS via rsync over SSH. When I first started I stored everything on NFS but the resource usage was too high so I ended up opting for a simpler solution. The only service that still uses NFS directly is Jellyfin; this is solely because movies and TV shows consume significant amounts of storage space.

To leave TrueNAS behind, I needed something to replace it. My original plan was simply to stand up a Rocky Linux or Debian VM running ZFS with my PCIe card passed through. However, I wanted to mix things up so I opted for a different route: instead of running separate ZFS instances inside guests, I moved my storage pool directly into Proxmox. To run storage services, I have a Rocky Linux VM that has a seperate Zvol formatted as XFS. XFS on ZFS does introduce some overhead but it still but lighter than running TrueNAS in a VM. Overall I’m pretty happy with this setup and I figured I should post it somewhere. In the future I’m looking to introduce some automation into the mix using Forgejo actions but for now everything is still rather manual.

1 Like
2

Sounds fun!

I have a Rocky Linux VM that has a seperate Zvol formatted as XFS.

You could run this as an LXC container instead of a VM, that way you can bind mount the ZFS mount into the container directly.

Also Proxmox is just Debian, so you can easily install other services alongside without issue. If LXC containers don’t work for you, you can also do podman / docker containers (systemd quadlet works well).

3

That was my though initially but NFS can’t easily run inside a unprivileged container

4

Ditto. But my primary reason for moving off of TrueNAS were the breaking upgrades. Many of them. Many many of them. From docker to k3s back to docker, from apps migration “deadlines?!” to enforcing smartdrv breaking spindown, many many breakages for what should be just simple upgrades.

This is precisely my setup as well on Proxmox (shout-out to Jeff @ craft computing).

I also ran multiple TrueNAS instances. Think data only, another for public apps only, another for security apps, etc.

The security concept is that I treat all of these “apps” as untrusted, exploitable apps. So I severely limit their access to other apps data (ACLs) while isolating them to monitored public VLANs/networks.

“When, not if, an app is exploited and unauthorized access is granted to the apps namespace, what access does that app have to other apps, data, shares, etc?”

i landed on NFSv4 and strict ACL on a per app, per UID setup to isolate everything. PITA really.

This approach saved my ass about 17 years ago when uTorrent that had a backdoor/exploit and it got ransomewared on a Windows box (I used the same deny by default back then). Only the app and it’s data/config files got ransomewared though, as it didn’t have any other access under it’s Deny by Default file permissions on that box. Torrents were moved off by a background services as soon as they completed, and then reshared as read-only access back to uTorrent. So it had RO access to my Linux ISOs, but nothing else than the apps own configuration. And none of them got ransomwared either.

So I carry that philosophy and mindset through to today.

I’ve already migrated my apps off to simple LXCs running docker-in-docker, now with compose files and Portainer managing it all. Simple, reliable, upgradable, etc. Same ACL structure over NFS.

What I’m struggling with now is letting go of the point-n-click TrueNAS data plane of ZFS tooling, scrubbing, scheduling, backups, etc. Going to Proxmox as the data layer means I have to manage all that manually.

I’ve started down an Ansible path for this, as a checks and balances approach I use for clients. It works well, but I miss the webui click-and-done approach of new apps. I know, more Ansible, more modules, more config. But it’s turning into a coding chore more and more than I have time for.

Then there’s the whole Dashboard/holistic view of statuses and active monitoring and alerts project to work on, while balancing spindown. Ugh.

5

I advise against this. If you treat the Proxmox host as an appliance that needs to be supported by the manufacturer (e.g. Proxmox GmbH), and never touch anything on the system outside of the GUI, then you will have a very clean and simple upgrades path. Many many upgrade paths.

I’ve been managing Proxmox clusters for 10+ years for clients and homelab. The many upgrades do take a toll, and custom packages and services overly complicates the upgrades - if things don’t break the upgrade path entirely.

My advice is to leave the host alone: if it doesn’t exist in the webui, then the upgrade scripts don’t know about it. Put any custom services or packages you want into a LXC/VM. The host doesn’t need anything but Nvidia drivers - and that’s only warranted if you have to share the vGPU across multiple LXC/VMs (this is why I pick AMD over Nvidia personally - AMD is built into the kernel). Otherwise, just pass the GPU device into the VM or LXC and manage the drivers there.

The point is to treat the host as a pet, but any LXC/VM as a ephemeral that can be thrown away and rebuilt.

I have a few instances that started from Proxmox 4.x (10+ years ago), that have gone through every upgrade cycle to current 9.x and are continued to be supported, without issues.

2 Likes
6

I concur with @eduncan911 it’s best to keep your Proxmox host as vanilla as you can for a plethora of reasons.
It’s a few clicks to restore a VM or LXC backup if a package or bad update broke something, it’s a lot more annoying work to have to restore the host from backup or clean install because the same happend.
I’m usually oké with diagnostic packages, like htop and I believe lsscsi are not installed from iso but I require them sometimes.
Opinions my differ where the line is, but it’s commonly agreed the less you mess with the host the more stable it will run.

As for a Truenas alternative if you havent heard of Openmediavault yet it might be interesting NAS appliance to check.
It’s not the most flashy in NAS operating systems, but it’s turnkey and comes with a webui plus just about all features you can ask for in a NAS.
Also there is a official plugin for ZFS support.
Alternatively I believe 45drives uses cockpit with a few plugins for their webui.
https://cockpit-project.org/ Should run on all Linux flavors and most RedHat based Linux like Fedora or Almalinux usually have it already included.
It also works in a LXC container for a really lean install.

1 Like
7

The big advantage (ab)using Proxmox as a NAS OS has is that the installer can set up a ZFS mirror for the boot drives, takes a lot more fiddling with the alternatives unless something has changed in recent years.

1 Like
8

You’re right Proxmox installer can make ZFS mirror’s for boot drives.
The main reason you see so little ZFS in general Linux and even less in Linux installers is because ZFS and Linux have both different opensource licenses and conflict on some parts.
Thad’s why ZFS is not part of the main Linux kernel and has to be installed usually as a separate package.
Making it unavailable in most Linux installation media, developers need to manually add ZFS to their installation media to make it available as option during installation. (I’ll spare you the licensing challenges it might create)

And remember raid is not a backup, so I’d still advise all services being hosted in a VM’s and/or containers.