I’ve been using Proxmox recently after switching from command line LXD/LXC containers on Ubuntu.
Adding a NAS to the mix.
I just assumed I would virtualize TrueNAS Scale on a Proxmox node that had the requisite hardware for disks/etc.
Someone here commented on that assumption that I should consider using TrueNAS Scale AS my hypervisor on at least that node.
Can someone share the pros/cons of such a decision?
I had been assuming TNS on Proxmox because I like to generally manage my nodes and specifically my backups using the built in backup solutions and management features.
Do I gain or lose anything material in the options?
not true. KVM in Scale and beehive in Core allow for fully fledged VMs.
I had TrueNAS Core as a VM in Proxmox but eventually streamlined it back to Proxmox managing my ZFS pool. With SMB/NFS containers for filesharing and not having to passthrough disks and controllers to then share it back to Proxmox for VMs, was a big drop in complexity and easier administration.
Running a hypervisor with ZFS inside a hypervisor with ZFS just doesn’t make sense for me anymore. TrueNAS GUI is good, but CLI and Cockpit GUI is plenty for my storage and share needs and feels like a more clean approach.
Also results in less memory fragmentation and redundant caches which is always a thing with both Proxmox and the VM caching stuff. I’ve seen less memory usage because of no double/triple caching of the same stuff.
Proxmox runs ZFS out of the box. Use it. If you need more GUI than Proxmox can offer (likely), try Cockpit.
Or just run TrueNAS Scale on bare metal, it is built for doing all that stuff.
They both run Debian kernel and KVM under the hood…so they’re very similar. TrueNAS has focus on storage and docker/kubernetes and gluster cluster, Proxmox is mainly a hypervisor with strong storage options like ZFS , Ceph (emphasis on RBD) and a dedicated backup server , LXC containers and clustering via Corosync (and/or Ceph).
I run Proxmox using ZFS for its OS and for the VM images but when it comes to the SATA disk farm for TruNAS Scale I simply pass the entire sata controller in full through to the TruNAS instance. I suspect that is nearly native performance.
I have not passed the network card through so there is some hypervisor sharing going on there.
you don’t run ZFS inside of ZFS when it comes to TrueNAS. At max you have TrueNAS’s boot drive on Proxmox’s ZFS and thats it. Everything else is being passed-through into TrueNAS : HBA, NVMes
So there is 0 of double-caching or fragmentation. You’re just misleading people here.
TrueNAS is a NAS primarily and using it for KVM is just using wrong tools to do the job.
I think the point was that you already have ZFS available in your hypervisor OS; you don’t need an additional NAS OS.
As for caching: you are better off with one ARC for all data than separate ARCs for different parts of the data, since then ZFS can do a better job figuring out what data to discard/prioritize. (I.e. fragmentation.)
Also, although the ARCs won’t contain the same data, you will still likely have multiple caches of the same data between e.g. the ARC on the NAS VM and the filesystem cache for your VM storage shared back to the hypervisor. (I.e. double-caching.)
I got the point, but as I said above you are better off using right tools for right job.
Just because Linux supports ZFS,SMB,KVM,NFS out of the box, doesnt mean that you have to run everything inside of a single OS with crap tooling around to support it.
TrueNAS has built great tooling to manage the storage, ACLs and to export this storage to outside world.
Can you do this inside of any Linux OS ? sure you can, with lots of command line, tweaking, random github repos and so on.
Yeah TrueNAS is easy and point and click and stuff. It’s super nice until you realize that it REALLY doesn’t want you to treat it like a distro. (Literally no apt. Sad.) If this doesn’t matter to you (and you’re sure it never will) then by all means, it’s a good product.
On the other hand, I never found ZoL to be too much work. And in $currentyear you can ask any halfway competent LLM for the correct command line if you’re trying to do things that you don’t remember or know how.
TrueNAS is a great NAS and a not so great Hypervisor
Proxmox is a great Hypervisor and a not so great NAS
NAS and Hypervisor have different hardware needs
and the most important point
Unless you really don’t have to space to have two systems, please don’t try to get jack of all trades device.
It will perform worse, probably be more expensive, it will for certain be more complex, prone to fail and harder to troubleshoot. If you still need an all in one system, UNRAID is probably a better fit.
Yeah, yeah I know, someone will come and boost about my generalization is not true because his/her specific niche case and his/her bifurification ready mobo and NIC blablabla will work great with TrueNAS inside Proxmox. I think, if you have to ask about running TrueNAS inside Proxmox, you probably shouldn’t do it
I mean, if you forego the GUIs and use a basic linux server distro with ZFS and libvirt installed, it really doesn’t get much simpler. No complex, buggy, and functionally limited GUIs to contend with! There’s certainly a steeper learning curve at the beginning though, but especially for a NAS I feel that you really should know what you’re doing anyway.
And as for hardware needs I find that the two functions complement each other nicely: a NAS needs IO but not much CPU; a hypervisor needs CPU but not that much IO. Both needs memory but cache is flexible so more can be used when VM memory pressure is low and vice versa. And with the already limited IO on consumer HW it’s nice to be able to completely forego the NICs that would be needed if you run the NAS and the hypervisor on different machines.
If you’re talking about data center sized generalized storage and compute then it’s a different matter of course, but for a home lab I find that a “converged” setup works great for me at least!
It really boils down to what performance you need.
I am a little bit out of the loop when it comes to new homelab hardware, and your needs may differ. Here is what I want from my systems.
CPU performance isn’t important.
NAS needs 8 SATA ports for RAIDZ, 2 SATA ports for special vdev, and something for boot.
Hypervisor at least two fast NVME SSDs.
Both fast NICs.
Hypervisor needs iGPU or GPU.
Both need ECC RAM.
Getting 128GB of RAM two times for two system is easier and cheaper than getting 256GB for one.
Cheapest option was two cheap Supermicro boards instead of one board that supports all of that in one mobo.
Yeah, my requirements are more suited to consumer hardware:
I need some reasonable CPU power for running a few game servers. Nothing an AM4 APU can’t handle.
32 GiB of ECC RAM has been plenty/enough.
4 SATA. (Would like more since this feels a bit crowded, but it has been enough so far.)
2x NVMe.
Space for a fast NIC (e.g. PCIe 3.0 x8). Although only using 2.5 GbE for now.
Solution: Asus ProArt B550-CREATOR + Ryzen PRO 4650G.
Since the APU supports x8x4x4 bifurcation and the motherboard has two slots split x8/x8 plus a CPU-connected M.2 slot this allows me to run up to 3 CPU-connected NVMes plus one CPU-connected x8 network card.
The MB also has 4 SATA ports and a chipset-connected M.2 slot where I could hook up one of those 6-port M.2 to SATA controllers - or an additional NVMe.
I could also get up to 128 GiB of ECC ram if I really wanted to (or actually needed it ).
It’s really quite powerful compared to what I need.
I pass my SATA controller through to the TrueNAS VM on Proxmox and everything works very well, so well in fact if I power down the TrueNAS VM the disks shut down! Highly recommend the pass-through controller technique. It really stays out of the way.
Overall I’m disappointed by TrueNAS. I was expecting a NAS-focused OS to manage drives with sensibility by default, with under the hood access to tweaks for the pros… I thought it could grow with my proficiency.
What I’ve found so far is a lot of “you should’ve known” rationale. I should’ve known it won’t monitor disk health with SMART by default? That’s absurd. A 20 year old HP laptop does that. Or at the very least ask during setup.
Are you really disappointed by TrueNAS because of that? Or is there something else?
If it is just about SMART, don’t worry. You won’t need it.
Fortunately TrueNAS runs by default a scrub once a month.
If something is wrong with your drive, it will act up there.
From my anecdotal evidence, scrub is even more sufficient in finding bad drives than SMART is.
Many times I had drives throwing ZFS errors while SMART showed no errors or only during an offline test.
Don’t get me wrong, I would still enable SMART. Scrubs and SMART are two very different things. But it also is not a problem if you don’t have it enabled, that is what redundancy is for.
While we are at this topic, more important than SMART would be an e-mail alert that is also not enabled by default. Otherwise you end up like LinusTechTips, because while there have been SMART and scrub errors, he never went into the system to see them
I was going to rant about the lack of notifications too, but i searched it before I said it and found there is a “Slack” integration - and Discord can accept “Slack style” webhooks for channels.
They should do more work to highlight that capability, call it Discord directly, etc… there’s a feature request from over 6 months ago just to allow general configurable webhooks support but they seem a bit resistant. Seems like an easy win, it’s already in there for Slack.
You are speaking from a homelabber perspective.
TrueNAS is a business product.
Most customers will use SNMP.
Switching to SCALE and the “all in one” approach brought in a whole slew of home users with different expectations. Still think this wasn’t the right call.
As long as you don’t use the terminal… then the locked-down nature of TrueNAS will be a hugee limiting factor.
^ this
It shouldn’t have made sense to anyone to begin with. I’ve long been the “hipster” to tell people to manage ZFS in proxmox and if they need NFS or SMB, they can use proxmox itself.
You realize that TrueNAS Scale uses the same tooling that exists in Linux, don’t you? It’s just a GUI for already existing tools. Same with TrueNAS Core, but that’s based on FreeBSD and its utilities.
You don’t need any particular tweaking when it comes to ZFS, SMB or NFS. Since it’s Proxmox we’re comparing it to, then KVM configuration goes out the window, because Proxmox is superior to TrueNAS for managing VMs.
For SMB and NFS, you can argue you need tweaking, but using the defaults is just fine, until you need it to do more (like root-on-nfs). The same point applies to TrueNAS as well, if you need custom server options, then you’ll need to set them up there (only that there are some options present in the GUI).
I never heard of anyone using random github repos to setup ZFS, SMB or NFS. I might hand it to you if you meant really old stuff, like compiling a ZFS DKMS many years ago (I’m not familiar what options you had at the beginnings on ZFS-on-Linux, but I suppose you’d have some kind of code you’d run to compile your module, instead of just simply installing zfs-dkms from your main distro’s repo). Today this is clearly not the case on a vast majority of distros.
You shouldn’t do it regardless.
I’ll have to disagree. The libvirt package is way more complex than what proxmox offers (qm + pcm). It works for virt-manager and multi-hypervisor platforms like opennebula, because they have something to integrate with and don’t have to write their own modules for kvm, xen, esxi, bhyve, lxc, virtuzzo and all that. But from both a setup and from a complexity level, libvirt is definitely harder to manage (just making that horrible XML is lame, I believe libvirt was working on switching to json or yaml IIRC). For qm, you just set up the VM conf file and you’re done (even if you have to use the CLI, it’s way easier to follow).
That doesn’t mean proxmox is necessarily better (in fact, the tooling built around qm, i.e. the GUI, is still buggy / crapshoot in some places). But it’s what we have. I really wish we’d have a better platform, but nobody’s building it and I doubt I have the skills to build one myself (at least I could design it architecturally, kinda like pseudo-code, but the implementation would probably be insecure AF, even if I would write it in rust or go).
But I’ll agree with your point, I find the GUI to be limiting and buggy fr fr.
I’m nitpicking, but I think you mean single-node or combo storage + compute.
I’ve managed 70+ users in Samba with different ACLs and additional read-only permissions for others. And we had more than 100 shares. I did that using groups. No user had permission to a single folder, only the groups had in the smb.conf. I would just add the user to the group, reload the config and they’d be up to the races.
You’re telling me I could have managed all these using a GUI, easier? I kinda wish that was true, but I have high doubts and I would absolutely love to be wrong (because it’d make my life easier outsourcing my job to someone else, if they can handle the GUI). Well, atm I’m not managing samba anymore, but that was among my fun rebuilding projects I had (permissions were handled with symlinks before, UURGH).
Or a more decent platform, like zabbix-agent or prometheus node_exporter (to be fair, idk if there’s any zfs-specific modules, but you can write basic stuff yourself in case there’s nothing, like grepping for fail or error from zpool status).
Slack integration is definitely an enterprise thing and I absolutely hate it. People are so used to not look at the zabbix or graphana dashboards for alerts (probably because they bloat them with so much stuff and don’t configure the thresholds, or outright just ignore stuff until too many errors are pilled up, that they simply start ignoring it). Then some brilliant dude came up with the idea to send error alerts on slack and telegram instead and now you have a filled dashboard and a filled up message inbox that you just see, acknowledge and do nothing about, other than ping others and maybe open a service ticket with the right team. That’s how enterprises work (mostly because everyone’s just overworked, not because they’re lazy).
What a silly idea, the standards in my homelab are very flexible compared to what I’m willing to bring into the office.
The fact is TrueNAS implemented these features but they did it rather poorly. It looks and handles like a quality issue.
Canceling a wizard because I have to go create a dependency first? It’s a wizard - just add the selection/creation of the prerequisites to an earlier page in the wizard. This is basic UI stuff. You can’t miss it if you ask for feedback from new users.
They support Slack webhooks for notifications, but leave out Discord and Teams? The feature request for a generic webook option has been on their site since September. It would probably take 20 minutes to remove code from the Slack implementation to make a generic copy.
).
Highly recommend the pass-through controller technique. It really stays out of the way.