Homeserver setup dilemma ft. TrueNAS Scale Apps, Virtualization and Proxmox

Software & Operating Systems Linux
, ,
1

Hello Level1Forums! This is my first post here and it’s gonna be a lengthy one so better brace yourselves! For the past 2 years I’ve been using a very janky homeserver setup. I use ubuntu 22.04 (not fresh install, continually updated from 20.04, yeah I know, bad practice) and even though I have QEMU KVM set up I didn’t do any virtualization on it. I run all of my services on bare-metal ubuntu because why would I spawn VMs when my server is literally connected to a monitor and all of my services are compatible? In fact I do most of my server stuff directly with a monitor and keyboard even though I have VNC and SSH set up. Weird right? Anyways, the services I use are:

  • PiHole + PiVPN (Wireguard)
  • qbittorrent + jackett
  • plex media server
  • a minecraft server
  • 2 discord bots

With the current server I am not hesitant to break stuff and spend a full day repairing it. It doesn’t store any critical data anyway. It’s just a useful home appliance that can have downtime. So after breaking my drivers multiple times and fixing them, breaking and reinstalling plex at least 3 times, breaking and somehow reviving qbittorrent when transitioning from snap to apt, SMB nightmares, discovering, reporting and finding a fix for a bug in ubuntu/firefox and much more, it really doesn’t matter in the end if it all goes to hell tomorrow. And honestly as much as it was annoying it was also kinda fun fixing issues and making it all work.

But recently after a bad experience with OneDrive, seeing how my mom asks me on a yearly basis to copy her phone images to her PC and how my dad has his google drive full 24/7 and documents everywhere I decided it’s finally time to fix all the plaguing issues in my family’s storage of data. Actual important data. I’ve been planning to turn my server into a NAS for a while now but this was the push I needed to finally get started. Yeah, get some nextcloud set up as well so the data from all our phones and PCs is nicely and automatically backed up. Beautiful. And unlike with the current setup I want this one to be stable as hell. Because for the first time I’ll be making something that needs to store important data and not risk losing it. Maybe I can get a system specifically for messing around later but for this one I can’t make any unnecessary risks.

So I’ve been doing my homework. Watching youtube, reading up and planning my setup. When TrueNAS Scale dropped it sounded really promising, ideal for my use case: a NAS that does all the virtualization for the services I need, in a single box. Truly an Ultimate Home Server OS right? Or so I thought. Just as I was planning the setup wendell dropped a video about TrueNAS with Nextcloud and Tailscale which really opened my eyes as to how broken TrueNAS Scale actually is and I had no idea about it. If I hadn’t seen this video I would’ve walked into this minefield and stepped into every single trap it has laid out for me.

Initially I was planning on using the apps tab for services like nextcloud, plex and qbittorrent. The idea of all my storage-based services having direct access to storage instead of through an NFS share sounded promising to me. But after seeing this video I came to know about the issues I previously didn’t know existed. Yeah turns out that both apps and virtualization on TrueNAS Scale are kinda broken. Apps make weird uninitiated snapshots of container persistent volumes and makes lots of them. I’ve seen a temporary solution here involving a cron job and a docker-prune command but I am not really comfortable with this solution as I feel like it could break very very badly. Another drawback with containers is that it would probably be a nightmare getting them to talk to each other (qbittorrent+jackett) although I could be wrong. Plus I have 0 actual experience with containers.

Next in line are VMs. Sure I can just as easily set up a VM with said services as I can using containers. The only drawback would be having to use an NFS share to access storage instead of having direct access which won’t really matter for performance because all of these apps are limited by network anyways. But behold! VMs are also broken. From the video I’ve learned about a weird debian default/bug that was fixed more than 10 years ago but somehow still exists in TrueNAS Scale. Yeah talking about the one where the VM can’t ping the host at all. The solution to this is setting up a bridge connection through the CLI (because GUI is broken of course) which is all so unnecessarily difficult for something that should just work out of the box. But I guess this is a one-time thing and then smooth-sailing right?

Well, another thing that really pushed me away from TrueNAS occurred to me when reading the docs and doing further research. Apparently a TrueNAS install takes up the entirety of your boot drive however large it is. You can’t put anything on the drive TrueNAS is installed in. Given that I have a 1TB boot SSD I intended to use to store VMs as well, losing >900GB of storage for basically nothing is a massive waste. TrueNAS doesn’t let you store anything other than the OS on this drive and messing with the linux underbelly to try to remedy that breaks stuff. Partitioning the boot drive is not supported officially but community managed to succeed, although I am not comfortable with using this as it can kill my install AND all my VMs if it breaks. I happen to have a 250GB SSD in an M.2 enclosure because I thought I need it but I honestly don’t. So I can steal that SSD but that’s still a loss of >200GB.

So I’ve reached my last option and the one I intended to use before TrueNAS Scale even existed. Use Proxmox as my hypervisor and TrueNAS inside a VM. I am aware Proxmox supports ZFS and network shares out of the box but TrueNAS makes it infinitely more simple and time-saving. This perfectly remedies all of the above issues: VMs will be handled by Proxmox which has way more virtualization features anyway, TrueNAS will do what TrueNAS does best which is just storage and file sharing, and since it will be installed on a virtual disk, I won’t have to sacrifice any SSD space. The drawback with this setup is that it adds complexity. I would have to learn to use Proxmox and reading through the docs left me with more questions than answers, mostly storage and backup related i.e. idk which filesystem to use for the install. Another thing is that managing snapshots and backups would be more difficult as I would have to deal with proxmox snapshots as well and I would want my VMs and/or certain data from VMs to be replicated to TrueNAS as well.

So help me decide, which setup to go with? Are there any more traps I don’t know about, ready to ruin my day and eat my data?

2

Howdy! Reviving this because I’m having a similar dilemma. What’d you end up going with?

1 Like
3

I think it’s terrible that no one answered your question because they just don’t take the time to read it all. Yeah, I only skimmed it :laughing:

Anyway Proxmox is what you want for Virtualisation, it’s the best. Yes I am biased but trust me, it will save you having to make difficult choices.

TrueNAS is the best for file storage and file serving but do you actually need something this good? Running TrueNAS inside a Proxmox VM is not the best solution but can be done. You can pass the actual drive controller through and TrueNAS will have hardware access to the drives like it wants.

However you could just install a Linux based file server as a normal VM and let Proxmox handle all the ZFS. ClearOS or Neth Server are two I can think of or an LXC based file server for SMB.

I would not use TrueNAS Scale yet, it’s advantage is it’s VM support which is still rubbish from what I’ve heard. Core is tried and trusted.

I was unable to decide how to fit everything in one box so I have two TrueNAS boxes, one Proxmox box and one Proxmox box with TrueNAS Core VM. A bit of a mess.

2 Likes
4

There’s nothing stopping the OP from tinkering with truenas or proxmox, running ZFS with a couple of cron jobs/systemd timers on Ubuntu, and adding next cloud … or whatever desired phone backup strategy their parents need.

1 Like
5

I’m doing the same thing. Proxmox and truenas

6

still nothing, I kinda left the whole thing hanging, but hopefully some developments soon when I finally get the drives. I did play with Proxmox on that 250GB drive just as a test a few months back and it was pretty alright. Managing VMs in proxmox is a breeze. I also installed a TrueNAS VM on that test drive and it was alright. NFS is a pain in the butt though. And probably insecure although I can just pass it off with the usual “it’s a home use case” excuse.

TrueNAS Scale has probably improved a lot in the past few months so I’m in a dilemma again. Ideally I’d have 2 servers, one for NAS, one for VMs but living room space is limited. So I have to make a choice and there simply isn’t a right answer, only 2 wrong ones.

TrueNAS sucks for virtualization and containerization but when baremetal it’s a solid and stable NAS. And stability is important as I can’t afford to lose data. A bonus is that I won’t need an HBA to easily use my drives.

Proxmox on the other hand gives me perfect virtualization but if I run TrueNAS as a VM it comes with complications such as VM boot order and backups. Actually my main concern are backups. It’s difficult to set them up in a way that’s truly stable and fool-proof. And backing up VMs to another VM in the same hypervisor is probably not a best practice. And I’ll have both Proxmox snapshots and TN snapshots. And then Proxmox would snapshot TN snapshots. Or at least that’s how it’s in my head. I’ll probably just end up with an ugly mesh of cyclic backup jobs that make unnecessary duplicate files. It’s hard for me to wrap my head around that.

If only there was a best of both worlds…

2 Likes
7

I am aware that I can get NAS functionality from Proxmox or a basic linux VM but there’s a simple reason why I don’t want to do it. Ease of use. You may not like that answer but after testing TrueNAS a little, I don’t want to settle for anything else. It’s king when it comes to permission and share management. Everything else is a pain the ass compared to it.

1 Like
8

I want to escape my Ubuntu setup already. It’s a huge pain. I’ve played with Proxmox and TN a little just as a test. They’re both great at their own job but not the other’s. That’s kind of where this whole dilemma comes from. If TN has gotten better with running VMs and containers maybe I’ll just eat a 200GB loss of SSD space and call it a day. I’ll do more research about it.

9

How has it been so far? How are you managing backups?

10

I’ve made a decision. After tinkering a little with TrueNAS I have decided that it’s too much of a pain in the ass for virtualization and even moreso containerization so I will use Proxmox with TrueNAS as a guest. Also setting up the arr-suite, qb and plex is infinitely easier with a docker-compose file than clicking through the apps GUI. Plus the apps are not very up to date in some cases. The only real loss is having to settle for NFS (although I’d have to use it anyway even if my VMs were hosted inside TrueNAS) and complexity but after laying out a plan it’s not too bad. I don’t know if I should use Core or Scale. Core is supposedly more stable so I guess that’s the better option.

1 Like
11

This is the route I went except I am running Jellyfin and arr apps in their own vm rather than containers.

1 Like
12

This is what I’m still struggling with. I’m on my 4th server where I am bouncing between Proxmox and TrueNAS. I’m attempting to make a Workstation with a GPU passed through to a VM which also happens to run TrueNAS as a VM. It will function like a desktop PC but secretly it will be Proxmox hosting both the PC and the TrueNAS and any other VMs I care to create.

Probably a crazy idea but if I’m going to put two major functions in one box then I may as well make it three. At least file access to the server will be fast because it won’t need to go over the physical network.

1 Like
13

I agree, TrueNAS is amazing as a file server. It’s not just the ZFS storage that’s out of this world it’s functions like ACL and all the saved versions available from Windows File Explorer.

1 Like
14

I am running ubuntu on zfs without desktop gui, and put all my service in the lxd container and kvm. Although the initial setup took years to finalize, right now it is pretty stable. Thanks to zfs, zfs just never dies, if any thing wrong, I just rollback to a working snapshot.
I don’t use TrueNas or FreeNas. I only have very basic samba share.
If I run TrueNas, I would put it in VM. I won’t let it manage all my file system. Zfs wouldn’t die, but TrueNas would die and drag the underline zfs with it. I don’t want that to happen.

15

TrueNAS is actively trying to force users into “their” methods. They have/are removing all support for native docker leaving only their Kubernetes implementation (which I find to be way worse and slower than docker for home user).

I was using docker in a systemd container, which they are now removing as well… can’t have people not using their method.

I have also found their VM implementation to be lacking. ie. Proxmox and KVM on Ubuntu had zero issues running a windows guest, but on TrueNAS it is laggy as hell and just doesn’t seem anywhere close to the other systems. All on the same box and drives so it is purely the HOST system. Also, with various updates I have lost access to my VMs till the next update came along and fixed the bugs, yet introduced others.

Basically TrueNAS Scale is a hostile environment for any AIO type system, it is only good for file serving, so you might as well use CORE. I am currently using it as a VM on my Proxmox with PCI sata card passthrough, but am currently investigating UnRaid. Newest version has ZFS support built in. Reason for this is TrueNAS seems to be using a ton on CPU just for SMB hosting.

I also just want to run mainly docker with maybe a VM or two. Current setup is way overkill and am trying to eliminate TrueNAS but still maintain some file sharing etc.

16

Yeah from wendells video and my own tinkering I’ve come to a similar conclusion. Apps and containers are terrible and virtualization is subpar. They are definitely trying to force things for whatever reason, maybe something like ease of troubleshooting? Although it creates the opposite effect because of how buggy everything is.

Also a little update: I finally got the drives yesterday and an HBA beforehand. Installed them and all is nicely detected. Then opened up KVM virt-manager on Ubuntu to test HBA passthrough before installing proxmox. Aaaand it wouldn’t work. Turns out I need to pass through the entire IOMMU group and the motherboard I have is an MSI B550 Tomahawk which has the laziest most horrible IOMMU grouping - everything that connects to the chipset is in the same group. Including the PCIe slot that the HBA is plugged in. So I basically can’t pass it through. If I swapped the GPU and HBA, the HBA would be isolated but the GPU would suffer the same fate so I wouldn’t be able to pass it through.

So basically if I want to carry out my original plan I’d have to buy a new motherboard. Or change plans. So I can either use proxmox or some other linux distro for manual zfs and file sharing and spend ages configuring it or use TrueNAS as a host and pray for the best.

17

After running TrueNAS with a passhtrough PCI adaptor to make the VM as close to hardware as possible, I cam to the conclusion that it is not worth it.

Essentially I was running TrueNAS to manage the ZFS and shares. Thats it. Samba shares were causing a lot of CPU load for no reason at all and other stuff just seemed to eat the cpu even while idle. Performance was subpar as well.

My solution… remove TrueNAS completely and manage the ZFS command line.I used Proxmox as base OS, installed docker, samba and nfs sharing. Used conf files to setup shares, and sanoid/syncoid to manage snapshots and mirroring of zfs folders. Everything is running like a dream now, including the nvidia container kit for various docker things. I even did an upgrade from proxmox 7 to 8 (which including going from debian 11 to 12 and nothing broke. Never had anything not go wrong with minor truenas updates.

Bonus is that I have a PROPER VM and container system with a UI for when that is needed, got a FULL docker implementation and don’t have to fear proxmox screwing things up for me.

Negatives, ZFS is now manually done through cronjobs etc. no UI. Also, shares have to be configured CLI as well.

Thats my story. :slight_smile:

4 Likes
18

As a straight file server TrueNAS is excellent. On Core I’ve not had much joy with the VMs. I did get a couple working but then they broke and it was too fiddly to bother with them. I already could do these things on Proxmox.

I also did TrueNAS as a VM a couple of times. First time was a disaster because I had the wrong idea about why one would do such a thing. I was trying to use TrueNAS as a SAN for Proxmox and the whole system disappeared into a singularity.

The second time I had a genius idea. customer needed a server and a desktop. I used Proxmox to create one PC that did both. Passed through a GPU and USB devices for the Windows VM and passed through a SATA card for the TrueNAS SSD drives. Used a single 1TB NVMe drive for all of Proxmox.

It works beautifully.