For the past nine years or so I’ve had a NAS running as a home server. Since then, I’ve also added a pihole + piVPN, as well as a simple minecraft server with a management panel. The thing is, all of these run on different devices (NAS, raspi 2, and a spare parts build respectively.)
For years I’ve also wanted to expand and host other services and I’ve gotten as far as running nextcloud in a jail on my NAS, but never really much further. Cutting to the chase, I picked up a Seeed Reserver on sale recently and I’d like to actually start hosting more than a NAS with a cobbled together NextCloud setup. Here’s a list of what I have in mind to run on the Reserver:
Nextcloud
Jellyfin, or similar
Airsonic, funkwhale, or similar
Minecraft server
Bitwarden^
SearX^
SteamCache^
Photoprism^
Pihole/VPN
^ These are all options I’ve considered, but I haven’t read up enough on to know if this is the best solution for me. I wanted to add them for full consideration, though.
The big question that I have right now is how to go about hosting these services. For one reason or another I’ve always just ended up with one service per machine, so running multiple things on one machine is something I’ve not tried before and I don’t really know the best way to do that or manage everything.
From looking into this and reading and watching where I can, it looks like docker or some kind of container management is probably best. Portainer seems to be a common solution, are there other things I should look for? Would it be better to just host everything on the OS itself and have some sort of management program? I guess what I’m looking for here is some advice on what to look into and where to start with this.
Beyond how to go about hosting different services, obviously I’ll be moving the minecraft hosting over to this new server. That said, would it be a good idea to move the pihole + PiVPN and Nextcloud over to it as well? I would like to have everything in one spot for convenience and organization’s sake, but I don’t know if there’s actually any advantage one way or the other.
I’d appreciate any advice or tips anyone wants to share, and thank you in advance!
In short, get an OS/distro you’re comfortable with or will to learn and start setting things up one by one. What you should consider is however that you’re prioritizing, if storage is your primary target you likely want to look at FreeBSD or possibly some Linux distro that use a somewhat recent kernel and have decent support for ZFS and go from there.
@TimHolus is there an advantage one way or the other? Is there a reason I wouldn’t want to, for example?
@diizzy for now I plan to keep the NAS and the other services separate. I’ve worked with FreeBSD with Free/TrueNAS and their jails system, it’s been a slog to even get basic things working. I have more experience with linux which is why I’d like to have two machines - one managing storage (where I’m familiar with the layout, etc. for storage), and one for hosting other things like nextcloud, jellyfin, etc. I guess my question is just how to handle everything on the non-storage server, if I just install stuff and let it all run or if I’d want to have things running in containers or virtualized like Tim mentioned.
You should try TrueNAS scale (it is Linux based) but as Wendell says, TrueNAS doesnt approve if you “color your crayons outside the line”. Stuff breaks or will break in the future if you do things outside of their prescribed web GUI. But at least TrueNAS wont get in your way directly and prevent you from doing stuff you like to do.
See also TrueCharts for additional software outside of the official TrueNAS apps.
Not comprehensive but I run a comparable setup with a Debian machine running Linux apps (Jellyfin, Mealie, Uptime-Kuma, Monica CRM) and a TrueNAS Core box running some jails (nextcloud, transmission, etc) and core services (samba, nfs, etc).
On my Debian box I run both native apps and containerized apps. No need for VM’s in my case. Native apps I use are Nginx+certbot to reverse proxy everything for https because the dockerized version is a headache, Jellyfin because GPU passthrough for hardware transcoding is a nightmare, and Cockpit for a simple management UI.
Everything else I run with docker-compose because it tends to “just work” whereas podman-compose always gives me some kind of problem (could be user error I dunno). Some self hosted applications only support a containerized version, so it may be necessary to run both at the end of the day.
For docker-compose apps, I run with the data stored locally and backed up nightly to the truenas box. I run the backup scripts on the debian machine, then have the truenas box ssh in and copy the files, that way the backups run even if there is some kind of connection issue.
I’m going to say that TrueNAS isn’t ideal for when it comes to do anything else than what’s it’s meant for, you have much better tools available in FreeBSD compared to the customized variant for NAS purposes. Irregardless, you want something that’s easy to maintain and keep up to date especially Internet facing services. I’m going to say that virtual machines probably wont fall under that category overall and you likely want to look into a distro with decent docker support if that’s what you’re going for but that itself comes with a few downsides.
Honestly, after spending a couple of days trying to color inside their lines, it still isn’t a great experience at the moment on Scale. Admittedly, my anchor point is Synology’s environment, but things that were five clicks have turned into five hours, five documents (minimum), and in one case a bug report that has the devs arguing amongst themselves on a public forum.
I’m still playing with it, as I’m still waiting for a HDD enclosure to hook up to the box so right now it’s a throwaway / test environment anyway. But still, I’m pretty close to just throwing debian or proxmox on the box and standing up a separate storage host that runs TrueNAS Core instead. Let the tools do what they’re good at for simiplicity/sanity’s sake, and all that. Maybe if I’m lucky it’ll win me over before then, or I’ll have the ‘ah ha!’ moment where their system makes sense.
To the main topic of the thread, a lot of what you’re running is similar to what I had set up on Portainer (i.e. docker with a web UI) on the syno box. Not entirely familiar with all of the items you’re intending to run, but the main one that sticks out as potentially wanting some dedicated hardware allocated is the game server.
The other potential thing that might tip whether to containerize, virtualize, or just install on bare metal is if things have to share specific hardware. As an example, say you want to pass an integrated GPU to jellyfin for transcodes, but also have some other thing that needs GPU access as well to run faster (Photoprism, for example) – containers probably aren’t the answer there, and deciding between bare metal or virtualized will likely depend on the GPU you’re using. I’m still early into my research in this part, though, so ymmv (and/or I’m probably holding it wrong).
If you can, try to get as many things running in LXC / LXD and Podman / Docker and anything that cannot run in an OCI or OS container, run in a VM.
As for the OS, like always, I suggest you stick to what you’re already familiar with. If you want an appliance OS, I’d suggest you just pick a hypervisor that also doubles as a NAS and have a VM or container inside that mounts to a NFS share (or local FS) on the host.
I’d keep these 2 on the rpi 2, only because DNS, in my experience, should be running outside of your hypervisor. The VPN should be fine on the Pi depending on your internet’s upload speed, but if you want to stream stuff outside your home, you might want to move it to the hypervisor and keep just pi-hole on the pi.
At the moment (22.12.3.1), if you encrypt the root/parent dataset in a pool, and then install the apps into that pool it will create the ‘ix-applications’ dataset as unencrypted. That dataset being created unencrypted during app setup is by current design. However, having an unencrypted dataset as a child of an encrypted dataset is a bad practice/broken scenario that breaks things like replication. However, the only warning you have that you are going to put yourself in this scenario comes after some health check job fires after you’ve gotten yourself into this state, and there isn’t a good migration path out of it (save tearing it out and starting over).
Containers are more lightweight. If the image(s) you are using are well maintained might be easier to setup. It might also be easier to update down the line. That depends. Say if the software you are running runs on java you would not even need to be aware of that if you just setup the container. You never had to install java. The image maintainer had to. And if java disappears and gets replaced. As long as the image is maintained in an upwards compatible manner, you may not even notice.
But if the image is not so well maintained or does not support the setup you want you either have to set up the software in a VM or build your own image.
Building your own image is similar to setting up a VM, except it forces you to make everything repeatable in a scriptable manner. So it´s like setting up a VM with extra steps.
I´d say some of both makes the most sence for a home setup. If there is a well maintained non-scetchy image available for your software. Go for it. Otherwise, it´s less friction setting up a VM a lot of the time.
The main thing you gain from both approaches is some seperation of concern in that if one software needs a specific version of a runtime you do not need to juggle around multiple versions of it. Can get a lot more complicated if the software is maintained for a different base distro than you are using (RHEL vs debian flavored).
If you run everything within one operating system, there’s nothing wrong with that. Some will say that theoretically it creates a bigger mess and in some extreme case it may negatively affect security.
Among other things, for this purpose there is conterization. That is, keeping and running applications in an isolated environment. As a result, such an application is less likely to mess up the OS if it breaks off the chain.
Virtualization is simply running an operating system on a system. So we isolate not only applications but the entire OS. In the case of virtualization, of course, we need much more hardware resources.
Sorry for the delay, unfortunately I’ve been away. Tim and Maximal, thanks for the explanation on when to use containers vs just running everything on the base OS. I think that makes a bit more sense to me now.
For a lot of the other discussion about TrueNAS or FreeBSD vs. linux, I think I did a bad job explaining my situation. I currently have, and will continue to have, two servers. One is TrueNAS Core, and I plan to keep running that as opposed to switching to scale and running everything on one machine. The second machine is likely going to run debian (just some kind of linux, I’m not too picky) and will run everything else specifically because TrueNAS - especially Core - doesn’t always play nice with other software.
One of the lingering questions that I had was why waste resources on containerization if everything could run on the base OS. Hypervisors and VMs I get, as there are plenty of instances where you need to run more than one OS, etc. etc. With containers, docker, etc., the main advantage as I understood it for something like this was if something breaks it’s easy to remove or stop without breaking other stuff on the server in case they share dependencies or have similar settings. Thinking about that though, I’m fairly certain most package managers can account for and handle that sort of thing, so I guess there’s maybe not the benefit I had expected, and probably an additional layer of complexity as well due to the containerization.
Docker containers specifically only share the kernel with the host. They ship all the required OS packages. So you can have containers that are based on a different distro. Just your base OS has to be Linux for you to be able to run Linux containers on it directly. On Windows Docker Desktop will spin up a Linux VM in WSL2 to run Linux containers with.
Personally, running proxmox with one VM for docker + simple software that won´t cause problems like nginx. And then some other VMs where it made more sence to me to put it into a VM. For instance gitlab runners I very much want to be it´s own VM.
The unifi controller I also set up in it´s own VM back then. It´s still on debian 10 because I can´t seem to be able to upgrade that thing to past debian 10 without breaking it. So that´s another reason why virtualization and containerization can be good. If I had everything on one OS install I´d be stuck on debian 10 now. But instead I can have most things on debian 12 without any upgrade issues. I´ll just have to figure out what to do with unifi when debian 10 goes EOL, but that does not impact anything else.
I am currently looking at Unraid or Proxmox as my TrueNAS Scale replacement… will keep TrueNAS as NAS for storage only, but the apps and virtualization story on TrueNAS is not great…
I am using Proxmox since a few years. I like doing “storage” things in CLI, e.g. creating ZFS pools with my own parameters and editing config files for samba and NFS. I know Proxmox can create ZFS pools in webgui, but I want my own parameters.
I also like the Proxmox webgui for VM/LXC a lot.
If you separate everything, maybe TrueNAS Core and XCP-NG can be interesting to you, as the youtube channel Lawrence Systems does it.
I also wouldn’t know what to do, if I start all over
Native ZFS encryption is a nice thing, TrueNAS Scale has it, Core not. I use it in Proxmox, with custom systemd script to unlock the pools.
hey, have you had any good sources to influence the decision between them two? (unraid vs proxmox)
I run only one machine rn (proxmox) virtualising docker on 1 vm and true nas in 2nd vm. And still not comfortable with lxc, tested out true nas as a single solution for bare metal, and wasn’t enjoying it. Thinking about staying on proxmox and exploring lxc more in depth or going unraid…
The processes in the container still run on the host OS kernel, so they should be pretty similar for CPU and ram usage to running the program completely natively. There might be higher CPU for the extra networking stuff going on (depending on the setup), although I’m not sure about that.
Containers are less efficient disk space wise, although we are still only talking in the tens or hundreds of MB extra for most containers.
Yes, package managers can solve this. However, many web applications are not packaged, or not packaged by the distro maintainers.
There are some issues that are hard to deal with, unless you are either proficient at compiling stuff from source yourself or are using something that brings the dependencies with the application (e.g. containers, flatpak, snap, appimage).
For example, each Debian version only has an official package for one version of PostgreSql. So if you are on Debian 12, then only Postgre v15 is available, but what if the application only supports up to v14. You either will have to compile v14 from source, find another party that packaged v14 for Debian 12, or patch the application to support v15. While if you were using a container, this whole issue gets sidestepped.
