Seeking advice on a NAS build

Operating Systems & Open Source BSD
,
1

Hi there,

so… I’m sorry in advance but this is going to be a long one.

Background (you can skip this)

I’ve been tinkering building a new NAS for a while and now that I have some time on my hands I finally got around to it. The hardware is done, but I’m still debating on the software side.

So originally I was looking at just installing Rockstor as the NAS OS, the reason being that initially I didn’t want to deal with some of the downsides of ZFS (different sized drives, expanding the pool), and also Rockstor seems to have a fairly solid plugin system based on Docker, and it’s fairly trivial to add and run docker containers as plugins that aren’t in their registry as well.

However, shortly before I set this up I read up again on their licensing and it turns out the stable channel updates are only available with a subscription. I’m not keen on paying for yet another subscription so sadly this is off the table.

TrueNAS Scale or Core?

So I decided to go with TrueNAS after all, as this was my second choice to begin with and it seems to have fairly good support and a big community if help is ever needed. I know OpenMediaVault exists but the website doesn’t tell me much about features specifically regarding plugins.

So I was wondering whether Core or Scale would be the right fit. I will be honest I don’t really understand the difference between the two (other then Core being FreeBSD while Scale is Linux), and their buzzword salad didn’t help me on that either.

Essentially I’m wondering if Scale would be “overkill” for a small NAS with a couple services running, or if it doesn’t really matter.

Plugins and Docker containers

What I need initially would be the following:

  • Automatic Ripping Machine (I have a lot of physical media and a drive that can RIP it all)
  • Deluge
  • jellyfin
  • Pyload
  • (maybe) Scrutiny (is this actually as useful as I hear some rave about it?)
  • Heimdall for easy access to all the services

And down the line:

  • Nextcloud
  • Paperless-ngx
  • piHole
  • Syncthing

So I know all of these are available as Docker containers and that is why I was leaning towards Scale because Docker → Linux → Scale makes sense, right? But then I read in wendell’s article TrueNAS Scale: Ultimate Home Setup incl. Tailscale. that apparently it doesn’t just let you willy nilly set up any Docker container, so it seems that point is moot.

It seems I would have to go with a VM that then hosts the Docker containers instead, at which point I wouldn’t think it matters whether this is done on Core or Scale since both can run VMs just fine.

This also begs the question whether it makes sense to use TrueNAS plugins at all or if at that point it’s just more convenient to use the one Docker VM to rule them all.

The next question for the Docker VM would be whether it matters what Linux it’s running. My main OS is Fedora so I was leaning towards Rocky (or Alma), however wendell runs Debian on Scale “because it’s the same kernel version as the host OS”, although I’m not entirely sure how that’s relevant. On Core on the other hand I wouldn’t think it matters at all? It seems everything wendell’s doing in Scale is also possible on Core.

Data Storage

So for storage I bought 2x 16TB drives that will be set up in a mirror configuration for now. When they inevitably fill up I was thinking the drives will have gotten cheaper and I can just buy 4 of them and throw 3 in a new RAID-Z1 pool, and the last one + the two existing drives would be going into a second vdev on that pool. I could also have 2 new + one “old” drive per vdev to spread out failure potential but I’ll see about that when I get to that point. Is that idea sound?

From my frankenstein NAS a couple years back (old MacBook running Windows and all the drives via USB hubs, if you were wondering) I still have all the drives. They are still functioning fine and so I would like to use them for unimportant data, i.e. temporary download storage, ripped media that still needs sorting and the like.

The issue is that those drives are different sizes. I honestly don’t know all the drives off the top of my head (and they are buried in a shelf somewhere right now), but I believe it is 2x 2TB drives and 2x 1TB drives (possibly it’s just 1x 1TB though).

On Rockstor I would have just put them all in a JBOD and be done with it, however as I understand there is no such thing as JBOD in ZFS so I can’t just use the entire drive space. Am I remembering correctly that I would have to create separate pools for every drive to be able to use the full disk space? Is there another way I could “unify” them on TrueNAS?

Boot drive for VM/configuration storage

Another slight annoyance with TrueNAS is that it seems using the unused space on the boot drive for literally anything is not supported whatsoever. So I have a 250 gig SSD, the bootpool takes the entire drive but only uses about 10 gigs of it, which… seems a bit of a waste.
And I know why that is a thing and I also know that iX Systems is not interested in changing this behaviour anytime soon.

There is however a community guide on how to do this:

It does bring up some questions though. First and foremost, in the warning at the top it says:

will break the ability to replace failed drives

And I’m confused on that. Are they referring to only the boot drive? Why would it break the ability to replace failed drives on the data pools that are entirely separate from the boot pool?

Also, has anyone done this? Any experiences with it or things I should watch out for if I were to do this?
I would like to put the VM storage on there and if I’m going to use Plugins maybe also the configuration data that they create. It just seems reasonable to put the frequently used data on the SSD and have the entirety of the hard drive space for media storage.

I’m hoping this is everything :thinking: at least I can’t think of any more right now.

Thanks for reading and hope for some advice.

2

If you are planning on running docker services and serving content that way I would actually steer away from TruNAS and utilize Proxmox/DebianZFS + docker and VM’s for serving content. I am not super happy with TruNAS Scale and many others have issues with their docker implementation. If you deviate from their way of doing things at all every update something will break and you will have to fix it.

1 Like
3

Well, running Docker was never the plan but not everything I want to run is available as a native plugin, so Docker is the only alternative.

I haven’t looked at Proxmox extensively, but doesn’t that require a subscription as well?

I’m confused… I thought it wasn’t running Docker? Or is it running Docker and just won’t let you launch random Docker containers? I’m just confused why Wendell would be going through a VM to set up Docker if Docker is already available (albeit with a bit of work to configure).

4

Not necessarily. You can use it for free, but you need to change from enterprise repos to community repos once installed (I believe there is even a button in the UI somewhere now to do that). You get a popup every time that you don´t have a valid subscription, but you do get all the features regardless.

Also running it like this, maybe I´ll buy it at some point don´t know didn´t yet. As I understand it as a community user you are more or less a beta tester, but it has been rock solid stable for me since proxmox 6 whenever that was. Installed samba on proxmox directly, it already ships with zfs. Running docker in a VM.

5

That’s pretty much the same as Rockstor then and I just don’t like the idea of that tbh.
Don’t get me wrong I have no issue with paying for things but I don’t like being forced into being a beta tester.

6

Like I said it´s been stable for quite a long time for me, proxmox 6 came out 4 years ago and did 2 major version updates. Take that for what it´s worth.

I´m just not doing many changes to the base os. I would not want to install docker on proxmox directly (it would work though). But youd have to install another repo. Maybe docker breaks on updates. I mean it broke on fedora before with cgroups v2 for like half a year or so. Maybe they add their own docker down the line. Maybe you want to pass a nvidia gpu to your container now you gotta install nvidia driver on proxmox. I don´t want to do that either. If something breaks I prefer that to be in a VM that is easy to restore with snapshots.

7

A lot of this will be more of a pain than useful and I second that a NAS distro isn’t suitable for a all-in-one box which is what you’re trying to do. Just go for a plain distro be it FreeBSD or some Linux-variant, I would personally go for FreeBSD but that’s up to you.

8

Well the idea was to use the plugins that are provided by TrueNAS (or whatever other software) but as it turns out not everything is available as a plugin :sweat_smile: I’m not sure where I’d even start with a complete DIY setup and honestly I don’t really feel comfortable with that. I’d rather have some software where the majority of the work is already done. If I have to do some legwork for a plugin or just have to do without one that’s fine with me. The primary goal is to have a storage box after all.

9

No.

It is using their own implementation of kuberneties/docker which is really unwieldy for most use cases. I have docker compose running natively on it using the guide in the forum.

Proxmox would be my choice if you are planning on running VM’s. If you can containerize everything then I would just do Debian with ZFS and docker + portainer.

10

As harsh as it may sound is it a better idea to have something (hopefully) running and no idea how to fix it when it breaks? That sounds like a good way to loose data. You’re aiming for a NAS oriented distro, you can in theory try to shoehorn X but that’s usually not a great idea at all. It’s like buying a car and trying to use it as a RV.

Any generic application frontend is more or less geared towards a very specific software setup so if you’re deadset on Heimdall (I would recommend against using such software) you should look up what they recommend / require and go with that. I’m quite sure that will rule out any NAS distro available.

Start small and go from there (just Samba and possibly NFS), most of this will be handled by a regular package manager which is by far easier to maintain if you’re not familiar with Unix/Linux in general than Docker, Containers and whatnot.

11

I’m just going to believe you here, but I will be honest I have no knowledge of Docker or Kubernetes other then they exist so I can’t really follow you here :sweat_smile:

I’m not sure I’m following you here. I’m not trying to shoehorn anything. I was planning on using the functionalities that are provided by the software in whatever way the software is intending them to be set up.

Reading wendell’s article it intentionally changes nothing on the base system and works within the boundaries of what Scale provides. It’s setting up a VM, which is officially supported, and using an NFS share for that VM, which as a NAS appliance is also supported.

I’m not, it was just the first thing I came across when I tried out Core as it was one of the native plugins it provided. It doesn’t seem to be available on Scale but there’s other similar things.
Also are we sure we’re talking about the same thing? I’m talking about the Heimdall Application Dashboard, it’s just a webpage for Bookmarks and Live Stats of linked services. I think you might be talking about the Android Flashing Utility which I can assure you even if I wanted to run, I would not do that on a NAS :stuck_out_tongue:

12

FWIW I run a lot of similar apps (nextcloud, jellyfin, heimdall, syncthing) and settled on a TrueNAS Core for simple storage and backups as well as a Debian 12 tiny/mini/micro for power efficient QuickSync transcoding and Docker containers where it makes more sense than a FreeBSD jail. Best of both worlds for slightly more money (gen 6 or 7 intel was cheap last I checked, I believe because they don’t officially support windows 11, which doesn’t matter in your use case).

If that drive fails you have two problems, instead of one. A truenas boot device can be replaced and restored from backups in minutes without this complication, add a $20 ssd if you like.

1 Like
13

Yeah that’s kind of what I was thinking too… if you just backup the TrueNAS config regularly then swapping out a failed boot drive isn’t a big deal to begin with. The wording just made it sound like it would also break replacing data drives which made no sense to me.

But yeah I might just throw in 20€ and buy a second drive so I can mirror them

14

You want to use a distro that targets using a device as a NAS but then you want to add on a bunch of services and functionality on top that’s not related which will be messy a best.

I’m assuming that Heimdall needs to know where things are located but oh well…

15

I mean, you’re right in that they are not directly related to being a file storage, but NASes running more then just file storage has been the case for the last 10 years or so. Even when you buy a full built Synology or whatever brand you get the option of adding services by using their plugin system. I’m doing no different here, I’m using the plugins that they provide in the UI, nothing more. I’m not planning on changing anything to the base system.

I don’t mean to be rude but it seems to me that you have either never used TrueNAS to know how the Plugins work or you’re not understanding what I’m wanting to do. As I mentioned before altering the base system was never and won’t be me goal for this. I’m planning to use what is provided by the system.

Yes, but it’s configured right on the webpage. You select the type of service to link, you enter the address and possibly an API key and that’s it. The communication between Heimdall and the services for live stats and whatnot is handled through their respective APIs.

16

TLDR; by your description it sounds like this box will be your storage nexus where a large majority (if not all) of your data will live. If that is the case, I would definitely say TrueNAS is the right call, specifically TrueNAS Scale.

I was in a similar boat a few years back and settled on Scale as well and it’s been amazing.

On Distros:
Between TrueNAS and Proxmox, my rule of thumb is that TrueNAS is a Storage-First distro, while Proxmox is a Virtualization-First distro. They have heavy overlap, so it’s very common, and super acceptable, for people who have more experience with one to use it for the other usecase.

Scale vs. Core:
Tom Lawrence (Lawrence Systems on YouTube, and a champion of TrueNAS in the SMB space) said it best in a recent livestream: “every time a client hits a problem or limitation with TrueNAS Core, usually the answer is just to switch to Scale”. Because Scale is built on Linux it has a much wider gamut of bolt-ons you can MacGuyver on to the system, better driver support for modern hardware, and easier to troubleshoot online.

On Docker
Since Scale is basically Debian, Docker and Portainer are super simple to get working. Wendell posted a great initial guide here on the forum, and it’s constantly being updated by the community (TrueNAS Scale Native Docker & VM access to host [Guide]). There is currently a minor conflict with TrueNAS Apps, but only if you enable them - however because no one ever uses Apps and it conflicts with normal Docker TrueNAS, iXSystems is actually removing Apps in the next major revision.

On VMs
VM support has come along way, so much so that my gaming rig is a VM and is powered by a 3080 Ti via GPU passthrough. My wife just beat Stray by streaming to the Nvdia Shield in our living room via Sunshine/Moonlight and loved it - she’s working through Dave the Diver now, and I’m playing Starfield (as I’m writing this). She’s a lawyer and super non-technical and has had no issues at all (like I would find her at 4AM still playing).

On Disks
When I was young I was really all about using as much storage space as possible from different drives, but after losing data a couple of time, as well as spending days troubleshooting I’ve realized my time is worth more that the few dollars saved by using an exotic disk setup. ZFS is pretty robust, and has a healthy distrust of any storage medium, so multiple single vdevs/pools mounted to a single folder has worked best. If you’re in the US or Canada, I recommend checking out https://serverpartdeals.com/ - they sell refurbished drives at a great price and are super great at replacing drives that fail a burn-in test. I’m currently a fan of 12TB Helium Hitachi enterprise drives for spinning rust and Intel P4510 NVMe SSDs for primary storage.

On OS Backup
Backups on TrueNAS are just the best. The entire config can be exported to a single file - I have this backup to Backblaze & S3 every day. To recover from a failed OS drive, just reinstall TrueNAS and import the backup config when prompted. Had to do it 3 days after my first install, and was sold instantly. I have a bash script that reinstalls everything off-label (like Portainer) - everytime I add a new off-label component, I just copy the commands I used into the script as I’m entering them.
TrueNAS also has a builtin Cronjob manager and Cloudsync tool in the GUI so it’s pretty simple to manage.

On Data Backup
Having used enterprise software, TrueNAS’s ZFS native is just plain amazing. If you ever have the ability to put another box somewhere else, TrueNAS can perform differential backups to a remote TrueNAS install with a single click - even the disks don’t have to be the same. You can also use the cloud sync tool to backup just subsets of data to a remote site - All my work and personal documents get backed up to multiple cloud services every hour, and it was basically “Select folder, select destination service, select frequency, and DONE”.

On Licenses and Subs
Just not a thing, this is why Tom, Wendell and so many others favour it in the business space as well. (They also love XCP-ng on the virtualization side as well, for good reason, but I digress)

For context some of the components in my home network are:

  • TrueNAS Scale at the center of my env serving SMB shares to the network - running a AMD R3600 on a AsRock Rack X470 w/3080 Ti on passthrough and MCX4121a with Virtual Functions (VFs) and VXLAN for VM & Docker network isolation
  • Performance & Data Sensitive VMs/Containers run on TrueNAS on dedicated ZFS datasets that can then be individually backed up via the WebUI (like the gaming VM, by Seafile (document sharing) docker, and Gitlab docker)
  • Proxmox HA Cluster running on 3x Lenovo M720q Tiny for fun and multi-node development work (they use a MCX354a 40GbE ring with VFs between each other for VM connectivity)
  • TrueNAS/Portainer running on a M720q Tiny for core house services like Home Assistant, AP Controller, Dashy, and Prometheus/Grafana - all are backed up to the core TrueNAS server.

Hope it all works out on the NAS!

1 Like
17

Hey, just to clarify, it is not improving on Wendell’s guide, it’s a similar solution that has more performance than going the VM Route. If you have any questions, feel free to ask in that topic as well.

Right now we are preparing for Cobia which will remove docker from TrueNAS Scale, so far our tests have concluded that users that use my script should not even feel the effect of that after upgrading.

1 Like
18

Thanks for your insight. So if I’m reading this right you’re voting for always going Scale and just skipping Core entirely.

I’m not sure I’m following here. The linked guide sets up Docker and Docker Compose natively, however it refers to Wendell’s guide which specifically avoided doing those things to not alter the base system.
I’m not sure I’m really comfortable to doing that right away to be honest. Maybe down the line, but probably not now.

Why would noone use the Apps? From the bit of testing that I did setting them up was super easy and they just worked right away, seems like a no-brainer to just use them? The only downside I see for them (at this time anyway) is that not “everything” is available as an App, but I feel that’s kind of normal.

I also couldn’t find any articles online about Apps being removed, can I read up on that somewhere? I’m wondering what their plan on a replacement is since I’m not sure they’d just leave people that used it in the dust :thinking:

Through googling “truenas scale cobia” I just found the beta announcement:

And there’s no talk on removal, on the contrary it’s (supposedly) improving on the UI :thinking:

I definitely get that, which is why I don’t wanna fiddle with it too much and wanted to avoid altering the base system for less risk of breakage. A 2-way mirror also isn’t something I’d call exotic :stuck_out_tongue:
I don’t have much of a choice on the disk setup right now anyway unless I buy a bunch more drives :slight_smile:

The only thing I still need to figure out is how to use my old drives. But I guess I’ll just come back to that later once everything is set up, there’s still some stuff on there anyway that needs to go over to the new drives and then I’ll know what kinda drives I even have lol.

I’ll be honest I haven’t put much thought into backing up what’s going on it simply because it’s pretty much just my media library and if I were to loose that it’s not that much of a deal since most of the stuff I have physical anyways.

19

I would take that Youtuber with a bit of salt and your claim about driver support is based on what? Neither Core or Scale uses a “new” software base (stability > new features) and to my knowledge TrueNAS Enterprise is based on FreeBSD.

I agree however that if your goal is to use the box as some kind pseudo multipurpose server but still claim its a “NAS” SCALE is probably your best choice although there has been a lot of complaints about it being buggy and understandbly so because there only so much one company can do trying to shoehorn everything. There’s a reason why you see “applicances” professionally as far as products goes.

20

Because they are based on a weak implementation of Kubernetes, which in case you only run a single machine makes no sense to run. However, the point of Scale is just that, scaling up, which is why they went that route.

Not apps, docker is being removed, as in the package, they are doubling down on the Kubernetes mentioned.

If you are still debating scale vs. core, just look at the SAN video from LTT. They realize there that every time they start with core, and find something missing or not working, then end up switching to scale.

You’re not altering the system if you follow my guide, you’re enabling features that have been locked away by ixSystems. it will not bork your system if you enable a package that’s already installed.
If you go beyond and start downloading other packages that might conflict with base system, then yeah, there’s a risk.

But, from my personal experience, I went even further than my own guide. I enabled the Debian repos and updates to those packages, and nothing broke. So if you have any experience with Linux, you’ll know it’s quite easy to manage when you know what you’re doing.

As for your concern about needing to always maintain and tinker, I can give you 2 examples where I used both FreeNAS (the old BSD version of TrueNAS) and TrueNAS Scale in my workplace. Both were set and forget. They both cloud backup and have local snapshots. I only connect to the web interface to update the system when needed.

1 Like