Filesystem/scheme for Home-/Media-server/NAS

Operating Systems & Open Source Linux
1

I’m looking for some input on how to setup the storage on my new home server.
I was just using an old thinkpad with a single external 16tb hdd before, but I’m now in the process of migrating to an Intel Nuc 13th gen, 32gb non ecc ram, 4x16tb HDD (connected using an m.2 sata controller) and two 1tb SSDs I still had lying around.
I’m debating what filesystem to use. I’m aware that ZFS would be the somewhat obvious choice, but I’m not super convinced it really fits my use case. Specifically, I’d really like to setup SSD caching so that the HDDs can sleep a lot of the time, which from everything I’ve gathered is not really possible with ZFS/L2ARC/SLOG.
More specifically what I’m looking for:

  • Some (~Raid 5 level) redundancy. That being said, losing data wouldn’t be the end of the world, most of it is not really all that important (linux isos and the like), and I’ve got backups of the stuff I care about.
  • low power consumption/SSD caching so the HDDs may sleep regularly (i.e. keep linux iso torrents in cache for seeding)
  • decent (~2.5gig level) perfomance
  • easy extensibility would be nice to have, but not critical

Bcachefs sounds like it will be perfect for my use case at some point in the future, but for now their Raid5 support is still marked as unstable so that’s unfortunately out. So I’m considering some combination of the followings:

  • either mdraid or snapraid/mergerfs for redundancy
  • dm-cache or bcache for ssd caching (possibly writeback cache on ssd dmraid mirror, or writethrough cache on separate/striped ssds, or some combination of the two? (i.e. dm-writecache on raid1 and regular dmcache/writethrough on bare ssd?) on the block level
  • or some sort of mergerfs/script based caching solution, though from what I gathered those mostly operate “just” as a write/recently-written cache and don’t promote often accessed files to cache?

That’s basically where I’m at, but this results in a lot of possible combinations and I’m somewhat paralyzed to decide on one. I think my two favorites right now are
a) mdraid1 ssd writeback bcache for single mdraid5 hdd device (formated with btrfs), pros: simplicity, perfomance (?)
or b) 4 writeback cache/hdd combinations, mergerfs three together, snapraid to 4th nightly, pros: resilience (never lose everything), possibly lower power consumption (only spin up hdd containing accessed files when not in cache), cons: lose amount of data written since last snapraid update in case of failure (and from what I can tell basically random?), also snapraid update will probably mess with/invalidate the entire cache?

I’m aware there’s probably no clear cut answer here, I’m just interested in hearing what others are thinking or if I’m maybe missing some better options. :slight_smile:

1 Like
2

For anyone interested, after trying out some combinations I finally landed on the following approach. I might write up a more detailed guide at some point if there’s interest.

  • btrfs for the data drives, with bcache-writethrough cache to (primarily) cache metadata
  • roughly pre-sort data/media on the hdds (e.g. all movies on one HDD, all shows on another, all animes on yet another). doesn’t have to be quite this corse, e.g. on a per show basis should be enough. This is so that when a user streams multiple episodes of a show only one hard-drive has to spin up and stay active, instead of alternating between hdds.
  • mergerfs the HDDs to one mount point with mspmfs create policy and moveonenospc=true. this is to maintain the “presorting”
  • another mergerfs-pool as ssd-write cache and download/torrent directory in front of the data pool (ff policy, roughly as described on the mergerfs github)
  • rsync ssd-pool to backing pool nightly, then snapraid-btrfs sync to parity drive (snapraid-btrfs solves the “write-hole” issue I was concerned with before).

This way the drives stay idle for most of the time, even when e.g. download/moving new files to the pool, and usually only one drive needs to spin up (and stay active) to serve media.

In the future I’d like to implement some sort of jellyfin-precaching as well (I outlined my ideas here: Server-side pre-caching of shows/partially played movies · Jellyfin Feature Requests), but that’s just an idea for now.

1 Like