The Ultimate Home Server - Toward an Amalgamation

Introduction

The purpose of this post (series of posts!) is to gather information about how you, the community, use your home lab and home server(s). That’s a huge topic. So I’ve broken it into a bunch of sub-topics, linked to below. Put the reply in the most appropriate sub-topic but if you don’t see something, say something.

Muh vision

I’d like to start and maintain a guide, both a wiki of sorts and a curated list, of all the available software one could add to a home server to do a lot of common ‘home server’ tasks.

My vision of The Ultimate Home Server is basically a non-cloud AI that Does Everything. But down here in reality, we aren’t there yet. There is no One Program to do everything. So let’s talk about what we like, and dislike, about a lot of popular purpose-specific open source programs.

Eventually we might be able to package some of these up into an add-on for something like TrueNAS; packaged commercial systems like what Synology provides is also kind of close to this. I would say those systems are adjacent to the ideal system in my mind – lot of in-common features – but we can do better by designing the system with different goals.

What might you want in the ultimate home server?

  • Home Automation (link to thread)
    – home assistant
    – link to IoT series

  • Home Surveillance & Alarm tie-in

  • Home & Cloud PC backups
    – full disk imaging of PCs
    – maintain “retired” PCs at bootable VMs “just in case”
    – synchronize cloud services for offline backups
    – functionality for offline backup (i.e. usb disk rotation?)

  • Music & Media hosting
    – what’s good
    – what doesn’t require cloud services
    – etc

  • File Hosting
    – things like NextCloud can tick a lot of boxes here
    – Organize all your Adulting Stuff like bills, taxes, retirement, etc

  • VPN
    – host your own VPN to connect back to home resources

  • Knowledge Repository
    – perhaps the most important aspect of my home server is that it is my brain outside my brain. I need a wiki to store stuff and keep track of things. I need something to store and keep track of my books BUT ALSO my notes in books.
    – love something that integrates with all my old written notes that have been digitized. Used LiveScribe for almost a decade. The new pens don’t do it for me anymore, though.

Things that would be nice to have:

  • Secure VMs
    – for things like secure online banking or one time use

  • Easily create disposable VMs that are fully patched up

  • Wifi Portal Landing Page
    – offer guests access to… the printer? The media server? The remote coffee pot? Not a captive portal per se, but just a landing page letting home guests know how to interface with the TV in their room, the network resources, etc.

The How

What platform makes sense to host this? Obviously a robust platform that supports not just virtualization, but also containers for automation.

In an ideal world all of these components would also support federated login/user accounts.

Sure TrueNAS can install Emby and NextCloud… but can I manage users from one control panel? No. Commercial solutions such as NAS units from Synology go some ways to address this kind of thing, but often the actual implementation is a bit hacky.

One would hope that, after enough traction, we could just provide a pluggable auth module for the open source projects that are most popular with our community.

Awesome Self-Hosted

Someone brought this to my attention; it is awesome:

61 Likes

A post was split to a new topic: The Ultimate Home Server - Component: Home Automation

A post was split to a new topic: The Ultimate Home Server - Component: Home Security

A post was split to a new topic: The Ultimate Home Server - Component: Backups

A post was split to a new topic: The Ultimate Home Server - Component: Media Hosting

A post was split to a new topic: The Ultimate Home Server - Component: Knowledge Repository

Running an Unraid Home NAS with Plex, Nextcloud, Home Assistant, Roon, with the standard inclusions like Sonarr and Radarr have been using wireguard as my VPN thanks to the Level1Techsforum.

1 Like

So I’ve got two rackmount servers:

1 hosting a variety of containers including:

  • Traefik for reverse proxy. I actually hate Traefik, it’s way too complicated and abstracted and the documentation is atrocious. But it does work.
  • Home Assistant
  • OctoPrint
  • Synapse (which is federated and does support social login)
  • NextCloud - also hosting SubSonic (Music streaming)
  • BookStack (supports a variety of external auth)
  • BitWarden

The other server is hosting TrueNAS with a variety of SMB shares

Obviously, the biggest problems I have with this system is that it’s:

  1. All command line based (this may be solveable with something like cockpit)
  2. A nightmare to manage permissions and credentials across all these systems.

I’m in the process of trying to condense all of this down to a single Synology DS720+ because the rackmount servers are too noisy and take too much space that I don’t have at the moment. I’m trying to figure out what on earth makes sense there - do I fight against the Synology ecosystem and host everything in Docker still, or do I accept it and accept the limitations that come with it and hope that NextCloud can fill in most of the gaps?

My goal has always been one, single, unified system, and NextCloud does seem to come the closest to that. Maybe NextCloud + AD backend and then enable AD on all my home PCs and use that to login? But then that makes issues for my laptop when I’m not at my home network.

3 Likes

I can already tell this is going to be my favorite thread/series of all time. I think the first step is going to be creating a list of common requirements for each of these tasks so that when we start building out lists of software we can objectively compare which requirements each piece of software meets.

6 Likes

yes. and I for sure am not going to be able to do it alone. So lets’s build

12 Likes

aye aye cap’n

2 Likes

A post was split to a new topic: The Ultimate Home Server - Whats your Setup? (pics welcome)

Interesting question @wendell !

Background Experience (might be relevant)

I’ve been a Windows user for most of my life, occasionally dabbling with (shudder) Apple when needs must. I’ve also dabbled with Linux, sadly never for very long when I came across a severe issue. I’ve occasionally coded websites, tried some CMS but while they were easy to use, overall they just caused more hassle than they were worth, so I tended to return to HTML, PHP, etc. I did create a MySQL / PHP time sheet once, which I was quite pleased with, but the company I worked for back then didn’t want it because, “it might lose jobs in the company”. Hmm.

Onto servers, I was the backup guy for sometime, but I was never allowed to touch the actual server.

.

Who's responsible for me jumping into Server?

Well, it’s actually Wendell’s fault :slight_smile: Along with Lawrence Systems & Craft Computing.

.

What made me think of having a server?

Neither Windows or Apple implicitly admit when there’s been a disk issue, unless it completely dies, that disturbed me, as I have files that are critical and have an actual dollar value attached to them.

.

The quest starts (relating to Servers)

So I started looking into these server O/S’s that people mentioned, Byte My Bits mentioned UnRaid (along with a few other people) but after looking into it, I didn’t think it was a very good solution when it came to redundancy. Also, nothing that easy is worth having. So I wondered over to the (then) FreeNAS. The install went well, but of course I only used a USB drive and it was erratic and unstable…I lost heart. But then I found a spare SSD and installed on there…wow, it became bullet proof and 100% reliable.

.

A strange time to chose Free/TrueNAS

Of course FreeNAS turned to TrueNAS, then there was the SMR-gate, I landed at a peculiar time but for around a year I tinkered with TrueNAS, setting up lots of features that were really must have (like email notifying of issues, etc.)

.

What are these Jail things?

As I played with TrueNAS, I found these Jail things and wondered how helpful they were, at the same time I was thinking that my limited DVD / Blu-Ray collection would be better accessed without getting the disk out and having a player. Again, Byte My Bits mentioned Plex, I gave it a shot and I was very impressed with how it worked and soon started transferring my media, both entertainment and personal videos that I could share with other family members.

.

There are alternatives to Plex

I would like to try Emby and Jellyfin, especially as Self Hosted mentioned that the latter had progressed a great deal.

.

Dealing with files (SyncThing)

I’ve also been trying to figure out how I’m going to deal with files. Some are stagnant and only kept ‘just in case’, while others could really do with sync’ing of some sort. So I tried SyncThing - wow, what a decent piece of S/W. Installed on TrueNAS then on client machines and it works very well.

.

The Server config so far (not implemented yet)

So I’d like, but can’t necessarily justify having 3 servers:

TrueNAS 1 - 2 x Pools (5 x 4TB Iron wolf in RAIDZ2). One pool will have hard drives in standby mode as the data will only be accessed once a month or thereabouts. The other will be constantly spinning.

TrueNAS 2 - A collection of older hard drives that are equal in capacity to the above that works as a snapshot machine.

TrueNAS 3 - an off-site backup machine (well, in an outbuilding) that will provide me with my 3-2-1 approach. I did want to use Backblaze but as soon as you go from personal computer to NAS/Server, the price rockets up to around $800 dollars a year for my approx. 12-14TB.

.

Off-site access to files

This happens so rarely, so I’m keeping things simple and files are only accessible to the LAN only, though Plex does reach outside of that. I’ve got a laptop that’ll store live files via SyncThing, this includes examples in case I have to woo potential clients.

.

What I would like to try

I tried TrueNAS Scale and although it was familiar in places, I have no previous container/docker knowledge. I’ll play when time permitting as I’m sure it’ll eventually be a great alternative.

.

Smart Home Amateur

I started with some Govee gear, along with some Gosund Plugs. Then I found a website that sold pre-flashed Tasmota plugs that are Home Assistant friendly. I initially installed HA on my TrueNAS, but I didn’t really want to keep the server on just for that, so I bought 2 Raspberry Pi 3 B+. I put Unifi on one and Home Assistant on the other. They both work really well and I’m very slowly getting the hang of it.

3 Likes

A quick update finally I have had the best upgrade for my Home Server switching ISP’s gave me the chance to get a cheap Static IP well self hosting just got a boost.

1 Like

I’ve sort of been on a similar journey. I’m using UnRAID currently, mostly because it’s just so damned efficient when it comes to power utilization, but I hate how much of the management I have to do via the command line, as well as how much time I had to spend ‘working around’ UnRAID’s design…

For backups, it has a few useful features:

  • Borg backup - With the Vorta client now being functional on Mac, Windows, and Linux, it’s one option. Windows via WSL, which is kinda… Jank. But it does work.
  • I use borg for Linux, a TimeMachine share for Macs, and just a normal SMB share for Windows - basically trying to use whatever the manufacturer supplies for those which that applies, then borg for everything else
  • For phone backups, it’s basically ‘just photos’, and that gets handled by nextcloud.

Music / Media

  • Plex is unfortunately still the best solution I’ve found for multimedia. Emby, Jellyfin, they’re both good tools, but both have their own limitations (especially Jellyfin, much as I hate to admit it, and much as I keep rooting for them)
  • The 'arr stack for media management

File Hosting

  • Nextcloud for file sync / share - it’s just unavoidable as far as I’m concerned
  • Paperless-ng for adulting - really need a decent scanner for this one to truly show it’s worth imo… But once you add in a decent scanner, any time you get a receipt, a bill in the mail, whatever the case may be, you just scan it in and you’re done. As long as you’ve set up your tags properly in advance, they’re even automatically organized, and even if you haven’t, the fact that they’re all instantly searchable by text means that as long as you can remember just a single identifying word from the paper, you can find it. Biggest adulting time-saver I’ve found.

Booting bare metal as VMs

  • I’ve done this with three of my machines - tiniest bit of configuration after pulling their boot drives and throwing them in the server, now all three can be booted as VMs on the server itself and I’ve retired the hardware those drives came from. Added bonus of being able to image those drives locally to the server, much faster than a network backup (for me, without 10Gb everything hardwired lol)

Spash page -

  • Another project currently on my list - setting up a Heimdall instance with panels/links to each item someone visiting should have access to. You can set up a splash page in Engenius’ wifi system, but I’ve not gone down that rabbit hole yet; only just got the swtiches and AP’s in last week, but it’s on the agenda!
4 Likes

mostly just:

my media server

program for recursively comparing hashes and generating snapshots of mount points for historical comparison, so i can verify on any format/filesystem on local/remote whatever

working on version for only comparing file tree/file names as a faster way to verify the data sets are atleast in theory in place before wasting the time generating hashes

some small programs mostly for processing media to batch rename(for a given directory every file in the first level) insert/remove a string etc, remove unnecessary punctuation, brackets etc

I had great success lately with password management (I just had an Excel file before… yeah I know… not good). Vaultwarden a port of the Bitwarden server with more features and enhancement has been awesome.

3 Likes

I’ve heard a lot about this but I’ve been hesitant to give it access to my entire docs share.
Can I give it read-only access and use it primarily for searching files? Let it build its backend metadata database in a separate directory where it does have r/w permissions, but feed it the “docs” directory in a read-only manner?
This is how I use PhotoPrism (obviously for Photos, not documents) and it works wonderfully.

1 Like

The only real software I have on my server is an instance of Gitea. And I use it mostly to backup some .txt files.

The good thing about Gitea is that it is mostly ‘set and forget’. I like to play with computers but I don’t like having to babysit them.

Today I think the best upgrade for me would be some backup solution. This would let me keep working as I am used to and hopefully allow me synchronize data between devices.
When I did a short stint as a sysamin I implemented Bareos in the org was at. It was best solution, by far, that I could find and was free at the time(~2016). And the WebUI was kinda nice, albeit a little clunky to configure.

I just joined here based on the latest YouTube video, so I’m not sure the depth of what everybody gets into here.

Would it make sense to have a section for Management / Monitoring? Or maybe that’s getting too “enterprisey”.

I run a bunch of VMs, so I use things like:

  • CertBot to keep my SSL certs up to date
  • Zabbix for monitoring (mostly making sure VMs don’t run out of disk space)
  • Ansible for automating things (mostly applying updates and installing the Zabbix agent on new VMs).
2 Likes