The Ultimate Home Server - Toward an Amalgamation

So I’ve got two rackmount servers:

1 hosting a variety of containers including:

  • Traefik for reverse proxy. I actually hate Traefik, it’s way too complicated and abstracted and the documentation is atrocious. But it does work.
  • Home Assistant
  • OctoPrint
  • Synapse (which is federated and does support social login)
  • NextCloud - also hosting SubSonic (Music streaming)
  • BookStack (supports a variety of external auth)
  • BitWarden

The other server is hosting TrueNAS with a variety of SMB shares

Obviously, the biggest problems I have with this system is that it’s:

  1. All command line based (this may be solveable with something like cockpit)
  2. A nightmare to manage permissions and credentials across all these systems.

I’m in the process of trying to condense all of this down to a single Synology DS720+ because the rackmount servers are too noisy and take too much space that I don’t have at the moment. I’m trying to figure out what on earth makes sense there - do I fight against the Synology ecosystem and host everything in Docker still, or do I accept it and accept the limitations that come with it and hope that NextCloud can fill in most of the gaps?

My goal has always been one, single, unified system, and NextCloud does seem to come the closest to that. Maybe NextCloud + AD backend and then enable AD on all my home PCs and use that to login? But then that makes issues for my laptop when I’m not at my home network.

3 Likes

I can already tell this is going to be my favorite thread/series of all time. I think the first step is going to be creating a list of common requirements for each of these tasks so that when we start building out lists of software we can objectively compare which requirements each piece of software meets.

6 Likes

yes. and I for sure am not going to be able to do it alone. So lets’s build

12 Likes

aye aye cap’n

2 Likes

A post was split to a new topic: The Ultimate Home Server - Whats your Setup? (pics welcome)

Interesting question @wendell !

Background Experience (might be relevant)

I’ve been a Windows user for most of my life, occasionally dabbling with (shudder) Apple when needs must. I’ve also dabbled with Linux, sadly never for very long when I came across a severe issue. I’ve occasionally coded websites, tried some CMS but while they were easy to use, overall they just caused more hassle than they were worth, so I tended to return to HTML, PHP, etc. I did create a MySQL / PHP time sheet once, which I was quite pleased with, but the company I worked for back then didn’t want it because, “it might lose jobs in the company”. Hmm.

Onto servers, I was the backup guy for sometime, but I was never allowed to touch the actual server.

.

Who's responsible for me jumping into Server?

Well, it’s actually Wendell’s fault :slight_smile: Along with Lawrence Systems & Craft Computing.

.

What made me think of having a server?

Neither Windows or Apple implicitly admit when there’s been a disk issue, unless it completely dies, that disturbed me, as I have files that are critical and have an actual dollar value attached to them.

.

The quest starts (relating to Servers)

So I started looking into these server O/S’s that people mentioned, Byte My Bits mentioned UnRaid (along with a few other people) but after looking into it, I didn’t think it was a very good solution when it came to redundancy. Also, nothing that easy is worth having. So I wondered over to the (then) FreeNAS. The install went well, but of course I only used a USB drive and it was erratic and unstable…I lost heart. But then I found a spare SSD and installed on there…wow, it became bullet proof and 100% reliable.

.

A strange time to chose Free/TrueNAS

Of course FreeNAS turned to TrueNAS, then there was the SMR-gate, I landed at a peculiar time but for around a year I tinkered with TrueNAS, setting up lots of features that were really must have (like email notifying of issues, etc.)

.

What are these Jail things?

As I played with TrueNAS, I found these Jail things and wondered how helpful they were, at the same time I was thinking that my limited DVD / Blu-Ray collection would be better accessed without getting the disk out and having a player. Again, Byte My Bits mentioned Plex, I gave it a shot and I was very impressed with how it worked and soon started transferring my media, both entertainment and personal videos that I could share with other family members.

.

There are alternatives to Plex

I would like to try Emby and Jellyfin, especially as Self Hosted mentioned that the latter had progressed a great deal.

.

Dealing with files (SyncThing)

I’ve also been trying to figure out how I’m going to deal with files. Some are stagnant and only kept ‘just in case’, while others could really do with sync’ing of some sort. So I tried SyncThing - wow, what a decent piece of S/W. Installed on TrueNAS then on client machines and it works very well.

.

The Server config so far (not implemented yet)

So I’d like, but can’t necessarily justify having 3 servers:

TrueNAS 1 - 2 x Pools (5 x 4TB Iron wolf in RAIDZ2). One pool will have hard drives in standby mode as the data will only be accessed once a month or thereabouts. The other will be constantly spinning.

TrueNAS 2 - A collection of older hard drives that are equal in capacity to the above that works as a snapshot machine.

TrueNAS 3 - an off-site backup machine (well, in an outbuilding) that will provide me with my 3-2-1 approach. I did want to use Backblaze but as soon as you go from personal computer to NAS/Server, the price rockets up to around $800 dollars a year for my approx. 12-14TB.

.

Off-site access to files

This happens so rarely, so I’m keeping things simple and files are only accessible to the LAN only, though Plex does reach outside of that. I’ve got a laptop that’ll store live files via SyncThing, this includes examples in case I have to woo potential clients.

.

What I would like to try

I tried TrueNAS Scale and although it was familiar in places, I have no previous container/docker knowledge. I’ll play when time permitting as I’m sure it’ll eventually be a great alternative.

.

Smart Home Amateur

I started with some Govee gear, along with some Gosund Plugs. Then I found a website that sold pre-flashed Tasmota plugs that are Home Assistant friendly. I initially installed HA on my TrueNAS, but I didn’t really want to keep the server on just for that, so I bought 2 Raspberry Pi 3 B+. I put Unifi on one and Home Assistant on the other. They both work really well and I’m very slowly getting the hang of it.

3 Likes

A quick update finally I have had the best upgrade for my Home Server switching ISP’s gave me the chance to get a cheap Static IP well self hosting just got a boost.

1 Like

I’ve sort of been on a similar journey. I’m using UnRAID currently, mostly because it’s just so damned efficient when it comes to power utilization, but I hate how much of the management I have to do via the command line, as well as how much time I had to spend ‘working around’ UnRAID’s design…

For backups, it has a few useful features:

  • Borg backup - With the Vorta client now being functional on Mac, Windows, and Linux, it’s one option. Windows via WSL, which is kinda… Jank. But it does work.
  • I use borg for Linux, a TimeMachine share for Macs, and just a normal SMB share for Windows - basically trying to use whatever the manufacturer supplies for those which that applies, then borg for everything else
  • For phone backups, it’s basically ‘just photos’, and that gets handled by nextcloud.

Music / Media

  • Plex is unfortunately still the best solution I’ve found for multimedia. Emby, Jellyfin, they’re both good tools, but both have their own limitations (especially Jellyfin, much as I hate to admit it, and much as I keep rooting for them)
  • The 'arr stack for media management

File Hosting

  • Nextcloud for file sync / share - it’s just unavoidable as far as I’m concerned
  • Paperless-ng for adulting - really need a decent scanner for this one to truly show it’s worth imo… But once you add in a decent scanner, any time you get a receipt, a bill in the mail, whatever the case may be, you just scan it in and you’re done. As long as you’ve set up your tags properly in advance, they’re even automatically organized, and even if you haven’t, the fact that they’re all instantly searchable by text means that as long as you can remember just a single identifying word from the paper, you can find it. Biggest adulting time-saver I’ve found.

Booting bare metal as VMs

  • I’ve done this with three of my machines - tiniest bit of configuration after pulling their boot drives and throwing them in the server, now all three can be booted as VMs on the server itself and I’ve retired the hardware those drives came from. Added bonus of being able to image those drives locally to the server, much faster than a network backup (for me, without 10Gb everything hardwired lol)

Spash page -

  • Another project currently on my list - setting up a Heimdall instance with panels/links to each item someone visiting should have access to. You can set up a splash page in Engenius’ wifi system, but I’ve not gone down that rabbit hole yet; only just got the swtiches and AP’s in last week, but it’s on the agenda!
4 Likes

mostly just:

my media server

program for recursively comparing hashes and generating snapshots of mount points for historical comparison, so i can verify on any format/filesystem on local/remote whatever

working on version for only comparing file tree/file names as a faster way to verify the data sets are atleast in theory in place before wasting the time generating hashes

some small programs mostly for processing media to batch rename(for a given directory every file in the first level) insert/remove a string etc, remove unnecessary punctuation, brackets etc

I had great success lately with password management (I just had an Excel file before… yeah I know… not good). Vaultwarden a port of the Bitwarden server with more features and enhancement has been awesome.

3 Likes

I’ve heard a lot about this but I’ve been hesitant to give it access to my entire docs share.
Can I give it read-only access and use it primarily for searching files? Let it build its backend metadata database in a separate directory where it does have r/w permissions, but feed it the “docs” directory in a read-only manner?
This is how I use PhotoPrism (obviously for Photos, not documents) and it works wonderfully.

1 Like

The only real software I have on my server is an instance of Gitea. And I use it mostly to backup some .txt files.

The good thing about Gitea is that it is mostly ‘set and forget’. I like to play with computers but I don’t like having to babysit them.

Today I think the best upgrade for me would be some backup solution. This would let me keep working as I am used to and hopefully allow me synchronize data between devices.
When I did a short stint as a sysamin I implemented Bareos in the org was at. It was best solution, by far, that I could find and was free at the time(~2016). And the WebUI was kinda nice, albeit a little clunky to configure.

I just joined here based on the latest YouTube video, so I’m not sure the depth of what everybody gets into here.

Would it make sense to have a section for Management / Monitoring? Or maybe that’s getting too “enterprisey”.

I run a bunch of VMs, so I use things like:

  • CertBot to keep my SSL certs up to date
  • Zabbix for monitoring (mostly making sure VMs don’t run out of disk space)
  • Ansible for automating things (mostly applying updates and installing the Zabbix agent on new VMs).
2 Likes

After looking at my last electric bill, something that might be worth considering in the “what”/“nice to have” category is whether 24/7 access(low power) is required or not which might lead to 2 servers for some folks. It will be different for everyone but for instance I don’t need financial records or CAD drawings 24/7 but I might want audio-books and obviously home automation 24/7. Maybe a “reliable” wake-on-lan server with an equally reliable sleep function might fit the bill.

3 Likes

One software package I personally find quite useful is Mediawiki, which is the wiki engine that runs Wikipedia. I find it quite a natural fit for accomodating lots of otherwise sparse and disjointed information in one place, and hyperlinking/categorising the hell out of everything.

Some examples of open source projects using this to track collective know-how is the sunxi project (Allwinner chipset support for Linux), and the elinux wiki, which is a collection of useful info regarding running Linux in an embedded context.

2 Likes

I’ll generally have a many-core Debian Linux server of whatever’s dirt cheap to run. Currently an old Bulldozer that spins up local NFS and CIFS shares, HTTPS, plus some SSH secured remote services. Probably more I’m not remembering.

It’d be nice if there was something easier than that but I haven’t run across it yet.

I do like running Part Keepr as a webserver VM to organize my electronic components. It’s a PHP and webserver based software. I pushed up a pre-built VM if anyone wants to use it (or reference what software versions an install needs).

Edit: and KODI! Since my media’s just a well organized network-transparent share to begin with. I like it.

1 Like

automagic file sorting and such would be handy.

doc/xls/ppt/odt/etc could be considered “editable” documents, where as pdf’s would be documents to keep/file. The suck thing about pdf’s is they can have embedded text or be basically just jpegs of the original docuument, so they’d have to be ocr’d for easier sorting. something like gpdfreader works but its finnicky for getting the settings dialed so it isn’t trying to ocr a company logo or some shit. password protected archives could be considered backups. images/music/videos all have metadata that make them easy to sort/categorize. Musicbrainz can be used to organize/clean up tags on audio files fairly easily and accurately.

should the sorting be automatic or a cron job or something the user should have a button/right click option (in folder, right click, “sort this shit out”, automagically sorts files into appropriate folder structure.

Desktop search utilities have been done a million times, and i’ve never really found anything that works exactly how i want it. indexing/sorting/etc, documents with searchable snippets as metadata are sort of useful, but not all filetypes have those capabilities.

Also naming conventions can be another spaghetti bowl to untangle.

I remember several years ago google had made some sort of magical tool that can clean up massive data sets and correct minor differences (initial case, camel case, abbreviations (road vs rd) typo/trasposition) to have things in neater piles.

also in mixed platform setups (windows/linux) even stuff like backups can be a bitch because of file permissions and fs differences.

I don’t think there are enough fingers in a full football stadium to count all the times i’ve used linux to copy stuff to and from NTFS volumes to avoid permissions and hours of setting/changing permissions just to move shit from one drive to another, root said move, files fucking move.

There needs to be a standardized server base, and modular plugins bolted onto it for each functionality. NAS module, Backup module, Audio/video server, mobile sync module that handles ping things out to pants computers, book server, home automation system, firewall/vpn, maybe even something cool like a chat/federated file exchange thing.

The problem with having every module “containerized” is it sidesteps the common base and allows for breakage. The whole reason the google/microsoft/apple ecosystems are so damned useful and hard to avoid is the integrations from being on a common base. Gmail can see/do things involving your calendar, it can see the stuff you’ve bought on google pay, if you want to attach a file in an email it can just grab it from google drive instead of having to find out which drive it is saved on and upload it, instead of having to sneakernet a file over to someone or fuck with bluetooth/adhoc wifi transfer (and finding a program both people have on their phones) you just send them the link from google drive.

Convenience through integration.

the wheel doesn’t have to be reinvented. Not long ago i basically had a FTP server at home, because pretty much everything from my phone to my laptop, and probably even a toaster if connected to the internet can interact with FTP… Massively insecure, old as fuck, but the reason you can still get computers with serial ports/serial port cards/usb to serial adapters is because it works.

fugly isn’t always a dealbreaker, but broken/inconvenient always is.

idk, brain dumping ideas here.

also, on notes… i feel your pain.

I almost want something like terry davis had in temple OS. he had some sort of weird thing where you could have like images and 3d models directly in text files. i know terry was insane, but there were some nuggets in temple OS that were genuinely cool ideas.

I think in pictures and objects… Having a document format where you could have text/images/3d models/etc in one document would be handy for fleshing out ideas.

an interface somewhat like Mspaint/gimp drawing & measuring, simple 3d modeling, handwriting recognition, a graphing calculator, and an embedded python console or some sort of basic language to step through/automate things within your notes, excel-esque functionality baked in as well, some flowcharting capability would be nice, able to import images to trace or annotate. the ability to file away links/snippets of text and images from anywhere for later use. also being able to embed or link to files and manipulate them from within that interface (sifting through a csv file full of data thats analyzed and can be used as an object in the document) would be super handy.

google keep is barebones, but is easy enough to dump things into and do minor edits, but still super lacking. onenote is really nice and fairly close, but its still somewhat off and being tied to office kind of makes it an afterthought.

a proper notebook/brain dumping tool utility should be its own thing and not just a freebie tacked onto another suite.

onenotes on mega steroids…

2 Likes

I don’t have any server solutions, but have apps I can suggest.

I used to recommend notability before they turned into a f2p SaaS app. They had the not only the ability to sync audio with handwritting, but also the ability to apply OCR on handwriting (making it very easy to search through documents).

Recently I’ve heard of Xournal++, which has similar features. I might give it an install since my work doesn’t allow us to use personal devices

Currently I use marktext for all my typing notes. It is markdown, open source, and wysiwyg. If you want something more powerful than notepad, but lighter than word, I highly recommend it.
https://marktext.app/

On mobile, I use quillnote. It is my replacement for google keep. Since I use markdown for everything, it’s a nice feature. Furthermore, it also has an experimental nextcloud sync feature

This has more of a database/wiki format

https://wiki.gnome.org/action/show/Apps/Gnote?action=show&redirect=Gnote

Both of which I have heard of from this video

1 Like

That is a great point of conversation, the real cost of running a home server is actually the power.

I am half way though posting my home setup details, but I have done the math several times and it is something anyone getting a setup started should consider.

I have 2 servers, a HP 24 port PoE managed switch and a ubiquiti USG Pro. Attached to the switch is 1x wifi AP and 3x CCTV cameras.

This setup draws about 700 watt 24/7. I live in Australia and power is quite pricy at about 0.34c per KW/h. This comes to around $2000 aussie per year just to keep the lights on on quite a simple and small home lab.

Can’t wait to see others annual cost in power, just to make my self feel better!

2 Likes

I have one main box (truenas) and run everything else in docker on Pi’s or the app has its own dedicated Pi

Storage
• TrueNas
• But with openzfs on linux now, I have been considering just running a linux box.

Media
• Plex is still my go to - because of the wife approval
• I do use Kodi as a backup to Plex
• Emby does look good and will be checking it out
• Ebook I’m just using Calibre the app (can make notes etc) and Calibre-web on the go (running on a PI)

File hosting
• Nextcloud on a Pi.

VPN’s
• Running OpenVPN on a Pi.

The single pain of glass to access it all
• Homer - I like how simple it is to add new links etc.

Knowledge repo
• I’ve been using Joplin for about 2 years now, and I use it in my job. I just use a private repo to pull and push the changes.

Backups
• I only backup the data on my truenas, everything else I use ansible for with a private git repo with all the dot files etc and the apps are in docker with compose files.

For blow away vms etc I’ve been playing with Kasm and it been a good experience so far

This is a very interesting question, definitely got me thinking this morning. I think it will depend on how much you want to tinker and how much control you want.

I think my ultimate set up would be a linux distro for the base and storage with zfs, with containers for my apps and something if not cockpit for a user friend way to manage it all - which just describes truenas scale, BUT gives me full control.

1 Like