Home Server Goals and Issues

Communities DIY Corner
1

Let’s start by talking about the project itself, in order to give some sort of more complete context:

I’m working up what I consider a “complete” home/family server for 10~20 users.

Goals:

  • Replace any web-service where “you the customer are in fact the product” (gmail and other google services, etc.)
  • Replace any web-service where one pays for “access” to content and thus one does not “own” content (netflix, iTunes, Audible, etc.)
  • Access from anywhere
  • Various work and automation services
  • Maintain sufficient security while retaining decent reasonableness of maintaining and using the server as a whole
  • Respect user privacy, as much as possible, even from Admin eyes

How I intend to achieve this:

  • Docker
  • More Docker, Compose is great
  • Yet more Docker and also ZFS
  • FOSS wherever possible

A partial list of Docker containers:

  • Organizr for that fantastic front-end integration and organization
  • Monitorr for at-a-glance awareness of whether things are running
  • Jellyfin
  • Ombi
  • Ubooquity (probably to be replaced with something else for reasons below)
  • Traefik
  • Trilium Notes
  • Airsonic
  • Booksonic
  • Komga
  • bitwarden_RS
  • Some sort of OpenLDAP/Keycloak solution
  • Backup for user files, dropbox replacement
  • Paperless document storage
  • Contacts, calendars, etc
  • A website or 3
  • Image upload, pastebin, URL truncation
  • Likely an IRC server or similar
  • Game servers
  • SIP is likely
  • RSS server
  • Financial tracking
  • Some IoT/home-automation stuff
  • Likely docker-mailserver, or at least some similar stack
  • Etc, Etc

(General network privacy stuff like PiHole, OpenVPN, etc, is handled elsewhere. and so it’s not relevant here).

I mentioned LDAP but also Ubooquity - this is the reason I’m likely to replace Ubooquity. It doesn’t seem to support any sort of SSO implementation as far I’ve seen and that’s a problem - SSO makes life so much easier on the user side that it’s more or less essential for most services I intend the server to run.

A further problem I’ve been having is that Docker seems to like to semi-regularly corrupt or screw up the configs/userdbs from various containers. I can manually back these things up but that’s a lot of extra effort, so I’m looking to automate it. I’ve attempted to troubleshoot this for ages, and I’m not really making progress. I’m not currently looking for a solution to that, but I am certainly seeking to band-aid it at this point, at least for now.

As a result, I have three interrelated questions. The questions:

What’s my best way to implement SSO for the server as a whole, including each / nearly each service within it?

I can replace things here and there, and that’s fine for most things. For some services, such as Jellyfin for example, there are no open source alternatives. With privacy being a priority, closed-source is a terrible idea!

What can I do to go from individual dbs/configs to integrating that into a much smaller list (I’m thinking “normal” stuff and “security-sensitive” stuff as my groups here) and how can I easily automate a backup for that smaller list of files/dirs/dbs?

I’m not finding any sort of genericizable db management tutorials for Docker online, most of the stuff is specific to x or y service in a way I can’t parse through to use with anything, and I’m really not sure what my simplest and easiest backup solution for this is as well.

And lastly, what is the deal with internal vs. external volumes?

I’ve received conflicting advice here and would appreciate some sort of authoritative answer. Obviously some things need to be external (content library dirs come to mind) for automation and ease of maintenance. At the same time though, I keep being told that “docker is unreliable at handling [internal/external] volumes” as the likely cause of my config/db corruption issue.

3 Likes
2

I like Keycloak for my homelab SSO needs. It supports SAML and OIDC. It works great with Nextcloud and Gitlab (which ironically, you didn’t mention but seem to be pretty popular in homelabs). There is a way to federate with LDAP but I don’t use that so I can’t speak to it.

You also might want to look into Kubernetes/Openshift if you are going to be running that many containers. Might make things easier to manage, especially if you version control you deployment configs.

Also, if you find a good financial management container, please let me know…

2 Likes
3

I like Keycloak, too, in theory, but the frustration I have is how to make things aware of it and function properly behind it. Can’t just have a single user account for Jellyfin, for example, after all.

2 Likes
4

Yeah, I’m pretty sure Jellyfin has an LDAP plugin which can federate with Keycloak, but then you’re really in the weeds

1 Like
5

There’s an open issue for this but there seems to be some confusion on how to handle this. Hopefully they figure this out

1 Like
6

wow learned about so many great softwares from that first post.

do you mean SIP as in phones? i was thinking about some some sort of SIP or POTS conversion.

rather a fancy answering machine to vet and take calls form our POTS line. maybe i could install lenny?

EDIT: jelly fin’s LDAP/AD integration is broken ATM too. it can’t create the config file needed to make the plugin work on windows host.

EDIT: i’m really hoping windows server 2019 will get docker for windows 2.0 soon. it would be cool to run docker stuff more natvley like lan cache.

otherwsie i’m going to install ubuntu server 20.04 LTS as a hyper-V VM and choose the docket install option and go nested like that. i think my current i7-3770T could handel it so my new ryzen 3400G ought to be up to it. almost tempted to try run linux as the host but, BlueIris has to be the way it is.

EDIT: why airsonic? jellyfin has music too?

EDIT: Komga has features over Ubooquitiy? like authors?

1 Like
7

Jellyfin LDAP: YUP bleh, waiting game.
Airsonic: Variety of clients! For example, Clementine is a better option than, say, Kodi for music on win/linux since it’s purpose-built to that. On a TV I might go for Kodi for everything.
Komga is mostly about separating out kinds of books so that, like Airsonic, it can use different clients on different devices.

1 Like
8

have you considered freeipa?

wow, you’ve been busy!
Where do you find a 20 bedroom house?
/jk

1 Like
9

Luckily many of them have their own homes! But “work and friends and family server” just isn’t as linguistically pleasant, is it? Eloquence :smiley:

1 Like
10

Updates as of midnight on the 1st?

Well sure.

Running:

  • Jellyfin
  • Ubooquity
  • Airsonic
  • Organizr
  • Monitorr
  • Ombi
  • FreshRSS (new)
  • Linkding (new)
  • Planka (this is for work GTD) (new)
  • Grocy (new)

Running Notes: I’m watching for Jellyfin and Airsonic to corrupt their DBs again. It keeps happening, but it has at least slowed down, which is nice. I somehow lost all the cover scans from my music in the transition to airsonic, and that pisses me off a little bit. Meanwhile, Monitorr cannot see Jellyfin, and I’ve yet to figure out why - it parses just about everything else just fine. I’m working on filling out Linkding, Planka, and Grocy. FreshRSS could not import the full data export from my old RSS service, only the subscriptions, so I’ve got to go through and mark a few hundred articles as read.

Issues:

  • Vikunja (this is for personal GTD)
  • Traefik

Issues Notes: Vikunja and Traefik are both failing regardless of what I do to them. Vikunja’s functionality is spread across multiple container images, and the problem lies therein somehow: I’m fairly convinced they’re not talking to eachother “properly”. Meanwhile, Traefik’s setup is insane and each attempt so far has made no progress toward a working solution. Particularly annoying, because one of the priorities I set was to have the internal security (which Traefik provides much of) set up and functional before I went and made all of this internet-capable / reachable from anywhere. This of course isn’t great for a good percentage of my users, though as I work from home, it’s less bothersome personally.

Next Implemented:

  • Booksonic and Komga, probably
  • Slimming down DBs to categories, rather than just having a pile
  • Daily script to stop all containers, backup all data, and start all containers.

Next Implemented Notes: I’m sick of pulling backups manually. I would do calendar and contacts next, instead of booksonic and komga, but that really feels like it needs the personal and work email stacks and the whole thing opened to internet, so it’s all waiting. As far as DBs go, if I understand things correctly, I can run just a handful (conceivably just 1 if I were insane), and just have multiple containers make use of each. Probably going to categorize them: Security stuff (SSO, certs, bitwarden, etc), content, work, personal, etc. That’ll require a test stack, something I haven’t bothered with yet.

Thoughts on the server as a whole:

Eh, frustration with Vikunja aside, the more that I implement, the more that I come to believe is practical, feasible, and should be implemented, and the more that I believe I want to implement. This idea started out as “replace g services, netflix, and spotify” and it’s becoming much more.

2 Likes
11

oooohhhh i’ve only accessed anything from the web so far. still waiting for a JF LG TV client app. what kind of clients should i be looking at?

EDIT: curious to see how the family likes Grocy!

EDIT: can’t google LinkDing without getting linkedIn

2 Likes
12

Here, take a look at Linkding. It’s fairly basic, but it gets the job done in a browser-agnostic manner. I haven’t explored multi-user with it yet, but assuming that doesn’t introduce much more complexity, it’s fine.

The 5 actually local users here will probably despise Grocy… at first. And then love it, once the automation for shopping lists kicks in! Nobody likes the back-and-forth “Do we have…” “Some” “What’s some and do we have…” conversation. And I can lock everyone else out of seeing the link with Organizr’s user-type scheme.

As far as clients, alright, here’s what I’m using so far:

Ubooquity: I’ve been reading on my Android phone, primarily, since I’m not buying a new reader device right now. I’m using Moon+ Reader. It’s… alright. That said, though, IIRC it’s just an OPDS thing, So any ebook app or ebook reader device (with browser) should be 100% capable of acting as a client.

Jellyfin: the options here are actually fairly wide-ranging, but as you said, I don’t think there’s an LG TV native option. That said, I wouldn’t trust the TV anyway. My handful of Apple users seem to prefer MrMC but I like the Android app just fine for my phone, Kodi generally, and web on desktop.

Airsonic: As mentioned, I’ve been using Clementine. But there’s a great variety of options here and here. This may very well figure into my IoT/Home-Automation/Voice-Assistant stages in this project later (I’d much rather pay for nice speakers in the kitchen, than a nice TV in the kitchen, for example, since I think there’s far more use in a small tablet and good audio in there than a big screen).

On a side note, this is becoming a blog. I’m alright with this. :stuck_out_tongue:

2 Likes
13

Hey I have code to share. This backup batch script isn’t my work, K (one of my two girlfriends) kindly write this. Though it does seem to be the simplest way to do this, and I think it’s sufficiently scalable.

 @echo OFF

::Setup here to make things easier to read and change
set YEAR=%DATE:~10%
set MONTH=%DATE:~4,2%
set DAY=%DATE:~7,2%
set BACKUP=D:\ServerBackups
set FILENAME=%YEAR%-%MONTH%-%DAY%

::Set our list of containers in a more human readable format
set CONTAINERS=^
 organizr^
 monitorr^
 ombi^
 trilium^
 jellyfin^
 ubooquity^
 airsonic^
 planka planka-db^
 freshrss^
 grocy^
 linkding

::Empty line above in case of extra carat
::Stop the container, using our list from above
docker-compose stop -t 180 %CONTAINERS%
::Creates a zip containing the contents of the current folder in %BACKUP%
tar -c -f %BACKUP%\%FILENAME%.tar .
::Removes the oldest files in %BACKUP%
FOR /f "skip=7" %%A IN ('dir %BACKUP% /B /O-D') do del %BACKUP%\%%A
docker-compose up

Edit: It’s broken, sorry. I’ll post the powershell replacement she wrote and my sanitized docker compose later (yay .envs).

14

Now Planka is misbehaving.

Synchronous XMLHttpRequest on the main thread is deprecated because of its detrimental effects to the end user’s experience. For more help http://xhr.spec.whatwg.org/ lpfulllib.js:1:35437
downloadable font: kern: Too large subtable (font-family: "Open Sans" style:normal weight:600 stretch:100 src index:1) source: moz-extension://20c99096-be2c-478b-b73c-8eec07644ae9/fonts/opensans/Semibold/OpenSans-Semibold.woff2?v=1.1.0
downloadable font: Table discarded (font-family: "Open Sans" style:normal weight:600 stretch:100 src index:1) source: moz-extension://20c99096-be2c-478b-b73c-8eec07644ae9/fonts/opensans/Semibold/OpenSans-Semibold.woff2?v=1.1.0
Firefox can’t establish a connection to the server at ws://localhost:1337/socket.io/?__sails_io_sdk_version=1.2.1&__sails_io_sdk_platform=node&__sails_io_sdk_language=javascript&EIO=3&transport=websocket. websocket.js:124:12
The connection to ws://localhost:1337/socket.io/?__sails_io_sdk_version=1.2.1&__sails_io_sdk_platform=node&__sails_io_sdk_language=javascript&EIO=3&transport=websocket was interrupted while the page was loading. websocket.js:124:12
Firefox can’t establish a connection to the server at ws://localhost:1337/socket.io/?__sails_io_sdk_version=1.2.1&__sails_io_sdk_platform=node&__sails_io_sdk_language=javascript&EIO=3&transport=websocket. websocket.js:124:12
The connection to ws://localhost:1337/socket.io/?__sails_io_sdk_version=1.2.1&__sails_io_sdk_platform=node&__sails_io_sdk_language=javascript&EIO=3&transport=websocket was interrupted while the page was loading.

This is what Firefox spits out in web console after login - the splash login works just fine, then it just gives a spinny “loading” icon.

websocket.js:124 WebSocket connection to 'ws://localhost:1337/socket.io/?__sails_io_sdk_version=1.2.1&__sails_io_sdk_platform=node&__sails_io_sdk_language=javascript&EIO=3&transport=websocket' failed: Error during WebSocket handshake: Unexpected response code: 400

Seems to be docker-side, not browser-side.

15

i planned on letting my TV’s NIC sit and rot buuutt OLED settings aren’t ALL available for a PC input so only certain picture adjustments are available ONLY in the god damn apps! but, hey you wouldn’t want frame interpolation when playing Doom 2016 or running over hookers in a stolen SUV.

in the future i plan to block some LG IPs and stuff.

thanks for the links! I for one WELCOME this new blog! i haven’t heard of almost ANY of these software and i want many of them!!! Wendell has been slacking!!!

4h17zw

1 Like
16

All it needs is like a google and netflix logo on the damn fruit. I love it.

1 Like
17

do you mean SIP as in phones? i was thinking about some some sort of SIP or POTS conversion.

I never got back to this, but yes, I mean SIP as in phones. I don’t see any reason for me to maintain a pair (personal/family and work) or even three landlines on top of my cell phone. I’d really rather just have QoS rules on the router to ensure that the phones take absolutely priority, and order one of those nice desksets that can handle multiple lines for my desk, only place they have to cross together like that.

You can find the majority of the SIP options (and in fact a majority chunk of everything else!) here. And this is another excellent resource I’ve found in the same vein.

18

i was looking at SQM but, i’d probably make phones one of the top priorities but, not THE top priority but, hey gaming. pretty sure voip has to be up there or else tooo much delay and it gets garbled. one reason why i run TeamSpeak3, yeah discord is free but, TS3 lets you choose and tweak codecs!!! not SIP though. think i’m more interested in an automated answering machine with Lenny. However a nice voip handset would be pretty cool

19

I have solved this issue, at least for the time being. This turns out to be precisely the same issue I was having with Vikunja, though naturally expressed differently.

Ultimately, it goes like this:

If you have a service that depends on another service in an inherent manner, like frontend+backend or frontend+backend+api, built to work together as parts of a whole, you have two options.

  1. Do some hacky stuff with Docker Compose, particularly CLI arguments, and have a pretty good chance of it just being broken randomly.
  2. Actually run the damn reverse proxy.

I’ve stopped attempting 1 and have implemented Traefik into the stack. As a result, most of these networking issues appear to be fading away with the proper implementations of the traefik.toml, various .envs, and labels in the Docker Compose YAML.

This does not seem to effect “service + DB” setups nearly as much, though I’m not deep enough into any sort of code to be 100% on that.

So I’ve learned:

  • Favor single-container or container+db-container solutions where feasible, because networking between containers is sometimes wonky. (Of course, feature set will always matter! We don’t want monolothic software but we don’t want do-one-thing-in-one-way incompetence either!)
  • Traefik becomes more indispensable as the server stack continues to grow! (but also, run it with tecnativa’s docker socket proxy for unrelated reasons)
  • Further lesson learned: it’s delightfully easy to sanitize the compose YAML for public consumption when you’re using both Traefik and .env files.

Now, I promised the powershell script and the docker compose. Gonna use pastebin for now, but pretty soon I’ll have a pastelink service of my own on this very server! Let’s hope mods/admin here don’t mind if I use that.

First, the Docker Compose YAML. This may seem a little odd, with no HTTPS/SSL/SSO/Authelia/Letsencrypt/etc, but I have yet to feel ready to open the server to connections from the wider internet. As such, none of that stuff would have any meaning (yet - we’ll get there).

Here’s the Powershell backup script. . I actually really like this one, especially because it’s easily extensible, and so far, it seems to be working.

What’s coming next?

  • Basic net tools: img share, pastebin, link truncation. Especially if any of these come with, or have as addon, any sort of bookmarklet options.
  • Possibly a web assistant, something I can tack a voice assistant onto later.
  • Grocy of course still needs tons of set up. That has to happen before I can comment on results at all.
  • Some work stuff. One of the priorities at this stage is completely replacing Wave (I trust none of the financial services companies, but you have to use at least 1 bank and Paypal to be in reputable business online today ugh), so I need all those features. Soo… Akaunting, at least to evaluate.
  • I know I need some sort of monitoring and notifications beyond just Monitorr, so that’s probably going on the list of ‘next steps’. But I am thinking may have to hold that back until I have a chat solution implemented that fits the requirements, so it has a primary destination for alerts/notifications/
20

Yes! SQM, AQM, QOS, doesn’t much matter which term we use. This is going to be particularly important for me because I am on (and likely to stay on) an internet service where both upload and download can usually be measured in single-digit Mbps.

One of the consequences of living out here (but on the edge of being off-topic).