Multi-homelab design help

Operating Systems & Open Source
, ,
1

Hi, I’m looking to get some advice regarding a setup I want to have between multiple homes. My intention is to have nothing exposed to the internet, and everything wireguarded up. The VPS would have specific firewall rules for our home networks. There’s a few ms between homes, and ~35ms to the VPS.

Each home has its own NAS with the same capacity, so I would also like some advice if I were to synchronize say a folder between them(Is glusterfs still a thing people use? Should I just learn ceph for such a small use case?)
I already have the wiring up between the two homes, and from an external VPS(meant mainly as a frontend, it’s a bit underpowered) to each home. I am interested in hosting some services for both homes to access, with our NASes as backing storage. I’m looking for simpler stuff like nextcloud, immich, jellyfin, maybe LLDAP if I want to scale it further?

I dipped my toes in kubernetes a while ago, but it seems a bit overkill? I mostly just want to cluster up a few machines have the flexibility of each home running independent updates and maintenance without causing downtime to its own house or the other. I have portainer on a raspberry pi currently in my own home for my own stuff, and it seems to do a pretty good job, so it could probably cluster pretty well too, but I haven’t tried that feature.

I also am wondering if I can have a networking model/load balancer where depending on which house calls a service, it will try to prioritize the instance that’s in the same house and not route everything through the external VPS after the initial request.

I have not flexed my devops muscles in a while and I remember(5 years ago ish?) that technologies would come and go extremely fast as nothing was really proven, and instead of standardizing stuff people just had arbitrary proprietary kubernetes integrations that would make things easier to deploy, but way harder to understand what was actually going on.

Has anyone tried anything similar, had a use case like that at work or at home? Any feedback on possible choices? I want to balance something that is a learning experience and that would save toil in the long run with something that makes sense in the home.

Am I asking for too much? If yes, where should I compromise?

Leave your opinion and/or advice below!
Thanks.

2

Why not something like Tailscale to skip the middle man and potential insecurity of VPS?

3

Isn’t tailscale literally wireguard with different configuration files?

Also the VPS is firewalled off completely from non-home networks, it’s mostly there for uptime/external load-balancing.

1 Like
4

wow you reply fast… I was still editing lol

Consider using Proxmox VE+ Containers for high availability purposes? It has built in ceph integration.

I think Tailscale also solves this problem for you.

5

These are meant to be run on the same network and are latency sensitive. You can’t do that over the internet with them. Use ZFS send to replicate data sets.

You’d need some kind of replicated active-active setup with syncing between them. I doubt there’s such a solution for homeprod services, like jellyfin or others. You’re better-off hosting individual instances on each house and connect to them using a same hostname under a different subdomain (like jellyfin.alice-house.local and jellyfin.bob-house.local).

Kinda, but not quite? Tailscale uses the wireguard protocol for connecting 2 (or more) sites directly to each other + its own centralized authentication and coordination service (which is why headscale exists as an alternative). Your tailscale client authenticates with some credentials that will tell your tailscale network to add your client on the wireguard network.

In a 2 client configuration (your 2 houses example), your house’s tailscale client will first connect to tailscale servers. Since there’s nothing else there, nothing really happens. The other house’s tailscale client will now connect 2nd. It authenticates to tailscale servers and now the magic happens. The tailscale servers coordinate each client to connect to the other, by passing their public IPs and trying to use all available options to connect each other: ipv6, NAT-PMP, port-forwarding if defined manually etc. Once a tailscale client connects to the other client, a wireguard tunnel is established and will continue to live on without intervention. If a disconnect happens, tailscale servers will be contacted again to facilitate the reconnect (like if your ipv4 address was released and renewed or one site went offline or whatever else).

I personally like port-forwarding and DynDNS, because it doesn’t involve some 3rd party servers (well, technically dynamic dns does, but it’s only name resolution, not trying to brute-force poke holes through my network to connect my sites). Sometimes port-forwarding isn’t possible though (carrier-grade NAT for example), in which case a middle-man VPS might be justified. Tailscale still works on certain highly-restrictive setup (like the aforementioned carrier-grade NAT), but might take a couple of minutes to maybe a couple of hours to connect.

Check out Wendell’s article on tailscale.

1 Like
6

Mmm, seeing tailscale getting blanket recommended is becoming disturbing! While they have open sourced parts not all of the product. I personally still have concerns with tailscale, its a commercial product, that uses the freemium model, in which case weather your paying or not, you best believe your are the product. Sort of waiting for the other shoe to drop, it never fails to with freemium products at some point in their lifetime.

If you want a private vpn service, setup your own wireguard or openvpn its not that hard, mostly if anything it’s just tedious to not have it automagicly setup.

7

Agreed.

Same.

For once, I don’t agree with this. I believe the free part of it is to gain market adoption. Once it reaches a critical mass and becomes an established trusted option, more people will start paying for the premium subscription that has multiple accounts and more peers. Once it becomes a big name, the company can be sold for a couple million dollars and make the founder a fortune.

It’s a common trope played with many startups. This is literally a market in se. People are looking to get rich. How do you do that? Find a thing that people want / need, or invent something that nobody thought they needed / wanted. You don’t have a lot of investment capital to become a huge player and huge players are not going to invest into a risky business that might not succeed. So you get some loans or venture capital, grow your business enough to show what potential it would have and cash in early by selling to a larger company. Rinse and repeat. Some good example are github, skype and whatsapp. Sometimes you get lucky and become large yourself (like fitbit, but then it got bought by google and sometimes you lose and file for bankruptcy, like pebble).

Large corporations are known to be abusive to their users and extract massive amounts of wealth even from free services. Startups have a clean record and are generally good hosts for their users (otherwise their reputation plummets and they can’t sell well later). Both large corporations and startup founders have a lot to gain from this symbiotic relationship.

1 Like
8

OK remember company’s buy startups with large user bases, for one reason the users. If a startup is selling itself its really selling its users. Ergo you are the product being sold to the bigger company. Your argument supports what i said. A larger company buying tailscale and taking the product in an unfavorable direction to its user base and charging for something that was once free, after running off competition such that there are less options, is bad, and is the other shoe dropping.

What tailscale is doing is nothing new(from my viewpoint), i know of many companies over the years that sold or did the fremium model with similar vpn products (Cisco, GoTo(formerly logmein)… ). Tailscale is merely the new kid on the block doing it with the newest open source vpn software wireguard.

9

Open source folks are an opinionated bunch. You can be assured that if that happens people will fork it and I shall move according (pfSense → OPNsense comes to mind).

For now, Tailscale is fine.

10

This really has nothing do do with how open or closed tailscale is, it has more to do with how simple wiregaurd is to setup, see wireguard setup instructions, and the fact that tailscale is a freemium product. They need to link via vpn 3 servers(4 if they use a vps as thier talking about), it would take about 5mins to do with wireguard. Once setup, there really would be not much need to mess with the vpn for possibly years.

So in this case:

  1. why use a commercial fremeium product to use wireguard to vpn tunnel where the free tier can be rug pulled at any point, breaking your setup?
  2. In a server environment where you have no gui anyway, what are the advantages of the tailscale gui and simple user interface?

Your right it is fine, but this should not earn it the blanket recommendation in every case. Wireguard by itself is just as fine.

11

The wireguard setup is done actually, There is a site to site between the houses, and the server has two connections, one for each house. The fact that mikrotik now has a native wireguard UI really helps :slight_smile:

12

Not necessarily. They buy them for their technologies and patents as well. Was Nuvia a large company when Qualcom acquired it? Nokia, when going out of business, sold its phone division to Microsoft, when arguably almost nobody had Nokia phones anymore. Pebble sold its OS and software developers to Fitbit. I could go on, but it’s not always about the users.

There’s a lot of stigma to this. When people refer to “you are the product,” almost always it’s an association to user data mining and selling, sometimes close to fraud (like collecting your photos on your device to blackmail you or keylogging to get your SSN or bank info etc. - although obviously those fraudulent data mining operations are a minority, compared to what FAANG, info-brokers and similar wanna-be large conglomerates do).

In case of Tailscale, the users aren’t the product (at least not yet anyway), they’re a means to an end for the company. I hate to be the one defending Tailscale, I generally don’t even recommend them and encourage people to wireguard, port-forward and DynDNS, but what you’re saying does not match reality, so I’m forced to do it.

The way tailscale works allows for a very large number of users. It’s basically the ideal setup for a small team raking in millions of $$ (once they reach a large adoption). The only purpose of the tailscale servers is coordination between clients and the clients connect directly to each other. That means there’s no user data going through the tailscale servers (saving them tons of bandwidth) and the only traffic they’re getting is a few KB per device only when initiating a tunnel (I’m not entirely sure about it being only on establishing connection, but I’m absolutely sure the amount of data they get, given their design, is tiny).

Even their authentication servers are outsourced to github, google and whoever else, so they don’t even host user credentials. It’s pretty genius how little they can get away with doing.

Agreed, but so far it’s been blowing the old vanguard out of the water with their product (because of the new stack, not because they did anything revolutionary). What they did great, was like mentioned before, avoid doing as many things as possible and only focusing on that coordination process between clients.

Anyway, case in point. If you have restrictive networks (NAT over NAT over NAT) and port-forwarded can’t be done, have no access to IPv6 and you have a cheap VPS, but which would charge you too much for the traffic you’re making (if it were the wireguard “master” peer), then headscale is for you (you still need the VPS to coordinate the clients, but the traffic won’t go through the VPS). Obviously headscale uses the tailscale clients, so on proprietary platforms you get proprietary GUIs.

But there’s an argument to be made. If you use your VPS for more things, then it’s worth it to run headscale on it. Otherwise, if everything you do is in your tailnet, then using the actual tailscale service might be an option (particularly because it’s free, but I still don’t really want to trust tailscale, so I’d encourage people to look at using wireguard with IPv6 endpoints first, then port-forward, then headscale).

1 Like
13

So i have been thinking on the other two problems data synchronization and how to access, data synchronization is the one im not sure i have a good idea for, rsync is what comes to mind but that will probably be ugly. Centralizing the data would break the goal of redundancy for high availability.

As to access and high availability if all services are running on all three servers, Setup and nginx or apach server on each server to revers proxy all the services on each server, with it pointing to it self for the main server of each service, and the other two as backups. Really you would make one config then just switch around the ip address depending on which server your pasting it onto to deploy.

Idealy if you were running pihole on all three for their respective houses you could give each the same domain name, inside thier respective networks so any bookmarked links would work regardless of which location your at. Example if they all had the name server.local with the reverse proxy to get to jelly fin would be as simple as server.local/jellyfin in any house.

The missing piece here is if the whole server is down there is nothing to redirect you to anther server.