Hello,
In this tutorial I’m gonna tell you how to connect LXD container to ZeroTier network without giving the container much authority over the configuration and route all of its traffic via zerotier.
Part of this tutorial can be used for just connecting the container into a zerotier network, that is if you skip setting up the NAT and default gateway steps.
BEWARE:
I didn’t personally follow the instructions step by step, I had a lot of tria and error in between them and debugging issues, going a few steps back because I broke something etc.
This is more or less what you’ll have to do, and if it doesn’t work - let me know, I’ll try helping you out in this thread, but I am by no means a network admin.
Why
This section provides a little bit of a background why and how I arrived at this setup, if you are looking for step by step instructions, skip this section, nothing of value for you here.
So I wanted to host a few minecraft servers for me and my friends and random people I meet on the internet, I tried renting a dedicated server on hetzner (64GB RAM, some Ryzen 7 CPU, dont remember the exact model), but it turned out to be not performant enough.
The price I paid for it was the maximum I wanted to pay for (something around 25-35 EUR). So “just rent a better one, duh” ws not an option for me.
But, I know I want to host those things indefinitely, so we can calculate how much I will have to pay for hosting it in the current state for 2 years, and I’ll see how much of a difference would it make to just buy the hardware.
It turns out that buying a machine with a faster CPU (assuming no hardware failiure) is more or less the same cost as renting that server on hetzner for 2 years.
So, we have the hardware but the problem is network connectivity, I could make people I want to play with fool around with VPN software but let’s be realistic - nobody will want to do that to just play on a minecraft server.
So the obvious solution is to proxy the traffic from a public IP address over some VPN (mesh/point to point) to the “server” sitting in my apartment.
And with this goal in mind I bought a second IP address for my VPS hosted on OVH, now for the VPN, we’re gonna use zerotier because that’s what I’m already using and it works really well for me.
The end result we want to achieve
+---------------LXD-host---------------+
| |
+--The-internet--+ +------OVH---VPS-with-2x-IPv4------+ | |
| +-------+ | | +------lxd-container-A-------+ |
| +---+ | | | | | |
| | | | | | | | |
| | | | | | | | |
+----------------+ | | | | | | |
| | | | | | |
| | | | +----------------------------+ |
| | | | | |
| | | | | |
| +--+->Public IPv4 A | | |
| | | +------lxd-container-B-------+ |
| | | | | |
+------+->Public IPv4 B---->Zerotier IPv4--------------------+>Zerotier IPv4 | |
| | | | |
| | | | |
| | | | |
+----------------------------------+ | | |
+----------------------------+---------+
- All traffic going to Public IPv4 B will be routed to Zerotier IPv4 in lxd-container-B
- lxd-container-B has no connection to the host-os nor to the lxd-container-A
- all traffic from lxd-container-B exits via Public IPv4 B
Assumptions:
You know more or less how to use CLI tools on UNIX-like OS’es
The remote host with public IP addresss has two IP addresses - one for itself and the other to “proxy” (NAT) to zerotier
You have your ufw (or other) firewall configured in a way that doesn’t interfere with zerotier, they provide a handy troubleshooting guide, I personally never ran into a situation where I had to use it: Troubleshooting | ZeroTier Documentation
Both sides have no problems with communication on port 9993:
Default zerotier-one listening ports are:
9993
Secondary Port, randomized each start up and after being "offline" for too long.
Random Port for UPnP and NAT-PMP (UPnP or NAT-PMP is not required for ZeroTier hole punching to work)
For best results, a device needs be able to send to any IP address, on any UDP port. If you allow outgoing source:9993 and incoming related return traffic, it'll probably work OK.
Quote from: Corporate Firewalls | ZeroTier Documentation
The actual “tutorial”
So the first step will be joining two server(s) into one zerotier network:
on OVH VPS and LXD host :
zerotier-cli join $ZT_NETWORK
no on LXD host run this:
ip address
and save its output in in some text file, you are going to need it later
Now on the LXD host we want to rip the zerotier nic out of “default” cgroup and place it into the container cgroup:
# $LXD_DEVICE_NAME - this name is for you to pick, I recommend something that will tell you its purpose
lxc config device add $LXD_CONTAINER_NAME $LXD_DEVICE_NAME nic nictype=physical parent=ztyourjrks
If you want to learn more why and what this command does: Networking setups - LXD documentation
Now the interface is kind of a shrodingers interface - it technically doesnt exist from the perspecive of “host userland” but the kernel is aware of it and the LXD container userland is also aware of it, however the zerotier userland daemon is in charge of it, I have no idea why nothing breaks, it doesnt make any sense to me that the daemon running in “host userland” is not crashing.
But we’re here to get stuff done and not think too much, moving on.
Now you’ll want to open a shell in the LXD container:
lxc shell $LXD_CONTAINER_NAME
and we want to list out interfaces using:
ip addr
You might have noticed that we’re missing an IP address on our zerotier nic, this is why you were supposed to save the output of ip address ran on the host.
We need to also get rid of the “default” interface LXD inserted into this container. (technically you don’t need, but I have no idea how to make this setup work otherwise and next steps won’t work)
Generally speaking I don’t even know how to properly get rid of this interface, we’re just gonna turn it off
ip link set eth0 down
And now we want to turn on the zerotier nic, assign IP address to it and set our remote VPS on the other side as a default gateway and setup dns server
ip link set eth1 up
# $NETMASK - if you are unsure it's probably /24
ip a add $LXD_CONTAINER_ZT_IP/$NETMASK dev eth1
resolvectl dns eth1 8.8.8.8 # or other DNS of your liking
ip route add default via $ZT_VPS_IP dev eth1
Assuming the hosts can ping each other we’re gonna tell the remote host to act as a NAT
sysctl -w net.ipv4.ip_forward=1
iptables -t nat -A POSTROUTING -s $LXD_CONTAINER_ZT_IP -o $PUBLIC_IP_NIC -j SNAT --to-source $PUBLIC_IP_ADDRESS
iptables -t nat -A PREROUTING -d $PUBLIC_IP_ADDRESS -j DNAT --to-destination $LXD_CONTAINER_ZT_IP
iptables -A FORWARD -i $ZT_NIC -o $PUBLIC_IP_NIC -s $LXD_CONTAINER_ZT_IP -j ACCEPT
iptables -A FORWARD -i $PUBLIC_IP_NIC -o $ZT_NIC -d $LXD_CONTAINER_ZT_IP -m state --state ESTABLISHED,RELATED -j ACCEPT
if you are using ufw on the VPS, allow traffic on the NAT’ed IPs
ufw route allow in on ens3 out on ztyourjrks to $LXD_CONTAINER_ZT_IP
ufw route allow in on ztyourjrks out on ens3 from $LXD_CONTAINER_ZT_IP
You should be able to access the internet from your LXD container now, have fun.
One important thing to note here is that you’ll likely want to setup a firewall inside of this container since it’s exposed to the public internet.
All connections to $PUBLIC_IP are passed through into it :- )