Cross site proxmox clustering using Tailscale. Am I silly?

Operating Systems & Open Source Linux
,
1

Hi All,

Nnew to posting here so please pardon any formatting issues or common form post mistakes. I will try to lay it out my post in a way that makes sense rather than a fluid conversation since I find it easier to parse info that way.

Goal: 2 networks (separate houses), 1 proxmox host at each (Dell SFF PCs), the ability to replicate services that are network agnostic, and the ability to host services that are network specific. Stretch goal is to manage both from one interface.

Current setup:
House 1:

  • proxmox host, node has tailscale with routing to 192.168.1.0/24, with the following containers:
    • discord.py bot
    • game server - accessed via reverse proxy
    • adguard - serves local network
    • file server (copy party) - accessed via reverse proxy
    • reverse proxy (caddy) - friends who need access to game server have this machine shared out to them

Desired future setup:
House 1 (no change)
Add House 2:

*** means only active when House 1 is down

  • proxmox host, node has tailscale with routing to 192.168.2.0/24, with the following containers:
    • discord.py bot ***
    • game server - accessed via reverse proxy, storage replicated ***
    • adguard - serves local network
    • file server (copy party) - accessed via reverse proxy, replicated from house 1
    • reverse proxy (caddy) - friends who need access to game server have this machine shared out to them

Am I silly for trying to do this in proxmox? Should I instead be looking to rebuild this in a linux host with standard docker container with standard replication software? Should my game server and storage server just have different links with their data replicated and the services only enabled when the other is down? Thoughts? Keywords I should google? Frames of mind I should adopt instead?

All are welcome since I am lost.

Thanks,
pocget

2

won’t you need a third machine in a separate location with something like haproxy running to detect which house is offline (if any) and facilitate the failover from the 192.168.1.x server to 192.168.2.x? It doesn’t make sense to put such plumbing in either house. Othewise it seems like a relatively straightforward high availability setup. Proxmox is probably more well-suited to this than rolling your own with docker and a linux host.

3

The proxmox clustering mode assumes “proper” clustering. eg. if you don’t have the same storage configuration on each node it gets a bit messy, it kind of assumes 3 nodes minimum for elections, etc. You can do a 2 node cluster but for what you’re describing it’s more pain than gain.

Maybe look at the new proxmox datacentre manager? I’m not sure if it can replicate vms between nodes though.

4

Regarding requiring three sites to facilitate fail-over. I was hoping to just do some sort of setup where they ping each other, and google.

If other node is down, and google is up, take over. If other node and google are down, you’re the one that’s down.

One thing I am under-educated in is how the replication works from a network perspective. Since I want the adguard to have the local VLAN.

Regarding proxmox datacentre manager, those are new words to me and something to look into. Thank you!

5

Time to invest in house 3 to complete the proxmox cluster :slight_smile:

3 Likes
6

Corosync is very sensitive to latency, so creating clusters that sync over the internet is not a great idea (unless you aquire direct dedicated fiber line to the other house). I would try setting replication through proxmox datacenter manager.

7

In my experience tailscale is slow. It might be fast compared to other vpn’s but overall it’s slow. I’ve got a 1gig/1gig fiber at home and plenty of bandwidth at work, yet I struggle to break 20 Mbit/s over tailscale. It’s not saying tailscale is crap, it’s just rare to see an actually fast vpn.

For that reason I would drop trying to do any kind of clustering over a vpn.

My experience with running proxmox clusters is that network speed is paramount.

Even though being able to manage multiple proxmox hosts in a single interface is really nice, I think you’re setting yourself up for a world of hurt if you try to do it over a vpn. As soon as your proxmox hosts are clustered they get all kinds of picky with letting you start and stop and create vm’s. I think you’re better of just running them individually even if it means two management interfaces.

Migrating machines from one server to another over a vpn is going to be painful too. Transferring a 32 GB vm over a connection that struggles to do 20 Mbit/s is going to take hours.

What I recommend doing instead is running a proxmox backup server in each location. You could have them sync stuff to each other, but just having site 1 do backups to the proxmox backup server at site 2 and vice versa would allow you to quickly restore services at the other site if needed. The great thing about PBS is that it does incremental backups meaning that only changes needs transferring.

Lets say at site 1 you do nightly backups of your vm’s to a local PBS at 2 am. As it’s incremental it probably takes all of 8 minutes. Then at 3 am you do a backup of your vm’s at site 1 onto the PBS at site 2. As it’s incremental and over a vpn it might take an hour. if site 1 dies you have a local backup that is easily restored. If you need to order new hardware you have a backup of all the site 1 vm’s on the pbs at site 2, meaning that you can restore site 1’s vm’s at site 2 in half an hour if needed.

Only downside, as I see it, that you don’t get to manage both servers from a single web ui, you need two bookmarks rather than one.

8

That sounds a bit too slow though, are you sure it’s not relates to CPU load or resource allocation? VPNs can be pretty CPU-intensive at high speeds.

9

My pc at work:

thomas@beast:~$ inxi -Fw

System:
Host: beast Kernel: 6.8.0-101-generic x86_64 bits: 64 Console: pty pts/1
Distro: Ubuntu 22.04.5 LTS (Jammy Jellyfish)
Machine:
Type: Desktop System: HP product: HP Z240 Tower Workstation v: N/A
serial:
Mobo: HP model: 802F v: KBC Version 05.30 serial:
UEFI: HP v: N51 Ver. 01.35 date: 12/02/2016
CPU:
Info: quad core model: Intel Core i7-6700 bits: 64 type: MT MCP cache:
L2: 1024 KiB
Speed (MHz): avg: 812 min/max: 800/4000 cores: 1: 800 2: 800 3: 800
4: 800 5: 800 6: 900 7: 800 8: 800
Graphics:
Device-1: Intel HD Graphics 530 driver: i915 v: kernel
Display: server: X.org v: 1.21.1.4 with: Xwayland v: 22.1.1 driver: X:
loaded: modesetting,nvidia unloaded: fbdev,nouveau,vesa gpu: i915
tty: 80x43 resolution: 1: 3440x1440 2: 1920x1080
Message: GL data unavailable in console. Try -G --display
Audio:
Device-1: Intel 100 Series/C230 Series Family HD Audio
driver: snd_hda_intel
Sound Server-1: ALSA v: k6.8.0-101-generic running: yes
Sound Server-2: PulseAudio v: 15.99.1 running: yes
Sound Server-3: PipeWire v: 0.3.48 running: yes
Network:
Device-1: Intel Ethernet I219-LM driver: e1000e
IF: eno1 state: up speed: 1000 Mbps duplex: full mac: 70:5a:0f:48:e4:c4
IF-ID-1: tailscale0 state: unknown speed: -1 duplex: full mac: N/A
IF-ID-2: virbr0 state: down mac: 52:54:00:a8:29:5a
RAID:
Hardware-1: Intel SATA Controller [RAID mode] driver: ahci
Drives:
Local Storage: total: 476.94 GiB used: 106.43 GiB (22.3%)
ID-1: /dev/sda vendor: Micron model: MTFDDAK512TBN-1AR1ZABHA
size: 476.94 GiB
Partition:
ID-1: / size: 467.89 GiB used: 106.42 GiB (22.7%) fs: ext4 dev: /dev/sda2
ID-2: /boot/efi size: 511 MiB used: 6.1 MiB (1.2%) fs: vfat
dev: /dev/sda1
Swap:
ID-1: swap-1 type: file size: 2 GiB used: 0 KiB (0.0%) file: /swapfile
Sensors:
System Temperatures: cpu: 25.0 C pch: 46.0 C mobo: N/A
Fan Speeds (RPM): N/A
Weather:
Report: temperature: 10.17 C (50 F) conditions: clear sky
Locale: current time: Mon 09 Mar 2026 01:39:31 PM CET (Europe/Copenhagen)
Source: OpenWeatherMap.org
Info:
Processes: 259 Uptime: 11d 6h 3m Memory: 31.14 GiB used: 1.97 GiB (6.3%)
Init: systemd runlevel: 5 Shell: Bash inxi: 3.3.13
thomas@beast:~$ iperf3 -s

Server listening on 5201

Accepted connection from , port 41916
[ 5] local port 5201 connected to port 41922
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.94 sec 640 KBytes 2.70 Mbits/sec
[ 5] 1.94-3.11 sec 256 KBytes 1.79 Mbits/sec
[ 5] 3.11-4.80 sec 384 KBytes 1.86 Mbits/sec
[ 5] 4.80-5.81 sec 256 KBytes 2.09 Mbits/sec
[ 5] 5.81-6.46 sec 256 KBytes 3.19 Mbits/sec
[ 5] 6.46-7.68 sec 512 KBytes 3.45 Mbits/sec
[ 5] 7.68-8.58 sec 384 KBytes 3.48 Mbits/sec
[ 5] 8.58-9.83 sec 384 KBytes 2.53 Mbits/sec
[ 5] 9.83-10.29 sec 256 KBytes 4.47 Mbits/sec

[ ID] Interval Transfer Bitrate

[ 5] 0.00-10.29 sec 3.25 MBytes 2.65 Mbits/sec receiver

Server listening on 5201

^Ciperf3: interrupt - the server has terminated
thomas@beast:~$

My pc at home:

[thomas@cubus ~]$ inxi -Fw
System:
Host: cubus Kernel: 6.12.73-1-MANJARO arch: x86_64 bits: 64
Desktop: Xfce v: 4.20.1 Distro: Manjaro Linux
Machine:
Type: Desktop Mobo: ASRock model: B550M-ITX/ac serial:
Firmware: BIOS vendor: American Megatrends LLC. v: P3.90 date: 10/01/2025
CPU:
Info: 6-core model: AMD Ryzen 5 5600GT with Radeon Graphics bits: 64
type: MT MCP cache: L2: 3 MiB
Speed (MHz): avg: 400 min/max: 400/4668 cores: 1: 400 2: 400 3: 400 4: 400
5: 400 6: 400 7: 400 8: 400 9: 400 10: 400 11: 400 12: 400
Graphics:
Device-1: Advanced Micro Devices [AMD/ATI] Navi 24 [Radeon RX 6400/6500
XT/6500M] driver: amdgpu v: kernel
Device-2: Advanced Micro Devices [AMD/ATI] Cezanne [Radeon Vega Series /
Radeon Mobile Series] driver: N/A
Display: unspecified server: X.Org v: 21.1.21 with: Xwayland v: 24.1.9
driver: X: loaded: amdgpu unloaded: modesetting,radeon,vesa dri: radeonsi
gpu: amdgpu resolution: 1: 2560x1440~60Hz 2: 1920x1080~60Hz
API: EGL v: 1.5 drivers: kms_swrast,radeonsi,swrast
platforms: gbm,x11,surfaceless,device
API: OpenGL v: 4.6 compat-v: 4.5 vendor: amd mesa v: 25.3.5-arch1.1
renderer: AMD Radeon RX 6400 (radeonsi navi24 LLVM 21.1.6 DRM 3.61
6.12.73-1-MANJARO)
API: Vulkan v: 1.4.341 drivers: radv surfaces: N/A
Info: Tools: api: eglinfo, glxinfo, vulkaninfo
de: kscreen-doctor,xfce4-display-settings x11: xdriinfo, xdpyinfo, xprop,
xrandr
Audio:
Device-1: Advanced Micro Devices [AMD/ATI] Navi 21/23 HDMI/DP Audio
driver: snd_hda_intel
Device-2: Advanced Micro Devices [AMD/ATI] Renoir/Cezanne HDMI/DP Audio
driver: snd_hda_intel
Device-3: Advanced Micro Devices [AMD] Ryzen HD Audio
driver: snd_hda_intel
Device-4: Thesycon System & Consulting GmbH SMSL USB AUDIO
driver: snd-usb-audio type: USB
API: ALSA v: k6.12.73-1-MANJARO status: kernel-api
Server-1: PulseAudio v: 17.0-98-gb096 status: active
Network:
Device-1: Realtek RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet
driver: r8169
IF: enp6s0 state: up speed: 1000 Mbps duplex: full mac: 9c:6b:00:a0:36:18
IF-ID-1: tailscale0 state: unknown speed: -1 duplex: full mac: N/A
Drives:
Local Storage: total: 2.73 TiB used: 1.77 TiB (64.8%)
ID-1: /dev/nvme0n1 vendor: Samsung model: MZVLB256HAHQ-000L7
size: 238.47 GiB
ID-2: /dev/sda vendor: Crucial model: CT240BX200SSD1 size: 223.57 GiB
ID-3: /dev/sdb vendor: Crucial model: CT500MX500SSD1 size: 465.76 GiB
ID-4: /dev/sdc vendor: Crucial model: CT2000MX500SSD1 size: 1.82 TiB
Partition:
ID-1: / size: 232.91 GiB used: 188.78 GiB (81.1%) fs: ext4
dev: /dev/nvme0n1p3
ID-2: /boot size: 492.9 MiB used: 77.2 MiB (15.7%) fs: ext2
dev: /dev/nvme0n1p1
Swap:
ID-1: swap-1 type: partition size: 1.17 GiB used: 0 KiB (0.0%)
dev: /dev/nvme0n1p2
Sensors:
System Temperatures: cpu: 39.5 C mobo: N/A gpu: amdgpu temp: 58.0 C
Fan Speeds (rpm): fan-1: 0 fan-2: 1041 fan-3: 0 fan-4: 0 fan-5: 0
gpu: amdgpu fan: 1375
Weather:
Report: temperature: 12.2 C (54 F) conditions: clear sky
Locale: Kolding, 83, Unknown current time: 2026-03-09T13:40:03 CET
(Europe/Copenhagen) Source: OpenWeatherMap.org
Info:
Memory: total: 16 GiB note: est. available: 15.24 GiB used: 2.4 GiB (15.7%)
Processes: 336 Uptime: 7h 49m Shell: Bash inxi: 3.3.40
[thomas@cubus ~]$ iperf3 -c
Connecting to host , port 5201
[ 5] local port 41922 connected to port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 640 KBytes 5.24 Mbits/sec 0 79.1 KBytes
[ 5] 1.00-2.00 sec 128 KBytes 1.05 Mbits/sec 0 89.9 KBytes
[ 5] 2.00-3.00 sec 384 KBytes 3.15 Mbits/sec 3 67.2 KBytes
[ 5] 3.00-4.00 sec 128 KBytes 1.05 Mbits/sec 0 79.1 KBytes
[ 5] 4.00-5.00 sec 256 KBytes 2.10 Mbits/sec 0 89.9 KBytes
[ 5] 5.00-6.00 sec 256 KBytes 2.10 Mbits/sec 0 92.3 KBytes
[ 5] 6.00-7.00 sec 384 KBytes 3.15 Mbits/sec 0 92.3 KBytes
[ 5] 7.00-8.00 sec 384 KBytes 3.14 Mbits/sec 0 93.5 KBytes
[ 5] 8.00-9.00 sec 384 KBytes 3.15 Mbits/sec 0 97.1 KBytes
[ 5] 9.00-10.00 sec 512 KBytes 4.19 Mbits/sec 0 108 KBytes

[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 3.38 MBytes 2.83 Mbits/sec 3 sender
[ 5] 0.00-10.29 sec 3.25 MBytes 2.65 Mbits/sec receiver

iperf Done.

Not really sure why it messes around with the font like that?

I’ve removed the ip addresses just for the heck of it.

The numbers are through a tailscale connection on ipv4

If I run it over ipv6 through tailscale I get:

thomas@beast:~$ iperf3 -s

Server listening on 5201

Accepted connection from , port 48576
[ 5] local port 5201 connected to port 48578
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.18 sec 1.25 MBytes 8.90 Mbits/sec
[ 5] 1.18-2.10 sec 1.12 MBytes 10.2 Mbits/sec
[ 5] 2.10-3.11 sec 1.62 MBytes 13.4 Mbits/sec
[ 5] 3.11-4.09 sec 2.00 MBytes 17.1 Mbits/sec
[ 5] 4.09-5.11 sec 1.75 MBytes 14.5 Mbits/sec
[ 5] 5.11-6.16 sec 1.62 MBytes 12.9 Mbits/sec
[ 5] 6.16-7.14 sec 1.38 MBytes 11.9 Mbits/sec
[ 5] 7.14-8.13 sec 2.00 MBytes 16.9 Mbits/sec
[ 5] 8.13-9.10 sec 1.88 MBytes 16.2 Mbits/sec
[ 5] 9.10-10.09 sec 1.62 MBytes 13.8 Mbits/sec
[ 5] 10.09-10.17 sec 128 KBytes 13.2 Mbits/sec

[ ID] Interval Transfer Bitrate

[ 5] 0.00-10.17 sec 16.4 MBytes 13.5 Mbits/sec receiver

Server listening on 5201

^Ciperf3: interrupt - the server has terminated

[thomas@cubus ~]$ iperf3 -c
Connecting to host , port 5201
[ 5] local port 48578 connected to port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.62 MBytes 13.6 Mbits/sec 27 155 KBytes
[ 5] 1.00-2.00 sec 1.25 MBytes 10.5 Mbits/sec 0 180 KBytes
[ 5] 2.00-3.00 sec 1.50 MBytes 12.6 Mbits/sec 0 197 KBytes
[ 5] 3.00-4.00 sec 1.88 MBytes 15.7 Mbits/sec 0 204 KBytes
[ 5] 4.00-5.00 sec 1.50 MBytes 12.6 Mbits/sec 0 206 KBytes
[ 5] 5.00-6.00 sec 1.50 MBytes 12.6 Mbits/sec 0 206 KBytes
[ 5] 6.00-7.00 sec 1.62 MBytes 13.6 Mbits/sec 0 206 KBytes
[ 5] 7.00-8.00 sec 1.75 MBytes 14.7 Mbits/sec 0 211 KBytes
[ 5] 8.00-9.00 sec 2.12 MBytes 17.8 Mbits/sec 0 222 KBytes
[ 5] 9.00-10.00 sec 1.62 MBytes 13.6 Mbits/sec 0 241 KBytes

[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 16.6 MBytes 13.9 Mbits/sec 27 sender
[ 5] 0.00-10.17 sec 16.4 MBytes 13.5 Mbits/sec receiver

iperf Done.

So as I said, I struggle to get 20Mbit/s end to end over tailscale.

10

I only realized now that I practically necro’d the thread…
Anyway, the discrepancy in speed between IPv4 and IPv6 makes me wonder if it could be related to NAT? Could also be the ISP going “lol. n a h”.

Then again, my experience has also been that wireguard is much, much slower than OpenVPN. Would be interesting to run a test with Tailscale on a LAN just to remove any outside variables. Maybe I’ll get around to trying it out one of these days.

11

A month isn’t much of a necro.

I did some more testing. I run nextcloud in a virtual server on a server at home, that’s my main reason for tailscaling in the first place.

From my pc at home to my nextcloud at home tailscale gives me 802 Mbit up and 798 Mbit down.

If I run iperf3 -s on my pc at work, I get the same 3-4 Mbit/s using my home pc as an iperf3 client, but 13 Mbit/s when I use my nextcloud vm as a client.

If i use my pc at work as a iperc3 client I get 12 Mbit/s connecting to my nextcloud at home and 11 Mbit/s when connecting to my pc at home. The speed is in both directions and all on ipv4.

So with my pc at work vs my nextcloud at home I get the same speed no matter which is an iperf server and which in an iperf client. The speed tests are bidirectional, so it’s not because my workplace might have more bandwidth in one direction than the other.

But if I speed test between my pc at home and my pc at work I get around 3 times the speed if my work pc is the iperf client and my home pc is the iperf server than if the roles are reversed. Odd..

12

Yeah no, sounds more like a routing issue. My bet would still be on NAT.

Hard NAT makes establishing direct connections difficult (if not impossible). Tailscale will fall back to DERP relay servers based around the world.