Routing Packets Coming from a tun Interface out via default gateway

Hardware Hub Networking
1

my main goal: what is the most “correct” method of routing packets coming from a tun interface to the internet (via system’s default gateway.)

so what I have is a gre tunnel running over vpn to my vps.
to achieve this I have added an extra IRFC 1918 ip to my VPS’s WAN Interface (192.168.100.1).
and the gre tunnel is running from my route’s local ip (192.168.1.1) to this second ip.

I have OSPF running on my vpn so router knows how to get to 192.168.100.1 and the gre tunnel establishes.
Now I want to route some service to the internet using my vps’s ip.

I suspect that I need to snat packets coming from the tun interface to my vps’s public ipv4.
But I’m note sure why that’s necessary .

vps wan ip: x.x.x.x/32
vps second local ip on wan interface: 192.168.100.1/29
vpn endpoint ip on vps: 172.24.0.1/16
gre tun tunnel address on vps:172.25.0.1/30
gre tun tunnel adress on LAN : 172.25.0.2/30
vpn endpoint on local lan: 172.24.0.2/16
local lan net : 192.168.1.1/24

  • ospf running on vpn interface.
    look at the example below:
  • service with Ip address 192.168.1.5 sends packed destined to a wan ip.
  • I have a rule that sets gateway for all packets coming from 192.168.1.5 to be 172.25.0.1.
  • icmp destined to x.x.x.x gets a reply.
  • icmp destined to google.com gets to 172.25.0.1 and doesn’t get any further.
    ( tcpdump on VPS’s gre shows traffic like this : 172.25.0.2 >>> google.com)

my question is why gre tun doesn’t know how to route packets out via default gateway?
shouldn’t gre tun follow system routing tables?
can instead of SNAT can I configure OSPF differently so less manual configuration is required?

Note:
my main goal is to have many redundant paths to the vps’s network by way of multiple vpn tunnels between local router and vps which packets originating from 192.168.1.5 can take to go out of vps’s public ip.

correct me if I’m wrong, but with this current config if a vpn link goes down because of ospf running on all vpn interfaces, gre tunnel finds another path to 192.168.100.1 and recovers on it’s own.

Note 2 : Should I run OSPF on gre tunnel itself?

2

so im like 90% there, on the vps side snat ing gre tunnel traffic to vps public ip allows traafic to go out.
However I just realized gre tunnel changes packets source port for some reason so It seems I have to use something else.

3

an update on this.
despite port changes on packets traversing the tunnel the setup works.
How ever I’m note sure on how or Why exactly.

4

late reply … but I’m very confused by you even reaching towards OSPF.

What os / ui / routing software suite/os are you using? I sense it’s adding a lot of “stuff” into your routes?

in computer networking “routing protocols” is a misnomer, … what they really should be called are “route distribution protocols”. … there’s nothing stoping you from forwarding packets from one interface to another using manually defined “static” routes.

…in other words.

OSPF is what helps one router running OSPF tell another router running OSPF about subnets it can reach, so that the other router know who to talk to when packets destined to those subnets show up. if you know all the subnets ahead of time and they’re not changing, I don’t see why you need OSPF, … or any other “dynamic routing protocol”.

vps wan ip: x.x.x.x/32
vps second local ip on wan interface: 192.168.100.1/29

huh? what’s this 192.168.100.1/29 for?

vpn endpoint ip on vps: 172.24.0.1/16
gre tun tunnel address on vps:172.25.0.1/30
gre tun tunnel adress on LAN : 172.25.0.2/30
vpn endpoint on local lan: 172.24.0.2/16

what do you mean by endpoint, what are these /16 subnets for?

local lan net : 192.168.1.1/24
…

  • I have a rule that sets gateway for all packets coming from 192.168.1.5 to be 172.25.0.1.

“source based routing?” / “policy based routing” … what do you mean by a “rule” in this case?

my main goal is to have many redundant paths to the vps’s network by way of multiple vpn tunnels between local router and vps which packets originating from 192.168.1.5 can take to go out of vps’s public ip.

what’s this “vps’s network”? Is it 192.168.100.1/29 … which you have configured on wan?

So basically, your VPS is connected to multiple networks, therefore it is a router, and it should do SNAT on its wan - because that’s where packets from your private networks, that the ISP hosing your VPS doesn’t know about, go out onto the wider internet. You don’t really need any other NAT. (I guess you’ll have one on your LAN router to let computers on your network get to the internet directly, via your home ISP as opposed to everything using the GRE tunnel and the VPS on the other side of it).

On your VPS you need to route 192.168.1.0/24 via 172.25.0.2 , because when traffic from wherever comes to your VPS router, it needs to go back over the GRE tunnel.

And, you need that PBR / source based route / routing mark rule / … something to route 192.168.1.5 stuff, so it gets routed differently to other things passing through LAN router.

That’s what you need, but the OSs / routing suites / things you’re using may be putting other stuff into the mix which may or may not be helping you. (e.g. you may be blackholing rfc1918 subnets except for the directly connected ones somewhere?)

5

hey,
so here is a rough outline of my setup rn (two sites and a cloud vps, planning to add more sites later)

image
image1352×685 12.6 KB

Site A and Site B are running opnsense as their firewall/router.
VPS is running ubuntu server.
each site gonna have multiple wan connections AND multiple vpn links between themselves and VPS.
the Idea is to have redundancy incase for some reason one of the vpn connections drops.

I decided to run a automated route propagation protocol so I don’t have to keep adding/changing static routes plus I can add more sites/services without worrying about adding routes manually. (if that makes sense)

don’t take stated ip addresses / networks seriously I wanted to keep actual IPs private .(doesn’t make much sense I know)

so my justification for running a GRE tunnel on top of vpn tunnel to the vps is this:
Site A wants to serve a service by utilizing vps’s public IP addresss.
I can set a policy based routing rule (firewall rule in opnsense) to use wg tunnel, however but if the wg tunnel goes down the link breaks.

So my solution is to run a another tunnel over my vpn links between site A and vps like this:

image
image1744×560 14.1 KB

I will tell gre tunnel that parent for 172.25.0.1 is 192.168.1.1 and parent for 172.25.0.2 is 172.30.0.1 .
So if I run ospf on my vpn links between sites and vps, if a link carrying the tunnel goes down (say right now gre is running over wg and wg goes down for some reason) there will be a different path to 172.30.0.1 and eventually the gre tunnel will recover using another vpn to get to 172.30.0.1 ( that’s the idea at least ) .

I’m still not 100% on what happens behind the scenes when a nat (or routing in general) stuff happens.

I had this setup working to like this:
packet start from service on local lan (192.168.1.50 for example) with intent of going out of vps public ip to 2.3.4.5.
here is the path:
192.168.1.50:1000 > 2.3.4.5:500
172.25.0.1:1200 > 2.3.4.5:500 (packet source port changes when it enters gre tunnel)
172.30.0.1:1200 > 2.3.4.5:500

so two things happens above:

  • packet source port changes traversing to the vps
  • packet gets to 172.30.0.1 and stays there
    To get packets to go out of the vps, I had to do a SNAT packets from 172.25.0.1 TO 1.2.3.4 (vps public ip).

two things I’m not sure about:

  • why a packet on the 172.30.0.1 doesn’t simply use VPS’s default gateway to go out ? ( packet has a destination outside local/known addresses shouldn’t it default to going out via default gateway? basically the snat happens automatically)
  • how come packet source port changes do not interfier with a connection being made?
    for example my service on lan sends a packet using port 1000 this packet goes out of the vps via port 1200 (1000 PNATed to 1200 here), destination sends a reply to 1200 (same port it received the packed) this packet gets to local service using port 1300.
    so, packet from
    192.168.1.5:1000 goes out of 1.2.3.4:1200 to 2.3.4.5
    the reply from 2.3.4.5 to 1.2.3.4:1200 gets to 192.168.1.5:1300.

reading around, it seems routers keep some sort of table for ports but I’m not sure that explains what happens here.

doesn’t an application sending a request via port 1000 expects a reply via the same port?

6

Ah, I see now, … I’d really strongly consider Tailscale, if I were you.

In your setup you’d run it on two site routers with --advertise-routes=sit.e_a.sub.net and --advertise-routes=sit.e_b.sub.net and allow/enable the two advertised routes in your tailnet using the Tailscale console, and that’s it.

And if you don’t want Tailscale folks to know your using Tailscale, run headscale on the VPS - you’d run tailscale with --login-server https://myheadscale.mydomain.onvps.

Without Tailscale, and with links between sites going up/down/sideways, yes, dynamic routing makes sense, and I agree might as well go with ospf, there’s other options.

Wireguard breaks if underlying link breaks, and sometimes if underlying link breaks it recovers on its own (depending on how underlying link breaks exactly).

If you do:

[ site A ] — wireguard — [ VPS ] — wireguard — [ site B ]

site A and site B can have dynamic IPs from ISP, doesn’t matter, wireguard IPs will be static, and you can use them to route packets through, that’s all you need, it’s super simpl, you don’t need dynamic routing anywhere, nor do you need GRE tunnels.

Now, if you want to additionally add into the mix:

[ site A ] — wireguard — [ site B ].

… so that you have that triangle.

You end up with multiple links (via VPS and directly), you can make a route to site_a via site_b wireguard address, with a lower metric, so that it’s preferred, and packets don’t go over VPS unnecessarily.

If wireguard goes down, the route with an unreachable gateway gets marked as linkdown so the route is not used and the packets will go over the route with a higher metric, once wireguard is back, packets will go over the better route.

I don’t know how wireguard on OPNsense and FreeBSD handles this, this needs testing. It’s possible it might need some additional “reachability check script” to enable/disable a route. …or you could do OSPF over wireguard.

Your site A and site B have dynamic IPs, you could manually update them from DNS, or from VPS in some other way.

Your site A and site B also don’t have public IPs, and have NAT; there’s a technique called UDP hole punching, where the UDP server can, knowing the address and source port of the peer, send out a packet to that peer preemptively, to punch a hole in local NAT to create that mapping. This works most of the time, not always. With wireguard you can configure peer source and destination port statically and if you’re lucky you can rely on heartbeat to punch a hole.

Tailscale basically does this for you. If you’re fully DIY-ing it, headscale can act as your own coordination service. It makes every node in your tailnet know about every other node, it bypasses nats, deals with distributing keys, and propagates routes… and if a direct route is impossible, it can relay for you. (you can run as many relays as you want - headscale has one by default).

1 Like
7

Pretty much everything @risk said. I think you will save yourself a lot of headache using Tailscale/headscale.

8

hey,
another late reply here.
This setup exists in a quite hostile environment, one day you will wake up and all UDP traffic is blocked for one reason or another so for this reason I’m planning to have multiple links between clients (ex ovpn+wg+zt+tailscale from each site to vps AND tailscale+zt from siteA to siteB) and this was my reason for running an independent tunnel (gre) from sites to vps.
so if for some reason WG traffic gets blocked nationwide there will SOME failover.

right now I’m struggling to run bgp between sites and vps to properly advertise routes, I’m new to BGP and It’s not playing nice.

9

How hostile is hostile? e.g. what country?

Tailscale does point to point UDP, and if it can’t get through it fails over to DERP relaying over https.

As long as you can open arbitrary https connections, derp works.

By default, they have their own derp relays, but you can run your own for more performance.

It’s not built for cloaking traffic specifically, like some other protocols, but many of the great firewalls can be bypassed simply by obfuscating or xor-ing wireguard/tailscale UDP packets in iptables on both ends with some shared “passphrase”.

So… how hostile exactly? Do you need VPN over tor/OBFS4 - will the government send an army of people with $5 wrenches knocking on your door if they detect you using a VPN, or is it just wonky NAT-ing.

10

hostile in a sense that tomorrow almost all udp traffic might get blocked for whatever reason, therefore I can’t rely on just a single tunnel and
would like to have more than one backup path between nodes to try to keep the links online as much as possible.
I’ve had tailscale in mind to have in addition to my wg+ovpn+zt links but there is no official support for it in opnsense (as oppose to pfsense) so my current setup will have to do for now.

…

now for latest update to my setup,
Right now I have setup bgp routing between my sites and main traffic runs on a gre tunnel between sites (the gre runs on top of whatever vpn that currently linking nodes together wg,ovpn,etc.).
to get bgp to wok properly I had to setup some basic route maps to avoid recursive routing (ie tunnel running it’s own packets over itself).
So far the setup is working fine, however I will try to update this thread here if I had to some more major changes for the setup to work.

1 Like