Return to Level1Techs.com

[SOLVED] Bypass OpenVPN for web server

Hello, i am new to linux, openvpn stuff and hope you will point me in right direction.
I have Ubuntu Server with dedicated IP and it is running nginx, plex server, also it is a OpenVPN client.
I can tell that OpenVPN server is using redirect-gateway and all my traffic goes through the VPN tunnel, but i do not need it because every site i host on this machine becomes unavailable for the rest of the internet and uptimerobot says everything is down.
So…i need to use openvpn only for plex server traffic on port 32400 and leave rest of the traffic out of VPN. And i am not sure how can i achieve it.
IP Route list looks like this where tun0 is VPN interface and enp2s0 is Ethernet to ISP router.

Thanks.

If your openvpn server is pushing “redirect-gateway” you can use the “pull-filter” command in your config/cmdline to ignore the pushed “redirect gateway”:

pull-filter ignore redirect-gateway

The traditional way to route only specific traffic over a tun device is to:

  1. create a new routing table
  2. Add the tun device as the default route for that route table
  3. mark the specific packets in the PREROUTING chain of the mangle table
  4. add a rule to direct all marked packets to the new routing table.

If you need help with the specific commands, I am happy to get more specific.

Edit: Although with what you have described, as long a plex is listening on 0.0.0.0/32400 (which would include your tun device’s address 10.8.0.6), ignoring the “redirect-gateway” should be enough if all of your plex traffic is coming from a 10.8.0.0/24 address.

2 Likes

I have some old scripts from a Ubuntu router I ran a year ago. Just tried to paste them, but it’s a nightmare copy/paste on phone. Will post something tomorrow when I’m at work.

Setup I ran was similar to yours. Split routing for hosting my own VPN server, load balancing on several VPN client connections, while also hosting public services on public IP.

Cburn11 has the right approach, and also makes it seem simple. Which it is, when you’ve tinkered with it, up until then, it can be somewhere between frustrating and infuriating.

1 Like

Thank you. Yes, please tell what i am doing wrong.
I added pull-filter ignore redirect-gateway to OpenVpn config file and plex switched to my ISP’s ip.
With sudo su created new routing table plexvpn

echo 1 plexvpn >> /etc/iproute2/rt_tables

Added 10.8.0.5 as GW with tun0 device

ip route add default via 10.8.0.5 dev tun0 table plexvpn

Marked all TCP 32400 packets with 0x400, i choose 0x400 for no particular reason just saw this in guide

iptables -t mangle -A PREROUTING -p tcp --dport 32400 -j MARK --set-mark 0x400

and then to direct everything 0x400 to plexvpn table and flushed cache

ip rule add fwmark 0x400 table plexvpn

Plex has an option to specify its listening interface in Settings > Network > Preferred Network Interface that defaults to any. If you can select your tun that might be good enough. If not you may have to add a rule to the INPUT chain of the filter table that drops traffic destined for port 32400 that does NOT arrive on your tun interface.

If you want to route the traffic from the plex server, you should use --sport instead of --dport (the destination port in this case will be whatever the client arbitrarily selected)

The rest of your commands look good, although you may need to add a route in your plexvpn table that specifically routes traffic destined to your openvpn server’s ip address to the default gateway of your main routing table.

But I don’t think the marking packets method is necessary if all of your plex traffic comes from the vpn network:

Since you are no longer adding the tun device as your default route [edit: of the main routing table], as long as there is the route 10.8.0.0/24 via 10.8.0.5 dev tun0 in the main routing table the plex traffic will be routed over the tun device (assuming in came in over the tun device in the first place).

Think i just do not understand completely how plex works. I can not chose tun0 (only enp2s0) in settings and i can not re-route plex’s traffic either even with marking and blocking 32400 on enp2s0, it still goes through my ISP and not tunnel. But if i allow redirect-gateway it works fine.

Alright. After you have established your openvpn connection with the pull-filter command, will you post the output from:

netstat -tlpn and
ip route show ?

and then will you post some of the output from tcpdump -n -i tun0 port 32400 while one of your plex clients tries to connect to your server?

I know it’s a bit of a mess, I can’t seem to find the complete setup script i wrote, just the bits and pieces from before i did the final setup. Think it’s still on my old router, which is in a box somewhere around the house.

However, you’ll at least have a few of the basics to work from, and hopefully it’ll help you get your setup up and running. Everything below is from a router setup, where things were hosted on my lan, so some adaptation will be required.

If you lock yourself out because of iptables, just do a reboot, you have to actively make iptables and route persistent or load it during startup in rc.local or setup a system service to load your settings everytime you reboot.

I’m nipping to Italy on an extended weekend, and won’t be online until start next week, so if something, I won’t be able to answer much till then.

I added these to my VPN client profiles to make sure no route was added, so I could setup my own

route-nopull
dev tun0

You’ll need to enable fwmarks and forwarding, if I recall correctly, a reboot is required after setting the following two. It’s also possible to “echo 1 > /some/long/ass/path/to/nowhere” which doesn’t require a reboot. Someone else here will know if you want to do it that way,

sysctl -w net.ipv4.tcp_fwmark_accept=1
sysctl -w net.ipv4.ip_forward=1

You’d need to have the main table routes in your new “public access” table, used to route back from your server through your wan interface.

wan_gw = the gateway wan interface uses
#ip route flush table 100
ip route show table main | grep -Ev ^default | grep -Ev linkdown | grep -Ev tun | while read r ; do ip route add table 100 $r ; done
ip route add table 100 default via $wan_gw
ip rule add fwmark 100 lookup 100

Saves and restores fwmarks, this one is required for fwmarks to function, makes sure that previously marked packets get remarked if needed, and also makes sure that new ones coming in are marked for their return back out.

iptables -t mangle -A PREROUTING  -j CONNMARK --restore-mark
iptables -t mangle -A PREROUTING  -m mark ! --mark 0 -j ACCEPT
iptables -t mangle -A POSTROUTING -j CONNMARK --save-mark

An example for how I redirected traffic to my server and back out wan interface. The first three lines are general and not port related If the server that does the routing also hosts the services, I would think you’d have to use INPUT and OUTPUT iptables chain.
(Someone correct me if I’m wrong please).

srv_ip = ip of lan server
wan_ip = public ip adddress
wan_if = wan interface (enp2s0)
lan_if = interface for lan (br0)
iptables -t nat    -A PREROUTING  -i $wan_if -m mark --mark 100 -j DNAT --to $srv_ip # incoming to server
iptables -t nat    -A POSTROUTING -o $wan_if -m mark --mark 100 -j SNAT --to $wan_ip # outgoing from server
iptables -t nat    -A POSTROUTING -o $lan_if -j MASQUERADE

These are required for the specific ports/services. Sets a mark on new tcp connections from wan

iptables -A PREROUTING -t mangle -i $wan_if -p tcp -m state --state NEW -j MARK --set-mark 100
iptables -A FORWARD -d $srv_ip -p tcp --dport 80   -m mark --mark 100 -j ACCEPT # inc - HTTP on $srv_ip
iptables -A FORWARD -s $srv_ip -p tcp --sport 80   -m mark --mark 100 -j ACCEPT # out - HTTP on $srv_ip

used this piece to setup routes on specific tables for each vpn connection. It doesn’t have to be a loop if you only have one vpn connection just replace the “for f in…” line with “f=/path/to/your_vpn_file.conf” and remove the “done” in the end

for f in /etc/openvpn/*.conf; do
 iface=$(cat $f | grep dev | awk -F' ' '{ print $2 }')

 if [ -d /sys/class/net/$iface ]; then
    # log "Setting up $iface routes"
    tun_srv=$(cat $f | grep remote | grep -v random | awk -F' ' '{print $2}' | nslookup | tail -n 4 | head -n 1 | awk -F' ' '{print $2"/32"}')
    tun_net=$(ip a s $iface | grep peer | awk -F' ' '{print $4}' | awk -F'/' '{print $1}')
    tun_gw_net=$(echo $tun_net | awk -F'.' '{print $1"."$2"."$3".1/32"}')
    tun_gw=$(echo $tun_net | awk -F'.' '{print $1"."$2"."$3".1"}')
    tun_ip=$(ip a s $iface | grep peer | awk -F' ' '{print $2}' | head -n 1)
    tun_tbl=$(cat /etc/iproute2/rt_tables | grep $iface | awk -F' ' '{print $1}')
    ip route add $tun_srv via $wan_gw # table $tun_tbl
    ip route add 0.0.0.0/1 via $tun_net # table $tun_tbl
    ip route add 128.0.0.0/1 via $tun_net # table $tun_tbl
    ip route add $tun_gw_net via $tun_net # table $tun_tbl
    #ip route show table main | grep -Ev ^default | grep -Ev linkdown | while read r ; do ip route add table $tun_tbl $r ; done
 fi
done

I was really happy to have this one enabled, helps a great deal when you’re debugging.

if [ $ipt_log_enabled -eq 1 ]; then
  iptables -N INPUT_LOG
  iptables -N OUTPUT_LOG
  iptables -N FORWARD_LOG

  iptables -A INPUT -j INPUT_LOG
  iptables -A OUTPUT -j OUTPUT_LOG
  iptables -A FORWARD -j FORWARD_LOG

  iptables -A INPUT_LOG   -m limit --limit 50/m -j LOG --log-prefix "ipt i_log: " --log-level 7
  iptables -A OUTPUT_LOG  -m limit --limit 50/m -j LOG --log-prefix "ipt o_log: " --log-level 7
  iptables -A FORWARD_LOG -m limit --limit 50/m -j LOG --log-prefix "ipt f_log: " --log-level 7

  iptables -A INPUT_LOG -j DROP
  iptables -A OUTPUT_LOG -j DROP
  iptables -A FORWARD_LOG -j DROP
fi
1 Like

I had a similar issue where a Linux server was trying to establish a connection to the admin vlan over it’s default gateway instead of on the mgmt interface where the request was coming in. From what I remember, this was the solution:

https://www.thomas-krenn.com/en/wiki/Two_Default_Gateways_on_One_System

I believe BSD’s behave like this by default, always responding to requests on the interface they were received on instead of using the system default gateway regardless of incoming interface.

Ubuntu answer via default route. To do what op wants to do, have to use split routing.

Edit: however, if the server hosting the service that has to be exposed, it might not be necessary to use ‘ip route’ could be that it’s enough to use fwmark and input/output chain. For fwmark to work, they have to be enabled. I also had a method that worked where I disable rp_filter, which is bad because of IP spoofing.

Thank you all for replies. Especially @cburn11 and @hem i really appreciate your help.
So in the end instead of fighting plex and his weird remote access, i pulled out all my other services traffic form VPN tunnel, left redirect-gateway enabled and let openvpn add routes.
Here is everything i did with sudo su:

  1. net.ipv4.ip_forward = 1 and net.ipv4.tcp_fwmark_accept = 1 to /etc/sysctl.conf and run sysctl -p /etc/sysctl.conf to enable now and leave it persistent
  2. New routing table echo 1 rt2 >> /etc/iproute2/rt_tables
  3. Routing commands in /etc/rc.local to execute them on startup. Where wan_ip = dedicated ip of my ISP and wan_gw = ip of router in local network.
iptables -t mangle -A PREROUTING  -j CONNMARK --restore-mark
iptables -t mangle -A PREROUTING  -m mark ! --mark 0 -j ACCEPT
iptables -t mangle -A POSTROUTING -j CONNMARK --save-mark
ip rule add fwmark 100 table rt2
ip route add default via $wan_gw dev enp2s0 table rt2
ip route add $wan_ip/32 via $wan_gw dev enp2s0 table rt2
iptables -t mangle -A PREROUTING -p tcp --dport 80 -j MARK --set-mark 100 #repeat for every port which need to bypass vpn

And it works just fine.

Great to hear you got it working. Glad I could help.