If anyone is interested I found a way to solve my problem. Well I am not sure if it is a solution or a workaround that is just masking my routing problem. But after some more digging, I stumbled accross:
IMPORTANT: We received a report that MASQ and SNAT at least collide with marking packets. Rusty Russell explains it in this posting. Turn off the reverse path filter to make it work properly.
Linux Advanced Routing & Traffic Control HOWTO / Chapter 11. Netfilter & iproute - marking packets
Testing here reveals that the route filtering and mark don’t play well
together. Try:
for f in /proc/sys/net/ipv4/conf/*/rp_filter; do echo 0 > $f; done
echo 1 > /proc/sys/net/ipv4/route/flush
Rusty Russell: routeing, SNAT, MASQ, and fwmark
To test this out, I setup openvpn and routing table as above with the following rules:
clifford@router:~$ sudo iptables -t mangle -A PREROUTING -s 192.168.1.12 -p icmp -j MARK --set-mark 1
clifford@router:~$ sudo ip rule add fwmark 1 table AWS priority 10
And then setting up some logging:
echo 1 > /proc/sys/net/ipv4/conf/all/log_martians
to show traffic dropped by the rp_filter. And to show how far the traffic actually gets before being dropped:
iptables -t raw -A PREROUTING -i tun0 -p icmp -j TRACE
iptables -t raw -A PREROUTING -i enp7s0 -p icmp ! -s 192.168.1.14 -j TRACE
(192.168.1.14 is a host that produces a lot of icmp traffic that is a pain to sift through but isn’t relevant for this post.)
With that in place, a simple ping from host 192.168.1.12 to google is predictably dropped:
ping -4 -n 1 google.com
Pinging google.com [72.195.166.57] with 32 bytes of data:
Request timed out.
Ping statistics for 72.195.166.57:
Packets: Sent = 1, Received = 0, Lost = 1 (100% loss)
which is logged on the router as:
clifford@router:~$ tail -f -n 0 /var/log/syslog
May 10 10:24:14 router kernel: [76133.452124] TRACE: raw:PREROUTING:policy:3
IN=tun0 OUT= MAC= SRC=72.195.166.57 DST=10.8.0.6 LEN=60 TOS=0x00
PREC=0x00 TTL=45 ID=42399 PROTO=ICMP TYPE=0 CODE=0 ID=1 SEQ=17
May 10 10:24:14 router kernel: [76133.452132] TRACE:
mangle:PREROUTING:policy:2 IN=tun0 OUT= MAC= SRC=72.195.166.57
DST=10.8.0.6 LEN=60 TOS=0x00 PREC=0x00 TTL=45 ID=42399 PROTO=ICMP
TYPE=0 CODE=0 ID=1 SEQ=17
May 10 10:24:14 router kernel: [76133.452136] IPv4: martian source 192.168.1.12
from 72.195.166.57, on dev tun0
So it is the reverse route filtering that is dropping traffic.
Loosening the policy results in traffic going through: rp_filter = 2):
root@router:~# echo 2 > /proc/sys/net/ipv4/conf/tun0/rp_filter
And the subsequent ping attempt:
Pinging google.com [70.186.10.23] with 32 bytes of data:
Reply from 70.186.10.23: bytes=32 time=92ms TTL=42
Ping statistics for 70.186.10.23:
Packets: Sent = 1, Received = 1, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 92ms, Maximum = 92ms, Average = 92ms
Oddly enough, turning off the policy altogether:
root@router:~# echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter
doesn’t work. Which I don’t really understand.