SSLH Routing - Protocol Multiplexing or "Into docker, out of it and back in again"

StillNewToThis · July 28, 2024, 10:19pm

Hello dear enthusiasts.

I have been a silent watcher and reader of L1 and this forum for a while, but today I’m active the first time since I really need some help.

I’m trying to set up a little homelab, with the primary and lonely host running Ubuntu-server 22.04, 5.15.0-117-generic.

It’s also the primary VPN gateway into my homenetwork, and it is running docker with some http applications, the reverse proxy of my choice being traefik.

Of course I also have SSHD running locally on the host.

To be as little annoyed by overreaching firewalls and censorships of other kinds, I stubbornly decided to funnel all encrypted TCP traffic through port 443.

To realize that, I want to run the FOSS SSLH (GitHub - yrutschle/sslh: Applicative Protocol Multiplexer (e.g. share SSH and HTTPS on the same port)) inside of docker, as that is the easiest way to update it.

OpenSSH and OpenVPN are running directly on the host.
Traefik of course is running in docker.

The goal is to run SSLH in transparent mode, so that I can still use fail2ban in SSH, so that my http applications will end up with correct logs.

For this the repository provides a two part instruction:

github.com

yrutschle/sslh/blob/master/doc/simple_transparent_proxy.md

# Transparent Proxy Configuration Using IP Routing#
This documentation is another explanation of the transparent proxy with the goal, beeing secure and minimalistic. Besides this documentation will explain, how and why this configuration works.
The explanation will only describe the connection to sshd, so the target sshd can be replace with any other target service, sslh supports.

## Introduction in the data flow ##
This chapter can be skipped, if you just like to configure things fast.
This chapter is a little excurse to the dataflow. First point of all is something, which you will unfortunately not see in the nice routing diagrams for iptables or netfilter-tables (nft) like: [Iptables at wikipedia](https://upload.wikimedia.org/wikipedia/commons/thumb/3/37/Netfilter-packet-flow.svg/2560px-Netfilter-packet-flow.svg.png), [Netfilter Flow](https://wiki.nftables.org/wiki-nftables/index.php/Netfilter_hooks#Netfilter_hooks_into_Linux_networking_packet_flows).

Packets from local application talking to other local applications are routed through the loopback-interface. They leave postrouting to lo and reentering from there prerouting, without passing ingress/egress.
This has nothing to do with the "**route_localnet = 1**" trick, which only makes, that the the local ip range 10.0.0.0/8 gets routed!
As you can read in many articles, this is nothing you should do, as you may bring your system at risk, because it allows to leak packets from outside to applications, which feel themselves secure, by using those unroutable addresses.
#### A Simple Simulation ####
You can prove this behaviour with a simple test:
```
# In one terminal start socat as a local echo server
# this is simulating sshd

socat TCP4-LISTEN:2000,bind=SERVER_IP,fork EXEC:cat

# In the next terminal start another instance of socat,

This file has been truncated. show original

github.com

yrutschle/sslh/blob/master/doc/scenarios-for-simple-transparent-proxy.md

# Three Scenarios for the simple transparent proxy setup #

![Simple Transparent Proxy Examples](./sslh-examples-v3.png)

## Introduction ##
The first example is the configuration, which was described in the previousd document. I omitted the loopback interface "lo" in those diagrams, trying not no overload the picture. 
The connections have two different endings, showing the direction of the opening connection (SYN flag) and the answer connection (SYN-ACK flags). This is important, as the traffic in the transparent proxy setup flows somewhat unexpected.

## Example 1 ##
The first example shows the setup, which is described in the [previous document](./simple_transparent_proxy.md). You see the Client connecting to sslh (red connection). When sslh accepts this connection, the SYN-ACK packet is send back to the client, which sends the first data packet(s) together with the ACK for the SYN-ACK. So the bidirectional tcp connection is fully open.
Sslh opens now the blue connection to sshd and needs for that elevation rights, as it uses the clients IP address as its own address for opening this connection. Now things are becoming complicated: Sshd send back the first packet with SYN-ACK flags (green line), addressed to the clients IP (dotted line). As already described, that would go wrong, so our routing trick makes this packet beeing deflected back to sslh, so this tcp connection is also opened. But we have here now an asymetric behaviour, that the read and write pathes of the tcp connection are going different routes.
The sslh process shuffles now all the bytes coming from sshd from the green line to the red line, vice versa for the packets from the client.

## Example 2 ##
In this example sshd is running on another server. No matter, if this is docker, kvm, virtualbox or another physical host, connected with an ethernet cable. Here we need no dummy interface, so we need another way, to configure our routing deflection.  The principle is the same: We need to force packets coming back from sshd going to sslh and not directly back to the client.
In this case its your decision, where those rules will be tied in, options are:

* the startscript of sslh
* the docker or kvm configuration
* the configuration of the outgoing interface

This file has been truncated. show original

What I have got so far:

SSLH container:

services:
  sslh:
    image: ghcr.io/yrutschle/sslh:master
    container_name: sslh
    environment:
      - TZ=Berlin/Europe
    security_opt:
      - no-new-privileges=false

    cap_add:
      - NET_ADMIN
      - NET_RAW
      - NET_BIND_SERVICE
    #sysctls:
     # - net.ipv4.conf.default.route_localnet=1
     # - net.ipv4.conf.all.route_localnet=1
    command: --listen 192.168.172.55:443 --transparent --foreground --ssh 192.168.255.254:22 --tls 192.168.255.254:8443 --http 192.168.255.254:8080 --verbose=3
    network_mode: host
    ports:
      - target: 443
        published: 443
        protocol: tcp
        mode: host

    restart: unless-stopped

Created a dummy0 interface at 192.168.255.254:

user@ubuntuserver:~$ cat /etc/network/interfaces
auto dummy0
iface dummy0 inet static
    address 192.168.255.254/32
    pre-up modprobe dummy
    ## Attention! with kernels, not automatically creating a dummy0
    ## interface after module loading the following line should be:
    pre-up modprobe dummy; if [ ! -e /sys/class/net/dummy0 ]; then ip link add dummy0 type dummy ; fi
    post-up ip rule add from 192.168.255.254 table sslh
    post-up ip route add local 0.0.0.0/0 dev dummy0 table sslh
    pre-down ip route del local 0.0.0.0/0 dev dummy0 table sslh
    pre-down ip rule del from 192.168.255.254 table sslh

Predifined a routing table:

user@ubuntuserver:~$ cat /etc/iproute2/rt_tables
#
# reserved values
......
111     sslh

With this configuration, applications running locally on the host work perfectly, and transparently even. SSHD logs the actual public IP of my client.
Even a small python http server I set up for testing is accessable.

However, if I try to access either a small Nginx-demo or traefik back in docker, I get nothing.

What I already tried:

What I already checked and tried:
-in principle everything can reach (as in ping) everything. The sslh container can ping my Nginx-demo container ( for the final config I would be running traefik here instead)
-If I try to access the nginx-demo on localserverip:8443, it works
-If I route directly to the demo container instead of the dummy0 interface, it doesn’t work, neither if I route to local host or the eth0 ip (localserverip or 192.168.172.55 as in the sslh config)

The configuration guide makes mention of this, but does not expand sufficiently for a tomato like me to actually understand it. I quote:

Remote Setups
This concept can also be adapted for several setups, where the sshd (or any other target service) is running in a container, kvm-virtual machine, etc. Precondition is, that the target system is the next hop and uses the sslh-hosting system as default gateway. In addition you need to bind an additional ip-address, solely used for sshd on the corresponding interface. Than you can adapt the routing rule, routing traffic coming back from this ip to the sslh-routing-table. Its also possible, to forward to an next hop system, which has its own default gateway back, bypassing the sslh-host. In this case, you need to add a special route back to the sslh host, for all traffic with the sshd source ip address. This can be done similar to the two rules described above:
    # first define a name for the table in /etc/iproute2/rt_tables e.g. sslh-routeback
      ip rule add from IPADRESS-OF-SERVIE table sslh-routeback
      ip route add default via IPADDRESS-OF_SSLH-HOST dev eth0 table sslh-routeback
The details are depending on your network settings. Als long, as the forward chain to the hidden service passes systems under your control, you can add backroutes on each system in that route. Precondition: The used ip address produces no conflict on those systems.

And while I understand a lot of these words on their own and not together, after trying and messing around for a week or so now, I just can’t get it done.

I’d be incredibly grateful if somebody could take me figuratively by the hand and explain this to me in the way a prosumer but not exactly network engineer can configure this.
Thank you all