HAProxy-WI -- Run lots of public services on your home server

Wikis & How-to Guides
106

Any of these should work. You would need to link the file in sites-available to the sites-enabledfolder, but that goes for any files in sites-available.

Nginx looks in /etc/nginx/nginx.conf, but the default version of that file includes any .conf files in the conf.d folder, and any files in the sites-enabled folder.

I would actually suggest putting it in a new file in sites-enabled or conf.d. That way, the configuration is separated out into individual files for each site, and the nginx.conf file could get overwritten if needed.

2 Likes
107

Thanks for your reply. Iā€™ll place it in sites-enabled then.
Just to be clear - will the config in sites-enable overwrite the settings in /etc/nginx/nginx.conf?

1 Like
108

Iā€™m trying to follow the how-to but I canā€™t get it to work.

At the moment I just try to have the Linode VPS act as a reverse proxy for a simple webserver. When this works I will add more backends.

I donā€™t know what Iā€™m doing wrong but I canā€™t get NGINX to serve the default static page (port 81). There seems to be something wrong in my configuration but I canā€™t find it.

When I enter my domain name (no subdomain) my browser throws this error message: The page isnā€™t redirecting properly

Here is my configuration files.

HAProxy.cfg

frontend main
        mode http
        bind :::443 v4v6 ssl crt /etc/letsencrypt/live/example.com/fullcert.pem
        acl url_static  path_beg        -i /static /images /javascript /stylesheets
        acl url_static  path_end        -i .jpg .gif .png .css .js

        acl root_dir path_reg ^$|^/$
        acl host_web hdr(host) -i web.example.com
        use_backend web if host_web

        use_backend static if url_static
        default_backend app

backend static
        balance roundrobin
        server  static 127.0.0.1:4331 check

backend app
        mode http
        balance roundrobin
        server app1 127.0.0.1:81 check

backend web
        mode http
        balance roundrobin
        server web xx.xx.xx.xx:10878 check ssl verify none

NGINX config (site-enable/default)

server {

        return 302 https://$host$request_uri;

        listen 81 default_server;
        listen [::]:81 default_server;
        server_name example.com;
        return 404;

        root /var/www/html;
        index index.html index.htm index.nginx-debian.html;

        location / {
                # First attempt to serve request as file, then
                # as directory, then fall back to displaying a 404.
                try_files $uri $uri/ =404;
        }
}

When entering my domain name in the browser (without subdomain) HAProxy outputs these log messages:

Jun 13 20:30:03 linodemachine haproxy[615]: ::ffff:xxx.xxx.xxx.xxx:5938 [13/Jun/2021:20:30:03.492] main~ app/app1 0/0/0/0/0 302 361 - - ---- 1/1/0/1/0 0/0 "GET / HTTP/1.1"
Jun 13 20:30:03 linodemachine haproxy[615]: ::ffff:xxx.xxx.xxx.xxx:5938 [13/Jun/2021:20:30:03.506] main~ app/app1 0/0/0/1/1 302 361 - - ---- 1/1/0/1/0 0/0 "GET / HTTP/1.1"
Jun 13 20:30:03 linodemachine haproxy[615]: ::ffff:xxx.xxx.xxx.xxx:5938 [13/Jun/2021:20:30:03.521] main~ app/app1 0/0/0/0/0 302 361 - - ---- 1/1/0/1/0 0/0 "GET / HTTP/1.1"
Jun 13 20:30:03 linodemachine haproxy[615]: ::ffff:xxx.xxx.xxx.xxx:5938 [13/Jun/2021:20:30:03.536] main~ app/app1 0/0/0/0/0 302 361 - - ---- 1/1/0/1/0 0/0 "GET / HTTP/1.1"
Jun 13 20:30:03 linodemachine haproxy[615]: ::ffff:xxx.xxx.xxx.xxx:5938 [13/Jun/2021:20:30:03.551] main~ app/app1 0/0/0/1/1 302 361 - - ---- 1/1/0/1/0 0/0 "GET / HTTP/1.1"
Jun 13 20:30:03 linodemachine haproxy[615]: ::ffff:xxx.xxx.xxx.xxx:5938 [13/Jun/2021:20:30:03.569] main~ app/app1 0/0/0/0/0 302 361 - - ---- 1/1/0/1/0 0/0 "GET / HTTP/1.1"
Jun 13 20:30:03 linodemachine haproxy[615]: ::ffff:xxx.xxx.xxx.xxx:5938 [13/Jun/2021:20:30:03.585] main~ app/app1 0/0/0/1/1 302 361 - - ---- 1/1/0/1/0 0/0 "GET / HTTP/1.1"
Jun 13 20:30:03 linodemachine haproxy[615]: ::ffff:xxx.xxx.xxx.xxx:5938 [13/Jun/2021:20:30:03.600] main~ app/app1 0/0/0/0/0 302 361 - - ---- 1/1/0/1/0 0/0 "GET / HTTP/1.1"
Jun 13 20:30:03 linodemachine haproxy[615]: ::ffff:xxx.xxx.xxx.xxx:5938 [13/Jun/2021:20:30:03.615] main~ app/app1 0/0/0/0/1 302 361 - - ---- 1/1/0/1/0 0/0 "GET / HTTP/1.1"
Jun 13 20:30:03 linodemachine haproxy[615]: ::ffff:xxx.xxx.xxx.xxx:5938 [13/Jun/2021:20:30:03.630] main~ app/app1 0/0/0/0/1 302 361 - - ---- 1/1/0/1/0 0/0 "GET / HTTP/1.1"

When I access the root domain I get 21 identical outputs in the /var/log/nginx/access.log:

127.0.0.1 - - [14/Jun/2021:07:04:52 +0000] "GET / HTTP/1.1" 302 161 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
127.0.0.1 - - [14/Jun/2021:07:04:52 +0000] "GET / HTTP/1.1" 302 161 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
127.0.0.1 - - [14/Jun/2021:07:04:52 +0000] "GET / HTTP/1.1" 302 161 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
127.0.0.1 - - [14/Jun/2021:07:04:52 +0000] "GET / HTTP/1.1" 302 161 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
127.0.0.1 - - [14/Jun/2021:07:04:52 +0000] "GET / HTTP/1.1" 302 161 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
... (21 lines)

When running ss -4 -tlnp only port 81 is listed. Shouldnā€™t HAProxy be listening on port 80 and 443?
LISTEN 0 128 0.0.0.0:81 0.0.0.0:* users:(("nginx",pid=4205,fd=6),("nginx",pid=4204,fd=6))

*Edited the post to add more information and log outputs.

I donā€™t know how to identify my error so any help will be highly appreciated.

1 Like
109

In the youtube video Wendell talks about NGINX listening on port 81 and HAProxy on port 80. In the HAProxy configuration snippet that he shares I can only see HAProxy listening/bind on port 443? Or am I missing something?

I have some problems with the redirect so maybe this is my problem?

110

They are appended.

So if you have the same config in both, it will probably complain that you have duplicates (for sites), or it will use the setting that is lower down in the configuration.

111

Even though it is very simple I have some trouble making it works. Iā€™m ending in a redirect loop when trying to access the static webpage on the root on my domain.

Iā€™m not sure that I understand the underlying concept. Here is my understanding of what happensā€¦ and please correct me where I am wrong.

  1. The user enters example.com in the browser
  2. The browser adds https:// in front of example.com
  3. Binded on port 443 HA-Proxy directs the request to default_backend because there is no subdomain specified in the URL
  4. default_backend sends the user to localhost /127.0.0.1:81 on port 81
  5. NGINX returns 302 https://$host$request_uri; sends the user to https://example.com, which reloads the page
  6. And then it starts with step 3 again.

What am I missing here?

EDITED:
Okay, I have narrowed it down to this line: return 302 https://$host$request_uri;

If I remove this line the website on the root domain is displayed. So, what is wrong with this line? And why doesnā€™t work in my setup? And can anyone explain the purpose of this line in Wendellā€™s example?

I have really tried to fix this redirect issue, but without any luck. So all help or ideas to how to debug this is highly appreciated.

Thanks.

112

Ah perfect. Thanks for the clarification.

113

I think I have found a solution to my redirect issue. I have set up two frontends, one for port 80 and another with my ACLs on port 443.

Here is my configurations files. If you have any feedback or suggestions to optimization please share.

/etc/haproxy/haproxy.cgf

  global
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
        stats timeout 30s
        user haproxy
        group haproxy
        daemon

        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private

        # Default ciphers to use on SSL-enabled listening sockets.
        # For more information, see ciphers(1SSL). This list is from:
        #  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
        # An alternative list with additional directives can be obtained from
        #  https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy
        ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
        ssl-default-bind-options no-sslv3
        ssl-dh-param-file /etc/haproxy/dhparams.pem

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

# Settting up nameservers to resolve my DDNS
resolvers mydns
        nameserver ns1 123.456.789.1:53
        nameserver ns2 123.456.789.2:53
        nameserver ns3 123.456.789.3:53

# Frontend accepting http/port 80 traffic - used to redirect http traffic to https
frontend http_in
        mode http
        bind *:80

        default_backend app_server_http

# If request is https then forward to servers on any subdomain or serve a static web page for the bots
frontend main_https
        mode http
        bind :::443 v4v6 ssl crt /etc/letsencrypt/live/example.com/fullcert.pem alpn h2

        acl host_web hdr(host) -i web.example.com
        use_backend web_server if host_web

        default_backend app_server_https

# Backend redirect http -> https. Listening on port 81
backend app_server_http
        mode http
        balance roundrobin
        server app1 127.0.0.1:81 check

# Backend for serving static webpage for bots. Listenin on port 82
backend app_server_https
        mode http
        balance roundrobin
        server app2 127.0.0.1:82 check

# Forwarding to my web server
backend web_server
        mode http
        balance roundrobin
        server web1 my-ddns-domain.com:18878 check resolvers mydns

/etc/nginx/sites-available/default

 #Redirect configuration http->https
server {

        return 302 https://$host$request_uri;

        listen 81 default_server;
        listen [::]:81 default_server;
		
        server_name _;

        return 404;
}

/etc/nginx/sites-available/example.com

# Configuration for example.com - the static website for bots
server {

        listen 82;
        listen [::]:82;
		
        server_name example.com www.example.com;

        root /var/www/html;

        # Add index.php to the list if you are using PHP
        index index.html index.htm index.nginx-debian.html;

        location / {
                # First attempt to serve request as file, then
                # as directory, then fall back to displaying a 404.
                try_files $uri $uri/ =404;
        }


}

Any feedback will be highly appreciated.

1 Like
114

@Dynamic_Gravity - how do you manage to get a minecraft server (tcp) to work with mode http as shown in your example? Iā€™m having trouble to get it to work.

I have tried but with no luck so far.

115

That backend is for Plan, which runs a webserver about the servers Minecraft metrics.

The Minecraft server itself should be behind a tcp LB but I had difficulty so I did SRV records instead.

116

Ah okay. Thanks for your reply. I will try setting up the server with a SRV record.

117

Would love a full walk-through on something like this. Just afraid I am going to break something bad!

118

Probably a little bit late to the party on this one.
the only way I can think of to get a minecraft server working with HAProxy, is to leave it until last and have it as a default backend.
have it so that all your ACLā€™s have to fail before it tries to connect to your minecraft backend.
You may even need a dedicated front-end for your minecraft, which checks all your ACLā€™s, forwards them through to your other front ends, and then if all else fails, connects to your minecraft backend.

1 Like
119

Hi @wendell, First I wanted to thank you. Lately you have been covering every
topic I have been working on / planning for: Workstation build, Home Server Series
Zettlekasten ( I started litterally reading about this topic 2 weeks before I
saw it on your YT channl), ā€¦ I can say leaving
TS was the best thing you ever did :+1:

Now back to the topic. I would say Iā€™m a heavy tinkerer with home workstations
services and proxying. In the last years most of my services have been running
in the cloud and I started recently upgrading toward a full at-home hosting so
this thread was perfectly timed.

I am still in the process of the migration and trying to figure out the best
architecture. I will first describe my previous current architecture that served
me well in the last years, then the new one I am upgrading toward. Hoping this
writeup might help someone, it will probably intersect with the Home Server
Series thread.

Table Of Content


#1. Current Architecture
## V1 - Nginx Reverse Proxy + DOCKER
### DNSMASQ and Wireguard
## V2 - Caddy Proxy + Docker
## V2 - Exporting services through TOR

#2. Upgrade to home hosting
## Automated forward service proxying ?

1. Current Architecture

V1 - NGINX REVERSE PROXY + DOCKER

Simplified Diagram (see Example Scenario Below) :

Docker Services :: NG rev proxy :: Cloud Host(A)  -----WG-----(WG PROXY VPN(B))----WG----- (OpenSense)| -> HOME
    |dnsmasqs                         |dnsmasq          |            |dnsmasq        \__ Laptops,phones ...
                                                        |______ secure machines

I was previously a heavy nginx user, especially the reverse proxy features. I
remember around 2012 using Nginx reverse proxy to automatically setup
subdomains to host git branches for testing. I also used to manually setup my
cloud services and proxy them through nginx. Then Docker appeared and I jump
on the wagon right away. Around the same time also appeared Wireguard, it
was the base to my current architecture which have been serving me for almost 6
years without a single major issue.

My setup allows me to have a safe access to my services from anywhere just by connecting to my VPN proxy(B). Almost everything runs on docker, and
all my machines including the cloud server are interconnected with wireguard.

I discovered later nginx-proxy for docker
and it turned proxying services to a whole new level. I could now provision
docker services and have them instantly and automatically proxied through nginx
by just adding the right labels to the containers. Everything is connected
through wireguard from my host server to my home and my phones/laptops and friends/family who join my VPN. I also run a speparate VPS as a pure VPN to avoid mixing my personal trafic and the rest of the trafic generated by the services.

DNSMASQ and Wireguard

I have multiple dnsmasq instances handling network translation between all my
machines as well as the docker containers that I select.

On Docker, I run a special docker image of dnsmasq
which automatically handles the network translation of my docker containers. So
I run one instance for each docker network that I use for proxying services.
Now the little magic is to name the networks the same way I name my DNS
suffixes.

There is an other dnsmasq instance on the host itself which does some extra forwarding with my VPN so that I can have my services reachable.

Then there is the VPN Host (B) which also runs dnsmasq. This allows me to have access to proxied services as soon as I join my VPN by just knowing the name of the
proxied service + the domain suffix.

The domain suffixes with dnsmasq and wireguard allow me to emulate subdomains
but for my private intranet over internet.

EXAMPLE SCNEARIO

Letā€™s use a calibre service for ebooks as an example. The service name is
calibre Letā€™s also say my DNS suffix for reaching my containers on docker is
wg.services. I create a docker network named wg.services and run the dnsmasq
instance over it. On my phone after I connect to my VPN, I can just type
calibre.wg.services topic access my ebook server.

V2 - Using caddy and caddy-docker-proxy

My next update was moving from nginx to caddy which
was much more minimal and flexible for me and comes with automated letencrypt certificates out of the box. Itā€™s minimal size means I can run as many caddy instances as I need and I often use it to host static HTML pages as portals to my services. The
Caddyfile structure is very simple and can even be generated by code if needed.

The automated proxying is handled by caddy-docker-proxy. It has the same function as nginx-proxy and can handle any caddy directive that can be written in a Caddyfile.

Exporting services through TOR

As an extra backup option in case my VPN is down I started testing exporting
services through TOR using docker-tor-hidden-services.
Big advantage, I donā€™t need to worry about SSL or Wireguard. I can just setup
strong passwords just in case. Theoretically I should be the only one knowing
the onion address and the traffic is encrypted by default.

I also have portal to my services through TOR with just a few lines of Caddyfile.

2 Upgrading to home hosting

So I built this Threadripper workstation to host all my services and I started
thinking about a way to safely make them availble publicly without exposing my
IP. Since I have a lot of services that are TCP/UDP based and not only HTTP, I
need a good multipurpose proxy service.

Automated Forward Service Proxy

My goal is to automate the export of the services running home to my cloud VPS
proxy machine. To do that I need a reverse proxy solution that offers an API so
that I can add a script that updates the proxy when new services come up
or down.

So far I these are my options:

I am planning to do a progressive migration where I bring services slowly to my
home server while experimenting with different options. My choice will boil down
to the least long term effort solution.

2 Likes
120

Hey, Iā€™ve recently had the good fortune to have the opportunity to have Google Fiber as my ISP (I live in Huntsville), so I want to try this; but the problem is that I need to deal with Dynamic DNS because they wonā€™t sell me a static IP Address for some reason, and Iā€™d like to use Wireguard as a direct connection between my Linode running HaProxy and my home server. I donā€™t really know how to do this given the whole DHCP problem on the server behind my network. I know how to set up DDNS on my PFSense box which serves as the primary Gateway for my network, but I donā€™t know how to do it from the Linux-powered box which is what I would prefer to do. Can anyone point me on how to get started?

Please note I do not need a DDNS provider as I use ClouDNS as my DNS provider and it supports DDNS.

121

Or better yet, could I use the Wireguard (or OpenVPN, but wireguard is preferred because imo it is easier to configure) connection to forward HAProxy traffic to the host behind my home network? Thereā€™s a few reasons why I donā€™t like DDNS tbh.

122

Itā€™s fairly easy to set up a tunnel between you pfsense and your linode box. Set pfsense as you would set a normal client, connect it to the VPS, but for allowed IPs, only allow your home serverā€™s IP to go through it (i.e. not all traffic, so avoid 0.0.0.0/0). After that, just configure your HAProxy domain, subdomain or simple URL to point to your internal IP address of your home server and youā€™re set.

You can use either Wendellā€™s or PLLā€™s guide on Wireguard setup.

123

Does this involve using DDNS? How do you connect PFSense to the VPS?

124

Awesome guide, Iā€™m doing something similar with haproxy running on my opnsense router pointing to internal services.

I have avoided pointing to my Plex internally because Iā€™m worried that video traffic will get routed through my haproxy on the router.

Is there any special configuration that allows me to use Plex.domain internal but not proxy all the video traffic?

125

Ok, Iā€™ve been banging my head for a long time now.
Some information about my network:

  • I have CGNAT.
  • My router is pfsense. Its inside my network so I have my isp router before that but I canā€™t open ports and do anything on the router.
  • For my cloud provider Iā€™ve chosen oracle cloud because of the free tier.

From what I understand I need to get wireguard from pfsense to the cloud. From there do I install HAProxy on the same host where wireguard is, or I configure it on the pfsense box?

PS I know that I can forget about my problems with cloudflare tunnels but I donā€™t want to be that reliable on cloudflare