VPS with HAProxy & Wireguard to local pfsense HAProxy & Wireguard

Ever since Wendells article on how to host everything on your local network I’ve been trying to achieve this. However there is a problem and I can’ seem to find it. So please help :slight_smile:

Disclaimer if a step is not described I haven’t done it! Guess that will help troubleshooting :man_shrugging:

Cloud

I’ve got Oracle clouds always free tier account and setup an ubuntu 22.04 instance there.

  • on my cloud virtual network I’ve opened ports:

    • 51820/udp
      80/tcp
      443/tcp
      ssh/tcp

wireguard cloud part

  • in my instance I’ve installed wireguard
sudo -i
apt update && apt upgrade && apt install wireguard wireguard-tools resolvconf -y
  • created my keys
    cd /etc/wireguard umask 0077; wg genkey | tee privatekey.key | wg pubkey > publickey.pub
  • configured my interface
    vim /etc/wireguard/pfsense_lab
[Interface]
Address = 192.168.22.22/24 # oracle_wireguard tunnel IP
MTU = 1500
SaveConfig = true
PostUp = iptables -A FORWARD -i %i -j ACCEPT;
PostUp = iptables -A FORWARD -o %i -j ACCEPT;
PostUp = iptables -t nat -A POSTROUTING -o enp0s3 -j MASQUERADE
PostDown = iptables -A FORWARD -i %i -j ACCEPT;
PostDown = iptables -A FORWARD -o %i -j ACCEPT;
PostDown = iptables -t nat -D POSTROUTING -o enp0s3 -j MASQUERADE
ListenPort = 51820
PrivateKey = REDACTED # privatekey of the oracle_wireguard

[Peer]
PublicKey = 76iLP1IxfvPeGmqn0GbaS7b9i5doIrcJcUhOUoNUUhA= #pfsense wireguard
PresharedKey = REDACTED # preshared key from my pfsense wireguard
AllowedIPs = 192.168.22.0/24
  • enabled the wireguard tunnel in systemd
    systemctl enable --now [email protected]_lab
  • enabled the ports in the firewall
ufw allow 22/tcp
ufw allow 80/tcp
ufw allow 443/tcp
ufw allow 51820/udp
ufw enable
  • added the ip_forward for ipv4
    echo "net.ipv4.ip_forward = 1" > /etc/sysctl.d/100-ip_forward.conf

HAProxy cloud

  • after that I’ve installed haproxy
    apt install haproxy -y
  • this is my haproxy config:
    vim /etc/haproxy/haproxy.cfg
global
  log /dev/log  local0
  log /dev/log  local1 notice
  chroot /var/lib/haproxy
  stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
  stats timeout 30s
  user haproxy
  group haproxy
  daemon
  maxconn 10000

defaults
  log   global
  mode  http
  option        httplog
  option        dontlognull
  timeout connect 10m
  timeout client  10m
  timeout server  10m
  timeout http-request 10m

# I know you can use front and backend separate but I've gone with the listener and kept all in one place
listen https
  bind *:443
  mode tcp
  option tcplog
  server default 192.168.22.222:443 send-proxy

listen http
  bind *:80
  stats enable
  stats show-node
  stats uri /REDACTED #uri for the stats to be shown
  stats auth $USERNAME:$PASSWORD # set your own username and password for haproxy stats
  server default 192.168.22.222:80 check
  • check to see if there are any problems with the configuration:
    haproxy -f /etc/haproxy/haproxy.cfg -c
  • start and enable haproxy on boot
    systemctl enable --now haproxy.service

This summs up the VPS section

if I don’t make that work I’ll ditch it completely and install pfsense on the vpc and do site to site VPN. But I hope I can still learn where my mistake is and not go that route.

Cloudflare

cloudflare disclaimer
I’ve transfered to cloudflare from namecheap because there were some problems with ddns between pfsense and namecheap. With this setup I hope I can go back to namecheap and ditch cloudflare.

I’ve added a dns record pointing to the public ip of my Oracle VPC

Type A
Name gitea
Content REDACTED # IP of your public VPC
Proxy status DNS only
TTL 1 min
Actions Edit

PFSense in local lab

Gitea web server

don’t let the name confuse you!!!
I have a simple webserver listening on port 80 nothing special
ip working:

haproxy working

Wireguard in pfsense

  • install wireguard on your pfsense

  • go to VPN / Wireguard / Settings and enable wireguard

  • go to VPN / Wireguard / Tunnels and configure your tunnel

  • go to Interfaces / Assignment and add your new wireguard tunnel interface then enable it

  • go to VPN / Wireguard / Peers and configure your peer

Endpoint should be your public VPS IP that you’ve also setup in you DNS provider.

  • go to Firewall / Rules / WAN and add rule for wireguard

  • go to Firewall / Rules / WireGuard and add allow all rule. This is for TESTING purposses only. We will tighten to rule later!

not sure if necessary but just in case I’ve added the same rule on the WG interface as well.

  • go to Firewall / NAT / Outbound set outbound NAT Mode to hybrid outbound NAT and add a mapping for wireguard:

DNS in pfsense

I’m giving all my servers a static DHCP lease.
my test instance is in the 192.168.77.0/24 DMZ network

  • I’m assigning a hostname to gitea1

  • then we need to assign a static DNS record in DNS Resolver that points to the firewall
    • go to Services / DNS Resolver / General Settings and at the bottom of the page there are Host Overrides. Click on Add to add a new one.

DNS disclaimer
I’m not sure about that part so please correct me if wrong but in order to have local direct connection to the server and not one thru the HAProxy I’ve added 1 at the end of the hostname so when I want to ping local without HAProxy it is gitea1 and with HAProxy it is gitea

ACME cert for haproxy

  • install acme on your pfsense
  • go to Services / ACME / Accountkeys and add a new key

  • go to Services / ACME / Certificates and add a new certificat

I’m reissuing the cert every 60 days and restarting haproxy. Check picture above for reference.

HAProxy in pfsense

  • install haproxy on your pfsense
  • go to Services / HAProxy / Settings enable HAProxy and add maximum connections otherwise its complaining …

  • go to Services / HAProxy / Backend to configure your back end gitea

Should be noted I’m using port 80 unencrypted traffic this is why encrypt(ssl) is not with a tick. If you use port 443 check the encrypt(SSL). Also if you have only a wild card cert you should not check SSL Checks

  • go to Services / HAProxy / Frontend to configure your frontend end gitea



I’m using a wildcard cert for my domain and it is the default cert. Since I have cert for my webserver I’m adding it to the additional certs tab

It does not work!!!

If I’m on my LAN, HAProxy is working after I remove accept-proxy in the Frontend HAProxy external listener. However both options does not work from external network. I’m testing on my mobile thru my phone. I’ve tested with Chromium, Firefox & Brave

when I try http://gitea.MYDOMAINNAME I get error

503 service unavailable
No serve is availabe to handle this request

which is expected because I’m not using port 80

when I try https://gitea.MYDOMAINNAME I get error

This site can't be reached
gitea.MYDOMAINNAME unexpectedly closed the connection
Try:
Checking the connection
ERR_CONNECTION_CLOSED

I’ve also tried adding DNS = 192.168.22.222 to my oracle cloud VPS wireguard and then the error changed to

This site can't be reached
gitea.MYDOMAINNAME took too long to respond. 
Try:
Checking the connection
ERR_TIMED_OUT

I’ve added all the steps so the more knowledgeable people in here could let me know where I’ve fucked up. Thanks in advance