Reverse Proxy Nginx traffic to my home server through wireguard

CodeDragon57 · July 17, 2022, 2:53am

Hopefully it isn’t the wrong thing for me to take this off of the main thread, but I don’t want to clutter it too much. I am continuing a conversation I had in Wendell’s HAProxy-WI post. HAProxy-WI -- Run lots of public services on your home server - #131 by Biky
I don’t have anything against the way Wendell did it; it is just that I don’t have the money for HAProxy-WI. I want to accomplish functionally the same thing Wendell did in his guide. Currently my idea is this:

Create a Wireguard Tunnel between my Linode and my Home server.
Proxy HTTPS traffic sent to my Linode back to the home server via the Wireguard tunnel so that traffic remains secure.

Seems pretty simple, but it isn’t working for me. Here is my Wireguard configuration for each server:

Linode Wireguard:

interface: tun_wg0
  public key: wxs+MiIwEj+24gAPeGp+DNhdc4j3Mgxcwvu/q9ebEWE=
  private key: (hidden)
  listening port: 51820

peer: CxZIh0B+kn2IKKqql0sy7E4t9COfVl4sJ+bTQej8yAc=
  allowed ips: 10.0.0.0/30

Home Server Wireguard:

interface: tun_wg0
  public key: CxZIh0B+kn2IKKqql0sy7E4t9COfVl4sJ+bTQej8yAc=
  private key: (hidden)
  listening port: 51820

peer: wxs+MiIwEj+24gAPeGp+DNhdc4j3Mgxcwvu/q9ebEWE=
  endpoint: 104.237.130.46:51820
  allowed ips: 10.0.0.0/30

I have also attempted to open Port 51820 on my PFSense box:

I am able to send and receive ICMP Packets between the two servers over the Wireguard interfaces on each, but I haven’t been able to get HTTPS to work. Either it is a problem with my Wireguard configuration, home network’s firewall, or my Nginx configs. I am using Nginx’s ability as a reverse proxy to keep things simple, since I am already fairly familiar with Nginx. If there is a better way; or if some other piece of software - such as HAProxy - would accomplish this better, I am open to that.

NGINX Configuration on the Linode:

   1   │ # vim: filetype=nginx
   2   │ 
   3   │ # HTTPS PORT 443 CONFIGURATION
   4   │ server {
   5   │     server_name         codedragon.dev www.codedragon.dev;
   6   │     listen [::]:443     ssl http2;
   7   │     listen 443          ssl http2;
   8   │ 
   9   │     location / {
  10   │         root /var/www/codedragon;
  11   │         index index.html index.htm;
  12   │         include /etc/nginx/headers.conf.d/0-security-headers.conf;
  13   │         include /etc/nginx/headers.conf.d/1-codedragon.dev-headers.conf;
  14   │     }
  15   │ 
  16   │     include /etc/nginx/ssl.conf.d/0-options-ssl.conf;
  17   │     include /etc/nginx/ssl.conf.d/1-codedragon.dev-ssl.conf;
  18   │ 
  19   │ }
  20   │ 
  21   │ 
  22   │ # HTTP PORT 80 CONFIGURATION
  23   │ server {
  24   │     if ($host = www.codedragon.dev) {
  25   │         return 301 https://$host$request_uri;
  26   │     }
  27   │ 
  28   │     if ($host = codedragon.dev) {
  29   │         return 301 https://$host$request_uri;
  30   │     }
  31   │ 
  32   │     server_name         codedragon.dev www.codedragon.dev;
  33   │     listen 80;
  34   │     listen [::]:80;
  35   │     return 404;
  36   │ }

NGINX Configuration on my home server:

   1   │ # vim: filetype=nginx
   2   │ 
   3   │ server {
   4   │     server_name techdragon.repair www.techdragon.repair;
   5   │     listen [::]:80;
   6   │     listen 80;
   7   │ 
   8   │     location / {
   9   │         root    /var/www/html;
  10   │         index   index.html index.htm;
  11   │     }
  12   │ 
  13   │ }

Thanks guys!

kobberholm · July 17, 2022, 5:23am

Am I just blind or is there nothing about proxying inside your Linode Nginx config?

I am running a Jellyfin server on my LAN, which connects to a Digital Ocean droplet over Wireguard.
On the DO droplet I have an Nginx reverse proxy, forwarding the traffic to the IP of my Jellyfin server, but I don’t see anything about proxying inside your Nginx config.

There’s a bit more in my config, but here’s what actually forwards the incoming http connection to my server at home. 10.200.200.6 is the internal Wireguard IP address of the Jellyfin server.

    server {
        listen 80;
        set $jellyfin 10.200.200.6;

        location /socket {
            proxy_pass http://$jellyfin:8096/socket;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header X-Forwarded-Protocol $scheme;
            proxy_set_header X-Forwarded-Host $http_host;
        }

        location / {
            proxy_pass        http://$jellyfin:8096;
            proxy_buffering   off;
            proxy_redirect    off;
            proxy_set_header  Host              $http_host;   # required for docker client's sake
            proxy_set_header  X-Real-IP         $remote_addr; # pass on real client's IP
            proxy_set_header  X-Forwarded-For   $proxy_add_x_forwarded_for;
            proxy_set_header  X-Forwarded-Host $http_host;
            proxy_set_header  X-Forwarded-Proto $scheme;
        }
  }

CodeDragon57 · July 17, 2022, 1:34pm

My bad, that was the wrong config file. I wasn’t Paying attention well enough cause I was tired and ready for bed.

   1   │ #vim: filetype=nginx
   2   │ 
   3   │ # HTTPS PORT 443 CONFIGURATION
   4   │ server {
   5   │     server_name         techdragon.repair www.techdragon.repair;
   6   │     listen [::]:443     ssl http2;
   7   │     listen 443          ssl http2;
   8   │ 
   9   │     location / {
  10   │         proxy_pass                              http://10.0.0.2:80;
  11   │         proxy_redirect                          off;
  12   │         proxy_set_header Host                   $http_host;
  13   │         proxy_set_header X-Forwarded-Host       $server_name;
  14   │         proxy_set_header X-Real-IP              $remote_addr;
  15   │         proxy_set_header X-Forwarded-For        $proxy_add_x_forwarded_for;
  16   │         proxy_set_header X-Forwarded-Proto      $scheme;
  17   │         proxy_set_header X-Forwarded-Protocol   $scheme;
  18   │ 
  19   │         include /etc/nginx/headers.conf.d/0-security-headers.conf;
  20   │         include /etc/nginx/headers.conf.d/2-techdragon.repair-headers.conf;
  21   │     }
  22   │ 
  23   │     include /etc/nginx/ssl.conf.d/0-options-ssl.conf;
  24   │     include /etc/nginx/ssl.conf.d/2-techdragon.repair-ssl.conf;
  25   │ 
  26   │ }
  27   │ 
  28   │ # HTTP PORT 80 CONFIGURATION
  29   │ server {
  30   │     if ($host = www.techdragon.repair) {
  31   │         return 301 https://$host$request_uri;
  32   │     }
  33   │     
  34   │     if ($host = techdragon.repair) {
  35   │         return 301 https://$host$request_uri;
  36   │     }
  37   │ 
  38   │     server_name     techdragon.repair www.techdragon.repair;
  39   │     listen 80;
  40   │     listen [::]:80;
  41   │     return 404;
  42   │ }

ThatGuyB · July 17, 2022, 2:02pm

The config looks decent. I can only assume that nginx doesn’t listen on the wireguard interface, because it may be launching after the server starts. Can you enable wireguard, then restart nginx on your home web server?

kobberholm · July 17, 2022, 4:47pm

You have to narrow down where it issue is occurring.
Is your web server even listening to 10.0.0.2:80?
On your server, try running curl -v 10.0.0.2:80
If that gives you the output you’re expecting, SSH into the Linode server and run the same command.
You should get an idea of where the connection breaks.

I noticed, that you are redirecting to the HTTPS interface from HTTP, which I think is only happening on the Linode side, but I’m not certain.
If you’re doing the same thing locally, I don’t know if that works correctly with the proxy_pass command. Also since you’re already communicating inside a tunnel, there’s no reason to to TLS(HTTPS) encryption at the same time.

CodeDragon57 · July 17, 2022, 5:48pm

I was thinking that that is what the listen; directives did…

I tried that and got a 403 Forbidden response code. I do not know why that is the case; so I will search for it, but any pointers you could give me would be helpful. Also, on the Linode side, the connection to 10.0.0.2:80 fails to connect because of no route to host. Could that be because of the home network’s firewall?

Correct. That is only happening on the Linode side. The entire reason I want to use the Wireguard tunnel is so that I do not have to set up TLS locally.

kobberholm · July 17, 2022, 6:39pm

I just noticed that your Wireguard output doesn’t show any information about the connections. Normally it should look something like this on the client (your local server)

peer: PXeK8RTJ45BneJYbY05iQO+wbHCej5Yc9UAJLFqlhDQ=
  endpoint: 112.222.131.112:51820
  allowed ips: 10.200.200.0/24
  latest handshake: 1 minute, 6 seconds ago
  transfer: 46.09 KiB received, 186.24 KiB sent
  persistent keepalive: every 21 seconds

And like this on the Wireguard server

peer: 7ljIUHASDMRGApu9si7uvqehj3b1LIOUHADSuNNY5mc=
  endpoint: 11.11.12.12:46880
  allowed ips: 10.200.200.8/32
  latest handshake: 6 seconds ago
  transfer: 42.33 MiB received, 10.63 MiB sent

Yours doesn’t have a “latest handshake”, indicating you’ve never successfully connected to the Wireguard server.

Regarding the Nginx server, you likely have incorrect permissions for /var/www/html, but it’s hard to say. Make sure there’s a file at /var/www/html/index.html and that it’s readable by everyone.
See here for more 403 troubleshooting:
How to fix NGINX 403 Forbidden

The firewall should be irrelevant, since you aren’t creating incoming connections, you’re establishing an outgoing connection from your LAN and all firewalls should accept that by default.

CodeDragon57 · July 17, 2022, 11:38pm

Thanks. I fixed the permissions.

Also, for the record, I do have lots of services already setup on my Linode - mostly Docker containers. For example, https://git.codedragon.dev is my Gitea instance. I am merely wanting to move them to my hardware.

So it seems like my main problem is the Wireguard configuration, so I will work on that a bit more.

ThatGuyB · July 18, 2022, 12:05am

Dang, and I wrote my config previously, but decided to kill it, because I felt it wasn’t relevant. Well, I am doing things in a weird way to get going without a NAT, but that’s besides the point.

VPS:

[Interface]
Address = 10.0.0.1/24
PrivateKey =
ListenPort = 51820

[Peer]
PublicKey =
AllowedIPs = 10.0.0.2/32

Your home web server:

[Interface]
Address = 10.0.0.2/24
PrivateKey =

[Peer]
PublicKey =
EndPoint = ip:51820
AllowedIPs = 10.0.0.0/24
PersistentKeepAlive = 25

Try the above.

Then, after you enable both, restart nginx web server. No need to restart the proxy nginx.

Regarding this, it is important that you allow port 80 to not get redirected on the home nginx. On Linode, you redirect 80 to 443 and then on the 443 vhost, you proxy_pass to your home nginx on port 80.

Also, if you want to move your containers locally, you will have to run them on the same server you are runing nginx on, otherwise you really need to set up the tunnel on your router. Which is not too hard.

My setup is weirder, I have a Pi running wireguard and set up the Pi as the default gateway in the network, because I can’t set routes on the devices in there. So I can either route them through the Pi, or pass the traffic from the Pi to the ISP router. Messy, but gets the job done. Highly not recommended to imitate what I did.

CodeDragon57 · July 18, 2022, 5:54pm

It doesn’t get redirected on the home Nginx. I don’t know why this keeps getting brought up. The vhost file is in the first post and it is labelled.

Thanks. It works like a charm now. It seems like my subnet mask was the problem. I changed it from the 10.0.0.1/30 to 10.0.0.2/24 like in your config.

CodeDragon57 · July 31, 2022, 3:47am

Update:
I screwed up the Linode’s Nginx configuration a few days ago somehow. I still don’t know what I did. At first, I couldn’t get the Systemd service to restart because some other service took over port 80 after I stopped Nginx. Then, I couldn’t get the domains to connect. From what I can tell, even though Nginx was “running”, it wasn’t receiving and/or routing traffic. I still don’t know what was going on there. It guess Texas was experiencing some freak cosmic ray storm lol (Texas is where my Linode’s datacenter is). Now, I have finally got everything back… mostly. For some reason, the Linode Nginx doesn’t like the response it receives from my local server’s Nginx now. I think it’s got to be something I’ve done on the Linode side, but I cannot think of what. I am fairly certain that I haven’t messed with the local server’s Nginx config. Here is /etc/nginx/nginx.conf:


# vim: filetype=nginx

user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log notice;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    server_tokens       off;


    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
}

CodeDragon57 · July 31, 2022, 4:14am

Turns out that it was cause by SELinux the whole time. ( I moved the Linode over to Alma Linux so that I could become more familiar with RHEL. I wanted to move to RHEL, but I decided Alma was better).