Setting up a home server behind CGNAT -- but wait, there's more!

Hardware Hub Networking
,
1

I am currently working towards setting up a home server for most of the regular reasons plus maybe a couple slightly unusual ones, and am in no hurry to get it up and running and am as interested in the process as I am the result. I’ve got most of the hardware purchased and the deliveries are flowing in and in a week or two I should have it all here and ready to assemble and begin the setup process. But, one thing that I knew was going to be a major challenge was figuring out how to get this server connected to the internet and accessible from outside the home, because my internet setup is downright strange.

In the title I state that I am behind CGNAT, but to be totally honest with you I’m not even sure that term applies here. You see, I don’t even HAVE a router in my apartment. I have one RJ45 port in the wall that connects externally, and a WiFi access point that I am completely locked out of. My apartment complex is basically one giant LAN behind one single router, and they’re using passwords and accounts to keep the devices in each apartment separate from one another. I don’t rightfully know if that counts as CGNAT, or if that is what it means to go even further beyond.

I figured my first step was to call my ISP and ask what if anything they could do for me to assist in this process. Turns out, they were able and willing to assist! After getting transferred a bunch, I got to someone who just straight out sent me a modem and router to connect to the coax ports already in the apartment complex, at no additional cost to me. I didn’t even have to ask nicely – seems like they’d done this before for people who wanted normal networks despite living in this strange setup.

And now that’s where things get interesting. The hardware arrived the next day, I plugged it all in, and it works just fine – standard internet setup with a dynamic IP (they don’t offer static to residential customers, but that’s fine I understand there are workarounds). However, I noticed two things:

  1. These are two completely distinct networks. I can connect to each of them individually and they are both live at the same time.
  2. The upload speed through the router/modem/coax path is substantially slower than the RJ45/WiFi AP path. The original connection is approximately 750 down, 150 up, the new one is about 550 down, 20 up.

And now the devil on my shoulder has me thinking. What, if anything, can I do with two completely distinct networks that I couldn’t do with one? I could just use the new connection and move forward with that, but I’m not sure if that’s enough upload to stream video externally, and even if it is a part of me wants to see what all shenanigans I can get into.

Can I possibly:

  • Use pfsense or some other home router software to combine these into one ‘logical’ network that shows to my home network as one connection with 1300 down, 170 up?
  • Set something up so that the server sends data externally through the old network, but receives data from the new network? Basically have a device configured via wireguard / tailscale / what have you, it establishes the connection via the new network with the public IP, then does the file transfer across the old network via tunnel?
  • Do any other sorts of interesting shenanigans with this?

Basically, is it worth bothering with trying shenanigans with this, or should I just go forward with the new network with the public IP? I totally understand it would be complicated and beyond my current understanding – I’m fine with that, learning these sorts of things are half the reason why I wanted to set up a home server, and I’m in no rush and on no timetable to get it up and running.

2

Did a bunch of reading and some more testing over the weekend. The original internet connection definitely is CGNAT – I get the same public IP address whether I connect by wire to the RJ45 jack in my apartment, or if I connect by WiFi regardless of being on one end of the apt complex or the other (they service the whole complex and their WiFi covers the whole span via one AP in each apt). However, the port used by the CGNAT varies with every connection, just sitting and refreshing an IP checker site I get a different port every time.

So, if I understand networking correctly, and I’m sure I don’t entirely, the router is what encodes the packet that gets sent out into the internet, right? Or, at least, it converts the private IP address of the origin to a pairing of public IP address and port as part of the NAT protocol. Then it sends that packet out into the internet with a destination and lets all the magic work out there.

If that’s how it works, what if I set up two router VMs, one connected to each internet source, and I used some sort of routing rule or script to have VM A (the one with the public IP) act as the primary router for the network and send (most) of its outbound traffic to VM B instead of straight out to the ISP? VM B would then send the data out to the internet via its connection, but the packet would still have a return address of the public IP, and the recipient wouldn’t even be aware that any shenanigans had occurred – they sent a packet to the public IP address and got a packet back from the public IP address, nothing out of the ordinary to see here.

It wouldn’t load balance the traffic inbound to the network, but it would load balance the traffic outbound. But, I could connect some server VMs directly to the secondary router, namely the YouTube scraper I intend to run for short term archival purposes, which doesn’t need unsolicited inbound connections but will use a lot of bandwidth.

Network diagram:

image
image1286×578 58 KB

3

So you can definitely set it up that way if you want a High Availability pfSense setup but you can have multiple WAN connections on one pfSense box and do the routing/load balancing in one instance. There is a lot more that goes into a HA setup:

https://docs.netgate.com/pfsense/en/latest/highavailability/index.html

Give this a read and then get back to us if you have more questions as I think this will cover most of the bases:

https://docs.netgate.com/pfsense/en/latest/multiwan/index.html

If you don’t have multiple WAN ports you can also look into using a managed switch and VLAN’s

https://docs.netgate.com/pfsense/en/latest/multiwan/single-interface.html

4

Oh! It sounds like it might just work right out of the box without any major headache. That’s great news, now just to test it.

image
image729×234 24.5 KB

1 Like
5

You only need a single router. (if you need a router at all).

If you’re the only user, maybe it’d be enough for you to setup Tailscale on the server to be able to reach it from the internet. - generally if you’re streaming video you won’t need more than 150 - 200 Mbps peak for 4k UHD Blu-Ray.

If you want it publicly reachable over the internet, the question is if you need only https, or a couple of ports, or if you need a whole IP to yourself to run all kinds of services.

In the last case, the most cost effective solution is to rent a cheap VPS somewhere (e.g. can probably get unlimited bandwidth at 1Gbps for $5 a month, at least in most places you can). You can tunnel your home traffic through the VPS.

For other simpler cases there’s cloudflare or other tunnel, vpn, various other ddos protection proxies.

1 Like
6

Yea to get around CGNAT you will need to use tailscale/headscale or a external server that you vpn to and route your traffic through.