Need help with Network Isolation with risque people and risque ISP

Up until now, I have used this method through trial and error to configure my currently complex home network:

Internet -> FTTB -> VDSL2 Modem (Kasda Router) -> ASUS RT-AC68U (PPPoE login as WAN) -> ASUS RT-AC86U (WAN NAT off, OpenVPN is WAN)

The AC86U is my own private network, separated from my family’s network, and I really hope turning off WAN NAT is enough.

The AC68U is not separate from my Modem, which is a reconfigured router, and I can’t turn off WAN NAT or the PPPoE connection fails, but I want isolation from the ISP router. Currently, that is not possible unless you assign WAN as DHCP and you turn off DHCP on the ISP router, which causes DHCP to be overworked. I then made it a Static IP which helped somewhat, but the isolation is gone at that point and the Web GUI from the ISP router is accessible again, meaning LAN access.

Both ASUS routers are running Merlin, and unfortunately Merlin doesn’t have an intuitive VLAN creation GUI. I basically want several layers of isolation:

  • Isolate the ISP router LAN traffic from the AC68U, bearing in mind the ISP does not provide an admin login for the ISP router. (only user permissions)

  • Isolate the AC68U LAN traffic from the AC86U, making OpenVPN the only accessible WAN connection.

The big reason I need network isolation on my private network is because my Dad is still on Windows 7 and often goes to risque sites. And WannaCry HOPS to other PCs on the network. He’s stated “Why worry about viruses? You young people think malware is a big deal, but it’s just an annoying banner ad.”

The other reason is I’m on a Chinese ISP inside Canada. (YES, this exists)

I honestly only want to resort to pfsense or OPNsense or EdgeRouter as a last resort. As the the network will become extremely and overly complex for a home network.

I presume all you can do is separate sub nets until you can get an old/simple managed switch with vlan capability?
You say the Merlin interface does not easily allow for vlan creation? Could you install pfsense or something on Your Asus switch?

Other option is DD-WRT but that has extremely bad WiFi performance.

Next step I’m looking at is the EdgeRouter 4 with each interface on a separate VLAN, while retaining WAN access somehow. I need to also QoS the network traffic since that’s still needed on a 100/50 connection. The AC68U is being pushed to it’s limits doing QoS on the CPU, so I don’t know how I’m even able to improve it without more hardware.

Even if I get that, I’m not formally trained in large networks so I would be so lost in the Ubiquiti interface for setup of my complex network’s needs.

Could be worse.

I mostly stick to openwrt, the drivers / firmware are buggy, but at least they’re on GitHub and not going away, and I’m reusing hardware I already have. Ubiquiti proprietary network stuff is definitely higher quality.
(e.g. doesn’t require me writing a script to detect high/unusable 2.4 latency and unfuck the 2.4GHz wifi once this happens every few days).

Ubiquiti stuff is fairly popular - you’ll likely be able to find how to do stuff you need relatively easily online. I’d go as far as completely discount whatever perceived learning curve you might expect. Your setup also doesn’t sound overly complex (make one VLAN for yourself and for admin-ing things on the network, and another for your dad/family, maybe one or two more if you want iot/guest to be separate). Things like VLANs, multiple SSIDs per accesspoint, some forms of QoS are supported out of the box. I wouldn’t go with edgerouter, but would instead use the residential UDM (one with WiFi and quad core ARM - good enough for basic qos stuff) + nanohd/flexhd for additional accesspoints all around if you have range issues. (nanohd or other APs then pull their configs from a ubiquiti controller running on UDM over http/https, if https they check the cert unlike some of their competition. APs are fairly secure and don’t run much. Controller is in Java/mongodb and ubiquiti gives Linux repos and installers for windows/Mac if you want to run your own - easy to put behind nginx and secure if that’s what you want. The company has regular updates for everything, and if you have multiple APs they can do automatic rolling updates if you so choose.

Alternatively, I’d consider an x86 box instead of UDM and would just put basic Debian with maybe one usb/ethernet adapter for your ISP and another one for your dad’s network. And would turn those ac68u into dumb accesspoints… you can probably just turn off the http ui on them and leave only ssh running, right?

Okay, yeah using a UDM is too complex. I plan to take my AC86U with me because it’s such a good OpenVPN router. So I’m for now going with the Subnet idea. A 4 port EdgeRouter is pretty much the extent of how big my networks are going to be. The SFP port is a just in case for Gigabit Fiber in places where I might live in the future.

Edit: Uh oh, Latency significantly increased with different subnet masks.

Extremely confused about Subnet Mask topology specifically for network isolation, while retaining latency.

Once I started to use subnet masks, OpenVPN performance went south pretty fast.

If those aren’t mutually exclusive, only VLANs are worth it.

How’s your qos working. Is it just firewall classifying packets/connections somehow before they go into an interface queue? Is it possible some of it Is bypassing this classification because firewall has IPs somewhere?

Also, are your subnets entirely separate, or overlapping (different netmask lengths but common prefix?)

Also2, for OpenVPN… x86 is king, because aes-ni and sha extensions that openssl gets to use underneath. There’d be plenty of cpu leftover for qos and stuff.

I don’t know. That’s the problem.

So here’s how it’s now laid down after a few hours:

Modem (255.255.0.0) -> ASUS RT-AC68U (255.255.255.0) -> ASUS RT-AC86U (255.255.255.128)

But their IPs are different. Modem is 10.254.1.1, AC68U is 192.168.50.1, and AC86U is 10.1.1.1. AC68U connects to WAN over PPPoE, so I severed the LAN connectability for the Modem by using a automatic private IP (Set static IP to 169.254.1.254) and the WAN still worked. There’s one down.

The WAN on the AC86U is not isolated from the 68U unless I turn off NAT. The 10.1.1.1 is a dead giveaway that I want to use this fully isolated from the 192.168.x.x network and only use the VPN as WAN.

The AC86U actually has AES-NI and I’ve been experiencing pretty good speeds.

I did double check direct connections and it looks like my Chinese ISP added 17-20ms of lag since last night. fast[dot]com used to be 8ms to Seattle, now it’s 25ms.

It’s either the ISP or it’s internet backbones starting to crumble with everyone sharing protest stuff.

Edit: Found out why. Shaw’s Fiber infrastructure is adding 30+ms of lag in Vancouver right now. The Shaw hop in traceroute was consistently more latent than the rest of the hops. Blame Shaw.

1 Like

Yeah, I just don’t trust turning off NAT for the WAN isolation on the AC68U to AC86U connection. Though I don’t seem to be losing latency this time having a /32 prefix (255.255.255.255) on the AC86U WAN port with NAT off.

If I had to buy something, I might have to VLAN with a better router, then turn the wireless routers into APs, with the 86U AP being a OpenVPN client.

My biggest fear is ransomware being able to cross routers with NAT disabled on the WAN that’s connected to the other compromised network’s LAN.

How likely is that with VLANs? If the risk is equal I might just stick to the NAT off and Subnet solution.

some iperf3 numbers for it say 200mbps, I guess as long as it’s faster than your Shaw connection.

Zero.


risk’s quick and dirty wtf is a VLAN guide.

Without VLANs, and even without overlapping subnets, every machine is exposed to multicast and broadcast traffic from every other machine, can learn their IPs and can access whatever credentials are stored on the machine, or whatever credentials it might sniff out from the user to try and spread.

(I don’t know how much of today’s malware actually takes advantage of LANs, it was a big deal back when universities and companies ran "one big lan"s for entire buildings with a bunch of windows machines sharing local folders. )


With VLANs, you’re segmenting the network into separate LANs. (Think like 2 physically separate networks). “Virtually”, in this case means you can dynamically assign ports on a switch to each LAN (also called “access ports” - ports where hosts plug into); it also means if you have a pair of VLAN capable switches or a VLAN capable router you can designate ports as “trunk”, and prefix each Ethernet frame being received or sent over that port-cable with a VLAN tag header.


What do VLANs let you do? For example, you could just dust off that raspberry pi 3 or raspberry pi 4 with that single gigabit port, attach its only Ethernet port to a trunk port of a VLAN capable switch, and make it a router. It would be configured with a number of virtual interfaces parented by that physical real interface. On that trunk port (cable), one VLAN carried would be e.g. Shaw_WAN, another one would be DADS_LAN, and another one would be MY_LAN.
You’d then plug your modem into another switch port configured as an access port for SHAW_WAN, and the port next to it could be the access port for your dad’s desktop. And a port next to it could be the access port for you. Wifi accesspoints could connect into a trunk port, and accesspoints themselves would then unbundle this trunk into separate Ethernet interfaces, and would then have a separate SSID for each VLAN.

(btw: rpi3/rpi4 are a bad idea for OpenVPN, they’re like the only 64bit arm CPUs out there without some AES acceleration).


What happens when you connect a plain old device into a trunk port carrying tagged traffic? This can happen intentionally or unintentionally. In either case, a VLAN aware switch can have a PVID (physical vlan id) configured on a port telling it to “treat all untagged traffic as traffic belonging to a VLAN with some id you choose”. Symmetrically, you can ask a switch to not tag one of the VLANs being emitted from a trunk port. In all of these cases you’ll have a mix of tagged traffic and untagged traffic - we call these hybrid ports, or hybrid trunk.

A desktop/workstation computer plugged into a hybrid port could either ignore tags altogether, or ignore all tagged traffic. It’s a bad idea, tagged (trunk/hybrid) ports are most useful for network devices. You could also have a VLAN for untrusted VMs on your workstation, but then your host OS needs to deal with networking for that VM, in my book that makes it a network device as well.


Anyway, I think Merlin supports VLANs via CLI… It’s just not as clicky-clicky-follow-the-web-wizard as it would be in any of the ubiquiti stuff.

There’s just too many options for how to configure networking, sadly on either AC68 and on AC86 anything but Merlin just sucks.


Options:

  • UDM (it’s an all on one, if your house is not 4000sqft and you’re ok with a few hundred Mbps VPN performance on it, throw Asus into the skip).

  • UDM + another unifi AP - for extra range.

  • router on a stick : Some VLAN capable switch + rpi4 as a router/firewall/wireguard VPN thing… Yeah, can work. It’s a bad NAS because no crypto acceleration makes HDD expensive, but Wireguard works at 500+ Mbps. Use Asus/Merlin for multiple-ssid wifi.

  • Openwrt on R7800 (Netgear xs4), it’s got VLANs and OpenVPN in GUI, pretty good wifi.

  • small x86 with aes-ni with many ports like them qotom boxes that pfsense people like to use. Merlins for wifi.

  • router on a stick: odroid h2 (x86 with aes-ni; small; add storage and ram and it’s $200; two gigabit port) add a VLAN capable switch for more ports - there you go. Use Merlins as multi-SSID APs.

  • router with many ports: ryzen 3 on a B350 in an el-cheapo case / psu with a tiny bit of ram and some storage ($200), add a 4 port gigabit PCIe nic, or add a $20 10gig SFP+ and use a $130 mikrotik 24 port gig switch with 10gig SFP+ uplink to your Ryzen as a many port breakout box. … could be a nice nas.

  • Bunch of mikrotik or a bunch of ubiquiti stuff or a bunch of opensource everything stuff, or a pfsense + ubiquiti for wifi stuff or a pfsense plus Merlin or pay a few thousand per year for your salesmen friendly HP/Aruba corporate subscription licensed rent-a-gear wifi bs solution…

I find UDM (or a UDM + nanohd or flexhd for garden/outdoors or for more wifi) reasonable for home VLANs for a not-deep-into-diy-networking guy, all things considered.

2 Likes

The Dream Machine is $500CAD. Good design, but way over my budget. It also looks like the software is in beta state.

Still looking pretty intently on the EdgeRouter 4 unless that has features missing vs the Dream Machine. For instance, do I really need IPS and IDS? Is that for the most paranoid or is it justified considering my ISP is run by the communist party?

Er4 It doesn’t have wifi, meaning you’d have to rely on Asus/Merlin for , and runs edgeos. You get 4 calcium cores for $200 us (not sure how much in cad) … In the no-wifi category for $200, I’d prefer a Ryzen 3 + b350 shoebox router

Unfortunately, I’m not an x86 guy in the near term because I can’t justify building a system just for routing. I’m fine with the 2 Wireless routers, I just need something wired to replace the DHCP servers in those routers with the separated VLAN.

And is IPS or IDS really necessary? or is VLAN separation enough?

Edit: Oh, IPS and IDS is if you’re running a publicly facing server or enterprise scenario.

IPS/IDS is just a fancy regex filter that does man in the middle across all your traffic - if you have the CPU to do it. These days most malware infrastructure runs behind https cloud load balancers and uses websockets and not IRC anymore. That makes matching based on ip/port packet size and without provisioning certificates pointless.

All you need is some kind of low latency qos / traffic shaping/policy support and some kind of VPN.

I wonder how complicated or how easy it would be to get a separate vlan (another dnsmasq and set of nat/firewall rules) with your Merlin setup. Technically that’d be zero budget spent.

It’s iptables, but getting commands to run at boot time is “hard” to say the least. It depends on the JFFS which if it goes south, means a bricked router.

I just use ExpressVPN to bypass the local network over OpenVPN right now, but I’m just not sure if traffic from the AC68U LAN will leak through on the AC86U, despite setting /32 Prefix on both Router’s WAN ports.

It seems pretty cut and dry that VLAN separation on a managed wired router is what I need before the 2 wireless routers.

Maybe something like a “Mikrotik hex” or an “edgerouter-x” (same hardware between the two) is what you’re after. They should cost around $60, and support VLANs, PPPoE and basic iptables firewalling as needed out of the box. They’re both bad for OpenVPN, but either should do ipsec in hardware if you want to do that with express vpn. (Or can keep AC86 for OpenVPN).

Between the two I prefer er-x and its “smart queue” (basically just fq_codel underneath) slightly over plain old SFQ that Mikrotik let’s you set up. Mainly because ubiquiti lets you apply it on uploads only, and save some CPU if you’re on asymmetric links like I am (e.g. 500/50).

Mikrotik also has a hap ac2 which is a 4x700MHz arm quad-core for slightly more money 70usd and it comes with ok-ish 2x2 ac wifi, possibly good enough for “stereotypical parents use case”. But for then again you don’t need wifi (you already have it), and for $70 you could kit out a rpi with a pair of usb3 gigabit ethernet adapters (3 interfaces total), and it’d be a way better router and could maybe use it for other useful stuff despite it only being able to do like 200mbps of OpenVPN.

You sure the EdgeRouter X is enough for QoS on 100/50? My case is the down limit is set by the VDSL interface and that causes the most buffer bloat on hitting the limit. There’s minimal buffer bloat on upload, since that’s an upstream 50mbps limit.

@FurryJackman It sounds like you have two challenges:

  1. You want “isolation” from the ISP… something. Let’s skip that for now.

  2. Your residential network has untrusted devices. Let’s look at that.

Your “untrusted devices” issue is fundamentally the same as having Internet of Things devices in your home. That has been discussed quite a bit.
Steve Gibson proposed an approach on “Security Now” which he called “Three Dumb Routers”, and a search on those words turns up the original idea as well as critiques & suggestions. The original podcast (transcript available) explains why the configuration you propose is not the most secure.

Three Dumb Routers is a simple, straightforward way to tackle this issue.
(With respect, I notice that you disclaim networking expertise but plunge into use of VLANs and subnet masks - not sure you’re on the right track.) You already have two of the three routers needed, maybe all three.

In your case, the three routers would be:

  1. WAN Router - facing the Internet; just beyond PPPoE (I think)

  2. Untrusted Router - wired & WiFi connection for untrusted devices

  3. Trusted Router - wired & WiFi connection for trusted devices

Your AC86U is the Trusted Router. It sounds like the AC68U is serving as both the WAN Router and the Untrusted Router. It would be best to use another device for one of those purposes - maybe the Kasda Router could be used for PPPoE and WAN Router?

I had to look up PPPoE, but 2 minutes later, I am a Wikipedia Expert on the subject. It sounds like PPPoE is usually done by the telco/ISP equipment. But you refer to an “ISP router” which in context sounds like the Kasda router (and DSL modem). I speculate that perhaps the Kasda was originally used for PPPoE, then replaced by the AC68U to get WiFi? If so, then going back to the Kasda for PPPoE would let it serve as the 3rd router.

The EdgeRouter, or other boxes you mention, could play a role - but ideally they would go between the WAN router and other routers.

Good luck with getting a secure setup.

P.S. You ruled out pfSense, but for others who read this post I will say that pfSense, in dedicated hardware, is an excellent way to address the “untrusted devices” issue. Yes, there is some complexity (which brings very useful capabilities), and some expense. A good approach for many.

1 Like

Yeah, if it can handle the QoS needs of my upstream connection, my WAN router is the EdgeRouter.

The ISP basically made the Kasda Router dumbed down to only passing packets from the Modem without PPPoE, (so it’s only a modem) instead having the AC68U login to the tunnel setup for the WAN port… but knowing Steve Gibson and his excellent work, the EdgeRouter will work, if only it can handle the QoS needs of multiple VLANs. I would have a VLAN for the AC68U (untrusted), one for a Grandstream VoIP device, and one for my AC86U.

Sounds like I’m on the right path, I just need further upstream QoS and VLANs. I really hope the Edgerouter is fast enough, since if both download and upload are maxed, my 800mhz dual core AC68U is 100% on one core and 40-60% on the other core using fq_codel on Merlin. With that QoS setup, all hardware accelerations are turned off and all traffic is through the CPU.

You’d be setting 90/45 probably as the limits in HTB.

It should work out of the box even with pppoe and VLANs (it does work without pppoe). If not there’s a way to tweak HTB parameters to make the traffic accounting slightly more cpu friendly (increase quantum and burst sizing in HTB). There’s this thing called SFE (kernel patch) to make QoS work on er-x at 500Mbps, I doubt ubiquiti would be allowed to ship it on a mediatek device like the er-x, I doubt you’ll need it for 135 - 150Mbps.