Hello everyone! Recently I have faced some issues with my current ISP, which provides me a 300/150 fiber connection (with no data caps), which is awful for me since I work from home.
Due to some contract issues, I decided not to cancel it, but rather get another link from a different ISP, which provides 500/250, but with a 2TB data cap and use mwan3 to balance those.
I currently have a TP-Link WDR4300 with OpenWRT and a mesh wifi system as my APs. Since my currently router is long overdue, and isn’t able to handle a 300+500 link aggregation with SQM, I was planning on buying a new device to replace it. Doing load balancing alone it maxes out at ~400Mbps with the CPU pegged at 100% (way more than I expected to be honest).
My current option is a NanoPI R5S, which has 3 ethernet ports (two of those being 2.5Gb), and should be fast enough to handle my scenario whilst being reasonably cheap at less than $100.
The R4S, while cheaper and with a more powerful CPU, only has 2x 1GbE ports, making it hard to have 2 uplinks + a switch. I also looked into some x86 options, but they are more expensive, consume more power, and I have a weird kink for having different ISAs at home (like some MIPS and RISC-V devices that I have laying around). The R6S, on the other hand, is plenty powerful but way more expensive.
Does this seem like a good idea, or are there any other better options available?
I’m not currently using it in my router (not like it could handle it anyway), but since the new router will be kinda hefty, I’m planning to have stuff like grafana, minio, gitea, loki, and many other DIY images that I might come up for my applications (I already have some that I run in my desktop for DDNS, network quality, etc) running on it.
Using Docker for me makes doing so pretty easy and painless without messing with the base system in any way.
Ah right… like i kinda expressed in a similar thread today IMO for the wife/family factor I would likely suggest that you keep your router as a standalone device and keep your dockers/pihole on a separate device. I have found that is the most robust solution when your a tinkerer like me.
Have you ever had any issues when spinning up up/down containers in a device like so?
My original plan was indeed to have a different device for that and toy with k8s without giving a damn, but until then I can’t see how such basic servers in their own sandbox would interfere with the router itself, but I’m all ears to keeping those ideas at bay if issues are common to arise haha
If you have issues with any sort of container/VM/device requiring reboots or docker lockup it takes out your internet for a period of time. If you have issues with the host device due to whatever like hardware failure or drive issues, your internet/DNS again goes flakey/out. If your not at home and your family is home while issues arrive they likely cant diagnose issues while your out apart from rebooting it, which means they have no internet or worse (your local lan serving video files or whatever also stops working). If you have issues with a single pihole instance or whatever with no failover or something else higher up in the chain handling DNS (like a NextDNS) again you have internet issues. These are various things I have experienced and while my setup now is rock solid, part of that is in my belief now that i can take down various instances/services yet my main internet and DNS all function for the rest of the family which allows me to tinker more and hear complaining less lol.
You will probably top off at 2.something gbps total throughput (in+out) if you don’t use SQM, less if you use it, so I am skeptical you’ll be able to reach 800Mbps
For these kind of speeds and the way you want to use it I am afraid you’ll need more CPU, and possibly move to a platform that uses Intel network chips instead of realtek (even more $$$)…
And forget about using docker in top of that, if you really need the throughput…
I would advice against going for a device that has no support upstream as OpenWrt device support very rarely gets upstreamed (FriendlyElec’s track record is very poor) so you’re basically going to be stuck with OpenWrt which given your plans you will probably find very limiting. You don’t have that many options going for a router/firewall and being cheap except for RK3399 based boards such as RockPro64 and Rock Pi 4 (RockPi4 might be a more troublesome as it doesn’t have a “full size” PCIe slot). You might also want to look into Solidruns Marvell-based solutions but they’re a bit more pricey but I’m not up to date on mainline support (I think U-boot is a dead end). Most of what you list can easy be maintained within a few daemons and should be handled just fine with any decent package system.
Your can keep your wdr4300 as a 5 port managed switch + and do a “router on a stick setup” with e.g. one 1Gbps link per ISP would let you simultaneously max out both, it’s a bit wasteful in terms of port count it leaves you with 1 LAN port.
Belkin rt3200 is pretty much permanently discounted to $100 and below.
Older wrt1200/1900/3200 have crappy wifi but worth $30-$50 in your use case, ebay for them or look at discounts.
Odroid h3+ is around $300 if you add a network card and ram, but fairly compact and nice for the port count.
Building a B660+12100 is also about $300 , not as compact but a beast.
Regardless of CPU issues, I’d highly advise against running Docker on a router/firewall box, at least not without Docker running in a separate VM on the box.
Docker changes stuff in iptables for it’s own needs, and that’s the last thing you want on your router.
Do you have an idea on how much perf you trade when using a simple SQM algo like piece_of_cake? Losing almost a third of throughput would be really harsh.
The RockPro64 is kinda hard for me to source (same applies to Solidrun), but I will take a look at the Rock Pi 4, seems like it’s in the same price range as the R5S whilst being more powerful CPU-wise.
Most of the options you mentioned are hard for me to source, since I’m pretty much stuck with on Aliexpress here in Brazil
However, the idea of getting one of the previously mentioned devices with USB NICs + my current router as a managed switch is something to really think about, thanks for that!
outside of us/canada/uk/eu/aus/… and a few other places where it’s easy to order something and generally just either receive an email that you’re supposed to pay some predictable customs rate … and don’t need to stress about thieving couriers and warehouses in addition to your thieving neighbors…
… I’d just recommed doing your own 11th/12th gen intel ddr4 or am4 ddr box – with whatever locally sourced parts you can reasonably get… the overhead of your own time and nerves is huge, and dealing with it for one box + a couple of upgrades down the line makes for an easier way to live.
For the first time docker user, … yes absolutely I agree, you should learn about docker networking before venturing into this deeply, regardless of whether you click install random plugins for something that happen to be within containers in some random UIs, or whether you use e.g. docker compose or something else.
TL;DR; … if you want any rules to rule before docker rules, add them to the DOCKER-USER chain.
...
What docker (and similar) do is documented and is generally not unreasonable, and once you get a hang of how docker manages networking for you, you’ll understand also why. It’s not hard to get everything to work together nicely, … but …
… if you install random containers off the internet that people who don’t understand containers wrote to learn containerization, and without you yourself understanding how they’re setup and with somehow randomly setup docker networking … exposing all the ports and bridging everything and with --privileged or running as root or doing CAP_NET_ADMIN where not needed all the things like that (sudo give me a sandwitch), … then you get random holes in your firewall probably – it’s not much better … or might even be worse than curl xxx | sh and giving the script your password without understanding what the thing does.
… you can basically look at output of iptables-save and see what it does when you fiddle with the first few containers.
I personally like using docker compose, and I ensure all my services have their own networks (or no network), and that those networks that end up with bridges end up with named bridges so that I can understand the iptables rules later on and so that I can refer to them in DOCKER-USER chain, and because it’s a text file, I can check it into git and evolve it and improve it over time. I also have some macvlan networks – that’s the easiest way to give an entire interface to a container e.g. if you want your router and primary firewall to be in a container.
it works nicely eventually, it needs a bit of “getting the hang of of it”.
I consider both Docker and firewalls a tool O use, not my main focus. And, frankly, never learned iptables. Had a simple situation: bridged kvm and Docker on host. When Docker was running I just couldn’t get traffic to the VM. After over half a day of trying to figure it out, I gave up and chucked Docker into it’s own bridged kvm. Later migrated both to LXC. Works.
That said, I only ever use first party containers, my own containers (for builds, not services), or big projects like LinuxServer.io
Another point to consider, how will an appliance OS like OpenWRT or pfsense react to a foreign application messing with iptables.
pfSense is FreeBSD based … I don’t know if it can even run OCI containers… (yes I know FreeBSD kernel can emulate/translate Linux syscalls … Jails aren’t containers, bhyve-ing a Linux kernel to run containers doesn’t count as FreeBSD running containers…).
For something like OpenWRT, there’s some documentation that sadly doesn’t answer your question here: [OpenWrt Wiki] OpenWrt as Docker container host (I guess one could just opkg install docker* and have a look?)
TBH I find the other way more interesting given that each container has it’s own network namespace, tell docker to clone a pair of macvlan interfaces and attach them as WAN/LAN to an OpenWRT image running a luci web ui in a container…
… and then use whatever Debian or Alpine or something simple and mostly immutable on the host - something bare bones close to upstreams with minimal updates… basically like a hypervisor would come with some stripped down set of tools, just throw that on the host to manage containers.
I don’t really have an issue with delivery, it’s just a matter of trying to import from other sources ends up way more expensive than Aliexpress. I personally want to avoid a custom box like that, both due to power consumption, cost and it would be yet another boring x86 machine at home.
As for your docker tips, yeah, I usually know my way around docker, but never really cared about what it does with the firewall since I always trusted my router’s firewall to keep things at bay. Well, using it IN the router brings a new perspective now lol
Pine64 store is located in .cn just like Aliexpress
You also have vxlan in FreeBSD which might be of interest but if you’re deadset on docker there isn’t much of a choice…
(way more than I expected to be honest).
