Return to Level1Techs.com

Phaselockedloopable- PLL's continued exploration of networking, self-hosting and decoupling from big tech

already have some raidZ2 arrays… probably will use that if I need more

https://lxd.readthedocs.io/en/latest/clustering/

Interesting. I will look into this…

I prefer the Pis im trying to run ARM as much as possible with existing hardware. The pi4s OC’d to 2.1 GHZ and with SSDs are stupid… stupid fast

This is a good idea. Maybe a 16 RPI4 cluster? 64 ARM64 cores… lol. Thats basically a damn server LOL. Im going to need different power delivery … KEK

They all have large passive heatsinks on them (see somewhere above in the thread)

1 Like

For the main / host OS that would run LXD, SD cards are just fine. But SSDs will definitely make a difference for the service that run inside LXD. OC’ed means even “more better.”

Understandable. I also want to get away from x86. Check out the Wiretrustee SATA and also this article.

At this point, a Gigabyte ARM Server with Marvell-ThunderX2 makes more sense. Maybe 2 for redundancy with 1 Pi for tie-breaking.

2 Likes

well see I already have 8 (some not in diagram) so 8 more would be less than the marvel thunder

2 Likes

Following up Biki … and @Dynamic_Gravity you might as well be included in the loop since we spoke of SPOFs

Im going to attempt instead of multiple NGINX handling the talking of the nodes to each other…

Im going to instead push for this

Which means Im going to have to figure out multiple tenancy on wireguard on OPNSense

The planned centers: (reason is I travel globally alot. and thats kicking back up)

Budget 40 dollars a month max. Im think 5x$5 linodes? and a balancer (Closest Hop not Round Robin)

The question really is how much sense does it make. All of those locations would just be reverse proxies and ultimately would have to route back. I guess making for the closest proxy location is a form of optimization right?

I travel a lot within the US and europe so I was thinking concentrating the server/reverse proxies there… 2 in NA 1 in Europe 1 in AU and 1 in Asia

Also if any one server or data center goes down it doesnt take anything out

But the question is do NodeBalancers also have a location or are they mirrored to all locations. I really dont know if I am doing the right thing here


An alternative route is forgetting that and upgrading the server to Dedicated CPU… but I really dont think that would net much benefit

1 Like

You probably know I love frugality and scaling down to the bare minimum and KISS, so my answer may be a little… unsatisfying.

If the NodeBalancers have to make the trip back to your main server, then this will only make sense if Linode has leased lines between their locations. Not sure if that’s the case, but if it is, you will probably see a difference in speed or latency. And traffic has to travel through their leased lines and I doubt the users are allowed this privilege (that’s usually reserved for in-house traffic, like management or really latency-sensitive communications).

Another case in which it would make sense is if those NodeBalancers are caching webpages and requests and stuff. I have no idea how they work though. Reading a little about it, it appears they are just basic load-balancers. So just like any LB, these things only make sense when you got loads and loads of traffic going through your server (1k+ connection at a time from each NodeBalancer). Are you selling stuff or own a very popular website? I don’t think this makes either technical, nor financial sense for 1 user.

But if security is the issue, well, like any LB or reverse proxy, it is an added security feature. NodeBalancers communicate with the Linode web servers through private IPs (and if you’re on a limited bandwidth in Linode, this will help a little), so the main web server is not directly exposed to the internet.

I read there are some issues with hosting NodeBalancers with SSL on the web servers. You can’t get the real IP of the source, just the IP of the NodeBalancers and some websites don’t show up correctly (the steaming pile of garbage that is WordPress). To avoid this issue, it’s recommended you keep the SSL certificate on the NodeBalancers and have unencrypted traffic from them to your web servers. When it’s your own infrastructure, this should be fine, but I don’t know how it works in Linode, are you the owner of the NodeBalancer, or are they shared among multiple customers? If you want SSL on both the load balancer and on the web servers, HAProxy is the better solution.

And again, it’s certainly a cool factor to add redundancy and load balancing to your services, but if it’s just for you, this doesn’t make much sense. You can play around with VMs / containers in your own home lab and set things up as if it was a large corporate intranet. Paying additional money for 1-100 people doesn’t make much sense.

In the end, most likely you won’t see much of a difference in performance. It might add some privacy if you are connecting via the clearweb to your Linode web server, your traffic will show up as connecting to a regional server, which then goes through Linode’s VPN to your main instance, instead of you going directly to it. But then again, a reverse DNS lookup will show what you were connecting to, so kinda a moot point. If privacy is a concern, you can make a script to spin up Wireguard on new Linode instances and connect to them when you travel (or have them constantly running for ease of use). Bonus points for VPNs is that you can also use the private IP of your Linode instance to connect to it, so you also save on bandwidth costs.

Again not worth the investment. Unless you are serving customers that demand your service not be dog slow and consequently you or both of you losing money, shared resources are better. It’s not often that services are pegging the CPU constantly (and if they do, sometimes VPS providers move the demanding VMs around not really demanding ones).

3 Likes

Just another thing I have been working on. In blog post for index post

1 Like

YAY Series 7 done

Links to Infrastructure Series and Blogs:

Blog: Phaselockedloopable- PLL's continued exploration of networking, self-hosting and decoupling from big tech
Series 1: Infrastructure Series -- Native Dual Stack IP4+IP6
Series 2: Infrastructure Series -- Wireguard Site to Site Tunnel
Series 3: Infrastructure Series -- Recursive DNS and Adblocking DNS over TLS w/NGINX
Series 4: Infrastructure Series -- NGINX Reverse Proxy and Hardening SSL
Series 5: Infrastructure Series -- Taking DNS One Step Further - Full DNS Server infrastructure
Series 6: Infrastructure Series -- HTTP(S) Security Headers! You should use them! [NGINX]
Series 7: Infrastructure Series -- Use NGINX to inject CSS themes

2 Likes

Well, appears I shall need another pi for NGINX… Unless I can run it off my 24/7 PC I use…I can switch over the Linux for everything on it I suppose. (Would use the Ryzen 5 4650G x300 ASRock Mini PC/ 32GB ram… plenty i suppose?)

As a novice I will say, I understand the concept of DNS, reverse DNS sort of, but recursive…not so much. I think I also need to have my E-mail alerts configured properly (I still have yet to learn this), I did get NGINX installed and working properly on a test machine with Fedora as the base OS… I assume I could possibly ru8n a headless distro like debian and install it and get it running as well, but I dont know what kind of resources it will need access to.

For the time being I may have the firewall in a loop to two ports on my switch so I can play with it, see if I can route through it to test its working properly to just this one machine I am on, then move to put it between the ISP modem and the switch to the AP. I do like the idea that @Biky gave me of having 3 VLANs for different trust levels… I myself for my sanity would probably need different IP ranges to keep in straight, then of course see what I allow to cross talk to each other on the switch…Ok head is full now… going to see what I can break. lol

In general I’ll all also have to redo permissions on some of my Samba shares… I could never get them to work how I wanted, or I may wimp out and use TrueNAS in a VM.

But a netmap is forming. yet the wife becons agaion for the 10th time as I write this…lol

2 Likes
1 Like

yeah hes a good resource. Also more currently frequent to the forum. I feel bad that ive taken to an absence but its generally been busy

4 Likes

As I’ve said no worries at all. You do you! Lol I’m having fun bumbling around. :grinning:

2 Likes

I can often be caught on discord more often the level one discord.

My handle is “rear” admiral dox

The issue I have is also that I don’t open the browser that often I don’t live with notifications

2 Likes

There’s always doing clever shit with RSS

1 Like

No. I live a non live notifications life is the life. I disconnect regularly. I dont always live attached.

2 Likes

Someone thinks that I have notifications for things.

2 Likes

I did not say you lol

2 Likes

I need to go “back to school” on this…lol My books are proving not as useful as I’d hoped lol

Try looking online for CCNA resources. If you want the certification, buy the courses. CCNA is in more high demand only because of brand recognition and they arguably offer some more advanced knowledge than CompTIA. They are Cisco specific most of the time (in some labs, you may do OSPF and LACP, which are open standards), but if you know one, you basically know them all, just a difference in terms and commands used.

2 Likes

Took a while to figure out how to mitigate the 53 DDoS.

Basically my situation was I could not stop the incoming flood traffic, well guess what …? iptables string-match rules can be used as a last resort. Keep in mind that string matching is resource-intensive and may slow down your network speed and your servers btw… however if you need to shut out an attack, this option is better than letting your server continue to answer the abusive queries to port 53

so the first thing I did was get a hex-dump of the packets I wished to block. tcpdumping them out

tcp6dump -c10 -pntxi eth0 not udp src port 53 and udp dst port 53

The option -c10 specifies that you only want to dump out 10 packets at a time from port 53 only looking at input.

I was seeing this bullshit

IP [05e9:3aa4:3230:86b9:0737:5c22:13ff:f2ad].59174 > [2600:3c04:0:0:f03c:92ff:fec6:2030].53: 92732+ [1au] ANY? . (28)
        0x0000:  4500 0038 a48c 0000 f111 4e02 4509 64f2
        0x0010:  d857 54d3 e2f4 0035 0024 0000 aa3d 0100
        0x0020:  0001 0000 0000 0001 0000 66ff 0432 1189
        0x0030:  5348 0000 0000 0000
IP [05e9:3aa4:3230:86b9:0737:5c22:13ff:f2ad].16326 > [2600:3c04:0:0:f03c:92ff:fec6:2030].53: 92733+ [1au] ANY? . (28)
        0x0000:  4500 0038 d50a 0000 f111 2220 636c 41f3
        0x0010:  d857 54d3 4700 0035 0024 0000 9f8b 0100
        0x0020:  0001 0000 0000 0001 0000 66ff 0432 1189
        0x0030:  5348 0000 0000 0000

Guess what theres a common conseutive set of bytes here… 0000 0001 0000 66ff 0432 1189 5348 0000 so the iptables rule, you want to remove all the spaces between bytes, and drop the results into a rule like this:

iptables -I INPUT -p udp -m string --hex-string "|0000000000066ff0432118953480000|" --algo bm --dport 53 -j DROP

BINGO Running this rule will immediately shut out all matching queries, preventing your server from wasting time and bandwidth trying to answer them.

The next steps were basic

Establish throttling:

iptables -A INPUT -p udp -m hashlimit --hashlimit-srcmask 24 --hashlimit-mode srcip --hashlimit-upto 120/m --hashlimit-burst 30 --hashlimit-name DNSTHROTTLE --dport 53 -j ACCEPT
iptables -A INPUT -p udp -m udp --dport 53 -j DROP
iptables -A INPUT -p udp -m string --hex-string "|00000000000103697363036f726700|" --algo bm --to 65535 --dport 53 -j DROP
iptables -A INPUT -p udp -m string --hex-string "|0000000000010472697065036e6574|" --algo bm --to 65535 --dport 53 -j DROP 
iptables -I INPUT -p udp -m string --hex-string "|0000000000066ff0432118953480000|" --algo bm --dport 53 -j DROP
iptables -A INPUT -p udp --dport 53 -m string --from 50 --algo bm --hex-string '|0000FF0001|' -m recent --set --name dnsanyquery
iptables -A INPUT -p udp --dport 53 -m string --from 50 --algo bm --hex-string '|0000FF0001|' -m recent --name dnsanyquery --rcheck --seconds 60 --hitcount 4 -j DROP

Disclaimer: *You should always apply to INPUT before FORWARD or OUTPUT

Floods BTFOd

Specific series updated: Infrastructure Series -- Taking DNS One Step Further - Full DNS Server infrastructure - #2 by PhaseLockedLoop

1 Like

Yeah, if fail2ban is not an option, the last resort is --limit rate and --limit-burst in iptables. But as you mentioned, it’s a little CPU demanding (like any QoS for that matter).