Return to Level1Techs.com

Phaselockedloopable- PLL's continued exploration of networking, self-hosting and decoupling from big tech

Post purpose: To blog my continued exploration, difficulty, frustrations, blood, sweat, tears, rewards and happiness in experimenting with as open hardware as I can find within reason to do most of my cloud stuff at home. Occasional discussion of my projects ran on the server or devices will be talked about. Basically JAB (Just another blog)


A little theme music:

It has long come time to be the architect of my own set of network hardware, cloud and NAS.
tenor

In any case for the longest time I wanted to move away from the growing technological cloud onto something I created. I wanted to do this across all my devices but Android without google for example is rather terrible (and lol still is). I digress lets get into a brainstorm of the plan.

Government botnet= Telework system. I am an engineer for the Department of Defense.

UPDATED NETWORK MAP.


The ryzen server (planned)

The ryzen system supports 6 drives. So when I turn it into a server my ultimate plan is the following:

6 8 tb drives - RAID6. Shucking drives from WD Easystores seems to be the cheapest route and 8 tb is the price sweet spot atm. Excluding the amazing deal on 16 TB EXOS drives recently. @wendell @SgtAwesomesauce you are the Subject Matter Experts on this. ZFS or RAID6? I am thinking RAID6 but I will be upgrading to the full 64 GB of RAM so ZFS would be doable with its advanced features.

Management software:

Reason over unraid: I dont want to pay
Reason over proxmox: My primary focus is not VM’s. This software is more well rounded imho
Reason over truenas: I do still want to do VMs and truenas is less than intuitive here.

Thanks @Novasty for letting me stress-NG LOL I mean test out cockpit on your systems. Its really nice. It will simplify my multi system management. That is the end goal its not just a single host system here!

OS of Choice: CentOS 8
https://wiki.centos.org/

System Specifications (When built)
Ryzen 7 1700X
GPU: GTX 980 Ti (left in from desktop use||Might use as RCUDA pool)
64 GB RAM
6x8TB drives (Shucking seems easiest - OK to buy CMR Enterprise drives)
1 GBE (planned upgrade to Intel 10 GBE)

Doing this is contingent on me finding the Laptop I want first. See below in laptop section.

What it will do:
It will have the following software
Nextcloud - My google drive/cloud replacement
Jellyfin- My media center provider
Collabora - Google Docs and Collaborative office replacement
Self Host gitlab - Likely for my Android projects (see below after laptop)

Links to software pages:

https://jellyfin.org/


What to do with my RPI 4 - 8GB?

Now I know you are thinking what, your diagramn says 4 GB? LOL well long story short amazon sent me the 8gb version. OOPS. It will run the following software.

The focus being to block ads and acting as the DHCP Authoritative on the network.

I have room for two other planned things. I want it to be my open smart hub. So I have looked into software for this. OpenHAB seems the most feasible.

Voice Assistant : Pycroft

https://mycroft-ai.gitbook.io/docs/using-mycroft-ai/get-mycroft/picroft

I currently have smart RGB lights at home that run over WiFi. Intregrating them into this should not be too difficult. I dont shy away from a bit of light hacking/coding

Its arrived and I assembled it with a heatsink because I do intend to overclock it to 2.00 ghz with a OV of 6


The firewall

I chose the protectli with coreboot.

Whats cool about it?

I can trust the hardware. Coreboot as described in its project is a Simple. Clean. Secure open alternative with the Intel Management Engine disabled. This appeals to me. The J3160 is a true qaud with no HT so I dont take a performance hit when hardening the OS underneath.

Firewall software of choice:
OPNSense
https://opnsense.org/

I stumbled on this as an alternative to pfsense. What I do like about their project is that OPNsense has a nicer user interface and seems to be implementing new features faster than pfSense. This is important to me. I intend to run a full IDS/IPS system on this device (called inline IDS)

Specifications:
Processor: Intel J3160
SSD: 120 GB Kingston
RAM: 8 GB
Firmware: Coreboot
Connectivity: LTE Failover module (Chosen over wifi)

I will use my RPI2 as a 2.4/5ghz airspace watcher to watch for wireless attacks


The Netgear R7800

Why?

The fastest prebuilt OpenWRT router on the market and also the most well designed. I am an RF engineer by trade. Looking at the board disassembly its very well put together.

Let’s take a look:
Manual shows the exterior

The interior and specifications are more important. This is a MU-MIMO router. Its important that the amplifiers for each antenna are independent and are not a singular to avoid mixer interference and Imod interference. A setup that has an amplifier for each antenna per frequency is the most robust wifi board setup you can have. Its also expensive.

Photo Source: FCC

The four squares next to each radio are as follows. The top radio is the - QCA9984 4x4 MU-MIMO 802.11ac radio which has one Skyworks SE2623L 2.4 GHz power amp per antenna The bottom radio has the bread and butter of what I like. The best manufactured power amplifiers for a 5Ghz router atm. The QCA9984 4x4 MU-MIMO 802.11ac radio with the RFMD RFPA5542 5 GHz PA module per antenna Now I am going to trade out the antennas for a set of professionally designed Antennas.

Here are the specifications for them:

  • VSWR: 1.7:1 (nominal); 1.95:1 (Max - DFS Spectrum)
  • Very Low Boresight Error specification - Ill get into this more if someone asks. TLDR its important for beamforming
  • A Lower S21 Q_l factor (A low loaded Q factor on a high gain antenna is hard to do and results in a balanced performance of transmitting and recieiving. Often high gain antennas are good at recieving and far more poor at transmission)
  • Low resistive loss. (83% Efficient nominal through the frequency sweep)
  • Perfect 50 Ohm Match
  • Vertically Polarized (Ground planes are important to consider)
  • 2.4 Gain 7dBi; 5Ghz Gain 9 dBi

This fixes the routers one particular weak point. Its 2 dBi antennas. Now note that due to me doing this I really have to be careful with my TX power. I am going to do some careful calculations to keep it under 30 dBm or 1 Watt. When you change antennas on the router its no longer certified like it was by the FCC. Most routers will do fine with crappy poorly matched consumer antennas but if you go professional you must absolutely be careful not to violate law. In fact @ me if you want to talk about how crappy consumer antennas are and how all this stuff works. I trully trully would enjoy the conversation.

Why OpenWRT?

I prefer an opensystem. Now according to their project page; The OpenWrt Project is a Linux operating system targeting embedded devices. Instead of trying to create a single, static firmware, OpenWrt provides a fully writable filesystem with package management. This frees you from the application selection and configuration provided by the vendor and allows you to customize the device through the use of packages to suit any application.

In my terms its the debian of router OSes if DDWRT is the overloaded Ubuntu. It will require setup out of the box but it allows me the customization I need and the packages I want with nothing extra. I can also audit its base.


The laptop.

Well its an absolutely crappy time to buy. I want an open laptop. So I was targetting this:

Coreboot is a requirement. Just as it was with my firewall. (though ill void that requirement for a ryzen 5k series mobile). The issue I have is the 3K series nvidia cards and the new AMD cards just kicked off GPU Wars 7.0. So now I am stuck in the wait and see category.

I am completely fine with a coreboot Intel + Nvidia 3K setup however my dream is a (preferably coreboot) AMD 5K series mobile + AMD RX 6800 XT (mobile). I love the new GPUs. Amazing at raster and have an open ISA which is cool for my own tinkering. I do dabble in GPU accelerated machine learning projects.

I want a GPU onboard given eGPUs on linux are still buggy. I do want to still game as I would like so having all this in a laptop has become my requirement. I can wait patiently. Now it would be even cooler if SRVIO and stuff were on the system76. Then I could virtualize on a laptop haha! (death to my battery life)


My current phone:

Google Pixel 3XL 64 GB Black Codename: Crosshatch
Network Provider - TMobile Unlimited
OS - Lineage OS (custom signed by me) (Signature baked into Titan M security chip so I may lock the bootloader for more security) (NO ROOT)
Case: Spigen Armor Case + TG Screen Protector

Do I build lineage OS? Absolutely and its not that hard check out their docs!
https://wiki.lineageos.org/devices/crosshatch/build

Why lock the bootloader and whats the Titan M?

Relevant Articles:

It has still yet to be cracked. It does not mean it wont be but given googles wonderful openness about how to sign the chip etc. It makes me wonder do I need root? No. So why void my security by having an open root system and unlocked bootloader? If someone gets to my phone they can grab the data. Its on me and if I lose it thats bad even if I dont store much data about me on it!

So I ventured down the documentation of GrapheneOS on how to harden and do my own work to LineageOS with some changes and my own key

I lightly modify the lineage OS kernel. I dont go to the Graphene extent because I do still want to use a couple google services and location.

I sign the Titan M and my TWRP recovery with it. TWRP has a password protecting it. If someone gets ahold of my phone and tries to inject their own recovery the Titan M will destroy all data on the device. :yay:

Expected Lifespan: As long as 5G is not mainstream; As long as I can logistically maintain

18 Likes

(Reserved for diagramn of Milestone 1 completition)

1 Like

(Reserved for indexing later content)

Index:

The Overview

Post 15 - ZFS doesnt require as much memory as thought

Post 21 - Why ZFS is good

Post 23 - Installed suricata/Rule Interface Screenshot

Post 25 - She is starting to look good (Themes and nearly complete setup)

Post 26 - More Important ZFS talk

Post 27 - Reverse Engineering Unavoidable

Post 29 - What cockpit looks like

Post 31 - PiHole Online!

Post 35 - Convenient SELinux

Post 41 - Pre Antenna Test Settings

Post 114 - Ultra Secure Configuation of TLS/Certbot

Post 153 - Pi-Hole DNS Setup with block rules in OPNSense

Post 173 -Link to wiki post on Wireguard Setup

Post 174 -Link to wiki post on Dual Stack IP4+IP6 setup

Post 185 - Link to wiki post on adblocking and self hosted recursive DoT server

Post 206 - Security headers! Series 6 out!

Post 218 - Ceramic Coats Car Detailing. (Not really related)

Post 317 - UPDATED NETWORK MAP.

Any services discovered in this guide are pursuant to the following Policy: https://services.utangard.net/privacyLegal.html

2 Likes

After some testing I found out. The protectli does but like crucial memory whatsoever and will not run with it. Kingston and Samsung seem okay with Samsung being hit or miss

Yay :yay: its up now to put it back together and set things up

2 Likes

I got way too many of these cables for basically free… a data center in the city south of me closed. Really nice. Up to spec. Way overkill…I have some black cat 6s near the wall as well for the short ones but need a use for these so why not. You can never have too much cable

2 Likes

Janky temporary cabinet setup to work on it

Also RPI4 makes a lot of heat

1 Like

Pretty routine setup. I am going to update some packages first. Will touch back on how I feel about OPNsense’s initial setup

In a true me fashion. The systems are named after norse gods and concepts


Domain is fitting. Going to switch to LibreSSL soon ™

1 Like

VLANs are fairly routine. setting up a firewall rule so the government computer is completely isolated

1 Like

RAIDZ-2 is raid6.

1 Like

The big internal debate is which to use. Plain Raid6 or Zed 2… and ZFS goodies

1 Like

Well, in my book, there’s literally no reason to go traditional raid.

If your controller dies, so does your array. See, raid controllers usually have a proprietary storage format and all that fun stuff. RAIDZ is just ZFS, so you can plug it into any old computer and get to work.

Small performance sacrifice, but how bad is it really?

3 Likes

That was my thought but given mdadm exists. which is more advantageous over the other?

1 Like

mdadm has speed.

ZFS has reliability and feature set.

If you have 8GB of ram that you won’t use, go ZFS every day of the week.

2 Likes

64 GB of WAM so in theory I can support up to a 48 TB array and still take full advantage of all of ZFS’ great features

1 Like

wut

The only RAM requirement for ZFS is that you have at least 4GB to support the ARC and other things it does in ram.

There is no array size dependent memory requirement. (except for dedup, which we don’t talk about)

2 Likes

Novasty was saying something about 1 GB of ram per terabyte or something. Is it some feature that requires that?

1 Like

That’s only for dedup, and even that’s questionable now that SSDs are properly supported.

IIRC CERN uses ZFS and Ceph in the multiple petabyte order to do their LHC work.

2 Likes

Well its going on spinning rust (not budging on that)… so Dedup seemed like something I wanted to have?

https://wiki.freebsd.org/ZFSTuningGuide

I read a bit into this

1 Like

Dedup is something you want only if you do WORM operations and actually plan on storing deduplicatable data.

If you need performant writes, dedup is basically like tying a 5 ton anchor to a diesel truck. It’ll go, but not quick.

3 Likes

I see. so Tell me more. This is just going to be a massive storage bank. Its going to store whats in the OP (reference it for what I am doing with the server). Is de dup or any other feature really necessary. What features increase the reliability of the array. ( I want to minimize rebuilds).

If I am not limited by the size of disks… man how big can I go… Whats a good sweet spot of CMR drives (preferably ENT drives or NAS drives)?

1 Like