I am a True Level1…and it’s All Your Fault!

[Mark-D-Stroyer]

You did this to me! I stumbled across the YouTube channel several months ago - never matter exactly how many - and slowly been tumbling down the rabbit hole since, bouncing off the walls the whole way.

I am traditionally more of a power user, the go-to tech support guy for everyone who knows me, that whole story. I’m OS agnostic, had more of a hardware focus and have done a lot of device repair. Testing out the sysadmin skill set has been a real wake-up because a lot of it is outside my normal wheelhouse. I mostly used command line to launch games on DOS and to format drives, not…do everything. And I have no idea why, but I like it and can’t get enough, and am trying to do it now. I’ve been following the channel and lurking in a few different Reddits, trying to absorb as much general context and best practices, because I’ve gotten a decent amount of theoretical understanding over the years in passing…but applied is a very different story.

The Journey So Far - Baby Steps

I’ve been reclaiming some castoff, marginal e-waste computers for my multiple kids to largely play Minecraft and/or Fortnite, from quad-core 775 to the best of which has an third-gen i5 and a GTX 660! And there was a request from kids and their friends to play Minecraft together, all on incompatible platforms…unless I made a server. So we made a server, got it online and it even got played on!

Next, my computer has the gaming laptop curse of about an hour of battery life. I noticed while adding a dual-boot to my wife’s MacBook Pro that the Windows side was drastically hungrier and actually had about the same battery life. Well, one thing leads to another, which led to a Pop!_OS dual boot. I’ve always had a penchant for using FOSS software - call it DIY attitude, call it distrust of Big Tech’s ethics and competence, call it being cheap - but I’ve been using Firefox, GIMP, OpenOffice (yes, ~nobody~ told me about even Libre), etc for years anyway, and I’ve been really enjoying the heck out of it. Most of my business work is webapp or email, so there’s actually been no issues, and it turns out a legitimate tripling of useable battery life that way too. I’ve been learning some terminal stuff just getting the system set up and tweaked, but of course it’s been primarily a

And then we come to the business case. My wife and run a business that until very recently had a brick-and-mortar retail location, which we got rid of for huge overhead reasons. Well, we didn’t have time to do anything other than just pay a platform for our website, for Trello that we use for project management, for email, all that. Plus, on a certain level I’m not fully comfortable with other companies having the whole of all of our customers’ personal data on their servers with how often there are breaches, and I’d like to take control of all that. Now there’s time and all these things are expensive, and the way I see it we can use our own hardware instead.

The Setup

We have an older house with lots of add-ons, closed-up chimney, signal issues everywhere. We had an older Velop four-node mesh system to deal with that. I happened to get a deal on a Belkin RT3200 for the shop, so I set it up here with the mesh system bridged as an AP. So funny story, the Belkin has bugs with DHCP doing it that way, but it turns out that OpenWRT is crazy easy to install and solves that problem! (Actually understanding that system is also on The List.) We did manage to get a business internet account at our house, so we don’t have those personal ISP issues.

The server I’m running is as follows:
-Gigabyte G41 LGA 775 board, E5700 processor (yes really)

-Stock cooler with a broken fan and a second fan zip-tied to it (which it almost doesn’t need)

-6 GB 1333Mhz DDR3 (because the board has compatibility issues with 4GB DIMMs and I only had one that will post)

-300 GB 2.5” hard drive from a 2006 Mac Mini

-Ubuntu Server

-All in an Emachines case I picked up off the side of the road that still proudly proclaims “AMD Sempron”!

-A “run what you brung” enthusiasm

The Plan

I sat down with a self-guided Docker course and walked myself through some basics. I’m also using Portainer, which has been both very helpful and also not. I’m also slowly working through a Linux self-guided course too. This is what my diving into it has come up with, all WIP or early planning.

I’m running Tailscale, currently as secure remote access for when that comes up, also need to set up because we have family with Down’s that we help with Netflix, etc.

I do have a Minecraft server running too, I just haven’t figured out how to configure it with the same plugins and setup I had before.

As a Trello replacement, since we use it so extensively, I’m looking to use Vikunja, since it seems to be a good implementation of all the features that we use. I had started messing around with it standalone, haven’t looked at the Docker implementation yet.

For replacing our paid website, NGINX for a Hugo site.

Mailcow for Email (probably a secondary one)

I hear Pihole is just good to have?

And after investing in some non-jank hardware with actual storage:

Google/Apple Photos backup, possibly replacement.

Media server for the non-smart TVs we have, it’s mostly for streaming and not a huge library, we have some older consoles currently.

Steam Library/other gaming backup

Inventory management that actually understands production

And I have a close friend who’s also looking for some of the same backup stuff, so also cross-storage to be mutual second-locations.

Like I said, I’m currently running myself through courses after thinking about it for a long time, and I really feel like I’m drinking from the fire hose with everything. I keep having to remind myself that having imposter syndrome just means you’re learning things. I feel like I’m flailing forward and can’t stop, which is the weirdest kind of fun. Anyway, I have ~goals~ here, I have no idea what I’m really doing and I hope to have slightly more tomorrow, and the day after. I’m sorry if I ask stupid questions, because I feel like I have a lot of them. I have even more tabs open right now, if that’s any consolation.

So here’s looking forward to Level 1.0.01, and we’ll see you there!

[Mark-D-Stroyer]

I feel like it’s obligatory, but here’s what we’re at right now.

The aforementioned OpenWRT-flashed Belkin.

Emachines, E-waste, but out of the way enough for now.

One big win has been putting RetroPie on a Pi3 I had used mostly for music at the shop, printed and attached in incredibly nonstandard manner to a printed VESA wall mount.

[ucav117]

Welcome to the club!!!

@Mark-D-Stroyer

I keep having to remind myself that having imposter syndrome just means you’re learning things.

If you follow through on even half the projects you mentioned then congratulations! You are now more experienced than 50% of the sysadmins out there.

[Mark-D-Stroyer]

Thanks. Reddit makes this look like this is what people do in their sleep.

[ucav117]

Yeah… Reddit……

[MazeFrame]

Quite the ambitious journey, but, you will come out of it as a self-made IT-God.

One piece of advice: Two backups, one for daily oopsies (shadow copies, previous version, etc.), one that is the actual backup.

[Mark-D-Stroyer]

1.0.01

Today I learned how to take something broken and ruin it further!

I tried starting with the most important part: getting the Minecraft server running today, and I started by killing it. I’m used to the standard way of stopping it - editing config files - and then restarting, rinse and repeat. So based on my experience, I did the sensible thing and exported the server image, went into the layer archives, edited the configs, and imported as a new image.

This is apparently not what you do! Completely broke it. Okay, fine, we’ll do it more directly: SSH in, go to the volume with the files, nano edit, and restart. This…also is not what you do, and broke it, but this time in a way where the volume is permanently ruined regardless of image pull. So new volume, and ~this~ time, we look into doing it somewhat more properly.

As it turns out, Docker Compose makes the whole concept work way better in my head, as the, “we have a file that sets up and overwrites default values” clicks in ways that normal container startup often does it. So we got it properly executing…and it should be working but it’s not working. Like it totally says it’s running and I can do terminal commands, but actually seeing and joining the server is no dice. I even opened ports just to see! I’m probably missing some network configuration, or I hope so.

So I try the compose file approach to Tailscale too. Aaaand speaking of working but not working, I don’t know why any of the flags to set up exit node or even key login just aren’t working, so I’m missing something there.

I also use Vikunja’s guide to Compose an instance, which…fails. For no stated reason. When I edit out the nginx proxy section it does actually build and I can go to login screen, but none of the modules can actually talk properly. So yet another should be working but isn’t.

Now on the wins side, diving into the OpenWRT software I was able to set up a Guest zone, attach to an otherwise unused 2.4GHz antenna as a dedicated SSID, and we have a working Guest/Amazon Echo can go pound sand box! And of course we learned a lot about actually using Docker Compose and stack execution, which is the real progress.

The flailing will continue until morale improves!

P.S. I’m not actually putting this out there as a big “would somebody solve my problems so I don’t have to use a Google Equivalent Search Engine (G.E.S.E. (Yes, it’s DuckDuckGo))”. Obviously any wisdom bombs that you are willing drop are very much appreciated. I’m doing this halfway as a thought-externalizing/accountability thing, other half as just trying to make the thing I would want to read in the same position.

[ucav117]

Docker compose is going to be amazing for you once you get the hang of it. Make sure you understand volumes and keeping the “permanent data” separate from the other cruft.

IF you want to be real hot shit either spin up your own git (I recommend gitea or gitlab) or make an account on github. Then you can have portainer pull your docker-compose.yml from your git repo directly and manage changes that way.

[Mark-D-Stroyer]

Ooh, okay. Yeah I’ll definitely keep that in mind for the future.

[Mark-D-Stroyer]

1.0.02
Today, more Compose learning, trying to absorb some of the structure and make it make sense in my head. Some of the environment/flags stuff is starting to.

Ended up with just a little bit of time, tried spinning up Vikunja locally…and it worked! Except then I got fancy, tried to do specific versions to make some database importing stuff work, and broke all of it and now I don’t know how I made it work in the first place. So that’s fun.

For Minecraft I started updating the Windows-based server as a control test to see if it’s a network config problem, and speaking of networks I had to disable device isolation on the guest network to make the Echos play music together, but the separate router zone means the traffic is separated, so that work paid off.

I also, just for fun, started looking up Mailcow setup, and…well, I’m going to have to dive some more into domain hosting DNS settings and do my research on that first.

[Mark-D-Stroyer]

1.0.03
More messing with things for Vikunja, trying a couple of GESE’d solutions. I have made ~some~ progress based on the container logs on startup not having repeated failures, but…not there yet. Maybe I don’t understand SQL well enough.

I took a step back and ran myself through through several days of the Linux Upskill Challenge that I stumbled across in Reddit, just as a foundational basics thing, to make sure I wasn’t just…missing huge chunks. I actually blitzed through way more than I expected from already knowing it, which is a weird feeling at my stage.

[Mark-D-Stroyer]

1.1.01

The Day Job Strikes Back!

It is a dark time for the Upstart Server Alliance, for their failure to maintain the Ubuntu Server Base and inability to get its capabilities operational has led to its destruction and a hurried retreat, pursued relentlessly by the Evil Day Job Responsibilities! Now hidden in the Covert Under-Desk, the Server Alliance regroups and attempts to utilize a newly-found weapon…Prox-Mox…only time will tell if this will be enough to drive off the Day Job…for Good…?

~Pan to dust on window~

Yeah, so it got a bit busy with life there. There was some looking things up and reading during brief downtime, but it was too packed to do almost any sitting down and doing anything, and those brief moments I did were more of the same. I did discover that apparently as part of the Ubuntu Server install I had incidentally installed…Nextcloud, and it was sitting there doing nothing, and that could have partially contributed to the weird issues.

When I was doing the first Research The Mess You’re About To Get Yourself Into phase, the beginner advice was largely, “Figure things out bare metal first, then mess around with doing them in containers, don’t rush into doing everything at once.” And I figured, I had done enough of that failing up to this point, it was time to hit the reset button and try this from the top with everything we’ve learned so far.

The reset button – if you hadn’t guessed – was Proxmox. And boy, I definitely get why people like this. Built-in VNC terminals, super high-level views on everything? Let’s go, Fire Hose! We also have a Ventoy dongle on our keychain now, by the way, that is who we’ve become, so we pave everything and we have a super what-the-big-boys-do stack of:

Proxmox → VM Ubuntu Server → Docker

And…it works! Literally the things I’ve been beating my head against just aren’t a problem, Tailscale just…deployed, no problem, Portainer is back up, and we even have a Fully Armed and Operational Minecraft Server!

…not that the kids care, but y’know. That was as much Virtual work as we could fit in between other Real Life things, but for some reason, things just…aren’t being a problem the same way. A little bit of configuration but not the beating my head against the wall wondering why nothing is happening like it says it should.

Just wait: Tomorrow I’ll probably try to spin up Vikunja and it’ll all go downhill way fast!

[Mark-D-Stroyer]

1.1.02

The day job is very weekend-ramp heavy, so it’s been a lot of that and just having kids around…I still managed to get Traefik installed! Just basic running, but it’s still a good feeling to have things actually ~working~ instead of just…not. Still not sure what I did to børk the original install, but I’m just grateful I’m making progress. The flailing - now more purposeful and intentional wild gesticulation - is starting to make progress!

I did do a back-to-basics command line getting previous things set up with Docker instead of going into Portainer and Compose, just to not overcomplicate things. With Traefik, it seems like Compose actually is the better way to get it set up and run it. Still need to figure out the plugins for fail2ban etc.

I did a bit of look to spin up mailcow…and holy cow that is way heavier on resources than I thought it would be! The earnest-enthusiasm build is going to need a full platform upgrade to one that actually supports memory before we get too deep into that.

On side notes: I installed a demo mode script onto the Retropie, but I think it was tailored for the Pi4 and the Pi3 I’m using has different implementation. The Minecraft server (highly relevant) has a seed that actually did get some kid attention and now has a cluster of small houses and farms, and a couple developed mine-caves.

[Mark-D-Stroyer]

1.1.03

I’m in a weird spot where I feel like I am actually starting to understand a bit of what I’m doing with Compose. Like I’ve absorbed enough of the syntax to understand what I’m doing…but still there’s so many variable in everything going on that I’m not getting it all! So I’ve made zilch actual progress in getting Traefik in a configured enough to operate properly level, but I definitely feel like it’s more a troubleshooting thing rather than a sheer flailing.

Part of the problem is version control! I am, in fairness, trying to synthesize like three different guides that have different things - fail2ban, letsencrypt, auto-discovery - but each one of these is also using a combination of different Compose versions and config file formats! Part of me does want to do a back-to-basics and do just a text-config and command line start up, but I know that the headache now will help down the road when everything is Live and On Fire.

Since we have not gotten far enough where we’re trying to spray a chemical agent in every direction, we’ve be doing the old-fashioned thing of working on a website you pay for, but at least I’ve figured out how to do some HTML embedding that will carry over when I need it to rather than paying for higher tiers of Proprietary.

In other news today, fixing broken filament, dismantling a broken PS3 for keychains, aggressive spreadsheeting, and coming up!: Desoldering! More at eight.

[CHESSTUR]

Meanwhile, back at the outhouse, things are piling up.

[Mark-D-Stroyer]

Ohhh yeah, that’s that CPU doin’ real man’s work!

…Math!

The filthiest computer I ever saw was one I pulled out of a garbage pile off the side of the road. It was the Emachines this server system is using as a case (and power supply!) right now. That poor AMD Sempron had inches of pure dirt for cooling.

[CHESSTUR]

Git down and git durty. Probably should have posted it on the cringe thread. It’s all cleaned up now. lol How’s the desoldering coming along?

[Mark-D-Stroyer]

It’s not. That’s a problem for Future Me! ~Stands in silhouette with cape flowing.~

It’s an HDMI replacement on an Xbox, not crazy difficult but not my wheelhouse, I’m normally pretty comfortable with parts that are…separate? It’s got pushed back to morning.

[CHESSTUR]

Yeah, it’s a whole different kind of messy bizznizz. Not like taking a quick vac to it and you’re done. Flux can be tricky at the best of times.

[Mark-D-Stroyer]

1.1.04

Yeah so the HDMI on Xbox Series S replacement job kinda spiraled out of control. I took a crack, thought I was just…straight missing some fundamental soldering thing since I’ve only done a mild amount of basic stuff and it’s been a minute. I tagged in my dad who’s been doing electronic soldering since the seventies…who ran into the exact same issues so that was weirdly reassuring.

I borrowed his tools, and it turns out that even his pretty nice soldering iron couldn’t put in enough heat and the huge amount of high-temp stuff they put on the components was just a hard wall, low-temp stuff didn’t want to even ball or flow on top of it.

It turns out, the nuclear option of “just heat gun it” worked out great! Melted old off, popped the new one on, let it cool back on, and ‘Robert is a man who your family claims to be your Uncle but you’re pretty sure he was just your dad’s first college roommate’. Y’know, close enough to call it the same thing.

What else has happened? …Yeah, not much.