Sysadmin Mega Thread

Was talking to my cousin yesterday and he does big government adjacent security contracts and he said no one is using yubikeys because of cost. If they need hardware tokens due to gov compliance they buy the same smart cards the gov uses. Otherwise they don’t bother with hardware tokens.

TLDR: I don’t think Yubikey gives much if any discount and the cost of implementing hardware tokens on top of that makes in a non-starter for many business.


I commend you for your efforts. Whatever they are paying you, it is not enough. You better get positive job references for life!

Also, everything thing that you just went through shows how to not do things properly. As you move up in the world remember these as anecdotes when someone tells you to do something dumb in place of convenience.

Bravo Zulu my dude. Bravo Zulu.

1 Like

Oop first breach and it was CEOs email account

After today’s security incident, I got a feeling that I’m going to get a blank check for yubikeys to enroll the C-Suite into Google Advanced Protection programme


Make sure to order some spare yubikeys


Yeah, you just got a blank check. migrate the decimal point a couple spots to the right when you fill it out.

Call this one “foreseen consequences”


Today I took down most of our entire finance backend for like 5 minutes.

Turns out that the previous IT designed most things to stop working if his workspace account was disabled, but doing a quick ownership transfer to a different workspace account managed to get things back up running.

As I needed to do some deployment tasks, which required some files only IT had, I also took ownership of the whole GDrive. Turns out that most of the installers, documents and ISOs were removed, and all that was left were empty folders.

The more I’m within this company, the more it feels like the previous IT guy was either so malicious that he rigged up things to sabotage the new sysadmin, or he was THAT incompetent.


Its a Hanlon’s Razor kind of thing…

Its likely just laziness. You got things working and you should have taken steps to keep it working but its fine right now so we’ll deal with it later when an actual problem comes….


Thought of you when I read this @thunderysteak




This hurts.


I spent the last 4 days troubleshooting why I can’t get a blasted port on a switch to light up.

I tested cables, I tried different equipment, and I spent who knows how long digging through docs…

I was plugged into the wrong port.


I once spent about a day and a half trying to figure out why I couldn’t get my InfiniBand to link up. Different Mellanox drivers, different QSFP transceiver…

Turns out it helps if you actually plug the other end of the fiber into the switch on the other side of the house.


PSA to always start at layer 1 folks!


I also did that just the other day while moving stuff around in the rack and didn’t notice I hadn’t fully seated my LAN cable in the modem.

Hey, why don’t I have an IP now?

scramble scramble

visible confusion

checks cable



I once plugged a usb cable into the ethernet port of a laptop and kept dozens of people waiting for an hour while I tried to figure it out. Once I did, I swapped the cable and claimed the other 2 I had tried were broken…

Think of the OSI model as be “how tricky the problem is to identify” but in a defcon sort of way where the lower numbers are more tricky.

Layer 1: goddamn it, I spent 4 days looking at configs.
Layer 8: we’ve designed everything and it’s waiting to be implemented, but it’s waiting on CTO sign-off.


Wait what?

Lol. I legit thought its a new real layer.

1 Like

It’s as real of a layer as Project Manager is a job.

blows smoke off barrel
holsters revolver


It’s the biggest layer I’ve found out as I’m in month 4 of trying to get a system online that would have taken a few hours to actually deploy. But hey, layer 8 pays the most too…