Sysadmin Mega Thread

You got C-Suite buy in. Everyone else can eff off unless they are going to help you. I would recommend still dusting off that resume and making sure to update this. They are still making a killing off of you so make sure that they pay for some of the certifications for the stuff that you are doing.

Sounds like a lot of work for one person to be honest but if you have no obligations that need you home at a certain time and you are being compensated accordingly, you are picking up some valuable skills and doing the lord’s work there.

9 Likes

Regarding the part of too much work for one person, that’s what I discussed before I was even hired as I was concerned about that. I originally declined the job offer because of that, but they scheduled a meeting with me to ensure that if the workload is too much, I’ll get a second person within the six months. With the promise of getting a second person I accepted.

5 Likes

Has anyone administered or worked with Odoo? It looks very extensive but maybe to the point of being unfocused. Still it’s nice to see an open source project provide business services like payroll and crm.

The money likes it…

I need help building a really simple RPM, which take a single tarball that is a compressed directory, call it receiver-0.0.1.tar.gz, and extract it into /opt/receiver-0.0.1/

That’s it. However, the docs really don’t explain this because it’s either a single file, or a more complicated task.

Please help me figure out how to write the spec file for this, or any guidance on creating an rpm for a single directory.

cotton

FPM looks promising.

Your spec file should basically extract the tar ball to the build directory, copy it to the install directory, and mark all the files in the install dir to be included in the RPM.

Once you have a spec file place your source file (a .tar.gz) file in rpmbuild/SOURCES. Then use rpmbuild to create an rpm - “rpmbuild -bb file.spec”. (First b - build from spec file, second b - build the binary package).

Assuming you’ve got the rest of the boiler-plate bits done already, something close to this should work, though it’s late and I’m just spit-balling, haven’t tested.

Name:           receiver
Version:        0.0.1

Source1:        %{name}-%{version}.tar.gz
BuildArch:      noarch

%prep

%build

%install
install -m 0755 -p %{SOURCE1} -D %{buildroot}/opt/%{name}-%{version}/

%files
%defattr(-,root,root,-)
%dir /opt/%{name}-%{version}
/opt/%{name}-%{version}/*

Does anyone using the IaC model use LaTeX or some kind of templating tool (erb, jinja, etc) to automate the upkeep of their documentation as their codebase changes?

1 Like

I don’t think anyone uses LaTeX for that. Ansible has the documentation built into the roles/collections and it’s all markdown/reStructuredText. It’s not really automated though.

The disaster saga continues and I have some updates, mainly with good news

  • Got a budget of 50k for AVs, Veeam backups and 2x TrueNAS R20 systems (Thanks @SgtAwesomesauce for the suggestion btw)
  • About to close the purchase on the first TrueNAS system, you would not believe how much PAPERWORK there is if you’re an European company buying something from America
  • Began deploying Sophos Endpoint Protection onto employee’s laptops remotely from the licenses I unearthed that were not used at all
  • Audited our systems in terms of access, disabled around 50 user accounts of employees that left for the last two to three years
  • Previous sysadmin used all UK vendors, so I turned my family visit trip to a business trip out of pocket to have in person meetings with new potential vendors based in EU (I just mainly wanted to not interact with family). Also ton of paperwork dear lord.
  • Previous sysadmin disabled Google’s MDM for employee’s phones. Why the fuck? Trying to get it enabled now.
  • Began the work of getting CMDB in terms of asset management implemented. The SNOW guy was smart enough to bring in contractors.
  • Not even two months in and already being thrown to do a certification for SNOW Sysadmin for free
  • Pushing for Zabbix monitoring, CTO gave double thumbs up for it “looking nice” and dumping alerts into chat, networking guy gave me thumbs up for it supporting LDAP login
  • Went to the datacenter last week with the CTO. Mentioned that we need to kill some of these ancient boxes and he just began to pull power cables for them. “If someone needed those, they’ll scream”, and nobody has to yet even peep about those boxes being gone
  • All 3 racks are full of ambers, spent 2 hours just trying to break into out of band management for a single server with the forbidden Dell knowledge and no login to the server or being able to shut it down. Also establishing OOB management now in works.
  • HASHICORP VAULT! C-Suite gave me thumbs up on local hosted vault as long as it works at 2 in the morning
  • Unearthed some stuff that was out of scope for the audit, found more security holes than swiss cheese
  • Engineers in meetings get visibly nervous when I raise hand or join their meetings because I’m the company’s security enfoncer. Pls don’t be scared engineers JUST LOCK DOWN THAT UNPROTECTED ELK STACK!
  • Managers in standup amazed every single time with the amount of work I did the previous day and the engineers are praising me like the second coming of christ because I’m giving them stuff like local storage in their geographic office, server maintenance, ability to use Google SSO or even a UPS for their server rack in the office

There’s even more, but I’m shitposting this on a company time and I got a meeting in a minute so oops

EDIT:

I just had engineers threatening to quit the company because I want to implement 2FA onto VPNs and critical systems.
What.

EDIT:

After I alarmed the C-Suite about this, they told me that they were joking. Their joke got me into trouble. Smh.

9 Likes

Yikes. Nice catch.

That seems actionable…

Good, give it another two months and you’ll start receiving bribes.

Ok, you know where the door is.

They probably were genuinely upset about 2FA. They just decided to throw you under the bus because at this point, the writing’s on the wall that you’re the big dick in the locker room and anyone not with the security program is going to get asked to leave.


This has been a joy to read. I’m very happy to see the impressive amount of progress you’ve made in 2 months or so.

7 Likes

NGL if I knew what company he was working for I would avoid them like the plague, not recommend their services, and boycott anything until their provided a clean bill of health from a reputable pen-tester.

If a company is that far in the gutter in terms of security they deserve all manners of hellfire that befall them.

3 Likes

This - if it’s anything like my company (whom I do like, though), you’ve got a bunch of old guys that don’t like what they perceive as just a new IT person telling them what to do and the “Oh haha they were joking around” was to just shrug it off. :person_shrugging:

They never had to before, so why should the new guy be allowed to change that? That’s the attitude I see around here a lot.

Or maybe I’m completely wrong lol

4 Likes

To be fair, TOTP is really annoying and easily broken.

U2F or whatever it’s being replaced by is way more secure and much more convenient.

  1. Type username
  2. Type password
  3. press button on security fob
  4. log in

Hell, I use my key to sign my git commits along with logging into most services at this point.

4 Likes

The joke is in poor taste and those people should not get a pass like this. Those kinds of jokes are reserved for only the closest friends and inappropriate for work acquaintances.

The management ought to slowly replace them…

2 Likes

Those dinosaurs need to hurry up and retire or die and move over and let the people who know what they’re doing get shit done already.

Bit of a hot take I admit, however, they’re what led to the current state of things which is inherently unacceptable.

It is a wonder that some companies can even function. Like 95% of the work is done by 5% of the people.

I commend OP for the herculean task they have before them.

2 Likes

Soooo, something I forgot to mention that happened on Friday.
I saw the draft report of the security audit aaaand…

office

90% of things on the network is labeled as critical or high severity, and most things were found just running unauthenticated nessus scan on the network

Guess I gotta throw deployment of Security Onion into the box alongside Zabbix and Hashicorp Vault.

For context, he wanted Google SSO for the Gitlab instance, but I told him that because we use 2FA with Workspace, he needs to enable mandatory 2FA for regular accounts per ISO 27001 requirements, and that VPNs will require 2FA too.

On Friday, I had a follow-up meeting with my manager, and he basically told me to stop scaring the engineers lol. Need to run any drastic changes by him now (Why was I allowed to do drastic changes without telling him from day one tho???).

TOTP instead of security keys is unfortunately a financial decision. I keep pushing for Yubikeys to be rolled out, but I already unfortunately proposed a budget for the next six months that was already approved, so I can’t scrape 5k+ out of nowhere.

Okay you will laugh but I looked up what happened to the second IT guy in the other geographic office and he retired. Then two months later the main sysadmin left the company to be a software dev.

Anyway, I hope that with the audit report in the hand, I can finally get the push and support to get stuff fixed

3 Likes

Hey, at least you can make a nice big dent into that list without any real effort!

Call it… unforseen consequences.

4 Likes

Yubikeys get really expensive really quick

2 Likes

there’s got to be a volume discount.

1 Like