Sysadmin Mega Thread

Operating Systems & Open Source
4123

Thanks for all the recommendations

There’s no CIO or CISO or an acting one unfortunately. Turns out that there used to be an IT person for the other site, and he left recently as well so I’m all alone for this

Here’s the update so far from today

I scheduled a meeting with the company CEO to explain everything is terrible, what steps are needed for security and that a budget will be needed for it.

I also pulled a few senior management engineers into meetings to get more information and that information definitely took like 10 years worth of my lifespan in an hour. Previous Sysadmin was pushing back for getting backups because it was more work and there’s no suitable hardware for me to use for backups that are worth around 55TB of data.

I’m going to be working with extremely limited budget for a backup server, so I’m grabbing a NetApp disk shelf from a decommissioned NetApp server and a refurbished HPE ProLiant DL20 Gen10 for under 800€ to put TrueNAS onto it.

Anyone got any recommendations for what RAID/HBA card to go for? The E208e-p that is supposed to be officially compatible with it is 1200€, which is out of budget completely if I want to buy drives as well. Alternatively rackmount server with raid/hba combo that doesn’t cost an arm and a leg.

4124

They can’t spring for a SAN or a real NAS?

I would rather trust my data to a refurbished NetApp solution than to build one myself if they have “NO BACKUPS AT ALL”. Otherwise, the backup solution is another thing that you are going to have to manage.
Hell something that takes LTO6 tapes is going to cost ~800 - 1200 USD on eBay and the tapes are about 35USD a piece. That can be paired up with any server, and you can run that sucker with straight up unix commands if they do not want to buy backup software.

écoute, mon ami, I would highly recommend you start looking for employment else where. This does not seem like a trash island fire. This looks like scorched earth policy from the top down.

2 Likes
4125

ahh, yes. This is why alcohol.

hahaha, if you want on-site backups, I’d look at truenas. The X10 is around $10k for a 100TB system IIRC. It’ll do all the good network protocols, and it’s zfs.

If you’ve got a good pipe and you want offsite backups, I would look at something like backblaze or s3.

Just saw this. Try to budget for new hardware. Don’t buy anything until that meeting with the CEO is done. If you can get your hands on a truenas direct from IXSystems, you’ll get access to their support, which is invaluable. It means you have access to some of the smartest engineers on the planet.

Okay, get the CEO on board. Be his friend. This is a drama, and you’re playing the role of the hero who gently breaks him the bad news and comes to the rescue.

He needs to understand that it’s not going to be a seamless or fast transition to “in good standing” but it’ll be valuable.

If he’s totally on board and gives you a big budget, make a list of projects that you can either outsource or hire contractors to help with. This can speed things up and lighten your load. Makes this much less dreary work.

All in all, I think you’re handling this shitshow well. Keep your head on a swivel, vent to us or friends when you need to and just remember that when you drag this company out of hell, you’ll be the hero.

Can I ask, what sort of scale this company is? 20 employees? 200? 500? 2000?

What’s the windows:linux ratio in terms of infrastructure? Are they taking advantage of cloud services or is on-prem more their deal?

And what industry?

I don’t want to dox the company, but this info can tailor our advice.

3 Likes
4126

:+1: Not a sysadmin by any definition, but this right here - things tend to be a lot easier if you can convince the right people that “this is going to be a huge problem but I’ll take care of it if you just sign off on the budget.”

The thing that’s worked the best for me every time has been to write up a proposal, budget and all, for what it would take to ‘fix’ the problem within reason, and that to take it to them at the same time you break the news with the message of “Here’s a solution, I can make this problem go away if you just approve this.” All the management I’ve had to deal with has responded positively to having a solution ready they just need to sign off.

Emphasis on “for me” because YMMV obviously. :rofl:

</thoughts>

3 Likes
4127

Yeah, I wouldn’t show up to the first meeting with a full proposal, but what I would do is start off the meeting with something to the effect of “I’m near a solution for our vulnerability, but I want to make sure we’re on the same page with direction before I put too much effort into anything.”

Then outline the high level problems. CEO doesn’t care about the details at this point, he wants to know the problems and then your solutions. The best thing you can tell him is “These are easy to solve, but will just take time and money. I’ll handle it all, you just authorize budget and changes.”

I’m assuming there isn’t a CTO here or you’d be meeting with that C suite.

This x 1000

If you can have a proposal for one problem, in writing, I would present it to the CEO at the end of the meeting. “if you could look this over and sign off on it, then I can get started”

Bonus points for writing up 2-3 proposals and present the one that’s going to get him the most in your corner based on your first meeting. :wink: The other two can be used at a later time, so it’s not wasted effort.

4 Likes
4128

Yup -

"We have no backups and could lose everything. This [sheet] would fix that.

We’re at risk of having a massive security breach. This [other sheet] would fix that too.

etc etc etc"

Hey, seems to work well for engineering requisitions at least :person_shrugging:

2 Likes
4129

Essentially, after presenting all the problems to the CEO, solve his biggest concern first.

2 Likes
4130

We got two DCs in two different countries, so the plan was to have the two sites do each other’s offsite backups as cost saving measurement. I also already proposed Backblaze to one of the main engineers that’s on board with the backups and trying to come up with a solution.

Alright, will try to aim for a brand new system if the budget allows it. Wanted to use an “open box” R640 system as the head unit with the NetApp shelf as Wendell did back in 2017. This is currently one of the “we got zero budget” options.

Right now the primary option is a proper refurbished Supermicro storage server with warranty loaded with drive caddies, as it’s less hacky and doesn’t require an external HBA

I’ve got most of engineering on board and I had a quick chat with him today where I actually managed to schedule a 30 minute meeting with him to go over my notes about the company’s IT security. I’m in the process of writing a technical action plan to give to him, as I already written up a 3 page doc with the issues and either solutions I know will work, or potential solutions. Turns out some of the solutions for half of the problems are sitting in my home lab when I was in a meeting with another management engineer.

For outsourcing, I contacted an old company I used to work for if they would be interested, and mostly what I was told was “What the fuck?” and “I don’t think we want to touch this with a 10 feet pole”

Going to be honest, this is somehow kind of fun trying to unfuck everything???

Company scale is approx 100 employees

A single lone Windows server surrounded by everything on Linux

Combination of using GSuite and SNOW in the cloud and rest on-prem

…would you believe me if I told you that the company’s industry is cybersecurity?

3 Likes
4131

How does on-prem fit into your DR plan? What happens if the lines are cut on both DCs? Who’s going to crawl through broken glass muttering vague threats about uptime?

Just something to consider.

It’s a good solution if you have zero budget, but I highly recommend trying to use the emergency nature of this to get more budget.

You can absolutely fit 55TB of backups on a single 3-4u storage server these days, with no external shelf.

Oh, it’s definitely fun. Enjoy it.

I’d wait till you have a plan and ask to bring on either a contract company or bring in a contractor for specific tasks.

Ok, then I would get your security in order as soon as you have a working backup. Everything else is secondary.

Good, this shows proof of concept.

Engineering being on board is important. I can’t tell you how great it is to have the wind of engineering in your sails when you need to make large changes.

3 Likes
4132

Looks like a job for Samba-AD and simplify your life.

Do they work federal government contracts? Sounds like they work government contracts.

2 Likes
4133

The company is able to still function at very limited capacity even with both DCs down as mail, ticketing and both internal and external communication channels are with Google. The primary colocation datacenter is literally physically sandwiched between multiple AWS DCs, so if a fiber for that gets cut, not even the cloud can save me for uptime.

For DR regarding destruction of hardware, all of the backups are supposed to be ZFS replicated across the sea. If both DCs are destroyed, time to pack-up shop because a thermonuclear war most likely just started.

Yeah well turns out that NetApp shelf is still in production with live data, so can’t use that until its migrated away.

I did the math and basically getting used hardware (R730, 55TB of drives, HBA330, 64GB RAM,…) is almost the cost of a brand new TrueNAS R20 box.

Requested callback from iXsystems to get a proper quote to submit.

I’m requesting Veeam licenses for anything critical as part of “back out plan”. I’m not touching anything until its backed up.

Also I managed to dig up an active FreeIPA server today that I have no idea what it is used for

Today (or rather yesterday because 4:20 AM here hehe weed number) one of the engineers just gave up and send me logins to most of the infrastructure because I kept asking questions.

Imagine my horror finding TWO silver Compaq/HP rackmount servers with Pentium 4 in production in this company

I shit you not it straight up looks like a rackmount form factor of the early P4 Compaq machines with the same silver front and blue power button and oldschool HP logo

image

With having to manually go over the entire infrastructure with a haircomb and dinosaur of servers like this, I’m starting to feel like a technology archeologist.

2 Likes
4134

I have been defeated by a missing F-F adapter.

FFS Cisco.

image
image1920×2560 224 KB

6 Likes
4135

Gov’t contract confirmed /s

Be careful. You find uncover something that cannot be contained.

1 Like
4136

That is the problem.

3 Likes
4137

Get yourself a USB to Console cable and save yourself the trouble. The one I got works with the majority of different brands.

3 Likes
4138

I grabbed one off Amazon, just need to wait for it to be delivered.

You have recommendations? I have a Dell switch just to see how I’d like their CLI, but I’m not opposed to trying different things.

4139

I recommend most of their competitors. I have used Juniper, Sonicwall, Fortinet, Brocade, Arista, and Microtik (only [pro-]consumer).

They all have their pluses and minuses but I hate how Cisco takes a standard, then puts their secret sauce on it and then sells it to you at a premium, all while supporting on the basics of the standard that they highjacked to decrease interoperability with other vendors. Even down to serial connectors.
/rant

Use what you have, just know that Cisco is going to piss you off if you have a mixed vendor environment.

3 Likes
4140

Good to know. Glad I have folks like you with the “been there done that” experience.

2 Likes
4141

If this happens, please call the moderators so we know to back up the forum, before skynet hits.

2 Likes
4142

Rita Repulsa: After 10,000 years, I am finally FREE!

Or worse, Zeiram.

1 Like