Sysadmin Mega Thread

Adubs · July 25, 2019, 3:28pm

Anyone got ideas for how to do this?

Tool for Monitoring Linux Servers Logs and Access? Linux

The title is a bit generic, but ok. At work, i’m in charge of managing all of our customers Linux Servers. I’ve grown into this position over time and a lot of those Servers where not configured by me. Since attacks are ramping up and exploits get tested for ever faster (sometimes minutes after discovery) and a lot of those servers are, by design, accessible from the internet, i really want to step up my game in terms of keeping those things safe. Now, i don’t want to talk about security best practices here. I’m working to retrofit a lot of stuff on existing servers and have an extensive list of stuff i do on new servers to keep risk to a minimum. But this only goes so far. I’d really like to put a system in place, that monitors my Servers for certain security relevant stuff. I already maintain a Nagios System to watch for Resource usage and Services. This works as intended and makes sure the servers are up and healthy. A coworker installed Nagioslog Server a few months back, but the trial is expired and my company isn’t sure yet, wether to pay for the license. What i’d like to have is a tool that Reports any and all login attempts to a server (ssh, ftp etc.), maybe reports on installed versions of packages that are relevant (apache, nginx, php etc.) and maybe even alerts when there is a newer Version in place. Additionally i’d like to be able to log/monitor access to certain sub-sites. Say, the server has wordpress installed. In that case i’d like to see the amount of logins to server.com/wp-admin. What would you set up to manage all this stuff? I’m looking around for various options and am willing to test a lot of those. I have enough resources available to test a lot of stuff. I’d prefer something that offers at least a free tier or long’ish trial for testing purpose. Open-Source would be nice, but i’m not to picky if it gets the job done nicely. Anything you can recommend, or have personal experience with?

nx2l · July 25, 2019, 4:22pm

I dont.

do you plan on having multiple users on your proxmox?

oO.o · July 25, 2019, 4:32pm

Not really, but my usual way of working is to give a machine a local admin user with a random name and crazy password for initial config and recovery and then plug it into samba AD and just use my domain login for ease/continuity/whatever.

But yeah, also allows me to grant access to someone else on the domain instead of giving them the local admin user credentials.

Can’t do this all of the time of course. Network devices typically just have local admin users and FreeNAS only allows root to login to the web GUI (which I find pretty annoying tbh).

nx2l · July 25, 2019, 4:34pm

i usually default to doing local accounts that can authenticate via kerberos to AD/LDAP.

That way I can still control stuff easily (for me)

oO.o · July 25, 2019, 4:39pm

That sounds similar.

nx2l · July 25, 2019, 4:39pm

anyone know what this has to do with redash?

Eden · July 25, 2019, 4:41pm

Nothing directly?

nx2l · July 25, 2019, 4:44pm

someone at my work recently asked me to build a centos or rhel system so they could setup mcafee SEIM … they setup docker and docker complied from/with redash

Token · July 25, 2019, 6:06pm

Paying for any licenses already such as SolarWinds?

I drank the Splunk kool-aid, so I highly recommend it if a) environment small and slow enough that the free license suffices b) if the company can swing it, get a license.

Then there is ELK- its free but requires more administration to setup IMO.

oO.o · July 25, 2019, 8:16pm

Has there ever been an appliance based on OpenBSD? Seems like a obvious choice for a gateway, but maybe the support cycle is too short to be practical.

oO.o · July 25, 2019, 9:22pm

So I’m playing with a tape drive I eventually need to put into production. I wrote a few gigs to it with tar and then issued an erase command… apparently erase means overwrite the entire tape with no way of cancelling it (other than rebooting which I can’t do atm).

Damn it. Hopefully it’ll be done tomorrow.

SgtAwesomesauce · July 26, 2019, 1:29am

Remember that tapes can only be written to a few times. An erase command counts as a write.

oO.o · July 26, 2019, 2:41am

I have a “practice tape”.

In other news, I’m running some cable tonight. Client doesn’t like to see messy cables or cable trays or J hooks… So this is what I ended up with:

thro · July 26, 2019, 2:43am

Happy SYSADMIN DAY

https://sysadminday.com/

Goalkeeper · July 26, 2019, 2:45am

thro · July 26, 2019, 2:47am

I’ve repaired an AIX box before with cat (basically cat /etc/fstab, then cat >/etc/fstab to fix an error).

A muppet had overnight edited the fstab file, then the box failed to reboot properly, and there was no editor available.

Of course, said application support vendor did it

on friday night
with no heads up
from remote
and no phone support on the weekend to call

Was my first exposure to AIX Was working on a 24*7 mine site operation as on site IT… we needed that box working to process warehouse stock, raise POs, etc.

oO.o · July 26, 2019, 3:31am

thro · July 26, 2019, 4:02am

In other news, i’m about to embark on policy-based routing adventures.

Just migrated our Steelhead cache setup at hQ from WCCP to PBR so i can do PBR on the core switch for multi-homing traffic routing decisions (as cisco do not support the use of WCCP and PBR on the same device).

Network really becoming fairly complex (well for my standards) now. In HQ alone - 2 OSPF areas, BGP, PBR, caching, IPSEC tunnels, L2TP over IPSEC, 3 WAN links, etc…

thro · July 26, 2019, 4:19am

Go go linux multi-monitor support. Check that clock

Goalkeeper · July 26, 2019, 4:21am

Lol. The “4” is missing on the left monitor in the number “14”