Sysadmin Mega Thread

I always forget Jenkins exists, although I see it more like a CI/CD thing, but if it can run shell and ssh, it should be able to serve as an Ansible GUI, but I’d give this one a try last. I have not used it myself, I was just maintaining one, java devs were using it.

Bit OT, but learned about a new algorithm I’d never heard of last night - quickselect.

Hey :slight_smile:
I’m having some issue under debian buster with crontab in /var/spool/cron/crontab being a simlink pointing to another folder. (It point to the right application crontab file based on a blue green flip)

anyone better at C can tell me why this don’t work ? https://salsa.debian.org/debian/cron/-/blob/master/database.c#L213

1 Like

Is this running as a user or as the default cron service account?

if ((crontab_fd = open(tabname, O_RDONLY, 0)) < OK) {
		/* crontab not accessible?
		 */
		log_it(fname, getpid(), "CAN'T OPEN", tabname);
		goto next_crontab;
	}

Looks like it is attempting to open up the file descriptor. If it cannot open the file, then it bugs out and logs it.

It may be because you are trying to open a file that points to another file. You may have to treat symlinks differently or follow it to its inode first and then open the file!? It has been a long time since I have dealt with symlink and C code.

it’s a user crontab, in the log when i restart the cron service, i’m getting cron[pid]: (user1) CAN'T OPEN (crontabs/user1)

I made sure that the file the simlink point too have the correct permission (user1:crontab) so my guess was that it couldn’t get the FD from the symlink yea …

But the weird thing is that it look to be a debian specific issue …

That very well could be the case.
what is the output of file /var/spool/cron/crontab?

/var/spool/cron/crontabs: sticky, directory

That is the issue. You will have to deal with the symlink specifically in your code. I don’t remember how to do this.

The alternative is to make this a hard link instead of a soft link.

yea … sadly cron only allow a single hardlink for security reason
and i probably won’t be the one pushing a PR to debian repo XD

What do you see as an issue in that result ? the sticky ?

Sticky. Yeah.

Remember, everything is a file. So if you are trying to access a file that points to another file, you have to handle that logic.

i’ve try to put the cronfile in a similar kind of folder structure, /tmp/ without success.

/tmp/: sticky, directory

i’ll look into how sticky bit can affect symlink and open() function

1 Like

Good luck. I wish I could be of more help but I have not touched this stuff in over a decade. I am simply a GNU/Linux user now instead of a tinkerer.

1 Like

Anyone have a link to the firmware for a Cisco Nexus 6296UP? The Cisco downloads page is missing the product, at least from where I’m sitting.

One of the security requirements I’m being asked to implement is “SSH 2FA” at work, and it needs to be done in such a way where the engineers won’t run at me with pitchforks.

We use FreeIPA which only allows TOTP 2FA authentication with every new single SSH connection, and I’m pretty sure that this would drive every single engineer into insanity.

The idea that I’ve gotten is to create a script that a user launches at the start of the day, authenticates with LDAP credentials and 2FA TOTP and uploads a freshly generated keypair with set 12 hours long key lifetime that people will then use for key only based SSH auth (and LDAP for sudo)

What I’m planning on doing is following this but modify it a little bit

  1. User has FreeIPA LDAP password with an 2FA TOTP
  2. Script at launch removes pre-existing key if it exists already
  3. Script generates a new SSH key with 12 hour lifetime
  4. Script uploads new key to FreeIPA
  5. Script tests SSH connection to server and if successful exits with message that new key is renewed

Is this a good way of handling SSH 2FA? I’m completely out of ideas

1 Like

Duo SSH MFA had a similar issue, and I solved that with a custom pam module to whitelist IPs for 24 hours after a successful authentication. You could do a similar thing here, though custom PAM modules require some careful testing.

Is the SSH 2FA for systems that are externally connectable, or internally connectable? If external putting it behind a VPN will “solve” it.

IIRC there are also ways to delegate temporary SSH access using Hashicorp Vault and SSH Certificates, but I’ve never needed to go down that road. The idea is you run a special command and you’re given an SSH Certificate for temporary access. (If you haven’t heard of them, SSH Certificates are special keypairs that are signed with an expiry date. The server just needs to trust the CA’s public keypair for them to work). Then I’d put the MFA on the Vault access.

1 Like

The SSH 2FA is supposed to be for internally connectable servers. Its one of the insurance requirements that even internally facing server access or infrastructure admin stuff requires 2FA, so I have to be doing dumb stuff like 2FAing the TrueNAS servers

Yes I wanted to use SSH Certificates at first, but FreeIPA doesn’t support them, and Hashicorp Vault is still in the “research” phase where I’m learning how to deploy it and secure it in a lab, before reaching the integration stage. My deadline is basically end of November.

As we already use FreeIPA for stuff I’d prefer not to tackle on additional systems unless they’re really required.

2 Likes

SSH authenticating with a password-protected key is technically 2FA, all by itself.

SSH can also require both a public key AND prompt for a password. If the engineers will be opening a number of simultaneous SSH sessions to a server, they can use SSH Multiplexing to only need to authenticate once.

I know, this doesn’t solve your problem.

Unfortunately it’s not, the matrix hack from 3 years ago is a good example of what can go wrong - Post-mortem and remediations for Apr 11 security incident | Matrix.org. Though it also shows how dangerous ssh-agent can be, especially with default SSH Forwarding and no agent confirmation.

1 Like

I’ve actually never used SSH with FreeIPA. Does LDAP store the private keys or just the public?

If there’s merely a public key association with the account, you could use a yubi, but not if it’s holding the private keys also…

Yeah, I’ve seen a couple talks about time-based certificates and jump boxes being used at competent enterprises like Netflix.

I think it was this:

1 Like

It stores only the public keys, but the issue is budgetary as we can’t buy yubikeys for everyone.

I’d prefer something better to be implemented, but this works so far and has been tentatively given green light due of time constrains.

also omg I just by total accident solved most of our 2FA integration woes
Enabling 2FA with password and TOTP token in FreeIPA will propagate to anything using LDAP to use pwd+totp

2 Likes