Setting up 2FA on SSH on CentOS Stream 9 works incorrectly

Software & Operating Systems Linux
1

I’m spinning up a new CentOS Stream 9 on a cloud VPS. It will be used host a bunch of stuff (nextcloud, VPN server, etc)
The OS is installed through the VPS provider’s image. CIS Server Level 1 security remediation is applied on first boot.
SELinux is enforcing, firewalld is running. EPEL and EPEL Next Release are installed, system is fully updated. SSH keys are installed and I can log in correctly with no problems.
My next step is to configure SSH to use 2FA (TOTP).

CentOS Stream 9, unlike its previous versions or the Fedora 34 that it’s based on, does NOT include the awesome google-authenticator package. So I’m using the oathtool & pam_oath packages to set up TOTP.

Things that work:

  • I can login with my ssh keys if I don’t use the oath PAM.
  • The vps and my mobile are producing the correctly synced TOTP codes
  • If I define the oath pam in pam.d/sshd, I do get asked the TOTP code

Things that DON’T work:

  • The login process ALWAYS asks for a password with the TOTP. Using the -v option on ssh shows that there’s no communication with the server before asking for the password, ie the server expects both the TOTP code AND the password to authenticate
  • The server accepts any 6-digit code, even incorrect ones, and will happily authenticate if the password is correct.
  • Making pam_oath.so “required” in the /etc/pam.d/sshd makes login impossible.

I have my sshd configs in a seperate config file inside sshd_config.d but running “sshd -T” shows that I have the necessary bits setup correctly as far as sshd_config is concerned:

ChallengeResponseAuthentication yes
PasswordAuthentication no
UsePAM yes
AuthenticationMethods publickey,keyboard-interactive

I’m guessing I’m messing up with my /etc/pam.d/sshd configuration but I can’t figure out what
This is how it looks currently:

# PAM-1.0
auth       sufficient   pam_oath.so usersfile=/etc/users.oath window=30 digits=6
auth       substack     password-auth
auth       include      postlogin
account    required     pam_sepermit.so
account    required     pam_nologin.so
account    include      password-auth
password   include      password-auth
# pam_selinux.so close should be the first session rule
session    required     pam_selinux.so close
session    required     pam_loginuid.so
# pam_selinux.so open should only be followed by sessions to be executed in the user context
session    required     pam_selinux.so open env_params
session    required     pam_namespace.so
session    optional     pam_keyinit.so force revoke
session    optional     pam_motd.so
session    include      password-auth
session    include      postlogin
# auth       sufficient   pam_oath.so usersfile=/etc/users.oath window=30 digits=6

As it currently stands, the login process asks for OTP first, password second. If I comment out the first line and remove the comment on the last time, it asks for password first, OTP second.

Furthermore, if I make the line “auth required pam_oath.so…” It keeps asking for OTP + password over and over and fails after 3 tries. According to logs authentication fails with no explanation as to why it failed.

Questions:

  1. How do I make it so that it only asks for OTP and not the password?
  2. How do I make it so that it only accepts the correct OTP and not just and 6 digit number
  3. Optional question: One of the users has an SSH key with yubikey, is there any way to configure it so that it asks for TOTP for users with regular keys and no OTP for the user with yubikey protected SSH key?

Thanks

PS: I’m not well-versed in Linux, I arrived at this skill level by headbutting problems so explanation on how things work would be especially welcome.

2

I’m not that familiar with Centos, but try commenting out/removing the line above.

I’m not sure about the random 6-digit number, but it might be related to the password above.

3

Also- if you are trying to log in as the root user, you need
PermitRootLogin yes in your sshd_config.d dir

If those don’t work, try - I can’t remember if it is order sensitive
AuthenticationMethods keyboard-interactive,publickey

/etc/ssh/sshd_config.d/totp.conf

AuthenticationMethods keyboard-interactive,publickey

#PermitRootLogin prohibit-password
PermitRootLogin yes

# To disable tunneled clear text passwords, change to no here!
PasswordAuthentication no
#PermitEmptyPasswords no

# Change to yes to enable challenge-response passwords (beware issues with
# some PAM modules and threads)
ChallengeResponseAuthentication yes
4

This doesn’t work. After getting the OTP, it tries to continue authentication with keyboard-interactive, can’t find anything and instantly closes connection.

That’s bad practice and against the policy settings.

Yes it’s order sensitive

A bit of more digging around and I found out that SELinux is getting in the way. Even though I don’t like putting in SELinux exceptions without understanding the implications, i went ahead and applied the suggested exceptions but I still can’t get it to work. SELinux prevents access to a file being created at runtime during authentication ("/run/pam_oath.lock") and adding in exceptions to a file that doesn’t exist doesn’t work. I’ll dig around SELinux a bit more.

1 Like
5

UPDATE:

/etc/pam.d/sshd updated as follows:

auth [success=done new_authtok_reqd=done default=die] pam_oath.so usersfile=/etc/liboath/users.oath window=30 digits=6

I’ve also set

semanage permissive -a sshd_t

So with SELinux not getting in the way, 2FA works correctly. SSH key is accepted, then the user is asked for the OTP and only if the correct OTP is provided is the login complete.

UPDATE 2:

Setting SELinux to permissive then setting it back to enforcing removed the complaint about not getting read access to lnk_lock file but this time it complained about remove_name access on the directory pam_oath.lock. Putting in a custom policy solved the issue, now 2FA works correctly.

1 Like