Return to Level1Techs.com

Help with failing ssh connection to Debian Server

As stable as all our Linux servers are, when problems come up they certainly are something…

I have a Debian Webserver that’s been chugging along nicely. Yesterday, problems started to appear. First our Nagios Server wasn’t able to monitor it anymore. The NRPE couldn’t find it’s plugins allthough they worked locally.

Now, i’m unable to connect through SSH from two machines (on internal, one external). I don’t care for NRPE, but ssh is a problem. I’m able to work on the VM through Console.

The Problem is rather non-descriptive. My Client just tells me:

Connection to 10.101.0.100 closed by remote host.
Connection to 10.101.0.100 closed.

On the Server, in auth.log, i’m getting this:

So far, i’ve looked at hosts.allow and hosts.deny, both are empty. Allowing sshd for all doesn’t change anything.
sshd is configured to not allow root Login, accept Public Keys (which i’m using) and i temporarily enabled Password Auth to try it, without success. There is no fail2ban, no firewall (firewall is sophos hardware and didn’t change) on the server, no selinux (debian machine) etc.

I’m completely lost. All i could google up where some Problems with openssl and openssh a few months back, but those should be fixed and i have newer versions than where those fixes where added.

Additional Information:
OS: Debian bullseye (don’t ask me why this thing is running testing. No clue. Maybe downgrade to stable?)
OpenSSL Version: 1.1.1d
OpenSSH Version: 7.9p1

Feel free to ask for any further information. I’m willing to test anything to get this working. I can snapshot that thing and try stuff out. The website is still up and working, so there’s no influence on produciton so far.
Worst case, i can restore from 2 days ago, but i’m not sure if this will help long term. I’d rather find the Problem and fix it.

Edit 1:
SInce i’m working on this live, i’ll add information as it comes up.
I’m now trying to downgrade to stable through pinning to that release.
Not sure if this applies to me, but here is an issue with openssl 1.1.1d that seems to be related. No clue if debian backported a fix or if this is specific to certain openssh versions. I think downgrading should roll this back to an earlier version, so we’ll see.
Makes me wonder what happens, when Debian bullseye becomes stable and if this Problem would persist.

Alright, we’re back to a working system. What a sh*t show…
Downgrade to stable went semi smooth. mysql-server decided to uninstall itself. Other than that, we now have a stable, working, up-to-date system with ssh access.
I have no idea if the OpenSSL bug is related or not and if it’s currently in debian testing. All i know is: DON’T RUN TESTING ON PRODUCTION SERVERS. No idea who freakin configured this.
A total of 4 hours of work to fix this mess, but the website had minimal downtime, so only semi-bad. I’ll just leave it at that a won’t start a witch hunt in our company to find out who is responsible for that mess…

3 Likes

Did you try ssh -v to get more info from the client? Add more vs to get more spam :smile:

Glad you resolved it though :slight_smile: