Sshd on port 22 spamming your logs? here's a solution

Networking Networking Software
1

Here’s my notes from doing something basic, (for the gazilionth time, again :frowning: ) …

Specifically, I ran journalctl to check on something on my home router which is just a linux debian system, and was annoyed because my router logs were getting filled up with failed ssh attempts - so, here’s what I did to fix that.

step 1. make an ipset

I ran this:

# ipset create hash:ip recently_ok timeout 86400

… to serve as a 24h long allowlist of IP addresses that can ssh without any rate limits, and then I then added this to my ~/.ssh/rc file which gets executed on successful logins:

#!/bin/bash

ipset add -exist recently_ok $(echo "${SSH_CLIENT:-127.0.0.1}" | cut -d ' ' -f1)

… which adds or refreshes an entry in that IP set (and only works because this is actually my router, and I always just ssh in as root, so at that point I can totally modify ipsets).

I then ran netfilter-persistent save and then I logged in a few times and looked at output of ipset list recently_ok to make sure this is working ok.

step 2. firewall rules

in my /etc/iptables/rules.v4 files, I added the following.

-A INPUT -i wan -p tcp -m tcp --dport 22 -m set --match-set recently_ok src -j ACCEPT
-A INPUT -i wan -p tcp -m tcp --dport 22 -m state --state NEW -m recent --set --name ssh --mask 255.255.255.255 --rsource
-A INPUT -i wan -p tcp -m tcp --dport 22 -m state --state NEW -m recent --update --seconds 600 --hitcount 4 --name ssh --mask 255.255.255.255 --rsource -j DROP
-A INPUT -i wan -p tcp -m tcp --dport 22 -j ACCEPT

I then did netfilter-persistent load to apply those.

This rate limits failed ssh attempts to 4 (or is it 3) per 10 minutes, and I could see it working from iptables -Lnv and from cat /proc/net/xt_recent/ssh | sort -nk3

downsides and upsides of this approach:

  • needs iptables-extensions installed for “recent” and “ipset”
  • needs way of storing/restoring firewall on reboot (e.g. netfilter-persistent)
  • very low dependency - doesn’t really require strange software suites from wherever
  • fairly quick to set up
  • ipv4 only (my isp won’t give me ipv6 without losing ipv4, so I’m IPv4 only on the internet)
  • ~/ssh/rc file runs as root (might not be an issue for purpose built server machines)
  • ~/ssh/rc is openssh-server specific / e.g. it won’t work with tailscalessh, but might not matter for tailscale
  • kind of hard to figure out you’ve locked yourself out and debug, because you just get nothing back on 22
  • still get some spammy logs
  • lacks any kind of integration with central databases of bad ips across the internet (could be added)
  • lacks any kind of long term reporting / analysis
  • failures aren’t triggering any reverse probes or scans.
  • not a honeypot like cowrie - you have no idea what’s trying to log in
  • you’re not really wasting any attackers resources like with tarpit / endlessh

other options I see people on the internet doing:

  • moving ports to something obscure and random
  • tarpit
  • endlessh
  • fail2ban
  • rate limiting only (which kind of works with ControlMaster feature of sshd)
  • port knocking

… and many many others. @paulwratt has SS:FragWhare which, despite its name, is actually a lot more advanced than my 6 lines of code I’ve put up in this post.

6 Likes
2

My brain hurts today…

Why do you have such unsuccessful attempts in the first place? Personally, I do not expose anything to the world, let alone sshd. :confused:
If anything, only for a specific IP and everything else DROP all.

If I had to play such actions every now and then I would go crazy. :wink:

Can’t you just set your IP only access on the WAN side?
Or set up a tunnel to some server and connect there. Or have the server connect the sessions to you instead of you to the server?

You can block ssh traffic completely and write a script that will check your email account and receive messages and get a new IP address then create a new ACCEPT rule for your current IP address.
Or the same only via the web server and php + sh script.
The php script would call the sh script and give the new IP value and in the script itself usually the iptables command. Of course, all https and login / pass.

I am doing something similar not for fw but for initializing udp packets in Lan after the appropriate cloud action. A total mess, it’s a shame to even mention it. :wink:

Firefox uses a plugin and watches the corresponding page where JS generates live values that can change randomly. When it detects a change, a simple GET is made to the web server and a specific php script is called. The task of the php script is to run the sh script for the local user which generates a specific syntax and the udp packet is sent to the specified address in the lan. Where another machine listens for udp and reacts accordingly when the correct packets arrive and runs the correct macro. I think it would be okay to do something similar for FW and add a new IP so you could connect. :confused:

Alternatively, as a newer version of F2B you could try https://www.crowdsec.net/

3

That’s fine, but some people have to do so for a desired workflow so this is not a thread about should you but how to do so securely.

We often are :crazy_face:

With a large org this is pointless. You would typically also require a vpn and some other backend auth system like FreeIPA.

This is an awful idea. If your domain is ever hijacked you’ve just created a scenario when you can lock yourself out. So long as you have physical access you would be able to recover though. This is a worst case scenario obviously and I’m sure 99% of time its a solution that works well enough for the average person.

I really like what OP has posted, thanks for sharing.

1 Like
4

More about how not to clutter your logs. :slight_smile:

You have to be truly crazy to have the courage to consider absolutely any solution purely theoretically. :slight_smile:

I am fully aware of this. :wink:

Perhaps I had the wrong direction of the narrative… "on my home router which is just a linux debian system, and was annoyed because my router logs were getting filled up with failed ssh attempts" It confused me a bit. Perhaps you have knowledge of whether an OP uses work-to-home connectivity. I didn’t know that. :slight_smile:

You could, of course, argue on the principle…

kind of hard to figure out **you’ve locked yourself out** and debug, because you just get nothing back on 22

Don’t let the domain be stolen, and if someone is able to steal your domain, maybe he will be able to get to your sshd. :slight_smile:
Or, don’t use a domain, just IP per dedicated machine. :slight_smile:

Nobody tries to sell such a solution as a golden mean to tackle problems. :slight_smile: It’s just the loose thought of a madman. :slight_smile:

I mean my post gave the feeling that I did not like the post from the OP? :slight_smile:

5

Nah, all good buddy.

We just have quite a mix here of homelabbers and enterprise sysadmins and I was getting the vibe this was more targeted at the former, however, the information is useful and not too dissimilar to what some of us might do on smaller corporate networks.

Especially for edge security.

6

I personally stopped exposing ports to the Internet long ago, these days if I need connectivity between two systems it goes over a VPN tunnel, like IPSEC or Wireguard.

If you absolutely must (I didn’t see this previously mentioned) expose SSH to the Internet, I recommend using keys only and disable password authentication.

7

I don’t actually need it accessible from the world, I just need something accessible from outside for myself, if not sshd, … what do I use instead, and how do I make the thing doing the checking fail-open when things aren’t quite right, or how do I make it reliable ? :slight_smile: :upside_down_face:

The ssh being accessible from wan, at the moment and in this case, is just me not being “brave enough” to turn it off to make the access internal only - it’s failsafe that’d allow me to get into the home network and fix things when something isn’t quite right for some reason.

Maybe I just pull the plug one day on having anything exposed services and leave only Tailscale, or some other thing.

I’ve done that, and the box auto-updates so it typically runs whatever is latest patched ssh available (apparently ssh is up to 9 now: OpenSSH_9.0p1 Debian-1+b1, OpenSSL 3.0.4 21 Jun 2022 … I should catch up on release notes)

8

I’d highly recommend installing and configuring fail2ban. Not just for SSH, but for EVERYTHING. My logs quieted down immensely once I configured fail2ban to check Apache logs and block IPs that send repeated requests to 404 (non-existant) resources. SSH logs quiet right down, too.

Using a VPN doesn’t solve the problem, it just moves it around a bit. You’re still exposing a service to the public internet. You just have to pick the most secure service to expose, and put in any reasonable defenses you can. Of course there is some benefit to obscurity (VPNs are much less popular services than SSH and HTTP, so less bot traffic).

9

I get my fair share of random wordpress urls /wp-login.php and similar as well, but those just get drowned in the noise from legit requests, and mostly don’t seem to come back after after a 308 https location redirect – probably less than ssh.

10

When I did the L1 Linode VPS thing, some Linode (I think) guide said set up ufw and fail2ban. The logs total about 20 to 25 MB, and are compressed, rotated, and cleaned up automatically, painlessly.

11

Well… probably no matter what we choose, the point of the matter will still be the same… something is exposed to WAN. :slight_smile:

I’m not saying sshd is bad here, because it’s not… statistically looking at it. Only the philosophy of the concept turned me a bit.

More than once I have thought about issuing access to something and not doing it at the same time. :slight_smile:
I had a small network project where a firewall stood centrally and at the same time everything was to be closed from the WAN side, but LAN services required access for selected people from the WAN side. Of course, this can be done in many ways. But none of them really suit me fully. :wink:

So the thought is… Create a daemon on the LAN side that will regularly check the selected server and email account. When a message appears in your account from a specific address for a specific address, start processing the message.
From the content, take the items out of the pgp / pgp box. Then use the defined key for decryption. Then get the IP address. Then add it to iptables.

Crazyeeeeeeeeee! :slight_smile: I know! :slight_smile:

And what would the end result of this be? So that everything is blocked all the time. And when an authorized user sends an email from the correct address to the correct address and the content is properly encrypted, he will be able to add a new IP address from which he has to get to the LAN through the firewall. :slight_smile:

The effect would be that we are completely cut off from the WAN all the time and we leave absolutely nothing. It is from the LAN side that the control is initiated on the basis of an independent source, which would be an email.

Of course, this solution can also be done in many ways without the use of email. :slight_smile:

And I’m probably inventing the wheel… probably something created by someone already exists on this principle. :slight_smile:

The solution with encrypted email also provides an additional security feature in case the external command mechanism is compromised. :slight_smile:

Before someone shouts “UDP hole punching” or revproxy or Tailscale. It is all very dependent on certain patterns that can be a negative vector.

The email, however, is in this case a solid medium so independent of any anomalies that may occur in the case of the above. Instead of email, you can also go towards the web and the cloud.

I already do something like that, although it is not entirely adequate to your situation and would require a lot of adaptation and improvement of safety.

The mini version could theoretically be very simple and do not pass a specific IP, only discover sshd on the WAN on request only for a few moments so that you could establish a connection and add your IP and then block all.

But at the end of the day, everything based on the third factors will have some weak vectors.

12

So basically a bulletin board … implemented as an email inbox, … and used as a sort of command’n’control.

email’s a bit cringy to me for this use case, I’d prefer either very simple http, or a maybe even mqtt. e.g. There’s a small utility used by alpine linux for notifying mirrors that it’s time to rsync new packages, it’s this one ncopa/mqtt-exec.

Ideally, like email, something would have to host it, and it would have to be secured somehow,… theoretically one can secure anything just by making some certs and wrapping the comms into openssl s_client .. .

13

Lots of possibilities and solutions. :slight_smile:

I thought about email because it creates some semblance of reliability, security and some standardization and, above all, it is an ecosystem off the shelf without the need to use less common solutions.

When I mean email, I mean gmail accounts… absolutely not hosting your own mail server. Of course, the email can be replaced with something else, but in my opinion it is a more independent and more stable solution than a typical http solution. Web-based or targeted communication solutions have some disadvantages. They need to be hosted and made available to the world in a form that is very specific and this can create an attack vector.

Even the best hosting can theoretically be less stable / secure than just gmail.

The attacker would need to know your two addresses and obtain the key. But the main motive here is that we are hiding in the shadows. It is a similar concept to what intelligence agencies did / do in communicating orders to agents in the field. I’m talking about number stations… which just broadcast a normal radio signal but the message is encrypted with a one-time pad, from the counterintelligence side (the attacker) is not able to detect the spy’s position. Because the method of sending the commands is by no means end-specific and anyone anywhere can be a recipient of this message.

A similar principle would be a solution that implements email itself. The attacker would need clairvoyance to locate Command and Control.
The reverse situation is in the case of a botnet where some try to hide the CC and others find it.

This can also be compared to “dead drop” where firewall and CC handle the command exchange.

How can an attacker come across CC? If we assume that it is not a deep leak on our resources. The attacker is unable to knowingly or accidentally target a host that could turn out to be a CC resource. Because it does not exist, in simplified terms. :slight_smile:
Yes, the penetration of the google ecosystem or the penetration of our private resources creates a risk vector, but…

My point is that there is no central permanent CC that an attacker can touch even by accident.

The attacker will not have knowledge of the two dedicated email addresses. It will not have gpg keys. There is also a low probability that it will be able to penetrate the ecosystem belonging to, for example, google and catch our dead drop from the masses.
Our CC simply does not exist from the point of view of the attack vector. And if something doesn’t exist, it’s hard to get there and do the wrong things.

It’s just a thought :slight_smile:
I would call my book “Crazy thoughts - firewall and penetration vectors” by madman :slight_smile:

14

@risk thanks for the mention. I really appreciate your solution in the OP. A lot of default Linux installations will have the required software to assist and extend this approach. Anything above that requires an investment in time, analysis and research to get more out of it (incl. SS:FragWhare) if you want it to be bleeding edge up-to-date. That “more advanced” you mention comes from me constantly observing and adding checks for newly found attempts.

Having written the rest of this post first, to assist with the reason for the OP, I would suggest running a modified version of the SS:FragWhare standalone Gerka, that simply removes the related sshd log entry as the IP address is blocked (if you are interested, I’ll add it to the script #'d out). However this could be problematic in some instances (editing a file thats constantly getting written to).

BTW there is a monthly updated IPv4 blocklist archive available in the repo.

It seems your OP stirred up a “Quake vs Unreal map construction” type response. Engine
Rendering-wise, Quake maps are built more like a Lego set, one piece at a time, until you get what you want (hence sometime you can “see between the pieces” - you get air gaps), where-as Unreal maps are built more like Michealangelos’ David, you take a block and sculpt away pieces, until you have what you want.

If you are talking about automated and “script kiddie” attacks, maybe … Your opening response was brutal dude, so brutal that someone else felt the need to step in and say something (at length) defending the OP. You need to be aware of that in the future …

I often want (and therefore setup) something similar to the OP. Moving off default ports can cause issues (yes, often only corner cases, but still, I often find myself pushed into them for whatever reason). Mostly, I dont have a permanant IP address, or even IP range, that I could use in an “allow list” . Note that email is not less vulnerable than any other port, and maybe more-so because its plain text as opposed to ssl tunnel.

That said, aside from the OP solution (elegant in my opinion), and taking into account the discussion on Command and Control (I did not read any of the other posts - appologies), I would add my own personel favourite technique to the equation, “keyless entry”.

Applied in this situation, I would hit a webserver url that “turns on a service”. If that service does not get used for X period of time it automatically turns off. Thats the port security taken care of, the side effect meaning “no service, no log entries” (the reason for the OP).

There is no reason that service could not be an “email Command and Control” service. I would still prefer sshd, but it depends on requirements, preferences, and corner case uses, and why you need access in the first place.

Use of a VPN and an outside server (only logins are accepted from this IP address), that way you can block by default, might also be useful, but there can be reasons why this is not practical or usable. That said if you use Linode, you can move the default service and still get in on the default port from the command line … ssh -t [email protected] my-pretty-server. Then from there you can SSH into your home box, even without a VPN (this is kind of how TailScale works), you can apply “block all by default” rules to you home server (which may or may not reduce log entries).

Personally, I believe anything on a Home Network should be behind a VPN’d front server that collects the garbage that is “daily internet traffic”. But again this can have unforseen consequences too (what happens if you cant get to that server).

15

9 posts were merged into an existing topic: /dev/null

19

Indeed. Even just setting sshd to listen on port 2022 or 2222 instead of 22 cuts down the randos trying to get into your system by several orders of magnitude. It’s not a security system by any means, but it’s comical just how many automated scanners only ever try the default ports.

Really makes you wonder how many outdated applications full of vulnerabilities remain unexploited simply due to automated scanners not hitting anything besides 80 and 443.

1 Like
23

@TimHolus @paulwratt … I appreciate both of your honesty, and I kind of see both your points on the meta-discussion and appreciate both your inputs regardless of the styles/tones and I like your interest in styles/tones, but I’m kind of more curious about the purely technical discussion here.

If you could help out the mods / edit some of the posts, that’d be awesome.

Re ports 22/222/2222/2022/22222 … I don’t know if I mentioned it, but I have lots of internet facing physical boxes as well as VMs running various distros used for different purposes.

2222 was my favorite, and it used to work great until like 2-3 years ago, … but I think some scanners are catching on. I have 222, and that’s fine for now.

Re IP blocks. Most of the port 22 badness I’ve seen comes from various compromised cloud VMs, including sadly Linode.

Do either of you know of a reliable way to automatically report abuse?

24

Thread cleaned up. Y’all behave.

Have a nice day.

1 Like
25

remember people, ther is no one right way to do things with computers, only “a way to do it” and “a better way to do it” (and there is always a better way) - no-one aspires to do it “a worse way”, unless maybe you are a government employee with “highly knowledgable superiors”.

(guess thats sorted then - and the initial responder was a mod too - shame on you - ziiiiiiiip)

using other ports does not really work, I’ve checked the various (common) scriptable “probe” options, and they can spam ports for known protocols, not just spam known protocols on known ports. some dont even wait for a response (yay lets ping 10 million different IP adresses per second)

not automatically. Linode has controls in place to catch and disconnect “compromised” servers. What constitutes a compromised server is up for debate, but it includes, and is not limited to:

  • “super” high volume outgoings (external, vertical - same IP, different ports)
  • “super” high volume outgoings (internal - DMZ, inside hosting zone)
  • spam volume outgoings (internal - DMZ, inside hosting zone)
  • spam volume outgoings (external, horizontal - same ports, different IP)

I dont know it that makes any sense to anyone (without getting into elaborate details), but thats (part of) the gist of it.

They dont shut down the server, they lock it off (iscolate it internally), send a warning saying “if this continues we are going to shut it down”. They are pretty quick about it too.

On the SS:FragWhare proto server (on Linode Texas) I do get hit from inside the DMZ for that zone from time to time, and I could have manually blocked a few more, but when I checked their source they were gone already. Last month I undid a couple of blocked IPv4 from a few other zones they have. Yes its on the rise, but all-in-all, Linode have a good handle on things (not so much anyone else)

For anyone interested Automated Threat Response was how SS:FragWhare came about, and there is at least 2 posts specifically about this question, one being why it was not implimented.

EDIT: tried to clarify those network details above, and made IP/sec accurate (10 not 14)

1 Like
26

A note on different ports:

If you dont have a service on a port, you wont get a log for that port. That is not the same as “the port is not being attacked”. Your server can still be DDOS’d even if you have nothing processing that port or protocol , it will just have a higher thresh-hold before success, ie 100’s K fails per second needed ( unprocessed fails soak up network bandwidth ), instead of 1 K per second successful fails ( processed fails soak up processor bandwidth).

Also just becuase you have a specified service on a particular port , does not mean the attack source is using the same port protocol to probe or attack said port (and if it isn’t the same, it wont appear in any log either).

2 Likes