[Devember 2021] brutally simple firewall

(this is not specific to SSFW:FragWhare, so I thought I’d post it for those who might find it useful)

added a utility script ldom.sh to make it easier to automate backups:

#!/bin/sh
# ldom.sh - created with 'mkshcmd'

if [ "$1" = "--help" -o "$1" = "-h" -o "$1" = "" ]; then
  echo "Last Day of Month"
  echo " outputs the last numerical day of the month"
  echo "usage: ldom.sh [now|_date_]"
  echo "options:"
  echo "  now           as of the current month"
  echo "  _date_        either '2022-02' or '2022-02-25'"
  exit 0
fi

D="$1"
if [ "$D" = "now" ]; then
  D="$(date -Iseconds | cut -d \T -f 1)"
fi

X="$(echo $D | cut -d \- -f 2)"
if [ "$X" = "" ]; then
  echo "error: not a valid date '$D'"
  exit
fi
if [ "$X" = "$D" ]; then
  echo "error: not a valid date '$D'"
  exit
fi
if [ $X -lt 1 ]; then
  echo "error: not a valid month '$D'"
  exit
fi
if [ $X -gt 12 ]; then
  echo "error: not a valid month '$D'"
  exit
fi
if [ $(echo $D | cut -d \- -f 1 | wc -c) -ne 5 ]; then
  echo "error: not a full year '$D'"
  exit
fi

isLeapYear () {
  if [ $(($1 % 4)) -eq 0 -a ! $(($1 % 100)) -eq 0 ]; then
    return 0
  elif [ $(($1 % 400)) -eq 0 ]; then
    return 0
  fi
  return 1
}

lastDayOfMonth () {
  Y=$(echo $1 | cut -d \- -f 1)
  M=$(echo $1 | cut -d \- -f 2)
  M=$(($M + 0))
  case $M in
    1)  DoM=31 ;;       # January
    3)  DoM=31 ;;       # March
    5)  DoM=31 ;;       # May
    7)  DoM=31 ;;       # July
    8)  DoM=31 ;;       # August
    10) DoM=31 ;;       # October
    12) DoM=31 ;;       # December
    2)  DoM=28          # February
        isLeapYear "$Y" && DoM=29
	;;
    *)  DoM=30 ;;       # April, June, September, November
  esac
  echo "$DoM"
}

lastDayOfMonth "$D"

exit 0

Besides leap-year checks, it does century checks as well


FYI: at some point (years ago, while working on another project) I was creating so many .sh scripts per day, that I found it easier to create a mkshcmd command to touch, edit (via e link) AND chmod the new command, plus supply generic “shell command” content (the 1st if block) , generated into ~/bin/..., automatically for me, all in one go (ie after editing the command is usable).

Cheers

Paul

8980 blocked : 0 in last hour : 34 in last 24 hours

With almost 9000 blocked IPv4 addresses, the “redo” script that runs at 58min past the hour to ensure all “processed IPv4 blocks” are actually present in ip route show, now takes minutes to finish, AND it is only ever going to get exponentially longer, because TWO lists of 9000 are cross-referenced.

I just made a tweak to stop grep -m1 after the 1st match.

I could cull the initial output reference (which goes to file) so that the IPv4 is the only thing present per line, AND do a “pre-lookup” on the 1st number match (eg grep -nm1 ^192.), AND even “sub-cull” those out to a seperate input file for grep. But this all adds to processing time, even if it reduces it overall.

A better way would be to not cross-reference at all, and rather just “add the IPv4 block” again, redirecting errors from ip. ATM it will generate at least 9000 error text per run, if I were to do it this way, but at least it would be infinitely faster .

The only good thing about this script is that its using an “inline commandline grep match”, which means that even with 9K plus lines of input, its still only taking up “one line” in bytes of memory (which will never get over 24 bytes per line match) on top of what grep and the current shell uses.

In the longer term, it would probably be simpler and faster to cross-reference blocked IP address if they were present in the sysfs (/sys/) somewhere. But that in itself may or maynot present its own drawbacks (RAM?, Kernel Time?).

Oh the Joys of Firewall Fluff

Its not needed for the current (and still only sshd) Gerka, just the hourly Web Server Log scanner, because that’s run as the webserver user and ip (by default) only works for the root user (which is how the Gerka’s also run).

updated blocked lists:

9862 blocked : 0 in last hour : 12 in last 24 hours

Made a couple of tweaks over the last week, mostly more grep “.” related. But I did expand the webserver log “crawler”, which is an hourly check for what URL’s have been captured. And I did add the sshd kex check before the hourly IPv4 check (that everything logged was actually instanciated/blocked in IP).

The kex blocks really only need to be run once per log file just before rotation, but I have seen an increase in them, and it was anoying knowing I would miss any if I missed a log rotation (which is about every 10 days atm, but can be every 3 hours when getting hammered).

Did a couple of range blocks on MS Azure 40.* and an over-range block on an Afrinic 156.224.0.0/11 that was being used by a few different (world) locations to perform sshd attempts.

Its a bit un-nerving when the Gerka process stops creating (sshd) blocks, but its only been just over 12hrs, and those ranges are not part of any Linode network, so I guess they are doing there job (for now), but the webserver log “crawler” is still picking things up.

Could be that Linode have added some upstream protection of some sort, but I doubt that would be the case without an announcement.

A couple of weeks ago I sent 2 “abuse” emails, both happened to still have active servers at the end point at the time of writing. The automated responses were “rotton” to say the least. One was “GoDaddy” where you need to “sign-up” to post an “abuse report” and both provided no way to report the type of abuse I was seeing, their best assumption being “abusive content”.

I know that some places are serious about abuse, often I dont even have to make a report as the IPv4 has been “de-listed” (it goes no where), and the server no longer present (ping, tracert, or DNS).

Might get a change at Internet Storm Center too, they had listed some IPv4 that are used as “master infection servers” (as opposed to being source originators in log ouput), but the “comment” contains “- none -”. They have the same info I have, and although these particular addresses change based on who is doing what, they do hang around for extended period, and they never get used to do probes, only as “bounce” or “response” to a specific URL request, and mostly for certain types of routers that have not been patched yet.

It would just be nice (for most people) to know what such a low ranked security rating (2) was getting captured for (they are normally Mozi.a, Mozi.m and jaws “GET” servers that will connect the device to a BotNet or similar command-and-control network).

9933 blocked : 0 in last hour : 10 in last 24 hours

sshd attacks have dropped right off, 2 blocked every 3 days, but the caputered web urls (haxor-access.log) are producing 10 a day (which makes sense, there are more checks now). And no kex entries either.

Overall, I would say (besides what I have blocked already) sshd attacks worldwide have dropped off, as I can’t see ALL of them being routed through Azure. ISC showed a slowdown too, when I was there last (which I thought was odd at the time).

It maybe that “higher up the router foodchain” (internationally) more blocks have been put in place, but I dont believe that either, as alot of the “one off” sshd attacks were from random Asian sources.

Who knows, maybe its just the “quiet before the storm” …

10023 blocked : 0 in last hour : 11 in last 24 hours

well the end of last month saw a severe slowdown in the number of sshd blocks produced by the Gerka, this month (12 days so far) see no sshd blocks by the Gerka. So I checked the logs and sure enough, no sshd entries for root or invalid user, but still some kex (key exchange) failures.

However what I do see are the odd banner exchange: invalid format entries. These are new to me, but I think they are backdoor overflow attemps (thats a guestimate, based solely on the fact they fail, and there are no user attempts).

Again, I am puzzled by this lack of attack presence, especially considering the uptick in webserver blocks. I cant believe I have managed to fully block all sshd attackers . Remembering that those 10k blocked IPv4 dont include the 700+ ranges. Then again, maybe those nearly 800 ranges is enough to cover most compromised botnet devices and virtual machine services (but I highly doubt that).

EDIT: hmm… those banner entries match the pid (process ID) of the kex failures - banner line contains invalid characters. However the blocked kex IPv4 dont equate to those associated messages …

10138 blocked : 0 in last hour : 10 in last 24 hours

1 sshd attempt before log rotation 2 days ago, and 1 today, so the Gerka is definitely still working. A couple of kex blocks as well.

Still, a wierd month with more regular kex blocks than sshd password attempts, and those dont include the malformed banner either.

POST /scripts/WPnBr.dll HTTP/1.1

Thats something new in the captured haxor log. Guess they are looking for windows based webservers? Shame the logs dont capture the POST data, but then again it could be megabytes long, so no point in worrying about it, just nice to know it still captures new stuff properly.

A note here, basically the only reason I can capture these is because I dont have POST setup on the downstream server, so it always returns a 405 error. I guess I could add a .dll check to the URL checks for when I do finally have POST set up. Not sure how well that would go down on a WSL server, would definitely have to test before making further presumtions.

10449 blocked : 0 in last hour : 8 in last 24 hours

Well its been a while (46 days since the last post) but the sshd attacks are finally back in full swing, since 11am UTC on the 9th on June. They were not gone, the were more than a few kex entries blocked in that time, and that 1 sshd attempt, might have been close to 60 days of near silence in total.

EDIT: just checked the end of the block list and its all quiet again by 6am UTC on the 10th June.

Nothing else to report atm, except that the web server log generated blocks are “raging a storm” . The power situation here is fixed but the weather has caused issues, so no further work done yet …

1 Like

10803 blocked : 7 in last hour : 69 in last 24 hours

still racking up the web sever blocks, but since the 27 July sshd attacks have ramped up again (from 0-ish), with the last week of logs being 90% sshd based blocks. Over the past month there has been an increase in attacks from Serverion, a Netherlands based server farm operator that also hosts non-european IP address (eg Russia), and has a connection direct connection to OOO SibirInvest in several countries. It almost looked like someone was targeting their servers for staging, as there were only one or two attempts recorded before another address from a different location (but same operator).

The over-range blocks (besides various mentioned above) are mostly new DigitalOcean ranges and new Microsoft Azure/Cloud assigned server addresses (ie more from the usual).

The only other real news was that the ISC (Intenet Storm Center) stopped being functional a couple of weeks ago (maybe the weekend of the 4th July), which I only noticed after trying to contact staff with a possible new Mozi.m style infection server url, spreading something called aqua.mpsl (with multiple architecture binaries found at the url), very similar in size and attack type (presumption) to the jaws vectored z0r0 binaries found at the url supplied in those web attacks.

EDIT: I did spend a couple of days putting together a page that makes it easier (visually) to decide if a block-range or over-range block is nessecary, as it references everything in the log file (which gets 0’d each month), not just one type of block, which is what I have been using for xref up until now. This allowed me to examine possible control panel integration with other more commonly used web hosted applications or suites (eg pihole). (ED - this is working out really well, highly useful, even if the process is still manual)

EDIT 2:
I did find find a bunch of 0 length files floating in various places on the file system, but research showed it was (probably) due to the Apline Linux Web Administration being started by default (I turned it off by default after I found a couple of entries). Its protected by password, but it appears that a valid user is not checked until after the file is created (and hence no content is recorded). I need to look into this with a new Alpine Server release as this app is quite complex in how it hides actions from the web user. (Knowing me, I just found another bug, but we will see).

1 Like

so its been about a week since the last post, around the beginning of the month, which is when the logs are scrubbed (copied to archive folder, then emptied or deleted - depending), and even though things where starting to change before then (sudden uptick in sshd attacks), it has become even clearer that something else is going on, as the webserver blocks are way, way down, about a 3rd of what they normally are, while the sshd blocks are up about 300% (if 100 blocks a week were regularly performed).

So whats going on here, is there any way to “devine” what is happening? Well maybe… but its not due to anything related directly to whats in the log files, its more about whats not there, and what we have learned from the last 12 months (yes I did mention starting the project before #devember2021 ).

Most of web server attacks are either:

  1. from an infected source (part of a command and control network ), or
  2. command line tools (aka script kiddie attacks )

As it happens, a certain percentage of (normal?) sshd attacks are of a similar groupings (where script kiddie attacks try multiple usernames in rapid succession), along with the regular “security probe” (web) services (note that alot of this data ends up in the wrong hands for monitary gain).

Without taking into account the possibility of the hosting service applying some form of screening, service blocking based on analyrics, or IP region blocks, it appears there was a re-organisation of co-opted “command and control” networks.

Basically, whoever was running their own C+C either lost control of their network, possibly sold their networks, or were going through an organisational change, either way “they are back in business”.

On the webserver side, it is obvious when a “script kiddie” is trying to get in - 80 different url paths from the same IP address. Webserver attacks from compromised units or C+C networks are not like that, they know the urls they are trying to access are device specific so there is at most only one probe (possibly 2) from the same IP address.

Although I say above that webserver attacks are down to a 3rd there regular rate, 20 different IP addresses for a 7 day period is “quite normal”. Those are the sorts of numbers (20-25 per week) I was seeing around the time after I started applying range-blocks.

As mentioned in a couple of previous posts, there looked to be some sort of organisational use of hosting services, particularly rotating them, which may indicate that over the previous 2 months they were using those services again, and since their IP range is blocked, I dont see anything in the log files of the SSFW server.

As a side note, this leads to a problem covered in the initial couple of posts on both this project page, and the original post thread - how to detect “threat level” at the interface, after blocks have been applied, so that “threat response” can be lifted, or made permanant (ie dont check it again).

As you can see there are no real answers here, we are “guestimating” based on previous data, and research of that data, and that is the point at which some form of high level (or AI) analysis-over-time application would come in handy. Without a bigger data set, like that collected by the Internet Storm Centre (ISC), a visualization tool like those used for repository commits might be more practical in the interim.

It is worth noting the a similar “down time” appeared over the Christmas - New Year period. School holidays have just finished in certain parts of the world, but that only accounts for a 2 week period (over the last month), not 2 months.

Well, I guess you might classify this as more of a “gut feeling” assessment, rather than a thoroughly factual based assessment, and some might even say thats just a “knee jerk reaction”.

But it is clear there was a shift about 2 months ago, and there has been another shift again.

EDIT: with the research required for IPv4 range-blocks, there are some details that come through over time, that portray a certain scene .

  1. certain attacks from South American countries are performed by the same person.
  2. certain attacks from Southeast Asia are performed by Chinese nationals.
  3. certain attacks from Ukraine, Crimea, Estonia are performed by Russian supporters.
  4. Dutch / Holland / Netherlands is still the highest individual attack source vector.
  5. England and USA are right up there with China and Russia as “threat antaganiosts”.

EDIT2:
I suggest any tool more than visualization, needs to assess not only a larger data set for any given time period, but also need to xref geolocation news story corelations, as it may show why there is an associated shift, when that occurs.

EDIT3:
after checking 50 IP addresses of the 300 newly blocked sshd spam, it appears that the majority of them are from “dial-up” service providers all around the world (200 services in 150 countries) , and those that are not are “hosting” service in Russia or Hong Kong, and to a lesser degree Korea, Sweden, Netherland, UK & US (yes there are a few universities in there), as well as multiple from Isreal, Iceland & Iran.

Bumped with Brutality Bonus updated in the OP

get your “MWHahaha” shirt ready, and be prepared to asume the position chanting “I’m not worthy” if you are on the end of one of those entries :wink:

Gees, this would have been so much more effective if everyone was still on dial-up, but then again no one bothered to waste there valuable bandwidth hacking via web urls
:slight_smile: