[Devember 2021] brutally simple firewall

(this is not specific to SSFW:FragWhare, so I thought I’d post it for those who might find it useful)

added a utility script ldom.sh to make it easier to automate backups:

#!/bin/sh
# ldom.sh - created with 'mkshcmd'

if [ "$1" = "--help" -o "$1" = "-h" -o "$1" = "" ]; then
  echo "Last Day of Month"
  echo " outputs the last numerical day of the month"
  echo "usage: ldom.sh [now|_date_]"
  echo "options:"
  echo "  now           as of the current month"
  echo "  _date_        either '2022-02' or '2022-02-25'"
  exit 0
fi

D="$1"
if [ "$D" = "now" ]; then
  D="$(date -Iseconds | cut -d \T -f 1)"
fi

X="$(echo $D | cut -d \- -f 2)"
if [ "$X" = "" ]; then
  echo "error: not a valid date '$D'"
  exit
fi
if [ "$X" = "$D" ]; then
  echo "error: not a valid date '$D'"
  exit
fi
if [ $X -lt 1 ]; then
  echo "error: not a valid month '$D'"
  exit
fi
if [ $X -gt 12 ]; then
  echo "error: not a valid month '$D'"
  exit
fi
if [ $(echo $D | cut -d \- -f 1 | wc -c) -ne 5 ]; then
  echo "error: not a full year '$D'"
  exit
fi

isLeapYear () {
  if [ $(($1 % 4)) -eq 0 -a ! $(($1 % 100)) -eq 0 ]; then
    return 0
  elif [ $(($1 % 400)) -eq 0 ]; then
    return 0
  fi
  return 1
}

lastDayOfMonth () {
  Y=$(echo $1 | cut -d \- -f 1)
  M=$(echo $1 | cut -d \- -f 2)
  M=$(($M + 0))
  case $M in
    1)  DoM=31 ;;       # January
    3)  DoM=31 ;;       # March
    5)  DoM=31 ;;       # May
    7)  DoM=31 ;;       # July
    8)  DoM=31 ;;       # August
    10) DoM=31 ;;       # October
    12) DoM=31 ;;       # December
    2)  DoM=28          # February
        isLeapYear "$Y" && DoM=29
	;;
    *)  DoM=30 ;;       # April, June, September, November
  esac
  echo "$DoM"
}

lastDayOfMonth "$D"

exit 0

Besides leap-year checks, it does century checks as well


FYI: at some point (years ago, while working on another project) I was creating so many .sh scripts per day, that I found it easier to create a mkshcmd command to touch, edit (via e link) AND chmod the new command, plus supply generic “shell command” content (the 1st if block) , generated into ~/bin/..., automatically for me, all in one go (ie after editing the command is usable).

Cheers

Paul

8980 blocked : 0 in last hour : 34 in last 24 hours

With almost 9000 blocked IPv4 addresses, the “redo” script that runs at 58min past the hour to ensure all “processed IPv4 blocks” are actually present in ip route show, now takes minutes to finish, AND it is only ever going to get exponentially longer, because TWO lists of 9000 are cross-referenced.

I just made a tweak to stop grep -m1 after the 1st match.

I could cull the initial output reference (which goes to file) so that the IPv4 is the only thing present per line, AND do a “pre-lookup” on the 1st number match (eg grep -nm1 ^192.), AND even “sub-cull” those out to a seperate input file for grep. But this all adds to processing time, even if it reduces it overall.

A better way would be to not cross-reference at all, and rather just “add the IPv4 block” again, redirecting errors from ip. ATM it will generate at least 9000 error text per run, if I were to do it this way, but at least it would be infinitely faster .

The only good thing about this script is that its using an “inline commandline grep match”, which means that even with 9K plus lines of input, its still only taking up “one line” in bytes of memory (which will never get over 24 bytes per line match) on top of what grep and the current shell uses.

In the longer term, it would probably be simpler and faster to cross-reference blocked IP address if they were present in the sysfs (/sys/) somewhere. But that in itself may or maynot present its own drawbacks (RAM?, Kernel Time?).

Oh the Joys of Firewall Fluff

Its not needed for the current (and still only sshd) Gerka, just the hourly Web Server Log scanner, because that’s run as the webserver user and ip (by default) only works for the root user (which is how the Gerka’s also run).

updated blocked lists:

9862 blocked : 0 in last hour : 12 in last 24 hours

Made a couple of tweaks over the last week, mostly more grep “.” related. But I did expand the webserver log “crawler”, which is an hourly check for what URL’s have been captured. And I did add the sshd kex check before the hourly IPv4 check (that everything logged was actually instanciated/blocked in IP).

The kex blocks really only need to be run once per log file just before rotation, but I have seen an increase in them, and it was anoying knowing I would miss any if I missed a log rotation (which is about every 10 days atm, but can be every 3 hours when getting hammered).

Did a couple of range blocks on MS Azure 40.* and an over-range block on an Afrinic 156.224.0.0/11 that was being used by a few different (world) locations to perform sshd attempts.

Its a bit un-nerving when the Gerka process stops creating (sshd) blocks, but its only been just over 12hrs, and those ranges are not part of any Linode network, so I guess they are doing there job (for now), but the webserver log “crawler” is still picking things up.

Could be that Linode have added some upstream protection of some sort, but I doubt that would be the case without an announcement.

A couple of weeks ago I sent 2 “abuse” emails, both happened to still have active servers at the end point at the time of writing. The automated responses were “rotton” to say the least. One was “GoDaddy” where you need to “sign-up” to post an “abuse report” and both provided no way to report the type of abuse I was seeing, their best assumption being “abusive content”.

I know that some places are serious about abuse, often I dont even have to make a report as the IPv4 has been “de-listed” (it goes no where), and the server no longer present (ping, tracert, or DNS).

Might get a change at Internet Storm Center too, they had listed some IPv4 that are used as “master infection servers” (as opposed to being source originators in log ouput), but the “comment” contains “- none -”. They have the same info I have, and although these particular addresses change based on who is doing what, they do hang around for extended period, and they never get used to do probes, only as “bounce” or “response” to a specific URL request, and mostly for certain types of routers that have not been patched yet.

It would just be nice (for most people) to know what such a low ranked security rating (2) was getting captured for (they are normally Mozi.a, Mozi.m and jaws “GET” servers that will connect the device to a BotNet or similar command-and-control network).

9933 blocked : 0 in last hour : 10 in last 24 hours

sshd attacks have dropped right off, 2 blocked every 3 days, but the caputered web urls (haxor-access.log) are producing 10 a day (which makes sense, there are more checks now). And no kex entries either.

Overall, I would say (besides what I have blocked already) sshd attacks worldwide have dropped off, as I can’t see ALL of them being routed through Azure. ISC showed a slowdown too, when I was there last (which I thought was odd at the time).

It maybe that “higher up the router foodchain” (internationally) more blocks have been put in place, but I dont believe that either, as alot of the “one off” sshd attacks were from random Asian sources.

Who knows, maybe its just the “quiet before the storm” …

10023 blocked : 0 in last hour : 11 in last 24 hours

well the end of last month saw a severe slowdown in the number of sshd blocks produced by the Gerka, this month (12 days so far) see no sshd blocks by the Gerka. So I checked the logs and sure enough, no sshd entries for root or invalid user, but still some kex (key exchange) failures.

However what I do see are the odd banner exchange: invalid format entries. These are new to me, but I think they are backdoor overflow attemps (thats a guestimate, based solely on the fact they fail, and there are no user attempts).

Again, I am puzzled by this lack of attack presence, especially considering the uptick in webserver blocks. I cant believe I have managed to fully block all sshd attackers . Remembering that those 10k blocked IPv4 dont include the 700+ ranges. Then again, maybe those nearly 800 ranges is enough to cover most compromised botnet devices and virtual machine services (but I highly doubt that).

EDIT: hmm… those banner entries match the pid (process ID) of the kex failures - banner line contains invalid characters. However the blocked kex IPv4 dont equate to those associated messages …

10138 blocked : 0 in last hour : 10 in last 24 hours

1 sshd attempt before log rotation 2 days ago, and 1 today, so the Gerka is definitely still working. A couple of kex blocks as well.

Still, a wierd month with more regular kex blocks than sshd password attempts, and those dont include the malformed banner either.

POST /scripts/WPnBr.dll HTTP/1.1

Thats something new in the captured haxor log. Guess they are looking for windows based webservers? Shame the logs dont capture the POST data, but then again it could be megabytes long, so no point in worrying about it, just nice to know it still captures new stuff properly.

A note here, basically the only reason I can capture these is because I dont have POST setup on the downstream server, so it always returns a 405 error. I guess I could add a .dll check to the URL checks for when I do finally have POST set up. Not sure how well that would go down on a WSL server, would definitely have to test before making further presumtions.

10449 blocked : 0 in last hour : 8 in last 24 hours

Well its been a while (46 days since the last post) but the sshd attacks are finally back in full swing, since 11am UTC on the 9th on June. They were not gone, the were more than a few kex entries blocked in that time, and that 1 sshd attempt, might have been close to 60 days of near silence in total.

EDIT: just checked the end of the block list and its all quiet again by 6am UTC on the 10th June.

Nothing else to report atm, except that the web server log generated blocks are “raging a storm” . The power situation here is fixed but the weather has caused issues, so no further work done yet …

1 Like

10803 blocked : 7 in last hour : 69 in last 24 hours

still racking up the web sever blocks, but since the 27 July sshd attacks have ramped up again (from 0-ish), with the last week of logs being 90% sshd based blocks. Over the past month there has been an increase in attacks from Serverion, a Netherlands based server farm operator that also hosts non-european IP address (eg Russia), and has a connection direct connection to OOO SibirInvest in several countries. It almost looked like someone was targeting their servers for staging, as there were only one or two attempts recorded before another address from a different location (but same operator).

The over-range blocks (besides various mentioned above) are mostly new DigitalOcean ranges and new Microsoft Azure/Cloud assigned server addresses (ie more from the usual).

The only other real news was that the ISC (Intenet Storm Center) stopped being functional a couple of weeks ago (maybe the weekend of the 4th July), which I only noticed after trying to contact staff with a possible new Mozi.m style infection server url, spreading something called aqua.mpsl (with multiple architecture binaries found at the url), very similar in size and attack type (presumption) to the jaws vectored z0r0 binaries found at the url supplied in those web attacks.

EDIT: I did spend a couple of days putting together a page that makes it easier (visually) to decide if a block-range or over-range block is nessecary, as it references everything in the log file (which gets 0’d each month), not just one type of block, which is what I have been using for xref up until now. This allowed me to examine possible control panel integration with other more commonly used web hosted applications or suites (eg pihole). (ED - this is working out really well, highly useful, even if the process is still manual)

EDIT 2:
I did find find a bunch of 0 length files floating in various places on the file system, but research showed it was (probably) due to the Apline Linux Web Administration being started by default (I turned it off by default after I found a couple of entries). Its protected by password, but it appears that a valid user is not checked until after the file is created (and hence no content is recorded). I need to look into this with a new Alpine Server release as this app is quite complex in how it hides actions from the web user. (Knowing me, I just found another bug, but we will see).

1 Like

so its been about a week since the last post, around the beginning of the month, which is when the logs are scrubbed (copied to archive folder, then emptied or deleted - depending), and even though things where starting to change before then (sudden uptick in sshd attacks), it has become even clearer that something else is going on, as the webserver blocks are way, way down, about a 3rd of what they normally are, while the sshd blocks are up about 300% (if 100 blocks a week were regularly performed).

So whats going on here, is there any way to “devine” what is happening? Well maybe… but its not due to anything related directly to whats in the log files, its more about whats not there, and what we have learned from the last 12 months (yes I did mention starting the project before #devember2021 ).

Most of web server attacks are either:

  1. from an infected source (part of a command and control network ), or
  2. command line tools (aka script kiddie attacks )

As it happens, a certain percentage of (normal?) sshd attacks are of a similar groupings (where script kiddie attacks try multiple usernames in rapid succession), along with the regular “security probe” (web) services (note that alot of this data ends up in the wrong hands for monitary gain).

Without taking into account the possibility of the hosting service applying some form of screening, service blocking based on analyrics, or IP region blocks, it appears there was a re-organisation of co-opted “command and control” networks.

Basically, whoever was running their own C+C either lost control of their network, possibly sold their networks, or were going through an organisational change, either way “they are back in business”.

On the webserver side, it is obvious when a “script kiddie” is trying to get in - 80 different url paths from the same IP address. Webserver attacks from compromised units or C+C networks are not like that, they know the urls they are trying to access are device specific so there is at most only one probe (possibly 2) from the same IP address.

Although I say above that webserver attacks are down to a 3rd there regular rate, 20 different IP addresses for a 7 day period is “quite normal”. Those are the sorts of numbers (20-25 per week) I was seeing around the time after I started applying range-blocks.

As mentioned in a couple of previous posts, there looked to be some sort of organisational use of hosting services, particularly rotating them, which may indicate that over the previous 2 months they were using those services again, and since their IP range is blocked, I dont see anything in the log files of the SSFW server.

As a side note, this leads to a problem covered in the initial couple of posts on both this project page, and the original post thread - how to detect “threat level” at the interface, after blocks have been applied, so that “threat response” can be lifted, or made permanant (ie dont check it again).

As you can see there are no real answers here, we are “guestimating” based on previous data, and research of that data, and that is the point at which some form of high level (or AI) analysis-over-time application would come in handy. Without a bigger data set, like that collected by the Internet Storm Centre (ISC), a visualization tool like those used for repository commits might be more practical in the interim.

It is worth noting the a similar “down time” appeared over the Christmas - New Year period. School holidays have just finished in certain parts of the world, but that only accounts for a 2 week period (over the last month), not 2 months.

Well, I guess you might classify this as more of a “gut feeling” assessment, rather than a thoroughly factual based assessment, and some might even say thats just a “knee jerk reaction”.

But it is clear there was a shift about 2 months ago, and there has been another shift again.

EDIT: with the research required for IPv4 range-blocks, there are some details that come through over time, that portray a certain scene .

  1. certain attacks from South American countries are performed by the same person.
  2. certain attacks from Southeast Asia are performed by Chinese nationals.
  3. certain attacks from Ukraine, Crimea, Estonia are performed by Russian supporters.
  4. Dutch / Holland / Netherlands is still the highest individual attack source vector.
  5. England and USA are right up there with China and Russia as “threat antaganiosts”.

EDIT2:
I suggest any tool more than visualization, needs to assess not only a larger data set for any given time period, but also need to xref geolocation news story corelations, as it may show why there is an associated shift, when that occurs.

EDIT3:
after checking 50 IP addresses of the 300 newly blocked sshd spam, it appears that the majority of them are from “dial-up” service providers all around the world (200 services in 150 countries) , and those that are not are “hosting” service in Russia or Hong Kong, and to a lesser degree Korea, Sweden, Netherland, UK & US (yes there are a few universities in there), as well as multiple from Isreal, Iceland & Iran.

Bumped with Brutality Bonus updated in the OP

get your “MWHahaha” shirt ready, and be prepared to asume the position chanting “I’m not worthy” if you are on the end of one of those entries :wink:

Gees, this would have been so much more effective if everyone was still on dial-up, but then again no one bothered to waste there valuable bandwidth hacking via web urls (not to mention the months it would have taken to upload the 4.5Gb file to the server for the Brutality Bonus to be of use).
:slight_smile:

ok, its only been a few days since the previous 2 posts, but the new (consolidated) IPv4 block list view is making range-blocks and over-range blocks really easy to “see”.

What’s to note from today’s range-block and over-range session is (12-ish), yes from many places around the world, but there is often a pattern to “probes”, 3-5 days apart, 1 sshd and 1 webserver, both from different IP within the same service (sub-net), AND the majority (50-60%) are from mobile (ie cell phone) service providers.

In the US, there is another round of FranTech / PonyNet attack sources. And more international Amazon AWS attack sources, this time India (last time it was Hong Kong).


For those who are interested:

I use (linked in output) CentralOps Domain Dossier to do lookup and cross reference IPv4 haxor sources.

If I want to see some history of that same IPv4 (also linked in output) I use ISC Internet Storm Center (click IPv4 in resulting page to see current history list).

If I want to check what ASN is assigned to a sub-net I then use DNSlytics IP2ASN API and find other (sometimes associated) sub-net’s assigned to them with Dan’s BGP Lookup Tool .


so within 2-3 clicks I know where the IPv4 is (as well as if its still active, and if so, what basic services are running), who owns the sub-net, and who runs that part of the sub-net, and what the hostname and any alias to that same server (if any) exists, and where the domain name owner is, and its location (if different), and (visually) wheather a sub-net over-range block (110.36.0.0/13) should be applied, instead of just a sub-net block (110.39.34.0/22).

This is the part (combined with the IPv4 block-list and what it was blocked for) that I want to apply some AI or other algorythmic automated block analysis (and in my case, automated over-range blocking).

This is really quite complex, not due to the data that need to be analysed, but due to the fact that different whois nic services (RIPE, AFRINIC, etc) have differing data sets, or differing data construction sets.

Some of the US, UK and EU also apply privacy tags to the data, which makes tracing and tracking more differcult, and sometimes its impossible to get a direct result without doing a search engine query.

For example: AKAM.NET this is Akami (new owner of Linode BTW), but there is no server or IP address assigned to that domain, however it is used for sub-domains, most of which are nameservers. And you can’t get that info without doing a search engine query, as “Akami” does not appear in any of the network records.

2 Likes

just spent a couple of hours blocking another bunch of ranges and over-ranges. This time they were just about all hosting services, and looking at the previous 2 sessions of range blocking, there is a concerted effort by individuals who are abusing network service owners (as opposed to compromised routers used in botnets).

What to I mean by this:

  1. OVH is an Australian hosting service provider, today it was their Canadian network, yesterday it was their French network, last week it was their (can remember) network, last month it was their Singapore network, etc

  2. I have not blocked any Japanese ranges for a while (since the sshd slowdown 2 months ago), but today I blocked cnode.io in Japan, and cnode.io in Thailand.

Again these are not multiple probes from a single IP address, they are multiple single probes from different IPv4 inside the same sub-net (inside the same server farm and/or DMZ - either way, same physical service location).

I think today there was only one mobile related range block, and the IPv4 sources were dated yesterday (I ran out of time halfway through the list last night).

So there is definitley a conserted effort, and I would say coordinated as well, in an orcastrated way, as opposed to an organised associated way, IE. more like how cells work, common goal, specific non-overlapping tasks, but without collusion between them (except maybe in the collecting of the resulting data).

On top of the obvious Russian (Eastern block) and Chinese (SE Asia) efforts, there seems to be another group that has (possibly) spent the last month (lack of sshd attacks) developing a more indepth strategy based on previous experience (eg. I’ve mentioned blocking just about all digitalocean near the begining of this thread). The various Serverion and Hostinger and some other Dutch sources starting to show up gave me a heads up to something like this happening in the background, but based on the number and dates in my logs, “its fully under way”, whatever that strategy is …


I mentioned DigitalOcean above, not only because of the original abuse pattern I saw (at least 1 IPv4 from nearly every range they owned and assigned to different locations around the world), but also because today I found a new range of which one IPv4 was registered to a .ru service location. I saw a similar network owner abuse at the begining of the year with FranTech and their sub-net partner services (another PonyNet range was blocked yesterday) .

EDIT: I found another new DigitalOcean range in the logs, but the sub-net is spread all over the place. One IPv4 traceroute’s through New York City, another through Telstra (AU) Singapore-China router, another through different Telstra (AU) Singapore router (not China). I’ve seen this before, but I can’t remember if it was a DigitalOcean IPv4 addresses I was researching.

FYI :
These are TelstraGlobal SouthEast Asia backbone and peer routers, some of the biggest in the area. Telstra is Australian owned, and Australia is part of Five Eyes .

The Nginx web server at the end of the Singapore-China connection has a Quisk SSL certificate, “Quisk is a global technology company that partners with financial institutions and others to digitize cash and provide safe, simple and secure financial services and cash-less transactions for anyone with a mobile phone number.”

1 Like

11653 blocked : 4 in last hour : 45 in last 24 hours

So another week goes by, and another multi-hour range block session is nearing an end, so whats new, well a secondary “underlying” pattern appears, still methodically using hosting services ATM (Serverion RU,NL,DE,BG,Denver, Hostinger and Hetzner too), but this time I catch them because 7 days has passed (literally to the day) since the last attack probe from the same sub-net. Nothing mush else to say about that, except that it confirms the suspicions of the last few posts … ce-la-vie

EDIT: 100 ranges added, hers the 1st and last:

835:2022-08-19T15:49:39+0000#19152#0#range#added: 14.32.0.0/11 kornet-kr
933:2022-08-19T21:26:03+0000#8567#0#range#added: 220.128.0.0/16 over-range-various-tw

2 Likes

27215 blocked : 1 in last hour : 59 in last 24 hours
5608 since last archive

log clean tomorrow (my time), only 2nd time since Feb 2023, used to do this monthly, but been side-tracked, and only did the last one because Akamai (who bought Linode) insisted on resetting my server without notifying me

Here is a sample of the HTML table output i use to view what been blocked since last log clean (makes it easier to see new ranges that need blocking):

...
 45.56.98.98 	 1  	2023-07-17T06:56:48+0000
 45.61.49.164 	 1  	2023-05-06T09:53:25+0000 *
 45.61.162.124 	 1  	2023-05-03T15:53:08+0000 *
 45.61.186.237 	 sshd 	2023-06-30T00:16:51+0000
 45.66.230.26 	 1  	2023-06-02T21:55:53+0000
 45.66.230.69 	 4  	2023-07-04T16:00:07+0000
...
 45.79.163.53 	 sshd 	2023-07-17T05:56:00+0000
 45.80.158.38 	 1  	2023-06-05T21:56:05+0000
 45.81.39.8 	 1  	2023-06-19T15:55:18+0000
 45.81.39.254 	 4  	2023-06-04T15:56:16+0000
 45.81.243.4 	 1  	2023-07-10T09:56:27+0000
 45.83.122.140 	 sshd 	2023-05-15T13:10:20+0000
 45.88.8.95 	 sshd 	2023-06-20T09:22:12+0000
 45.88.90.105 	 1  	2023-07-02T12:56:09+0000
...

What I see visually in a web browser, if the date is grey text (* here), that IPv4 was blocked by the cron job that scans the web server log files (and always run at 53 mins after the hour), otherwise it is black, because it was blocked by the script that is the SSFW process .

there french mate not australian.
they have servers all over th world and provide cloud services in australia. but as i said, a french company.

1 Like

I seem to have realised the OVH reference at some point. I think at the time the only OVH “attacks” I had in the log were from Austalian servers and that they were probably newly commissioned too, hence their over-abuse

Later on I tracked down OVH servers in the log from Netherlands (whats new) USA, Canada, somewhere else (SouthEast Asia maybe, cant remember now) and finally France.

On that note, it often seems that the rise in abuse of a service coincides with publication of new servers / services / locations (like MS “cloud” servers mentioned further up).

Will post another “monthly clean logs” result next week on Wednesday, only the second one in a year - mass blocked all Tencent ASN ranges just now, had to tweak the ASN4IP script, as the source output had changed, still around 1-200 maybe got blocked, of which I had already manually added some ranges over a year ago (the reason this time is that they were all SSH attempts)

8178 since last archive

When ever I do a “server archive HTML”, which basically zero’s the log output, but records the HTML output of those logs generated by my PHP pages, I also run a script that allows me (and you, its in the git repo) to automate an updated download of the blocked IP addresses in various formats.

Looking at the .ranges.ipv4 file (which is hidden so it does not interfere with ls output from the block directory in the scripts) in the ./_blocks/ipv4/ directory, I can see those last TenCent IPv4 ranges I blocked (because they are sequential and start at 1.12.0.0/14), and there are 2129 ranges block (in an automated way by a piped shell command).

To be clear, I knew there would be alot of ranges to block. However in the past I have taken some time to see who owns associated sub-ranges, and wheather there are any IPv4 blocks (or how many) in “a similar range”, and produce a much simplified single range block that encompasses all ASN owners.

I am not doing the above for every single TenCent range, as there are 2129 and I just dont have the time or the patience for that.

This does point to yet a further need for some better automation (mentioned elsewhere in this thread), as especially here, a bit of basic maths on the parsed ASN IPv4 subnets would also produce a considerably condensed list of block ranges for ip (or an any firewall software ) to deal with.

No headway has been made on v2 as of yet - there simply is not a demand for it. 2ndly from a security point of view, nowadays you are better off setting a default allow=no on watever IP blocking you are using (incl. IPv6) and use allow lists instead. That way you could send a unique ID to any user, that clocks into an Enable Only, Frontend Bounce Server (on a different address), and the resulting server would only have to deal with a limited set of IP adresses and / or ranges

This does also bring up an idea of a default list for publically facing servers, that includes known services that allow even temporary abuse (which is most VM hosting platforms - HoHum) -Note that this does not eliminate broken / hacked routers etc from try to access and compromise your (usually, but not always) web services.

So the only other notable thing worth mentioning here is, Nginx had chrashed for some reason.

Turns out, Nginx does not post a log entry until the HTML / PHP /(insert other server here) has finished sending its content (which I had already figured out a couple of years ago).

Why does this matter?

Well in my case (OP topic (sic)), if I know the requester is nefarious , I dump a 2.1G DVD file back to them at 1K per second . Seems like a few people dont care how much data they collect from a web server response or even if it is HTML, so basically if you get too many Nginx worker threads serving data for too long , the main Nginx Process will timeout (possibly because it holds data structures for the final log output), and crash.

I dont know if its even worth posting anything on the Nginx dev list, as most likely people will say “well thats not how Nginx is normally used / setup”, and basically ignore the problem. I have had that sort of response from other projects and they always get bitten in the a.s.s latter on (even if it takes a few years)

in this case, the solution would be simple, but most web servers (actually most internet facing servers in general) are not setup to cope with the required changes.

dont wait for sub-process to finish before creating a log entry

dont wait for “total byte count in response” before creating a log entry

And that seems to sum up the collective way of dealling with security threats on the internet - record something after it has happened, as opposed to when it starts .

(quoted from the projects GitHub page)

searching “super simple firewall” now shows this project 1st.

Turns out this is still true for Google but no longer for Brave (which I switched to about 3 years ago - RPi) - it appears Brave ignores “super” in the search .

Not sure what to make of Bing, but it is on the first page, so … (???) …

28669 blocked : 2 in last hour : 14 in last 24 hours

things are a bit quieter after 2029 TenCent ranges were blocked … ish


just re-read the Github readme, the only things worth noting are:

  1. after 4 years and 28K block IPv4 (not including ranges) , by the end of the month, the CPU (on a 1 core 1G VM) is pegged at 100.15% usage when running the block checker a couple (4x ?) times an hour, but this is fine, its a shell script, so the Kernel is getting an oppotunity to do something every time a command executes, so it wont slow down any web server processes.
  2. the saving grace of the IP block checker cron jobs, is that they dump the contents of ip to file first, before doing any queries, and its the processing of this file via grep which pegs the CPU, but its via a pipe, so its non blocking anyway (ie the Kernel could care less).
  3. also, the Gerka’s dont scan entire logs, the SSHd Gerka culls the logs for SSHd entries only, and only works on the last 20 (every 4 seconds), and only does something if these entries were not the same as the previous 20 from 4 seconds ago.

NOTE: even though 100.15% CPU usage is mentioned, that amounts to 16% CPU usage for any one 24 Hour period, on a 1G single core VM


One problem I have seen is where ip has a record of a blocked IPv4, but the script can not find the matching IPv4 filename in the block directory - this appears to be becuse the proces that just updated the block, has not finished writing that marked IPv4 to disk yet.

This is a similar but different problem to log entries being created faster than they can be written to disk (by the OS / Kernel) . It may be a result of the VM containers, the infrastructure, the OS, or the Kernel I/O, but either way, there is nothing that can be done about it.

From SSFW point of view, I could write a flag file when a cron job is running and have the Gerka check for that and skip is task, but that is A) counter productive, and B) more needless disk access every 4 seconds, so overall, not worth worrying about, the resulting file turns up anyway.

If said file can truely not be found (by checking for it manually) it turns out that there are 2 files with the same name (one is corrupted), and you can tell by the timestamp of the error in console (its a kernel log message) and the timestamp in said dual file. Again, this is something to do with the OS or infrastructure latency, not the shell scripts.

On that topic, I have recently been looking again at a 0node filesystem because of the limited content of the files in the block directory (a timestamp) and the wasted space due to allocating the IPv4 filenames (four x3 digit + 3 dots max).

Even if I record more data in the file (why it was blocked, and from which service its was detected, and what type of detection was used), that content is still considerably less than 4K of data which does not help the longevity of (current) SSD technology (which has a minimum 4K re-write block), and we will start to see 8K and 16K blocks as sizes of SSD keep getting bigger.

Anything on a tmpfs gets cached by the Kernel through the Kernel Swap mechanism - ramfs does not, but you can not dynamically “grow” a ramfs mount, even though it will dynamically fill up .

In theory , if you were to have a sparse volume file (disk / partition image) on a ramfs and use that as a tmpfs substitute the resulting files would not pass through the Kernel Swap mechanism AND the are already in memory even if they do get cached .

So, what other ways can you record an IP address that is easily searchable via (already available) command line tools. Well you could use folders for each network part, but then what about a subnet mask , and what is recorded at the end of that directory path. Plus, each folder takes up an entire inode just to distinguish itself from any other in the same folder or on the same drive.

0nodefs says that you either dont record seperate directories, rather the whole path (to that endpoint), or you dont actually record the resulting file on said media, but a link to its actual location, or you do both. And if you need to record file contents, it is always going to be much less than the (minimum) block size defined but the current 0nodefs structure.

The “format structure” of 0nodefs is predefined by the minimum block size of the underlying media (so 4K ATM) , and part of that structure records a counter in an index of endpoint (file or folder) in “human searchable” form.

<< intrupted for dinner & movie >>

1 Like