Server inexplicably inaccessible overnight: help!

Hello all. Over the past few weeks, I have been helping a friend set up a small Debian server at their house. After about a week of work, the server had nginx, an ssh server, and a Minecraft server all up and running. We additionally had Tor hidden services working for nginx and the ssh server. Then one morning, we found every service on the server to be inaccessible beyond his local network.

Trying to access the website on port 80 for HTTP, or 443 for HTTPS via the domain name does not work. It also does not work when the public ip address is substituted. The ssh server is in the same situation: it cannot be accessed via the domain name nor the public ip address. Both of these are confirmed to work on the local network. The port forwarding rules are still in place, and the local ip address of the server has not changed. The systemd services for nginx and sshd are running and not reporting any errors. The server has internet and can ping external addresses successfully. Debian’s default iptables have not been altered.

An nmap -Pn of the server did not find anything.

What perplexes me is that the Tor hidden services for ssh and the website also no longer function (Tor’s systemd service is still running, too). Tor does not require any port forwarding, so this seems like a very fundamental networking issue. What would stop all of these different services from being accessed?

TL;DR overnight our server has become inaccessible from outside of its local network. Both clearnet and Tor traffic cannot reach it.

I will solicit any diagnosis, no matter how crazy. Like Car Talk.

Has any of your public / external IP addresses changed?
Do you just need to update the domain name?
Can you curl http://ext.ern.al.ip/booya and a get a 404? From your machine? How far towards the server can you get?

@risk Our public IP address has not changed. It matches the domain record. We can curl, update packages, and ping websites succesfully from computers on the network, and from the server itself. We can access every service on the server from the local network, and the server can reach the outside internet. The server just cannot be accessed from outside devices.

This is what I asked you to try, curl your own server from outside by your public IP address. And see how far the packets go.

If thay does not work, you can also try using mtr with TCP packets on a particular port, which can help you determine whether your ISP is blocking you.

I see what you’re asking. Here’s my curl attempt.

[n8r@fedora ~]$ curl xxxxxxxxxxxxxx
curl: (28) Failed to connect to xxxxxxxxxxxxxx port 80: Connection timed out 

I also ran traceroute to the ip address, and it seems to jump from addresses associated with my ISP to ones associated with my friend’s, but it ultimately fails:

25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *

Thank you for bringing mtr to my attention. I will do some learning and try it out.

Unless you are paying for a business internet connection, MOST ISPs in the USA port scan and will block external access to your web server. It is very likely that they got you and you need to call and upgrade your plan.

Mastic_Warrior is correct in ISPs blocking.
If nothing has changed on Server, fault is between server and WAN

Check:
Router settings for NAT, Port forwarding changes, Firmware (router could be hacked)
Firewall for ports blocked/updates/rules change
Firewall on server

While I am inclined to believe that the ISP has blocked external access, I do not think this would explain the failure of the hidden services. Tor does not depend on any ports being open. To my knowledge, an ISP would have to go out of their way to block Tor traffic. This suggests that there is a more fundamental problem.

Why not start with the bare minimum?

On that network, open up the GRC ShieldsUP tool and have it scan for common ports. This will tell you if the ports are open or not.

If the ports are not open and you can confirm that they are on the router, call up your ISP and complain.

This wont solve the problem, but it will help provide more information.

Just to be safe…

Double check there’s free space on the device with df or something.

2 Likes

Good thought. There is plenty of space:

Filesystem      Size  Used Avail Use% Mounted on
udev             16G     0   16G   0% /dev
tmpfs           3.2G  828K  3.2G   1% /run
/dev/sda2        90G  2.5G   83G   3% /
tmpfs            16G     0   16G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
/dev/sda1       511M  3.5M  508M   1% /boot/efi
tmpfs           3.2G     0  3.2G   0% /run/user/1001

We have temporarily fixed the issue by completely reinstalling Debian. Bullseye was just released, so it’s not entirely in vain. The process took two or three hours.

It’s all up and working now. There are a few differences between our current config and the one that failed. One is that we had installed XFCE the first time. Another is that we assigned the static ip via the router instead of /etc/network/interfaces.

Now it’s time to wait it out and see if the server has another failure if left alone for some time.

Yeah, you should always make a static reservation on the router for a webserver, always.

Also, do you know what you were using to manage the network stack? DHCPd, systemd, Network Manager, and etc.

Did you have any system upgrades applied between when it was working and when it stopped working? If it was not an ISP issue, sounds like it was a system issue.

Also, do you have any exceptions in your firewall on the machine? Are you running a firewall on the machine?

Are you using AppAmour (Default for Debian) or SELinux?

I am glad it is working, but if this is going to be something used for production, then you really want to do an after action report to get a better understanding of what went wrong and how to fix it in the future.

Oh, my sweet summer child. ISPs are Monsters, at least here in the USA. Sometimes they go out of their way just to nickle and dime you for normal, on label usage. With that said, if reinstall and reconfiguration fixed your issue, then that means that something was wrong with your setup. Ask yourself the above questions and try to answer them. Knowing is half the battle.

It’s a production server insofar as we want to play Minecraft :slight_smile:

Believe me, I’m not one to underistimate the evil of ISPs. It just seemed unlikely that my friend’s would be making any direct effort to block Tor. They’re continuously updating the cryptography to make Tor traffic harder to discern: https://www.youtube.com/watch?v=YZKoBiQkl_Y

The network stack must be the problem. We did no updates, no firewall/iptables changes, no overnight updates, etc. We used just ifup and etc/network/interfaces to connect to the internet. There must have been a misconfiguration that caused this strange, partial breakage. I am doing some learning about networking to try to get to the bottom of this.

1 Like

It’s been two weeks, and there have been absolutely no hitches since reinstalling. We’ve configured a cron job to make sure the domain name is pointing at the correct IP every hour. I’ve done some more experimenting with networking, and I can say confidently that this was a misconfiguration.

1 Like

I am glad that it is working for you. If the original mis-configuration cannot be found, let’s hope it was just a typo or a blip in the Matrix.