I run a microscopy facility at an R1 university and we just got hit with ransonware. I noticed cluster jobs failing to write to network storage last night at 9 and emailed IT. This morning at 8:08 a colleagues lab computer running Windows 10 had a big ransomware notice on it.
I unplugged all my facilities servers, storage, and computers (that I had to fight tooth and nail to have in my facility on a subnet instead of in a central location), and it appears to be fine. Including my Server 2012 running one of our cameras I just firewalled (against university policy) two weeks ago.
But we have 300 TB of data on university central storage that is inaccessible, status unknown (the stuff that failed at 9:00 PM last night).
We got an email at 10 saying they were having “network” issues, and it took until 12 to admit it was ransomware. They finally shut down the network 4 whole hours after the attack was discovered.
At least from my facility, while datasets are sometimes confidential, they’re dozens of TB and would take forever to transfer. So I think we’re safe there.
The central storage we used was compromised, but supposedly there’s a cold backup from 2 days ago. I’m being told tbat it’s only cold because the backup server broke 2 days ago and has been sitting powered off lmfao.
The rest of the campus is in complete disarray, The network is offline, and the domain controllers may be compromised.
You would think universities with all of the smart people would be last to get hit by security issues, but having attended one recently… the software is woefully out of date, and the infrastructure is just scraping by on tattered remains.
It always surprises me a bit how you can have such an unprepared infrastructure that hostile code is able to propagate. A well-thought-out configuration of the environment already strongly limits the propagation. As usual, probably no one bothered to make the necessary changes and as always, as long as it worked and somehow it will be.
People dealing with the IT infrastructure must finally understand that the construction of the structure must be done on the basis of zero trust and every host in the internal network is a hostile machine that does everything to cause us problems.
It’s sick that one desktop where someone clicks hostile shit puts everything else, servers, other desktops to the grave. Rarely are we talking about sophisticated code using 0day here. Usually, these are unsecured and poorly configured resources that are vulnerable to even very old and simple attack vectors.
I can understand that we lose the battle when we have resources exposed to the world because there must be and then someone manages to penetrate the machine … but the fact that an infection from the inside on Mrs. Samantha’s desktop in accounting crashes everything in the company is a sick joke, especially in 2023.
So when I hear stories about someone in the company clicking something on dekstop and the whole company went to sleep, that’s all I can say to the responsible IT team… Change your profession because you simply don’t know what you’re doing!
Budget is one of the problems, but in recent years, hiring and keeping jobs based on “ideology” also leads to massive problems.
A perfect example recently from another industry was Budweiser, where, according to ideology, the wrong PR move is introduced to the company model that only brings losses.
There is no approach to the case, let’s do it as best as possible. It is used no matter how bad, as long as it is in line with the ideology…
I would think having each department handle their own stuff would be good since then an attacker would be looking at multiple possibly completely different targets.
If you worked in IT you know that people like to do this. If you don’t have some level of policy, you end up in an unmanageable heterogeneous ecosystem/mess that involves much more work and money to run. And budgets are a thing, not just in Universities. And you want to be efficient to offer as much services as possible to your co-workers. Centralizing is very efficient. That’s why we buy a NAS or use Clouds after all.
But as we can see in this thread, there are downsides that may or may not be a concern in everyday use. Decentralization may just increase costs without the benefits ever manifesting. It depends on the perceived security need. And we humans like to prefer immediate benefits over committing ressources regarding drawbacks that “might happen”. That’s just risk management and maths.
This is a rational and economic choice after all and I can’t blame decision-makers for doing so. But these kinds of attacks become ever more visible and threatening, so it’s harder and harder to justify the default and previously reasonable behaviour.
I have a degree in business computing and I know both sides of the equation all too well.
It will happen anyway, unless you send armed guards at night to confiscate any non-centrally managed IT equipment, and even then, the hiding spots and shadow-networks will just get better hidden.
Most universities are large enough to have at least one server room per department anyway. So, embrace the decentral nature!
Centralizing is nice and all, but also turns EVERYTHING into an inconvenience since nobody has it their way now. A system that caters to everyone can never be secure.
Alongside all the smart people you have a lot of students, only some of whom will turn out to be smart. Also even smart people make mistakes if it’s not their area of interest. I once worked with an email admin at a US university. They had security on the inbound email gateway but nothing going out. The reason he gave was that they had 40 thousand students with little attention to security best practice. They secured what they could control but left the rest to fate.