They would need to reset the server to some sort of safe known state before handing over a server to the next user.
Any hard drives and SSDs would have to be entirely wiped. But what of the numerous pieces of firmware? Who knows what the previous user might have installed there.
Which got me thinking, what do the big dedicated providers actually do between one customer renting a specific server and the next?
I’d like to think they do more than just a wipefs and new partition table, but the commercial world has a history of overlooking security until it gets bad enough to impact profits or stock…
That is an interesting question. Some of dedicated server vendors use consumer grade hardware and I wonder what they do with it when transferring to another customer. I went trough Hetzner FAQ and Docs, but can’t find anything about it. I think an email is in order, and I’ll report back when I hear from them.
There is no IPMI access and even KVM is on request.
In 2019, we at Hetzner decided to no longer provide customers with additional IPMI or KVM modules to install on their dedicated root servers. There are two exceptions: DELL servers and DELL servers from the Server Auction. Just in case there is a fault, we have kept a few of these IPMI modules.
A KVM Console is free for 3 hours. If you need it for a longer period of time, you can book it for 3 more hours for just € 8.40 (not including VAT).
Yes you should encrypt stuff but yeah - you basically give physical access to the next customer.
I do hope you don’t have the illusion hosting providers haven’t thought about such scenario
SSD’s are cheap, replacing them simple with removable trays. Heck, some might even just take out the used server and replace it entirely. Data centres are major users of off-the-shelf servers like Dell, Supermicro et all and their discounts are considerable as a result. Writing off an SSD or even the entire server is part of the price the customer pays anyway.
I’m sure they did, but I doubt they’d be sharing their “”“trade secrets.”“” I’ve heard some cheaper ones just format everything with 0s (idk, probably with something like dban), while others just replace the drives entirely. All of them will charge for the hardware (more or less than what it actually costs), then subsequent monthly hosting fees (that will more than make up for the hardware).
I seriously doubt that. You can get a dedicated server for a month for like 45€ at Hetzner with no commitment and no setup fee. And given that it runs on Ryzen 3600 from 2019. looks like they don’t throw shit away just like that.
As for SSDs wouldn’t secure erase do the job in a few seconds? I’m more interested in HDDs.
HDDs, at least more modern ones <5y old for sure, support Secure Erase.
When you rent whole servers from hyperscalers, it’s the NIC that implements a gazillion pcie devices, like fake nics, fake hba, fake USB controllers and devices. And the NIC/BMC end up reflashing everything. Those NICs are expensive.
We have a standardized wipe process if the hardware is defect or our customer terminate the contract (This is also stated in our technical and organizational measures on page 13 under the heading “transfer control”.) Just use the following link: https://www.hetzner.com/AV/DPA_en.pdf to see our TOMs.
Hetzner uses various techniques for deleting data from hard disks and the like. These range from multiple overwriting with defined characters and checking the characters at the end of the wiping process to using the secure erase function of the hard disks. All erased hard disks are checked again after the wiping process. Only after the check has been completed and successful is the reuse determined. Defective drives that cannot be securely deleted shall be destroyed (shredded) directly in our data center.
Unlike the US of A who has their Suite B public so any flaws can be found, Germany operates on security by obscurity, which seems to occasionally spring catastrophic leaks…
It took more than a month, but they did reply again:
We have no procedure to prevent potential firmware level rootkits after cancellations. When security vulnerabilities become known, we check the impact on our server models and configurations. In doing so, we weigh up possible options, such as updates or further actions.
Thanks for asking vlvante, about what we expected but hoped wasn’t the case
@Dutch_Master while I’m sure staff have thought of such things, it is another thing to allocate the money to negate it. I believe addressing the threat would be rather expensive, people have to be assigned to do the work, the system provisioning process becomes more complicated, and perhaps worst - for those with diverse off the shelf hardware there is unlikely to be a universal cost effective mechanism.
To not spend a significant amount of resources countering an attack vector customers rarely think about, hasn’t manifested as a reality for the hosting company and has no compelling narrative regarding how it’s going to grow this quarters profits is to be expected.
Sure, but can they afford not addressing it? It may save them in the short haul, but when a client is compromised, can they afford the legal bills that come with a conviction in Court? Pray said compromised client isn’t a Gov’t entity, 'cause then they really have a problem! (loosing Gov/t contracts and/or the ability to bid for them is one thing, loosing their business licence is a considerably higher existential threat, never mind the public weighing in on the issue by ditching them completely)
@Trooper_ish Good point(s), sure. But if the source of the compromise was traced as a BIOS root kit the hosting company failed to remove prior to reprovisioning the machine to another customer, regardless whether they were actually aware of its presence or not, for the sake of saving a few bucks/quid/euros/whatever, pretty much any Court would hold them responsible for not addressing the threat pro-actively, as it can reasonably expected of a professional hosting company to assume the threat was there in the first place. Which, depending on which country said hosting company resides in and/or the jurisdiction the legal case plays out, may lead to seriously high fines and compensation claims. Particularly the US Court system has a reputation to uphold where these amounts basically have no upper limit.
(as an example, the recent high-profile Carroll VS Trump deformation cases resulted in seriously high compensation for the victim awarded by the Court)
Can’t really blame a party that does not know about it though, until you institute a framework demanding regular / routine checks for such things though
Normally, I agree. But a professional company who’s sole purpose is to rent out server space can, must and shall be expected to have measures in place to mitigate the risk inherent to the job of renting out servers to customers by a Court. That includes getting rid of BIOS root kits and the like before reprovisioning that particular hardware to a different customer.
