Problems in defragmenting ReFS on HDD file server

High-Performance Computing Workstations & Servers
1

Hello and welcome to ponder upon a rather rare(?) problem.

I have a 12 TB Western Digital Gold HDD formatted to ReFS, that contains over 14 million “SQL-esque” files that are in need of a defragmentation. The problem is, that because of the ReFS metadata fragmentation and the degree of fragmentation in general, every defragmentation software is very slow at determining where to place the files (even in their fastest defragmentation mode), when ever there is existing files in the nearby sectors they try to place the files to. There is plenty of space left (over 8.8TB), so the slow defragmentation rate is not because of low space, but because there is millions of files, ranging from 4KB to 2.2MB.

Few notes about the situation;

  • When I first used NTFS for this HDD, the disk usage was nearly constantly 100% and the NTFS filesystem was corrupted very fast within 1-4 days, according to check disk - even with a 64k Cluster size.

  • The StorJ app (essentially a decentralized cloud storage solution, which allows people to rent their HDD space to the people using this service as cloud storage) currently downloads/writes about 38gb worth of files in average daily, which amounts to over 100 000 files per day typically.

  • My current work around has been to consolidate all the files to the inner sectors of the disk, before trying to consolidate them on the faster sectors, which has had limited success. The problem is, that I can’t keep the StorJ server offline long enough to defragment the whole disk that way, without risking my server to be disqualified. And, that the metadata is too fragmented for the defragmentation software to place files fast enough among them.

  • Thus far I have tried O&O Defrag, Perfect Disk, Defragler (thus far seemingly the most responsive when moving files in between the metadata or other files), Ultimate Defrag (DEMO mode only free option), Ultradefrag, Disk SpeedUp (crashes, because it is only 32-bit) and Auslogics Disk Defrag 11 and 12 (both crash, only 32-bit versions available).

So, I have a few questions about this;

  1. Is there any way to defragment the ReFS metadata files, like it was possible to do with NTFS metadata on some defragmenting software, or am I out of luck with that?

  2. Is there any enterprise grade and/or open source defragmentation software that can handle tens of millions of files more efficiently?

  3. What would be the best alternative to ReFS in multimillion file server situations, that could be used in Windows 10/11 environment, defragmented or not even need defragmentation and setup with or without some tinkering?

The end goal would be to find a solution, that provides as fast access times / latencies to files as possible, without the urgent need to upgrade to NVME.

Bonus questions for future reference:

  1. Does NVME read and write latencies matter much in situations, where low latency to serve the files is essential or are they so fast that the impact of latency differences on NVME drives are not that important?

  2. What would be the best bang for a buck NVME drive to handle tens of millions of small (4KB - 2.2MB) files and hundreds of thousands writes per day?

  3. Is there any NVME with SATA connections that has at least nearly as fast access times / latencies as their M.2 counterparts?

Thank you for your help in advance :slight_smile:

2

I think you are troubleshooting entirely different class of problem than you should. If your setup experienced random fs corruption under heavy load, your setup need deep and detailed check, something is failing hard.

That never happens just due to load, there is hardware failure involved.

Isolate that first, then deal with minor issues like fragmentation.

fast access times / latencies to files as possible

Solution here is any ssd, not defrag of spinning rust. Look around for older enterprise sata ssds on the cheap.

Does NVME read and write latencies matter much in situations, where low latency to serve the files is essential or are they so fast that the impact of latency differences on NVME drives are not that important?

Almost any* ssd will outperform hdd, nvme or not, even if data is fragmented. I would wager the perf gap will widen in that case.

What would be the best bang for a buck NVME drive to handle tens of millions of small (4KB - 2.2MB) files and hundreds of thousands writes per day?

Any besides trash tier QLCs for writes. But even QLCs will outperform HDDs on read heavy tasks.

Is there any NVME with SATA connections that has at least nearly as fast access times / latencies as their M.2 counterparts?

No, they do not exist. Putting nvme drive behind sata interconnect would like putting trabant engine into brand new porsche. Waste of both performance and production cost for nonexistent market.

3

Make that Porsche. Now part of the VW Group.

4

Lots of things don’t add up here.

Please tell me they’re not all in the same directory.

SQL is a language, what is a SQL-esque file, and why are they getting fragmented? Are they getting lots of random writes? Does the (I’m assuming some sort of database) software not support CoW append or log-stuctured merge modes of updates?

That might well be expected depending on the workload.

That is a completely different issue that is not expected: This is hardware instability or a kernel-level driver which is corrupting memory. Fix that first.

There’s a nice trick to defrag any filesystem containing any number of files: copy them to a new filesystem. Robocopy to a second HDD, you get defrag for free. Then use the second HDD. On spinning rust this happens at linear read/write speed, which is orders of magnitude faster than random read/write speeds that defrag software would do.

2 Likes
5

I thought ReFS was not supposed to need defragmentation?
And just like ZFS, if you want to do a defrag on the files you basically just select all files and write to a new drive and you get the defrag in the process to consolidate the free space to a continuous block. AFAIK, ReFS has the same copy on write, dynamic block allocation, and built in scrubbing that ZFS has. Basically a “ZFS on Windows” type file system that keeps fragmentation low to begin with.