Hello from Ukraine - Software Developer and Data Storage Researcher

Communities
1

Hi! I’m a software developer from Ukraine. I’ve been programming for over 25 years, and for the last 15+ years I’ve been working on data recovery technologies at Hetman Software.

My main interests are RAID, NAS systems, file systems, encryption, virtualization, and low-level storage analysis.

One of the things I particularly enjoy is experimenting with real hardware. In our lab, we work with Dell PowerEdge servers, various hardware RAID controllers, Synology and other NAS devices, as well as Proxmox, TrueNAS, FreeBSD, and Linux.

We regularly build test environments, fill them with data, and deliberately break things to see what happens. For example, interrupting RAID expansion, removing disks from an array, deleting encrypted partitions, or damaging virtual machine storage.

Then we investigate what’s left on the disks, how the storage structures work, and whether the data can still be reconstructed. Not every experiment succeeds, but that’s part of what makes it interesting.

If you have an unusual RAID, NAS, ZFS, or virtualization failure scenario you’d like to see tested, I’d be happy to hear about it.

5 Likes
2

Slava Ukrain! I’m using a box-standard 6-drive RAID6 on Linux (mdadm) with an LSI-9300 series HBA so nothing special here. Probably safe to say, if you value your data, keep yandex out of your IT chain :fire: :rofl:

:netherlands: :heart_hands: :ukraine:

1 Like
3

Heroyam Slava! That’s a solid and reliable setup! Are you using it as a file server?

4

As a NAS/backup, basically. It has a workstation mainboard so IPMI available and I use it to power the machine on remotely. That way I don’t have to go downstairs to push a button, which is a good thing as the old big tower no longer has a front panel, hence no power button either :stuck_out_tongue:

Board is an Asrock Rack X470D4, AMD R7 1700 (non-X) CPU, 64 GB RAM (when it was “cheap“ :anguished_face: ) and aforementioned LSI-9300 series HBA. Drives are refurbished enterprise 16TB from various manufacturers, that way any 2 drives can fail and still retain my data.

I do have some EPYC server systems, but these are offline due to power and noise, I’m contemplating putting them on for the upcoming LTT folding month (november). Not committed yet. (2x 7551P+128GB ECC RAM each, one 3151 with 64GB ECC RAM)

5

Enterprise drives are generally quite reliable, especially when running continuously. Frequent power cycles can introduce additional wear, so keeping them running 24/7 often makes sense.

Overall, RAID6 provides good protection against one or even two drive failures.

In my experience, one of the more challenging situations comes when you need to expand the array, either by adding more drives or replacing existing ones with larger-capacity models. RAID reshaping and rebuilding can be particularly demanding on large arrays.

Have you ever had to expand an mdadm RAID6 array, or are you still running the original configuration?

6

I started out with 4x 500GB drives, back in the late 2000’s. Then upgraded to 4x 2TB, next was 4x 4TB. Every time I swapped out 1 drive at a time, let it re-silver then after the final drive, expand the filesystem to the full capacity. The current RAID was build from scratch, I had a period where a single 8TB drive was my “backup“ (money was tight back then). When finances allowed, said 8TB drive was full, so I switched to 16TB drives as the difference in price back then wasn’t that big.

I did experiment a few years back with some low capacity SATA SSD’s I got for essentially free. I had enough to make a RAID5 out of RAID6’s. So a RAID56 :light_bulb: Did some quick testing, forgot the results as it’s unlikely I’d be building something like that for actual “production“ use.

7

That’s a really interesting setup! I wasn’t even aware that you could build a RAID5 on top of multiple RAID6 arrays. RAID56 is a new one for me! :slight_smile:

Did you use mdadm for both RAID levels? And do you happen to remember or still have any benchmark results from your testing? I’d be curious to see how such a configuration performed.

8

Yes, RAID can be nested and mdadm was used for all arrays, including the top RAID5. Testing was rudimentary and not persistent, hence no results. I do recall I wasn’t overly impressed, but as a “just for the sake of it“ it was a fun thing to do. I doubt there’s a practical use case.

If you’re not familiar with them yet, look up LTT on YT. They’ve lost data before. Twice :roll_eyes: It’s already (quite) a few years back. Please be advised that LTT is primarily “info-tainment“ not so much geek-tech (like L1T) although you’ll find a surprising overlap in audience between the two channels :wink:

9

Depends on your use case what RAID method you use. Once the number of drives and TB gets up there, storage cluster and object storage becomes easier to manager.

I migrated my home storage to a Ceph cluster with some Dell R710s and PowerVault shelves (in it’s current form, it started off much smaller). I like having the ability to add or remove drives at anytime and the cluster just expands and shrinks as needed.

I also did a durability test on the cluster (in it’s early days) by removing every storage drive in every storage server (HAL9000 style), while the cluster was online and I was copying data to it. The data copy did stop, but after I reinserted the drives and rebooted the storage servers, it recovered in a few minutes and the file copy resumed. No issues.