Best Filesystem For Drive(s) With Bad Blocks? To Keep Using My Half-Dead Drives!

[Nerdgasm]

I’m looking into which filesystem is going to be best to use with drives that are known to have bad blocks and could have more blocks fail. There’s a lot of filesystems out there. While I know a decent amount about some of them, there’s no way I would know enough about all of them. I’m okay with this discussion taking a long time as I am in no rush.
Things to keep in mind:

1. This is for HDDs, not SSDs

2. Drives would have bad block scans and space tests performed before formatting

3. Drives would be freshly formatted with this filesystem

4. No data put onto these drives is mission critical and can be lost

5. Doesn’t need to be specific to any Operating System.

6. Consider all the factors around resiliency and performance. Everybody may weigh every factor differently and this is okay!

Why I want the information:
I have many old HDDs that have bad sectors on them, but are still able to be formatted and used. I want to put them to use until they are actually dead in one way or another. I used to do some BURST coin mining years ago on drives like this and it worked out well enough. I think BURST has now changed to Sigma or something like that. Chia operates on the same principle but came up later and stole the show.

I might consider an experimental setup for a NAS, doing a sort of JBOD setup with a couple new larger drives to act as parity along with versioning and recovery, but have the primary storage of it be a bunch of these half-dead drives.
This is all for fun, don’t bother convincing me to just get rid of the drives. Other fun use suggestions would be cool, but the focus of this thread is to narrow down the filesystem to use.
I’ve attached a picture of most of the drives I would be using. It’s not quite all of them, but it is most of them. I figured some of you would be curious.
This post also marks my desire to start actively participating in the forum so hurray!

PS. I have this same thread on the LTT Forums, but haven’t gotten much traction. I figure that Level1Techs forum might be a bit more specialized on this topic so I probably should have posted here first.

[Hydrosaure]

ZFS with pools configured with redundancy (RAIDZ) and possibly also set copies=2 on filesystem for extra protection

[twin_savage]

I would think ZFS would be a very poor file system for this use case primarily because of the different sized drives. Also the likely more frequent resilvering needs are going to push even more drives over the edge, possibly resulting in a kind of cascading failure.

I don’t think its the file system you want to enable this use case but some kind of erasure coding scheme or perhaps unraid (I can’t believe I’m recommending it because it is awful imo).

[mith]

I keep a zfs raid running on some bad drives that are over 15 years old. They are so old that they used to run in an opensolaris server. Its been interesting to see how ZFS handles all the bad blocks. ZFS has never puked on these drives. Its losing data, of course, since the number of failing drives exceeds the parity, but the damage is still limited to just a few files. Incredible stuff. It still mounts and I can access data that hasn’t been permanently lost. If you run the status command with -v it actually lists the files with errors.

        NAME                                    STATE     READ WRITE CKSUM
        zspace                                  DEGRADED     0     0     0
          raidz1-0                              DEGRADED   198     0     0
            ata-SAMSUNG_HD103UJ_S13PJ9xxx  DEGRADED   217     0     0  too many errors
            ata-SAMSUNG_HD103UJ_S13PJ9xxx8  DEGRADED     0     0     0  too many errors
            5247442656480345445                 UNAVAIL      0     0     0  was /dev/disk/by-id/ata-SAMSUNG_HD103UJ_S13PJ9xxx-part1
            ata-SAMSUNG_HD103UJ_S13PJ9xxx ONLINE       0     0     0
            ata-SAMSUNG_HD103UJ_S13PJ9xxx  ONLINE       0     0     0
            ata-SAMSUNG_HD103UJ_S13PJ9xxx  ONLINE       0     0     0

errors: 5 data errors, use '-v' for a list

[adman-c]

What about BTRFS? I would otherwise hesitate to recommend it, but for this use case it might be perfect. You can use its version of RAID1 and pile a whole bunch of different sized disks in a single pool. As long as no single disk is bigger than all the other disks combined (I believe, it’s been a while since I used it), it uses all of the available space on the disks. See here for a BTRFS space calculator. You could even increase the number of copies it keeps to 3, which would be the approximate equivalent of a 3-way mirror. Like ZFS it’s COW and uses checksums to detect data errors/bitrot and correct them.

In any case, this seems like a great experiment! Looking forward to seeing the results!

[jode]

This is probably not an uncommon, but certainly special use case for HDDs.

Since the use case is defined by use of failing drives I recommend starting the discussion with ways how drives can fail and then designing a solution that takes those into account.

I think (from experience) the most common way for drives to fail is developing bad blocks, meaning some blocks are for some reason no longer accessible by the drive. HDDs are designed to deal with this situation by having spare space set aside and firmware that will map out bad blocks and “swap” in spare blocks.
There are ways to control this using SATA drives (e.g. linux - How to use hdparm to fix a pending sector? - Server Fault), I think SAS drives are designed to do this transparently (please correct me if wrong).

Other ways to fail include firmware bugs that eventually run amok (Seagate had some well known ones in their series of 3TB consumer drives).

Of course, there can be catastrophic failures, e.g. issues to mechanics, heads or disks. These could be caused by sudden power loss (e.g. HDDs doesn’t have time to park heads). In this case the solution is to a) have a backup of data, b) remove/replace failed HDD.

For convenience it is desirable to have a single filesystem spanning multiple (potentially all) HDDs. On the other hand, an issue on a single HDD may corrupt the file system that holds data on healthy HDDs.
Given, that any HDD may fail in small (bad blocks) or large ways at any time you will want to create a system with redundancy that can absorb failing HDDs.

The other issue to consider is that as HDDs age and failures occur, it may take some time until you notice the issue.

Taken all of these things into account, I’d recommend using a ZFS filesystem, because it allows creating a single pool of differently sized HDDs, its checksumming allows detecting issues on the block level as part of regularly executed scrub operations.
The drawback of having a single pool, of course, is that you need to be able to physically connect all HDDs and power all of them up, even in case you only want to access a targeted small amount of data. It’s obviously possible to create several smaller pools to accommodate this.

I would organize the pool into a set of mirrors for two reasons. First, it allows creating mirrors of different sizes (just pair up same-sized drives), and secondly, a pool of mirrors can be altered (add and remove drives). The same is not possible with raidz vdevs.

Depending on your willingness to take risks or your knowledge of the state of individual HDDs you can create 2-way, 3-way or higher level mirrors for added redundancy.
In case of a drive failure, not all drives in the pool are affected, but only the drives in the same mirror.

Edit: enhance the section discussing pro/cons of single pool layout

[MSMSMSM]

This is something I’ve actually wanted to try myself but I’m experiencing RMA hell and a ton of regret buying stuff to the degree that I don’t want to spend a dime for a long time…

But, if I were to build something like this, I’d do something like this:

• Do the usual bad block and space scans as you’ve mentioned

• Use a disk burn in script like Spearfoot/disk-burnin-and-testing to separate the wheat from the chaff.

  It’d be nice to separate the drives who are next in line to meet the Grim Reaper than ones who are maybe, a couple dozen months away.

• Sort the drives by size and make sure you’ve gotten yourself enough drives of one capacity to make an array of say, for example, five drives.

  You can mix and match drives of larger capacities but IIRC, you’re limited to the size of the lowest capacity drive in the array.

  Since you have plenty of drives, I can assume that you’d be able to sort your drives so that you have the least amount of space lost to mixing and matching.

Assuming you’ve got all that sorted out, now it depends how you plan on using the system. If the drives are in a NAS-like standalone system, then ZFS (configurable with TrueNAS SCALE) is the way to go.

If you’re using the system itself and don’t want to worry about having to install ZFS when distro-hopping, then LVM2+[a standard Linux filesystem of your choice] (configurable with OpenMediaVault) might be the way to go*.

EDIT: I’m partial towards RAID1 and RAID10 personally, I’m not sure if RAID5/6 is viable for such a setup

* - I’d still go with TrueNAS Scale and ZFS if I were you

[jode]

Any mirror technology would work, of course (mdadm, LVM, btrfs, …). However, zfs has another trick up its sleeve in that in case of an issue it only has to resilver occupied data blocks as opposed to traditional raid1 solutions that will copy all blocks.

The benefits of a zfs resilver are a) it completes faster, and b) introduces less load on the already wounded drives.

[TimHolus]

@Nerdgasm

Other fun use suggestions would be cool, but the focus of this thread is to narrow down the filesystem to use.

I don’t know if in any special sensational way some particular FS will change something to the plus here.

Scan the surface and determine which sectors are bad or slow. Then create the partitions so that they are not on and near these sectors. But if a drive has progressive bad sectors, it’s only a matter of time before it shows up somewhere within a partition. Even if FS manages to some extent, it will affect performance and at some point it will become unusable. imho

P.S
I had many hdds that were almost dead, I could start and initialize them and query the controller but the number of bad sectors and their constant progression meant that the responsiveness of the disks was at the level of unusability … from what I remember it was about 3k bad sectors and it was growing, ZFS or nothing else here it wouldn’t help.

[twin_savage]

This can be a double edged sword too, when the vdevs are more empty or not fragmented then resilvering is more performant over traditional raid rebuilds, but if the vdevs are more full or especially if fragmented then the resilvers can be significantly more stressful/slow than traditional raid rebuilds. Same paradigm would apply to scrubbing as well.

I had just assumed the drives would be filled to the brim because burst and chia were mentioned. Although if that is the only thing stored on the disks, fragmentation might not be a very prominent issue.

[Peter_Blanchard]

It’s not a file system per se but Greyhole might be of interest. Sits on top of of file system It’s not tiered storage, nor is it RAID (although it will work with/beside RAID arrays). Basically, it does drive concatenation and file duplication (with some very basic rules) but with some lateral thinking you might get something genuinely useful.

Don’t expect stellar performance though.

[Nerdgasm]

Thank you all for responding so far! I ended up getting quite busy the last few days so I couldn’t take the time to fully respond to reach of you until now.

@Hydrosaure acknowledged
@twin_savage Good point!
@mith Fantastic real world experience! Quite useful to refer back to.
@adman-c This is one of the reasons I asked this question. To learn more about it all! I run BTRFS as my main filesystem, but didn’t think about this usage. I’m leaning away from pooling them, but still this is a consideration.
@jode The drives I have ad have failed for any number of reasons. I have tested and filtered out the ones that would not be usable for this purpose. So the only ones we are talking about here are bad blocks. Some with far more than others. They may have some level of mechanical issue, but not enough to stop functioning… Yet. Once again another vote for ZFS, with some good reasoning and some appreciated setup information.
@MSMSMSM The script you mentioned look good. I’ve been doing those tests manually myself. It hasn’t been too big of an issue because they are in my terminal history so I just press up a couple times and maybe adjust a letter. The idea of LVM2+whatever is pretty interesting. I acknowledge your RAID preference and vote for TrueNAS Scale with ZFS.
@jode Good note on the benefit ZFS
@TimHolus Totally get and agree with where you are coming from. I’m just looking for the best way to have these set up to run them into the ground properly. I’m not trying to kill them, but I want them actively used until they die.
@twin_savage I haven’t used Chia or what Burst has turned into, but they were high up on my potential use case for them as I had been somewhat successful (didn’t get anything, but no errors along the way) doing Burst mining on drives like these before. I know with Burst in the past I was able to create smaller plot files, so I could possibly spit the plots into a bunch of smaller pieces, allowing for some plots to get corrupted, just unassign those, and keep using the rest of the drive. Not 100% going down that route, but just some brainstorming for sure.
@Peter_Blanchard Making note of Grayhole and will be looking into it further.

A bit more detail on what has been my process in the past:
If I know the drive is damaged I start at step 1. If I don’t know if the drive is damaged I run step 2 then go back to step 1 before continuing up the steps.

1. Connect to older computer that has a BIOS option for SATA to run in IDE Mode directly with SATA. Run HDD Regenerator to try and repair any bad blocks. (HDD Regenerator recommends IDE mode for highest likelihood of success)
2. Connect to Linux computer via SATA to USB and run badblocks on it
3. Use f3probe destructive size check both wipe it and confirm it can be fully written to.
4. Run SMART Conveyance test if available
5. Run SMART Extended test
6. Format with desired filesystem

Initially I wasn’t wanting to pool them all together, but more and more I lean towards pooling. If anybody wants to respond further and give more suggestions I am all ears! I was expecting more debate over filesystems, but maybe I shouldn’t be surprised that most paths lead to ZFS. 4 votes for ZFS, 1 vote for BTRFS, and (in LTT forum) 1 vote for RFS.
I’d love to hear some crazy ideas of why some random other filesystem would be good.

[Gnuuser]

All the above are good suggestions and disk mapping can flag bad sectors for the os to ignore, buy once bad sectors start growing it wont be long before the disk is totally unuseable.
But even a dead drive still has uses.

Dissassemble the drive and make wind chimes out of the platters( if they are aluminum you can also engrave images or poetry on the disk)

Be careful drilling them! Lappy drive platters are often made of glass

The magnets are very powerful and you can find lots of uses for them.
The bodies are machined aluminum and easily recycled

For ide drives i use d-ban to wipe and overwrite.
This not only forensically clears the drive but refreshes the magnetic flux field in the blocks.

However if it fails d-bans brute force testing ,the drive gets scrapped.