I’m just wondering, can one cobble together a zfs-like solution using it and md-raid, lvm, btrfs and bcache?
From what I can read in the documentation it looks like the mainline kernel basically has all of the primitives that could allow someone to basically build their own version of zfs.
I know there is a problem o reliability of this setup in case of a disk going offline for a second or two which zfs from my experience handles pretty reasonably. But from a perspective of just trying it out, it should cover all of the most fundamental features zfs has:
block level checksums handled by dm-integrity
raid levels and journal log for those handled by md-raid
creation of block devices handled by LVM
creation of “datasets” and individual quotas and snapshots handled by btrfs
I know that btrfs in this mix duplicates some of the functionality, because it can on its own support raid and block level checksums but from what I still hear on the internet raid 5/6 is buggy. I threw it into this mix because I think it’s gonna be more space efficient with snapshots than taking a snapshot of entire ext4 volume on LVM.
Mostly a thought experiment. I just find it interesting that the basic facilities are basically already in the kernel. Also, its something that on paper should work but I’m curious about all of the cursed problems I’m missing that would be caused by this setup…
That being said, I have some spare HDDs… I wonder if I’ll have too much free time this weekend.
Btrfs raid 5/6 is and has always been experimental with a large list of todos. The raid 1 raid0 and raid10 are ok and usable in production.
It seems more like nobody gaf about running btrfs in raid5/6 in the businness world, probably because zfs has that on lockdown
Its not even beta and its a piece of shit on multiple levels. The scrub logic for example is hilariosly inefficient for raid 5/6. It can take weeks to scrub an array zfs scrubs in days. The btrfs tools literally warn you and discourage its use when you create it.
Even on a single disk you get the nice things like compression, snapshots, etc.
Back up to a ZFS NAS with zfs send, etc.
Even on a single drive if you’re worried about bad sectors wiping out individual files, you can do copies = N on a single filesystem (which could be limited to a small number of important files) so that there’s a better chance of recovering data in tact with small numbers of bad sectors. There will be a write performance penalty of course but sometimes data is important…
Hmm… from the doc I linked it seems like dm integrity and dm crypt have some cooperation implemented between them, should LUKS be directly atop integrity or on top of LVM?
Going with the first option means that I won’t leak any metadata, taking a snapshot of encrypted volume gives hints as to what regions were changed.
But in the second options I get multiple, separate LUKS volumes that can have different encryption settings and keys
Ah yes, I was once trying to recover btrfs from a disk that started shitting itself, the recovery tools couldn’t fix or recover anything but they ran for a few days on a single disk
I’m not sure this is accurate in regards to btrfs as a filesystem itself. I think facebook is running it on their servers and SUSE defaults to it for the system partition.
Yes and no, the built in encryption support, caching devices and raid{5,6,7} are painfully missing from btrfs… that’s why I came up with the idea of this cursed setup.
Zfs encrypts at dataset level, so it works like your second option.
Btw i just remembered that RedHat is trying to do a similar thing to what you are attempting with the Stratis project. They have been working at it for like 10 years and i have no idea what % of feature parity they are at, but it is still going https://www.phoronix.com/news/Stratis-Storage-3.9-Released
I thought this was a nice write up of a comparison between ZFS and alternatives:
Conclusion from the article:
Final notes
With mdadm you can setup any kind of RAID pool for disk redundancy and you can even expand with further disks if later needed. Then you can add dm-integrity for error detection and error correction at the block level. If you add dm-crypt+LUKS to that the data integrity is protected with native authenticated encryption. And if you use the RAID pool as LVM physical volumes you can do snapshots too. But it cannot by a long shot compare to ZFS or Btrfs!
And I would never use something like mdadm+dm-integrity as a replacement for ZFS or Btrfs.
It is true that you don’t have to worry about compiling a kernel module every time the kernel is updated (as with ZFS on Linux), or using annoying Solaris compatibility layers, or to run with an external and un-official and often out-of-sync repository (Arch Linux developers, yes I’m talking to you! We need ZFS in the Arch Linux official repositories ASAP! Even Debian has ZFS in its repositories!). And you don’t need to worry about any “write hole” issues as with Btrfs.
However, not only do you need to carefully study the documentation of each piece of technology you put together with mdadm and make sure you understand how you put these things together and how you best deal with potential problems, but you’re still limited by the “regular” filesystem you put on top of all that, and you don’t get any of really well designed and superior protection or management that ZFS or Btrfs provides.
I know, but it also takes care of the metadata afaik. In this setup I’m considering if the metdata that can be leaked by snapshots is really that much of an issue, I’m not a crypto expert so I’m just thinking out loud in hopes that someone more competent in this topic will chime in.
“metadata” is generic and can mean anything, the man of zfs encryption tools spells out what is and what is not encrypted by zfs. So we know what we are actually talking about here.
ZFS will encrypt file and volume data, file attributes, ACLs, permission bits, directory listings, FUID mappings, and userused/groupused data. ZFS will not encrypt metadata related to the pool structure, including dataset and snapshot names, dataset hierarchy, properties, file size, file holes, and deduplication tables (though the deduplicated data itself is encrypted).
This is imho comparable to having a LUKS volume inside a LVM.
As a veteran of the btrfs vs ZFS wars myself, on btrfs side at the time, I’ll tell you he is out of line, but he is right.
Btrfs is usable and ok and can do a bunch of cool stuff as long as you stay the hell away from experimental raid 5/6 and don’t use a lot the still very squirrely quotas feature, but we were promised a lot more than that back in the day, a true zfs competitor. The hype was real. A lot of people (me included) are still hurt in their feels about the lack of raid 5/6 support and lack of native encryption (this is supposed to come eventually through the FSCRYPT kernel feature btw, so it’s not a “blatant layering violation” or something like zfs).
While we watched zfs get ported over to linux and get feature after feature after feature. btrfs TODAY has less features than zfs had when the “next gen filesystem wars” started (which is fair, considering zfs codebase was made a while back and was already in production on solaris by then). It has some edges somewhere but overall it’s inferior.
The mail linked in the article is peak LKML.
The bachefs dev starts drama (as usual) and a facebook engineer comes in and drops some info about facebook use of btrfs to own him and then says pearls like
Emails like this are why a majority of the community filters your emails to
/dev/null.
