It seems we got a new toy to fiddle with and if its good enough for Linus to accept commits is good enough to me to start playing with it.
This file system has come to take on both ZFS and BTRFS and its written mostly by a lone wolf dude. The best kind of open source software. Seriously the code is quite good.
For me, personally, the draw is that it aims to solve a few problems not solved by any other file systems (not even ZFS):
Read/write cache (promote/foreground targets)
Elimination of the RAID write hole problem
Tiering compression and RAID levels: recognizing that compression and erasure coding is detrimental to high performance, bcachefs introduces the concept of compressing and erasure coding data written to background devices while metadata, read, and write caches remain uncompressed and mirrored for performance.
There’s also this:
btrfs doesn’t have this problem, but suffers from a few other dealbreakers.
I’m curious as to how this pans out in real workloads.
I learned about bcache some years ago, just before the guy started serious dev work on bcachefs. Looks indeed promising and as I’m a strong proponent of RAID6 I’m hoping bcachefs can handle RAID6.
If someone could write a shim that lets me zfs send from bcachefs to ZFS that would be amazing. I would love to try it out but don’t want to risk my real data set, and I don’t have any need for these filesystems outside of a NAS.
I’ll try to reinstall my fun laptop with bcachefs encrypted root. It uses 2 nvme as system drive. Downloading the steam library is normally a good way to test throughput. Will report disasters.
Installation on Nixos unstable went fine but had to change UUIDs to raw device paths. It seems systemd-boot has an issue finding the UUIDs of multi devices and/or encrypted volumes.
It’s raid1 encrypted. It doesn’t use LUKS though.
I’m on 6.8.1. Other than the weird way the array UUID is set (which makes other user land programs to not be able to see it) everything is alright so far.
I plan to eventually test this out [and might publish benchmarks here] once I have all my “new” hardware set up (see my username). Does anyone know if BCacheFS has an effective alternative to ZFS’s ARC, my reading of the documentation I could find suggests no. Personally I share concerns others (see Proposal: Consider adding warnings against using zfs native encryption along with send/recv in production · Issue #494 · openzfs/openzfs-docs · GitHub ) seem to have about newer ZFS features [ mainly block cloning and native encryption] and instability/corruption and I do see a future where for SSD’s where either https://www.purestorage.com/ or something akin to it is the path forward, there still needs to be a good alternative that is more generic.
Also does anyone know if there is anyone who has done work on making Bcachefs work with QAT as I did pick up a QAT device that I hope gets good use, right now I have a lot of ideas on how to use it mainly for ZFS, but also for other things.
I got the dreaded deadlock issue on fsck after my computer started to shutdown (for unrelated issues, suspect temperatures). FSCK wouldn’t work so I had to prepare a Nixos ISO with 6.9-RC5 so I could run the new FSCK code. It froze the computer while running it but it worked.
Extremely fast and low latency FS even though I was running full disk encryption (native not LUKS). Userland tools are kinda weird as the dm and systemd doesn’t recognize the array UUIDs, df shows the summed size of the array instead of available etc.
It was a good run. I’m reinstalling NixOS so everything should be back to normal. Only hassle will be to download the Steam library again.
I think I’m going to migrate from btrfs raid5 (yes I know read below about why I was only a little afraid to use this) to bcachefs raid5. I plan to make the switch because I want:
Better write speeds (i’ve had to move certain write-intensive jobs to another drive, and I’m hoping to move it back to my pool using write-caching)
native file encryption (so I can sell drives easily)
I’m a little scared though. Most of my data is… replaceable media, but there are photos and stuff that I wouldn’t like to lose. you might think that I’m moving from a provably unsafe FS to a largely untested one, so that’s an improvement in reliability, but I’m not sure. Maybe! The thing is the write hole in btrfs is probably over-blown. I’ve read that it basically only possible during a server crash, at at write-intensive moment, and even then is highly unlikely. So I used a UPS to greatly mitigate that risk. ATM, I’m a little concerned about the future of bcachefs in the Kernel! So I’ll back up the important stuff, and pull the trigger soon. I’ll let you all know how it goes!
Reading the openzfs issue tracker is indeed somewhat of a nightmare fuel.
Last I looked, bcachefs lacks anything like zfs’s send/recv though, which (along with the checksumming) is the main reason I use zfs. So that’s a bummer.
(The Replication mentioned as a feature of bcachefs doesn’t relate to replication of a file system or snapshot, but rather to redundancy of data blocks within a file system.)
Send and receive is in the Roadmap, but they talk about needing to implement a “network protocol” which is a bit disconcerting (only need stream to/from stdio like zfs). Makes me think whoever wrote that has never even used zfs send/recv.
Well, the list of things you need to do to add a drive to an existing btrfs raid5 array would include:
luksformat the drive, probably using a passkey file present on the root fs
add that file to the array using btrfs commands, that might be adding, or replacing a current drive
ensuring that luks automatically unlocks the drive without user intervention at boot
Then when it’s time to remove/sell a drive, then you’d have to
remove or swap in a new drive (btrfs commands)
destroy all keyslots on the old drive
That’s all doable, a couple of bash scripts could probably do it, but I’ve come to believe that Luks + BTRFS is really bad for performance, specifically write performance, and because I’m already having write performance issues, I’m probably not going to do that. additionally, it’d probably take me a while to nail-down the scripts described above.
For now, I’ll follow your advice and hold off on bcachefs on my FS for now and keep doing what I’ve been doing.
However, I’m considering using it on my desktop for an easy way to accelerate a large hdd. Maybe that’d be a better way to play with bcachefs for now. I plan to nuke and pave my old Manjaro install in favor of Nixos (which I’m now using on my laptop).
I’d strongly urge for a plan B and C if you had pinned your hopes on bcachefs.
bcachefs’s encryption could be moved to hardware encryption for performance (if you trust it). That takes over bcachefs’s encryption (which can only encrypt the entire file system anyway). Such a drive that is also T10-capable could also handle corruption (i.e., bit-rot) when pair with MD RAID or dm-raid. If you put a lean file system like ext4, you get three of the key features of bcachefs, which are integrity, recoverability (from corruption), and full disk encryption. With only one software layer, I imagine there is little to no performance degradation relative to putting the file system directly on top of the hardware block device. It’s also the setup I’m going to experiment with as bcachefs seems to be heading in a questionable direction.
What do you lose? Well… C.O.W., snapshots, compression, etc. But the beauty of having encryption, integrity, and RAID pushed down to hardware and MD RAID is that your file system choices are no longer contingent on having those features.
I agree about not using it on anything you care about but it’s funny because I’ve lost more data to btrfs than to bcachefs and bcachefs crapped out on me on a hard shutdown (I lost power). I managed to recover it with a fairly quick command found in the docs vs utter failure on btrfs (in fairness the btrfs failurtes were 5ish years ago, I haven’t used it on anything since then. zfs for stuff I care about in 3 places plus a borg backup of the stuff I really care about to another place for that stuff)
Yeah I mostly agree with that. But definitely don’t go with ext4. XFS is much better as it can give you some of the CoW features with reflink copies. If you for instance host VMs on that fs, being able to to reflink copy the raw VM disk to make a snapshot is just great.
Mainly because I expect people who develop file systems to not be normal people. Kent can be as weird and antisocial as he wants but his file system is the good stuff.
I’d be more worried about using something from LinkedIn brain developers to be honest.
Also my backup game is strong even with different geolocations (as in, I’ve managed to convince my parents overseas to run a server in the corner).
It’s fast even with the native encryption. I got clevis and tpm2 unlock with secure boot working.
Recently I’ve converted my main big file pool to bcachefs.
I absolutely love that I can set data_replicas=1 to folders containing stuff you can just fetch or rebuild again from the internet (in my case the bitcoin full chain and electrum index), caches, locally mirrored repos for my forgejo instance and so on.
You can literally choose a directory and tell it to not use mirror/raid logic on that one.
I’m also using a second pool of mixed sata ssd + nvme devices. Being able to mix different devices is cool.
No data loss so far. Back when I started this thread I had a split brain issue with a mirror array but an fsck fixed it. Back then I managed to recover the data and move away from it but eventually tried again. Since then it’s been smooth sail except for stuff that I did out of stupidity but thankfully managed to easily recover.
Considering switching my game pc to a similar configuration as I have a fast 1tb gen4 for OS and a slow 2tb gen3 for games. If I just throw everything on the same pool and set the gen4 devices as frontend it will definitely help and the storage on the gen4 won’t be wasted.
I still use XFS on single device machines out of simplicity.
Please do not take this as an endorsement as I’m an advanced user. DYOR but I’ve been loving it. But don’t brush the thing off because slopfluencers milked the kernel drama for clicks.