I’m planning out a small home storage project for “fun” purely out of an itch to mess around with tiering, and different types and speeds of storage. I’d like some general guidance and input on different approaches to storage tiering, my mind mainly focused on open-source solutions.
I want to create an unreasonably complex storage configuration involving a plethora of different types and speeds of storage, encroaching on Rube Goldberg machine territory. What I have in mind would be something like:
RAM
Optane(writes)
NVMe SSD
SATA SSD
“Fast” Spinning Rust(7200RPM?)
“Slow” Spinning Rust(5400RPM?)
where whatever solution I use would move data based on usage to different tiers such as “hot, warm, cold”, etc. excluding caching layers.
Outside of this project I currently have a main OpenZFS storage server with spinning rust(RaidZ3) on Linux(Fedora) that has optane(SLOG) and nvme(L2ARC) set up working very well, along with a backup server running mergerFS with SnapRAID. With these and other projects I’ve worked on I’m pretty comfortable trying and tinkering with different storage solutions that are out there.
That said, I’m having difficulty finding a feasible software stack to build this type of configuration. ZFS has some great built-in “tiering”, but once I hit the SATA SSD and HDD layers, I haven’t really found a good solution to handle “tiering” those layers. It looks like bcachefs despite the project’s turbulence, has(had) some good roadmap items to do kind of what I’m talking about, but isn’t quite there yet. I found the autotier project that 45drives had created, but it doesn’t look like it has been maintained for some time.
I’m debating just going with ZFS to manage the storage pools and then throwing mergerFS on top to string them together, using some scripting in the background to move files between the different tiers, but obviously an automated solution or even better a block-level solution would be ideal.
I’m not solving for any specific use case in terms of what storage needs I have for an application or similar; this is purely a home lab project just for fun where I have a possibly unreasonable itch to build something like this and play around with it. Any input would be appreciated.
I saw a write up on Ceph where they did this sort of thing using RGW and some lua scripting. If you want overly complicated storage adventures, Ceph is a good bet.
If you haven’t invested in Spinning Rust yet, I’d say… Just let it go. Tape is still cheaper bulk storage, more durable, and as soon as production can meet demand, SSD storage prices will collapse - it is only a matter of time now.
By 2028 you should be easily able to get a 256TB E3.S drive for ~$2.5k-3.0k. Hard drives simply cannot compete with those prices.
One half-solution is a ZFS Z Device. This let’s you keep all metadata and optionally small data blocks on a separate vdev. Probably the best speed up you can get for a spinning rust pool, in lieu of upgrading everything to SSDs.
LVM has a feature called lvmcache that may work for this. Or you could manually make a script that issues pvmove to blocks when they’re hot or cold, maybe with an EBPF script to track hot/cold blocks? Lots of options there. Post about lvmcache. Six Months of lvmcache on My Desktop - Patshead.com Blog
This is from my gaming pc. I couldn’t sleep last night. And I recently realized recently that I’m at a place in life where I should just get all the dumb crap I want in life so I bought a 5090 and 64GB ddr5-6000/30. So I decided to play with PrimoCache, it’s fine for what it is and at $30 it was money chasing away my boredom well spent. So this sequential score was from a WD blue HDD with 4000MB tier 1 cache of ddr5 ram, 13gb of intel optane as an l2 cache. It performed much worse in a longer test- I moved 285GB of random junk and got more like 250-600MB/s, it’s ultimately still a spinner and that’s gonna be a Hangup. But if you want to nerd out on storage you can. My home NAS is 8 HDDs in 2 RAIDZ2 vdevs with a mirrored 32gb Intel optane m10 L2 cache, 64GB ddr4 ECC ARC cache, 2 mirrored WD BLUE SATA special vdevs for metadata- it’s almost all full of movies lol. Hardware is the fun part. But be learning how to bend it to your will is the satisfying part.
