Hey folks,
I’ve just wrapped up a first‑time adventure building an SSD RAID array, and I wanted to share the lessons I picked up along the way. If you’re thinking about moving from spinning disks to flash, hopefully my story will help you avoid a few headaches or you will just get a laugh reading about someone trying to figure this out!
My Experience Up To This Point
I’ve been running RAID‑5 (sometimes ZFS’s RAIDZ1) on a bunch of HDDs for years in my homelab. It works fine for bulk storage, but the latency is obvious when I’m juggling VMs and containers.
I’ve used plenty of larger storage arrays at work but this time I was actually building the system myself and frankly I thought I knew what I was doing from my experience with RAID arrays with spinning disks, but there are some gotchas that I was not aware of.
Needed A Machine With Real Performance
I’m working on a project that deals with containers and virtual machines. To get serious performance I bought a PowerEdge or two with:
- 2 × 20‑core CPUs
- 256 GB RAM
I needed a fast, reasonably sized pool, so I ordered three consumer‑grade SATA SSDs from Newegg—just enough for a RAID‑5 layout.
First Lesson – RAID5 & SSDs Don’t Mix Well
Almost immediately I hit the wall: SSDs in RAID‑5 suffer from write amplification. The parity calculations force a lot of extra writes, and the drives’ internal wear‑leveling gets hammered. My solution? Add a fourth SSD and switch to RAID‑10, which should be friendlier to flash.
The Problems Persisted
Even with RAID‑10 the system felt sluggish. Simple dd or fio tests showed decent sequential write speeds, yet under real workloads the machine would intermittently crawl.
iostatshowed w_wait spikes into the hundreds of milliseconds.btop’s IO% hovered at 100 % almost constantly.
What was going on?
Second Lesson – Not All SSDs Are Created Equal
I discovered I’d bought consumer, DRAM‑less SATA drives. Those models rely on a tiny SLC cache and an HMB (Host Memory Buffer) that lives in the system RAM. When the cache fills—as it does with the random read/write patterns typical of containers—the drive falls back to its slower NAND, and latency explodes.
The (Hopefully) Final Upgrade – Enterprise‑class SAS SSDs
I replaced the consumer drives with Samsung enterprise SAS SSDs (with dedicated DRAM). The change was night‑and‑day:
| Metric | Before (DRAM‑less SATA) | After (Enterprise SAS) |
|---|---|---|
| Avg. write latency | 150 ms (spikes > 300 ms) | 0.3 ms – 0.5 ms |
| Random read IOPS | ~100s IOPS | ~12.5 k IOPS |
| Overall system responsiveness (btop IO%) | 100 % (blocked) | < 10 % (idle) |
Now my VMs and containers spin up instantly, and the storage pool can sustain the mixed‑workload traffic without choking.
Closing Thoughts
Building this array was a steep learning curve, but it reinforced a core principle: flash storage is a different animal than spinning disks. The right hardware (DRAM‑backed enterprise SSDs) combined with an appropriate RAID level (RAID‑10) turned my “high‑performance” machine into the fast, reliable platform I needed for virtualization and container workloads.
A couple other notes if anyone else is thinking of doing something like this. SAS, unlike SATA, allows for parallel read and write which also improves IOPs. Ensure your storage controller is not a bottleneck, my machine has a pretty low end controller so I flip it in HBA mode and handle RAID with software.
I made some mistakes along the way but hopefully I learned something while completing this build. I had fun with it anyway.
Thanks for reading my story!
Feel free to ask any follow‑up questions or share your own SSD‑RAID experiences. I’m still tweaking the setup (tuning ZFS recordsize, experimenting with L2ARC), so any tips are welcome. If anyone has any resources for folks like me to learn more about the different things to think about when building high speed storage that would be great!