FIrst Time Building Solid State Drive RAID Arrays

Hey folks,

I’ve just wrapped up a first‑time adventure building an SSD RAID array, and I wanted to share the lessons I picked up along the way. If you’re thinking about moving from spinning disks to flash, hopefully my story will help you avoid a few headaches or you will just get a laugh reading about someone trying to figure this out!


My Experience Up To This Point

I’ve been running RAID‑5 (sometimes ZFS’s RAIDZ1) on a bunch of HDDs for years in my homelab. It works fine for bulk storage, but the latency is obvious when I’m juggling VMs and containers.

I’ve used plenty of larger storage arrays at work but this time I was actually building the system myself and frankly I thought I knew what I was doing from my experience with RAID arrays with spinning disks, but there are some gotchas that I was not aware of.


Needed A Machine With Real Performance

I’m working on a project that deals with containers and virtual machines. To get serious performance I bought a PowerEdge or two with:

  • 2 × 20‑core CPUs
  • 256 GB RAM

I needed a fast, reasonably sized pool, so I ordered three consumer‑grade SATA SSDs from Newegg—just enough for a RAID‑5 layout.


First Lesson – RAID5 & SSDs Don’t Mix Well

Almost immediately I hit the wall: SSDs in RAID‑5 suffer from write amplification. The parity calculations force a lot of extra writes, and the drives’ internal wear‑leveling gets hammered. My solution? Add a fourth SSD and switch to RAID‑10, which should be friendlier to flash.


The Problems Persisted

Even with RAID‑10 the system felt sluggish. Simple dd or fio tests showed decent sequential write speeds, yet under real workloads the machine would intermittently crawl.

  • iostat showed w_wait spikes into the hundreds of milliseconds.
  • btop’s IO% hovered at 100 % almost constantly.

What was going on?


Second Lesson – Not All SSDs Are Created Equal

I discovered I’d bought consumer, DRAM‑less SATA drives. Those models rely on a tiny SLC cache and an HMB (Host Memory Buffer) that lives in the system RAM. When the cache fills—as it does with the random read/write patterns typical of containers—the drive falls back to its slower NAND, and latency explodes.


The (Hopefully) Final Upgrade – Enterprise‑class SAS SSDs

I replaced the consumer drives with Samsung enterprise SAS SSDs (with dedicated DRAM). The change was night‑and‑day:

Metric Before (DRAM‑less SATA) After (Enterprise SAS)
Avg. write latency 150 ms (spikes > 300 ms) 0.3 ms – 0.5 ms
Random read IOPS ~100s IOPS ~12.5 k IOPS
Overall system responsiveness (btop IO%) 100 % (blocked) < 10 % (idle)

Now my VMs and containers spin up instantly, and the storage pool can sustain the mixed‑workload traffic without choking.


Closing Thoughts

Building this array was a steep learning curve, but it reinforced a core principle: flash storage is a different animal than spinning disks. The right hardware (DRAM‑backed enterprise SSDs) combined with an appropriate RAID level (RAID‑10) turned my “high‑performance” machine into the fast, reliable platform I needed for virtualization and container workloads.

A couple other notes if anyone else is thinking of doing something like this. SAS, unlike SATA, allows for parallel read and write which also improves IOPs. Ensure your storage controller is not a bottleneck, my machine has a pretty low end controller so I flip it in HBA mode and handle RAID with software.

I made some mistakes along the way but hopefully I learned something while completing this build. I had fun with it anyway.

Thanks for reading my story!


Feel free to ask any follow‑up questions or share your own SSD‑RAID experiences. I’m still tweaking the setup (tuning ZFS recordsize, experimenting with L2ARC), so any tips are welcome. If anyone has any resources for folks like me to learn more about the different things to think about when building high speed storage that would be great!

Why SAS instead of NVME?

Spinning disks also have their own issues, like CMR vs SMR.

SATA is not fast. RAID5 is not fast.

Write amplification has less to do with RAID vs. mirror and more with volblocksize.
Sure pool geometry and padding are also a thing, but changing a 16 volblock in 3 wide RAIDZ1 does write 24k in total, while in a striped two way mirror writes 32k in total.

DRAM backed is not exclusive to enterprise. And there are many consumer drives that are faster (at the same price, in terms of seq read write) than enterprise drives. Main advantage of enterprise drives is PLP which helps speed up sync writes, and sync writes only.

In general, good advice is this: don’t!
Defaults are pretty great. Also recordsize is probably not even what you are looking for. If you use RAW ZFS drives and not QCOW (which you should not use, else you have CoW on top of CoW), they are stored in zvols with a volblocksize setting. Record size only applies to datasets.

1 Like

Beginner’s Guide to ZFS:

  • Don’t use block storage for files! Instead, use datasets for files.
  • Separate your files from VM data.
  • Your VM data is stored in zvols and block storage. For block storage, use NVMe mirrors (or at least SSDs).
  • Don’t use RAIDZ1.
  • Don’t use RAIDZ2 for block storage.
  • Use RAIDZ2 only for large files that are sequentially read and written.
  • Max out your RAM and set the zfs_arc_max value, so you don’t have empty RAM.
  • If you need a special vdev, add it before moving data to your pool.
  • Don’t use L2ARC or SLOG from the start. There’s a high chance your workload won’t need them. If needed, you can still add them later.
  • SLOG speeds up sync writes only, never async writes. Async writes are “chached” in 5s transaction groups in RAM already.
  • Don’t use CoW on top of CoW by either putting qcow in a zvol or using zfs in a VM guest.
1 Like

Just because I got a good deal on the SAS drives :slight_smile: