ZFS HDD Setup - What pool configuration for max speed?

Hi, I have 8x14TB drives, they get about 210MB/s R/W. What setup should I be going with if I am looking for maximum speed?

I will be running TrueNas, thanks in advance.

I edited your post title to be more descriptive.


But on to your question, the absolute fastest ZFS configuration is striped vdevs, which is the equivalent of raid 0. This configuration would give you both the fastest speed and the largest amount of usable storage. However, this configuration means that if any of the disks fail then ALL of the data is gone and also there is no data to recover bitrot. So this is NOT recommended in almost all circumstances, especially with a large number of drives.

If you still want good speed, but also redundency, then striped mirrored vdevs is the best option, which is like raid 10. It is fast, and does have redundancy so it can tolerate at least one drive failure. However, you only are able to use %50 of the total space on the drives.

For good space efficiency while also having redundancy, look into RAIDZ, but this is slower then the other options.

4 Likes

generally:

adding more vdevs into your pool usually improves the IOPs your pool can do

more drives in a vdev help with sequential performance (to a point)

5 Likes

56TB is more than enough for me so i will go with striped mirrored vdevs.
With that said, how many vdevs should i have and how many drives in each vdev? Will that also effect speed, or should i have a single vdev with all the drives assigned?

1 Like

I believe you create four vdev with two drives each. The two drives in each vdev are mirrored(raid 1), and then the four vdevs are all striped(raid 0 the vdevs). But its been a bit since I ran Free/TrueNas, so I might be misremembering.

Ahh ok, thanks. What are you running now?

My 2 two cents would be RAID-Z2 double parity. So two drives can fail and you have 75.6TB usable space. To improve performance you can use a SSD L2ARC cache. There are multiple articles on STH. https://www.servethehome.com/buyers-guides/top-hardware-components-freenas-nas-servers/top-picks-freenas-l2arc-ssds/ Also check reddit r/DataHoarder and r/homelab

1 Like

OpenMediaVault with BTRFS.

It was not anything against TrueNAS NAS capabilities, I wanted more flexibility to do home server type things.

What added flexibility do you have with OpenMediaVault? It’s something i might consider.

OpenMediaVault is based on stock Debian. Which means that:

  1. There are a bunch of packages in the default repositories, definitely way more then in any of the BSDs(not counting ports)
  2. It has an official docker package, so most home server type thing that do not have a debian package have a docker container available, say for instance nextcloud. Also, there are lots more docker containers available as compared to the TrueNAS jailed plugins available.
  3. There are a lot more people using Debian as a general server OS then there are people using TrueNAS, so there are a lot more guides and other information available. Not necessarily less buggy, but it is more likely that someone has already run into the bug and has posted a workaround.

Then I also use Debian in general and have more or less standardized on it. Minus my phone, I think all of my computers that have Linux installed are on Debian.


That is not to deter you from using TrueNAS it if fits your use case. TrueNAS is excellent as a NAS and it has the best ZFS support. But it is maybe not the best as a combination NAS and general server, although it can work as such. However, there is something to be said for having a box that does one thing and does it well.

2 Likes

There are two good articles here on ZFS pool performance from Ixsystemes them selves :slight_smile:

2 Likes

Wrong question.

Better question would be “what speed do I need to meet the requirement of my workload?”

You may be better off putting your massive drives into a highly resilient array with striped mirrors or raidz2 or even Raidz3 and building a second smaller array of all flash for IOPS intensive tasks. Set up a job to periodically more older data or redundant data to the disk array. If this doesn’t work for your workload then stick with striped mirrors. Note striped mirrors still technically only give you one disk redundancy but if you have a hotspare the risk of data loss is still lower than raidz so it is “better”. Striped raidz2 vdevs is the best middle ground.

A couple of blog posts from zfs forums have said that the so called hybrid array is coming very soon and may give functionality to use SSDs for proper caching (not just arc / slog). I’ll believe it when I see it but I suspect if it does come that will be a boon.

2 Likes

if the persistent l2arc feature was released soon… that would be a huge leap. IMO

1 Like

This already came up to an extend, but asking it anways:

  1. What type of access will you use to stress your IO?
  2. Is redundancy something you want to have? if yes, how many disk are allowed to fail?
  3. Do you want more read, write, both?
  4. Would adding an SSD for caching be an option for you?

yes this will effect speed to a reasonable degree, as ZFS internals work on different speeds depending on the disk setup.
There is a great read I can recommend on this topic: FreeBSD Mastery: ZFS (Chapter 2, p.41. It says FreeBSD in the title, does apply to linux as well).

That was in the works since before encryption and ssd trim…

I’ll believe it when I see it.

3 Likes

Got it all sorted now. Beautiful. I also have it running in ESXi ;). Runs perfect.

1 Like

Was it not in TrueNAS CORE BETA1?

1 Like

I watched a 1.3TiB L2ARC persist today :wink:

3 Likes

Oh nice!

That’s gonna be huge. No more killing SSDs semi-annually for me!

1 Like

You reboot too much :stuck_out_tongue:

1 Like