ZFS Metadata Special Device help needed

Operating Systems & Open Source Linux
1

Hello all,

I am planning to upgrade my home server/nas and I would like to implement ZFS Metadata Special Device.
Currently on my NAS the file size structure looks like this

1k: 128934
2k: 51018
4k: 41406
8k: 30902
16k: 21645
32k: 14579
64k: 13777
128k: 12690
256k: 10106
512k: 17154
1M: 14118
2M: 15163
4M: 72485
8M: 32683
16M: 7574
32M: 1468
64M: 742
128M: 1340
256M: 351
512M: 503
1G: 389
2G: 256
4G: 219
8G: 63
16G: 109
32G: 14
64G: 7

The storage is
NAME STATE READ WRITE CKSUM
nas-storage ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
sdc ONLINE 0 0 0
sdb ONLINE 0 0 0
sdd ONLINE 0 0 0
cache
nvme0n1 ONLINE 0 0 0

sdb, sdc and sdd are 16Tb Seagate Exos X16

sda(16Tb Seagate Exos X16) will be added as spare

nvme0n1 Samsung SSD 970 EVO Plus 1TB used as cache

ZFS Metadata - 2x1TB mirrored
nvme1n1 Samsung SSD 980 PRO 1TB
nvme2n1 Samsung SSD 980 PRO 1TB

nvme3n1 Samsung SSD 980 PRO 1TB

My goal is to improve the write/read performance over a 10Gbe network keeping the data safety in mind.

My questions are:

  1. I can add sda as a spare HDD for the pool, does it make sense? Or is better to have 4 HDDs in the pool?
  2. Considering the file size structure, what do you recommend to use for 1Tb HDD special_small_blocks and zfs_max_recordsize ?
  3. Can I set zfs set special_small_blocks=128K for folder where I keep large files? For <16M I have 715Gb of data and < 8M I have 596Gb of data.
  4. For ZFS Metadata I intend to use 2x1TB mirrored. Can I add one more as spare?

Have a great day.
Ion

2

Hi,

  1. iirc z1 like’s odd HDD amounts, z2 is even slower.
    2-3. don’t know
  2. i think you can. also iirc metadata is already mirrored on each drive by default? so when you go 3x1 it would make sense to disable that.

One thing i like to add because i am thinking about it myself:

Why the redundancy when i (will :sweat_smile: )have tape backup?
When there ever would be a bitflip or hd failure i could rely on the Tapes.
The Z1 seems to just waste money, energy and course grey hair since in the end it still can fail no matter how many spares (more investment more possible pain).

Better adding SSD’s with Z1 for my home directory to have live redundancy there and sync to a tape sized hdd each day until it gets unloaded to tape.

3

The only use of a ‘spare’ is to act as an active swap-in device for many zpools, and whatever zpool loses its redundancy first, the ‘spare’ fills the gap. You only have one raidz1-pool so running a ‘spare’ is pointless, you may as well run raidz2 and be doubly protected.

As far as the meta-special-mirror goes, yes the triple-mirror would work, but understand if you lose the meta-special-mirror you lose ALL of the raidz1-pool’s data too, so don’t go any lower than a triple mirror.

4

Hi,
To make the triple mirror I will have to use
zpool add nas-storage -o special mirror /dev/nvme1n1 mirror /dev/nvme2n1 /dev/nvme3n1 ?

5

Yes close, but leave out the option switch.

$ sudo zpool add nas-storage special mirror ‘list-your-drive’s WWN here’

Please don’t ever use /dev/path because that can change, always use WWN (or UUIDs)
You can get that info from;
$ ls -lh /dev/disk/by-id

6

What about questin 3? How should I set the parameters considering the hdd size and file size distribution?