FreeBSD is interesting as a choice. I don’t think it’s unfair to say it’s less well developed than Linux/KVM for a VM host, though I don’t see that it couldn’t work for you, only suggest to understand that going in.
If you’re going manual (great!), the main thing I’d recommend is make a trial install on an old device and play with it as a host, create pools (on loopback files if you need to), replicate datasets from a to b. At least try each operation once (including replace).
This is more the of an expert view -
but to quickstart, for zfs -
ashift=12 means 4k block size (not to be confused with recordsize), the other parameters people would typically tune are later configurable (I don’t think there’s a strong reason not to enable compression in most cases, you can toggle this property later but if you already copied everything, then you might have to again).
There’s some level of opinion not to use the top level dataset, e.g. make a zpool/stuff even if you only think you’ll need one dataset.
Into the more subjective, I’d be tempted to configure all the 2T SSD, & directly to the host, like:
2=2
2=2
2=2
= 6t pool, 1 failure tolerant
Additional resource for your consideration
benefit:
a fast configuration, can later replace a 2 then a 2, on one of the pairs then you have additional storage.
If you were to arrange for the OS vhd files to be here, and your ‘fast’ served files from a VM with a vhd here that should be performant. Generally it’s not ‘clean’ to serve anything from the host directly (maybe you could run it from a jail), but you don’t have to start with that, and it’s always best to break migrations down into different problems you can attack at other times.
Snapshot, and schedule replicate the pool to a dataset on the HDD RAIDZ2.
To which:
- an optane/nvme slog (mirror) can be attached, or ignore it and sync=disabled
- special vdev for metadata is a point of failure but elsewhere on this forum is demonstrated to offer performance benefit. However, I think, if you’re not multithread writing it 24/7 it’s probably not worth it, considering you will have a fast pool.
- L2ARC - if you have a spare nvme left, then certainly attach it.
I think it’s fun though to consider yourself how to best deploy, and you’ll probably want to make some adjustments over time.