First post, and have tried having a dig through the forums, Wendell’s ZFS/Optane Youtube clips and reddit but was unable to get a clear direction.
Basically building a Trueness system, and had obtained 2* Octane 118Gb drives with the intention of using these as metadata special devices. Unfortunately, I didn’t do enough forward planning and don’t think these will end up being large enough potentially with the amount of spinning rust.
I just purchased some Iron Wolf Pros at a good price (was considering Exos/WD HC550’s but the price on these was too good), so I have 9 * 20TB on the way, planning to do 4 * 20TB mirrors in the pool. Reason for buying 9 was to mitigate risk of one drive arriving DOA and then having to buy a single at a premium (happened in the last batch I ordered) and worst case I have a cold spare.
The base PC has the following:
- W2XXX CPU
- 192GB DDR4 RAM
- in X16 Slot: 4*4 NVME Holder (handles the x16 interface lane splitting)
- in X8 Slot: 2*10Gb SFP+ Card (Intel X520-DA2 maybe)
- in X4 Slot: LSI 9201-16e (have this now working in Truenas with some test drives)
- GPU (in x1 slot, might do some Plex transcoding etc, so assume this doesn’t need more than x1)
- in X16 Slot: SPARE (considering another NVME carrier)
- There is also U.2 Flex bays (currently booting off 1TB, but could go to Sata boot drives instead then utilise the U.2’s in a mirror.
2* Intel Optane 118Gb P1600X
2* WD SN850X 4TB (plan to mirror, VM Storage, Bulk video editing storage etc)
8* Ironwolf Pro 20TB Drives (plan to run in mirrors for ~80TB)
In terms of data storage, was planning to use this to backup 2 * Synology NAS units that have 16TB Mirrors in them, and not much space used (they are currently backing each up other using Synology HyperBackup tool). Will also be storage GoPro footage, PC Backups, TimeMachine MacBook backups and then obviously general other stuff. Unlikely to come anywhere near filling the total 80TB, mostly spanning more mirrors for speed (just upgraded to 10/20Gb bonds throughout home). Eventually this server will also house IPCAM footage.
I would say generally speaking, bulk of the data is larger files, especially on the hard drives, but I do have some reason sized data pools containing heaps of small things too. Also want to run a Steam/Game cache so I don’t need to keep everything on Laptop and PC all the time.
Basically, looking for recommendations on the best setup to go. I.e. get 2 more P1600X’s, and have roughly 240Gb of Metadata Special for the spinning rust pool (@40TB data that gives me 0.5% ratio approx). My biggest worry is creating a metadata special now and then it not being big enough long term. Reality is I probably won’t have heaps of data at the start, so could easily add and then re-write the data to refresh the metadata on a special vdev if I decide to add it. Alternatively, do I find half decent 1TB NVME and use that as the metadata special (in theory will be easier to replace quickly with off the shelf NVME in future.
Things to clarify:
- Do I bother with an SLOG, or given VMs would be on half decent storage its not a big deal. I could use the Octane’s for SLOG on the spinning rust if I don’t go with metadata special.
- Metadata special - do I bother, or do I need more storage (obviously have to know since the data set isn’t loaded in ZFS yet and will just grow over time).
- If I add more NVME for either of the above options, would I be better having the bulk storage NVME on a seperate PCIE carrier board so its less likely I accidentally pull the incorrect drive in the even of a fault (planning to physically label drives with serial numbers etc so I can more easily correlate in Truenas, but my understanding is if I have a free slot there I can install new drive, then “repair” without having to remove faulty drive initially)
- I understand if I am using mirrors throughout (i.e. Not RaidZx) that the metadata special can be removed and migrated back onto the spinning rust drives.
Sorry for the large amount of info - I have seen too many threads of people detailing how they wish they had setup their Truenas system differently and now it’s too late to make major changes