Recommendation for using Optane with TrueNAS Mini XL+?

The 2.5" bays on that chassis are, at least AFAIK, only support SATA drives. So, unfortunately, you wouldn’t be able to Optane in those bays. The internal M.2, on the other hand, you could use an Optane P1600X.

You can install TrueNAS on a USB SSD (not a thumb drive).
Something like this:

With something like this:
Amazon.com: Kingston 120GB A400 SATA 3 2.5" Internal SSD SA400S37/120G - HDD Replacement for Increase Performance , Black : Electronics

or something like this, with the drive that’s already in the M.2 slot preinstalled with TrueNAS:
Amazon.com: SSK Aluminum M.2 NVME SATA SSD Enclosure Adapter, USB 3.2 Gen 2 (10 Gbps) to NVME PCI-E SATA M-Key/(B+M) Key Solid State Drive External Enclosure Support UASP Trim for NVME/SATA SSDs 2242/2260/2280 : Electronics

Then it’s just a matter of changing the boot order to boot from USB rather than an internal drive. This will free up that internal slot for some data caching drives.

In ZFS, the “Read Cache” is called L2 ARC. Generally, it is not really going to help much unless your system has >64GB of RAM. This is partly because pointers to the L2ARC need to be stored in RAM, which is also the L1 ARC (first level) cache. So space which should be used for caching your data is instead used to point it to a slower (but still faster than HDD) SSD.
Because of that you need to be careful of the ratio of ARC-To-L2ARC. According to TrueNAS’s community documentation, that wasted space is about (favors such as record size matter) 25MB per 1GB of L2ARC. So if you were using a 118GB Optane in the M.2 slot, you would lose about 3GB of RAM for L2ARC pointers. For that reason, I would say skip the L2 Arc unless you have at least 32GB of RAM.

IX Systems also goes further to talk in their own documentation as for their recommendations:

From IX Systems:

Like all complicated features, deciding whether L2ARC is effective or not requires a strong understanding of your storage environment, performance goals, and the software you’re using.

However, we have a few recommendations for L2ARC devices:

Using multiple L2ARC devices helps reduce latency and improve performance.

Random Read Heavy workloads can benefit from large capacity L2ARC SSDs. L2ARC SSDs are faster than the existing data storage drives.

Sequential or streaming workloads need very fast, low-latency L2ARC devices. We recommend Enterprise-grade NVMe devices. L2ARC device capacity depends on how much faster it is than the data storage devices. An L2ARC device that is much faster than the data storage devices makes better use of its larger capacity.

With some more information they link to here:
PowerPoint Presentation (snia.org)

Since your system is a small, low-power unit it doesn’t have alot of PCIE lanes to really get the most out of an L2ARC. FWIW, I don’t think you should bother.

Optane is a great choice for a SLOG (separate log device) But In your use case:

I don’t think you need to worry about it. SMB file sharing writes asynchrously by default, and so it would bypass the write cache anyway. If you were actually storing your VMs on the NAS instead of Proxmox, with NFS or ISCSI, that’s when you would want to buy a SLOG.
ZFS sync/async + ZIL/SLOG, explained – JRS Systems: the blog (jrs-s.net)

If I were you, I would read up on the ZFS SPECIAL device, which I think may be more beneficial for you in the long run, given the constraints of the system. Also, buy as much RAM as you can afford.

ZFS Metadata Special Device: Z - Wikis & How-to Guides - Level1Techs Forums

One Optane 118GB in the M.2 slot and the other can go in something like what was posted above, since you likely want a 10GBe card anyway. Though, you will be forced into SCALE with that specific card, as AFAIK there are no drivers for that chipset in CORE (FreeBSD)

If not, your best bet in that chassis would be to use SATA SSDs for a SPECIAL device in your pool. But obviously, while they aren’t nearly as fast as Optane, they are still faster than HDDs.

1 Like