Recommendation for using Optane with TrueNAS Mini XL+?

Hardware Hub Storage
, ,
1

New forum user, recently found @wendell’s videos and I have been loving the content, even if I have trouble following some of it because of the firehose of information.

I watched his video “Intel Optane is DEAD. Long Live Intel Optane! Buy them cheap right now…” and I have some questions I hope someone can help me with.

I haven’t built hardware since two decades ago, but I’ve spent the past week immersing myself in the topic via YouTube videos and blog posts, so I consider myself to have just enough knowledge to be dangerous at this point. IOW, please correct me if anything I say or imply is wrong!

I just ordered a TrueNAS Mini XL+ with no drives and I have eight (8) Seagate 16TB IronWolf drives I plan to install in it. I still need to decide on read cache and write cache and am thinking that optane 1st gen might be good for that?

There is the 118Gb P1600X for $76 and the 960Gb 905P for $399. Although I have not seen the box yet I think it has two (2) open 2.5" bays — one hot swap and one internal — and one internal M.2 slot, or at least that is what their configurator implies.

They told me the unit will be delivered with TrueNAS Core installed on the one an SSD card using the M.2 PCI-Ex2 slot so that slot is probably not available for a p1800x unless I get a SATA DOM and install TrueNAS Core on that instead? Should I do that, or does the PCI-Ex2 mean its not worth it?

It only has one yellow SATA port so I don’t think it would be possible to get two (2) SATA DOMs to mirror them (or would it be possible?)

Regarding my use-case I am a software developer who works from home and this NAS will just be for my own use to store personal and work files and to interact with a SuperMicro server running VMware ESXi server and an ASRock server running Proxmox, all connected over 10GbE. I also may start working on some very niche YouTube videos, but I have not decided to do that for certain yet. If I do though, it will just be occasional and I won’t be quitting my day job as a programmer.

So, would it make sense to put a p1600x in the M.2 drive for write cache and a 905P in the 2.5" for a read cache, or no? Is the 118Gb too small for this? Would I be better off buying something larger like a 2TB IronWolf M.2 NVMe?

The motherboard in the unit is a SuperMicro A2SDi-H-TF motherboard with only one PCI-E 3.0x4 slot, and I don’t know if it will have anything in it that I will need to keep when I get it (maybe someone here knows?). If that slot is available, would it make sense to get a PCI-E card to be able to add another 905P for a write cache?

BTW, I am not too constrained by cost, assuming it is not outrageous; the cost of the p5800x is in the category of outrageous! Since I plan this to last me for many years it is worth building it right so I can worry about other things — like coding for my day job.

Thanks in advance for any help.

-Mike

P.S. I should have mentioned that long-term reliability is probably more important to me than absolute speed.

2

If younvan finagle two m.2 using the 118gb optane as a mirrored metadata special device is likely to have the biggest impact on percieved speed of the thing.

Aynology had this x4 card thats 10gbe and dual m.2 thatd work i think. I forget if ifs x4 or x8 actually

2 Likes
3

It’s x8 PCIe lanes actually. Interesting card.

Only PCIe Gen3, though.

From reviews it seems to require the AQC107 driver.

4

The 2.5" bays on that chassis are, at least AFAIK, only support SATA drives. So, unfortunately, you wouldn’t be able to Optane in those bays. The internal M.2, on the other hand, you could use an Optane P1600X.

You can install TrueNAS on a USB SSD (not a thumb drive).
Something like this:

With something like this:
Amazon.com: Kingston 120GB A400 SATA 3 2.5" Internal SSD SA400S37/120G - HDD Replacement for Increase Performance , Black : Electronics

or something like this, with the drive that’s already in the M.2 slot preinstalled with TrueNAS:
Amazon.com: SSK Aluminum M.2 NVME SATA SSD Enclosure Adapter, USB 3.2 Gen 2 (10 Gbps) to NVME PCI-E SATA M-Key/(B+M) Key Solid State Drive External Enclosure Support UASP Trim for NVME/SATA SSDs 2242/2260/2280 : Electronics

Then it’s just a matter of changing the boot order to boot from USB rather than an internal drive. This will free up that internal slot for some data caching drives.

In ZFS, the “Read Cache” is called L2 ARC. Generally, it is not really going to help much unless your system has >64GB of RAM. This is partly because pointers to the L2ARC need to be stored in RAM, which is also the L1 ARC (first level) cache. So space which should be used for caching your data is instead used to point it to a slower (but still faster than HDD) SSD.
Because of that you need to be careful of the ratio of ARC-To-L2ARC. According to TrueNAS’s community documentation, that wasted space is about (favors such as record size matter) 25MB per 1GB of L2ARC. So if you were using a 118GB Optane in the M.2 slot, you would lose about 3GB of RAM for L2ARC pointers. For that reason, I would say skip the L2 Arc unless you have at least 32GB of RAM.

IX Systems also goes further to talk in their own documentation as for their recommendations:

From IX Systems:

Like all complicated features, deciding whether L2ARC is effective or not requires a strong understanding of your storage environment, performance goals, and the software you’re using.

However, we have a few recommendations for L2ARC devices:

Using multiple L2ARC devices helps reduce latency and improve performance.

Random Read Heavy workloads can benefit from large capacity L2ARC SSDs. L2ARC SSDs are faster than the existing data storage drives.

Sequential or streaming workloads need very fast, low-latency L2ARC devices. We recommend Enterprise-grade NVMe devices. L2ARC device capacity depends on how much faster it is than the data storage devices. An L2ARC device that is much faster than the data storage devices makes better use of its larger capacity.

With some more information they link to here:
PowerPoint Presentation (snia.org)

Since your system is a small, low-power unit it doesn’t have alot of PCIE lanes to really get the most out of an L2ARC. FWIW, I don’t think you should bother.

Optane is a great choice for a SLOG (separate log device) But In your use case:

I don’t think you need to worry about it. SMB file sharing writes asynchrously by default, and so it would bypass the write cache anyway. If you were actually storing your VMs on the NAS instead of Proxmox, with NFS or ISCSI, that’s when you would want to buy a SLOG.
ZFS sync/async + ZIL/SLOG, explained – JRS Systems: the blog (jrs-s.net)

If I were you, I would read up on the ZFS SPECIAL device, which I think may be more beneficial for you in the long run, given the constraints of the system. Also, buy as much RAM as you can afford.

ZFS Metadata Special Device: Z - Wikis & How-to Guides - Level1Techs Forums

One Optane 118GB in the M.2 slot and the other can go in something like what was posted above, since you likely want a 10GBe card anyway. Though, you will be forced into SCALE with that specific card, as AFAIK there are no drivers for that chipset in CORE (FreeBSD)

If not, your best bet in that chassis would be to use SATA SSDs for a SPECIAL device in your pool. But obviously, while they aren’t nearly as fast as Optane, they are still faster than HDDs.

1 Like
5

Total nonsense. If working set is > memory, L2 helps a lot, even with 8GB of memory.

This is wrong. L2 headers are dependent on record/block size and it is more like 1GB per TB, not 3GB per 118GB lol.

L2ARC headers in ARC are vastly overrated and their impact exaggerated. (persistent) L2 is great and cheap.

see this table for reference:

1GB per 2TB of L2ARC with default recordsize of 128k. Get as much L2 as you can. Optane is kinda wasted for L2 anyway because L2 is like 95% reads and 118GB isn’t really much space (I got more memory).

6

I hear you, but I am referencing the TrueNAS community’s reference manual directly FWIW:
Introduction to ZFS | TrueNAS Community
image

BUT IF we look at the math outlined here:
OpenZFS: All about the cache vdev or L2ARC | Klara Inc (klarasystems.com)

(L2ARC size in kilobytes) / (typical recordsize -- or volblocksize -- in kilobytes) * 70 bytes = ARC header size in RAM

image
with a 118GB Optane as L2ARC and128k block size, you’re right it’s cheap in this example, about 64MB. That’s a neat little chart :slight_smile:

Regardless, your goal should be to keep your ARC hit ratio as high as possible, if your ARC cache has a >90% hit rate you’re not necessarily going to feel the benefit of the L2ARC.

This is my pool, as an example:

image
image1265×652 67.5 KB

There are times where (mostly when I’m seeding alot) I hit 30-50% L2ARC hits, but mostly it’s going unused. Obviously this is workflow dependent. My ARC hits never go below 90%.

root@prod[~]# arc_summary

------------------------------------------------------------------------
ZFS Subsystem Report                            Fri Dec 23 03:32:16 2022
Linux 5.15.79+truenas                                            2.1.6-1
Machine: prod (x86_64)                                           2.1.6-1

ARC status:                                                      HEALTHY
        Memory throttle count:                                         0

ARC size (current):                                   100.2 %  126.0 GiB
        Target size (adaptive):                       100.0 %  125.8 GiB
        Min size (hard limit):                          6.2 %    7.9 GiB
        Max size (high water):                           16:1  125.8 GiB
        Most Frequently Used (MFU) cache size:         64.7 %   80.8 GiB
        Most Recently Used (MRU) cache size:           35.3 %   44.1 GiB
        Metadata cache size (hard limit):              75.0 %   94.3 GiB
        Metadata cache size (current):                  6.2 %    5.8 GiB
        Dnode cache size (hard limit):                 10.0 %    9.4 GiB
        Dnode cache size (current):                     2.6 %  252.9 MiB

ARC hash breakdown:
        Elements max:                                               2.7M
        Elements current:                              73.0 %       2.0M
        Collisions:                                                 1.6M
        Chain max:                                                     4
        Chains:                                                    56.9k

ARC misc:
        Deleted:                                                    3.6M
        Mutex misses:                                                474
        Eviction skips:                                             2.2k
        Eviction skips due to L2 writes:                               0
        L2 cached evictions:                                   676.1 GiB
        L2 eligible evictions:                                 225.1 GiB
        L2 eligible MFU evictions:                     65.6 %  147.6 GiB
        L2 eligible MRU evictions:                     34.4 %   77.5 GiB
        L2 ineligible evictions:                               116.1 GiB

ARC total accesses (hits + misses):                               453.3M
        Cache hit ratio:                               99.0 %     449.0M
        Cache miss ratio:                               1.0 %       4.3M
        Actual hit ratio (MFU + MRU hits):             98.8 %     447.9M
        Data demand efficiency:                        96.2 %      61.2M
        Data prefetch efficiency:                      16.4 %       1.4M

Cache hits by cache type:
        Most frequently used (MFU):                    93.2 %     418.3M
        Most recently used (MRU):                       6.6 %      29.6M
        Most frequently used (MFU) ghost:               0.1 %     462.2k
        Most recently used (MRU) ghost:                 0.1 %     353.7k
        Anonymously used:                               0.1 %     326.6k

Cache hits by data type:
        Demand data:                                   13.1 %      58.9M
        Demand prefetch data:                           0.1 %     225.4k
        Demand metadata:                               86.5 %     388.5M
        Demand prefetch metadata:                       0.3 %       1.4M

Cache misses by data type:
        Demand data:                                   54.0 %       2.3M
        Demand prefetch data:                          26.5 %       1.2M
        Demand metadata:                               17.8 %     774.8k
        Demand prefetch metadata:                       1.6 %      70.9k

DMU prefetch efficiency:                                           25.5M
        Hit ratio:                                     22.8 %       5.8M
        Miss ratio:                                    77.2 %      19.7M

L2ARC status:                                                    HEALTHY
        Low memory aborts:                                             0
        Free on write:                                                 3
        R/W clashes:                                                   0
        Bad checksums:                                                 0
        I/O errors:                                                    0

L2ARC size (adaptive):                                         235.0 GiB
        Compressed:                                    93.8 %  220.4 GiB
        Header size:                                  < 0.1 %    8.9 MiB
        MFU allocated size:                            82.2 %  181.1 GiB
        MRU allocated size:                            17.6 %   38.7 GiB
        Prefetch allocated size:                        0.3 %  569.0 MiB
        Data (buffer content) allocated size:         100.0 %  220.3 GiB
        Metadata (buffer content) allocated size:     < 0.1 %  100.8 MiB

L2ARC breakdown:                                                    1.1M
        Hit ratio:                                     12.0 %     126.9k
        Miss ratio:                                    88.0 %     930.9k
        Feeds:                                                    222.9k

L2ARC writes:
        Writes sent:                                    100 %     105.1k

L2ARC evicts:
        Lock retries:                                                 15
        Upon reading:                                                  0

Solaris Porting Layer (SPL):
        spl_hostid                                                     0
        spl_hostid_path                                      /etc/hostid
        spl_kmem_alloc_max                                       8388608
        spl_kmem_alloc_warn                                        65536
        spl_kmem_cache_kmem_threads                                    4
        spl_kmem_cache_magazine_size                                   0
        spl_kmem_cache_max_size                                       32
        spl_kmem_cache_obj_per_slab                                    8
        spl_kmem_cache_reclaim                                         0
        spl_kmem_cache_slab_limit                                  16384
        spl_max_show_tasks                                           512
        spl_panic_halt                                                 1
        spl_schedule_hrtimeout_slack_us                                0
        spl_taskq_kick                                                 0
        spl_taskq_thread_bind                                          0
        spl_taskq_thread_dynamic                                       1
        spl_taskq_thread_priority                                      1
        spl_taskq_thread_sequential                                    4

Tunables:
        dbuf_cache_hiwater_pct                                        10
        dbuf_cache_lowater_pct                                        10
        dbuf_cache_max_bytes                        18446744073709551615
        dbuf_cache_shift                                               5
        dbuf_metadata_cache_max_bytes               18446744073709551615
        dbuf_metadata_cache_shift                                      6
        dmu_object_alloc_chunk_shift                                   7
        dmu_prefetch_max                                       134217728
        ignore_hole_birth                                              1
        l2arc_exclude_special                                          1
        l2arc_feed_again                                               1
        l2arc_feed_min_ms                                            200
        l2arc_feed_secs                                                1
        l2arc_headroom                                                 2
        l2arc_headroom_boost                                         200
        l2arc_meta_percent                                            33
        l2arc_mfuonly                                                  0
        l2arc_noprefetch                                               1
        l2arc_norw                                                     0
        l2arc_rebuild_blocks_min_l2size                       1073741824
        l2arc_rebuild_enabled                                          1
        l2arc_trim_ahead                                               0
        l2arc_write_boost                                        8388608
        l2arc_write_max                                          8388608
        metaslab_aliquot                                         1048576
        metaslab_bias_enabled                                          1
        metaslab_debug_load                                            0
        metaslab_debug_unload                                          0
        metaslab_df_max_search                                  16777216
        metaslab_df_use_largest_segment                                0
        metaslab_force_ganging                                  16777217
        metaslab_fragmentation_factor_enabled                          1
        metaslab_lba_weighting_enabled                                 1
        metaslab_preload_enabled                                       1
        metaslab_unload_delay                                         32
        metaslab_unload_delay_ms                                  600000
        send_holes_without_birth_time                                  1
        spa_asize_inflation                                           24
        spa_config_path                             /etc/zfs/zpool.cache
        spa_load_print_vdev_tree                                       0
        spa_load_verify_data                                           1
        spa_load_verify_metadata                                       1
        spa_load_verify_shift                                          4
        spa_slop_shift                                                 5
        vdev_file_logical_ashift                                       9
        vdev_file_physical_ashift                                      9
        vdev_removal_max_span                                      32768
        vdev_validate_skip                                             0
        zap_iterate_prefetch                                           1
        zfetch_array_rd_sz                                       1048576
        zfetch_max_distance                                     67108864
        zfetch_max_idistance                                    67108864
        zfetch_max_sec_reap                                            2
        zfetch_max_streams                                             8
        zfetch_min_distance                                      4194304
        zfetch_min_sec_reap                                            1
        zfs_abd_scatter_enabled                                        1
        zfs_abd_scatter_max_order                                     13
        zfs_abd_scatter_min_size                                    1536
        zfs_admin_snapshot                                             0
        zfs_allow_redacted_dataset_mount                               0
        zfs_arc_average_blocksize                                   8192
        zfs_arc_dnode_limit                                            0
        zfs_arc_dnode_limit_percent                                   10
        zfs_arc_dnode_reduce_percent                                  10
        zfs_arc_evict_batch_limit                                     10
        zfs_arc_eviction_pct                                         200
        zfs_arc_grow_retry                                             0
        zfs_arc_lotsfree_percent                                      10
        zfs_arc_max                                                    0
        zfs_arc_meta_adjust_restarts                                4096
        zfs_arc_meta_limit                                             0
        zfs_arc_meta_limit_percent                                    75
        zfs_arc_meta_min                                               0
        zfs_arc_meta_prune                                         10000
        zfs_arc_meta_strategy                                          1
        zfs_arc_min                                                    0
        zfs_arc_min_prefetch_ms                                        0
        zfs_arc_min_prescient_prefetch_ms                              0
        zfs_arc_p_dampener_disable                                     1
        zfs_arc_p_min_shift                                            0
        zfs_arc_pc_percent                                             0
        zfs_arc_prune_task_threads                                     1
        zfs_arc_shrink_shift                                           0
        zfs_arc_shrinker_limit                                     10000
        zfs_arc_sys_free                                               0
        zfs_async_block_max_blocks                  18446744073709551615
        zfs_autoimport_disable                                         1
        zfs_btree_verify_intensity                                     0
        zfs_checksum_events_per_second                                20
        zfs_commit_timeout_pct                                         5
        zfs_compressed_arc_enabled                                     1
        zfs_condense_indirect_commit_entry_delay_ms                    0
        zfs_condense_indirect_obsolete_pct                            25
        zfs_condense_indirect_vdevs_enable                             1
        zfs_condense_max_obsolete_bytes                       1073741824
        zfs_condense_min_mapping_bytes                            131072
        zfs_dbgmsg_enable                                              1
        zfs_dbgmsg_maxsize                                       4194304
        zfs_dbuf_state_index                                           0
        zfs_ddt_data_is_special                                        1
        zfs_deadman_checktime_ms                                   60000
        zfs_deadman_enabled                                            1
        zfs_deadman_failmode                                        wait
        zfs_deadman_synctime_ms                                   600000
        zfs_deadman_ziotime_ms                                    300000
        zfs_dedup_prefetch                                             0
        zfs_delay_min_dirty_percent                                   60
        zfs_delay_scale                                           500000
        zfs_delete_blocks                                          20480
        zfs_dirty_data_max                                    4294967296
        zfs_dirty_data_max_max                                4294967296
        zfs_dirty_data_max_max_percent                                25
        zfs_dirty_data_max_percent                                    10
        zfs_dirty_data_sync_percent                                   20
        zfs_disable_ivset_guid_check                                   0
        zfs_dmu_offset_next_sync                                       1
        zfs_embedded_slog_min_ms                                      64
        zfs_expire_snapshot                                          300
        zfs_fallocate_reserve_percent                                110
        zfs_flags                                                      0
        zfs_free_bpobj_enabled                                         1
        zfs_free_leak_on_eio                                           0
        zfs_free_min_time_ms                                        1000
        zfs_history_output_max                                   1048576
        zfs_immediate_write_sz                                     32768
        zfs_initialize_chunk_size                                1048576
        zfs_initialize_value                        16045690984833335022
        zfs_keep_log_spacemaps_at_export                               0
        zfs_key_max_salt_uses                                  400000000
        zfs_livelist_condense_new_alloc                                0
        zfs_livelist_condense_sync_cancel                              0
        zfs_livelist_condense_sync_pause                               0
        zfs_livelist_condense_zthr_cancel                              0
        zfs_livelist_condense_zthr_pause                               0
        zfs_livelist_max_entries                                  500000
        zfs_livelist_min_percent_shared                               75
        zfs_lua_max_instrlimit                                 100000000
        zfs_lua_max_memlimit                                   104857600
        zfs_max_async_dedup_frees                                 100000
        zfs_max_log_walking                                            5
        zfs_max_logsm_summary_length                                  10
        zfs_max_missing_tvds                                           0
        zfs_max_nvlist_src_size                                        0
        zfs_max_recordsize                                       1048576
        zfs_metaslab_find_max_tries                                  100
        zfs_metaslab_fragmentation_threshold                          70
        zfs_metaslab_max_size_cache_sec                             3600
        zfs_metaslab_mem_limit                                        25
        zfs_metaslab_segment_weight_enabled                            1
        zfs_metaslab_switch_threshold                                  2
        zfs_metaslab_try_hard_before_gang                              0
        zfs_mg_fragmentation_threshold                                95
        zfs_mg_noalloc_threshold                                       0
        zfs_min_metaslabs_to_flush                                     1
        zfs_multihost_fail_intervals                                  10
        zfs_multihost_history                                          0
        zfs_multihost_import_intervals                                20
        zfs_multihost_interval                                      1000
        zfs_multilist_num_sublists                                     0
        zfs_no_scrub_io                                                0
        zfs_no_scrub_prefetch                                          0
        zfs_nocacheflush                                               0
        zfs_nopwrite_enabled                                           1
        zfs_object_mutex_size                                         64
        zfs_obsolete_min_time_ms                                     500
        zfs_override_estimate_recordsize                               0
        zfs_pd_bytes_max                                        52428800
        zfs_per_txg_dirty_frees_percent                                5
        zfs_prefetch_disable                                           0
        zfs_read_history                                               0
        zfs_read_history_hits                                          0
        zfs_rebuild_max_segment                                  1048576
        zfs_rebuild_scrub_enabled                                      1
        zfs_rebuild_vdev_limit                                  33554432
        zfs_reconstruct_indirect_combinations_max                   4096
        zfs_recover                                                    0
        zfs_recv_queue_ff                                             20
        zfs_recv_queue_length                                   16777216
        zfs_recv_write_batch_size                                1048576
        zfs_removal_ignore_errors                                      0
        zfs_removal_suspend_progress                                   0
        zfs_remove_max_segment                                  16777216
        zfs_resilver_disable_defer                                     0
        zfs_resilver_min_time_ms                                    3000
        zfs_scan_blkstats                                              0
        zfs_scan_checkpoint_intval                                  7200
        zfs_scan_fill_weight                                           3
        zfs_scan_ignore_errors                                         0
        zfs_scan_issue_strategy                                        0
        zfs_scan_legacy                                                0
        zfs_scan_max_ext_gap                                     2097152
        zfs_scan_mem_lim_fact                                         20
        zfs_scan_mem_lim_soft_fact                                    20
        zfs_scan_strict_mem_lim                                        0
        zfs_scan_suspend_progress                                      0
        zfs_scan_vdev_limit                                      4194304
        zfs_scrub_min_time_ms                                       1000
        zfs_send_corrupt_data                                          0
        zfs_send_no_prefetch_queue_ff                                 20
        zfs_send_no_prefetch_queue_length                        1048576
        zfs_send_queue_ff                                             20
        zfs_send_queue_length                                   16777216
        zfs_send_unmodified_spill_blocks                               1
        zfs_slow_io_events_per_second                                 20
        zfs_spa_discard_memory_limit                            16777216
        zfs_special_class_metadata_reserve_pct                        25
        zfs_sync_pass_deferred_free                                    2
        zfs_sync_pass_dont_compress                                    8
        zfs_sync_pass_rewrite                                          2
        zfs_sync_taskq_batch_pct                                      75
        zfs_traverse_indirect_prefetch_limit                          32
        zfs_trim_extent_bytes_max                              134217728
        zfs_trim_extent_bytes_min                                  32768
        zfs_trim_metaslab_skip                                         0
        zfs_trim_queue_limit                                          10
        zfs_trim_txg_batch                                            32
        zfs_txg_history                                              100
        zfs_txg_timeout                                                5
        zfs_unflushed_log_block_max                               131072
        zfs_unflushed_log_block_min                                 1000
        zfs_unflushed_log_block_pct                                  400
        zfs_unflushed_log_txg_max                                   1000
        zfs_unflushed_max_mem_amt                             1073741824
        zfs_unflushed_max_mem_ppm                                   1000
        zfs_unlink_suspend_progress                                    0
        zfs_user_indirect_is_special                                   1
        zfs_vdev_aggregate_trim                                        0
        zfs_vdev_aggregation_limit                               1048576
        zfs_vdev_aggregation_limit_non_rotating                   131072
        zfs_vdev_async_read_max_active                                 3
        zfs_vdev_async_read_min_active                                 1
        zfs_vdev_async_write_active_max_dirty_percent                 60
        zfs_vdev_async_write_active_min_dirty_percent                 30
        zfs_vdev_async_write_max_active                               10
        zfs_vdev_async_write_min_active                                2
        zfs_vdev_cache_bshift                                         16
        zfs_vdev_cache_max                                         16384
        zfs_vdev_cache_size                                            0
        zfs_vdev_default_ms_count                                    200
        zfs_vdev_default_ms_shift                                     29
        zfs_vdev_initializing_max_active                               1
        zfs_vdev_initializing_min_active                               1
        zfs_vdev_max_active                                         1000
        zfs_vdev_max_auto_ashift                                      14
        zfs_vdev_min_auto_ashift                                       9
        zfs_vdev_min_ms_count                                         16
        zfs_vdev_mirror_non_rotating_inc                               0
        zfs_vdev_mirror_non_rotating_seek_inc                          1
        zfs_vdev_mirror_rotating_inc                                   0
        zfs_vdev_mirror_rotating_seek_inc                              5
        zfs_vdev_mirror_rotating_seek_offset                     1048576
        zfs_vdev_ms_count_limit                                   131072
        zfs_vdev_nia_credit                                            5
        zfs_vdev_nia_delay                                             5
        zfs_vdev_queue_depth_pct                                    1000
        zfs_vdev_raidz_impl cycle [fastest] original scalar sse2 ssse3 avx2
        zfs_vdev_read_gap_limit                                    32768
        zfs_vdev_rebuild_max_active                                    3
        zfs_vdev_rebuild_min_active                                    1
        zfs_vdev_removal_max_active                                    2
        zfs_vdev_removal_min_active                                    1
        zfs_vdev_scheduler                                        unused
        zfs_vdev_scrub_max_active                                      3
        zfs_vdev_scrub_min_active                                      1
        zfs_vdev_sync_read_max_active                                 10
        zfs_vdev_sync_read_min_active                                 10
        zfs_vdev_sync_write_max_active                                10
        zfs_vdev_sync_write_min_active                                10
        zfs_vdev_trim_max_active                                       2
        zfs_vdev_trim_min_active                                       1
        zfs_vdev_write_gap_limit                                    4096
        zfs_vnops_read_chunk_size                                1048576
        zfs_wrlog_data_max                                    8589934592
        zfs_xattr_compat                                               0
        zfs_zevent_len_max                                           512
        zfs_zevent_retain_expire_secs                                900
        zfs_zevent_retain_max                                       2000
        zfs_zil_clean_taskq_maxalloc                             1048576
        zfs_zil_clean_taskq_minalloc                                1024
        zfs_zil_clean_taskq_nthr_pct                                 100
        zil_maxblocksize                                          131072
        zil_nocacheflush                                               0
        zil_replay_disable                                             0
        zil_slog_bulk                                             786432
        zio_deadman_log_all                                            0
        zio_dva_throttle_enabled                                       1
        zio_requeue_io_start_cut_in_line                               1
        zio_slow_io_ms                                             30000
        zio_taskq_batch_pct                                           80
        zio_taskq_batch_tpq                                            0
        zvol_inhibit_dev                                               0
        zvol_major                                                   230
        zvol_max_discard_blocks                                    16384
        zvol_prefetch_bytes                                       131072
        zvol_request_sync                                              0
        zvol_threads                                                  32
        zvol_volmode                                                   2

VDEV cache disabled, skipping section

ZIL committed transactions:                                         3.7M
        Commit requests:                                          421.1k
        Flushes to stable storage:                                421.1k
        Transactions to SLOG storage pool:            0 Bytes          0
        Transactions to non-SLOG storage pool:      108.8 GiB       1.2M

From my own testing, I have had systems with 32 or 64 GB of RAM that have high ARC hit >90%, and upgraded their RAM to 64 or 128 GB. They have remained at >90% hit rates with the larger ARC cache.
However, adding small 128GB L2ARCs to systems with the same 32 or 64 GB of ram and similar workloads (mostly security camera recording servers) hasn’t yielded >90% L2ARC hit rates.
While that is anecdotal evidence, I’m certainly not the first to bring this up.

7

Thanks @wendell (Typos aside! :laughing:)

Never occurred to me to use a Synology card in something that is not a Synology NAS.

That’s is actually where I started — to upgrade my Synlogy 1817+ — but then realized it wouldn’t work, and the rabbit hole I followed brought me to iXsystems.

BTW, why the card with 10GbE and not just a card with two M.2? They have two (2) 10GbE on the motorboard.

P.S. I have been watching a lot of your videos lately, and I am in awe of your knowledge! (And I rarely find people whose knowledge awes me at this point in my life.)

Much respect!

8

Wow @NickF, thanks for all the help. MUCH appreciated.

Thanks for the links.

But not a SATA DOM? Reason?

So, let’s say I upgrade to 256Gb RAM once I get it. Then? Still don’t bother?

Currently planning to keep VMs on my VM server, so I guess I don’t need write cache then, either?

Will do!

ZFS Metadata Special Device: Z - Wikis & How-to Guides - Level1Techs Forums

Do I (really) need a 10GbE card when I have two (2) 10GbE ports on the motherboard?

I will definitely check out that option. Thanks again!

9

Interesting.

Do you have specific recommendations for my configuration/use-case?

10

Development and VMs (especially with Proxmox server getting all the zvols via iSCSI I imagine), a special vdev for metadata makes sense. Problem is the lack of slots/bays for these extras. I was considering to buy the mini XL+ myself, but decided on a all-in-one Ryzen server.

There are 8x 3.5" bays for your HDDs and 2x 2.5" for SATA SSD. You can either get 2x SSD for special metadata vdev or either 2x L2ARC or 1xL2ARC (read-cache) and 1x SLOG (“write” cache, not really though, but helps with sync writes if you like to keep sync on). Problem with SATA SSDs is that 500MB/s on a cache hit for sequential data isn’t that great. Two L2ARC devices will saturate your 10GbE connection however.

I don’t think you should tweak around with the NVMe x2 slot as you still need a dedicated drive for the boot-pool.

TLDR: 2x consumer SATA SSDs as L2ARC or 1x consumer SATA SSD + 1x enterprise SATA SSD as SLOG with high write endurance.
Alternative: 2x consumer SSD as special metadata mirror, but basically forfeits any opportunity for read/write caching because no bays left.

But I’d personally rather take two L2ARC devices, keep boot-pool as pre-configured by iX Systems with sync=disabled and just let it run and be happy.

You really have to decide if you need sync or not. You really want a SLOG if you do, because random writes on 8x RAIDZ HDDs are horror incarnate.
You can tell ZFS to disable sync and it leaves everything in memory and ZFS tells the application “Well, everything’s already on-disk, next pls”, which is fast, but less secure as memory is volatile. In this case you don’t need a write-cache as your memory serves as such.

Well if you only write stuff, you don’t read very often. Try reading 500GB of data with and without an L2ARC. With a proper L2, most comes from memory+SSD. Without L2, you get a couple of GB more from memory and the majority is fetched from HDD. ARC hit rate is a highly dubious metric and I personally don’t trust it anymore as it is misleading and values quantity over quality. Which is valid, but often misleading.
Most of the L2ARC talks are based on ancient assumptions and from a time where dirt cheap NVMe wasn’t a thing.

That TrueNAS Community link also portraits worst case scenarios with those numbers (4/8k blocksize (ZVOLs) across the pool), which doesn’t reflect the stuff people use here on the forums including this thread. Sure, some machine with 8GB RAM and 2TB L2ARC along with 4k/8k recordsize is madness. But that’s an exception.

2 Likes
11

@Exard3k — Thank you for elaborating!

I will say it is going to take me numerous re-reads and a fair amount of googling to let all that sink in, but I won’t be getting the device until early January so I have time.

Mind if I follow up later after I get the unit if I still have questions?

12

Sure, that’s what the forums here are for. The good thing is that you can add things like L2ARC, SLOG and special after pool creation without troubles. L2ARC and SLOG can also be removed at will, so you don’t lock into anything there.

So just let it run with 8x HDD when it gets delivered and see where performance needs tweaking and decide accordingly.

1 Like
13

I only suggested that method because a SATA DOM is a tiny little thing with very little NAND on it for a very high price. You’re generally going to have more write endurance in a real SSD for less money.

I think that’s something to re-evaluate based on telemetry from the ARC based on your usage.

I wouldn’t bother if you’re using a system that has PCI-E lane constrictions like yours. If all you’re doing is async writes there is no reason for it.

I thought the board only had gigabit, disregard!

SMB file sharing does not write synchronously by default anyway.

I think the entire spirit of my post was to illustrate just that: L2ARC only matters in specific workloads and under certain circumstances. ARC size always matters, and SPECIAL DEVICES are not cache, your data is permanently stored on them and always accelerated.

In my opinion, based on information I have seen here in the forums and from my own experience, more RAM + the fastest SPECIAL devices you can fit are going to have more benefit than a dedicated L2ARC device. At least, for homelab type general purpose workloads.

The only realworld example I can share that I have personally seen maximum benefit from an L2ARC device was a fileshare that I built for a client of mine that had roaming Windows profiles and a dedicated network share. But that was with a dozen people accessing those share all day long. In that example, enough commonly accessed files were present to actually enter the L2ARC and not get evicted or show basically no hit rates.

Ironically, recently, I’ve migrated him to OneDrive/SharePoint so that use case doesn’t even apply any longer. From what I’ve seen most folks and small businesses are going in that direction anyway.

1 Like
14

Thanks to both of you!

15

This topic was automatically closed 273 days after the last reply. New replies are no longer allowed.