so each pool has 96Gb/s, 2x SFF-8643 are 8x12Gb/s and the theoretical maximum of your 20 SSD is 10.8GB/s, the connection of the backplane is a bit tight but OK
I meant the CX-6 in your workstation, is the card extra cooled?
no, with ZFS you should create a datasets for each application, depends on the database but some using 16k, qcow2 64k, movies >512k and so on.
The NAS has direct-attach boards. No SAS expander. It’s fully-bandwidth SATA3 to each port over SAS3.
It’s my offsite NAS w/ HDDs that has the SAS expander.
I wouldn’t call 170k iops and 10GB/s slow by any means… Enterprise hardware that costs six figures to start with does not reach this kind of performance …
As for why your bigger pool tops off at that speed, are you using deduplication? I see you have created a dedup special device …
I have deduplication drives, but I decided not to enable deduplication yet.
I’m still on edge about using it.
Good, then my next suspect would be the L2ARC ssd pair .. it may be what limits your pool sequential reads
You could try removing it from the pool and see if/how performance changes …
Good point if the goal is to maximize inbox performance, regardless of whether it makes sense in his use case or not.
I would have rather gone to 256GB of RAM, if money is not a concern, as Truenas limits the available RAM for ZFS to 50%.
So 64GB RAM looks a bit unbalanced given the storage system.
But this does not explain his bad Samba performance.
because we had talked about it recordsize for large files
recordsize 1M
[manja01 ~]# dd if=/dev/zero of=/store01/test1/fio1.tmp bs=1G count=10
10+0 Datensätze ein
10+0 Datensätze aus
10737418240 Bytes (11 GB, 10 GiB) kopiert, 3,09422 s, 3,5 GB/s
recordsize 512k
[manja01 ~]# dd if=/dev/zero of=/store01/test2/fio2.tmp bs=1G count=10
10+0 Datensätze ein
10+0 Datensätze aus
10737418240 Bytes (11 GB, 10 GiB) kopiert, 3,20663 s, 3,3 GB/s
recordsize 128k
[manja01 ~]# dd if=/dev/zero of=/store01/test3/fio3.tmp bs=1G count=10
10+0 Datensätze ein
10+0 Datensätze aus
10737418240 Bytes (11 GB, 10 GiB) kopiert, 3,75711 s, 2,9 GB/s
Absolutely, but the question at this point was why the 10 vdev all ssd pool was performing the same as the 20 vdev one … if all reads go through the cache, and the cache is only two SSDs 10GB looks right about where it would peak?
Anyway, if the storage performance is 10GB/s-100K iops, the network IPerf tops the 10Gbit link, I’d say the issue with the Samba performance is to be sought somewher else (either the win client or the Ubiquity flow control or the human running the tests 
)
dd is not really useful for validating storage speed, especially on ZFS .. it is single threaded, if compression is enabled you will be testing how fast a single core can compress zeros and write the metadata around it …
you are right, but in this specific case it shows that a lager recordsize helps with big files.
When I have later time, I can repeat it with .mkv files.
Just bought another 128GB of RAM, but it won’t be here for at least a week. I don’t think I really need it, but I always planned to buy it at some point.
You’re right about those two cache SSDs, but they’re only 3.5GB/s max each. They’re Intel Optane 905p drives which are PCIe 3.0.
I originally bought them thinking the lower latency would be better, and the fact that they were faster than my SATA drives. Now that I have so many drives, they may be a bottleneck, but the read speed at 10GB/s is more than the theoretical read speed of these two combined (~7GB/s).
I tested without the cache drives and got the same results. 9-10GB/s. Something funky’s going on.
# fio --filename=/mnt/Bunnies/Test/fio4.tmp --direct=1 --rw=read --bs=64k --ioengine=libaio --iodepth=64 --runtime=120 --numjobs=4 --time_based --group_reporting --name=throughput-test-job --eta-newline=1 --readonly
throughput-test-job: (g=0): rw=read, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64
...
fio-3.25
Starting 4 processes
Jobs: 4 (f=4): [R(4)][2.5%][r=10.7GiB/s][r=175k IOPS][eta 01m:57s]
Jobs: 4 (f=4): [R(4)][4.2%][r=9.97GiB/s][r=163k IOPS][eta 01m:55s]
Jobs: 4 (f=4): [R(4)][5.8%][r=10.1GiB/s][r=165k IOPS][eta 01m:53s]
Jobs: 4 (f=4): [R(4)][7.5%][r=9.93GiB/s][r=163k IOPS][eta 01m:51s]
Jobs: 4 (f=4): [R(4)][9.2%][r=9723MiB/s][r=156k IOPS][eta 01m:49s]
Jobs: 4 (f=4): [R(4)][10.9%][r=10.2GiB/s][r=166k IOPS][eta 01m:46s]
Jobs: 4 (f=4): [R(4)][12.5%][r=10.1GiB/s][r=166k IOPS][eta 01m:45s]
Jobs: 4 (f=4): [R(4)][14.2%][r=10.0GiB/s][r=164k IOPS][eta 01m:43s]
Jobs: 4 (f=4): [R(4)][15.8%][r=10.1GiB/s][r=166k IOPS][eta 01m:41s]
Jobs: 4 (f=4): [R(4)][17.5%][r=10.2GiB/s][r=168k IOPS][eta 01m:39s]
Jobs: 4 (f=4): [R(4)][19.2%][r=10.1GiB/s][r=165k IOPS][eta 01m:37s]
Jobs: 4 (f=4): [R(4)][20.8%][r=10.1GiB/s][r=165k IOPS][eta 01m:35s]
Jobs: 4 (f=4): [R(4)][22.5%][r=10.2GiB/s][r=167k IOPS][eta 01m:33s]
Jobs: 4 (f=4): [R(4)][24.2%][r=10.1GiB/s][r=165k IOPS][eta 01m:31s]
Jobs: 4 (f=4): [R(4)][26.1%][r=10.1GiB/s][r=165k IOPS][eta 01m:28s]
Jobs: 4 (f=4): [R(4)][27.5%][r=10.1GiB/s][r=166k IOPS][eta 01m:27s]
Jobs: 4 (f=4): [R(4)][29.2%][r=9.00GiB/s][r=164k IOPS][eta 01m:25s]
Jobs: 4 (f=4): [R(4)][31.1%][r=9.99GiB/s][r=164k IOPS][eta 01m:22s]
Jobs: 4 (f=4): [R(4)][32.8%][r=10.1GiB/s][r=166k IOPS][eta 01m:20s]
Jobs: 4 (f=4): [R(4)][34.2%][r=10.1GiB/s][r=165k IOPS][eta 01m:19s]
Jobs: 4 (f=4): [R(4)][35.8%][r=10.1GiB/s][r=165k IOPS][eta 01m:17s]
Jobs: 4 (f=4): [R(4)][37.5%][r=10.2GiB/s][r=167k IOPS][eta 01m:15s]
Jobs: 4 (f=4): [R(4)][39.2%][r=10.6GiB/s][r=173k IOPS][eta 01m:13s]
Jobs: 4 (f=4): [R(4)][40.8%][r=10.5GiB/s][r=171k IOPS][eta 01m:11s]
Jobs: 4 (f=4): [R(4)][42.5%][r=10.2GiB/s][r=167k IOPS][eta 01m:09s]
Jobs: 4 (f=4): [R(4)][44.2%][r=10.0GiB/s][r=164k IOPS][eta 01m:07s]
Jobs: 4 (f=4): [R(4)][45.8%][r=9.00GiB/s][r=164k IOPS][eta 01m:05s]
Jobs: 4 (f=4): [R(4)][47.5%][r=10.2GiB/s][r=166k IOPS][eta 01m:03s]
Jobs: 4 (f=4): [R(4)][49.2%][r=9.98GiB/s][r=164k IOPS][eta 01m:01s]
Jobs: 4 (f=4): [R(4)][50.8%][r=10.1GiB/s][r=166k IOPS][eta 00m:59s]
Jobs: 4 (f=4): [R(4)][52.5%][r=10.2GiB/s][r=167k IOPS][eta 00m:57s]
Jobs: 4 (f=4): [R(4)][54.2%][r=10.0GiB/s][r=164k IOPS][eta 00m:55s]
Jobs: 4 (f=4): [R(4)][55.8%][r=9.98GiB/s][r=164k IOPS][eta 00m:53s]
Jobs: 4 (f=4): [R(4)][57.5%][r=9.97GiB/s][r=163k IOPS][eta 00m:51s]
Jobs: 4 (f=4): [R(4)][59.2%][r=10.1GiB/s][r=166k IOPS][eta 00m:49s]
Jobs: 4 (f=4): [R(4)][60.8%][r=10.0GiB/s][r=164k IOPS][eta 00m:47s]
Jobs: 4 (f=4): [R(4)][62.5%][r=10.2GiB/s][r=167k IOPS][eta 00m:45s]
Jobs: 4 (f=4): [R(4)][64.7%][r=10.6GiB/s][r=174k IOPS][eta 00m:42s]
Jobs: 4 (f=4): [R(4)][65.8%][r=10.7GiB/s][r=176k IOPS][eta 00m:41s]
Jobs: 4 (f=4): [R(4)][67.5%][r=10.9GiB/s][r=178k IOPS][eta 00m:39s]
Jobs: 4 (f=4): [R(4)][69.2%][r=10.0GiB/s][r=180k IOPS][eta 00m:37s]
Jobs: 4 (f=4): [R(4)][71.4%][r=11.6GiB/s][r=190k IOPS][eta 00m:34s]
Jobs: 4 (f=4): [R(4)][72.5%][r=11.4GiB/s][r=186k IOPS][eta 00m:33s]
Jobs: 4 (f=4): [R(4)][74.2%][r=10.5GiB/s][r=171k IOPS][eta 00m:31s]
Jobs: 4 (f=4): [R(4)][75.8%][r=10.6GiB/s][r=173k IOPS][eta 00m:29s]
Jobs: 4 (f=4): [R(4)][77.5%][r=10.2GiB/s][r=168k IOPS][eta 00m:27s]
Jobs: 4 (f=4): [R(4)][79.2%][r=10.1GiB/s][r=165k IOPS][eta 00m:25s]
Jobs: 4 (f=4): [R(4)][80.8%][r=9.99GiB/s][r=164k IOPS][eta 00m:23s]
Jobs: 4 (f=4): [R(4)][82.5%][r=10.1GiB/s][r=165k IOPS][eta 00m:21s]
Jobs: 4 (f=4): [R(4)][84.2%][r=10.0GiB/s][r=164k IOPS][eta 00m:19s]
Jobs: 4 (f=4): [R(4)][85.8%][r=10.1GiB/s][r=165k IOPS][eta 00m:17s]
Jobs: 4 (f=4): [R(4)][87.5%][r=10.1GiB/s][r=166k IOPS][eta 00m:15s]
Jobs: 4 (f=4): [R(4)][89.9%][r=10.1GiB/s][r=166k IOPS][eta 00m:12s]
Jobs: 4 (f=4): [R(4)][90.8%][r=10.0GiB/s][r=164k IOPS][eta 00m:11s]
Jobs: 4 (f=4): [R(4)][93.3%][r=10.1GiB/s][r=166k IOPS][eta 00m:08s]
Jobs: 4 (f=4): [R(4)][94.2%][r=10.1GiB/s][r=166k IOPS][eta 00m:07s]
Jobs: 4 (f=4): [R(4)][95.8%][r=10.1GiB/s][r=165k IOPS][eta 00m:05s]
Jobs: 4 (f=4): [R(4)][97.5%][r=10.2GiB/s][r=167k IOPS][eta 00m:03s]
Jobs: 4 (f=4): [R(4)][99.2%][r=10.6GiB/s][r=174k IOPS][eta 00m:01s]
Jobs: 4 (f=4): [R(4)][100.0%][r=11.1GiB/s][r=182k IOPS][eta 00m:00s]
throughput-test-job: (groupid=0, jobs=4): err= 0: pid=4053886: Fri Feb 17 05:51:14 2023
read: IOPS=168k, BW=10.2GiB/s (10.0GB/s)(1229GiB/120001msec)
slat (usec): min=4, max=8108, avg=22.94, stdev= 5.23
clat (nsec): min=1310, max=10324k, avg=1502601.52, stdev=106078.42
lat (usec): min=21, max=10356, avg=1525.63, stdev=107.50
clat percentiles (usec):
| 1.00th=[ 1188], 5.00th=[ 1336], 10.00th=[ 1418], 20.00th=[ 1467],
| 30.00th=[ 1483], 40.00th=[ 1500], 50.00th=[ 1500], 60.00th=[ 1516],
| 70.00th=[ 1532], 80.00th=[ 1549], 90.00th=[ 1582], 95.00th=[ 1614],
| 99.00th=[ 1827], 99.50th=[ 2008], 99.90th=[ 2376], 99.95th=[ 2442],
| 99.99th=[ 2704]
bw ( MiB/s): min= 8772, max=12988, per=100.00%, avg=10485.24, stdev=108.24, samples=956
iops : min=140356, max=207812, avg=167763.80, stdev=1731.85, samples=956
lat (usec) : 2=0.01%, 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%
lat (usec) : 750=0.01%, 1000=0.01%
lat (msec) : 2=99.49%, 4=0.50%, 10=0.01%, 20=0.01%
cpu : usr=3.72%, sys=96.26%, ctx=2028, majf=0, minf=2105
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=20130261,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=10.2GiB/s (10.0GB/s), 10.2GiB/s-10.2GiB/s (10.0GB/s-10.0GB/s), io=1229GiB (1319GB), run=120001-120001msec
I can remove them or add them as another metadata mirror. Since they’re low latency, they’re perfect for that workload. I just realized I already had 2 mirrors in my metadata already. Adding a third seems unnecessary.
I’ll make another thread with suggestions on the allocation size for different datasets. I’m storing tons of stuff, but only in certain places does performance really matter like images, videos, and backups.
I’m curious what I should use for Windows as well. When SSDs first came out, I set it to 64K, but these days, I leave it at the default 4K.
If 500MB/s is the most SSDs typically read, it makes sense that Wolves is at 10GB/s (but it averages at 11.5GB/s). That’s the max of 500 * 20.
That’s why the 34-drive zpool is frustrating because it’s also capping at the same amount.
I added a Test2 dataset to the 34-drive zpool with 1MB blocks.
# dd if=/dev/zero of=/mnt/Bunnies/Test2/fio4.tmp bs=1G count=1010+0 records in
10+0 records out
10737418240 bytes (11 GB, 10 GiB) copied, 4.49624 s, 2.4 GB/s
These numbers were a LOT slower. I don’t know how ZFS caches, but if I run these reads multiple times in a row, it’s the same speed each time.
Just as a sanity check, I did another fio from the Test dataset, and it was back to 10GB/s.
I also tried with 256K blocks and had similar read slowdowns, but only 1GB/s vs 4GB/s seen here.
I tried other things like disabling ZL4, sync writes, and not using ACLs, but there was no change.
# fio --filename=/mnt/Bunnies/Test2/fio4.tmp --direct=1 --rw=read --bs=64k --ioengine=libaio --iodepth=64 --runtime=120 --numjobs=4 --time_based --group_reporting --name=throughput-test-job --eta-newline=1 --readonly
throughput-test-job: (g=0): rw=read, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64
...
fio-3.25
Starting 4 processes
Jobs: 4 (f=4): [R(4)][2.5%][r=6485MiB/s][r=104k IOPS][eta 01m:57s]
Jobs: 4 (f=4): [R(4)][4.2%][r=6387MiB/s][r=102k IOPS][eta 01m:55s]
Jobs: 4 (f=4): [R(4)][5.8%][r=6206MiB/s][r=99.3k IOPS][eta 01m:53s]
Jobs: 4 (f=4): [R(4)][7.5%][r=6289MiB/s][r=101k IOPS][eta 01m:51s]
Jobs: 4 (f=4): [R(4)][9.2%][r=6175MiB/s][r=98.8k IOPS][eta 01m:49s]
Jobs: 4 (f=4): [R(4)][10.9%][r=5966MiB/s][r=95.5k IOPS][eta 01m:46s]
Jobs: 4 (f=4): [R(4)][12.5%][r=5961MiB/s][r=95.4k IOPS][eta 01m:45s]
Jobs: 4 (f=4): [R(4)][14.2%][r=5924MiB/s][r=94.8k IOPS][eta 01m:43s]
Jobs: 4 (f=4): [R(4)][15.8%][r=6050MiB/s][r=96.8k IOPS][eta 01m:41s]
Jobs: 4 (f=4): [R(4)][17.5%][r=6119MiB/s][r=97.9k IOPS][eta 01m:39s]
Jobs: 4 (f=4): [R(4)][19.2%][r=6198MiB/s][r=99.2k IOPS][eta 01m:37s]
Jobs: 4 (f=4): [R(4)][20.8%][r=6398MiB/s][r=102k IOPS][eta 01m:35s]
Jobs: 4 (f=4): [R(4)][22.5%][r=6299MiB/s][r=101k IOPS][eta 01m:33s]
Jobs: 4 (f=4): [R(4)][24.2%][r=6480MiB/s][r=104k IOPS][eta 01m:31s]
Jobs: 4 (f=4): [R(4)][26.1%][r=6503MiB/s][r=104k IOPS][eta 01m:28s]
Jobs: 4 (f=4): [R(4)][27.5%][r=6473MiB/s][r=104k IOPS][eta 01m:27s]
Jobs: 4 (f=4): [R(4)][29.2%][r=6321MiB/s][r=101k IOPS][eta 01m:25s]
Jobs: 4 (f=4): [R(4)][31.1%][r=6117MiB/s][r=97.9k IOPS][eta 01m:22s]
Jobs: 4 (f=4): [R(4)][32.8%][r=6039MiB/s][r=96.6k IOPS][eta 01m:20s]
Jobs: 4 (f=4): [R(4)][34.2%][r=6219MiB/s][r=99.5k IOPS][eta 01m:19s]
Jobs: 4 (f=4): [R(4)][35.8%][r=6073MiB/s][r=97.2k IOPS][eta 01m:17s]
Jobs: 4 (f=4): [R(4)][37.5%][r=6244MiB/s][r=99.9k IOPS][eta 01m:15s]
Jobs: 4 (f=4): [R(4)][39.2%][r=6391MiB/s][r=102k IOPS][eta 01m:13s]
Jobs: 4 (f=4): [R(4)][40.8%][r=6535MiB/s][r=105k IOPS][eta 01m:11s]
Jobs: 4 (f=4): [R(4)][42.5%][r=6416MiB/s][r=103k IOPS][eta 01m:09s]
Jobs: 4 (f=4): [R(4)][44.2%][r=6415MiB/s][r=103k IOPS][eta 01m:07s]
Jobs: 4 (f=4): [R(4)][45.8%][r=6455MiB/s][r=103k IOPS][eta 01m:05s]
Jobs: 4 (f=4): [R(4)][47.5%][r=6419MiB/s][r=103k IOPS][eta 01m:03s]
Jobs: 4 (f=4): [R(4)][49.2%][r=6331MiB/s][r=101k IOPS][eta 01m:01s]
Jobs: 4 (f=4): [R(4)][50.8%][r=6406MiB/s][r=102k IOPS][eta 00m:59s]
Jobs: 4 (f=4): [R(4)][52.5%][r=6401MiB/s][r=102k IOPS][eta 00m:57s]
Jobs: 4 (f=4): [R(4)][54.2%][r=6498MiB/s][r=104k IOPS][eta 00m:55s]
Jobs: 4 (f=4): [R(4)][55.8%][r=6587MiB/s][r=105k IOPS][eta 00m:53s]
Jobs: 4 (f=4): [R(4)][57.5%][r=6400MiB/s][r=102k IOPS][eta 00m:51s]
Jobs: 4 (f=4): [R(4)][59.2%][r=6512MiB/s][r=104k IOPS][eta 00m:49s]
Jobs: 4 (f=4): [R(4)][60.8%][r=6257MiB/s][r=100k IOPS][eta 00m:47s]
Jobs: 4 (f=4): [R(4)][62.5%][r=6255MiB/s][r=100k IOPS][eta 00m:45s]
Jobs: 4 (f=4): [R(4)][64.7%][r=6583MiB/s][r=105k IOPS][eta 00m:42s]
Jobs: 4 (f=4): [R(4)][66.1%][r=6474MiB/s][r=104k IOPS][eta 00m:41s]
Jobs: 4 (f=4): [R(4)][67.5%][r=6509MiB/s][r=104k IOPS][eta 00m:39s]
Jobs: 4 (f=4): [R(4)][69.2%][r=6516MiB/s][r=104k IOPS][eta 00m:37s]
Jobs: 4 (f=4): [R(4)][70.2%][r=6572MiB/s][r=105k IOPS][eta 00m:36s]
Jobs: 4 (f=4): [R(4)][71.9%][r=6615MiB/s][r=106k IOPS][eta 00m:34s]
Jobs: 4 (f=4): [R(4)][73.6%][r=6372MiB/s][r=102k IOPS][eta 00m:32s]
Jobs: 4 (f=4): [R(4)][75.2%][r=6314MiB/s][r=101k IOPS][eta 00m:30s]
Jobs: 4 (f=4): [R(4)][76.9%][r=6369MiB/s][r=102k IOPS][eta 00m:28s]
Jobs: 4 (f=4): [R(4)][78.5%][r=6155MiB/s][r=98.5k IOPS][eta 00m:26s]
Jobs: 4 (f=4): [R(4)][80.2%][r=6109MiB/s][r=97.7k IOPS][eta 00m:24s]
Jobs: 4 (f=4): [R(4)][81.8%][r=5862MiB/s][r=93.8k IOPS][eta 00m:22s]
Jobs: 4 (f=4): [R(4)][83.5%][r=5937MiB/s][r=94.0k IOPS][eta 00m:20s]
Jobs: 4 (f=4): [R(4)][85.1%][r=6021MiB/s][r=96.3k IOPS][eta 00m:18s]
Jobs: 4 (f=4): [R(4)][87.5%][r=5793MiB/s][r=92.7k IOPS][eta 00m:15s]
Jobs: 4 (f=4): [R(4)][88.4%][r=5807MiB/s][r=92.9k IOPS][eta 00m:14s]
Jobs: 4 (f=4): [R(4)][90.1%][r=5896MiB/s][r=94.3k IOPS][eta 00m:12s]
Jobs: 4 (f=4): [R(4)][91.7%][r=5989MiB/s][r=95.8k IOPS][eta 00m:10s]
Jobs: 4 (f=4): [R(4)][93.4%][r=5883MiB/s][r=94.1k IOPS][eta 00m:08s]
Jobs: 4 (f=4): [R(4)][95.0%][r=5766MiB/s][r=92.3k IOPS][eta 00m:06s]
Jobs: 4 (f=4): [R(4)][96.7%][r=5671MiB/s][r=90.7k IOPS][eta 00m:04s]
Jobs: 4 (f=4): [R(4)][98.3%][r=5665MiB/s][r=90.6k IOPS][eta 00m:02s]
Jobs: 4 (f=4): [R(4)][100.0%][r=5684MiB/s][r=90.9k IOPS][eta 00m:00s]
throughput-test-job: (groupid=0, jobs=4): err= 0: pid=226229: Fri Feb 17 06:15:50 2023
read: IOPS=99.9k, BW=6244MiB/s (6548MB/s)(732GiB/120001msec)
slat (usec): min=4, max=6330, avg=38.82, stdev= 7.87
clat (nsec): min=1480, max=8925.3k, avg=2522746.36, stdev=262050.70
lat (usec): min=35, max=8964, avg=2561.70, stdev=265.64
clat percentiles (usec):
| 1.00th=[ 2245], 5.00th=[ 2278], 10.00th=[ 2311], 20.00th=[ 2311],
| 30.00th=[ 2343], 40.00th=[ 2376], 50.00th=[ 2409], 60.00th=[ 2474],
| 70.00th=[ 2606], 80.00th=[ 2704], 90.00th=[ 2900], 95.00th=[ 3130],
| 99.00th=[ 3294], 99.50th=[ 3359], 99.90th=[ 3752], 99.95th=[ 4113],
| 99.99th=[ 4686]
bw ( MiB/s): min= 5513, max= 6786, per=100.00%, avg=6247.77, stdev=71.39, samples=956
iops : min=88218, max=108588, avg=99964.28, stdev=1142.26, samples=956
lat (usec) : 2=0.01%, 4=0.01%, 50=0.01%, 100=0.01%, 250=0.01%
lat (usec) : 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=99.94%, 10=0.06%
cpu : usr=2.93%, sys=97.05%, ctx=1787, majf=0, minf=2109
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=11988998,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=6244MiB/s (6548MB/s), 6244MiB/s-6244MiB/s (6548MB/s-6548MB/s), io=732GiB (786GB), run=120001-120001msec
Here we go! I set the fio block size from 64k to 128k and man that sped it up!
Read speed is there, but I’m not sure about write speed. Seems to be limited somehow. Maxing at 2GB/s? Is that normal or should it be higher?
So then back to my original question 
: why am I not able to saturate the link over fiber in the same way?
One appears to be some weird issue with either Explorer or my NVMe drives in Windows. But then the CrystalDiskMark write performance was 500MB/s slower over fiber. Like someone suggested before, it could be that those 25Gb packets are being buffered, but I thought there was some link speed negotiation that took place.
I’m trying to buy a used ConnectX-6 for my NAS as well, but I might end up buying new.
ZFS Block: 128k
fio Block: 128k
# fio --filename=/mnt/Bunnies/Test/fio4.tmp --direct=1 --rw=read --bs=128k --ioengine=libaio --iodepth=64 --runtime=120 --numjobs=4 --time_based --group_reporting --name=throughput-test-job --eta-newline=1 --readonly
throughput-test-job: (g=0): rw=read, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=64
...
fio-3.25
Starting 4 processes
Jobs: 4 (f=4): [R(4)][2.5%][r=18.4GiB/s][r=151k IOPS][eta 01m:57s]
Jobs: 4 (f=4): [R(4)][4.2%][r=17.7GiB/s][r=145k IOPS][eta 01m:55s]
Jobs: 4 (f=4): [R(4)][5.8%][r=17.5GiB/s][r=143k IOPS][eta 01m:53s]
Jobs: 4 (f=4): [R(4)][7.5%][r=17.8GiB/s][r=146k IOPS][eta 01m:51s]
Jobs: 4 (f=4): [R(4)][9.2%][r=17.8GiB/s][r=146k IOPS][eta 01m:49s]
Jobs: 4 (f=4): [R(4)][10.9%][r=17.7GiB/s][r=145k IOPS][eta 01m:46s]
^Cbs: 4 (f=4): [R(4)][12.5%][r=17.9GiB/s][r=147k IOPS][eta 01m:45s]
fio: terminating on signal 2
throughput-test-job: (groupid=0, jobs=4): err= 0: pid=329360: Fri Feb 17 06:37:31 2023
read: IOPS=149k, BW=18.1GiB/s (19.5GB/s)(277GiB/15279msec)
slat (usec): min=11, max=520, avg=25.90, stdev= 4.46
clat (nsec): min=1530, max=5103.3k, avg=1696057.51, stdev=164266.31
lat (usec): min=25, max=5164, avg=1722.07, stdev=166.66
clat percentiles (usec):
| 1.00th=[ 1221], 5.00th=[ 1336], 10.00th=[ 1516], 20.00th=[ 1631],
| 30.00th=[ 1663], 40.00th=[ 1696], 50.00th=[ 1713], 60.00th=[ 1729],
| 70.00th=[ 1762], 80.00th=[ 1778], 90.00th=[ 1844], 95.00th=[ 1893],
| 99.00th=[ 2114], 99.50th=[ 2245], 99.90th=[ 2704], 99.95th=[ 3032],
| 99.99th=[ 4424]
bw ( MiB/s): min=17490, max=22232, per=100.00%, avg=18588.56, stdev=333.00, samples=120
iops : min=139924, max=177860, avg=148708.47, stdev=2663.97, samples=120
lat (usec) : 2=0.01%, 4=0.01%, 50=0.01%, 100=0.01%, 250=0.01%
lat (usec) : 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=98.29%, 4=1.68%, 10=0.02%
cpu : usr=3.82%, sys=96.16%, ctx=277, majf=0, minf=2116
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=2270298,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=18.1GiB/s (19.5GB/s), 18.1GiB/s-18.1GiB/s (19.5GB/s-19.5GB/s), io=277GiB (298GB), run=15279-15279msec
ZFS Block: 1M
fio Block: 256k
# fio --filename=/mnt/Bunnies/Test2/fio4.tmp --direct=1 --rw=read --bs=256k --ioengine=libaio --iodepth=64 --runtime=120 --numjobs=4 --time_based --group_reporting --name=throughput-test-job --eta-newline=1 --readonly
throughput-test-job: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.25
Starting 4 processes
Jobs: 4 (f=4): [R(4)][2.5%][r=12.5GiB/s][r=51.3k IOPS][eta 01m:57s]
Jobs: 4 (f=4): [R(4)][4.2%][r=11.2GiB/s][r=45.7k IOPS][eta 01m:55s]
Jobs: 4 (f=4): [R(4)][5.8%][r=12.6GiB/s][r=51.6k IOPS][eta 01m:53s]
Jobs: 4 (f=4): [R(4)][7.5%][r=12.6GiB/s][r=51.4k IOPS][eta 01m:51s]
Jobs: 4 (f=4): [R(4)][9.2%][r=12.4GiB/s][r=50.9k IOPS][eta 01m:49s]
Jobs: 4 (f=4): [R(4)][10.9%][r=11.8GiB/s][r=48.5k IOPS][eta 01m:46s]
^Cbs: 4 (f=4): [R(4)][12.5%][r=11.5GiB/s][r=47.2k IOPS][eta 01m:45s]
fio: terminating on signal 2
throughput-test-job: (groupid=0, jobs=4): err= 0: pid=334710: Fri Feb 17 06:39:14 2023
read: IOPS=49.1k, BW=11.0GiB/s (12.9GB/s)(188GiB/15665msec)
slat (usec): min=17, max=2047, avg=80.20, stdev=15.39
clat (nsec): min=1800, max=13426k, avg=5127752.41, stdev=592162.92
lat (usec): min=89, max=13622, avg=5208.07, stdev=601.05
clat percentiles (usec):
| 1.00th=[ 4686], 5.00th=[ 4686], 10.00th=[ 4752], 20.00th=[ 4817],
| 30.00th=[ 4817], 40.00th=[ 4883], 50.00th=[ 4948], 60.00th=[ 5014],
| 70.00th=[ 5145], 80.00th=[ 5276], 90.00th=[ 5800], 95.00th=[ 6259],
| 99.00th=[ 7373], 99.50th=[ 8848], 99.90th=[ 9765], 99.95th=[10552],
| 99.99th=[12387]
bw ( MiB/s): min= 9569, max=13106, per=100.00%, avg=12285.11, stdev=211.50, samples=124
iops : min=38276, max=52426, avg=49140.45, stdev=846.02, samples=124
lat (usec) : 2=0.01%, 4=0.01%, 100=0.01%, 250=0.01%, 500=0.01%
lat (usec) : 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=99.90%, 20=0.08%
cpu : usr=1.50%, sys=98.43%, ctx=351, majf=0, minf=2125
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=769716,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=11.0GiB/s (12.9GB/s), 11.0GiB/s-11.0GiB/s (12.9GB/s-12.9GB/s), io=188GiB (202GB), run=15665-15665msec
ZFS Block: 1M
fio Block: 512k
# fio --filename=/mnt/Bunnies/Test2/fio4.tmp --direct=1 --rw=read --bs=512k --ioengine=libaio --iodepth=64 --runtime=120 --numjobs=4 --time_based --group_reporting --name=throughput-test-job --eta-newline=1 --readonly
throughput-test-job: (g=0): rw=read, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=64
...
fio-3.25
Starting 4 processes
Jobs: 4 (f=4): [R(4)][2.5%][r=10.0GiB/s][r=22.5k IOPS][eta 01m:57s]
Jobs: 4 (f=4): [R(4)][5.0%][r=9.94GiB/s][r=20.4k IOPS][eta 01m:55s]
Jobs: 4 (f=4): [R(4)][6.6%][r=10.5GiB/s][r=21.5k IOPS][eta 01m:53s]
Jobs: 4 (f=4): [R(4)][8.3%][r=10.2GiB/s][r=20.9k IOPS][eta 01m:51s]
^Cbs: 4 (f=4): [R(4)][8.3%][r=10.3GiB/s][r=21.0k IOPS][eta 01m:50s]
fio: terminating on signal 2
throughput-test-job: (groupid=0, jobs=4): err= 0: pid=337417: Fri Feb 17 06:39:54 2023
read: IOPS=21.5k, BW=10.5GiB/s (11.3GB/s)(108GiB/10307msec)
slat (usec): min=31, max=2028, avg=184.13, stdev=44.28
clat (usec): min=2, max=21432, avg=11711.67, stdev=972.86
lat (usec): min=195, max=21715, avg=11896.01, stdev=985.81
clat percentiles (usec):
| 1.00th=[10159], 5.00th=[10421], 10.00th=[10683], 20.00th=[10945],
| 30.00th=[11207], 40.00th=[11469], 50.00th=[11731], 60.00th=[11863],
| 70.00th=[11994], 80.00th=[12256], 90.00th=[12649], 95.00th=[13173],
| 99.00th=[15008], 99.50th=[15926], 99.90th=[18744], 99.95th=[19530],
| 99.99th=[20841]
bw ( MiB/s): min= 9413, max=11528, per=100.00%, avg=10752.85, stdev=152.05, samples=80
iops : min=18826, max=23056, avg=21505.70, stdev=304.10, samples=80
lat (usec) : 4=0.01%, 10=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
lat (usec) : 1000=0.01%
lat (msec) : 2=0.01%, 4=0.02%, 10=0.35%, 20=99.57%, 50=0.04%
cpu : usr=1.01%, sys=98.90%, ctx=249, majf=0, minf=2165
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=221609,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=10.5GiB/s (11.3GB/s), 10.5GiB/s-10.5GiB/s (11.3GB/s-11.3GB/s), io=108GiB (116GB), run=10307-10307msec
ZFS Block: 1M
fio Block: 1M
# fio --filename=/mnt/Bunnies/Test2/fio4.tmp --direct=1 --rw=read --bs=1M --ioengine=libaio --iodepth=64 --runtime=120 --numjobs=4 --time_based --group_reporting --name=throughput-test-job --eta-newline=1 --readonly
throughput-test-job: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
...
fio-3.25
Starting 4 processes
Jobs: 4 (f=4): [R(4)][2.5%][r=11.5GiB/s][r=11.7k IOPS][eta 01m:57s]
Jobs: 4 (f=4): [R(4)][4.2%][r=11.4GiB/s][r=11.7k IOPS][eta 01m:55s]
Jobs: 4 (f=4): [R(4)][5.8%][r=11.5GiB/s][r=11.7k IOPS][eta 01m:53s]
^Cbs: 4 (f=4): [R(4)][6.7%][r=11.5GiB/s][r=11.8k IOPS][eta 01m:52s]
fio: terminating on signal 2
throughput-test-job: (groupid=0, jobs=4): err= 0: pid=333736: Fri Feb 17 06:38:46 2023
read: IOPS=11.6k, BW=11.3GiB/s (12.2GB/s)(97.7GiB/8609msec)
slat (usec): min=106, max=2240, avg=342.52, stdev=23.82
clat (usec): min=2, max=31929, avg=21657.58, stdev=1011.32
lat (usec): min=375, max=32335, avg=22000.30, stdev=1020.36
clat percentiles (usec):
| 1.00th=[19792], 5.00th=[20579], 10.00th=[20841], 20.00th=[21103],
| 30.00th=[21365], 40.00th=[21365], 50.00th=[21627], 60.00th=[21627],
| 70.00th=[21890], 80.00th=[22152], 90.00th=[22676], 95.00th=[22938],
| 99.00th=[23987], 99.50th=[24249], 99.90th=[27919], 99.95th=[29492],
| 99.99th=[31065]
bw ( MiB/s): min=10594, max=11914, per=99.87%, avg=11602.24, stdev=80.62, samples=68
iops : min=10594, max=11914, avg=11602.24, stdev=80.62, samples=68
lat (usec) : 4=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.02%, 10=0.07%, 20=1.36%, 50=98.53%
cpu : usr=0.51%, sys=99.43%, ctx=112, majf=0, minf=2225
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.7%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=100010,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=11.3GiB/s (12.2GB/s), 11.3GiB/s-11.3GiB/s (12.2GB/s-12.2GB/s), io=97.7GiB (105GB), run=8609-8609msec
ZFS block: 128k
fio block: 128k
# fio --filename=/mnt/Wolves/Test/fio4.tmp --direct=1 --rw=read --bs=128k --ioengine=libaio --iodepth=64 --runtime=120 --numjobs=4 --time_based --group_reporting --name=throughput-test-job --eta-newline=1 --readonly
throughput-test-job: (g=0): rw=read, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=64
...
fio-3.25
Starting 4 processes
Jobs: 4 (f=4): [R(4)][2.5%][r=11.5GiB/s][r=94.2k IOPS][eta 01m:57s]
Jobs: 4 (f=4): [R(4)][4.2%][r=11.9GiB/s][r=97.1k IOPS][eta 01m:55s]
Jobs: 4 (f=4): [R(4)][5.8%][r=11.3GiB/s][r=92.3k IOPS][eta 01m:53s]
Jobs: 4 (f=4): [R(4)][7.5%][r=11.5GiB/s][r=94.5k IOPS][eta 01m:51s]
Jobs: 4 (f=4): [R(4)][9.2%][r=11.2GiB/s][r=91.5k IOPS][eta 01m:49s]
^Cbs: 4 (f=4): [R(4)][10.9%][r=11.1GiB/s][r=90.8k IOPS][eta 01m:46s]
fio: terminating on signal 2
throughput-test-job: (groupid=0, jobs=4): err= 0: pid=394673: Fri Feb 17 06:55:55 2023
read: IOPS=91.2k, BW=11.1GiB/s (11.0GB/s)(146GiB/13110msec)
slat (usec): min=9, max=11930, avg=42.74, stdev=28.66
clat (nsec): min=1130, max=14835k, avg=2762550.82, stdev=463736.72
lat (usec): min=48, max=14880, avg=2805.44, stdev=470.26
clat percentiles (usec):
| 1.00th=[ 2245], 5.00th=[ 2278], 10.00th=[ 2343], 20.00th=[ 2409],
| 30.00th=[ 2573], 40.00th=[ 2769], 50.00th=[ 2802], 60.00th=[ 2802],
| 70.00th=[ 2835], 80.00th=[ 2868], 90.00th=[ 2933], 95.00th=[ 3130],
| 99.00th=[ 4686], 99.50th=[ 5080], 99.90th=[ 5604], 99.95th=[ 6456],
| 99.99th=[13960]
bw ( MiB/s): min= 7307, max=13370, per=100.00%, avg=11408.26, stdev=306.79, samples=104
iops : min=58456, max=106962, avg=91266.15, stdev=2454.35, samples=104
lat (usec) : 2=0.01%, 4=0.01%, 50=0.01%, 100=0.01%, 250=0.01%
lat (usec) : 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.23%, 4=96.09%, 10=3.66%, 20=0.02%
cpu : usr=2.73%, sys=73.63%, ctx=864292, majf=0, minf=2113
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=1195796,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=11.1GiB/s (11.0GB/s), 11.1GiB/s-11.1GiB/s (11.0GB/s-11.0GB/s), io=146GiB (157GB), run=13110-13110msec
Write speed test. Not sure if I’m doing this correctly, but it’s much faster than the fiber link. You’re right, I need 100Gb at least, 200Gb at most. Curious how to acquire those kinds of cards.
I’d do it just for kicks even though I’m lacking a 100Gb switch. Unless I can find one for 2 machines. The most I can do is 2x25Gb if I buy another ConnectX-6.
Still, this write speed is too slow. Could my log devices be slowing it down? The speed is the same for both 34 and 20 drive zpools.
# fio --filename=/mnt/Bunnies/Test/fio5.tmp --direct=1 --rw=write --bs=128k --ioengine=libaio --iodepth=64 --runtime=120 --numjobs=4 --time_based --group_reporting --name=throughput-test-job --eta-newline=1 --size=30G
throughput-test-job: (g=0): rw=write, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=64
...
fio-3.25
Starting 4 processes
throughput-test-job: Laying out IO file (1 file / 30720MiB)
Jobs: 4 (f=4): [W(4)][2.5%][w=6983MiB/s][w=55.9k IOPS][eta 01m:58s]
Jobs: 4 (f=4): [W(4)][4.1%][w=6508MiB/s][w=52.1k IOPS][eta 01m:56s]
Jobs: 4 (f=4): [W(4)][5.8%][w=4341MiB/s][w=34.7k IOPS][eta 01m:54s]
Jobs: 4 (f=4): [W(4)][6.6%][w=5161MiB/s][w=41.3k IOPS][eta 01m:53s]
Jobs: 4 (f=4): [W(4)][7.4%][w=4621MiB/s][w=36.0k IOPS][eta 01m:52s]
Jobs: 4 (f=4): [W(4)][8.3%][w=6235MiB/s][w=49.9k IOPS][eta 01m:51s]
Jobs: 4 (f=4): [W(4)][9.1%][w=5793MiB/s][w=46.3k IOPS][eta 01m:50s]
Jobs: 4 (f=4): [W(4)][9.9%][w=5637MiB/s][w=45.1k IOPS][eta 01m:49s]
^Cbs: 4 (f=4): [W(4)][10.7%][w=5796MiB/s][w=46.4k IOPS][eta 01m:48s]
fio: terminating on signal 2
^C
fio: terminating on signal 2
throughput-test-job: (groupid=0, jobs=4): err= 0: pid=441395: Fri Feb 17 07:01:00 2023
write: IOPS=46.7k, BW=5837MiB/s (6120MB/s)(68.4GiB/12002msec); 0 zone resets
slat (usec): min=17, max=79637, avg=83.64, stdev=644.95
clat (usec): min=2, max=98022, avg=5396.02, stdev=6404.77
lat (usec): min=58, max=98093, avg=5479.87, stdev=6462.56
clat percentiles (usec):
| 1.00th=[ 2024], 5.00th=[ 2212], 10.00th=[ 2311], 20.00th=[ 2507],
| 30.00th=[ 2704], 40.00th=[ 2933], 50.00th=[ 3261], 60.00th=[ 3589],
| 70.00th=[ 4047], 80.00th=[ 5669], 90.00th=[11207], 95.00th=[17695],
| 99.00th=[33162], 99.50th=[40633], 99.90th=[63177], 99.95th=[73925],
| 99.99th=[93848]
bw ( MiB/s): min= 3948, max= 7638, per=99.92%, avg=5832.12, stdev=254.26, samples=92
iops : min=31584, max=61110, avg=46656.39, stdev=2034.02, samples=92
lat (usec) : 4=0.01%, 10=0.01%, 100=0.01%, 250=0.01%, 500=0.01%
lat (usec) : 750=0.01%, 1000=0.01%
lat (msec) : 2=0.76%, 4=68.26%, 10=19.36%, 20=7.78%, 50=3.59%
lat (msec) : 100=0.24%
cpu : usr=7.42%, sys=46.21%, ctx=100782, majf=0, minf=57
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=0,560441,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
WRITE: bw=5837MiB/s (6120MB/s), 5837MiB/s-5837MiB/s (6120MB/s-6120MB/s), io=68.4GiB (73.5GB), run=12002-12002msec
test again without –direct=1, then we get the cache performance
It would be interesting to see your performance with ISCSI.
Is your Samba configuration default or are there any changes?
But please make a review if you bought it, especially how loud the Switch is
Your SLOG is only used for synchronous writes, i.e. databases or VM workloads.
I disabled sync, is not so important to me on my private system, usually I have the data anyway 2x
Been looking at this switch, problem is what I really need is more SFP’s not just 100gb. So the 16x SFP has been on my to buy list.
Without --direct=1 or even with --direct=0, I don’t see any change run running fio. It’d be nice if I could know which blocks are in cache, so I could try that file.
Are you sure nothing else uses SLOG? I have no VMs other than the TrueNAS apps, and those are on a different pool anyway.
Even if I did make a VM, my idea was to put it on the TrueNAS apps pool, but I could put it on my main pool too. Don’t have enough information to know either way.
Updating the firmware on these SSDs, I was gonna do it through a Windows VM.
There’s a rumor floating around regarding some strange behaviour in the memory management of Windows 11.
Would you care testing throughput with following setting disabled:
Windows Security → App & Browser Control → Exploit Protection Settings → Force randomization of images (Mandatory ASLR)
