To put to bed on “not getting full Optane performance”, I was able to reproduce the 1.5M IOPS 4KB Q8T8 test that @wendell posted here: How fast is a P5800X Optane?
I used this script: https://gist.github.com/snizovtsev/1e57da4bf1cdae16599d59619600ee07 with SIZE restored to 1024. fio --version fio-3.29-31-g2b3d . I think Wendell might have a different version of fio, but I got comparable results.
It looks like the issue was mostly just reproducing the right test for apples to apples numbers. Though the correct cable did help get full PCIE 4.0 performance from the P5800X.
I’m not sure if Allyn from Intel would like this ( Intel Optane and a Whole lotta IOPS: A Chat with Allyn Malventano - YouTube ) but I’m really seeing the value of NVME RAID w/ consumer stuff over Optane. 4KB Q8T8 for NVME RAID is at least same order of magnitude, and it’s hard for me to think of even a modern database workload where you wouldn’t be able to benefit from parallelism. Latency can definitely be a big deal for niche stuff (e.g. financials), also the durability is unrivaled.
I think for Optane to be more competitive, they need to release the drives in the M.2 form factor and in smaller sizes. You need to be able to have a higher ratio of Optane per PCIE lane. Maybe this is in the works and Intel was waiting to release CPUs / chipsets for PCIE 5.0? Maybe VROC team is behind a little?
The Samsung consumer 980 pro / 970 Plus have also fallen quite a bit in price lately despite the NAND shortage. For the retail price of a 1.6TB P5800X, or about 4x 400GB P5800X, today you can get about (retail) 14TB of Samsung 980 PRO. It’s hard to argue with 10x space when the perf (in RAID0) would be pretty close for most software.
I have a few other machine learning workloads to try for the P5800X, but for a lot of use cases (most deep learning, NAS, database, even large-scale virtualization?) it seems like NVME is the sweet spot. I’m hoping to see opportunities to put more PCIE lanes behind Optane-- it would become a much more appealing expansion versus memory.
Interesting Optane vs DDR4 RAMDISK:
- 4KB IOPS is roughly same order of magnitude, unlike NVME.
- Sequential read is higher than RAM, but maybe that’s just fio with tmpfs weirdness.
Optane 800GB P5800X
Sequential Read: 6368MB/s IOPS=6
Sequential Write: 4834MB/s IOPS=4
512KB Read: 4571MB/s IOPS=9142
512KB Write: 4346MB/s IOPS=8692
Sequential Q32T1 Read: 6320MB/s IOPS=197
Sequential Q32T1 Write: 5300MB/s IOPS=165
4KB Read: 463MB/s IOPS=118746
4KB Write: 388MB/s IOPS=99538
4KB Q32T1 Read: 1257MB/s IOPS=321965
4KB Q32T1 Write: 874MB/s IOPS=223939
4KB Q8T8 Read: 6097MB/s IOPS=1560904
4KB Q8T8 Write: 4954MB/s IOPS=1268395
For comparison, 4x Samsung 970 Evo Plus in mdadm RAID0:
Sequential Read: 9481MB/s IOPS=9
Sequential Write: 9922MB/s IOPS=9
512KB Read: 2600MB/s IOPS=5200
512KB Write: 2587MB/s IOPS=5174
Sequential Q32T1 Read: 9715MB/s IOPS=303
Sequential Q32T1 Write: 12337MB/s IOPS=385
4KB Read: 65MB/s IOPS=16734
4KB Write: 250MB/s IOPS=64068
4KB Q32T1 Read: 1328MB/s IOPS=340181
4KB Q32T1 Write: 1116MB/s IOPS=285933
4KB Q8T8 Read: 3665MB/s IOPS=938364
4KB Q8T8 Write: 5595MB/s IOPS=1432506
4x 980 Pro RAID 0:
Sequential Read: 11962MB/s IOPS=11
Sequential Write: 13726MB/s IOPS=13
512KB Read: 2831MB/s IOPS=5663
512KB Write: 3610MB/s IOPS=7221
Sequential Q32T1 Read: 11130MB/s IOPS=347
Sequential Q32T1 Write: 12736MB/s IOPS=398
4KB Read: 90MB/s IOPS=23189
4KB Write: 291MB/s IOPS=74650
4KB Q32T1 Read: 1153MB/s IOPS=295340
4KB Q32T1 Write: 790MB/s IOPS=202365
4KB Q8T8 Read: 4134MB/s IOPS=1058490
4KB Q8T8 Write: 4693MB/s IOPS=1201456
Ramdisk (8x DDR4):
Sequential Read: 4413MB/s IOPS=4
Sequential Write: 4684MB/s IOPS=4
512KB Read: 8722MB/s IOPS=17444
512KB Write: 5412MB/s IOPS=10824
Sequential Q32T1 Read: 4417MB/s IOPS=138
Sequential Q32T1 Write: 4688MB/s IOPS=146
4KB Read: 3080MB/s IOPS=788640
4KB Write: 2583MB/s IOPS=661311
4KB Q32T1 Read: 3108MB/s IOPS=795822
4KB Q32T1 Write: 2442MB/s IOPS=625343
4KB Q8T8 Read: 23644MB/s IOPS=6052940
4KB Q8T8 Write: 3015MB/s IOPS=771917