Im confused about why optane is so good. Wendell said in one of his videos that even a “mid level modern nand nvme” would beat optane in some cases. Is this the case for a windows boot disk? What about to run Vmware workstation vms from?
The thing about optane is it had very low latency, and was very consistent in the latency it had.
Capacity-wise it might be difficult to have multiple VMs running from it depending on the size of your drive and what you’re doing with it.
2 Likes
They were cheap and fairly fast, nothing special though…
Optane will respond to a data request much faster than NAND NVME (better latency), but it can’t accomplish the same bandwidth as a top-of-the-line NAND NVME drive can.
Think of Optane as a motorcycle, great acceleration from stop light to stop light; and think of a NVME NAND drive as a big heavy grand touring car, can reach high top speeds on the highway but doesn’t accelerate stoplight to stoplight very well.
Low queue depth storage workloads are like the stop light to stoplight case and high queue depth loads are like the highway.
2 Likes
Still dont understand. Could you tell me the use-cases that Optane is good for? Then I will get it. I hope. 
Low queue depth scenarios would basically be any time the computer doesn’t know exactly what data should come next like during searches for files, or when a program needs to accesses a bunch of small files randomly (like loading a game).
The high queue depth workloads would be transferring large files like a large file copy or perhaps streaming in some ridiculous 8k texture pack into memory upon game load.
It’s the low queue depth performance that makes a computer feel snappy, which is why Optane got such a cult following… well that snappiness and that it doesn’t wear out as fast as NAND.
Thank you. So Im assuming VMs would benefit?
VMs are definitely a kind of workload that would benefit from Optane (lots of low queue depth request from VMs generally)… but Optane is also kind of expensive for the amount of storage you get for a specific price so that may work against it too.
Ok great. Another question. I am thinking of putting an optane drive in my SSD Truenas system. If I did that and booted from iscsi from a Mellanox 100gbe, could I acheive NVME speeds if I used Optane as a SLOG/L2ARC or other type of cache?
hmmm… that setup would add a lot of overhead over “native” performance of Optane. I think that extra overhead from the NIC and to a lesser degree ZFS would be enough added latency to make Optane not worth it in my book. While technically it would have better latency than if you used NAND, it would only be marginally better than NAND overall.
I don’t have performance figures to back this up but that is my “gut feeling”
1 Like
Yes but my concern is the durability of the caching. Hence the optane for it. But yeah I see your point. What do you recommend for high durability and fast storage for truenas caching? Is there something cheaper and better than optane?
I’m sure there is something better out there but I’ve been happy with a Samsung PM9A3. the 4TB version has about the same write endurance as 480GB 905P Optane drive all while being cheaper and larger.
Since NAND drives can come in much larger sizes than Optane drives, their write endurance figures aren’t nearly as bad as some would have you believe.
One of the best use cases are busy databases. Busy=many different questions/many transactions being asked concurrently. The data volume is larger than fits into faster but even more expensive RAM.
Optane performance in this scenario is closer to RAM than it is to Nand SSDs, so is its price.
Optane is great in a ZFS based NAS, but for a different quality than speed. It’s extremely durable compared to Nand SSDs. No need for trim action and Optane sustains its performance even when filled to the brim.
The latency overhead of booting over a network is much higher than the Optane latency - in a way that would be a waste.
Also, the vdevs of a ZFS pool define the general performance characteristics of the pool. An Optane or Nand SSD can be used to overcome performance weaknesses in regards to small blocksizes or concurrency or sync requests, but these will not magically create top performance if the vdevs don’t offer that performance in the first place.
1 Like
My plan was mirrored vdevs of SSDs
What is the connection between size and write endurance? I thought the endurance was due to other factors.
Total write endurance almost always scales linearly with drive size; so a larger drive with comparatively inferior write endurance to Optane can sometimes still beat it in write endurance simply by being much larger.
For current enterprise TLC drives, they need to be about an order of magnitude larger than Optane drives in order to match their write endurance.
So in other words, a 7tb drive could possible match an 700gb optane drive in write endurance?
Yup. Also notice I specified enterprise TLC drive; most consumer TLC drives are ~2-4 times worse at endurance than enterprise TLC drives.
1 Like
That depends on which Optane product you are comparing to. I’ve been going through the product briefs for a number Optane products. It’s surprising how unequal they are—by almost 2 orders magnitude. I’ll let the numbers do the talking:
-
P5800X: 100 DWPD
- 400 GB: 73 PB lifetime
- 800 GB: 146 PB lifetime
- 1.6 TB: 292 PB lifetime
- 3.2 TB: 584 PB lifetime
-
P4800X: (DWPD varies)
- 100 GB: 11 PB lifetime (60 DWPD)
- 375 GB: 21 PB lifetime (30 DWPD)
- 750 GB: 41 PB lifetime (30 DWPD)
- 1.5 TB: 164 PB lifetime (60 DWPD)
-
905P: 10 DWPD
- 380 GB: 7 PB lifetime
- 480 GB: 9 PB lifetime
- 960 GB: 18 PB lifetime
- 1.5 TB: 27 PB lifetime
-
900P: 10 DWPD
- 280 GB: 5 PB lifetime
- 480 GB: 9 PB lifetime
-
Memory M10: (DWPD varies)
- 16 GB: 365 TB lifetime (13 DWPD)
- 32 GB: 365 TB lifetime (6 DWPD)
- 64 GB: 365 TB lifetime (3 DWPD)
-
Memory: 3 DWPD
- 32 GB: 182.5 TB lifetime
584 PB lifetime write for the 3.2 TB P5800X is hard to beat. But cost-wise with respect to endurance, it could be considered cheap. Let’s do a price comparison with a mix of obtanium and unoptanium products:
-
Micron XTR (1.92 TB): $675/210 PB; $3/PB
- This uses SLC NAND and was specifically engineered to encroach on Optane’s endurance advantage while being much cheaper. It can match the 60 DWPD of Intel’s P4800X for certain write loads, although Micron itself notes that it does not also try to match Optane’s latency.
- Intel P4800X (1.5 TB): $900/164 PB; $5/PB
-
Intel 900P (280 GB): $40/5 PB; $8/PB
- These can often be found on eBay for as low as $40.
-
Intel P5800X (3.2 TB): $5,400/584 PB; $9/PB
- This has the lowest latency with 512-byte random reads competed in 3 μs on average, and 4-kibibyte reads completed in 5 μs on average. A typical NAND SSD’s random read latency is measured on the order of tens of μs.
- Intel Optane 905P (960 GB): $339/18 PB; $19/PB
- Micron 9400 MAX (25.6 TB): $3,300/140 PB; $24/PB
- Samsung PM9A3 (3.84 TB): $346/7 PB; $49/PB
-
Intel 670P (2 TB): $70/740 TB; $95/PB
- This is among the cheapest SSDs available with respect to price-capacity ratio.
- Samsung 990 PRO (2 TB): $160/1 PB; $133/PB*
* Calculations use unrounded numbers.
If latency does not matter, the winner is clear; get the Micron XTR SSD when it becomes available.
5 Likes
Thanks for the chart @LiKenun !
Here’s info on a couple more:
- Intel P1600X (118gb): $59/1292TBW; $46/PB
- Intel P1600X (58gb): $33/635TBW; $52/PB
2 Likes