Wendell on Gamers Nexus - Enterprise SSD's will noticeably enhance desktop workloads?

Hardware Hub Storage
1

In this video at around 6:30 with Gamers Nexus: https://youtu.be/5Fz3P41emo8?si=ZVccE4z75t9NNV6R

Wendell says paraphrased: “The highest end enterprise drive used in a gaming computer would feel night and day different” he goes on to say it has to do with random IO latency.

Here is a chart of the Random Read Latency of some of the best consumer ssd’s on the market. Does anyone know how some of the enterprise SSD’s Wendell is talking about would compare to these? It’s difficult to find equivalent data on those drives to compare to the table here.

I hope Wendell makes a video on this subject in the future as I find it fascinating!

Lmd9UMLETrei3kBV68kAtn-970-80.png
Lmd9UMLETrei3kBV68kAtn-970-80.png970×546 22.3 KB

1 Like
2

Wasn’t it one of the Kioxia Enterprise units he showed off ?

I think he mentioned it was one of the lower capacity 4TB units but that the Enterprise devices don’t suffer from that issue he described.

Pretty sure the issue is how the design and firmware works on these devices vs. consumer units which are more price focused.

3

The difference here is that Crystal Disk Mark is being run in ideal conditions, basically fresh drives, and the point being made about the enterprise-class products is that while being used and filling up with data, consumer drives start very performant and that degrades substantially as things like cache and SLC get filled, where enterprise is designed to sustain performance during use.

8 Likes
4

These are good points. The big question in my mind is do the enterprise drives still perform considerably better under these ideal conditions? Or does that only come into play once a consumer drive is getting full exhausting the pSLC (or whatever the drive uses) cache?

Meaning can I as a consumer who just wants the most performant desktop experience possible buy a 2tb or 4tb consumer NVME drive that performs well on that chart, never use more than say 25% of the drive capacity hopefully maintaining similar levels of performance to the chart and thus negating any benefit of the enterprise drives Wendell is talking about?

5

Probably IF you are only going to use less than 50 % usage of that SSD.

I like to always fill mine near capacity and had not seen issues with performance however I don’t go out of my way to run Crystal Diskmark when they get to less than 10 % free capacity.

I have a set of both NVMe Gen 3 and Gen 4 SSD’s along with some SATA SSD’s.

6

It isn’t just exhaustion of pSLC cache that contributes to declining performance with consumer NVMe; another dynamic at play is NAND cell charge decay, consumer NVMe suffers from it while enterprise NVMe often doesn’t. Charge decay can degrade performance (bandwidth and latency) by 1-2 orders of magnitude over the course of 1-2 years on consumer NVMe, of course the usual benchmarking sites never capture this behavior.

3 Likes
7

Now that’s very interesting! Is this decay permanent or does it only affect data that has been at rest for that amount of time?

8

I think it is as well. I really really wish someone would take up investigating this behavior because it affects all us normies without enterprise SSDs.

Here’s a very poor attempt at documenting this phenomenon:

The decay only affects cells that were written to many months ago, and weren’t updated by garbage collection, trim or the user. Garbage collection could fix the issue if it went over all occupied NAND cells, but it doesn’t do this on consumer NVMe. A deliberate rewrite of NAND cells will fix the degradation though, as evidenced in the linked post.

2 Likes
9

Optane?
Like Intel p4800x, p5800x : day/night difference with any consumer/prosumer gaming SSD.

1 Like
10

Best storage for gaming is DRAM. So get your filesystem cache going. Storage really only matters on cold first start.

I don’t think storage is relevant to gaming in 2023. I’m bottlenecked by (gaming)software all the times, certainly not storage.

If you have to wait for 30sec of ads when starting a AAA game, a difference of 50ms is trivial.

1 Like
11

TLDR: Enteprise ssds outside optane and pSLC specialty models will not perform significantly better than average consumer drive. Will likely perform worse than top of line consumer models. Nearly impossible to get hard proof though.

TLDR2: Even speciality Z-NAND drive is not uniformly superior to top of line m.2 consumer product for the same price (new. vs second hand). To get +80% RR, you will have to sacrifice 30% RR@64T and 50% SEQ.

I have been trying to answer the same question as OP to myself, but there is almost no available data. Just indirect hints from other sources and random blog posts from enthusiasts that buy them second hand for shit and giggles.
So let me dust off my onenote scratchpad, and regurgitate what I found over the last year of poking around.

There are few specialized review sites that do enterprise grade products, but the reviews are tailored to domain experts and system integrators.

Random read at low queue depth is irrelevant metric in that area, since enterprise workload is almost never limited by that.
In rare occasions it is, its not something normal enterprise models are suited for, and its serviced by extremely specialized optane or Z-NAND/XL-FLASH nand products from samsung/micron.

What is important in this area is:

  • performance at heavy concurrent load
  • latency depending on increasing concurrent load
  • sustained performance levels depending on increasing concurrent load (hours+, not minutes like consumer products)
  • durability (not explicitly tested), DPWD levels
  • overall reliability under any operating conditions (not tested)

On design side this is achieved by:

  • increased levels of nand channels, increasing performance by leveraging parallelism
    • this is direct opposite of consumer sector due to cost increases
  • high or obscene nand overprovisioning (sandwich boards are not rare in high capacity models)
  • unknown controller differences and certification
    • pausing work to perform garbage collection is not permissible on device that is hammered by request 24/7 for example
  • nvme zoning and QoS support
  • dedicated ram or nvram write buffers (I’ve seen one product with small PCM cache even)
  • internal capacitors in case of power loss to safely flush data to nand
  • certified and well though early error detection and error handling
    • enteprise drive will rather offline itself early that risk mangling data, second hand shoppers are sometimes bit by this behaviour. Absolutely okay if you running an array and have spare ready, but massive headache for home user.

None of this, beyond increased amount of nand channels is capable of increasing random RW at low que depths.

I had found only one indicative benchmark for KIOXIA CD6-R U.3 Gen4 NVMe 3.84T SSD.

Results are much worse than you would expect, notice 4K value:

1687503656-194929445-g
1687503656-194929445-g998×469 181 KB

Lets compare it to normal high end consumer M.2 samsung 990 PRO:

image
image1313×1325 117 KB

So I would summarize my personal finding as:

“The highest end enterprise drive used in a gaming computer would feel night and day different” does not hold water outside few very special cases.

Why?

Desktop workload and enterprise workload do not meaningfully overlap and products in each segment are optimized for segment (duh :slight_smile: ).
While enterprise hardware is generally much more powerful, durable and stable, its not optimized at all for desktop use. Also the same performance constraints that limit consumer products are present here as well.
Its very nice to get them second hand, but if you want performance, you are looking in wrong space.

There are specialized enterprise products that could fill this desire, specifically the original products created to combat optane while it still existed as competition.
They are extremely hard to get and were extremely pricy. Most were not sold directly to anyone but OEMs.

Price was accordingly published as call us or no data available.

If you, dear reader, are interested in going into that rabbit hole, look for Z-NAND second gen product SAMSUNG SZ1735. One performance fiend bough recycled unit and tested in on LTT forum. Behold:

image
image1136×1064 92.6 KB

While original asking price for this unit was somewhere at least north of 3800 USD (todays price), it can be bought now on ebay for 400 USD.

4 Likes
12

That is the only metric I really check out for storage.

Consistent trhoughput for initial onboarding/saving is nice, but all drives basically saturate the origional source (internet)

Enterprise excells at caching, because en mass, the multiple users, often require similar data.

Consumers, from my non scientific view, are more random in what they want at a time, though certain files are accessed hundreds of times more commonly (OS fikes) than even game or media files?

13

Exactly, consumer workload can be generally summarized as mix of three modes:

  • random RW and QD <4, most often QD = 1
  • short bursts of SEQ RW which fit into pSLC cache area
  • rare long SEQ RW bursts that exhausts pSLC cache

Enteprise ssds are noticeably better only in third access pattern, while sacrificing effective performance in patterns 1 and 2.

Caveat emptor and do your research. Or get optane.

2 Likes
14

I was specifically thinking of the P5800X. Possibly the solidigm P5810 z-nand-like device since it is very very low latency too, but also nand.

So what gives me pause on consumer nand is more the direction its going in than cherry picking the best of the best currently on consumer drives.

For example, the Samsung 990 Pro is somewhat worse when near full than the 980 Pros. And if you routinely shuttle around 100-200gb projects, read and write, then a higher capacity 8/16tb u.2 nvme enterprise drives like the CM7 will generally hold up better than m.2 counterparts, ecen samsung, but especially cheaper consumer grade m.2. Once the consumer drives have a little wear on the nand cells read and write perf usually drops, sometimes significantly, which is part of the annoyance.

Most of the consumer tests also test the consumer drives in an idealized scenario and generally yes, that looks better on paper. But if you have a workstation doing workstation stuff, the enterprise nand is often a better choice, moreso going forward looking at it int he context of the 990>980 regressions. And that’ll be true moving to 5 level cells for bulk storage. (Real world? I’d rather have the 980 pro. Unless I was going to run the 990 pro drive empty or nearly so.).

The place to start is daily driving optane, which I think you’ve already conceded, is night and day difference, even against the best “consumer” drives like the 990 pros, and yes P5800x optane itself is an enterprise drive.

I was also thinking of consumer drives as the lower end kingston, pny, inland etc brands that pop up here where the performance really tanks in mixed workloads vs “commodity” “cheap” enterprise drives showing up on ebay. Its not atypical to be able to get a CD6 on ebay for ~$50/tb, used, but with plenty of wear left.

It does merit investigation; the P5800X is night and day different. But Maybe also the gen3 $400 1.5tb 905 optane would also be noticeably different for daily-driver use cases (games, working on videos/large projects/etc), maybe a raid 0 of the same would also be worth testing. Vs “commodity” consumer m.2 nand?

also true if you just Have More Ram then the pressure on the drive is greatly lessened. Nothing beats ram for speed…

Ps. you can also play games with namespaces on modern enterprise u.2 to get extra performance, depending on your workload:

4 Likes
15

I also wanted to get 980 PRO instead of 990, as last good MLC drive ™. Apparently not, 970 PRO was the last. Holy techpowerup ssd databse confirms that.

But prices never fallen to sane level (europe and small local market) and 990 PRO pricing was simply too good. 2TB drive has enough dynamic SLC cache so that I have never run out of it with my workloads.

Side benefit is also newer more energy efficient controller, that reduces incidences of heat induced throttling.

Compared to that, I cannot justify second hand enteprise drives. Too many caveats, not enough price reduction, duty hassles, subpar performance versus what I already have AND neccesity of getting HBA.

Caveat emptor. If you fit withing expected use case, consumer drives are superior performers. If not, then its time to experiment.

But I am intrigued with possibility of leveraging namespaces in consumer space, that pretty novel usecase.

EDIT:

How about making benchmark that trumps all benchmarks , and solves this once and for all?

Lets take well known common consumer drives from various epochs and compare them to interesting enterprise drives side by side. Same platform, same utilities, consumer workloads.

Then enhance the testing by long duration workload thats monitored continuosly, to detect cahce effects. Only one who does that is tomshardware from time to time (suprisingly enough).

We have the knowledge of testing caveats that have been assembled piece by pice over last ten years, and we can record and replay workload on linux via fio.

Nobody just ever made historical comparion and consumer vs enterprise.

If you ever feel like goofing around with team, this could have amazing technical and academic value.

Some basic trendsetter drives from memory:

  • intel X25-E (SLC SATA 2) that started the revolution
  • intel 320 (MLC SATA 2)
  • intel 520 or 525 (MLC SATA 3)
  • Samsung 850/860 EVO (TLC SATA3)
  • Samsung 850/860 PRO (MLC SATA3)
  • Samsung 870 QVO (QLC SATA3)
  • Samsung 970 EVO (TLC NVME PCIE3)
  • Samsung 970 PRO (MLC NVME PCIE3)
  • Samsung 980 (TLC DRAMless NVME PCIE3)
  • Samsung 990 PRO (TLC NVME PCIE4)
  • Samsung Z-NAND SZ1735
  • Any pure optane 1 gen reasonable size consumer + DC variant
  • Any pure optane 2 gen reasonable size consumer + DC variant
16

I think this benchmark needs to have read tests for data that was written to the drive more than several months ago because this seems to be an area that consumer SSDs fall flat on their face compared to *some enterprise (the ones with very aggressive garbage collection that implements a cell refresh algorithm).

I’m pretty sure this drive is TLC, not 2 bit MLC

2 Likes
17

Damn, remembered wrong. Good thing I didnt buy it then :facepalm: I’ll remove it from the list, since its totally redundant then.

970 PRO was last samsung MLC drive then? And that marketing fiasco around “3-bit MLC” claims that had samsung ridiculed is probably from 980 pro launch.

EDIT: Just update my notes, here is reference performance result for last MLC consumer drive from samsung (source):
Samsung-970-PRO-AS-SSD
Must have been beast in 2018, but nearly 6 years of development have bridged and overtaken the gap between MLC and TLC significantly.

18

yeah for their consumer line; but MLC still lives on in their enterprise line, they even make M.2 enterprise drives.

ohh I remember that, that was when they tried using the English definition of multi instead of the industry norm meaning of multi.

19

When regularly dumping 2TB (well, 1600GB-ish) on consumer SSDs, wow they suck.
First thirty seconds or so are fine, then most drives I tried fall down to about 1.5Gbit/s, which, for the 1200GB remaining at that point is annoyingly slow to say the least.

20

I should send you a “cheap” 4tb kixoxia CD5 as they are currently $40/tb and see how they work for your use case.

For ahem re-assembling rar files or reconstructing w/par2, its shockingly different too.

1 Like