Curiosity on SSDs, RAMdisks and benchmarks

[Zorak]

So, I’ve run a couple quick benchmarks on some drives just to experiment with them, but I’m not entirely sure how to interpret the results. In particular, I’m curious about the variation in relative speeds from test to test (frankly, I don’t have a full understanding on what each benchmark is exactly doing compare to each other, only sequential vs. random. And still I don’t know all the determinants of performance in each). Hence, I thought I may ask the hive mine for its thoughts.

The drives tested:

• C: a 1TB PCIe 3.0x4 SSD
• D: a 4TB PCIe 3.0x8 SSD (truly a RAID0 PCIe x8 AIC)
• M: a 1TB PCIe 3.0x8 SSD (truly a RAID0 PCIe x8 AIC)
• R: a 32GB RAM disk

I did two sets of benchmarks:

• Run all benchmarks for each drive, one drive at a time
• Run all benchmarks for all drives simultaneously

The benchmarks themselves are the 4 test CrystalDiskMark defaults to when freshly installed. The point of the second test is to potentially trigger bottlenecks elsewhere, such as the CPU’s I/O, memory or whatever.

These are the results:

Sequential:

Parallel:

Some results are boring (RAM > PCIe x8 > PCIe x4), but some made me wonder about the nature of the test and the determinants of drive performance, such as:

• The RAM disk is faster than the SSDS except the RND4K Q32T16 benchmark, where it turns to be the slowest reader.
• Most benchmark scores fall when running all drives simultaneously, but SEQ1M Q1T1 seems unaffected for the SSDs (the RAM disk still suffers)
• On the other hand, RND4K Q32T16 is unaffected for the RAM drive, while it’s the test where the PCIe x8 SSd suffered the most from running simultaneously.
• For RNDK Q1T1, only being RAM helps, the other drives show no significant difference.

So, what do you make of these? Is there anything else that draws your attention?

[Mark-D-Stroyer]

This really speaks towards just a lane splitting issue. Since consumer platforms don’t have a huge number of PCI-E lanes, a lot of storage lanes will frequently get split, frequently tied to the chipset that becomes a lane bottleneck.

Disk benchmarks are hugely just testing theoretical limits, since no program really hits them the same way.

[Zorak]

Thanks for your reply!

This really speaks towards just a lane splitting issue. Since consumer platforms don’t have a huge number of PCI-E lanes, a lot of storage lanes will frequently get split, frequently tied to the chipset that becomes a lane bottleneck

I’m not sure which result you mean by “this”. It’s an X399 Threadripper system, though, so all the SSDs are going through independent CPU lanes.

Disk benchmarks are hugely just testing theoretical limits, since no program really hits them the same way.

That’s fine, the point is not to assess the performance of these drives in some real-life scenario, but to understand the interaction between specific IO operations performed by the benchmarks and specific characteristics of the system. For example, what about an RND4K Q32T16 test makes it slower on RAM than on a drive? Why are some tests more resilient to running all drives simultaneously than others? Etc.

[alkafrazin]

I was wondering this myself, when using a ramdisk for various things and seeing similar results.

I think it has to do with kernel I/O management and how the ramdisk filesystem works around the OS’ expectations of block device access. I see some pretty massive kernel CPU use in the heavily threaded Q32T16 test.
Also worth noting that the linux side sees completely different results. Filesize also seems to make a very large impact on performance.

numbers

4GB, default kdiskmark

1GB, matched settings kdiskmark 0s cooldown 10s measure time

8GB, otherwise matched settings kdiskmark 0s cooldown 10s measure time

16GB, otherwise matched settings kdiskmark with 1s cooldown 5s measure time

Some very weird results indeed, notably that high QD and thread count gets worse results than low QD and thread count for ramdisking.
Consumer AM4(5900x) dual channel jedec 3200mhz bog standard 2dimm 4rank config. I would think your platform should perform a lot better than mine, but I seem to have better results in everything but sequential writes using linux tmpfs.

[Zorak]

Thank you for your reply, very interesting data! Indeed, I’m surprised by your write results for the last test vs. the third one. Then again, Q32T16 is also the test were my RAM disk performed worse than the SSDs. At the same time, your random Q32T16 read is faster than your sequential Q1T1, while it’s much slower for me. Might try running the tests in Linux later, to clear the OS effect.

Consumer AM4(5900x) dual channel jedec 3200mhz bog standard 2dimm 4rank config. I would think your platform should perform a lot better than mine, but I seem to have better results in everything but sequential writes using linux tmpfs.

I don’t know, I’m on second (or first and a half ) Threadripper, 2950X, quad channel 3200MHz 4dimm dual rank, using “local” mode. The CPU itself should be overall better on your computer, the IMC as well, and I don’t know if quad vs dual channel would help me in these tests

[wertigon]

I think you’ve simply run into a CPU I/O bottleneck here. Remember, modern PCs are not the relatively simple designs of the Altair and the 80286, but the die actually contains I/O, RAM and complex compute algorithms in addition to the load-instruction-from-RAM x86 assembly instruction parser your grandpa was used to.

[CHESSTUR]

Can I get in on this “old conversation” please? Being the semi-sorta retired, semi-non-normie, pseudo-plebish, PC builder, enthusiastish, curious but somewhat naive joneser, I was making plans to put my AMD X570 HERO in a NVMe RAID 0 config when I inadvertently stumbled upon this RamDisk thing once again. Now admittedly I’m not completely naive (but close) about RamDisk as it has been around for a couple decades BUT… I kinda sorta accidentally activated it on my build and thought “why not?” and now I’m wondering why I need RAID 0 at all. Advance apologies for bad spelling and grammar.

So I’ve been asking around on forums trying to glean what information I can about what the shortcomings of this might be and how reliable are the above specs and what are the advantages and possible disadvantages of using RamDisk over NVMe when the OS is on NVMe prior to loading into RamDisk. I’m not getting a whole lot of straight answers.

I mean, it seems to work okay. Data does seem to save from the RAM drive to the drive. Am I missing something here? Is there any reason why half the domestic end user world isn’t using this??

[Susanna]

Could you clarify what you mean by RamDisk? Because from the context it seems you’re referring to disk caching, which actually all modern operating systems do automatically. Unused RAM is used to cache async writes and recently read blocks, which is why a program will open quicker after you’ve already had it open, than right after a cold boot.
RAMdisk is when you set aside a memory region to pretend to be a hard drive. Performance is excellent, which is why it’s popular to use as scratch partition for various work software, but when the system powers down any data left in it is lost.

[CHESSTUR]

@Susanna

RAMdisk is when you set aside a memory region to pretend to be a hard drive. Performance is excellent, which is why it’s popular to use as scratch partition for various work software, but when the system powers down any data left in it is lost.

I trust that I am referring to the above in your words. The operating system loads from NVMe into memory but I still get to make changes and save data. On reboot my changes remain. I moved a 6.2 GB file in under 5 seconds and barely had time to PRTSc the event. On reboot the file was still there. Can you tell me more?

[Susanna]

I’m afraid not. Windows is a huge unknown to me, I mostly use macOS and Linux. It sounds like you’re using something that makes all writes asynchronous, which is to say that when you copy the file, it is read into memory, the process is declared complete, and then it actually writes to disk afterwards. It’ll work fine until you accidentally bump the cable and power the system down, then, anything that hasn’t yet been flushed to disk will be lost and the system will insist on running a chkdsk to fix the journal before letting you boot. It’s probably fine to use on your home computer, but I guess Microsoft may not want the liability of losing user’s data, since sync writes are generally sync for a reason, and blanket declaring them async can make things weird, so it doesn’t do that by default.

[CHESSTUR]

It has been my observation that Microsoft isn’t liable for anything at all when anything goes wrong on a PC for any reason. That’s what the big fat EULA agreement pretty much amounts to. Being a domestic end user, I am not overly concerned about this as I have little choice but to roll with the punches regardless of the OS. I’m not certain which cable you would be referring to but usually when I’m in the process of moving any data from one point to another I’m not in the habit of reaching inside my PC and jiggling cables around. Decades ago I did learn my lesson about power outages so all main PCs now run on UPS.

Come to think of it powering down a system in the middle of a data transfer generally results in mishaps regardless if one is using a Ram Drive or not so I don’t really consider that a factor, all things being relative. Still, i don’t think I’m writing back files to the NVMe drive at the rate of 97GB per second either. I am wondering if RAID 0 really gives me any advantage over a Ram Drive in my case and I’m wondering what sort of things associated with using a Ram Drive I should be vigilant about.

[Trooper_ish]

I think the cable was indeed the “kettle lead” power cable to the computer.

And just an example.others would be when windows just crashes, before ram disk written out.

Ram disks have been around a while, but SSD’s are cheaper.
Like a Lot.

$40 might get you a cheap Terabyte of SSD.

I am not sure you could get a terabyte of DDR3 for that…

I guess the guys before, were surprised that ram is excellent speed wise to start with, but higher threads unexpected ly meh, in comparison. Still beats Raid0…

For yourself, you know the risks, have the money for the ram, so could do it.

Personally, I’ve coppied the app I wanted to speed up to a ram drive, done the business I needed, save the date elsewhere and blown the drive away at the end of the session.

If one gets used to it around, it becomes part of the workflow. Then it gets annoying when it has to be recreated.
Same with Raid0. Copy project files/ scratch files while editing, then blow them away once done.
If you get used to using it, then you’ll be annoyed when its not available.
And all drives die, but Raid0 is succeptable to minor things that should not be breaking issues

[CHESSTUR]

I forgot to thank Susanna for responding so, thank you Susanna. It’s so hard to get the experts to talk about this and I can’t help thinking, what’s the big hold out?? What am I not being told about RAM drives that I should know before I go whole hog with them?

@Trooper_ish

I think the cable was indeed the “kettle lead” power cable to the computer. [/quote]

Yes, I wondered about that as an afterthought. This is a good warning in any case, applicable in all cases of PC operation, and not necessarily a RAM drive concern. Any power failure or system failure can affect any process regardless of the format or configuration or hardware being used.

[quote=“Trooper_ish, post:12, topic:201257”] Ram disks have been around a while, but SSD’s are cheaper.
Like a Lot.

$40 might get you a cheap Terabyte of SSD.

I am not sure you could get a terabyte of DDR3 for that…

I couldn’t agree with you more on that point and that could be a reason why so few people are using it despite the fact that it has been around for ages. It’s starting to give me some ideas. There was a time when people couldn’t afford that much RAM but now older RAM is being tossed away for so much e-waste. Hmmm… Just a thought.

To be true, it is not uncommon these days for people to have more RAM in their PC than they use on a regular, daily basis. The example I am using here only employs a 16 GB RAM drive, yet I can do all my daily tasks, transfer files, save data to the OS, and perform all the things I normally use my PC for with the main difference being that they are being done incredibly faster. For less than 40 dollars I could easily pull out a TB of DDR3 RAM from a recycling station (but I’d probably get in trouble if I got caught doing it).

@Trooper_ish

I guess the guys before, were surprised that ram is excellent speed wise to start with, but higher threads unexpected ly meh, in comparison. Still beats Raid0…

For yourself, you know the risks, have the money for the ram, so could do it.

Personally, I’ve copied the app I wanted to speed up to a ram drive, done the business I needed, save the date elsewhere and blown the drive away at the end of the session.

It would seem that my comments have generated some controversy in other forums regarding this matter and I wasn’t aware that this subject was a hot potato. That’s part of the reason I mentioned it here, where I find there are many level-headed, rational thinking, individuals, who might be more inclined to answer some of my concerns about RAM drives.

You mentioned knowing the risks. Well, that’s precisely what I’m trying to determine. Exactly what are the risks and the Ramifications (no pun intended) here as compared to using a typical SSD or say, NVMe in RAID 0? If the risks are comparatively negligible and a RAM drive is significantly faster then I think this is something worth looking into.

In the past I’ve done exactly this: Used a Ram drive to get the job done and then uninstall it. This time I didn’t uninstall it and forgot about it. The drive automatically loads on boot now and it happens in mere milliseconds. I no longer have the heart to disable it because it runs so well and I can’t help wondering, “what’s the catch?” The deal breaker is that RAM drives are so much faster than NVMe in RAID 0.

Anyone know of any Wendell videos where he explains RAM drives?

Thank you Trooper_ish. You have given me some food for thought and I appreciate it.

[Trooper_ish]

there was a product by Enmotus(?) that was offered as like a freebie with some early Ryzen chips / boards as a sweetner, and for a couple years it took off again.
but all drives die, including RamDisks and various bugs and crashes showed they are not super stable.

might have an uptime in months, rather than years. and when gone, it’s gone.
not everyone backed up the data realtime/ frequently

I presume, in the end, people just gravitated to more stable flash storage, as the space was larger, and they didn’t need to clear / populate it as much.

[CHESSTUR]

The program I’m currently using is from ASUS ROG. So far, so good. No bugs have been discovered this far. Data saves and for all intents and purposes it seems to behave much the same as a conventional drive except considerably faster. I’ll keep ya’ll posted if I run into anything worth reporting.

[Zorak]

It definitely sounds like RAM buffering, rather than a RAM drive. At least the way I set it up now or in the past, it is 100% volatile, meaning all content is wiped with each reboot. RAM buffering on the other hand will give you apparent acceleration to your non-volatile storage, as longas no crash or power loss interrupts the actual write process after the OS got the “completed” message, as explained by @Susanna

[CHESSTUR]

Okay, that might explain a few things then. Thank you for the clarification.

If that is the case then Crystal Disk benchmarks are reading the buffer, but wait… Crystal Disk reports the volume of the NVMe drive. How is this possible?

This is what the very slow NVMe gives me without it.

This is what I get with the “buffer”.

I’m not disputing what Susanna is saying so much as I’m stating that issues like this also exist with standard drive configurations. For example, if I’m transferring a file from a standard SSD to another standard SSD regardless if I’m using a buffer, cache, RAID, whatever. . . And I have a power outage with zero backup power I’m going to lose that data. I’m very likely going to encounter the very things she/he suggested so this isn’t specifically germane to using a RAM Drive/Cache/Buffer or whatever it’s called these days.

[Susanna]

The benchmark reports the size of the NVME drive because it isn’t really a RAM disk, just a disk cache for your SSD. Basically, what your “RAM drive” is doing is putting a memory layer between the operating system and the drive. When you write data out, it goes into this interim layer, and the software tells the operating system the transfer is done. The software then does the actual write to drive after the fact. When you read data in, the software keeps track of which blocks are read most frequently, and keeps these in RAM. All modern operating systems have this functionality built in. What I suspect is happening is that this benchmark uses synchronous writes, which basically means when it tells the OS to write data, it says “and don’t come back until it’s done”, and then reads those same blocks back. That means it bypasses the memory cache, and measures the actual performance of the drive. But when you put this “RAM drive” software in between, it writes into memory cache which then reports that the write is finished before actually putting anything on the drive, which will make the benchmark report greatly improved performance, since you’re now benching your memcpy performance instead of the actual drive, and every block the read test requests will already be in cache and therefore return immediately. In practice almost every write you do during normal work is going to be an async write, which uses the operating system’s memory cache and basically means “write this to disk when you have the time”, so the software you’re using won’t improve that performance one bit.

For a home user it’s usually fine to use async writes. On Linux you can set special mount flags to make the drive behave as if every write is async even if it’s explicitly a sync write, there’s probably something similar on Windows. As it is, I’m fairly sure your software only improves performance in the benchmark, and you could achieve the same performance boost in everyday use by just suspending the computer instead of turning it off entirely, so that your memory cache never gets flushed. I’ve been putting RAM drive in quotes because I’m fairly sure your software isn’t actually a RAM drive, but just some sort of disk cache that turns sync writes into async writes. Most good NVMEs (also SATA SSDs and even HDDs) will actually have a built-in disk cache, usually in the form of 512MB of DDR3 DRAM which stores important things like the map of blocks:sectors and the most recently used blocks, which means up to 512MB writes will feel very fast, and then once the cache is exhausted slow down drastically. Your NVME is possibly performing so poorly because it lacks this cache (many low-end M.2 SSDs do) and has to actually access the NAND every single read/write. A real RAM drive is a distinct volume and actually trying to use a terabyte large one is going to make you go OOM and crash on any computer that isn’t a workstation or server that actually has so much memory to spare.

[CHESSTUR]

@Susanna

I agree with you here, in general. This is certainly not the RAM Drive that I used of former years, but a RAM cache, as you have stated, and I would say that this is where I went astray in my thinking as I am new to this. I would also say that it’s definitely worth using on low end NVMe such as this Corsair MP400 I’m using. That I can move a 6.2GB file in under 5 seconds with this is quite impressive. I am certain that the file transfer I made was done in real time as it was made to an entirely separate drive. Whether the “cache” assisted in this process or not is dubious, I’ll admit, but I think perhaps yes because I did not make file transfers that fast with the NVMe before. Admittedly, I’m unfamiliar with the mechanics involved here and I think that should be apparent so I certainly thank you for your time and patience explaining this to me.

I am not an IT. I’m just an old grandpa in his mid 60’s who went “enthusiast” in his spare time and started building computers as a hobby. I think I’ve come to the right place to get my answers and all of this is very fascinating for me. I’ve also dabbled in Linux but I am barely a novice. At some point I plan on getting back into Linux just for the fun of it all but I’m tired of losing my Linux boot strap even when I have it on a different drive altogether. I suppose I’ll just have to get used to booting from BIOS or simply build another PC that is strictly for Linux. Anyway, I digress.

One thing I’ve gleaned from all this information is that using this RAM cache < is that it will probably save me some wear and tear on the cells in my NVMe. I think I’ll keep using it.

Once again, thank you for taking time to explain all of this to me.

[Laurent974]

I don’t know if this could help but did you know, CrystalDiskMark use thread to do the job of testing the disk and you are able to increase the number thread.

by default, CrystalDiskMark use 1 thread so when you do the test, you will see 1 CPU at 100% but this is a bottleneck for performance. When you use many disks, you have to use many threads and the test will get better. But with chipset motherboard, you still stuck at the performance of the chipset but with many thread the test will get better.