Poor RAID0 performance on Win Server 2025

Operating Systems & Open Source Windows
, , ,
1

Hi all I have a Dell Poweredge R7615 with the following spec:

AMD EPYC 9354P
256gb DDR5
Dual 512gb ssds in RAID1
x2 Kioxia CM6-R (Dell branding)
Intel E810 100G NIC

Win Server 2025 Datacenter

We bought this server for virtualisation of workstations with the intention of expanding storage down the line. What we have found instead is that the Kioxia drives have been very hit and miss with performance.

I have been using CrystalDiskMark to benchmark the disks/array throughout, I will just abbreviate it to CDM. If this tool is not reliable etc, I am open to suggestions :slight_smile:

We originally were suffering extremely poor response time when we were in a VM. The x2 Kioxias were in a storage spaces RAID0, everything left to default during setup. We were using fixed size vhdx, with a Gen2 VM. When I used CDM within the VM I got these results:
Seq1MQ8T1 - 3556MB/s - 4139MB/s
Seq128KQ32T1 - 2166MB/s - 2108MB/s
RND4KQ32T16 - 391MB/s - 203MB/s
RND4KQ1T1 - 21MB/s - 22MB/s

I’d have thought that Reads/Writes would’ve been a lot faster than this since no other activity was happening on the array. I took a screenshot of the response time during a CDM run, I had seen the response time go to over 20,000+ms during this run:
image

So I decided to test, I will just use a single disk and put my vhdx onto there. The performance seemed to be better on the benchmark but as soon as random tests started, the response time again within the VM just fell off dramatically.
Seq1MQ8T1 - 4284MB/s - 4008MB/s
Seq128KQ32T1 - 6891MB/s - 4016MB/s
RND4KQ32T16 - 1132MB/s - 331MB/s
RND4KQ1T1 - 26MB/s - 62MB/s

So next thing I did was benchmarking the array on host in RAID0 this time. A bit more promising it seems:
Seq1MQ8T1 - 13,901MB/s - 7967MB/s
Seq128KQ32T1 - 12,819MB/s - 7976MB/s
RND4KQ32T16 - 5901MB/s - 4380MB/s
RND4KQ1T1 - 45MB/s - 301MB/s

Finally I did single disk benchmark on the host.
Seq1MQ8T1 - 4283MB/s - 4026MB/s
Seq128KQ32T1 - 6957MB/s - 4030MB/s
RND4KQ32T16 - 6174MB/s - 3160MB/s
RND4KQ1T1 - 46MB/s - 312MB/s

I then proceeded to update the firmware of the controller card and then the firmware of the disks themselves. This is where it just gets weirder…

To start with single disk benchmark. So the Random reads fell off a lot, random writes halved almost.
Seq1MQ8T1 - 6869MB/s - 4028MB/s
Seq128KQ32T1 - 6193MB/s - 4031MB/s
RND4KQ32T16 - 3545MB/s - 3074MB/s
RND4KQ1T1 - 40MB/s - 169MB/s

Next I then did RAID0 in storage spaces again. the random speeds are within margin of error and sequential results are a bit better overall but also not the expected result of meeting the pre firmware update benchmark.
Seq1MQ8T1 - 8362MB/s - 7964MB/s
Seq128KQ32T1 - 5670MB/s - 5689MB/s
RND4KQ32T16 - 3437MB/s - 3227MB/s
RND4KQ1T1 - 40MB/s - 165MB/s

Lastly I have done a RAID0 within the S160 controller card, which is software RAID btw. I don’t really know what to make of these results either.
Seq1MQ8T1 - 7388MB/s - 7586MB/s
Seq128KQ32T1 - 6479MB/s - 6538MB/s
RND4KQ32T16 - 3453MB/s - 2948MB/s
RND4KQ1T1 - 40MB/s - 167MB/s

It seems that post firmware update has just caused performance on RAID to take a massive hit. I feel like I am starting to go insane with how far im looking into this, as I started with trying different block sizes and interleave sizes to improve random performance.

If anyone has any pointers or suggestions that would be greatly appreciated, we’re all scratching our heads a bit with this one.

2

Are you using SMB/CIFS for data transfer? That protocol is not the best/smartest/optimised for certain applications, just as Win-OS itself btw. Try installing Linux and rerun your tests.

BTW, how have you connected the Kioxia drives? Are they on a PCIe5 link or “just” PCIe4? Also, that test suite may not be the best choice either, get the Phoronix test suite for an industry standard:

HTH!

3

Our storage server which is part of the same network is running Ubuntu Server with minio. So using FTP to transfer data from the VMs to the storage server.

The Kioxia drives are PCIe4

I could install Linux temporarily to test speeds etc, however we cannot use it for production (as much as I would love to). We rely too heavily on the .NET framework and built in windows features to not be able to use it.

I will check out phoronix, thanks for the suggestion :smiley:

4

Just to help me understand your setup a little bit better.

You are using a Windows Server 2025, that uses storage spaces RAID0 for storage, and then use start a VM with I guess Hyper-V and run tests inside that VM?

I would probably start by testing the performance on the host itself to see if the results match my expectation and then go from there.
Linux is certainly a good idea.

I don’t know about the current state of Storage Spaces, since I moved to Proxmox and ZFS a long time ago, but back in the days Storage Spaces was known for extremely poor performance and stability.

5

Yes so correct that using Win Server 2025 as host. Using Hyper-V to create our VMs which are also running Win Server 2025, since the host is the datacentre version of server.

In the original post I did put in results for the host being benchmarked, might not have been clear apologies!

This is pre-firmware update, RAID0 using storage spaces benchmarked on the host:
Seq1MQ8T1 - 13,901MB/s - 7967MB/s
Seq128KQ32T1 - 12,819MB/s - 7976MB/s
RND4KQ32T16 - 5901MB/s - 4380MB/s
RND4KQ1T1 - 45MB/s - 301MB/s

Another pre-firmware update, single disk benchmarked on the host:
Seq1MQ8T1 - 4283MB/s - 4026MB/s
Seq128KQ32T1 - 6957MB/s - 4030MB/s
RND4KQ32T16 - 6174MB/s - 3160MB/s
RND4KQ1T1 - 46MB/s - 312MB/s

Post firmware update, single disk benchmarked on the host:
Seq1MQ8T1 - 6869MB/s - 4028MB/s
Seq128KQ32T1 - 6193MB/s - 4031MB/s
RND4KQ32T16 - 3545MB/s - 3074MB/s
RND4KQ1T1 - 40MB/s - 169MB/s

Post firmware update, RAID0 using storage spaces, benchmarked on the host:
Seq1MQ8T1 - 8362MB/s - 7964MB/s
Seq128KQ32T1 - 5670MB/s - 5689MB/s
RND4KQ32T16 - 3437MB/s - 3227MB/s
RND4KQ1T1 - 40MB/s - 165MB/s

Post firmware update, RAID0 using S160 controller card, benchmarked on the host:
Seq1MQ8T1 - 7388MB/s - 7586MB/s
Seq128KQ32T1 - 6479MB/s - 6538MB/s
RND4KQ32T16 - 3453MB/s - 2948MB/s
RND4KQ1T1 - 40MB/s - 167MB/s

Again as much as I would love to go with Linux, especially proxmox in this situation. The decision is unfortunatley not mine lol… Once this project is shipped I will be recreating this using Proxmox but that’s for a later date.

I am tempted to go back to WinServer 2022 again to test it out, but I’m running out of time to keep testing these things annoyingly. Was more hoping if someone would have an idea or similar issue before that was all :smiley:

6

I would simply install Ubuntu or Windows 11 and run the benchmark again on NTFS or ext4. Bare metal on one disk, and then with the PERC RAID afterwards.

If you still get these poor numbers, something is off with your setup.

If you don’t get these poor numbers, Storage Spaces is still trash :slight_smile:

4 Likes
7

FWIW we’ve hit similar looking problems with one of our 11 23H2 machines that was upgraded from 10 22H2 (so just host OS) and it seems like it’s not an especially uncommon problem with 11. We’ve been scratching our heads too and, absent any better idea, are planning to take the machine down for clean install of 11 at its next maintenance window.

I’ve also hit plenty of issues around the scheduler, up to 11 just suspending processes’ threads ~65% of the time even though there’s plenty of work to do. Given Microsoft develops both client and server Windows SKUs from a shared core I’d be surprised if 2025 was immune to these seemingly low level (kernel and driver) issues with 11.