So that other users aren’t confused since two scenarios are being mixed now:
At first I tested the setup of the OP (ASUS Pro WS X570-ACE with 2 Broadcom HBA 9400-8i8e).
Then I went on testing AMD NVMe RAID0 with an ASRock X570 Taichi (BIOS P3.00, have already been using it for a few weeks, it went public a few days ago).
On the Taichi, I’m using M2_1 (PCIe via CPU) and M2_2 (PCIe via X570) for the P4500. There I’m having the issues Wendell is currently trying to recreate:
Since the drives act that weird in two quite different scenarios and that funny response from Broadcom’s Tech Support I’d say I’m 75 % sure there is something up with the P4500’s current firmware.
However, it cant just be the firmware on the p4500 since linux is fine, and windows in at least one other scenario is also fine.
It “smells like” a sector size/alignment problem. But I did some quick experiments there and couldn’t really seem to make it better or worse. Windows wouldn’t let me create an intentionally misaligned partition, though, which might reveal that type of problem.
Sector size misalignment on raid but not windows native could explain the anomolay.
I reformatted the array and disabled cache during the raid setup, and set a sector size of 64kb. I could claw back some of the expected performance:
Maybe worth trying 256/128kb with cache disabled. Its interesting disabling cache in the amd raid thing makes the apparent speed in CDM that much faster. (2 threads running at once)
Thank you, at least now I know I’m not an idiot doing something wrong.
As I wrote in our DMs in the past weeks, for me the performance is like a “slot machine” and is determined the moment the RAID array is created in the UEFI. The best results I ever saw were around 5000 MB/s 1M/Q8/1T, the worst in the 2000 range (read). And the Optanes are always blasting by even though they should have lower SEQ1M/Q8/T1 results.
I couldn’t establish a pattern between 64, 128 and 256 kB stripe size and cache options, but maybe I was just a bit too frustrated after the first rather disappointing experiences with the Broadcom HBAs.
Do you have “someone” to contact at AMD to further check on this (maybe a RAID firmware bug in the UEFI module?)? Or might it be a symptom of the same underlying issue that is slowing the drives down when using them with the HBA or the low RND4K/Q32/T16 results when using Windows’ built-in software RAID0?
I contacted ASRock about this since I noticed this with the Taichi, but they outright said that they have no clue how to trouble-shoot this.
I can try but it probably won’t be a priority. The “slot machine” aspect really REALLY makes it sound like its a simple sector misalignment type problem. Something like that. With overhead about 5gb/sec is about what I’d expect so if you do it a bunch and about 1 out of 8 times it’s fine, then I would expect it to for sure be the sector alignment problem.
I bet the intel drives report their sector size in a funny way and that’s the issue. Since things do actually look for that. And if not sector size something like that.
What I’m struggling with is why then using the AMD NVMe RAID array in Linux seems to be OK (or did I misunderstand you there?) - that sounds more like that there’s something wrong with AMD’s NVMe RAID driver?
I’m still a bit confused when also thinking about Broadcom’s statements regarding the P4500, they didn’t say that a different OS would solve the underlying issue with the drive’s performance (with an HBA).
AMD doesn’t have a native raid driver for Linux. If the drives work in windows with windows driver and Linux with linuxs driver it suggests not hardware. But the and driver has this “roulette” speed function with one piece of hardware but is solid on another. It’s baffling.
Does it make sense for me to open a support request with Intel - I got the impression that you already have a connection to them regarding this drive model and its firmware versions.
I on the other hand as a mere end customer pleb would most likely need some time to get through the first level support drones reading from their scripts until I get to an actual engineer.
I’m not in it to point fingers at Intel or AMD but I would like to have properly functioning drives some time in the not to distant future.
After what I’ve learned about this model with Broadcom and the AMD NVMe RAID array stuff I’d even have a bad gut feeling re-selling them to someone else.
Cannot contact Intel’s tech support via their website. Created a “normal” account, checked the warranty of the drives via their SNs (September 2022) but I can’t get to a support form, it’s always looping me back to the general support overview site where I am to select the type of product I’d like to request support for.
Checked vanilla Firefox without add-ons, cleared caches etc., no change
Can anybody verify this? You don’t have to submit anything, just try to get to a form where you can actually describe an issue.
their website does that sometimes, but it should work. might try incognito. of all the p4500 I got, I did get one that was acting weird so I replaced it via RMA. RMA process went fine. By acting weird I mean it would disappear and not come back sometimes.
Doesn’t work either, also tried Safari for comparison, as soon as something by the name of SAML 2.0 Auto-POST form appears in the title bar of the browser window I get kicked back to the start, can’t even access their “support community” which seems to be something like a forum.
Just out of curiosity: Can you properly access Intel’s support websites when logged into your account?
Still no idea what’s going on there, some time ago I was at least able to check the drive’s SNs for warranty but at least for a week now even that has been impossible.
Tried multiple Windows, Linux, macOS machines and mobile devices, including mobile, cable and fiber ISPs.
Could finally create a support ticket with Intel a few minutes ago. I am curious how they’ll respond since I didn’t sugar-coat that third parties can recreate my issues and since the only thing in common in my two test scenarios is the P4500 I’m fairly certain that there is a general issue with this SSD model.
Got P4510 4 TB models for testing and the buggy AMD NVMe RAID0 array behavior persists, meaning the moment you create the array determines if the sequential read results are at least usable or complete trash.
Writing scales nicely with the P4510 that is a little faster than the P4500 but the best read values I got so far are around 4800 MB/s. As a reminder, with the previous P4500 drives there was a 1 in 50 (or so) chance to get an array with a little over 5000 MB/s.
The most consistent results (at least 4500 MB/s) are with 128 kB as the RAID0 stripe size with read cache enabled. With the default 256 kB stripe size I get mostly performance like this :
For reference the drives individually on the same PCIe interfaces as when used for an RAID array:
Based on disk selection and performance target, you are entering enterprise/business territory, where usually a raid controller is first chosen, then officially supported drives, not the other way around.
I understand that it sucks and it should work, but that is the reality, it does actually work but it needs optimization, controller vendors cannot test and optimize for every drive out there, neither could drive vendors do the same for every controller, all they can do is try to test and optimize for major operating systems when the drive is connected directly or through the basic passthrough controller, in case of sas/sata ports. So the industry standard is to lay the responsibility on the controller vendor, on the other hand they take responsibility only for drives they advertise as supported. It’s more or less similar to ram support on motherboards.
Other than that, you are mostly fighting windmills, your milage may vary.
You can always go the linux route and setup software raid, which I believe is better for many different reasons, including that it gives you better performance than lowend controllers anyway, even when they work correctly.
(The P4500 is to be removed in a revised list according to Broadcom’s Support, BUT the P4510 should have been fine, specifically asked them about this model)
By the look of it now I guess that Intel has messed something up in the SSD controller firmware (the P4510 and P4500 use the same controller) at least for some newer versions and since you cannot downgrade the firmware yourself as an end customer it’s a bit hard to trouble-shoot.
