It’s a little sad to have one bay of the MB699VP-B V2 empty, I’ll try to at least free some faster Gen3 M.2s to use them in the Icy Dock M.2-U.2 adapter…
The 3 CrystalDiskMark benchmarks were started at the same time (as fast as I could click and move the mouse);
Maybe the CPU (3700X) limits performance a bit, if I only launch one CDM benchmark at a time I also get the 7.350 MB/s sequential read just as with the SFF-8654 8i-to-2xU.2 cable directly attached to the SSDs;
Maybe good news regarding a SFF-8654 8i-to-2xOCuLink cable, the 0.5 m noname one I got from Amazon offers the same performance numbers as the expensive-AF 1.0 m Broadcom cable:
I can’t tell if it’s just the P411W-32P that tunes/amplifies the PCIe signals to the max so that shitty cables don’t matter thaaat much or if that cable by chance is actually good;
Am a little bummed out to not have more Gen4 SSDs to toy with, wanted to get some with some old hardware I had planned to sell but I chose to donate it to a cat café that is hopefully opening up sometime this year;
The Delock HBA was buggy AF with its default firmware from manufacturing and finally worked correctly after I bugged Delock’s level 2 support about firmware updates.
I got the P411W-32P some time before the first suitable cables arrived and I wanted to “prepare” it for smooth operation. It came with non-public early firmware 0.1.0.0 and I thought that it’s going to be the same spiel as with the Delock 90504.
Broadcom’s support didn’t help, they didn’t suggest to try an earlier firmware version since they “had just tested” it in their “lab” and everything’s fine there.
But yes, after some distance I think I was stupid not to think of it myself before sending my first unit into RMA.
I’m glad it finally works for you.
I hope I don’t run into similiar issues once my HBA 9500-16i will finally arrive.
I hope the SFF-8654 to U.2 cable ( 05-60005-00 ) will also arrive at the same time.
So the Amazon Cable is compatible/a replacement with/for Cable 05-60001-00 from Broadcom?
Good to know. But for the momment I don’t plan to have the SSD hotswappable.
Yes, that’s the correct (“guaranteed”) cable to connect one of the SFF-8654 8i ports of the Broadcom HBA 9500-16i to two U.2 SSDs.
It’s quite long in regular PC cases, please be gentle (as with all SFF cables) when cable managing it during installation.
I wonder if the 4xU.2 Icy Dock MB699VP-B V2 also works with the 9500-16i, I can feel a slight itch coming…
May I ask why your 9500-16i takes that long to be delivered? Checked locally here (Germany) and it is widely available.
If I don’t get my 9400-8i8e issues properly managed (now that the P411-32P is fine I got the time for them) I’d likely “have” to get 9500-16i as a replacement and adapt two internal to two external ports since for some reason there’s no 9500-8i8e - When getting HBAs new I’d only get PCIe Gen4 ones for more the efficient use of PCIe lanes.
Next I’ll try to somehow scrap together a total of 8 NVMe SSDs to try to reach the mountain top of 32 GB/s with the P411W-32P’s PCIe Gen4 x16 host interface. The available 3 Gen4 Samsung SSDs reach around 21 GB/s sequential read, so the remaining 11 GB/s should be easily doable with 5 PCIe Gen3 NVMe SSDs
Is there a comfortable tool to benchmark several independent drives all at once or is just opening 8 instances of CrytsalDiskMark, selecting a larger test file size and clicking as fast as you can the way to go?
According to their website, CrystalDiskMark is basically just a front-end to Microsoft Diskspd
The details on exactly what each of these tests does is vague, so I grabbed the command-line it’s passing to diskspd for each of them to see exactly what they are doing. The options passed to Diskspd for each of the tests was :
Test
Read
Write
Seq Q32T1
-b128K -d5 -o32 -t1 -W0 -S -w0
-b128K -d5 -o32 -t1 -W0 -S -w100 -Z128K
4K Q32T1
-b4K -d5 -o32 -t1 -W0 -r -S -w0
-b4K -d5 -o32 -t1 -W0 -r -S -w100 -Z4K
Seq
-b1M -d5 -o1 -t1 -W0 -S -w0
-b1M -d5 -o1 -t1 -W0 -S -w100 -Z1M
4K
-b4K -d5 -o1 -t1 -W0 -r -S -w0
-b4K -d5 -o1 -t1 -W0 -r -S -w100 -Z4K
As a side note, I went ahead and tried CrystalDiskMark in my windows VM:
“May I ask why your 9500-16i takes that long to be delivered? Checked locally here (Germany) and it is widely available.”
I’m located in Switzerland and ordered it via Digitec. Often products are marked with product available at external distributor, delivered within 3-4 days. When I’ve ordered it in April I think, I got one day later the notification that there’s a delay in delivery and expected delivery is January 2023 :-0
Luckily now the expected delivery date is middle of June.
The same happened with the 05-60005-00 cable. On order page it said 1 piece available at external distributor, delivery in 3-4 days, when I ordered last week. Then after I ordered it, I got a mail that delivery date is unknown. And now when I look into my orders page it also says expected delivery date also middle of June.
At least I’ve got now the 05-60006-00 cable which I also ordered when I ordered the 9500-16i.
Currently I don’t plan to use any Icy Dock enclosures, as I don’t have an (external accessible) 5,25 bay.
Funny coincidence, have a pleasant memory of digitec’s Zürich location, picked up my main living-room speakers there back in 2007 that I’m still using to this day
Just scraped 6 SSDs together, think I can get 8 soon and I think the poor 3700X is bottlenecking the performance, but no stability issues at all, even when going to Sleep/S3 while benchmarks are writing or reading from the SSDs connected to the P411W-32P
Hope I can test the P411W-32P maxed out in combination with a 5950X.
I think I encountered a hardware bug with the Icy Dock ToughArmor MB699VP-B V2 - can you confirm the following?
When looking at it horizontally the status LED of two bays that are located over each other seem to be activated in parallel.
Example: The drive in the top left bay gets accessed and that one’s and the status LED of the bottom left also flashes, even if there is 100 % no access to that drive. The same pattern can be observed on the opposite side.
If you’re using Windows with a system that has the P411W-32P installed in can you confirm my Bluescreen observation when using the latest drivers version 2.61.29.00 (ItSas35_Win10_Win_Server_2016_2019_PCIe_P18_0) with the HBA having any older firmware up to 4.1.2.1 (P411W-32P_4_1_2_1_HBA_signed_P14.2.fw)?
Only in combination with the latest firmware version 4.1.3.1 (P411W-32P_4_1_3_1_HBA_signed_P14.3.fw) the driver version 2.61.29.00 (ItSas35_Win10_Win_Server_2016_2019_PCIe_P18_0) doesn’t cause a Bluescreen for me on my two tested platforms (but then again, no SSDs are showing up at all with that latest firmware).
I’m trying to collect as much details as possible to maybe be able to resolve this in concert with Broadcom’s tech support.
Well… A lot has happened since I’ve last been here. I’ve had to perform minor data recovery (Windows ate dirt), troubleshoot hardware (H/W is fine at least XD), and reinstall Windows… It’s been fun.
For the P411W-32P:
There are several versions of drivers depending on which version of Windows is installed. And then, there are two parts to the driver package. One half appears to be for the management endpoint (ItSas35.inf and ItSas35.sys) and the other is for the hot swapping mechanism (lsinodrv.inf).
I tend to have issues with the first half - ItSas35.inf and ItSas35.sys. I have not tried previous versions of this file. (I’m still running firmware version 4.1.2.1) However, any attempt to install the driver on the last couple of versions of Windows I’ve had (1809, 20h2, 21h2) promptly results in a blue screen of death. If left alone, Windows will eventually install a version that does not cause crashes. Something to note, I have been using a third party utility to manage drivers.
It is on my to-do list to eventually try the latest firmware and driver combination. I honestly do not want to break a working configuration quite yet.
The driver read-me does warn about crashes if a particular patch (KB4482887) isn’t applied. However, Windows version 1809 and newer already has this patch applied. Weird…
I am running mine in PCIe 4.0 x8 and have noticed that I may have hit a bottleneck of about 14 ish GB/s with 3x Samsung 980 Pro 2TB SSDs. Damn 3080 consuming lanes… =P @aBav.Normie-Pleb, what do you peak at in a PCIe 4 x8 config?
I’d love to know what exactly their test rig is composed of…
Also, thank you for helping out with this.
The guide mentioned (just skimmed it) looks to be applicable for your use case.
I don’t remember off the top of my head what Manufacturing Data Region is. I’ll have to wait for when brain.exe decides to resume operation. Until then, you might want to consult someone who is more versed. In the meantime, play it safe and don’t nuke that portion.
ECC isn’t supported with APUs (particularly because the shared memory of the graphics unit doesn’t like ECC). AMDs workaround for this are the Pro versions of Ryzen (very rare in retail as they are mostly an OEM thing).
After testing the P411W-32P in several motherboards with the latest firmware (v4.1.3.1 - Shows up as v1.3.1) and the previous version (v4.1.2.1 - shows up as v0.1.2.0), I can definitively state the following…
If the motherboard firmware cannot see attached SSD, no OS will. Drivers won’t matter either.
The latest firmware hates Samsung SSDs. Nothing shows up regardless of OS or motherboard firmware. It plays a little bit nicer with WD SSDs and will at least show the (WD SandDisk) NVMe controller when attached, but still no storage medium.
I’m staring to think the particular reason why it works perfectly in the lab is because there aren’t any SSDs other than Seagates attached.
All in all, it is a firmware issue of some sort. Whether it is on the controllers’ end or the SSD, I won’t know until I have a larger sample size.
What I do know is:
Seagate has been using Phison for flash media controllers.
Both WD and Samsung are vertically integrated – they design everything top to bottom from flash media to controllers to firmware.
Also, no need to wipe everything in order to downgrade firmware on the P411W-32P, at least in my most recent testing.
Thanks for that additional data! Just so that there is no misunderstanding on my end you have already been able to test the P411W-32P on all the motherboards you listed?
In the meantime I was able to confirm my findings with another AM4 motherboard, an ASUS ProArt X570-CREATOR WIFI with UEFI 0801 and a 3900 Pro. Was the first time I was able to test X570+PCIe Gen4+AGESA 1207, had hoped that maybe an AGESA update would help out here.
Yes, I see the same behavior with simple firmware downgrading (=restoring functionality) not needing additional erase steps.
Interestingly I cannot confirm WD SSDs showing up at all with firmware 4.1.3.1 (0.1.3.1), tested:
SN550 2 TB
SN750 2 TB
Ultrastar DC SN630 7.68 TB
They only ever show up with P411W-32P firmware up to version 4.1.2.1, exactly the same as with the Samsung PM1733 7.68 TB units. Have you been able to use any SSD with firmware 4.1.3.1?
BUT I saw a crashed state of the P411W-32P after extensive S3 (Sleep/Suspend-to-RAM) testing where after reawakening the system with active SSD file transfers which had been paused by the system going sleep the SSD showed up as a PCIe device but wasn’t accessable as storage any more.
Rebooting the system fixed that again.
The same SSDs attached to the same motherboards directly with passive adapters function absolutely stable with S3 sleep.
This crashed state seems to occur more often with Broadcom’s drivers installed instead of letting the P411W-32P remain as an unkown “Base System Device”.
For me this is the main reason for pursuing Broadcom to fix the firmware and/or drivers since I want to use the P411W-32P in a workstation scenario where the system is also allowed to go to sleep.
My curiosity* got the better of me and I ordered a Broadcom HBA 9500 16i. If you have any tests I could help you out with while you have to wait for yours just drop a message here.
*What I want to find out:
Does it have the same firmware issues as the ones from the 9400 8i8e I’ve been experiencing?
Performance comparison Broadcom HBA 9500 16i vs. P411W-32P when using NVMe SSDs?
Neat thing: With active HBAs you can use PCIe Gen4 SSDs without automatically cutting their performance on Gen3 systems like current Ryzen APU builds.
Does the 9500 16i (as a “Tri-Mode” controller) really not work with Icy Dock’s SFF-8643 and OCuLink U.2 NVMe backplanes?
If the 9500 16i turns out to also be shitty I think I might get the closure I need to finally move on from Broadcom for good…
Hopefully the card will arrive in 1-2 weeks. Unfortunately the P5800X order that said delivery date start of June when I ordered it, now has a new delivery date of Jan 2023
I’m awaiting some explanation.
I wonder if I should order a Micron 7450 U.3, though I’ve only seen the 7400 on the supported NVMe drive list. All my other SSDs are M.2 installed in the Asus Hyper M.2 x16 Gen4 card (or a very old SATA SSD in my old PC).
I think the next SSD for “serious” data handling I’ll be looking at will be from Micron, too.
(-) Not the fastest
(+) Good value
(+) Anecdotal personal evidence: Never had issues with Micron SSDs, had a WD SN630 “softfail”, meaning it lost its data (even SMART) and it wasn’t accessable until it got secure-erased.
(+) Firmware updates available for retail end customers (not with Samsung’s DC SSDs)
(+) 5 years warranty available for retail end customers (not with Samsung’s DC SSDs)
Ever since the P4500/P4510 debacle I don’t trust Intel NAND SSDs anymore (see also my previous Broadcom NVMe HBA experiences), Samsung are dicks regarding OEM SSDs and end customer service, Kioxa is a bit higher priced (in Germany), WD is a bit “retardé” so Micron seems the only one left.
My 9500-16i arrived today, had a quick look at it:
The issues I’ve been having with my two 9400-8i8es are also occurring in the exact same way with the 9500-16i.
Wouldn’t use a Tri-Mode HBA for NVMe SSDs, the two generations I could test so far (Broadcom 9400 and 9500) eat up quite a lot of performance compared to the pure PCIe Switches.
Confirmed: Icy Dock’s OCuLink backplane doesn’t work with the 9500-16i.
Will report back wíth more details after I’ve had more time with the 9500.
- When getting HBAs new I’d only get PCIe Gen4 ones for more the efficient use of PCIe lanes.
