I don’t think durability works like in your calculation. The total durability will be determined by the min(durability of single device) not sum(durability of single device). Especially, if you plan on configuring all 21 m.2s in a RAID0 configuration - otherwise the config will not reach the expected 42TB of storage.
But what makes you think that in a RAID0 configuration the total bytes written would differ significantly from other RAID member drives?
Actually, this would apply to striping with ZFS, since ZFS can ‘load balance’ the writes, so that some drives that are faster would get more writes than drives that are slower. But with identical drives this shouldn’t cause too much of an issue even with ZFS.
Scratch that comment. I am thinking of durability as ‘expected time to failure’.
You are correct in that it is possible to write more data (TBW) to an array of disks than to any single drive (although I doubt you’ll get anywhere close to the numbers cited).
However, if you are concerned about durability and intend to use this array of drives as a single logical device, you’ll need to add redundancy in some form (RAID1, RAID5, RAIDZ1, …) which will reduce the usable capacity to less than the total capacity of the array.
Sure but MTBF is 2 million hours so 228 years (!). So what are the concerns about durability? Perhaps that the specs are wrong, that is possible.
Redundancy is not always needed. I intend to use two high capacity harddrives (18-20TB) in stripe to act as backup. And perhaps some online/offsite backup that i sync monthly. With ZFS you can do incremental backups nicely and efficiently, also remotely.
RAID-Z1/2/3 does not work well with random IOps. Unlike RAID5 where multiple drives can do individual I/O, so parallel I/O, this is not possible with RAID-Z1/2/3. One I/O will affect all drives and as such the IOps is the same as a single drive, which is not the cause for traditional RAID5.
Also, expansion for RAID-Z1/2/3 is still a work-in-progress (see: RAIDZ Expansion feature) and might take a year still. Expansion of striping has been available since ZFS version 1 i believe and is considered very safe, since it does not shuffle any data around. It also works instantly (<1 sec).
I value any feedback you and others have on my plan. But… it is pretty cool isn’t it? For not that much money you get ultra-enterprise grade performance, durability and capacity. The only downside is that the SSDs do not have PLP (power-loss protection) so they can corrupt themselves on improper shutdown. But since i would run RAID0 i will protect the data another way, using automated backups. Every morning the harddrives which act as backup would spinup and receive an incremental snapshot using ZFS script and then the drives spindown again.
Super low-power, low noise; this seems the holy grail of enthusiasts?!? 
Broadcom P411W-32P Summary for Direct Attached Devices
This is the latest working firmware:
https://docs.broadcom.com/docs/P411W-32P_4_1_2_1_HBA_signed_P14.2.fw.zip
Firmware newer than v4.1.2.1 requires a UMB backplane before it will detect anything storage related.
Be careful about drivers - it’s perfectely fine to let Windows track them down for you.
Edit:Windows 11 appears to be a different process. Caution is advised!!
These cables work at PCIE 4 speeds:
| Manufacturer | MPN | Cable Description | Adapter Connector | Backplane Connector |
|---|---|---|---|---|
| Broadcom | 05-60001-00 | x8 8654 to 2x4 8612, AltWiring 1M | 1 x8 SFF-8654 Slimline | Two x4 SFF-8612 OCuLink |
| Broadcom | 05-60002-00 | x8 8654 to 2x4 8643 (W), SMC 1M | 1 x8 SFF-8654 Slimline | Two x4 SFF-8643 Mini-SAS HD |
| Broadcom | 05-60004-00 | x8 8654 to 2x4 8654, 9402 1M | 1 x8 SFF-8654 Slimline | Two x4 SFF-8654 Slimline |
| Broadcom | 05-60005-00 | x8 8654 to 2xU.2 Direct, 1M | 1 x8 SFF-8654 Slimline | Two U.2 SFF-8639 |
| Broadcom | 05-60006-00 | x8 8654 to 8xU.3 Direct 1M | 1 x8 SFF-8654 Slimline | Eight U.3 SFF-8639 |
| Broadcom | 05-60007-00 | x8 8654 to 1x8 8654, 9402 1M | 1 x8 SFF-8654 Slimline | One x8 SFF-8654 (SlimSAS) |
| HighPoint | TS8I-8639-060 | x8 8654 to 2xU.2 Direct, 60CM | 1 x8 SFF-8654 Slimline | Two U.2 SFF-8639 |
| Micro SATA Cables | OCU-1708-GEN4 | x4 8611 OcuLink to x4 8611 OcuLink, 50CM | 1 x4 SFF-8611 OcuLink | One x4 SFF-8611 OcuLink |
Adapters and Backplanes that work at PCIE 4 speeds:
Icy Dock EZConvert M.2 to 2.5" U.2 SSD Adapter MB705M2P-B
Icy Dock 4x U.2 backplanes MB699VP-B 2 and MB699VP-B V2
SSD wise… Just pick one.
Or 32 of them…
Edited to remind folks not to update past a certain firmware version.
2 Likes
Got this baby which turned out to be exciting and disappointing at the same time—it works, but it blocks my second PCIe 5.0 slot. It was claimed to be PCIe 4.0 signal-compatible. I got them so short because I also expect them to work with PCIe 5.0 at that length. Unfortunately, I have yet to get my hands on a PCIe 5.0 U.2 drive to test it on.
Perhaps I should get a different bracket that holds the drive upright instead of on its side.
2 Likes
Hi, can you give me a little help with my P411w-32p?
I purchased a P411w-32p some days ago, but it will BSOD in the already installed Win11 system, and when I plan to reinstall Win11 system, it will also fail to install because system thread exception not handled.
However, it works fine under win10 (even the latest win10 22h2), and the firmware only works with 4.1.2.1, while 4.1.3.1 can’t find the hard disk (the latest firmware under win11 can enter the system normally but can’t find the hard disk, while 4.1.2.1 will BSOD).
My CPU is 13900K, motherboard is ASUS Z790 EXTREME, and P411w-32p can only run at x8 bandwidth because of the presence of the graphics card.
I can’t think of any better way to do it, can it only run under Win10?
In reference to the prior post, this is the M.2 to U.2 adapter: PCIe 4.0 x4 U2 Interface SFF-8639 To M.2 Key-M M2 Adapter Riser Card Ribbon
1 Like
I guess this’ll be an experiment to try out windows 11 with my card.
As far as I have seen in this thread, you’re the only one to have tried this with windows 11.
Known good working configuration is with windows 10, Linux, ect… and firmware up to 4.1.2.1 and no higher.
Firmware newer than 4.1.2.1 requires a UMB backplane before it will detect anything storage related. There are posts about this throughout the thread.
Update: I successfully used P411w-32p on Win11.
- Install another win10 system to update the firmware to 4.1.2.1 (or downgrade).
- removed the P411w-32p out of the system.
- removed all mpi2ses.sys, itsas35.sysh and itsas35i.sys related file names on the win11 system.
(Because I was using the 9500-16i before so many folders have these files, in fact a newly installed system should just delete the files in the DriveStore.) - Reinstall the P411w-32p device, you will find the system pex88xxx device and display error code 35, at this time to install the P16 (not sure if the updated driver can be used properly) driver and reboot can be used.
Because the system’s DriveStore has 95xx HBA/Raid card P25, P18 drivers cause P411w-32p match on these drivers and then blue screen.
And I use the backplane is icy dock MB699VP-B V3, can be used normally.
2 Likes
Now, please try to use system sleep (suspend-to-RAM, S3), wake the system up again and access and change files on a connected SSD 
No problem at all … I created a txt file in the hard drive connected to the P411w-32p, went back into the system after S3 sleep and modified the txt, the file was modified normally, no problems with unreadable, file hard drive does not exist, BSOD.
(The hard drives I use are intel’s P4610, P5510 and Samsung’s PM1735, all without problems)
1 Like
Welp! I had played around with the 2023-03 update of Windows 11 22H2 and my P411w-32p. Fresh install, no internet, and at first, all I was getting out of the ordeal were BSODs.
A word of caution to anyone who wants to try this in the future…
The driver available out of box will cause BSODs with firmware version 4.1.2.1.
My solution was to boot without the card, rename ltSas35i.sys in C:\Windows\System32\ to ltSas35i.sys.ori
Then reboot with the card, rename ltSas35i.sys.ori back to ltSas35i.sys, and disable the “Broadcom PCle Switch Management Port PEX880xx” in devmgmt.msc
Worked like a charm afterwards. No need to tinker with the firmware.
I’m definitely avoiding W11. That was awkward and painful af!
1 Like
Just to be sure: That changed txt file remained changed even after a reboot of the system? I’m asking since Windows’ memory write cache could lead to a file seemingly being changed even without it actually being changed on the drive.
I’m not trying to sound too hyperbolic but you are the first user EVER I’ve seen outside Broadcom’s “engineering lab” that has seen S3 and waking up working with the P411W-32P. It happens on all of my systems, even if no SSDs or cables are connected to the HBA.
Could you please give a complete report of your hardware configuration including motherboard BIOS version etc. (memory amount etc.)?
I don’t want to experience a new hope only to get hurt again eventually 
Guess I’ll also take a look at Windows 11 now.
I opened the txt again today after restarting the computer and took a look at it, and it is indeed the correct modified text …
However, I am using a Chinese system, although I checked the state of S3 sleep is indeed enabled, can not fully guarantee the difference between the region and the system version.
Device information from intel Extreme Tuning Utility program (intel XTU)
System version is win11 22H2 22621.1485 Professional
1 Like
Thanks for these additional details!
I have been following through this thread, and I wanted to see what people were recommending right now for generic cables to connect SFF-8654 to U.2 at Gen 4 speeds. It seems a lot of folks were recommending Broadcom 05-60005-00, but I wanted to ensure that this wasn’t just with the P411W-32P that everyone seems to be using. I would be using this on an Rome2D16-2T with onboard SFF-8654 ports. Thanks all.