@cmmcnamara Let us know how it goes once you get it. We could use info on more data points and more diverse setups. 
I got a Broadcom P411W-32P NVMe switch adapter coming my way for some experiments I wanted to try. I hadn’t thought to check first what kind of hot mess I’d get myself into with this AIC, but I guess It’s time to start reading this thread backwards.
How do I go about finding a server like this, a ready to go platform that I can drop drives in and out of that is not too expensive.
Or should I just build a 7950x put some ECC on it and call it cheaper?
You’d only need one HBA and a SAS 3 expander. They can be found on eBay for cheap. $65 USD as of last year I believe.
Just looked… $26 USD Brand new.
Be aware that currently the only AM5 motherboards that actually can do ECC are from ASUS , ARock Rack and Supermicro.
ASRock and Gigabyte seem to have implemented cost-cutting in their motherboard designs (left the additionally needed traces for proper ECC between the CPU socket and the DIMM slots out). It’s a shame, with AM4 it was only MSI that was that cheap 
PSA: Don’t confuse DDR5’s On-Die ECC with proper ECC, the former is only there to increase memory chip yields, reducing cost for the manufacturer.
I emailed ASRock about this, about there pro RS board and got a reply saying it would be working. But honestly I do t want to believe them I’ll probably just buy an Asus board and slap together a cheapy workstation.
It’s such a shame though, because I really like ASRock, the bios and there boards are pretty decent for there prices.
ASRock seems to have changed their memory support specifications to include ECC UDIMMs again (wasn’t there during the AM5 models’ initial launch).
Be sure to actually verify ECC functioning properly no matter what motherboard you pick, it’s relatively easy, just overclock the memory and run Passmark MemTest, refer to this thread where we did some experiments with AM4 motherboards:
Ok awsome. I’ll be building a spare server rig. But for the moment I have a PRO RS.
Do any of you guys have a link to a 48gb udimm ECC vendor by any chance. That will post internationally?
I came across the Broadcom 9500 i8/i16. The official specification states it can support up to x32 NVMe devices. I searched wide and far but I can’t comprehend how that is achived. It appears backplanes are necessary but everything from IcyBox takes x4 lanes per SSD. What backplanes are there to support x32 drives with a single x8 SFF-8654?
Thanks!
You should ask Broadcom support. Even if you breakout the x8 into 8x x1, each of those have to use some kind of SAS expander for NVMe or PCIe switch.
It’s essentially marketing bullshit to advertise higher numbers than commonly achievable. 8i is 2x NVMe unless you get a breakout cable for more and sacrifice throughput in the process.
Thanks mate. Sounded to good to be true from the start, but you never know until you know.
I’m struggling with the some consumer PCIe limitations and would like to connect more than the U.2 I already have.
The thread has been tremendously helpful!
Thank you for the testing and sacrifices you’ve made. I just installed a Broadcom P411W-32P and it was loaded up with the 4.1.3.1 firmware, which wasn’t working. Upgraded to 1.8.4.0 and found that it did not work either. Only downgrading to 4.1.2.1 worked.
It still sucks a bit that Broadcom isn’t trustworthy (for example removing advertised features during the life cycle of a product, in this case support for directly attached SSDs which is why newer firmware versions aren’t working for you and us all). I generally would like stuff that handles file storage to be at least somewhat trustworthy.
Out of curiosity: Can you try out S3 Sleep with your system and check if it will cause your system to also crash when trying to resume operation?
I’ve seen 3 kinds of crashes:
Immediate BSoD afterwards
System in a semi-crashed state: Windows still works (not installed on drive handled by the P411W-32P), but no files will be changed when trying to access on drives handled by the P411W-32P. Device Manager will freeze when you try to detect changed hardware.
Trying to reboot Windows in a orderly way crashes the system.
I’m using Ubuntu. Sleeping usually happens instantly, but it took a good half a minute to go to sleep this time. Resume worked without a hitch. I’m replying to the post on the same machine. 
Are you using a backplane or directly attached SSDs?
I have the P411W-32P, but I’d like to add that strangely, this NVMe switch seems to have the same limitation and it even extends to U.2 devices with their own PCIe switches. I have a QNAP QDA-U2MP which puts two M.2 NVMe SSDs behind an ASMedia 2810 PCIe 3.0 switch. Plugging it in reveals only one of the M.2 SSDs. When connected directly to the motherboard, both SSDs were exposed and even got their own IOMMU groups.
Something tells me the Broadcom P411W-32P isn’t a pure PCIe switch.
My purchase of the P411W-32P was also predicated on the fact that it’d support 32 SSDs… Bummer. One step forward (hot-plugging) and one step back (one device per x4 port). I’m poking around to see what I can do with it, and hoping anyone who gets further along posts their findings. Might join y’all in banging on Broadcom’s doors real soon.
A post was split to a new topic: Old supermicro x8dh-f – thoughts on replacing internal components vs purchasing newer?
Has anyone confirmed that the MB699VP-B V3 is backwards compatible with U.2 SSDs? I thought the U.3 wiring was incompatible with U.2?
I am using a SFF-9402 PCI Express 4.0 x16 to Two SlimSAS SFF-8654 8i Retimer from Linkreal from Aliexpress. So there is no logic on the card to switch around the pinouts. It that how U.3 works or required?
It should be fine.
The Icy Dock MB699VP-B V3 backplane is just “pseudo” U.3: Only U.3 NVMe SSDs work in the MB699VP-B V3, even if you connected it to a Tri-Mode HBA that supports PCIe NVMe, SATA or SAS.
I’ve personally tested the MB699VP-B V3 with an U.2 PCIe Gen4 NVMe SSD (Samsung PM1733) and a newer U.3 PCIe Gen4 NVMe SSD (Micron 7450).
I am curious about what the difference between the V2 and V3 is? Seeing as you have both, do you have any idea what the wiring change was? It must be very minor if the V3 still works with U.2.