Subjectively speaking I just don’t want to accept that situation that it all fails because of “shitty wires”.
While I already have non-RDMA 40 GbE, I mainly use Windows on my desktop systems that handle large files and only Windows “for Workstation” and “Server” supports RDMA, so I would like to use swappable NVMe drives locally to not throw performance away
Windows for Workstation on the other hand doesn’t support/accept settings that comply with General Data Protection Regulation (EU), so I’m currently staying with Enterprise - changing Server to a desktop OS is a bit of a pain-in-the-ass.
No idea about your physical situation, but I actually have my computer in the garage, and use a combination of optical DisplayPort cable and OM4 cable and network compatible extron usb extenders to bring video and usb to my room and it’s worked flawlessly for almost 2 years now. I finally have the perfectly silent and non-room heating desktop I’ve always strived for, with all the power I could ask for.
I do appreciate your documentation. Hopefully it saves others disappointment.
Those types of server chassis come with cables installed, as the compatibility and connection quality of the motherboard, cables and backplanes is verified/certified as a unit and product line. Hence the enterprise price delta over the raw parts cost.
It’s a half humorous, half “but seriously though in 2022 this is basically the only fucking way to deal with this shit” suggestion that’s overkill. Unfortunately nvme connectivity is a half baked enterprise affair that simply doesn’t have a clean ad-hoc solution for the prosumer level yet.
My core desire is to connect NVMe U.2 SSDs to motherboards that only have regular M.2 or PCIe slots and be able to easily swap the SSDs;
In combination with PCIe Bifurcation no additional logic like HBAs is required in general;
The length of PCIe traces is pretty narrowly designed to perform optimally, while at least no issues with “quality” parts with PCIe Gen3, just adding a passive part to bridge the distance in my experience always leads to PCIe Bus errors when doing the same with PCIe Gen4;
Irony: An old SAS3 cable (SFF8643) that was bundled with an Intel SAS3 expander (in 2015?) leads to the fewest PCIe Gen4 Bus errors when using it with the SFF8643 variant of the 4 x 2.5" Icy Dock U.2 backplane;
This observation tickles my tummy that these issues have to somehow be possible to solve with “wire quality” and connectors with better high frequency properties like SFF8611 or SFF8654.
Silent systems are also an interest of mine. I’m currently designing a multi-room watercooling solution for computers in two rooms that uses the heat (worst case about 3 kW) to help drying clothes in a third room where a couple of radiators are mounted to a wall.
While also using a file server in another room I “have” to use bare metal Windows systems since I don’t have the energy to fight against goddamn DRM mechanics that don’t like virtualization.
Can’t speak to the other things, but silver wire is significantly worse with bending/pulling forces than copper. Also copper produced with a single crystal growing method can match silver as a conductor. Which is used to help hallucinate about better sounding audio.
I could imagine this might be the source of issues with available cables: OEMs claim to use suitable high-quality (=expensive) copper alloys but in reality its aluminum painted with a brown sharpie.
But I seem to be too dumb to install the latest firmware (4.1.3.1), it came with version 0.1.0.0 which doesn’t sound too trustworthy, can anyone see what I did wrong here?
After “doing” the update in cmd (run as administrator) as shown above I did an orderly shutdown, disconnected power for five minutes and upon checking back in, it’s still on version 0.1.0.0 :-/
There doesn’t seem to be any example documentation for the firmware update tool:
Unlike Broadcom HBAs the P411W-32P doesn’t show up in the LSA management software so you would have to do a firmware update manually - not normie-friendly
First test with Amazon’s “generic” 0.5 m SFF8654 8i-to-2-x-OCuLink completely failed, even PCIe Gen3 U.2 SSDs aren’t showing up in Icy Dock’s “PCIe 4.0” OCuLink backplane.
To be extra sure I’ve ordered some of the absurd 1 m cables from Broadcom that are officially supported, maybe then you can do a noteworthy A/B comparison with various third-party cables.
I sincerely hope that this time - not like with the HBA 9400 with its need for proprietary cables - Broadcom really did do the standard pin layout for their connectors on the HBA…
The intend was to connect U.2 SSDs (Samsung PM1733 7.68 TB) directly to the HBA/PCIe Switch to verify its performance alone without the possibilty of the Icy Dock backplanes interfering.
The picture is exactly the same as Test #01, no SSD shows up, not during POST or in Windows.
SSDs connected via the PCIe Switch should be able to be used to boot off, could verify this with the previous Delock 90504 PCIe Switch AIC, SSDs show up normally in the motherboard’s UEFI as if they were directly installed.
Nothing with the P411W-32P
Tried:
Reseating cable connectors on HBA and SSDs;
Verified that the SSDs have power;
Tried each of the 4 SFF8654 8i port on the HBA;
Tried an UEFI CMOS Reset while every component is installed;
Tried 2 different platforms:
Platform A: ASUS ProArt B550 CREATOR/5750G/128 GB ECC Micron; HBA in first x16 PCIe slot (running at PCIe 3.0 x16);
Platform B: ASRock X570 Taichi/3700X/64 GB ECC Samsung; HBA in first x16 PCIe slot (running at PCIe 4.0 x8);
The HBA and its chipset get rocognized in Windows (including temperature readings from Broadom’s command line tools) but as described nothing connected shows up, also no error messages (i. e. Windows Even Viewer);
Option 1: HBA has a strange defect (optically in perfect condition and shows up in the operating system, but somehow every single port broken);
Option 2: Every SFF8654 8i cable so far has been completely defective (not just errors under full load);
Option 3: Broadcom’s firmware is sh**;
Option 4: Samsung’s firmware is sh**;
FUN
Does anybody have access to Samsung Enterprise SSD firmware updates?@wendell ?
The PM1733 SSDs are the only thing I cannot verify to have the latest firmware, bought them over a year ago and Samsung doen’t offer anything for retail customers of enterprise SSDs (was a calculated gamble but they have been fine so far).
Both parts are listed by Delock to be able to handle PCIe Gen4 (64 Gb/s with 4 PCIe Gen4 lanes);
These are the worst parts so far I have ever had where the connected devices also show up with a PCIe Gen4 interface link;
A benchmark run with CrystalDiskMark (2 GB test file size) with a Samsung PM1733 U.2 PCIe Gen4 NVMe SSD causes the Windows Event Viewer Error log to max out (20 MiB) due to the reported PCIe Bus Errors with PCIe Advanced Error Reporting enabled in the UEFI;
HWiNFO counts 2,267,047 PCIe Bus Errors (6 already occured in the time during Windows booting and launching HWiNFO before the benchmark is actually performed;
Made these images a bit more explicit, I hope that if other people search these part numbers image results will warn them more effectively than text alone.
Tiny update: Broadcom’s Tech Support didn’t find any obvious compatibility issues with the tested hardware configuration and suggested to RMA the HBA P411W-32P.
@aBav.Normie-Pleb@wendell
I too just purchased a brand new P411W-32P from Newegg and began playing with it. On initial 0.1.0.0 firmware, everything shows up. However, upon updating to the latest (v4.1.3.1 - Shows up as v1.3.1), absolutely nothing will show up regardless of motherboard (Supermicro, Asus, etc…) or settings. HBA shows up in both linux and windows though.
Any ideas would be greatly appreciated. i will try the second latest firmware and report back.
i’ve done HBA firmware updates via the efi shell.
Update:
The previous update (v4.1.2.1 - shows up as v0.1.2.0) is not botched. Everything shows up as it should.
Now let’s see if updating to the latest will now work…
Note:
The process to return the card to stock is as follows…
g4xflash -i 1 erase -r 0 -s
g4xflash -i 1 erase -r 1 -s
g4xflash -i 1 erase -r 2 -s
g4xflash -i 1 erase -r 3 -s
g4xflash -i 1 erase -r 4 -s
g4xflash -i 1 erase -r 5 -s
To update is…
g4xflash -i 1 dl -f firmware-update-file.fw
Edit-Update:
The latest firmware is most absolutely, positively botched!!
DO NOT USE version v4.1.3.1 (v1.3.1)!!!
Thank you for your feedback and welcome to the forum!
I’m happy and sad to hear that; happy that I’m not completely incompetent meaning I’ve not done something obviously wrong and someone else can confirm the issues I’m seeing.
Sad that these blatant issues exist that should be caught by QC/QA before a firmware is published.
My P411W-32P is currently at Broadcom’s RMA facility in the Netherlands and I’m eagerly awaiting an update from them.
The support technician assigned to my case said that they personally tested a P411W-32P in their lab with the same latest firmware and drivers version and said that there were no issues…
My gut says that something serious is up with Broadcom’s HBAs. Also have two HBA 9400 8i8e that have been experiencing the exact same stability issues during operation (crashing systems when waking from S3 sleep, drives disconnecting after about 15 sec of max load and immediately reappearing).
The HBA 9400 line vanished from Broadcom’s support download/drivers web site, it’s pretty weird.
I’m treating this thread as a personal blog and will update it if I can clear something up (even if it’s just a monologue) and I welcome anyone to participate with their own experiences!
I don’t have much experience with the CLI tools, can you elaborate these steps?
Why is it not enough to “just” flash an older firmware version but do these erase steps beforehand? Or is this just good practice to be sure that no traces of another firmware version remain?
It is a pleasure to help.
I was starting to pull my hair out over this too, and knowing I wasn’t the only one experiencing these issues is what prompted me to try other….possibilities…
Thank you @aBav.Normie-Pleb for sharing your experiences. Sanity Checks FTW!!!
Firmware wise, I honestly do not have any idea why this does not work on our consumer / prosumer / server hardware with the latest firmware. Perhaps @wendell or someone who has access to more enterprise hardware than I could play around with what systems or configurations that play nice with this switch card with up-to-date firmware.
If you’d be willing to ask what configuration and hardware the tech is using for testing, that would be potentially helpful in this quest for answers.
To be precise, I tested this in an ASUS ROG Crosshair VIII Dark Hero (Ryzen 5900 + x570), ASUS Crosshair VI Hero (Ryzen 1700 + x370), and a Supermicro MBD-X11SSL-F (Xeon E3-1220 v5 + c232). I managed to find Highpoint PCIe 4.0 sff-8654 8i to 2x sff-8639 cables for under 40 USD each. I have a few other systems I could test in at a later point in time (I don’t feel like tearing my main system apart so soon after this set of shenanigans).
Tool wise:
There are three environments the g4xflash utility will work under – Linux, Windows, and EFI shell. The first two require official drivers to be installed. EFI shell does not. Pretty much any recent UEFI motherboard should have an EFI shell built-in. (My Asus x570 EFI shell did not want to play nice… Yay secure boot!!)
g4xflash -i 1 erase -r 0 -s
g4xflash —The utility in use.
-i number —Selects which card in the system to operate on.
erase —The operation to perform. In this case, clearing out memory. (I’ll elaborate why in a sec.)
-r number —Which section of memory to work with. There is a primary & alternate and each number correlates to one (Boot/SBR, Firmware, Config, and Log just to name a few).
-s —Silent. Don’t ask…. Just do it!!
Why do a full erase…? This is a required step in a different post for converting raid cards to IT Mode (LSI 2008 & LSI 3008 based).
However, in this case, I was concerned it would just ignore the new ….cough….cough…. older version I just flashed.
I also wanted to verify that the card worked before any updates.
Broadcom’s technician just vaguely mentioned “Seagate” SSDs when doing their lab testing with a P411W-32P. This worried me a bit since the only Seagate U.2 NVMe SSDs I know of is the Seagate Nytro XF1440 line that is only PCIe Gen3.
The last contact I had with the technician was me asking if the P411W-32P really uses industry-standard pin layouts for its ports, back when I first bought the 9400 8i8e I wasn’t aware that you need special Broadcom cables to use its SFF-8643 ports for NVMe SSDs (SATA and SAS works with standard SFF-8643 cables). Haven’t received a response yet.
Could I ask you for assistance regarding HBA flashing?
I’ve only used the LSA web interface software to flash my 9400 8i8e units (that have been experiencing issues for some time), could you please describe how to do such “complete” flashes with erase steps to reduce any risk of old firmware stuff remaining?
While you can’t find the 9400 8i8e on Broadcom’s download site any more you can bypass this by showing all files for all HBAs:
Here is a link to the latest firmware package P22, note that the firmwares to flash consist of three parts (“Firmware”, BIOS" and “EFI”, a bit annoying):
Something else I find odd: There are two firmware types for Tri-Mode adapters, one supports SATA/SAS/NVMe and one supports SATA/SAS-only. I don’t understand the necessity for the latter if the controller is able to do all three protocols with one firmware type.
