Most people unfortunately don’t look for PCIe Bus Errors (picture lost packets from a network connection) or know what PCIe Advanced Error Reporting (AER) is or test that PCIe AER is actually working for the used PCIe lanes.
Up to a point the error correction can keep up without dropping the PCIe interface link from Gen4 to Gen3 for example (but the error correction is obviously eating up performance), depending on the platform this can lead to weird stability issues like stuttering of everything from the mouse cursor to audio, black screens and complete system crashes.
I had the Broadcom card in my cart and was ready to play the firmware dance when I saw this post. I decided to give ATTO a call and ask about their two Gen 4 PCIe/NVMe switch models. I spoke first to Mike in sales then Fred in tech support and thoroughly enjoyed chatting with them as they seem to be a company of techies.
They did not disclose the core chip OEM, but it was hinted it was not Broadcom. They also said they test their cards with direct connection and have no plans to follow Broadcom in requiring UBM communication before recognizing downstream devices. Great news, IMO!
I asked about drivers and it sounds like the device will work out of the box with drivers that have been upstreamed in Linux kernels found in latest LTS versions of the major distros. Their propriety driver is required if you want to use their monitoring tools, however, they said the out-of-the-box kernel driver will expose the devices as NVMe nodes and the NVMe MI over SMBus communication should work (e.g. for formatting custom namespaces with nvme-cli). I did not get a chance to ask which vanilla kernel module would take responsibility for the device. I’m afraid I have 0 experience with NVMe in Windows so wouldn’t have known what to ask there.
Given all this, I’m going to evaluate the ATTO ExpressNVM S48F NVMe Switch Adapter for a project if I can find one. At this $1320 price point (+ cables), I can no longer say “m.2 NVMe looks cheaper than SATA these days”, at least not at scale.
I don’t have an answer. It’d be neat if it were a pure PCIe switch in this way, but my own exploration was driven by storage needs. The kernel module used or lspci tree may shed some light. If you’re curious, consider giving them a call during Americas hours. If I end up talking with them again, will ask for you.
Realtek has an RTL9101 PCIe 3.0 x2 to 9-port SATA controller pending. I get no search hits on the RTL9101 at the moment but presumably something will turn up in the next year or so.
I have Jeyi’s Wareship Pro armored JMB585 M.2 5-port SATA in test. Runs warm, increasing to hot as fewer drives are plugged in, but is mechanically preferable to ASM1166 equivalents.
I’ve benched the JMB585 out to five JBOD drives and where it seems to saturate at ~1.5 GB/s. Looks fine for 3.5 performance and passthrough but can’t recommend the 60+ °C heatsink temperatures. Voids packed with thermal putty I think the JMB585 on the Jeyi might be ok with five drives plugged in. But even then the no-name ASM1166 card I’m also evaluating is a good bit cooler. Absent an armored ASM1166 (or ASM1164), M.2 my preference would be to load balance drives across ASM1166 cards in x1, x2, and x4 slots plus chipset SATA ports.
Dendryte. this thread is really long, so I’m sorry if this has been asked before.
What’s the recommended x16 to 4-port U.2 card that I should use to add Optane 905Ps using my x16 slot? I want to be able to install the SSD into the card without dealing with finding good cables. PCIe 4 support would be nice, but I only need PCIe 3 for now.
My Ryzen motherboard with Ryzen 5700G exposes x8/x4/x4 bifurcation, so I’ll be able to add 3 drives.
I see some recommendations for Delock adaptors, but I can’t seem to buy them from Amazon or Newegg.
BTW: Is there a tool to display the actual amount of data transferred per second over a specific PCIe connection no matter the type of connected PCIe device?
Has anyone seen anything on ASMedia or Promontory routing behavior between SATA ports and PCIe lanes? I’m going through a pile of measurements and there’s some things which hint at constraints on how drive transfers can flow across lanes. Could just be happenstance in the tests but I’m curious if there might be something structural behind it.
I’ve looked occasionally but haven’t had luck. From what I can infer it seems PCIe isn’t instrumented this way, just the endpoints.
I finally was able to get some working parts after ordering a slew of items. In a previous comment here, I was trying to move some M.2 drives elsewhere in my case for cooling reasons (not a typical need found here I know) and had been using some ADT-Link cables that would fail within a month or two of successful working.
What I have ended up with is an M.2 to Oculink ReDriver, U.2 to M.2 adapter and a PCIE Gen4 rated Oculink to U.2 cable. The cable unfortunately is on the longer side at 50cm but I wasn’t able to find anything shorter in these connection types. The good news is this is the first set of hardware chain that seems to work at all in terms of getting the drive to show up in Windows and be able to be written and read from successfully.
What I now want to do is make sure I am not getting WHEA errors and be able to see the speed that is being negotiated by the drive and monitor that to make sure this is holding up and doing what I want it to in the long term. Can anyone point me to how I might be able to do that? I am having trouble trying to figure out what speed the drive is negotiating and any WHEA errors. Thanks to anyone who can point me in the right direction.
For reference this is the list of parts I purchased:
However, I can only get PCIe 3.0 read speeds (~2.9GB/s) on both drives with this setup. I suspect it’s the cards (really because I don’t know what else it could be…) Or is there a something I might be missing? I don’t see any “PCIe Bus Error” in journalctl.
Now, since the board has a SlimSAS x8 NVME connector, I’m thinking about getting one of these two cables instead. They should be able to deliver PCIe 4.0 speeds, right? (I don’t have any experience with SAS and friends…)
Before you go out and buy more gear validate the connection speed of your drives.
As root for your Kioxia’s in the output of lspci -vv.
In the “Capabilities” section you should see a line like LnkSta: Speed 16GT/s, Width x4 in case it’s connected via gen 4 speeds or LnkSta: Speed 8GT/s, Width x4 in case it’s connected via gen 3 speeds.
In case of gen3 speeds identify all the devices between root and your Kioxia drives in this tree view: lspci -vt. Repeat the step from above and identify the first component that’s not running PCIe Gen4 speed.
Hah, your right! For both drives, I get: LnkSta: Speed 16GT/s, Width x4
But why could the read speeds be so low then? I’m running the Gnome Disks Benchmark w/ 1GB sample size. It always jumps to 3.8GB/s and then quickly drops to 2.9GB/s…
Is PCIe AER working for the PCIe Slot the PCIe-to-U.2 AIC is installed in?
Error correction working overtime could be a possible explanation for lower useable drive speeds even if the interface link is correctly negotiated with PCIe Gen4 x4.
Can you test the card in a different slot?
Can you try booting Windows (SCNR) and do a CrystalDiskMark benchmark run on the drive?
Try running the same test with the PCIe slot forced (in BIOS) to gen 3. Is performance now consistent?
That would be another indicator for issues due to gen 4 signaling.
Definitely check for Gen 4, but I’ve also gotten massive quantities of AER log entries with dodgy Gen 3 adapters. (But you only really need to check if one of the components in your link were obtained from Amazon third party, eBay, or AliExpress. )
)