The backplane is part of a kit and i have a hell of a time finding specs, so i am not sure if it’s hiding some relevant options for what I’m trying to do.
Will oculink → 4x sata or 8x sata cables just work?
Any trouble connecting oculink to SFF-8643 for the backplane?
Is there gen 4 pcie switches available that won’t break the bank? Gen 3 should be fine for my needs at the moment, but if i can get more versatility, i might as well.
I have not used pcie switches before but they seem like the right solution to getting more devices per physical port without too much of a performance hit. Is there anything I’m missing in general? Tips are very welcome.
Be mindful that a PCIe switch reduces your average bandwidth to a device by a significant margin. A 3-port switch (actually 4, but that includes the input port too) gives you 100% of the available speed on the PCIe bus just 30% of the time. The others likewise. The remaining 10% is overhead (commands to the switch itself by the kernel) and thus wasted bandwidth.
Having said that, you could Frankenstein a solution, provided that single 16x slot allows bifurcation. Use a PCIe to 4x M.2 adapter board and populate 2 M.2 slots with an SSD. The others is where the Frankensteining begins: slot 3 on the adapter gets a M.2 to PCIe x4 slot adapter cable. In this 4x slot is another PCIe to 2x M.2 adapter for the remaining M.2 drives. The last M.2 slot on the 4x M.2 adapter board is then populated with an M.2 to 6x SATA adapter. I know, it’s not quite 8 SATA ports, but at least you’re having 6 instead of none
Given all I’ve read about PCIe switches and cables, your best bet is probably to find a motherboard that has the connections you need and get that. It’ll probably both be cheaper and have a much higher chance of actually working than fiddling around with PCIe switches!
I don’t know about those white connectors or the signals they need so I have no specific motherboard suggestion though.
Finding the right one is difficult. This is the usual price/performance/power consumption chase, so my current try is making MC12-LE0 work. Finding good alternatives quickly becomes a bit expensive, while here i could potentially do it with a pcie switch and some cables.
Goal is all my storage, 10gbe, ecc memory and oob management basically.
After looking around and coming across this post by @aBav.Normie-Pleb , it seems like i can solve my problem with a pcie switch like this. Its either an oem part or a generic design from a bunch of chinese manufacturors. Possibly Linkreal.
With an adapter like this and those m.2 to 6x sata adapters in the link above and a bunch of cables, i should be able to get going, without throwing that much money at it…
I feel like there is a lot of jank in my future, wish me luck.
Should be doable with quality adapters since the ASM 1166 is only PCIe Gen3 x2, not Gen4 where things start to become a bit stressful.
BUT BE CAREFUL WHEN CHOOSING A PCIE SWITCH!
If it is advertised as something like “NVMe adapter” this CAN (doesn’t have to) mean that the firmware on the PCIe Switch only works with NVMe PCIe SSDs and not generic PCIe devices of every possible type.
This is done to add hot-plug support for NVMe SSDs without the motherboard/platform in general having to support that feature.
I could very well be making uninformed decisions. I’ve read a bunch of your threads and appreciate the testing you’ve done - if you have collected a list of suitable cards, please share. I’m getting slightly frustrated with my lack of knowledge
I read this thread and the " DiLinker LRNV9349-8I"adapter is the same as from other “brands” - shinreal, linkreal, lr-link and most likely a bunch more. All listed as using a Broadcom PEX8749, which should just do pcie switching? I’m not sure if they would do firmware changes that could trip me up. Their sites look exactly the same too, except different company addresses.
On top of that, Linkreal lists a card (can’t find the product but its in the sliding banner thing on their front page) that seems to be exactly the same as this one from Delock. Feels like a mix of crazy pills or the most niche and elaborate scam.
I would strongly prefer to be able to just configure things myself, but I guess they just change the minimum amount of stuff in the broadcom firmware…
The seller also included pics of “risers” (not sure of the proper term here) for compatible x4, x8 and x16 pcie slot adapters that i think is in their catalog.
I have had the adapter in my watch list and tried a few low offers without luck, but received a pretty decent offer back a little while after.
If you get that PCIe Switch from that eBay Seller could you ask for all possible Firmware configurations and post them here?
(x16+x16, x16+x8+x8, x16+x8+x4+x4, x8+x8+x8+x4+x4, x8+x4+x4+x4+x4+x4+x4, x4+x4+x4+x4+x4+x4+x4+x4, did I miss something?)
Thanks for getting me to test that, would have thought that it wouldn’t work and am honestly pleasantly surprised.
And pissed that Broadcom turned the PCIe Gen4 version of the same product (P411W-32P) into a piece of hot trash, that would enable fun and well-performing builds with PCIe Gen4.
Awesome! Thanks for testing that.
I’m grabbing the card i linked along with some cables and adapters. Price seems alright and the guy is helpful and quick to respond. Also means i can just get a regular AIC for more sata instead of m.2 shenanigans. I actually need more bandwidth than the ASM1166 would do
Once stuff arrives I’ll see about doing config. With luck, it’s just figuring out how ports are connected to the chip and faffing about with plxmon like it was suggested here. I’ll do a writeup if i can get the necessary information.
Slightly related: Ebays “no accepted offers in cart” rule along with “no combined shipping with volume rebates” can go jump off a cliff. It is just frustrating for no good reason and makes shipping and payment a hassle that has to be manually arranged.
Stuff has arrived and looks exactly as advertised. Still waiting for ram and cpu so i have not yet had a poke at it. Shinreal has been quick to respond and sent me the following information for configuring:
Start plxmon and click the EEPROM button in the toolbar
Load one of the included files or update manually and click “write”
Restart pc
Seems like the software allows for saving premade configs, so there’s probably a way to make this all cli friendly.
This is untested by me and there are still a few things i don’t understand. As for manual config:
Register 300h Port Configuration The port bandwidth configuration is valid only Port 0
Bit(s)
Description
Serial EEPROM andI2C
2:0
Port Configuration for Station 0 Port Configuration Link width, per Port, for Station 0. 000b = 0 = Reserved 001b = 1 = x16 010b = 2 = x8x8 011b = 3 = x8x4x4 100b = 4 = x4x4x4x4 101b = 5 = x8x2x2x2x2 110b = 6 = x4x4x2x2x2x2 111b = 7 = Reserved
Yes
5:3
Port Configuration for Station 1 Port Configuration Link width, per Port, for Station 1. 000b = 0 = Reserved 001b = 1 = x16 010b = 2 = x8x8 011b = 3 = x8x4x4 100b = 4 = x4x4x4x4 101b = 5 = x8x2x2x2x2 110b = 6 = x4x4x2x2x2x2 111b = 7 = Reserved
Yes
8:6
Port Configuration for Station 2 Port Configuration Link width, per Port, for Station 2. 000b = 0 = Reserved 001b = 1 = x16 010b = 2 = x8x8 011b = 3 = x8x4x4 100b = 4 = x4x4x4x4 101b = 5 = x8x2x2x2x2 110b = 6 = x4x4x2x2x2x2 111b = 7 = Reserved
Yes
31:9
Reserved
Need to figure out if this is applicable to just this Shinreal card or all PEX 8749 cards, including silicon revisions. And what “Station” is.
Will be fun to mess with once the rest of the hardware arrives
Meanwhile, the PEX 8749 product brief is an interesting read. I havn’t really seen any direct pcie machine ↔ machine stuff in the wild.
Seems like i will have to wait a while for the last pieces of hardware to arrive.
Meanwhile, the product brief says the 8749 can do some pretty neat pcie monitoring, including full gen 3 speed data generation and error simulation. I’m curious if this is just available on all 8749 cards.
If your card is in a machine with a gui, you should be able to get some interesting things via plxmon and the monitoring should have no performance impact. Having read some of your pcie error hunt adventures, there should be quite the rabbit hole there
