A Neverending Story: PCIe 3.0/4.0/5.0/6.0/Sci-Fi+ | Bifurcation, Adapters, Switches, HBAs, Cables, NVMe Backplanes, Risers & Extensions - The Good, the Bad & the Ugly - Will there be a final solution?

Hardware Hub Motherboards
627

You can still get 4 nvme drives onto 8 lanes its just not advisable to have them attached to a 9500. If you are going this route you should use something like the AOC-SLG3-4E2P (gen 3) hba or a gen 4 alternative. You won’t be able to attach sas or sata drives to it, but it would be much better than putting it through the 9500-16i. These kinds of adapters are a far more robust solution designed to handle 4 nvme drives at once and should be pretty platform agnostic, at least this one from supermicro. If you just want something to plug the drives in and don’t care much for the performance hit and can deal with a few issues, the 9500s would probably be fine. Sorry if it feels like I am giving you grief haha.

Also, that sas expander has a sister card that can sometimes be found for cheaper. Its the intel RES3FV288.

The gen 2 epycs might be in your ballpark, depending on what youre looking for. Some have gotten incredibly cheap, and the CPUs are cheap too if you don’t mind the lower clocks. The slightly more obscure chips that werent there at launch also tend to be cheaper. The supermicro boards for these are highly sought after and are way higher priced than the other boards on the market. Got a 32 core 7532 cpu and an mz32ar0 for ~800 iirc, was a pretty damn good deal. It’s definitely something to look out for as the market changes.

Good luck on your build, seems like itll be a fun one :).

1 Like
628

I gave a long thought again and after reading up I agree that the 9500 tri-mode is probably a bad idea.

Does anybody have experience with the Supermicro AOC-SLG4-4E4T ?
Will it work on a non-Supermicro board?

I would like to hook this up to a Icy Dock ToughArmor MB699VP-B V3, but now I’m struggeling to find the right cables for that. “SFF-8654 Port to ​SFF-8612” cables is an easy enough search term, but the range on offers is wide. Can anybody recommend a cable/vendor for a round about 40cm length? If I already get a controller with retimer etc. I don’t want to skimp on cables but the market seems like russian roulette for the non-initiated.

1 Like
629

The slg4 4e4t is a retimer card btw not a pcie switch card. All it does is repeat the signal to maintain signal integrity through cables and out to backplanes. It doesn’t do port multiplication at all, and will only put through the amount of lanes you give it (given it has power to do so), e.g. 8 lanes = 2 drives, 16 lanes = 4 drives. If you want gen 4 speeds out to something like that icydock (if it supports this), then it would be great to send the signal out to it from a pcie slot, but it would not be a good option if you are seeking to get 4 drives shoved into 8 lanes

2 Likes
630

I had that exact setup until recently:

  • Icy Dock ToughArmor MB699VP-B V3
  • Supermicro AOC-SLG4-4E4T

The following cable worked for me:

5 Likes
631

Anecdotal, but I have heard that their retimer cards are simply plug and play, and that their redrivers are less so. Retimers are far better than redrivers anyways if you have that option.

1 Like
632

Gorden-Question-1

I need a PCIe Controller card(s) to connect around 32x SATA/SAS Drives.
12 drives have a connection via 3x SFF-8643, the rest via a SATA port.

I don’t need any hardware RAID functionality and I have see the Broadcom HBA 9500-16i.

The question is, can I use the HBA 9500-16i with for example JBOD off, so that I don’t need any addional driver to see/use the discs in Linux. Like with an “Delock 5 port SATA PCI Express x4 Card (90498)” or I need an older one like the 9305-16i.

Thank for the support!

633

What you are looking for is called an HBA (host bus adapter). An HBA is used to convert pcie into ports that you can connect storage to. Some do sata, some do sas, some do nvme. A raid card is an HBA that has some additional functionality built into it for running things in raid configurations even without support from the operating system. You can disable the raid functionality on these raid cards and run them as simple HBAs to just show what is plugged into them to the OS. Usually this is called IT mode (with raid mode being called IR mode), though it might be called something different on some cards. The LSI cards are the most famous, well used, and well documented of he bunch in the world of HBAs and RAID cards, and for good reason. The LSI 92xx and 93xx cards are reliable, robust, and trustworthy options. The later generations (94xx, 95xx, etc) have been said to be less reliable/trustworthy at least on the nvme side, which I can back up with my own experiences, but I cannot say that I know much about how good they are as simple sas/sata controllers. It makes no sense to buy a 9400 or 9500 if you are simply using sas/sata drives, however, since it’s not like the drives need any more speed than the 93xx options have with their famed robustness. Older cards than the 92xx do not support drives larger than 2tb or speeds above 3gb. The adaptec cards are decently reliable, but have their issues. Again, anecdotal, but there is a decent consensus (that I agree with, having attempted to use some before) that they are the lesser option. Now, there is a slight offshoot of these HBAs that are usually called SAS expanders, which basically act as a network switch does and allows you to connect multiple different devices through the same uplink. If you plug one sas cable into it from the HBA and 2 cables to the drives, the maximum speed that it would be able to do is the speed of the link to the HBA. Some sas expanders accept multiple links to the HBA to allow for higher maximum uplink speed, and some do not. It depends on the model. It might make sense to avoid a SAS expander if you are using lots of high speed drives, depending on the the set up. These sas expanders are useful in that they do not require additional lanes/slots, and they will (most usually) simply add more ports to your HBA/RAID card without increasing the cost that much. You can also daisy chain them, to an extent. They need power, which some use the pcie slot for, however it does not need to come from the slot if it has a molex/sata power connector on it or if you use a pcie riser board that provides power(usually found in gpu mining setups). One last technical thing about HBAs that you should know, their ports are 4 lanes each, so if it is a cable that says it is a 6gb cable, it is actually 4 lanes of 6gb (24gb).

Real quick, heres the naming convention that HBAs and RAID cards from the broadcom/adaptec/lsi side usually follow:

LSI 9200-16i

  • The first number slot, 9, is irrelevant. Skip.
  • The second number, 2, tells you that this is a second generation card. 92xx cards support 6gbps per lane, 93xx 12gbps, 94xx 12gbps and gen 3 nvme, 95xx and 96xx 12gbps and gen 4 nvme.
  • The third and fourth numbers are different models within a same specification, some being models with different specifications, form factors, or technical design.
  • The number after the dash is a multiple of 4 designating the number of direct connections the card has. The physical connectors carry either 4 or 8 of these connections. A 16 means that you can directly connect 16 drives to the card.
  • The letter after the port number, either “i” or “e”, tells you whether it has external or internal ports. If it has both external and internal ports, it will tell you how many it has of each in the name.

Anyways, what you likely want is an lsi 92xx or 93xx card. Since you are only using sas and sata drives, you won’t need any fancier NVME supporting adapters (94xx or later). Now, depending on the type and speed of the drives you are connecting, you could go a few different ways. If you are using hard drives and a few sata ssds, it would probably be best to either go with a 9300-8i (cheaper than 92xx right now lol) and a 82885T sas expander, which supports both slot and molex power. This is the cheapest and easiest way to do it 32 drives in a reliable and trustworthy way. Well, at least that I can find right now lol. Slot or molex power to the expander and an sff 8643 to sff 8644 cable from the hba to the expander and you’ve got yourself 32 (36 if you count the other external port) ports to plug sas and sata drives in. Not bad. If you want something that will directly attach all of your drives, a great option is 2 of the 9305-16i. Quite a fair bit more expensive, but with no bottlenecks to be found for even the fastest 12gb sas ssds. The 9300-16i is another great option, but it consumes far more power. It can be had for cheaper, however, if that is what you care more about. It takes slightly more than the slot can support in terms of power, though, and the motherboard ight not like it, so it has a 6 pin pcie power connector on it.

Let me know if you want me to pick something out for you. I can probably find you something that is cheap and fits your needs if this doesn’t cover it.

Dont forget to cool these adequately by the way. They need active cooling. Use zip ties or twist ties to strap a fan onto the heatsinks of whatever solution you choose. A 40mm will probably be fine.

2 Likes
634

Hi Rat
Thanks for the explanation :slight_smile:

…and a 82885T sas expander, which supports both slot and molex power.

You mean to power the drives over HBA card? I don’t need to power the drives, they are connected directly to the power supply.

Maybe I should have added a little more details right away :slight_smile:
For the moment I will use normal SATA HDD (MG10AFA22TE) and some older ones drives in this slots.
The price for the 82885T is around 200 CHF and for the 9305-16i, 9400-16i and 9500-16i around 300 CHF. Not really a difference if I buy gen 3,4,5.

I have also read a bit on this topic and I was not really sure if its worth to take the newer HBA 9500-16i card or the older one like the 9305-16i, because of the change to tri mode and some other issues.
If I am not mistaken then there is also an issue with the heat with the older ones and with the newer once not, if you are not using NVMe drive.

My requirements are more or less to be able to connect the 32 drives, have a good speed when copy big data blocks to the drives from a NVMe which is connected directly on the mainboard and have a HBA where I don’t need to install special drives in the Linux OS.
I’ve already fallen into the trap: I’ve bought the RR2840C from Highpoint because they wrote on their website and in the manual to support Arch Linux, but now the support wrote…

Due to policy changes, we no longer support Arch Linux.

and the special drives don’t compile without having to invest much effort.

Best Regards

635

It varies by adapter but the range I’m aware of is ~27 W down to 6 W typical for the 9500-8i. For 9500-16i and 9305-16i, 9 and 16 W, respectively.

It’s unclear how Broadcom defines typical but presumably ~4-16 W watts idle, depending on the adapter and its ASPM support, plus a linear function of transfer rate would be a reasonable approximation.

2 Likes
636

Typical power consumption for the 9305-16i is 16 watts actually, per https://docs.broadcom.com/doc/pub-005245
The card you are probably thinking of is the 9300-16i, which was 26.9w iirc. Has an extra 6 pin, since the power standard for x8 is only 25 watts. Stricter boards would require the extra 6 pin. Had one of them, and yeah it ran hot, but its around the right size to perfectly fit a slim 80-90mm fan lol. Cable ties and a dream. It consumes so much power because they duct taped together 2 of their 8 lane supporting controllers with a pcie switch to get that one to work, no joke.
(Pcie switch left, sas 3008 controllers right)

image
image1016×582 30.9 KB

The 9305 has a newer chip that supports up to 24 lanes per controller. It was released later to fit low profile installations iirc. Also, according to this post, the 24 port variant of the original was a single 8 lane controller with an expander duct taped on. Go figure lol.

Not sure what typical means in these cases, but on the 9300-16i it just says “The power requirements for the LSI SAS 9300-16i HBA under normal operation are as follows”. Odd stuff.

2 Likes
637

Hahhahha no the cables from the hba and expander do not power the drives. The expander itself needs power, which is why I mentioned that it can be powered by either molex or a slot :slight_smile:

Ah, sorry, I made the mistake of assuming you were in the US. After conversion, that price seems absolutely absurd. What in the world lol. What is causing these prices? Import fees?

Any of them are reliable enough with sas and sata drives that it shouldn’t matter… probably. As I said, I haven’t heard much if anything about their reliability with sas/sata drives, and so they seem fine. I managed to get a 9400-16i for cheap to replace my 9300-16i and I haven’t experienced much issue aside from a lack of documentation. My experience with nvme over it has been pretty awful though lol. I am curious what the cabling is going to cost you depending on what you need, however. Newer HBAs will come with different connectors that will have to be adapted to either your drives or backplane. Newer cables usually means more expensive.

The power/heat thing is mostly (if not solely) due to improvements in the silicon or design by the way. Neither old nor new is an issue if you just strap a fan on em, which I beg you to do with any of these regardless of power consumption. The differences are pretty big between the two but not by a large actual amount by the way, with the 9305-16i saying around 16.2w (here, page 6 ) versus the claimed typical power of the 9500-16i of 8.5w. Huge difference between the two, but only ~8w difference all in all. It may factor into which one you will buy, who knows :person_shrugging: . If they are all the same price and the cables are pretty decently cheap and you care greatly about power consumption, then maybe go with the newer one. If you want tried and true, I always have a few 9200 and 9300 cards lying around to hand to those in need lol. Never had an issue with them aside from the learning involved in understanding how to update and flash them haha. Not sure if I am getting this right, but if google is to be believed, CHF is the swiss franc, in which case those prices for the cards are absolutely absurd. A 9305-16i being 350 US dollars is incomprehensible… I dont know if you have the ability to use a shipping forwarder or something to buy a card from the US where it is cheap and get one to you that way, but if you can do that, I would absolutely recommend it. Haven’t looked at prices for these in a while and they have gotten insanely cheap lately. An inspur badged 9300-8i is ~20 dollars here (~17 CHF), an LSI 9300-16i ~30 dollars (~25 CHF), a 9305-16i ~75 dollars (~63 CHF), and a 9500-16i ~190 dollars (~156 CHF). The 82885T sas expander is ~20 dollars here as well (~17 CHF). It is incredible how much cheaper it is here if what you are saying is true. If you want direct links to reputable listings where you can purchase these, let me know what you would like and what cabling and I will send you the listings for them that are from reputable sellers at reasonable prices. I hope youre able to work out a solution from this that will not cost you an arm and a leg… Good luck.

You should have no issues with this regardless of your choice at 93xx series and beyond with hard drives. The drives themselves are the limiting factor here, depending on config.

Do you need a raid card? IIRC arch linux has access to mdadm, which would allow you to raid your boot drive, removing any need for OS support of RAID. If you remove that need, you can just go with a bog standard HBA and avoid all of the raid card bs.

Oh, do you mean drivers? I guess that would make sense, however mdadm is a tried and true solution for this kind of thing that is more trustworthy than a RAID card. If you are alright with the rather small performance hit and are able to do so, go with the software RAID solution and avoid the hardware support requirements, penalties, and pitfalls entirely.

2 Likes
638

Okay :slight_smile:
I had see some cables with SATA and power connections to the SFF Port, but maybe this ones had an extra power connection.

Yes the currency is Swiss Francs.
And yes on this prices are Customs costs and taxes. The Taxes are everytime included on the shop for B2C. And I’m not sure how it is in the US, but we have in swiss and also in the most EU country 2 year warranty from the shop. Maybe this is also a reason for higher prices.
So far I know in US the prices on the websites are without taxes in the shop?
When you have some recommendations for US shops, that would be great. But I don’t know if they send to swiss, but lets check :slight_smile:

Here is a compare website over some shops in DE/AT/PL/UK with the controller cards:
geizhals.eu/?cmp=1203566&cmp=1507201&cmp=1717529&cmp=2282213&cmp=2944736&cmp=2944720&cmp=2944708&active=4

The prices are similar in swiss.

I was trying this card because it was cheaper as the other ones and they wrote support on their website.

Yes I mean drivers, I only have the German spell checker in my browser :sweat_smile:
Yes I already have a Software RAID running with mdadm, but with less drives and only 50TB. So far it is runing very well since 6 years now, but now its more or less full.
Thats the reason why I search for only a HBA without RAID functions. before I had only a SATA extender for 7 HDDs. As a software developer, I’m not so much into hardware and I haven’t had any experience with the HBA cards so far.

Thank you for your advice and support!
best regards from switzerland!

639

Can confirm I’m seeing similar prices in PL, e.g. here: HBA Controllers - Getprice

Seems to me like a supply vs demand issue. ebay is also not helpful and incurs high import costs (shipping and VAT). And most (sensible) offers are from China.

A spellchecker wouldn’t help you here, both drives and drivers are perfectly valid in this context :wink:

640

I’m offended on your behalf, these prices are so awful that its funny.

RIP.

Just checked a random one and the 9300-16i is very nearly 10x the price it would be in the US if translated to USD. Is it different if I view it from you? Am I misunderstanding this whole thing as being a huge price issue for you? If so, my bad. Ill continue anyways just in case they are really that expensive. It’s hard to believe.

The taxes (shipping on the ones I found were free) would add maybe 15% to the cost.

Some might. These say they ship to Switzerland, and it could even be true! lol
9400-16i: Lenovo 430-16i (LSI MegaRAID 9400-16i) 12Gb/s SAS SATA NVMe Tri-Mode HBA PCIe | eBay
or Lenovo 430-16i / Broadcom LSI 9400-16i SATA / SAS HBA Controller 12G IT Mode ZFS | eBay
9305-16i: SAS9305-16I LSI SAS 9305-16I 16 Port PCIe 3.0 x8 12 Gb/s Host Bus Adapter | eBay
9300-16i: LSI 9300-16i 16-port SAS9300-16I 12GB/s Host Bus Adapter 03-25600-01B | eBay
9300-8i: Inspur LSI YZCA-00424-101 Raid Card 12Gbps HBA HDD Controller 9300-8i | eBay
82885T expander: Adaptec AEC-82885T LENOVO 00LF095 36port 12Gbps SAS SATA Express shipping | eBay

Go ahead and give these a look. If some of them have anywhere close to earthen rates out to you then it should be significantly cheaper than what was listed on the site you showed me.

If we can’t find you a solution, I might have to start a service being a middleman for these things to get you guys at least relatively normally priced HBAs… ugh. Id be willing to send you some regular cost ones to europe if you guys have some way of getting them to you afterwards @gorden (and @Marandil if you want something as well). Not sure how that would work out but if it somehow ends up being cheaper to have me get them and then get a service to forward them to you, I could update them and verify they work before sending them your way. I’d like to avoid that if at all possible though, I have classes to do and am already wrapped up in a lot of hardware stuff lol.

Broadcom likes to do this too, as you will learn if you scroll up a bit lol. Hardware raid is fickle, embracing zfs is the boring inevitable solution that simply works.

Glad it’s running well. Anecdotally, I have heard absolutely terrible, awful things about sata expander cards over the years. Please dont trust these. Glad you are getting into HBAs, if you are using sas/sata drives and they show up and work, they are reliable for forever. Keep them cool and they stay happy.

I appreciate it. It takes a long while to write these out haha. Good luck with finding your hardware. Let me know if you end up buying one and you need some resources about updating or crossflashing them from IR (raid) mode to IT (direct attach) mode.

1 Like
641

artofserver on ebay (and youtube under the same name). American, super nice guy, sells all kinds of used enterprise gear, provides support, can’t recommend him enough. I have no idea if he ships abroad, but it’s worth checking.

1 Like
642

I don’t think this warrants a new thread since it’s just a quick question and fits reasonably well here:

I got a few D5-P5336 drives to play with (the big ones @ 61.44TB). Manufacturer claims 24W active power, 1M IOPS, 7000MB/s seq read.

I formatted the default namespace to 4k and ran a very quick very unscientific benchmark on read speeds. dd-ing the drive to /dev/null with 1MB block size gave me 5.5GB/s (fair enough), and I get this from fio:

marton@sapphirerapids:~$ sudo fio --name=test --filename=$SOLI1 --ioengine=libaio --direct=1 --rw=randread --bs=4k --iodepth=32 --numjobs=4 --time_based --runtime=10 --group_reporting
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
...
fio-3.36
Starting 4 processes
fio: file /dev/nvme0n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
<...snip...>
  read: IOPS=1357k, BW=5302MiB/s (5560MB/s)(51.8GiB/10001msec)

So… 35% higher read IOPS than mfg claims? Those numbers are usually impossible to hit, how am I exceeding them by a huge margin?

Even more puzzling, I set the drive power state to 10W:

marton@sapphirerapids:~$ sudo sst show -ssd $SOLI1 -DSSDPowerState select=current
<...snip...>
DSSDPowerState : The DSSD Power State is currently 25 watts.
marton@sapphirerapids:~$ sudo sst set -ssd $SOLI1 -DSSDPowerState DSSDPowerState=10
Set DSSDPowerState successful. Completed successfully.
marton@sapphirerapids:~$ sudo sst show -ssd $SOLI1 -DSSDPowerState select=current
<...snip...>
DSSDPowerState : The DSSD Power State is currently 10 watts.

Verified to persist across power cycles too, very good. I wanted to see how it affects read performance, and … it doesn’t? I am getting the same numbers from fio and dd as I was before with negligible variances.

Pleasant surprise, but what gives? Why am I getting such high IO numbers, and why is there no power cap penalty?

643

Agreed. Fantastic guy, would absolutely recommend. His shop is pretty expensive if you know or are willing to learn how to flash things though. It’s simple as dirt nowadays too.

644

Thanks for the recomendations.
I’ll give the 9500-16i a try and checked the shops for this card, but at the end the price ist more or les the same as here after tax.

I’m not sure if you have other prices in the US but for shipping to europe costs are 30-70$ and we must pay tax and customs fees what is for me around 10%.

  • 9305-16i 272$ + 50$ Shipping
  • 9500-16i 400$ + 56$ Shipping

Yeah flashing is not an issue.

645

Just order from Amazon DE and let it ship to Switzerland?

At least then you would get full warranty and 100 % no issues with Broadcom’s vanilla firmware directly released by them.

HBA9500_Schweiz.PNG
HBA9500_Schweiz.PNG1920×1320 197 KB

646

Yes, I ordered this yesterday via Amazon to Switzerland for 333€ including tax and customs.
In Switzerland, it would cost more than 500€.

1 Like