A san server that serves data via ethernet consists of several parts:
a head unit, with the cpu, ram, and sas cards, and network cards.
a sas expander, which is often internal on the head unit and disk shelves
one or more disk shelves
Notice how I am referring to sas hardware? you can’t just plug in a sata hard drive. You either have to use sas hard drive or use a sata hard drive with an interposer. The interposer is usually mounted to the hard drive sled, and is specific to the hard drive sled system you choose, though you can use any sas drive.
The disks can be in the head unit, or in a separate unit, often called a disk shelf or disk drawer. A disk shelf has disks accessed from the front of the unit, a disk drawer pulls out of a rack and has disks accessed from the top of the unit. A disk shelf may hold 12+ disks, a disk drawer may hold 60+ disks.
You can combine both spinning disks, and flash media, or use flash media to accelerate spinning media.
Sas cards have either 4, 8, 12, 16, or 24 channels.
Since it would be inconvenient to have a direct connection between the cards and every drive, there are sas expanders($90-$350). Sas expanders usually look like a pcie card, but they don’t send any data via the pcie data bus. They can either be powered by being inserted into a motherboard’s pcie slot, or via a molex power connector. A common number of ports for a sas expander to have is 36 ports. If you had that on a computer with a 8 internal sas ports, all connected to the expander, you could have 8 ports external, and still connect 20 sas hard drives internally to the computer.
to design better software, it is good to understand the hardware.
If you have 8 sas channels, you will get better performance out of an array with 8 or fewer sas devices vs an array with more than 8 sas devices, because during a read or write event, the sas card will need to hang up a connection to one set of devices and talk to the second set of devices before hanging up to talk to the first set of devices again.
If you had a card with 8i8e, you could have an internal array with 8 sas devices in one pool, and a pool with 3 vdevs each with 8 sas devices on a disk shelf, and transfer data between the pools at full speed. However duplications within the external pool would not be full speed, you would need to do a read, then it would transparently switch devices, do the write then transparently switch devices to do the next read.
Sub optimal is not impossible though. You can get a card in 8i8e and begin using it. If you realize that your raid is going slow, you can then add more resources, like a 16e card.
As flash started being used, nvme had not been invented yet, and flash was so much faster than spinning media, they started needing faster and faster busses to transfer data off of this new medium. At this point in time, it is not worth your time or money to pursue access to flash media over any bus that is not pcie. Flash is compact and fast, just put it internally. If you run out of space, put your spinning media externally, and now you have more internal room for flash on pcie. The fastest spinning hard drives available on the market are the new dual head drives which can transfer data at 280 megabytes per second, ie are just over 3 gigabit. 6 gigabit sas is more than fast enough to transfer data from drive to card. A bunch of the 6gigabit sas cards are only pcie2. So it is worth it to get 12gb sas if that will let you use an x8 pcie3 or x8 pcie4 slot instead of a x16 pcie2 slot, since your motherboard won’t will only have a x8 pcie3 or pcie4 slots, it is a waste of the slot and a slight increase in the cost of the card.