SAS HBA and cables experiments and questions

Basically, I did some experiments with SAS cabling, and I would like to double-check, that I am not doing something horribly wrong and damaging my hardware…

I have 2 unRAID servers. One is my main media server, and the 2nd is for testing, pre-clearing drives, etc.

I have 4x of these SGI Rackable SE3016 disk shelves, that I bought of ebay in 2013, IIRC.

I have LSI SAS 9201-16E 6GB/S Per Port HBA in each server.
And I have 2 of these shelves connected to my main server using external (SFF-8088 to SFF-8088) SAS cables to that LSI HBA.

I was not using unRAIDs parity, and I have only 1Gbps network. So, basically, I was perfectly fine with the speeds I was getting, over my LAN, because single drive was easily faster than 1Gbps network. And that was enough for me.

Then I decided to look into using unRAID parity, which works by reading all the drives at the same time… And here is where the fun begins :grinning_face_with_smiling_eyes:

I started experimenting on my secondary/testing unRAID server.

I used 12 drives for testing:

  • 8 WD Red 3TB drives
  • 4 Samsung 500GB drives

And then I ran controller speed tests, while switching things around…
Controller speed test basically tests the read performance of every drive by itself, and then reads all the drives at once, hence showing the total system read performance.

:house_with_garden: All 12 drives in one shelf, connected via 1 external ( SFF-8088 to SFF-8088) cable

This is the “base” performance. This is how I have my servers setup ATM.
Result: 625 MB/Sec - which is kinda abysmal :disappointed:

:test_tube: I tried using 2x SAS (SFF-8088 to SFF-8088) cables per shelf.

Basically, there was no difference.
:question: Is this because SAS IN can only be used for daisy-chaining?

:test_tube: 2 shelves - 8x WD Reds in the 1st and 4x Samsungs in the 2nd

I’ve split the drives in two groups and used 2 shelves connected via 1x SAS (SFF-8088 to SFF-8088) cable per shelf.
Result: 762 MB/Sec - which is better, but not by much :disappointed: I was hoping for more… well, double to be honest.

:test_tube: Very interesting case → I switched the HDD groups around

Basically I took all the WD Reds and put into the case where Samsungs were, and vice-versa. I do not actually remember why I did it, which is kinda funny, considering what happened.
Result: 456 MB/Sec - I mean… wat​:question::flushed:

I ran multiple tests, with pretty much the same result.
:question: How can there be such a difference? Same drives and same shelves and the speed is even worse than a single shelf?
:question: I am not an expert by any means, so I am completely baffled by these results. Can someone explain how can this be?

:test_tube: Bypass the expander by using SFF-8087 to SFF-8088 cables

These some ancient shelves, so I thought, I’ll test if bypassing the expander would help.
Here’s a picture of the insides of the shelf and the expander.

Here’s the internal SAS cables inside the shelf, before I got to them :grinning_face_with_smiling_eyes:

I used two SFF-8087 to SFF-8088 cables to bypass the expander all-together.

I ended up with:

  • 8x WD Reds in shelf 1 - bypassing the expander
  • 4x Samsungs in shelf 2 - regular external-to-external cable
    I had only 2 of those internal-to-external SAS cables… for now :grinning_face_with_smiling_eyes:

And… Result: 1.61 GB/Sec - :rocket: woohoo :star_struck:

:question: The most important question - is this a No-No?

:question: Basically - am doing something very wrong by bypassing the extender, and connecting the backplanes straight to the HBA using these internal-to-external SAS cables?
The drives seem to be working perfectly OK. The only downside that I can see, is that the blue activity lights don’t work now. The yellow power lights do work, though. But all I did was run a test and powered off… I wanted to ask and make sure that I am not killing my drives or some-such by doing this… Hence this post.
My knowledge and experience about this stuff consist only of google-foo. I need some smarter people’s advice… :grinning_face_with_smiling_eyes:

:point_right: The whole reason I did this - my main server

Here is the pathetic results of my main server with 32x various size WD Red drives in 2 shelves. Both connected via external-to-external SAS cables.

I have more LSI HBAs laying around.
I would very much like to add a 2nd HBA to my main server, and connect both shelves by bypassing the extenders. I would use 8x internal-to-external cables going straight to 2x HBAs.

:question: Can I do that? :star_struck:

:question: If I can… I think I would need to rig some kind of cooling for the HBAs. Would something, like a 120mm Noctua fan above the HBAs and blowing down onto them be enough? Any ideas/suggestions are very welcome.

Sorry for such a long post. Anyone still reading… thank You :grinning_face_with_smiling_eyes:

Thanks in advance for any help/comments.

This topic was automatically closed 273 days after the last reply. New replies are no longer allowed.