Dell Rack 7910 SAS vs SATA

High-Performance Computing Workstations & Servers
1

Hi all, I’ve been aware of something of a problem on my workstation for a while, and all the googling I do is just getting me turned around a bit.

I use a Dell precision rack 7910 as my desktop. I do a nontrivial amount of virtualization in addition to my homelab, which was recently upgraded to a Dell FX2s chassis and FC630 nodes. So this question applies to my other systems too but I use my personal system as a kind of test bed for “experimental” type solutions that I’m not confident will be fully valid or helpful when it comes to my homelab. I want my homelab to be more stable since it’s running things like home assistant and other home service software. The more downtime it has the more aggro I get from the spouse and family.

Long story short (TL;DR): I’m wondering if it’s worth the investment to get interposers, and whether it’s worth it to implement that across the board on my system(s). What interposers to buy, etc.

I use pretty much only SATA drives due to cost. I’m not rich enough to afford fancy SAS HDDs, and certainly not SAS SSDs. I have four storage arrays that are worth noting. One is in my precision rack 7910, an eight drive raid 6, on the higher end (but stock) perc controller… I think it’s the H710p ? I’ll have to verify that, but something similar. I like it because it integrates with the iDRAC on the system and can be monitored remotely. The remote monitoring is not something I want to give up.

The drives are fairly economical, they’re Samsung Evo 870 500G. I kind of feel like I screwed up on this choice because the Evo line is fairly performance limited in the first place, but the 870s were new when I got them, in the sense that they were purchased as new, and that they were the current tech at the time I bought them… At least for SATA SSDs from Samsung. I just slotted them into my 7910, configured the array as raid 6 for redundancy (which has already saved me a few times from having to reformat), and installed Windows. Drivers are up to date and the firmware was updated by the lifecycle controller when I set all this up. I updated the individual drive firmwares shortly after I installed them due to a firmware bug that would cause the drives to prematurely fail. I had to RMA a few drives because of that bug.

I can look and verify that only one link is in use and everything works but performance is still pretty underwhelming. I know raid doesn’t scale super well, and the drives aren’t exactly fast by today’s standards, so I’ve lived with it.

I know interposers are usually a good idea in these scenarios since it will allow for the dual channel SAS controller to have redundancy on the link to the drive, I didn’t consider this a huge priority since all the SAS links are internal on the system and the drives will bottleneck to SATA 3 regardless. I’m reconsidering that position.

I’m happy to spend a little bit, but not crazy amounts. Cost is always a factor. I can’t exactly afford to rip and replace all of this with a U.2 setup with a new controller, backplane, cables and drives, that would cause more wife aggro, no matter how happy it might make me. I also can’t afford the U.2 drives. But $50 or less for a set of interposers, even if it’s $10/each isn’t really a problem. I’d spend that just to find out if it will help with speed at all… So I’d like to get some as an experiment but then I hit another problem…

My main issue with even attempting this is that, I don’t know which interposers to buy. All of my drive trays on my 7910 have depth to spare to fit one in, but I can’t seem to find an interposer that will fit in the SFF/2.5" trays for Dell, without pulling my hair out. I keep searching for a 2.5" compatible interposer, but I can’t seem to find one.

I can find the 3.5" ones without issue, plenty of those around, especially on secondary markets like eBay, but the 2.5" interposers seem to be rainbow unicorns for the Dell SFF hot swap drive trays.

Does anyone know what Dell part number I’m looking for?

Also, I’d welcome any thoughts on whether this will help. My main concern is whether I’m overloading any SAS channels to the drives while in operation, where the additional links might be useful. I’m not certain that’s happening, but Dell doesn’t really give you the level of reporting you would need to figure it out. So it’s all a bit of guesswork.

As an aside, my other arrays are in similar situations, I have a power vault nx3200 (pretty old, I know), with 12 SATA HDDs in it. All the HDDs are WD Red “plus”, either 4TB or 8TB, where the 4TB 6 drive raid 6 is for virtual machine “OS disk” storage (the C:\ or root for VMs) and the 8TB6 drive raid 6 is for a data vault (large file storage). On another system (an R630), I have 8SATA SSDs for faster VMs, also raw SATA to a similar perc controller, and finally, the one I’m least worried about because speed doesn’t really matter, is the bare metal OS raid 1 array on my FC630 systems also SATA SSDs (this one only holds the boot OS for my host, all VM disk data is on the 4TB array over iscsi… Once the os loads, the disks are basically idle). I’m wondering if I can copy/paste the same ideas, and get interposers for those systems too to hopefully enhance speed a little bit, provided they help with my 7910.

Sorry for the long post. All opinions are welcome as long as they’re constructive. Any opinions posted which simply criticize my setup without any helpful input will be ignored. :slight_smile: have a good day.

1 Like
2

You’re saying performance isn’t good, but your post doesn’t seem to have any specific numbers…

Certainly RAID-6 is going to perform very poorly on write speeds (and latency).

How about just converting to RAID-10? You won’t lose a lot of capacity, and write performance will dramatically improve.

Do you have write-back caching enabled on the R710p, and a good, working battery?

Also, I think you’ll find older SAS drives on the used (or NOS) market to be pretty cheap, often cheaper than SATA as there’s less demand. I haven’t seen SAS drives perform any better than SATA, but I never was doing RAID-6.

2 Likes
3

I’m not opposed to refurb SAS SSDs and that’s viable on the whole. I’m not super concerned about write, since a lot of my disk use is read; so raid 10 likely won’t help much in that respect and carries with it a degredation in redundancy.

Raid 10 is “1.5” drive redundancy, where raid 6 is 2 drive redundancy. Since I’m using hardware raid, it should have dedicated coprocessors for the parity calculations, and speeds shouldn’t be significantly impacted by parity. I haven’t done all my homework on that, so I may be mistaken.

Either way, raid 10 is a valid suggestion. I thank you for that.

My only reservation with used/refurb/whatever enterprise SAS SSDs is the age and wear level of the drive. I prefer to use new disks to ensure the longest run time before failure. My only other concern with that suggestion is it will be rather costly vs what I’m thinking to do. “Upgrading” existing drives with interposers is cheap, most interposers are less than $5 each, and you can often find bundles of 6-10 for $20… They’re basically throwing them at you. Contrast that with a set of 8 SAS SSDs from a secondary market, whether new old stock or refinished or used, and each drive is going to be $20+ at the low end. It’s a significantly more expensive solution. It will certainly address the concern and it’s something that’s crossed my mind before.

Right now, before ditching everything and going to SAS SSDs in whatever form I can, I’m trying to experiment with the options which are less expensive. If they yield no net benefit, then the only way forward will be to buy SAS SSDs.

I don’t have figures in front of me from the last time I did performance testing on my raid array (I’m presently at work), but IIRC, the 8*500g evo870 raid 6 array peaked out slightly above 2000mbps read on crystal disk mark. That was the fastest speed on the benchmark, I believe it was seq, the more challenging disk operations were at or below 500mbps (again, read)… With writes being mostly in the toilet by comparison, maybe half as fast as their “read” counterparts.

The last time I benched the disks, I toggled all of the controllers caching and write-through/write-back options to brute force which options made it faster/slower, and my current configuration is a result of that testing. I can dig up the figures from that testing and my current settings once I’m finished my work day.

I’ll update it here.

1 Like
4

Just verifying the configuration:
Controller: PERC H730p 2G (mini)
vdisk settings:
Read Ahead Enabled
Write Through
Disk Cache Enabled

CrystalDiskMark, default profile, 5 count, 2GiB size
Original Test results, as recorded:
Sequential:
Read Q8T1: 5472.07
Write Q8T1: 2465.25
Read Q1T1: 3195.28
Write Q1T1: 499.29
Random:
Read Q32T1: 230.97
Write Q32T1: 91.16
Read Q1T1: 71.54
Write Q1T1: 17.22

I just retested (the system is active, ~5% CPU, and 28% RAM used):
same settings.

Sequential:
Read Q8T1: 5083.41
Write Q8T1: 306.60
Read Q1T1: 3041.84
Write Q1T1: 261.11
Random:
Read Q32T1: 224.39
Write Q32T1: 89.12
Read Q1T1: 64.08
Write Q1T1: 16.04

Some of the numbers are pretty decent, like high queue sequential, which is great, but pretty much anything non sequential or write is kinda trash.

Specs for the drives:
Samsung EVO 870 500G:
NAND Type: V-NAND 3bit MLC
DRAM Cache: 512MB LPDDR4
Seq. Read: 560MB/s
Seq. Write: 530MB/s
4K Random Read (Q1): 13k IOPS
4K Random Write (Q1): 36k IOPS
4K Random Read (Q32): 98k IOPS
4K Random Write (Q32): 88k IOPS

… and that’s PER drive, for 8 drives. a bit of napkin math says sequential read, if everything scaled perfectly, would be in the neighborhood of 4480 (8*560), and writes should be 4240 (8*530).
for IOPS, which is probably what I’m most concerned about, running CDM on the “Real world performance” profile, 5pass, 2GiB, I get RND4K IOPS of 16383.06 read and 3703.61 write with a latency of 60.7 for read and 269.26 for write at Q1, and on the “peak performance” profile (Q32), IOPS are 59282.96 read (Latency 522.42), and 20700.44 write (latency 1528.74). The Q32 performance is significantly reduced from even a single drive according to Samsung’s spec sheet. I didn’t record the IOPS from my earlier round of testing, but I believe they were similar.

If the numbers exceeded a single drive’s performance on all metrics on the spec sheet, I wouldn’t really care. I know RAID will reduce throughput and IOPS performance across the board. I understand that my numbers are quite good for a SATA array, as far as I can tell.

From my earlier testing I might be able to bump write performance by pushing writes to using writeback (forced), which may double or triple the throughput, I’m not sure that will have a net positive on speeds.

Either way, the thing I’m looking at mainly is that I have one connector utilized on my array, from what I’ve gathered, my RAID controller has two connectors, each 4x lanes at 6Gbps, for a total of 8 lanes, however, from the BMC, it’s showing that only one connector is in use (Connector 0) with all eight drives on there, that’s 4*6Gbps lanes, for 8*6Gbps (link speed) drives, which isn’t great. My best understanding of how it works is that Connector 0 is connected to the first link of all of the drives (SAS/SATA operation), and Connector 1 is connected to the second link (SAS only), so there’s nothing on Connector 1. I’m losing 4*6Gbps of bandwidth because the connector is connecting to nothing. With an interposer, and using MPIO, I can effectively reduce or eliminate the bottleneck on the lines, since each drive would be connected to both connectors, as long as I configure the H730p to use MPIO to the interposers, it should be able to use either connector to reach each drive, and it should optimize performance, since then I would have an amalgam of 2*4*6Gbps links, totalling 8*6Gbps links, which is linked to 8 drives, each running SATA III at 6Gbps.

Currently I’m basically cutting my possible bandwidth in half, and from what I can tell, sequential isn’t really bothered by this, probably due to some fancy caching on the controller (I selected 2GiB for the test with 5 runs to try to overrun the controllers cache), but IOPS seems to suffer significantly since the drives are basically running half-rate. This is one of the problems with RAID 6, it hits all the drives at the same time, all the time. So the rate-limits are fine at lower speed access, but fall over on highspeed data access (many IOPS).

The primary problem I’m having with trying to solve this is finding a 2.5" compatible interposer. It seems to be an impossible task to find an interposer that is compatible with the physical form factor for the 2.5" disk trays from a Dell system. Almost all the interposers I can find are 3.5" sized, and never anything smaller. I’m sure the technology has progressed to the point where it’s possible, but the industry has pivoted away from SATA and SAS entirely, so finding such a product is quite difficult.

1 Like
5

I think you’re having an incorrect understanding about interposers.

From:

“They translate SAS (SCSI) commands to SATA commands, and even implement a few themselves (like power- and spinup-control).
They do nothing else (the drives are still SATA, their performance characteristics are unchanged, their reliability is unchanged, etc.)”

Your tests prove that you don’t have a bandwidth problem. Using the second port on your RAID card will not improve the performance.

Yes, this is a drive problem - not a connector problem, not a lack-of-interposer problem.
It could be a RAID card problem. It’s possible your RAID card cannot process the IO fast enough. I’m not familiar with the Dell card terminology, but (as general advice) expect any 6gb SAS RAID card to be the bottleneck when connecting SSDs.

1 Like
6

I’m forced to agree here. I may still pick some up if I can find some that fit and work, but at present I’m not expecting a different result.

I think I’ll keep an eye on secondary markets for 800+ GB SAS SSDs, and maybe get some 400G (or so) for my hypervisors, and try to get away from consumer and SATA drives on my raid controllers.

My next array, which I’m already planning (waiting on a few things for this, including funding), will be entirely NVMe. That will come with its own set of challenges; I’m mainly wanting high IOPS performance above pretty much everything on it, so I can’t really get much faster, as far as I can tell, without going to NVMe. Even an array of “slow” NVMe drives will completely crush my best IOPS numbers on any of my SAS raid arrays.

If anyone knows of an interposer that is small enough to be used in a 2.5" tray, link me and I will appreciate you forever.

Everyone else, if you wish to comment further, you are welcome to, otherwise, I hope everyone has a great day.

1 Like
7

QDepth on Dell firmware H710 cards is something like 1/20th of LSI firmware H710 cards.
Just had a similar issue with an H710 card running stock firmware and 870 Pro’s from an existing installation an engineer was trying to troubleshoot.
Swapping cards solved that to the greatest extent possible. Ultimately it was decided to go JBOD with ReFS for that data pool. The client now gets nearline speeds.
For kicks, I flashed that same card to IT mode and got nearline speed though I would not deploy it in production for fear of a rogue Windows update reverting the card firmware.

1 Like
8

That’s new information to me. Thanks for that.

I have two main questions:
first, are there any complications with getting the LSI firmware on to the Dell cards? I don’t really want to replace the physical cards, as many of them are mini/micro variants, and replacing the card for the different firmware isn’t really a desirable outcome for me, since it will just eat more PCIe slots which I’m already fairly limited on.

Second, does the LSI firmware integrate with the iDRAC/BMC the same way the Dell version does? I prefer to use iDRAC for alerting, so for me, that’s a key feature for disk monitoring (to alert me if a disk dies). I’m sure there’s additional ways to do that, but the system I have already setup is working, so I’d rather not have to adapt a new alerting system for dead drives.

if it integrates normally with iDRAC/BMC and it only requires a reflash of the LSI firmware, I’ll give it a go. As long as I don’t need to buy a new controller card, I’ll probably try it on my workstation, to try it out. My only other concern is migrating the RAID config over, and I assume that’s just adopting the existing RAID on disk to the new firmware on the controller.

Any additional information you can provide would be appreciated. Have an awesome day.

2 Likes
9

-any complications with flashing LSI firmware to Dell OEM cards
Typically only with some mid level micro and mini variants, but I have not memorized the compatibility charts.
So far, I have been able to flash every card that comes across my desk.
Entry level and high end cards I know can flash easily, but I am typically after JBOD functionality as we use software RAID exclusively.

-Does LSI firmware integrate with iDRAC
Not the same, no
HP iLOM does, but not Dell iDRAC since it’s lookin for the Dell special sauce.

The majority of our installations are standalone or small clusters; so typically iDRAC is disabled as most small enterprises do not have full licenses and are not running dedicated management networks. vLAN is not an acceptable solution for most vendors since they simply set a vLAN tag but still rebroadcasts the packets.

Those tags are not respected when snooping the network and only high end switches allow that level of traffic management.

A dedicated network can be added for a couple grand, but only our managed customers opt for it.

We prioritize: Security > Uptime when the choice has to be made

-Import RAID configuration across vendors
From my experience the LSI firmware cards are the best at this
Provided the card is newer than the array, it’s seamless
I have successfully imported 10+ year old Dell & HP arrays into cards that are less than 1 year old without a problem.
Just imported an array from 2009 onto a Broadcom 9600-16i card without issue

2 Likes