LSI RAID Testing and Cachcade

High-Performance Computing Workstations & Servers
1

Hi all, I’m having some awkward numbers coming from Crystal Mark, and ATTO with my RAID 6 and RAID 0. Currently I am using a LSI 9266 8i with an intel 24 port expander. I currently have 3 different RAID groups. I have a RAID 0 with 2 SSD (testing prior to trying Cachecade), I have a second RAID 0 with 4 2TB WD Black, and I have a RAID 6 with 5 6TB RAID 6.

Prior to setting up the SSD RAID started getting unusual crystal mark numbers with my RAID 0 group, and ATTO reflects more normal/actual numbers in my tests.

So here are the results:
RAID 0 with 2TB drives = Drive W
RAID 6 with 6TB drives = Drive X

First, seeing better write speeds on RAID 0 really throws me off, and seeing that low of a read speed when a few months ago it was showing in the 500MB/s range is throwing me off as well. In addition, seeing better numbers from my RAID 6 is really messing with me, since RAID 0 should be performing better than RAID 6.

Does anyone have any thoughts on what could be the cause of this?

When I do a real world transfer of 3 files at 15GB, I get 650MB/s from RAID 6 with dips down to 400 when going to my RAID 0 with HDD, I get consistent 650MB/s from RAID6 going to the SSD RAID 0.

With the same files I get 580-600MB/s from my RAID0 with HDD to RAID0 SSD, and I get the same speed going to my RAID6.

Obviously my SSD RAID 0 is hitting the hard drive RAID write bottlenecks, but a max transaction on a RAMdrive is 1200MB/s so I’m reaching the full potential of the RAID card’s 12gbs throughput.

So here are my questions:

Why is my RAID6 out performing my RAID0?

Has anyone set up Cachcade before, and what are the best settings associated with it, particularly for video editing over 10GbE?

Stripe size, what is the best size for video? Considering Cachecade in addition to this, what are those best settings?

Is CrystalMark a good testing software, as it’s showing numbers that aren’t accurate? Is ATTO a better tool to use?

Is running multiple RAIDs on a single card going to cause trouble? I another card to separate my RAID groups? The final product will most likely consist of 2 RAIDs, RAID 6 and RAID 0. (SSD will turn into Cachecade if it will help video editing).

Note: I’ve also tested latency with the free trial of HD Tune Pro, and all drives fall in the 20ms or less range with the occasional dot that is outside that range when running test (mind you the SSD RAID only had like 1 dot fall outside the 2ms range).

Thanks in advance!

I forgot to mention my system specs:
Rosewill 4U 12 bay case (SATA connections)
LSI 9266 8i
Intel 24 port RAID Expander Card RES2SV240
Asus Px79 Pro Motherboard
intel i7 3930K
GTX 760
intel x540 t1 network 10GbE card
4x 2TB WD Black (RAID 0)
5x 6TB WD Black (RAID 6)
2x 512GB Samsung 850 Pro SSD (RAID 0)
1x 950GB Crucial

2

Your best bet is to experiment :) And see what shakes out. Cachecade is awesome, cachecade 2.0 that is, cachecade 1.. is.. less awesome.

3

Haha, thanks wendell. I was hoping to hear that Cachecade is an awesome idea. I'm hoping it all pans out with video editing. If anyone else has any experience with cachecade, or setting up a RAID for video editing, please leave a comment!

4

@wendell I watched a video recently where you utilized cachecade. Were those test results fresh after a RAID group build, or had you been using the RAID array for a bit so that the SSD Cache would be populated?

The reason I ask, is because I'm about to go through setting it up, and I wanted to prepare for results that weren't expected...

5

No idea. Probably that was cachecade 1.0 which is disappointing. Cachecade 2.0 is much better.

6

So, I have Cachecade 2.0, and I've noticed that it does perform better after editing for a little while, meaning the algorithm is working.

Have you heard when they will allow you to utilize more than 512GB ssd? I have 2 512GB Samsung 850 Pro, which I hoped to use in RAID0, but unfortunately, they only allow 512 still. Any word on this?

1 Like