Hey Wendell,
I wanted to drop you a line in particular because I know you use the C2750D4I personally at home and I wanted to see if you've come across this same problem before.
So I've just bought a C2740D4I, 4 x WD Red 2TB, 2 x Kingston 8GB ECC RAM, and a Corsair CS450M to power it all, all housed in a BitFenix Prodigy.
Now everything so far is working fine except for when I actually went to go write data to the drives and things were...underwhelming. At best it was 47MB/s.
Just to clarify, all the drives are plugged into the SATA III ports and have the appropriate cables and none of my disk disagnostics have led me to believe that a DOA disk might be what's causing an issue. I'm also testing locally from the FreeNAS box, not over the network.
My Google-Fu sorta leads me in the same direction - the FreeNAS forums where the crubling tolerance of the mods there end up being more abrasive than helpful.
Just wanted to know if you had the same issues and if you did, what you did to resolve it.
Thank you in advance.
You can use the IPMI (or just SSH in) to see if there are any messages on the console. FreeNAS has a great built-in diagnostic system, and can do long and short diagnostics, etc. You should set those up to perform weekly, probably.
For a drive failure, you can run diagnostics that use little more than dd to see how fast individual drives are:
http://superuser.com/questions/573768/zfs-raidz2-performance-issues-read-write-speeds-on-freenas
(note that you would have to setup a storage pool of one drive, and specify the right path).
For ZFS, 4 drives may not be appropriate. 3 drives in Raidz1 would give you 2* usable space, with the third for redundancy. You could, of course, use the 4th for additional redundancy (but this limits you to 8tb of storage) -- it might be better to buy a 5th drive so you get 4* sorage +1 redundancy.
I would suggest, for diagnostic purposes, you skip ZFS for now to try UFS and see if that's dramatically way faster.
If you then do a 3-drive raidz1 and that's also dramatically faster, and you saw no evidence of a drive failure from SSH or IPMI or SMART diags, you can 'level up' your ZFS skills by pulling one of the three ZFS drives, swap in the drive you didn't use and tell it to resilver the drive. You'd probably want to copy a few hundred gig to the 3 drive Raid Z1 for good measure, if it was performing properly, to get a feel for what that process is like. Will come in handy when it happens for real.
So, you're going to facepalm here...
Turns out, in my infinite wisdom, that I was doing the wrong dd tests locally. Once I corrected them, things reflected a more accurate reading.
For comparision of revised dd test to old one:
[root@freenas] /mnt/vol1/dataset1# dd if=/dev/zero of=tmp.dat bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 91.901052 secs (1168367283 bytes/sec)
[root@freenas] /mnt/vol1/dataset1# ls -alh
total 55
drwxrwxrwx 4 root wheel 5B Jul 28 09:29 ./
drwxr-xr-x 4 root wheel 4B Jul 28 05:16 ../
drwxrwxrwx 3 nobody wheel 3B Jul 28 05:52 .recycle/
drwxrwxrwx 3 nobody wheel 3B Jul 28 06:51 Plex/
-rw-r--r-- 1 root wheel 100G Jul 28 09:30 tmp.dat
[root@freenas] /mnt/vol1/dataset1# dd if=/dev/zero of=testfile bs=1024 count=5000000
5000000+0 records in
5000000+0 records out
5120000000 bytes transferred in 109.988534 secs (46550307 bytes/sec)
[root@freenas] /mnt/vol1/dataset1# ls -alh
total 55
drwxrwxrwx 4 root wheel 6B Jul 28 09:32 ./
drwxr-xr-x 4 root wheel 4B Jul 28 05:16 ../
drwxrwxrwx 3 nobody wheel 3B Jul 28 05:52 .recycle/
drwxrwxrwx 3 nobody wheel 3B Jul 28 06:51 Plex/
-rw-r--r-- 1 root wheel 4.8G Jul 28 09:33 testfile
-rw-r--r-- 1 root wheel 100G Jul 28 09:30 tmp.dat
[root@freenas] /mnt/vol1/dataset1#
Feel free to point and laugh. God knows I did. At myself. Oh God...
Thank you for your effort though, was much appreciated. I'll keep the info you passed on in a safe place just in case I need it in the future.
I will ask you this though: CIFS is horrendously slow for me (50MB/s over ethernet) but I have a feeling that my horrid Netgear router is to blame here. I'm yet to do an iperf to confirm, just wanted to see if you did anything special to squeeze out that extra bit from CIFS at all.
Once again, many thanks.
try turning on autotune?
http://doc.freenas.org/index.php/Settings
Worked for me. Can move 110mb /sec