I recently built am ESXi server to serve as a home NAS and server for some other VMs; the rig is detailed here.
Anyway I have transferred a few TBs of data to the BTRFS RAID 6 I've setup on the NAS VM(Centos 7 w/ Linux Kernel 4.6) and decided to run a scrub yesterday afternoon to get a baseline. When the scrub finished I had 24 uncorrectable errors on volume /dev/sdc and 13378 unverified errors on the same volume. The other four drives scrubbed just fine.
Seems to me like I have a bad drive, which is under warranty so that's not too big of an issue, but I wanted to make sure that's what's going on before I swap it. Also SMART isn't showing any issues with the drive. Here's some screen grabs of the output of various commands. Also I was getting a lot of messages in the command line similar to the ones at the top of the first pic during the scrub.
I have read over at Phoronix recently that BTRFS has quite a few issues when it comes to using RAID function, lots of DANGER things you can get caught with and loose data. If you head over there and read the BTRFS (windows kernel driver) article some people talk a bit about the issue(s).
I'll get to answering some of the other questions later today, but to answer yours.
The drives are connected to a LSI 9211-8i RAID card in HBA mode. This card is passed through as a PCIe device to the VM so ESXi doesn't even "see" the drives. ESXi has a couple of drives it uses connected to the motherboard's SATA ports.
Thanks for that website; I wish I had found it in my Googling of BTRFS back when I was planning this build. I had a very hard time finding up-to-date information on BTRFS; what I did find seemed to indicate that it was fairly stable now but that doesn't seem to be the case after all. It's looking like I'll want to rebuild the NAS portion of the system on something else. I didn't want to go with FreeNAS/ZFS in the beginning because I wanted the ability to expand the array as my storage needs grow.
Any suggestions? I'm still in a position where I can scrap the array and start over if necessary.
It is in a proper server with redundant PSUs and a UPS that runs 24/7. So far performance has been fine; I can saturate the gigabit link between my desktop and the NAS in both directions easily.
I would just move to FreeNAS. It's just stable and it just works. ZFS is a great file system, BTRFS is, at best, second place to ZFS as far as data integrity goes.
Yeah, that's probably going to be my best bet. After looking over the ZFS documentation it doesn't look like it will be as big of a problem as I initially thought to upgrade drives in the future. I'll probably get two more drives so I can run a 7 drive RAIDZ-2 leaving one bay empty so I can safely expand the pool in the future. Probably by then I'll be moving to 4+TB drives.
Well I made the switch to FreeNAS; now for a night of copying files back to the server again... I am noticing that my transfer speeds are around 25% slower on the same hardware compared to the BTRFS RAID 6. I'm using a RAIDZ-2 in FreeNAS with 5 drives at the moment. Currently it's not even saturating my gigabit LAN whereas that was the limiting factor on BTRFS. Know of any fine tuning I can do to get the speed up? I'll be adding 2 more drives later this week when they arrive.
If your house is, for example, entirely on Windows 8, you can specify a minimum protocol of SMB3. In theory your machines should talk to one another and decide on the best protocol, but I don't trust that too much. I snatched up about 10-15MB/s write speed to my FreeNAS box messing with this setting.
Just make sure to take it easy. One change at a time in case you need to roll something back.
Also for memory. Don't worry about it filling up the memory. That's just what *nix systems do, because free memory is wasted memory. As long as you're following the general rule of 1GB for every TB of stuff you have, with a minimum of 8GB.
I enabled Autotune and will switch enable SMB3 as the minimum when my transfers finish. I did a bit of googling and it was recommended that I check htop to see my CPU usage... It would appear that I may be CPU bound at this point since samba on BSD is apparently single threaded.
I had looked at the usage in the web interface but it didn't really show the pegged cores like htop does. I may end up needing to upgrade CPUs in my server if I want more performance. I should be able to upgrade to some X56xx chips fairly cheaply on eBay though. I will give the SMB3 setting a try first though. :-)
I'd want more verification on that before I went out and bought new hardware. There's a lot of tuning to be done in Samba in FreeNAS. The defaults don't cut it in any scenario I've seen. I've seen gigabit flooded on my build using a Pentium 4. CPUs didn't get much shittier than that until AMD released the E-350. :)
Good to know; I'll do some more research on it. Know any good resources? I should be putting the final version of the array together this weekend; I ordered 2 more drives Wednesday that are coming in tomorrow and will be rebuilding the array as an 8 drive RAIDZ-2. That will max out my enclosure and HBA with 2 TB drives and give me an upgrade path in the future when I want to make the jump to larger drives.
This was a pretty good resource for me. I ended up forgetting all I knew about tuning Samba, and ended up being happy with ~40-60MB/s transfer speeds. You can also hit up the FreeNAS forums. It's a bit of a scary place. The trolls who lurk there know FreeNAS very well, but you have to show that you're trying to solve the problem on your own, otherwise you will be presented with flat, unhelpful answers.
I'm coming back to this. 8 disks in a RAIDZ2, I feel like unless you need the space, a better solution might be two RAIDZ2s striped.
In a straight RAIDZ2, you'd get about 12TB I believe. You'd only be able to lose 2 disks. And your write performance wouldn't be boosted at all.
In a 2x RAIDZ2 striped, you'd get about 8TB. You'd be able to lose 2 drives in each vdev and still have the pool be functional. And your write performance would be increased.
Losing disks is pretty rare, so you're probably good with a 2 disk tolerance. And writes may be limited by gigabit anyways. But it's always nice to know that you have options, right?
Yeah, I don't think I want to give up that much disk space; I feel that 2 drive tolerance is enough given that I'll have other backups. I won't be needing 12TB today or even this year, but the amount of stuff I'm storing isn't going down... Also if all I can get speed wise for writes is ~70 megabytes a second I can live with that; I'm just hoping I can get it closer to 100 and if I need faster CPUs that's not too big of a problem.
It definitely is some food for thought though. Thanks for all the info!
Well after rebuilding the array with 8 drives in a RAIDZ-2 I can now hit 100+ MB/s read and write; not sure what made the difference but at least I don't need the CPU upgrade now.
On second thought maybe I do want a CPU upgrade....
My desktop's i5 is running circles around it in Handbrake transcodes. I guess it shouldn't surprise me that a newer quad core running twice as fast is faster than two old quad cores. I might want some hex core chips though instead of the better quad cores.
