I’ve built a testbed for playing with storage (60GB SSD OS drive,3x1TB 7200 RPM hard drives) but I’m really not familiar with what options are out there. I have built a RAID 5 array with mdadm and lvm and did some informal benchmarking (write speeds seemed really slow - 20MB/sec sequential.) However, I simply used ext4 on top of these arrays.
I know there are other filesystems I can use (xfs, zfs, btrfs) but it is unclear to me whether they are simply layered on top of lvm or mdadm like ext4, or if they have their own built-in RAID functionality. Is there a document somewhere that explains the different software RAID and filesystem implementations?
Hi checkout zfs my favorite filesytem for storage it’s so awesome. zfs and btrfs is a newer generation of file systems with inbuilt volume manager so they do not need lvm. Here is a link to read more about zfs http://www.allanjude.com/bsd/zfs.html
@habbis Thanks for the link. However it seems geared to BSD. Casual googling indicates that Fedora may not be the best distro to use ZFS with since its not supported natively. Are there any Linux distros with out-of-the-box support for ZFS?
There are other reasons to stay away from it. Snapshots/subvols exponentially degrade performance, there are occasional breaking changes and in general, it’s unstable.
@nx2l Read speeds were OK - 220MB/sec or so, but only 15-20MB/sec according to the GNOME Disk Utility. There was one other test I ran that I can’t remember the name of, that showed similar numbers. Any thoughts on what the problem might be? Keep in mind I’m using 3 disks, not 4.
@nx2l All three drives are 1TB 7200 RPM drives, but only 2 are identical Western Digital models. The third odd man out is a Hitachi drive. It shows 2 reallocated sectors, but still passes the SMART self-test. Casual googling finds that the WDs average about 120MB/s whereas the Hitachi is only rated at about 80MB/s.
Looking at iostat shows nothing unusual to my eyes, but maybe I’m missing something.
EDIT: Looking at top, I see two processes mdX_raid5 and mdX_resync using about 20% and 10% of CPU respectively. Is this indicative of the system still trying to build the array?
I should also note that I’m using SATA II ports for the spinning drives on this older system.
Just trying to run an experiment without breaking the bank. I have a wide variety of drives 2TB and less, but very few are alike. Since I already have the two identical WD drives I may go ahead and get a 3rd identical model.
try cat /proc/mdstat, see if it shows anything insightful.
there are also --assume-clean and --wait-clean flags that might be interesting in your case.
You can also try storing the bitmap/journal or whatever you’re using on flash on your OS drive in order to speed up writes, for sequential writes in your case there’s no reason you shouldn’t be able to get between 150 and 200 MB/s.
also atop will show you per block device stats, and you can use iostat -x to glean some performance data as well.
So I tore down my mdadm RAID5 array and rebuilt it in lvm, then waited for the build to complete. Same results - about 200MB/s read and 16MB/s write. At this point I got a bit frustrated.
So I broke out an old Dell SAS 6/iR controller I had from an old Dell tower. In the Dell tower, it only functioned as an HBA, but to my surprise when I put it in my ancient Asus motherboard it had RAID 0 and 1 functionality. So I mirrored the 2 WD drives and set up the Hitachi as a hot spare, and proceeded to benchmark.
That nets me 280MB/s sequential read speed and 150MB/s write speed. Much better, though now my 3TB of storage is reduced to 1TB. And the SAS 6/iR can only handle drives 2TB or less. But at least I didn’t have to wait hours for the array to build.