small question. I currently run a hardware raid6 with 6 discs and plan on migrating to ZFS. My Mainboard is a Gigabyte X570 Aorus Pro and has 6 SATA ports.
Currently I have a sequential throughput of about 450MB/s and I’d like to have speed like that on ZFS too. Is this possible with a simple onboard SATA controller or do I require a HBA for that?
I had bad experience and slow speeds with mdadm before which was the reason I choose a hardware raid but ZFS sound great with the compression and all. I would love to throw out the hardware raid card and use the pcie slot for a 10gb network card. When reading about ZFS I heart people pray that you need a HBA and others said as long as your Mainboard does have enough ports your good to go. Now I don’t really know what to think and who is right?!?
Onboard sata should be fine. Things like LSI HBA’s are nice for long term reliability, and the ability to handle a large amount of disks (especially when involving an sas expander).
However to get good performance from ZFS, you need to spend some time learning about it. It requires both intelligent vdev layout and pool configuration.
I concur with @Log. ZFS is agnostic of the port selection and you get no benefit at all from the “hardware RAID” as you are not using any of the RAID functionality. Each vdev runs at the speed of the slowest drive in that vdev. Therefore you will need to configure your vdevs and pools to optimise performance versus resilience. There are plenty of guides out there.
I was more worried about the SATA controller for example not being able to provide the throughput to write to e.g. 3 discs simultaneously. Not because it is not technically specified for 6GB/s each port but because the controller or the connection to the CPU bottlenecking the performance. I had SATA extension cards for PCI and PCIe in the past whos performance was just abysmal. Like writing 30MB/s to a single disc. As I mention I read that mdadm, on the onboard controller of another older Motherboard, should also be performant but my experience in the past was like 100MB/s read and 50MB/s write with the same discs that then did 450MB/s on the hardware raid. Linux offers solid software so I can’t really imagine that it was the software.
What? I have RAID6. I have double parity, hardware cache and increased speed, how do I not use the RAID functionality? I really do not understand this sentence.
From what I read about ZFS I did not get the impression that I need performance tuning except the basics like choosing the right sector size for the drives and stuff like that, but I might be wrong, I will have a look, thanks!
I understood it like that too. But I will have a look to make sure.
Striped mirrors is not an option, I am slowly becoming an data horder but I am on a budget. I buy high quality NAS drives with 5 year warranty but I need to gain as much space as possible. So RAID6 or now RAIDZ2 it is. I just can’t afford to waste space for entire mirrors.
If this is truly the case, you’ll want to pre-plan the disk layout for your first vdev because if you wish to expand your pool (1 or more vdevs) you’ll have to expand it with an identical layout. ie… If you have a RAIDZ2 vdev of 6 drives now, and you want to expand in the future, be prepared to get 6 more of the same drive and set it up in the same way to double your capacity.
However, if you were to create mirror pairs from the beginning, you could increase capacity by adding an additional vdev of another 2 disks at a time. Consequently, this provides good performance but also only provides for a single drive failure fault tolerance per mirror pair.
Moral of the story is, be very careful with how you layout your initial vdev because you’re tied to that configuration unless you do a full migration to a different layout.
It’s worth pointing out that RAIDZ expansion is being worked on for OpenZFS. No timeframe has been given, but it’s definitely in the pipeline. If they don’t plan on adding drives frequently, the feature might exist properly in a couple of years.
Hell, if future expansion is key and you want that sort of option now, you could look at unraid which provides for single disk expansion. Only thing is it’s not running on top of ZFS. But they did expand unraid a few years back to provide for dual parity drives to increase fault tolerance for the “pool”.
You are describing what you have today, not what you will have under a zfs scheme. Zfs and raid are not compatible. You will need to destroy your raid array and reset the controller to IT mode to benefit from zfs on the drives. This will undo any striping or caching functions of the raid controller that give you your current performance. Note you will also lose all the data on the array when you install zfs.
There is an introduction to zfs here that covers the basics
Ixsystems are also touting their hybrid array model for truenas core, which may be a more important feature if it means we can get high performance even from raidz2 schemes. The future of zfs looks exciting.
That much I know. I will do a complete backup before I migrate. Also I do not plan to use the raid controller in IT Mode. That is why I was asking if onboard SATA is enough. If I can get acceptable speed I’d like to get rid of the controller.
Correct
Only thing that currently is keeping me from migrating to ZFS: It seems using encryption with automatic mounting is a hassle. The archwiki basically creates a service that asks for the password. I do not want to enter an additional password at every boot, also I do not want to manually mount the dataset after every boot. I would like it using my keychain but I figure that functionality is out of the question.
If your data would be encrypted, have you considered setting the OS to boot without a password, and so you only have 1 passphrase/sentence/paragraph to enter?
I am certainly not going to boot without a password. Everything in my PC uses full disk encryption. But if ZFS is based on setting up different datasets and encryption is only possible per dataset I am not going to enter a series of passwords after every boot.
I definitely need a way so they get mounted automatically.
Not necessary. Encryption is setup at the pool level. While a pool is technically a dataset itself, you can have datasets within datasets so everything can be controlled with one encryption key. One password to unlock the pool and every dataset within.
Thank you! You are right, its possible. I just read an article about it. However I saw people not recommending it. If you need to change settings you need to destroy and recreate the whole pool and not just a new dataset. So people recommend to create the pool without a mountpoint and then one “main” dataset that is encrypted.
Also I am not sure anymore if I want to switch to ZFS. I was very enthusiastic about it, but the whole encryption functionality seems hacky. I have to choose to either use a password or a keyfile and can’t do both. To automatically mount I need systemd services that I do not know how to correctly create. I could only copy and paste from the arch wiki. Also I found several articles mentioning how the native encryption can leak metadata, but I was unable to find out what that includes.
As I said the whole ZFS thing seems cool, because of the data integrety but with encryption it seems hacky from my perspective. Correct me please if I am wrong here.
I don’t know if this has been suggested yet, but have you tried “gaming out” a ZFS configuration within a vm environment. You could essentially virtualize an entire environment with multiple disks and see how far you can go with creating the type of layout that you desire. All while not putting any real data at risk and being able to throw out and start from scratch. Just a thought, try before you buy.
I find this strange … mdadm on rust should be as good as any high-end raid controller even on older CPUs. What chunk size and filesystem did you test with?
Maybe it was not mdadm’s fault? what if it was a problem with port speed of the motherboards not being able to handle the throughput … that could answer your question.
I would retest with mdadm while running the HBA in JBOD mode and let the software handle the raid, this way you can determine if there are performance issues with onboard sata ports.
In all cases, I would not run a home storage with a hardware raid controller, simply because if the HBA card fails for whatever reason (and believe me they do fail), it would be a nightmare to recover the data … you would need an identical card that someone on ebay on the other side of the planet would charge you an arm and a leg for, if it had been discontinued and not available as new anymore, and when it does arrive, if it has older firmware it probably wouldn’t work, you would need to mine the internet for a firmware version you don’t know what it was (remember, the original HBA card had failed) and if you are lucky enough and the HBA manufacturer had not decimated or been acquired by another company, maybe you can land on the correct firmware version (and the tools to flash it) and recover the data finally … but it would have been 6 months already and you probably don’t need the data anymore or have learnt to live without it.
If you are determined on running raid on an hba, please buy another one and store it as spare.