Building a beginner-level ZFS pool - any advice?

TL;DR: What HDDs do I buy for ZFS pool?

Hi all!

I am not a beginner to Linux, but I am a beginner to self-hosting, and I fell down the rabbit hole quite a bit, to be honest :smiley:

My current setup is as follows:

  • Dell Optiplex 7050 micro (16GB DDR4, 500GB NVMe)
  • 4TB WD Red (connected via USB to the Optiplex)
  • Proxmox on the host, VMs inside

The future plan is to implement a ZFS pool for the NAS, since the single 4TB drive holds a bunch of data I wouldnā€™t want to lose. My idea is to get some dock for multiple HDDs (like the Orico Honeycomb or similar multi-HDD dock), and populate it with a couple of hard drives.

My question is this: If I buy 3 HDDs now of a certain size, can the ZFS pool then be extended with more HDDs of different sizes? Do I have to migrate my complete ZFS pool later if I replace all the drives with drives of a different size?

Thanks!

You may find this article from Klara systems helpful. I personally found I was able to use ZFS with a great deal more confidence after reading zfs mastery.

Each pool can be comprised of any number of vdevs; their topology and size donā€™t have to match. Each vdev is responsible for its own redundancy, and each disk in the vdev must store the same amount of data. Note that this means that you can have mixed size drives in a vdev but you can only use the capacity of the smallest drive. In my case I have a mirror of two drives: one with 3TB and the other with 4TB, so the vdev is 3TB in size.

To answer your question, say you build a pool with a raidz1 vdev from your drives. You can add a vdev of mirrored drives (two) later to increase the storage space of the pool. You could add another raidz1 vdev with larger (or smaller) disk sizes.

What you canā€™t do is convert the existing vdev type (say raidz1 with your three drives) without temporarily migrating the pool to another system. You can do an in-place upgrade of the disks by replacing them with larger capacity drives, although you have to do it one at a time which can be lengthy.

1 Like

With 4TB SATA SSDs now reaching below $150 and starting to touch $100 on promo deals, I can no longer in good conscience recommend HDDs unless you need more than 20 TB of storage. It just isnā€™t worth the hassle anymore.

While yes, you can get an 8 TB HDD for $110 and a 16 TB HDD for $260, these will be so atrociously slooooooow by now it will feel like being on a race track with a two cylinder engine. Consumer level SSD NAS is now a thing, still mostly for enthusiasts, but it is becoming rapidly affordable for high end and midrange buyers, too.

3 Likes

I wouldnā€™t say I ā€œneedā€ 20TB of storage. To be honest, I donā€™t really ā€œneedā€ the 4TB I have now. But I want it. Iā€™m content with my data running a little slower, this little bastard can encode 4K video just fine even from spinning rust, so I guess the price per TB is a bigger hurdle for me.

Iā€™ve checked my local prices (not in the US):

4TB SSD (Samsung 870): $211
4TB HDD (WD Red): $102

That is, in my mind, a good enough reason to stick with spinning rust at this point in time. I may be very content with having a ton of storage, because even though I only have 2TB used on the HDD, I am wary that this space may run out very soon.

Thank you for the book recommendation! Iā€™ll read that (and the article too, of course).

So basically I can add other vdevs later, and point the ZFS pool to ā€œalso use those ones as wellā€? Thatā€™s what I would love to be able to do, buy 1 drive now, another drive next month, etc. to progressively expand the storage space.

Thank you! :heart:

do not use RAIDZ1 (long resilver times, possible data corruption, the list goes on)

if you want the best and smallest way to be expandable in the future use mirror pools of 2 drives instead of any RAIDZ config.

for a home user (actually never) would i recommend mixing VDEV types in a pool.

you really should not try to add 1 drive at a time so if you buy them individually you would be able to add in batched of 2 by using mirrors.

2 Likes

If you want to be able to ā€œeasilyā€ add drives with zfs, you should do what @Zedicus advises and use mirror vdevs. That way you can add drives 2 at a time, which will not only increase your capacity, but your (theoretical) performance as well.

FWIW this is what I use (three vdevs of mirrors)

1 Like

So if I understand correctly, I get a DAS (anything I can fit drives in), and then put in 2 drives, mirror them, and later I can add 2 more drives to mirror that first 2-drive mirror?

That way, I give up 50% of the storage for a 2-drive fail tolerance, if I understand correctly?

Thank you for this!

1 Like

Do you have like a 2U server with drives in? or what do you put the drives in?

Thatā€™s one thing Iā€™m solving as well

Okay, for everyone involved, Iā€™ll have to pop this one feeling in and what I found that may have changed my mind (in addition to literally everyone here):

https://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-not-raidz/

This is the one thing I was worried about. I am a greedy, penny-pinching person when it comes to storage. Drives are not cheap, and I am now in a position where I have 1 4TB drive. I donā€™t want to lose it, but I also have no way to back that thing up. If it goes, itā€™s off to the data recovery center and they will charge much more.

So I believe the next plan is as follows (please correct me if I write anything stupid):

  • Get a rack to put drives in (4-bay, maybe, if I can find a 6-bay Iā€™ll save up for it)
  • Buy 2 4TB HDDs (I believe thatā€™s what I got in there now)
  • Set up a mirror vdev
  • Move data from my current 4TB HDD to the VDEV and keep it as a cold backup/expansion/replacement drive

This way, I get what I had before, but with plenty more resiliency and a backup!

After that, I can buy HDDs in pairs, as apparently the size difference between VDEV mirrors shouldnā€™t be bad for the overall storage (the 6TB would mirror each other and the 4TB would mirror each other). Now just to figure out if I can use the internal SATA drive on the Optiplex to hook up to a 5.25" HDD bay (finding external HDD bays is pain)

Am I thinking right, or did I miss some huge thing that will mess my day up?

2 Likes

yes*

say you have 4x 2 drive mirrors, you could loose 4 drives going down the row, but any 2 drives in a single VDEV and you would be SOL there.

when i started out i used an ATX tower with a bunch of 5.25 bays and 3.25 adapter trays. a NAS can be as simple or complicated as you make it.

2 Likes

Unless the docks are using eSATA itā€™s not adviced to use ZFS over USB HDDs due reliability.

I use the Silverstone DS380, it sits on a bookshelf in my living room.

I would think of it as 1 drive of resiliency. Maybe 1.5 drives? You can lose your pool with two drive failures (if they are in the same vdev), so itā€™s definitely less than 2.

My initial setup was a cheap SOC motherboard and throw away case. Definitely good for learning. As I transitioned into keeping important data, I found it was worth buying more reliable stuff (IPMI, ECC, hot swap bays), but thatā€™s hard to justify when itā€™s your first nas.

if you do not already have stuff, i recomend an H8SGL and an opteron 6366HE. (if you find a good deal on a board with different CPU included, it is probably fine. the 6366HE is the lowest power that gen, though) this will support ECC REG RAM, is DIRT cheap, and is reasonably low power considering its age.

Toss any PCIE HBA you can find on it and you have a brilliant reliable nas. (I am still using one of these as a NAS right now)

1 Like

If the data is really important and canā€™t be found again (family photos and the like). You really need to look into a backup plan. Raid is for uptime, not backup. I personally use Backblaze for about $7 a month and have been happy. There are a few other providers in this space.

1 Like

ā€˜Amazon Glacier Deep Archiveā€™ is another option, seems very cheap storage from what Iā€™ve seen if you donā€™t plan to access the data much or at all. It is not as user friendly, but once your backup system is setup it wouldnā€™t be an issue.

1 Like

Since you are now building a NAS proper, here is what I would get today:

Old school - 5 bay QNAP or Synology:

$649, Power draw ~60W, and you need to buy drives for this. Compare then with the Asustor Flashstor:

$449, Power draw ~30W, and six drives instead of five. Also, whisper quiet.

Now drives: What would I go for? Looking at HDDs 8TB seems to be the sweet spot between price per TB and actual price, so I am settling for the Seagate Barracuda Compute for $110 WD Blue for $125:

As for SSDs, any m.2 gen 3 drive will outperform the HDDs, but lets go with price per GB again. Crucial P3 4TB are $190 right now so let us go with that:

So, for the added cost of $65 per drive and 24 TB of raw storage instead of 40 TB (20 vs 32 if RAID5, 16 vs 24 if RAID6), you get incredible performance. Total costs, all decked out on both platforms, is $1275 vs $1590, or roughly 25% extra for superior speed, reliability and power consumption.

Furthermore HDDs are now getting more expensive to produce due to lower batch numbers. While 8TB and 32TB SSD costs will creep down eventually due to economics of scale, HDDs will inevitably increase in price, going forward.

This is why I no longer recommend HDDs for small NAS storage needs. Of course, if you require 40-50 TB of storage or more, which is rare for home use but it happens, then HDDs still make sense, but once 16TB SSDs becomes affordableā€¦

If youā€™re going to recommend hardware please do a bit of research first?

Shows that youā€™re recommending SMR HDDs for ZFSā€¦

The P3 Plus performs a lot better and is on 10$ more

Right you are. So what is the cheapest 8TB consumer drive that isnā€™t SMR then? I will upgrade to a $125 WD Blue but if you can find cheaper I will listen. This also means there is now a +$75 difference in price, which slightly helps my argument here.

P3 plus is nice but here we are talking storage / price not (excessive) perf / price. The only difference is the speed, both can saturate 10GbE easily.