Return to Level1Techs.com

LFA - ~24TB NAS build

I would like to build a “simple” NAS - NAS which will be mainly a way to remove HDDs from my PC and place it somewhere where I can ensure I can withstand a failure of one or more and and have an acceptable (but not 50%) capacity loss from parity drives.

In the future I want to run Jellyfin (or some other home media server w/o transcoding, just streaming and providing a unified interface - there will be time to do the research later though).

However, I am still missing some pieces, so I would like to ask you for a help as I think I am not sure how to move forwards now…

HDD Allocation + ZFS:

I would like to run ZFS - I like the benefits of speed and (imo) reliability plus regular error-checking and such. As for the allocation, I am considering (I hope a terminology is correctly used):

  • 3 stripped RaidZ1 Vdevs of 3 HDD per VDev (9 drives in total)
  • 2 stripped RaidZ1 Vdevs of 4 HDD per VDev (8 drives in total = 110€ saved)

which one would you suggest? Or is this as simple that whether I get more redundancy for more money or less redundancy for less money?

I will first build one Vdev and copy my ~7.5TB of data to it, then I will erase those, test them (as per this guide and if turn out well, I will add them together with the rest of new drives to the other Vdev (or in case of 3 Vdevs, each old HDD to one Vdev). Please correct me if this is not possible.

Hardware:

My current build draft:
CASE: Fractal Define R6 (already purchased, incl. extra HDD bays (10 in total available now))
PSU: Corsair RM550x (I will use one extra SATA power cable from my RM750x model in my rig)
MB: Supermicro X11SSH-CTF (unfortunately it is only for Intel’s 6th/7th gen, a shame they don’t have the same board for 8th/9th gen, I would then save 10€ on G5400 as opposed to an older G4560 (I cannot find anyone selling one))
CPU: Intel Pentium G4560 (for my use it should be plenty)
RAM: 2x Kingston KSM24ED8/16ME (not QVL’d by SuperMicro, but Kingston states X11SSH-CTF on its compatibility list)
HDD: 6-7x WD RED 4TB WD40EFRX (expensive for what it is, but whatever, I also need to live not far away from that server…)

Is there anything else you would recommend? Especially in terms of the board - I somehow know just of Supermicro in the “server world” and if so, I would like to have:

  • at least 10 SATA ports (whether straight or with additional ones via miniSAS
  • 10GbE NIC (RJ-45)
  • board where I can shove a cheap Pentium.

For me, 450€ is a lot of money and I lack an overview over this segment of the market, so maybe this price is an utter bargain (I would like to know if it is or it if it is just okay price or what not).

What about RAM, I think I could spend extra for 64GB of RAM, but would that be more useful in case I would have a bigger capacity array or does caching benefit from it in any case? I cannot really imagine what it would do for me in the real world in terms of a performance.

As far as the connection speed goes, I will buy Qnap 5GbE USB adapter for my PC (I use a vertical GPU mount, so there is no space left for an add-in card). I will then directly interconnect NAS and my PC for the maximum speed and use the second ETH port for the rest of the network (1Gbit).

I will also add UPS (for both, NAS and my PC - power outages are rare, but I hate when they occur and all cuts out), but I need to do more research first (especially what capacity I need), I heard about a protocol to communicate via USB and an option to leave NAS automatically shut down in such scenario, so I want to use it (will that affect my OS choice below??).

OS:

Which would you suggest? I don’t want to use UnRaid (I don’t need its VM capabilities and I don’t want to be required to have at least 1 UR array (from what I read in the piece with GN)).
From my mainstream point of view it leaves me with probably just two options, FreeNAS or OMV. FreeNAS could be the best to not require any (or the least) fiddling, while OMV would require me to install Proxmox Linux kernel and the ZFS plugin, but afterwards it should be all configurable via GUI afaik.

Just a note, I am not shy of CLI, but I work in CLI all day in work and I would prefer to have this in GUI form, I am not sure if I would feel comfortable to do a lot of ZFS configuration in CLI first.

Would you mind also advising me on how I can pair ID in OS with the particular HDD in the system? Is there a S/N readout in OS which I could pair with S/N from labels on HDDs?

Thanks anyone for sharing a thought or two, I really appreciate it.

I wouldn’t go with either of these. For this use case a single RAIDZ2 vdev of 8 drives would give you all the redundancy you need (2 disk) with simpler pool management. You can keep the 9th drive as hotspare.

Sounds complex. If your backup regime is good (2 copies in separate cold storage locations) you don’t need to do this. Just rely on your backups or use your hotspare drive as a temporary “third copy”. It is far better to create your array once and leave it alone than faff about creating testing and trashing vdevs.

At least look at a Xeon server CPU… You may find a used one at a similar price that adds way more features and support for future growth. Those Pentiums are “basic” and won’t give you a lot of ram options at higher capacities

I use these. Good drives.

Get a HBA. 8ports for €40 is easy to come by and the controller will give you an easier time passing through if you choose to virtualise the whole thing.

For beginners I’d normally suggest not jumping straight into a 9 disk raidz2 array setup, so if you want to practice build perhaps just start on one used disk and build the rest of your system, including the transcoder. If it works as you expect then spend the rest of the cash on drives. I am curious about your choice of 8/9 drives rather than 3 big ones (cheaper) or 5/6 modest ones (cost neutral). It is generally better these days to go for fewer larger drives to give the redundancy you want. Raidz2 works well with 6 drives. Mixing old and new to get that bit more capacity may be a false economy. You could just put the 3 old disks into an “archive” array for secondary backups.

Not worth it unless you upgrade the CPU On a 5Gbps network the 16GB buffer is plenty for that size array Caching on ZFS is not what we all want it to be (a fast write buffer) but more a working area for regularly accessed files. If all you are doing is one off transcodes and file transfers it adds nothing.

I think freeNAS would suit you. Any other OS needs a different CPU and more grunt. You just need basic storage and transcode via a jailed app. Freenas is fine for that.

I’m sure there is a proper way to do it, but I tend to just add the vdev’s worth of disks at the same time and create my vdev / pool. Then add the next set. Tends not to be an issue if they are all over the place on the controller. If you are worried about data, keep your live data off the server whilst building and copy it over the network. Makes a good test of the setup anyway.

Good luck

2 Likes

You will probably need a better CPU to make good use the that 10gb eathernet port. With the CPU you have selected you probably wont see 10gb and ZFS does require CPU resources especially but not only if you are using encryption. I would go with a quad core minimum something with a larger cache. Personally I would use an entry level Xeon here. I also have to remind you of Intel’s security flaws that have yet to be fixed for a system you will most likely leave running 24/7.

For those reasons I would go with AMD Ryzen. AMD Ryzen will cost less, offer more cores, perform better and support ECC as well. AMD Ryzen also does not have the security flaws. But you will have to buy something like am Aquantia 10gb NIC or get a newer higher end motherboard that includes the 10gb NIC.

1 Like

@Airstripone, thank you for your input. If I may, I would like to iterate on some points:

  • RaidZ2 - would Z2 not suffer the biggest write-hole? In case of a future resilvering, it would put read load on just other two or three drives instead of all 8 drives, would it not?
  • Backup regime - not existent, unfortunately, that is the main drive factor on building the server. Backup is difficult for me, when bulk of data are my ripped shows from BD, I can’t really imagine how to back up 7TB of data
  • Xeon - the problem used Xeons are literally nowhere to be found, even the entry model is over 200€, which is unacceptable for a mere quad core w/o HT, but I started looking at eBay, are such CPUs
    https://www.ebay.com/itm/352793754221
    recommended? Are those legit?
  • HBA - are you referring to ones such as Dell H310? I…honestly I know about those, but just as that offer from above, no warranty, no known origin and abuse throughout its lifespan and more HW, more potential points of failure – the very same reason I didn’t want to go with some random HBA, some random 10GbE Base-T NIC,…
  • amount of drives - I don’t know if I just have better basics, but for me it is the same reason why recovery of drives raises exponentially with an increasing capacity – having to rebuild 2TB drive is considerably quicker than having to rebuild 10TB drive and putting your array into the risk in the meantime - thus why 4TB drives (3TB drives cost just about 10% less), not mentioning that higher capacity drives present a much worse price per GB ratio, so the price would creep up substantially and also a potential risk - another problem, which is emerging is SMR, those drives are not appropriate to be used in NAS - those are good…for archiving and only a sporadic reads
  • RAM - thank you, I actually like to hear it, I think I will go with 32GB, the investment is not that high, I am quite comfortable with it
  • pairing HDDs - the reason I asked is that in case a drive fails how can I then know which is bad?

@OrbitaLinx, I was already looking at Ryzen, but you will not find any real AM4 server boards and consumer boards…well, especially with FreeNAS it may be a bad idea (driver issues), I looked at those Asrock Rack and we have literally just one AM4 available locally and it is just a bit cheaper than that Supermicro board I chose, but has just 1GbE NIC, only 6 SATA3 ports and no miniSAS. But I wish I could do it, Ryzen 5 1600 (AF) would be perfect.

ZFS uses COW ( so no write hole issue)

1 Like

They do a have a 10Gb version of that MB… you just lose come PCIe

I have the 1Gb version motherboard. But I installed a aquantia 10GbE and a sas/sata HBA for my drives.

This is a tough one.

My recommendation is to attach them to the pool by id. (/dev/disk/by-id). That way, you can see the serial number. If you can’t read the disk labels while they’re installed, I’d grab a label maker and print the serial on each and stick it to the side that sticks out. Some server enclosures provide backplanes that give you the option to put an indicator on a specific bay, but that’s expensive.

For example, this is truncated output of some of the disks in my workstation:

You can see the serials in the last segment (excluding the -partX)

As you can see above, I went 8x8TB. I got em for $150 ea, I believe. Obviously, they’re not all the same, but that’s because I’ve got a couple reds and the rest are white-label from externals. I’d highly recommend going that route if you need to save a buck.

1 Like

Yeah, that’d have been pulled from a server or overstock. It’s a used CPU, but totally fine. In fact, I had one of those exact CPUs in my server a few years back. It did the trick.

Thanks for the followup

No, that’s not how it works. ZFS is distributed parity so the resilvering is from all 7 remaining drives taking bits of data from each in parallel. You actually end up with 3 copies of some data in RAIDZ2. Read load is across the lot and not 100% on any one disk.

Not necessarily a knock off Dell but a used LSI controller is both cheap and plug&play. Cards like that don’t really “wear out”, they either work or they don’t. If you trust all 10 SATA ports on the mobo to meet your needs then fine, but if you want to virtualise then a HBA can be passed through. That’s all.

Not exponentially, but it does increase with capacity and in traditional raid it is 100% rebuild before you can use the array due to the way it resilvers. With ZFS you have distributed parity and improved bit error recovery, so much less likely to trash the array on rebuild. Plus ZFS will give you more warning of a failing disk to give time to take action. My CCTV array had been spitting out earnings for months and if I cared about the data I could have taken action before it fails. As it stands I’m using it as a test of how long ZFS can cope with a duff drive.

Agree don’t go for SMR drives but you aren’t there anyway. You are looking at the delta between two sizes of PMR drives. As @sgtawesomesauce says, look for cheap deals on bigger disks as it will save you money long term.

Covered by others but to add, you should have hotspare capacity (or spare header) to rebuild the array before shutting down and removing the failed disk. Unless you have a backplane with blinkenlights there is no trivial way to know which port the mobo had assigned to disks other than adding them one disk at a time and use a marker pen to write the ID on the cable (done that before). ZFS is pretty tolerant of shutdowns. We talk about drive failures here in theoretical terms, they are not that common so this isn’t a regular chore.

Saved the best for last. Backup 7TiB data using one 8TiB drive in a removable caddy. Take the drive out and put it in a box. Send it to a relative’s house with a tray of cookies saying “keep safe”. Once a month or so go pick it up and bake more cookies.

If you use good backup software you can just write the changes to the disk next time and save time and drive writes.

The next upgrade is to rotate 2 disks, one onsite and one off-site, with the onsite one doing incremental backups until you hit a checkpoint where you want to do a full drive write. Swap the disks and do a full write to the “B” disk. This is my preferred method, as it gives both two levels of drive redundancy and in the event of a data corruption, a higher chance that my B drive has a good version.

If your backup regime is good them RAIDZ1 is adequate for a home NAS. So it is cost neutral to have say a 5 disk RAIDZ1 array and 2 disk backup rotation versus 6 disk Z2 and 1 backup disk. Both better than Z1 or Z2 with no backups!

Choice is yours but from a disk perspective that is my suggestion. Others have covered your board and CPU options.

Good luck

Thank you all for follow-ups.

@nx2l, model incl. 10GbE NIC is X470D4U2-2T, isn’t it? Mhm, it is about 100€ (just ignore €, you can substitute it with $, it is more the less the same), which is not an insignificant amount, in terms of HBA, which would you suggest? If I could still save at least something compared to that Supermicro and I would get some 2600 (I have my doubts 1600 (AF) would work…), that would boost the CPU performance a lot compared to my Intel alternative.

And then, what OS solution do you use? How does it play with AMD HW? Any issues?

@sgtawesomesauce, fantastic, I will label them before putting them in the case. This way I can (almost) easily pair them.

And those drives…are all those shucked? Most of 8TB ones (especially white ones) are SMR and more I would have in my machine, more anti-depressive pills I would have to stock so when rebuilding occurs, I will not go insane that another one fails in the meantime. I don’t think shucking is worth doing anymore unless one builds an archival machine (on which some ultra-capacity archivation BRs may be more suitable anyway).

At the end of the day, I am just reaching 8TB, even 24TB will be plenty for me, and in the future when I need more storage we will have already much cheaper high-capacity SSDs which would result in pretty much noiseless NAS (I would put the same trust in a quality SATA SSD with TLC NAND as in PMR HDD).

@Airstripone, I see, RaidZ2 would work (well I would have to buy 8TB external HDD to copy all data to it, build the array and copy it back, but it would work), you mentioned that a single RaidZ2 would mean a simpler pool management, would you mind elaborating on that? Are there other reasons why to go with 1 RaidZ2 of 8 drives vs. 2x RaidZ1 of 4 drives each?

is there a particular HBA you would suggest (I don’t want to sound cheap, but…well, I would like not to pay 100€ for one (used on eB of course).


So to wrap up things I am wondering about now:

  • AMD build using Ryzen 2600 + Asrock Rack X470D4U2-2T (+ some HBA card, in such case, which is the least expensive option to plug in at least 8 SATA3 drives) or keep Intel build, but upgrading CPU to Xeon E3-1240L?
  • 2x RaidZ1 (4HDDx2) or 1x RaidZ2 (8HDD)?
  • does FreeNAS look really like the best option and there is no need to thinking about something else?
  • is there anything I am missing?

10Gb is still being refined for consumer use… with the 10Gb MB you cant upgrade that unless you get a PCIe card

the aquantia 10GbE isnt a X550 , but its certainly cheaper, and whatever replaces x550… wont be possible to update when its embedded .

But another argument can be made for when that actually happens, the MB will be obsolete anyway, but see the popularity of R710 dell servers…x58 era)

The 10Gb MB has more limited PCIe slots compared to the 1Gb version. I went with the 1Gb version, b/c adding 10Gb via pcie was cheaper than paying for the 10Gb MB version.

Its personal preference really.

The main difference is it is a one step build. Just add the 8disks and select 2 disk parity, it will do the rest. Your multi vdev option means building twice as many pools.

In the Z2 model you can lose any 2 disks from 8 and it will still run long enough to get your data off.

In the striped Z1 model, if you 2 disks in the same vdev that is it, you lose all 8 drives.

@Airstripone, I see, then RaidZ1x2 is actually a bit less secure as in such case you can have 2 failed drives, but only and only if each is in one Vdev, otherwise everything is lost. I haven’t thought about it that way. Albeit there seems to be a slight write speed advantage of RaidZ1x2 (source here), it hardly seems like a difference worth the risk. Thank you, I will go with RaidZ2, I think I will build it fully from new drives, copy data over and use one of my current ones as a spare and sell the other one. A bit more expensive upfront, but, well, life is for living.

Which basically completes the ZFS and allocation question once and for all and leaves only the last one on the table (I think I should not overthink the OS and FreeNAS will be the best for my needs (ZFS via GUI with the least of CLI work)).

AMD vs. Intel (FreeNAS as OS), what would you choose, AMD Ryzen 2600 + Asrock Rack X470D4U2-2T (+ some HBA card, in such case, which is the least expensive option to plug in at least 8 SATA3 drives), or Intel Xeon Xeon E3-1240L + current Supermicro X11SSH-CTF (more expensive, no need for additional HBA card)?

Glad you have resolved this. Just be sure to DBAN the drives before you sell them, or keep them as tier2 storage.

As for hardware, frankly it is a wash. I use intel only because it is used hardware and was cheap but gives me 64GB ddr3 ecc ram and 100% compatibility with freenas. I have an Intel x550 NIC and 2 LSI SAS controllers.

My production machine is threadripper and is better in every way, but ddr4 is expensive and I don’t need 24 threads on my NAS. If I were buying new I’d go amd for my NAS. I wouldn’t buy new though.

None of the 8TB disks in the WD EasyStores are SMR. Not sure where you got that from.

I’ve got 8 white-label shucked drives.

They are white-label reds or enterprise disks.

@Airstripone, so I am basically tossing a coin here. As for HBA, if I search “lsi hba card” on eBay, what should I be looking for? I see tons of unbranded cards from China, is there anything amongst those worth looking at (I am mostly interested where legit offers start, $30, $40?

@sgtawesomesauce, you are right, some time ago I was digging through an unending amount of reddit threads looking for such information, but it seems 8TB+ models should still be white-labeled REDs, however there are strange information around new RED “EFAX” models, where it seems lower capacity models already have both, PMR and SMR versions (SMR being those models with 256MB of cache)


but probably the 8TB models are still PMR, or maybe not…they should be direct about this, this feels sometimes more like a ghost hunting.

I tend to have more luck just searching for sas controllers as you get more used listing and rebranded LSI controllers, like a Dell PERC. The last ones I got were Toshiba branded but same chipset. Do your research as there is usually extensive documentation for “good” brands. Anything you can’t find a data sheet for should be avoided. A quick search on eBay shows a lot of HP and Adaptec branded cards. Avoid these as driver support is less optimal. Dell seem to be looked on kindly for ease of flashing the bios to IT mode.

I don’t live in the US but a reasonable used card price for a 2port 6Gbps HBA would be the equivalent of $40 here, so if you see new ones listed for less than that… avoid. Also avoid anything where it is just a photo of a card in a plastic tray.

If you are less fussed about peak performance an older 3Gbps card can be hard for $10 or so. Good way to get started cheap, test the rest of your setup then upgrade to a faster HBA when you find one.