LFA - ~24TB NAS build

Hardware Build a PC
,
1

I would like to build a “simple” NAS - NAS which will be mainly a way to remove HDDs from my PC and place it somewhere where I can ensure I can withstand a failure of one or more and and have an acceptable (but not 50%) capacity loss from parity drives.

In the future I want to run Jellyfin (or some other home media server w/o transcoding, just streaming and providing a unified interface - there will be time to do the research later though).

However, I am still missing some pieces, so I would like to ask you for a help as I think I am not sure how to move forwards now…

HDD Allocation + ZFS:

I would like to run ZFS - I like the benefits of speed and (imo) reliability plus regular error-checking and such. As for the allocation, I am considering (I hope a terminology is correctly used):

  • 3 stripped RaidZ1 Vdevs of 3 HDD per VDev (9 drives in total)
  • 2 stripped RaidZ1 Vdevs of 4 HDD per VDev (8 drives in total = 110€ saved)

which one would you suggest? Or is this as simple that whether I get more redundancy for more money or less redundancy for less money?

I will first build one Vdev and copy my ~7.5TB of data to it, then I will erase those, test them (as per this guide and if turn out well, I will add them together with the rest of new drives to the other Vdev (or in case of 3 Vdevs, each old HDD to one Vdev). Please correct me if this is not possible.

Hardware:

My current build draft:
CASE: Fractal Define R6 (already purchased, incl. extra HDD bays (10 in total available now))
PSU: Corsair RM550x (I will use one extra SATA power cable from my RM750x model in my rig)
MB: Supermicro X11SSH-CTF (unfortunately it is only for Intel’s 6th/7th gen, a shame they don’t have the same board for 8th/9th gen, I would then save 10€ on G5400 as opposed to an older G4560 (I cannot find anyone selling one))
CPU: Intel Pentium G4560 (for my use it should be plenty)
RAM: 2x Kingston KSM24ED8/16ME (not QVL’d by SuperMicro, but Kingston states X11SSH-CTF on its compatibility list)
HDD: 6-7x WD RED 4TB WD40EFRX (expensive for what it is, but whatever, I also need to live not far away from that server…)

Is there anything else you would recommend? Especially in terms of the board - I somehow know just of Supermicro in the “server world” and if so, I would like to have:

  • at least 10 SATA ports (whether straight or with additional ones via miniSAS
  • 10GbE NIC (RJ-45)
  • board where I can shove a cheap Pentium.

For me, 450€ is a lot of money and I lack an overview over this segment of the market, so maybe this price is an utter bargain (I would like to know if it is or it if it is just okay price or what not).

What about RAM, I think I could spend extra for 64GB of RAM, but would that be more useful in case I would have a bigger capacity array or does caching benefit from it in any case? I cannot really imagine what it would do for me in the real world in terms of a performance.

As far as the connection speed goes, I will buy Qnap 5GbE USB adapter for my PC (I use a vertical GPU mount, so there is no space left for an add-in card). I will then directly interconnect NAS and my PC for the maximum speed and use the second ETH port for the rest of the network (1Gbit).

I will also add UPS (for both, NAS and my PC - power outages are rare, but I hate when they occur and all cuts out), but I need to do more research first (especially what capacity I need), I heard about a protocol to communicate via USB and an option to leave NAS automatically shut down in such scenario, so I want to use it (will that affect my OS choice below??).

OS:

Which would you suggest? I don’t want to use UnRaid (I don’t need its VM capabilities and I don’t want to be required to have at least 1 UR array (from what I read in the piece with GN)).
From my mainstream point of view it leaves me with probably just two options, FreeNAS or OMV. FreeNAS could be the best to not require any (or the least) fiddling, while OMV would require me to install Proxmox Linux kernel and the ZFS plugin, but afterwards it should be all configurable via GUI afaik.

Just a note, I am not shy of CLI, but I work in CLI all day in work and I would prefer to have this in GUI form, I am not sure if I would feel comfortable to do a lot of ZFS configuration in CLI first.

Would you mind also advising me on how I can pair ID in OS with the particular HDD in the system? Is there a S/N readout in OS which I could pair with S/N from labels on HDDs?

Thanks anyone for sharing a thought or two, I really appreciate it.

2

I wouldn’t go with either of these. For this use case a single RAIDZ2 vdev of 8 drives would give you all the redundancy you need (2 disk) with simpler pool management. You can keep the 9th drive as hotspare.

Sounds complex. If your backup regime is good (2 copies in separate cold storage locations) you don’t need to do this. Just rely on your backups or use your hotspare drive as a temporary “third copy”. It is far better to create your array once and leave it alone than faff about creating testing and trashing vdevs.

At least look at a Xeon server CPU… You may find a used one at a similar price that adds way more features and support for future growth. Those Pentiums are “basic” and won’t give you a lot of ram options at higher capacities

I use these. Good drives.

Get a HBA. 8ports for €40 is easy to come by and the controller will give you an easier time passing through if you choose to virtualise the whole thing.

For beginners I’d normally suggest not jumping straight into a 9 disk raidz2 array setup, so if you want to practice build perhaps just start on one used disk and build the rest of your system, including the transcoder. If it works as you expect then spend the rest of the cash on drives. I am curious about your choice of 8/9 drives rather than 3 big ones (cheaper) or 5/6 modest ones (cost neutral). It is generally better these days to go for fewer larger drives to give the redundancy you want. Raidz2 works well with 6 drives. Mixing old and new to get that bit more capacity may be a false economy. You could just put the 3 old disks into an “archive” array for secondary backups.

Not worth it unless you upgrade the CPU On a 5Gbps network the 16GB buffer is plenty for that size array Caching on ZFS is not what we all want it to be (a fast write buffer) but more a working area for regularly accessed files. If all you are doing is one off transcodes and file transfers it adds nothing.

I think freeNAS would suit you. Any other OS needs a different CPU and more grunt. You just need basic storage and transcode via a jailed app. Freenas is fine for that.

I’m sure there is a proper way to do it, but I tend to just add the vdev’s worth of disks at the same time and create my vdev / pool. Then add the next set. Tends not to be an issue if they are all over the place on the controller. If you are worried about data, keep your live data off the server whilst building and copy it over the network. Makes a good test of the setup anyway.

Good luck

2 Likes
3

You will probably need a better CPU to make good use the that 10gb eathernet port. With the CPU you have selected you probably wont see 10gb and ZFS does require CPU resources especially but not only if you are using encryption. I would go with a quad core minimum something with a larger cache. Personally I would use an entry level Xeon here. I also have to remind you of Intel’s security flaws that have yet to be fixed for a system you will most likely leave running 24/7.

For those reasons I would go with AMD Ryzen. AMD Ryzen will cost less, offer more cores, perform better and support ECC as well. AMD Ryzen also does not have the security flaws. But you will have to buy something like am Aquantia 10gb NIC or get a newer higher end motherboard that includes the 10gb NIC.

1 Like
4

@Airstripone, thank you for your input. If I may, I would like to iterate on some points:

  • RaidZ2 - would Z2 not suffer the biggest write-hole? In case of a future resilvering, it would put read load on just other two or three drives instead of all 8 drives, would it not?
  • Backup regime - not existent, unfortunately, that is the main drive factor on building the server. Backup is difficult for me, when bulk of data are my ripped shows from BD, I can’t really imagine how to back up 7TB of data
  • Xeon - the problem used Xeons are literally nowhere to be found, even the entry model is over 200€, which is unacceptable for a mere quad core w/o HT, but I started looking at eBay, are such CPUs
    https://www.ebay.com/itm/352793754221
    recommended? Are those legit?
  • HBA - are you referring to ones such as Dell H310? I…honestly I know about those, but just as that offer from above, no warranty, no known origin and abuse throughout its lifespan and more HW, more potential points of failure – the very same reason I didn’t want to go with some random HBA, some random 10GbE Base-T NIC,…
  • amount of drives - I don’t know if I just have better basics, but for me it is the same reason why recovery of drives raises exponentially with an increasing capacity – having to rebuild 2TB drive is considerably quicker than having to rebuild 10TB drive and putting your array into the risk in the meantime - thus why 4TB drives (3TB drives cost just about 10% less), not mentioning that higher capacity drives present a much worse price per GB ratio, so the price would creep up substantially and also a potential risk - another problem, which is emerging is SMR, those drives are not appropriate to be used in NAS - those are good…for archiving and only a sporadic reads
  • RAM - thank you, I actually like to hear it, I think I will go with 32GB, the investment is not that high, I am quite comfortable with it
  • pairing HDDs - the reason I asked is that in case a drive fails how can I then know which is bad?

@OrbitaLinx, I was already looking at Ryzen, but you will not find any real AM4 server boards and consumer boards…well, especially with FreeNAS it may be a bad idea (driver issues), I looked at those Asrock Rack and we have literally just one AM4 available locally and it is just a bit cheaper than that Supermicro board I chose, but has just 1GbE NIC, only 6 SATA3 ports and no miniSAS. But I wish I could do it, Ryzen 5 1600 (AF) would be perfect.

5

ZFS uses COW ( so no write hole issue)

1 Like
6

They do a have a 10Gb version of that MB… you just lose come PCIe

I have the 1Gb version motherboard. But I installed a aquantia 10GbE and a sas/sata HBA for my drives.

7

This is a tough one.

My recommendation is to attach them to the pool by id. (/dev/disk/by-id). That way, you can see the serial number. If you can’t read the disk labels while they’re installed, I’d grab a label maker and print the serial on each and stick it to the side that sticks out. Some server enclosures provide backplanes that give you the option to put an indicator on a specific bay, but that’s expensive.

For example, this is truncated output of some of the disks in my workstation:

Screenshot%20from%202020-02-05%2010-13-39
Screenshot from 2020-02-05 10-13-39.png458×615 24.7 KB

You can see the serials in the last segment (excluding the -partX)

As you can see above, I went 8x8TB. I got em for $150 ea, I believe. Obviously, they’re not all the same, but that’s because I’ve got a couple reds and the rest are white-label from externals. I’d highly recommend going that route if you need to save a buck.

1 Like
8

Yeah, that’d have been pulled from a server or overstock. It’s a used CPU, but totally fine. In fact, I had one of those exact CPUs in my server a few years back. It did the trick.

9

Thanks for the followup

No, that’s not how it works. ZFS is distributed parity so the resilvering is from all 7 remaining drives taking bits of data from each in parallel. You actually end up with 3 copies of some data in RAIDZ2. Read load is across the lot and not 100% on any one disk.

Not necessarily a knock off Dell but a used LSI controller is both cheap and plug&play. Cards like that don’t really “wear out”, they either work or they don’t. If you trust all 10 SATA ports on the mobo to meet your needs then fine, but if you want to virtualise then a HBA can be passed through. That’s all.

Not exponentially, but it does increase with capacity and in traditional raid it is 100% rebuild before you can use the array due to the way it resilvers. With ZFS you have distributed parity and improved bit error recovery, so much less likely to trash the array on rebuild. Plus ZFS will give you more warning of a failing disk to give time to take action. My CCTV array had been spitting out earnings for months and if I cared about the data I could have taken action before it fails. As it stands I’m using it as a test of how long ZFS can cope with a duff drive.

Agree don’t go for SMR drives but you aren’t there anyway. You are looking at the delta between two sizes of PMR drives. As @SgtAwesomesauce says, look for cheap deals on bigger disks as it will save you money long term.

Covered by others but to add, you should have hotspare capacity (or spare header) to rebuild the array before shutting down and removing the failed disk. Unless you have a backplane with blinkenlights there is no trivial way to know which port the mobo had assigned to disks other than adding them one disk at a time and use a marker pen to write the ID on the cable (done that before). ZFS is pretty tolerant of shutdowns. We talk about drive failures here in theoretical terms, they are not that common so this isn’t a regular chore.

Saved the best for last. Backup 7TiB data using one 8TiB drive in a removable caddy. Take the drive out and put it in a box. Send it to a relative’s house with a tray of cookies saying “keep safe”. Once a month or so go pick it up and bake more cookies.

If you use good backup software you can just write the changes to the disk next time and save time and drive writes.

The next upgrade is to rotate 2 disks, one onsite and one off-site, with the onsite one doing incremental backups until you hit a checkpoint where you want to do a full drive write. Swap the disks and do a full write to the “B” disk. This is my preferred method, as it gives both two levels of drive redundancy and in the event of a data corruption, a higher chance that my B drive has a good version.

If your backup regime is good them RAIDZ1 is adequate for a home NAS. So it is cost neutral to have say a 5 disk RAIDZ1 array and 2 disk backup rotation versus 6 disk Z2 and 1 backup disk. Both better than Z1 or Z2 with no backups!

Choice is yours but from a disk perspective that is my suggestion. Others have covered your board and CPU options.

Good luck

10

Thank you all for follow-ups.

@nx2l, model incl. 10GbE NIC is X470D4U2-2T, isn’t it? Mhm, it is about 100€ (just ignore €, you can substitute it with $, it is more the less the same), which is not an insignificant amount, in terms of HBA, which would you suggest? If I could still save at least something compared to that Supermicro and I would get some 2600 (I have my doubts 1600 (AF) would work…), that would boost the CPU performance a lot compared to my Intel alternative.

And then, what OS solution do you use? How does it play with AMD HW? Any issues?

@SgtAwesomesauce, fantastic, I will label them before putting them in the case. This way I can (almost) easily pair them.

And those drives…are all those shucked? Most of 8TB ones (especially white ones) are SMR and more I would have in my machine, more anti-depressive pills I would have to stock so when rebuilding occurs, I will not go insane that another one fails in the meantime. I don’t think shucking is worth doing anymore unless one builds an archival machine (on which some ultra-capacity archivation BRs may be more suitable anyway).

At the end of the day, I am just reaching 8TB, even 24TB will be plenty for me, and in the future when I need more storage we will have already much cheaper high-capacity SSDs which would result in pretty much noiseless NAS (I would put the same trust in a quality SATA SSD with TLC NAND as in PMR HDD).

@Airstripone, I see, RaidZ2 would work (well I would have to buy 8TB external HDD to copy all data to it, build the array and copy it back, but it would work), you mentioned that a single RaidZ2 would mean a simpler pool management, would you mind elaborating on that? Are there other reasons why to go with 1 RaidZ2 of 8 drives vs. 2x RaidZ1 of 4 drives each?

is there a particular HBA you would suggest (I don’t want to sound cheap, but…well, I would like not to pay 100€ for one (used on eB of course).

…
So to wrap up things I am wondering about now:

  • AMD build using Ryzen 2600 + Asrock Rack X470D4U2-2T (+ some HBA card, in such case, which is the least expensive option to plug in at least 8 SATA3 drives) or keep Intel build, but upgrading CPU to Xeon E3-1240L?
  • 2x RaidZ1 (4HDDx2) or 1x RaidZ2 (8HDD)?
  • does FreeNAS look really like the best option and there is no need to thinking about something else?
  • is there anything I am missing?
11

10Gb is still being refined for consumer use… with the 10Gb MB you cant upgrade that unless you get a PCIe card

the aquantia 10GbE isnt a X550 , but its certainly cheaper, and whatever replaces x550… wont be possible to update when its embedded .

But another argument can be made for when that actually happens, the MB will be obsolete anyway, but see the popularity of R710 dell servers…x58 era)

The 10Gb MB has more limited PCIe slots compared to the 1Gb version. I went with the 1Gb version, b/c adding 10Gb via pcie was cheaper than paying for the 10Gb MB version.

Its personal preference really.

12

The main difference is it is a one step build. Just add the 8disks and select 2 disk parity, it will do the rest. Your multi vdev option means building twice as many pools.

In the Z2 model you can lose any 2 disks from 8 and it will still run long enough to get your data off.

In the striped Z1 model, if you 2 disks in the same vdev that is it, you lose all 8 drives.

13

@Airstripone, I see, then RaidZ1x2 is actually a bit less secure as in such case you can have 2 failed drives, but only and only if each is in one Vdev, otherwise everything is lost. I haven’t thought about it that way. Albeit there seems to be a slight write speed advantage of RaidZ1x2 (source here), it hardly seems like a difference worth the risk. Thank you, I will go with RaidZ2, I think I will build it fully from new drives, copy data over and use one of my current ones as a spare and sell the other one. A bit more expensive upfront, but, well, life is for living.

Which basically completes the ZFS and allocation question once and for all and leaves only the last one on the table (I think I should not overthink the OS and FreeNAS will be the best for my needs (ZFS via GUI with the least of CLI work)).

AMD vs. Intel (FreeNAS as OS), what would you choose, AMD Ryzen 2600 + Asrock Rack X470D4U2-2T (+ some HBA card, in such case, which is the least expensive option to plug in at least 8 SATA3 drives), or Intel Xeon Xeon E3-1240L + current Supermicro X11SSH-CTF (more expensive, no need for additional HBA card)?

14

Glad you have resolved this. Just be sure to DBAN the drives before you sell them, or keep them as tier2 storage.

As for hardware, frankly it is a wash. I use intel only because it is used hardware and was cheap but gives me 64GB ddr3 ecc ram and 100% compatibility with freenas. I have an Intel x550 NIC and 2 LSI SAS controllers.

My production machine is threadripper and is better in every way, but ddr4 is expensive and I don’t need 24 threads on my NAS. If I were buying new I’d go amd for my NAS. I wouldn’t buy new though.

15

None of the 8TB disks in the WD EasyStores are SMR. Not sure where you got that from.

I’ve got 8 white-label shucked drives.

They are white-label reds or enterprise disks.

16

@Airstripone, so I am basically tossing a coin here. As for HBA, if I search “lsi hba card” on eBay, what should I be looking for? I see tons of unbranded cards from China, is there anything amongst those worth looking at (I am mostly interested where legit offers start, $30, $40?

@SgtAwesomesauce, you are right, some time ago I was digging through an unending amount of reddit threads looking for such information, but it seems 8TB+ models should still be white-labeled REDs, however there are strange information around new RED “EFAX” models, where it seems lower capacity models already have both, PMR and SMR versions (SMR being those models with 256MB of cache)


but probably the 8TB models are still PMR, or maybe not…they should be direct about this, this feels sometimes more like a ghost hunting.

17

I tend to have more luck just searching for sas controllers as you get more used listing and rebranded LSI controllers, like a Dell PERC. The last ones I got were Toshiba branded but same chipset. Do your research as there is usually extensive documentation for “good” brands. Anything you can’t find a data sheet for should be avoided. A quick search on eBay shows a lot of HP and Adaptec branded cards. Avoid these as driver support is less optimal. Dell seem to be looked on kindly for ease of flashing the bios to IT mode.

I don’t live in the US but a reasonable used card price for a 2port 6Gbps HBA would be the equivalent of $40 here, so if you see new ones listed for less than that… avoid. Also avoid anything where it is just a photo of a card in a plastic tray.

If you are less fussed about peak performance an older 3Gbps card can be hard for $10 or so. Good way to get started cheap, test the rest of your setup then upgrade to a faster HBA when you find one.

18

If I may to iterate on the topic, I am actually just MB+CPU+RAM away from building the NAS, however last three Xeon 1240L v5 offers (2x eBay, 1x Aliexpress, all from Chinatastic sellers) didn’t go through, all were “out of stock”.

And in general it still seems not a good idea to buy those from China (e.g. different SKUs with a replaced IHS etc.), so here I am again, is there something you would recommend me to focus on instead?

Bin hunting old Xeon back from v2 gen or so from US/EU sellers on eBay (absolutely no idea what to look for)?
Buying a very expensive board with an embedded CPU such as X11SDV-4C-TP8F with both 10GbE+HBA?

Maybe yolo it with Ryzen? HW might be less of an issue, whether

  • I would go with a consumer board which supports ECC (maybe I would fine some good with B450 chipset and at least Intel 1Gbit to have a connection to the rest of LAN) , I can much easier find one with a quality VRM,
    +++ 10GbE NIC whether some Intel X540-based ones (I see them for around $100 on eBay (US/EU) or taking a risk with a similarly priced Asus XG-C100F with Aquantia which would not be a problem with Linux (there is an un-official driver for FBSD as well)
    +++ HBA from eBay (US/EU), I see currently some Dell H200 pre-flashed to IT mode for around $60
  • Or I would go with one of Asrock Rack boards, they have three models for AM4, all look good, pricing varies, but it could end up being cheaper (with an extra HBA card as above) than that Supermicro board I was planning originally, but man oh man oh man, those are absolutely lacking here in EU, I will try to contact both manufacturer and Scan in UK as they seem to be only ones who have them at least listed.

And whether Ryzen R5 1600 AF (=2600), 2600 or 3600, I would decide by the price, 3600 went down on price nicely as well.

I am trying to dig up some updates on a current experience with Ryzen and FreeNAS, but I may need to consider alternatives. Linux-based OS could be the better match for this HW, but what in such case, Proxmox? OMV (less painful addition of ZFS)? Unraid (more painful addition of ZFS)? One thing I know I want to do is ZFS, preferably via GUI, as I mentioned at the beginning I have absolutely no problem working in CLI, but I also like an ease of GUI.

19

Any reason to go for the L CPUs? In my experience they are usually overpriced and don’t exactly save you enough on modern architectures to be worth it.
Hell, even on my ancient LGA771 dual CPU system going from an E5335 to an L5410 only saved ~32Watt accross two CPUs, so ~16Watt/CPU. Which is really nothing on a system that sucks down over 300Watt at idle…

I mean, my pfSense box (E3-1240 v2) pulls <50Watt from the wall. Saving a few Watts off of that isn’t worth the premium the L’s tend to command, at least in my experience (not even at the expensive EU electricity prices we have over here).

Also FCLGA1151 (used by the 1240L v5 you have there) is still pretty new so prices will be quite a bit higher as a result.

Personally I tend to look in the LGA1155 to the LGA2011 range for motherboards/CPUs. That seems to be the sweet spot power-consumption vs price currently.

Handy reference on the sockets out there.

Ofc the above is imho, ymmv etc. etc.

20

I expected a lower power consumption by such L-chip. Honestly said, prices were all over the place, I could not find any median in prices for any of models. It will be a machine running 24/7 and I don’t need (and want) something made for a much higher performance-demanding use case.

And what I meant by “absolutely no idea what to look for” when mentioning v2 Xeons and bin hunting is that, there are dozens of Xeons per generation and motherboards just as difficult to spot any difference between and hard to navigate through them.

I have no problems orienting in consumer components, those are a second nature for me, but these old generations of Intel parts and motherboards…ufff.

If someone wants a super low-priced rig with used components I can always pinpoint components at every generation I know or simply said - I know what to look for specifically.

Then I take a breath and I try to do it logically, I throw a simple search phrase “supermicro c602” on eBay and what I get is all dual-socket boards with proprietary form factors and by the time I stumble upon single-socket ATX board I am already at $320 price tag. Or when I try something more reasonable such as “supermicro c216”, then even if I am lucky I find boards with a support for up to 32GB od RAM.

But in all fairness, Xeons are easier to find, like a second of search and I got this $20 incl. shipping 6/12 Xeon v2, but finding a motherboard for anything even distantly close to a great deal is a completely different matter. Our local Supermicro reseller of course has a ton of such boards, but starting at $350, but eBay? A ghost town.