NAS ZFS migration

Hardware
1

Hello all, I’m getting a new NAS in (ebay retired enterprise rack mount goodness!) and am not strong enough on ZFS to be confident in the following:
New NAS is a 12 bay, will populate 8 bays with 2TB HDDs, build a vdev and then pool, then RAID Z2. I then want to migrate data from my currently in use x5 2TB Z1 build over to the new NAS (so also build out a dataset and SMB share). I then want to take 4 of those old 2TB HDDs from the outgoing old NAS and fill the remaining 4 slots remaining on the new NAS, make a zdev and add it to the x8 2TB raid z2, making it a x12 2TB HDD raid z2.

From what I understand after building the x8 HDD pool and transferring the data over, then sliding in the 4 additional HDDs it would be --Storage -> Volume Manager, select existing pool under Volume to expand, select all the new disks, select RAIDZ2, Extend Volume.

Is this correct? Is it less than ideal (how data will be spread across the zdevs and pool)?

2

I don’t have time right this minute to fully parse what you are trying to do, I’ll have a look at it late tonight. Do NOT try to proceed, skimming your post is throwing some red flags.

Also, last I checked true raidz expansion (increasing the number of disks in the relevant vdev’s) was still a work in progress (though it could have been implemented without my noticing)
The alternative of adding a 3rd vdev (which is what I suspect that “Extend volume” does and what you are wanting to do will result in an unbalanced pool, which isn’t actually that big a deal, but not desirable either if avoidable.

Some additional questions
How much total data do you have to store?
How much are you expending to have to store in the next 1-2 years?
How many vdev’s and drives per vdev are you wanting to end up with?
What OS are you using, seems like freeNAS? All my ZFS experience is with ZoL and the commandline.
What is your backup situation? If you don’t have backups, how willing are you to lose your data (is it easily replaceable).

1 Like
3

My strong point is not in describing concepts, for sure. Looking forward to your analysis later.

I’m getting my info from the FreeNAS forums (answering Q 4)

One of my concerns for sure, I want to follow best practice here.

  • Right now about 4TB, if push comes to shove I should be able to move it off onto various sized externals to do this migration cleanly
  • I do not see a huge need for growth, this new array will give me 20TB at Raid z2, I expect to have about 16TB of data by the end of the year, but a large fall off in needed capacity after that.
  • I don’t know, not sure what best practice should be IRT vdevs, pools, datasets etc should be done at 20TB spread across 12 drives.
    • I want to offer iscsi for some VMs/VMware datastore, and for the majority of the storage used for a number of datasets, likely all shared/accessed via SMB.
  • FreeNAS 11.x
  • I wouldn’t even call it a backup solution, but I have the majority of the data found on the current FreeNAS on an external 5TB drive. I want to up my game in this department thus this purchase of the equipment in question. The data is not critical in nature- I would be peeved to loose it, but not devastated.
4

Is Raidz expansion active yet?
I didn’t think it was yet, and lookng at the freenas docs, it says “not possible” but doesn’t mention the big YET in the room?

The current extention [<-edit] is for adding another vdev to an existing pool, which should be the same size as the current vdevs, unless I’m reading it wrong?

In which case, maybe move the existing drives into the new rig, then add and additional 5 drives, configure them to a raidz, and add that raidz to the pool, then freenas will balance the data?

would only end up with two raidz (1 drive tolerance per vdev, rather than 2 per pool) or set up a large raidz2, and transfer the data over ethernet, and wait for the fetaure to hit freenas?

5

Another question is, what is the form factor for this disks? Are they 2.5" (laptop size drives), or 3.5" full size drives?
What size bays in the NAS?
What price did you get these drives for?

You may want to check your math on the storage you’ll have available. According to this calculator: https://wintelguy.com/zfs-calc.pl
1 vdevs, 12 disks raidz2 per vdev = 16.0TB (NOT RECOMMENDED, except as a backup that’s offline most of the time.)
1 vdevs, 12 disks raidz3 per vdev = 15.3TB (Okay as a backup, not as a primary storage system)
2 vdevs, 6 disks raidz2 per vdev = 14.0TB
3 vdevs, 4 disks raidz2 per vdev = 10.2TB

ZFS can still do fine past 80% capacity, but that’s the time you definitely need to look into how to expand. Once you get to 95% capacituy (assuming a balanced array) you see a severe degredation in performance.
Also keep in mind that the use of snapshots (HIGHLY RECOMMENDED) uses space as well, depending on how much gets changed. It’s basically free if nothing is deleted, but if a lot is deleted it acts like an unemptied recycle bin until the snapshot holding the data is deleted.

Having more than 8 disks in a vdev is generally discouraged. Disks of the same age tend to fail all about the same time, especially when stressed during rebuilds.

Unfortunately, with your estimated data storage needs, and current disk sizes, there are not any good options that don’t involve moving to 6,8, or 10TB drives.

Assuming you live in the US, thanks to sales of easystores and WD essentials, you can periodically find the following costs per TB if you are willing to “shuck” drives

6TB for 100 dollars = 16.6 dollars/TB
8TB for 140 dollars = 17.5 dollars/TB
10TB for 160 dollars = 16 dollars/TB (10TB drives are also guaranteed to be helium drives, and essential come from the same faucet as enterprise drives)

Another thing is to consider the power requirements of each disk running all the time, and the space each disk takes up ( which effect future adjustments).

Additionally:

  1. How slow are you willing to tolerate access to storage.
  2. Can your home network support the transfer speeds you want to achieve? You’ll find the 1G networks fucking suck, but 10G SFP+ networks are awesome and surprisingly cheap due to pulled hardware on ebay.
  3. Are you wanting to run these vm’s FROM your zfs storage?
6

By the way I encourage anyone else to speak up if you disagree with anything I say. Everything I “know” is from my own self taught hobby/experience, so if you have a better way of going about things, I’ll love to hear it.

7

My pool hangs sometimes and it’s at 92% prob gonna make my new pool and vdev for it tomorrow 6x8tb rz2

2 Likes
8

awwww, read so much better before the edit…

2 Likes
9

I’m safe for a little while yet with my 4 vdevs, 10TB mirrored pairs. This is a bit much for mirrors, but I have good constant backups and the performance is fantastic. But eventually I’ll have to move to raidz2-3 for bulk storage, and move active use items to a smaller array.

Haven’t decided what I’ll do for bays yet, as my max limit is currently an old 8 bay itx case.

10

Yeah not sure what I’m gonna do with my old disks (8x3tb) thinking about bunch if mirrors for performance storage pool for VM bulk crap but idk don’t really need the extra storage after new pool/vdev

11

This should answer the above:

A quick google:
“For raidz2, do not use less than 6 disks, nor more than 10 disks in each vdev (8 is a typical average).”
So with your advice and that, looks like best practice will be 2 vdevs of x6 disks, raidz2, resulting in 14TB of usable storage.

My expected data storage requirement is flexible, with 6TB of the data hardly ever needed to be called on so could maybe reside on external media.

Nothing crazy here, honestly should have just gotten a synology, but I like to take the hard way lol. House is on 1g, so servicing devices in the house is limited to that (and thats fine, I’m not remotely storing 4k vids / am not a content creator or data power user).

For #2 and 3, very good questions as thats the other part I have been googling about- for my use-case is 1G iscsi good enough ,or should I be looking into ebay 10G (pci card for the NAS, switch, and for whatever ESX server I get later). I do plan on running VMware datastore from FreeNAS storage, these are low I/O VMs, this is just a house. The VMs would be a Plex server, Splunk indexer, search head etc (for a very small amount of logs) *edit also the next cloud plugin or Linux VM as I’ve read the instability of freenas is with the plugins. I was thinking of multipathing as a band-aid, and to first try 1G just to see how it performs.

12

Yeah I’m thinking the best way to go is definitely 2 vdevs, six disks in raidz2. The redundancy there is very solid, just make sure you set up email alerts The 2TB disks are not ideal, but that’s a good motherboard and chassis.

Unfortunately, the best way forward is to do this is just as you mentioned, offload the data to your other disks and then cleanly set up all the disks at once. FYI, the 8TB WD element drives are $140 at amazon right now in case you’re interested.

So with that out of the way, I think you are good to go.

Here’s some other things to note:

  1. Make sure disks get added by-ID, not sbX labels. The labels are often handed out by order that the disks are found, meaning if a disk is added, removed, or slow, it could result in your array getting completely fucked up.
  2. Set up email alerts. Manually trying to periodically check for disk error status is a sure way to forget to do it.
  3. Read through this handy list of performance tips. Chances are you won’t have to do anything.
  4. Set up snapshots. I use sanoid/syncoid to manage mine, but I thing freenas has it’s own system. This lets you near instantly roll back to a previous state if you fuck something up. Also you can mound snapshots, and check/copy stuff individually.
  5. Set up automated scrubs. Every 2-4 weeks is good, I do 4 every weeks.
2 Likes
13

FreeNAS takes care of a lot of this for you

14

Thanks much @Log! Yeah it’s been very tempting to get larger disks, in fact the near purchase of 6tb hdds is what spawned this whole thing, but with the deal coming with x8TB drives plus the ones I am already sitting on, I can’t help but go with 2tb for now.

15

I have a thing for used Enterprise stuff, so if I need to go 10G, looking at these cards because IMO the VMware kernel (and FreeNAS) should have good drivers for them.

Then knock offs strait from China with included spyware:

What are level1’ers opinions on cache / L2ARC? Official documentation says RAM (if ample RAM) is primary and much better. Sounds like L2ARC is only beneficial if you have a lot of users accessing the same stuff.

1 Like
16

Basically, spend your money on ram

2 Likes
17

I have had fantastic success with my mix of mellanox cards. Dual port connectX-2 and single port connectX-3. I’ve never had any issues with them. From what I’ve seen, people that do just update the firmware and are fine. If you can, get both bracket heights, which will give you more options in the future.
I run an ubiquiti edgeswitch Ubiquiti ES-16-XG, however mikrotik has come out with some very affordable sfp+ switch options that deserve a look.

For transcievers I get Finisar FTLX8571D3BCL SFP+SR/SW 10Gb/s 850nm Multimode SFP+ Transceiver

For fiber cable I use cable matters

Slog(sometimes called a ZIL) can be useful when dealing with lots synchronous writes. I’ve seen some people find it very useful, and others have it do nothing for them. It’s not a big deal to try it out if you have some old tiny SSD’s laying around. These only need to be big enough to store several seconds of writes. Always mirror these. https://freenas.org/blog/zfs-zil-and-slog-demystified/

L2ARC is only useful when you are dealing with enterprise workloads, have an enterprise budget. It’s trading super fast ram caching (which is all you need or want), in exchange for using that ram to index a very large (but orders of magnitude slower) ssd cache. Basically what feqlabs said.

18

The advise by everyone has been greatly appreciated.

I’m thinking of just doing bridge mode 10g to 10g FreeNAS to VMWare and 1g the rest.

1 Like
19

Can you elaborate?

20

FreeNAS should take care of this automatically via the GUI, but it’s something good to keep in mind if you ever use the command line. Also, I misstyped, I meant sdX
So labels like sda, sdb, sdc, etc are what you don’t want to use. I’m not really sure what the proper term for them is.

On linux, the command “$ ls -l /dev/disk/by-id/” should give a bunch of results like (example is just the output spam from a single disk):

lrwxrwxrwx 1 root root 10 May 27 23:31 ata-WDC_WD2500BEVT-22ZCT0_WD-WXE908VF0470 → …/…/sda
lrwxrwxrwx 1 root root 10 May 27 23:31 ata-WDC_WD2500BEVT-22ZCT0_WD-WXE908VF0470-part1 → …/…/sda1
lrwxrwxrwx 1 root root 10 May 27 23:31 ata-WDC_WD2500BEVT-22ZCT0_WD-WXE908VF0470-part2 → …/…/sda2
lrwxrwxrwx 1 root root 10 May 27 23:31 ata-WDC_WD2500BEVT-22ZCT0_WD-WXE908VF0470-part3 → …/…/sda3
lrwxrwxrwx 1 root root 10 May 27 23:31 ata-WDC_WD2500BEVT-22ZCT0_WD-WXE908VF0470-part4 → …/…/sda4

The first “ata-WDC_WD2500BEVT-22ZCT0_WD-WXE908VF0470” is the actual ID of the disk that you are interested in, ignore the entries with “-partX”, which I think point to the partitions. With ZFS you always want to point to the whole disk.
edit Use ls -l /dev/disk/by-id | grep -v part to remove the partitions
I do not know what the freebas analog of this is.

So when manually creating a two disk mirror called “kpool” you’d use

zpool create kpool mirror /dev/disk/by-id/ata-WDC_WD2500BEVT-22ZCT0_WD-WXE908VF0470 /dev/disk/by-id/ata-WDC_WD2500BEVT-22ZCT0_WD-WXE908VF0471

Something like

zpool create kpool mirror /dev/sda /dev/sdb

would be incorrect. It would work fine, possibly for years, until something causes the a different device to become associated with sda or sdb, such as adding or removing a device, or an an issue with a disk having a delayed startup that cause you device order to get out of whack. I think freebsd uses a different three letter label than linux’s sdX, and probably has different sorts of checks and assumptions involved. ZFS does have some ability to look at the disks it’s given and use logic to use them correctly, but it can’t handle every situation.

This is also something good to keep in mind if you ever move the array to another system.

1 Like