dRAID Reporting strange available size, can someone explain it like I'm 5 yo :-)

Operating Systems & Open Source Linux
,
1

Hi
I an currently testing with a NetApp D460C shelf (60 SAS drives).
The drives are 6TB drives, reported as 5.5TB (with lsblk)
I just need this space to move some data, but I wanted to test dRAID2
So I did something like this:
zpool create ztmp draid2:28d:60c:1s /dev/sda[m-z] /dev/sdb[a-z] /dev/sdc[a-t]
(and yes I know 28 disks is a large number, but lets ignore that for now)

pool: ztmp
state: ONLINE
config:

        NAME                   STATE     READ WRITE CKSUM
        ztmp                   ONLINE       0     0     0
          draid2:28d:60c:1s-0  ONLINE       0     0     0
            sdam               ONLINE       0     0     0
            sdan               ONLINE       0     0     0
            sdcs               ONLINE       0     0     0
            sdct               ONLINE       0     0     0
        spares
          draid2-0-0           AVAIL

(I have omitted the other 56 drives)

df -h and zfs list reports 172T free space.
While zpool list shows 322T

If I understand this correctly (may use other lingo)
So the zpool create command created a pool with two RAID6 sets (two parity disks) and one spare disk. So there should be 28+27 = 55 data disks left over.
They should be 5.5TB large which is 555.5 = 302TB
Now lsblk -b reports “6001165074432” which ends up at 5.589 GB if you device with 1024 (and yes we do).
So 555.589 = 307.395 GB

Now I think to remember that zpool list shows the capacity including the parity…
So we have to add 4 * 5.589 = 22.356 + 307.395 = 329.751GB

But how do zpool list come to 322T ?

And why does the df and zfs list report a much lower value?

Just to test a bit more, I tried creating a similar pool with this:
zpool create -f ztmp raidz2 /dev/sda[m-z] /dev/sdb[a-p] raidz2 /dev/sdb[q-z] /dev/sdc[a-s] spare /dev/sdct

And here we get df/zfs list values of 287T
and zpool list value of 322T

Now this makes a bit more sense to me…

But is is just that dRAID is strange like this?
And should I care? Can I still dump 287T onto the dRAID even though it reports 172T free? Will it “adjust” along the way?

I like everything else about dRAID but this size stuff… :wink:

And please don’t let this be about why I shouldn’t create such large RAID groups :wink: This is just an example to prove a point…

2

Do you know, that minimum allocation size for your pool is 112k?
If not, feel free to ask why.

ZFS list assumes 128k blocks for the free space calculation.
That is based on the assumption that you use the default record size of 128k.
I don’t know if that is a good assumption (I think it might be not hard enough, because you could have even smaller block) but it is what it is.

To write a block that is 128k (after compression or no compression) you dRAID would need to offer two 112k stripes. So you need 224k to store 128k.
Almost half of your storage wasted. You get only 57.1%.
That checks out with reporting 172T instead of 322T (again, almost half or 53%).

What data do you want to move? How big are these files?

If you set your record size to 1MB, the math changes.
Now you need to offer 10 times a 112k.
So to store 1024k, you need 1120k.
That is 91% and way better.

3

This thread is a good example, why pool geometry is important.
No matter if RAIDZ or dRAID.
It is also a good example of how more data disks or wider RAIDZ does not automatically equal more storage.

Let’s us make an example to proof this point.

You decided for to go for a draid2:28d:60c:1s.
Since you have 28d blocks and two parity blocks, this intuitively leads us to belive we get
28 data / ((28+2 parity) + (1 spare)) * 100 = 90.32%%
or simpler
28 / 31 * 100 = 90.32%
storage efficiency.

But that is not how dRAID works.
You need padding. As calculated in the previous post, because of padding your stripe efficiency is down to 57.1%. Combined with the “parity loss” (90.32%) from above, this brings down total efficiency to 51.6%.

Ohh boy, that is close to mirror storage efficiency :slight_smile:

Let us compare that with a more sane dRAID like draid2:16d:60c:1s.
16d * 4k = 64k stripe. To store a 128k block, we simply need two stripes.
No padding loss here.
16 data / ((16+2 parity) + (1 spare)) * 100 = 84.21%
or simpler
16 / 19 * 100 = 84.21%
storage efficiency.

TLDR: At first glance this less wide dRAID looked less efficient.
But the pool geometry is a perfect match. So no loss in padding. That leaves us with 84.21% efficiency while your wider dRAID only offers 51.6%.

Again, this is assuming that you only write 128k blocks. Your milage WILL vary, depending on what you write to it.

Now, there are ways (or a combination) to combat this issue.

  1. Mitigate the problem. You could add a special vdev that stores smaller blocks.
  2. Choose a pool geometry that better matches your needs.
  3. Use larger files and split them up in larger blocks by using a larger recordsize.
1 Like
4

You should not create such a large dRAID group! :slight_smile:
ZFS people don’t say that to be mean to you, but because making something wider has diminishing returns.

Ignoring padding and pool geometry for a moment, do you really gain that much by using 28d instead of 16d?
Is getting roughly 6% more storage in theory, worth the poorer performance of a wider dRAID?
It is worth the higher rebuilding times?
All that with storage being so cheap nowadays?

2 Likes
5

Great explanation :slight_smile:
And it makes sense.
And the penny finally dropped… while raidz has a dynamic recordsize, dRAID doesn’t. Which is the reason it can resilver faster… But it does have to do this twice… one and fast sequential resilver, and then the “normal” check of the checksums, kinda resilver…
I am “used” to working with NetApp storage systems which is somewhat similar to ZFS. But I guess NetApp have solved this padding issue and are able to optimize the stripes a bit better with “compaction” as they call it, where they take all the small writes and pools them into one stripe. So we just need some hacker to port NetApp’s WAFL filesystem to Linux :wink:

Anyway, on the production pool we will of cause have a metadata device to speed up things, and store files smaller than 128k (typically).

It would be nice if someone have created an online ZFS calculator where you could add these values to reach the most efficient pool.

We have another setup with 24 disks… and what would be the optimal setup here?
My first suggestion would be 2 x dRAID2 but even with that, we would end up with stripes of 40k. And we want a record size which is larger than 40k ?
Which leaves us with 64, 128, 256, 512 and 1024, none of which is 100% optimal.
Where 128 is the one with the least padding?
It quickly becomes a puzzle :wink:

A way to have less padding would be just one dRAID3 with 21 data disks and 84k stripes and a recordsize of 256…

But then we end up in the old discussion of one vdev and performance, which is another rabbit hole… we have a few single vdev setups that works just fine. Some thinks you only get the performance of one disk, which of cause isn’t true.

In my example I ended up creating a RAIDz2 with 4 vdevs of 15 disks each (before I read all this), mainly because of performance… I am currently using zfs send to migrate a volume to this pool… will the receiving pool dictate the recordsize? Or does it follow the sending volume? :slight_smile: Because the sending volume is using 1M while the receiving volume is 128k :slight_smile:
(my guess is that it follows the receiving volume)
And of cause I can see that 128k isn’t ideal for this pool with 13d stripe, a better value would have been 256k… but I guess it’s not a good idea to fiddle with this while we are transfering? (this is just temp-storage and will be moved again)

6

Fund this calculator which is a bit “dare I say it” too complicated :wink:

https://jro.io/capacity/

1 Like
7

NO! :slight_smile:

RAIDZ has a dynamic stripe size, while dRAID has a minimum allocation size.
Record size is stil a variable max upper limit.
If your recordsize is 128k you can still get a 12k stripe or write.
volblocksize (zvol) on the other hand isn’t a max value but a static value.
By default this is 16k. But you won’t get only 16k writes there either, since it might be compressable down to 9k.

It is multiple things, the rebuild (I think that is how they call it for dRAID, not resilver) being sequential and also the spare being hot.

Just be aware of a mistake I recently did. If you have a 1,5GB whatever big mp4 file, and a 1MB recordsize, it might be the case that you can split it up into many 1MB chunks and then have a 990k tail. If you set the recordsize to 1MB and s vdev also to 1MB, all these tails will end up on the s vdev and you use way more storage than anticipated.

Depends on what you want and what your use case is.

Easy, do 128k / 4k = 32.
So it would be optimal for 32 data disks, or 16 or 8 or 4 or 2…

Yeah, I haven’t found any good information on that problem.
Is the IOPS maybe close to one disk, but not sequential writes?
I have no idea. Would be interesting if you could run some test on that.

That is pretty wide. How long was the resilver with that setup?

Not 100% sure on that one, since I set it manually on both. Do you use TrueNAS? There are multiple options like “Allow blocks larger than 128k” and “Include Dataset Properties”.

This really depends on the data.
What data do you have?

1MB is a save bet, since it is compatible with even older ZFS versions.
16MB is pretty new.
One could argue, that since this is only a max value, 16MB is the best setting. Best compression, geometry and so on.
Others argue that 16MB is maybe, potentially worse from a performance standpoint because CPU, PCI, bridge stuff I don’t really understand. Or scrubbing a file in final cut, you need to read the full 16MB even if you only want 1MB.

I really want to hammer this point home, you do understand that record size is just a max value?
You could have record size of 16MB for a dataset and fill that dataset with 4k writes and you would only have 4k stripes. So just because you set the record size to 128k, does not mean that you only get 128k writes.

Some more examples to make this clear.
If you have a 1MB file and a 128k recordsize, the 128k is the upper limit.
So your 1MB file will be split up into multiple 128k stripes.

No what I had a pretty hard time understanding is that compression happens AFTER the records.
And that while the stripe size is variable in general, for a single file it is NOT.

Again an example to make this more clear:

So lets assume you have 128k recordsize and a 159k file.
What does ZFS do now?
One 128 stripe is not enough.
And since the stripe size is the same for a single file, we need two times a 128k stripe.
That seems pretty bad right?
We need 256k just to store 148k?
ZFS will write the data into two 128k stripes, but the second one will mostly consist of zeros. We have 128k real data in the first stripe, 20k real data in the second stripe and then filled with 0000000000.

No thanks to compression, these 0 can basically be compressed to, well 0 :slight_smile:
So we don’t wast a lot of storage but only use roughly 148k to store 148k.

That is where the recommendation

never disable compression (unless you really know what you are doing)

comes from.

volblocksize or zvol or iSCSI on the other hand, really is a static value.
If you set it to 16k, the guest gets served 16k all the time.
But that also does not mean that you only get 16k writes, again because of compression!
That 16k block might be compressable to 13k.
Keep that in mind when thinking about pool geometry. You probably don’t know how big your writes are.

1 Like
8

I only have datasets with files on there that is mostly larger than 1G.
Compression and dedup is disabled on selected datasets where it doen’s make sense.
I ran the “zdb -Lbbbs” on a dataset to figure this out, and set the special device and “special_small_blocks” to 128K. so most smaller files land there…

The recordsize is a max value, got it :slight_smile: It’s the min. allocation size that is fixed and this only applies to dRAID.
The more I learn about dRAID the less attractive it gets :wink:
It seems like the normal RAIDZ is more optimised and store the data more effective. I don’t like the idea that my disks will have an unknown amount of “padding” stored on there which is dependent on whatever filesize I happen to write… Or maybe I have also not understood this 100% :wink:
Or is ZFS able to store multiple records/files in one stripe? So if you had a stripesize of 40k and wanted to write two files of 10k, would it fit that into one stripe? or waste two stripes? And when you then throw compression in the mix, or even dedupe, it becomes complicated.

Now since I already have a pretty big dataset on a dRAID3 pool, is there a way to calculate how much “padding” has been wasted? Maybe with zdb ?

9

So 16MB recordsize is probably a good setting for you.

Compression should always be enabled. LZ0 costs nothing and it can compress zeros you might have.

Dedup almost never makes sense.

But why? For your use case, it is pretty much ideal. I would not even bother with saving small files in s vdev.

Just use a draid2:16d:60c:1s and 16MB recordsize and you will get a sequential read and write blazing fast pool for your big files that can rebuild very fast. If you care about metadata performance (list directories) you can throw in 3 cheap and small (256GB) SSDs in a 3 way mirror as special vdev.

Not really. Since you have mostly large files, the pool geometry problem of RAIDZ and the min alloc problem of dRAID can mostly be ignored.

Since you should not go over 12 drives, for your 60 drives that would be 5 striped 12 wide RAIDZ2 which gives you 83% while the draid would give you 88% while offering better faster rebuilds than RAIDZ has resilvers.

You did not understand it 100% since you assume that this problem only applies to dRAID but not to RAIDZ :wink:

If you are interested in the math behind it you can read up on it here. But like I said, for a 16MB recordsize and mostly large files, this isn’t an issue to begin with.

No, it is not.

Other way round. Forget everything.
We make it simpler. You wan’t to write a 10k file.

Here is how that works on a RAIDZ2:

For that 10k file, you need to offer a 16k record (has to be a multiple of 4k).

That 16k record is only zeros after your 10k data and can be compressed to 10k.
Side note, this is the reason why you always want compression enabled! Even if your data is not compressable!

For RAIDZ to store a 10k write, it needs to provide 3 times a 4k data sector.
Sector size is 4k for almost all drives.

Now since you also want to parity sectors for that (so that 2 drives can fail), you also need two parity sectors.

So in total you stripe has 3 data sectors and 2 parity sectors, all 4k each because that is what your drives use.

This is in total 5 sectors.

But there is a problem. Padding. Each stripe in a RAIDZ has to be a multiple of (redudancy + 1).
Why? Because otherwise you would get empty spaces that can’t be used no more, because they are too small. It is described in the link I provided above.

We have a RAIDZ2 so (redudancy + 1) = 3.
So our sector has to be a multiple of 3.
We have 5 sectors.
5 isn’t a multiple of 3.

Damn. We add a padding sector.
Now we have 6 sectors in total
6 is a multiple of 3.

Hurray, that works. We needed 6 sectors in total for our 10k file.
6 sectors each 4k is 24k in total.

We needed 24k to store a 10k file.
41% storage efficiency!
That is the downside of ZFS over traditional RAID.

The mean part? You won’t see this directly. If you write your datasets full of 10k files, you are wondering why ZFS shows you more storage used than what Windows Explorer is telling you.

Here is how that works on a dRAID:

For that 10k file, you need to offer a 16k record (has to be a multiple of 4k).

That 16k record is only zeros after your 10k data and can be compressed to 10k. Side note, this is why you always wan’t compression enabled!

Since we have a min alloc of 16 x 4k, we will hand out a 64k data stripe. That stripe also has two parity, so in total we have a 72k stripe to store 10k.
Storage efficiency of 13%.

I am not sure to be honest.
Compare what Windows or Mac tells you, your dataset and what ZFS pool usage is.
Again, for your large files, the loss will be negligible.

1 Like
10

OK, thanks again… :slight_smile:
I am not actually going to use the 60 disk shelf as it is a temp setup with 6TB disks. The pool I am zfs send’ing over there now has to be send back again after I have reconfigured the pool… I have 24 x 18TB disks that I have to figure out the best setup for… and I would like to make it dRAID with a spare, just not sure if it should be one large dRAID3:20d:1s:24c or dRAID2:10d:1s:24c and that’s just it… the last example makes no sense in my head, because I am thinking of RAID groups of 10 data + 2 parity and 9 data + 2 parity… but as far as I can understand this is not actually how you should look a it because data is written as “tiles” split up kinda random on the disks… (I saw an older video from OpenZFS on this)…

Also while we are at it… we have the 24 x 18T diske, but we actually also have 12 x 20TB on the same system… would it make sense to add this to the same dRAID pool? This is possible with RAIDz with say 3 raid groups… and yet I know that the groups will be skewed a bit, but it would make other things easier to manage… This isn’t a high performance setup anyway :wink:

11

One additional thing I need to figure out… since we have datasets where compression is disabled… can I create a new dataset, enable compression, and then zfs send a volume that doesn’t have compression enabled to it? Won’t it just “overwrite” the volume setting?
Because then I need to use rsync instead…

12

Not going to lie… I’m a bit annoyed about ZFS now :slight_smile:
I am currently using zfs send to migrate a dataset across to my marge 60 disk pool… I created a snapshot which states that it is referencing 240TB…
Now with zfs send it now states that 264TB has been moved…
zfs list matches the 240TB if the snapshot, abd zfs list on the destination shows 264TB so there should be space right? well we are now down to 5TB avail and still going, and of cause of it would run full I need to start over… nice… I really hope this pans out positive… Maybe I should just start using md-raid, where you can count on your sizes :wink: (or maybe not)

13

Yeah… of cause it filled up :wink: So I nuked it and created the draid2:d16:c60…
And added a special device also to catch the small files…
And since I cannot seem to find out what will get transfered with a zfs send, I opted to use rclone… and it looks like it is faster than zfs send (copying 4 files at a time)… but it will be a few days before I can start re-creating the 24 drive draid

1 Like
14

Best setup for what? Sequential read and writes of huge files?

I think you got it right. The dRAID numbers don’t have to line up with the number of disks, because the dRAID is basically a stripe. Strange to describe, lets make a real example again, that is easier.

[quote=“beardmann, post:10, topic:236791”]
Also while we are at it… we have the 24 x 18T diske, but we actually also have 12 x 20TB on the same system… would it make sense to add this to the same dRAID pool?
[/quote] I think they will be used as 18TB drives. Just like in RAIDZ. But not sure on that one.

I don’t think so, but again not 100% sure.

Not enough information of what you are doing here. But in general, Snapshots, pool geometry, record size and volblocksize make these “free space” calculations a little bit complicated.
I would not say that this is ZFS fault though :slight_smile:

With what drives? I thought you have 30 drives?

You are again giving very little information, so I just have to guess and make assumptions :wink:
So here is my heads up:

  • s vdev is critical for you pool. If you loose s vdev, you loose everything. So when you say you added a device, I hope you mean you added multiple devices and run them in a 3 way mirror
  • s vdev does not catch small files by default, only metadata
15

A few answers :slight_smile:
The filesystems I have are mainly large files of 1GB and up to 200G…

My goal is to empty the existing draid3 that I have with the 24 18TB drives onto the 60 x 6TB disks that I have connected as a temp store, and where I am currently running them as a draid2:16d:60c:0s and I have attached 3 x NVMe drives as special device in a mirror just as I have on the main draid3 I am copying from… but because the sizes of the existing pool is so close to the temp pool, for this try I am using rclone instead of zfs send, the difference beeing that if it should run full I can still use the copied data and move some folders to another storage device that I have… I think it’s a bit sad that the difference in sizes can be so great dependant of a simple “mistake” in the configuration. I understand why this happens (or I think so) :wink: I guess the “fix” would be to allow variable record sizes… As I mentioned I have been working with NetApp storage for a long time, I am pretty sure they are able to mitigate this issue, but of cause you are capable of creating some not so performant configurations like what would be a raid3 with 29 disks in it :slight_smile: (which is maximum)… but it will work, and you get the space that the system reports… NetApp beeing NetApp you do loose some space with their “rightsizing” which is because they need to be able to use different vendors of say 20TB disks, and they rarely have the same number of blocks between the vendors… so they “shave the size down a bit”… :slight_smile: Anyway it looks like the rclone will be done in about 30 hours, so quite a bit faster than zfs send…

Before I build the final pool of the 24x18TB disks, I will update Ubuntu from 22.04 to 24.04 so that I get a but newer ZFS version which I hope allows for a max record size of 16MB instead of the 1MB which is the limit on this ZFS version 2.1.5…

PS: If you want me to perform some test on the 60 x 6TB SAS drives, let me know… Once I have completed the migration I will have to power them down as they eat way too much power to have powered on for “fun” :wink:

1 Like
16

Sad or math, call it whatever you want :slight_smile:

For dRAID? Getting rid of min alloc size? Yes, that would be a fix that breaks the only advantage of dRAID over RAIDZ, which is fast rebuilds.

I have no idea but I doubt that it is possible to fix this “problem” without suffering other even bigger drawbacks.

TrueNAS (or ZFS?) does the same btw.

Really recommend TrueNAS scale, makes life a whole lot easier.

17

I have been looking at TrueNAS Scale for some time… but I have several other scripts and things all arround which just needs some work unless you can just add your own stuff on top of TrueNAS?
One example would be that I use the vdev_id.conf so that the disks are shown as they are installed in the shelf, so my disks are called 01-00 → 01-12 for the first shelf, then 02.00 → 02-12 etc… 01 = HBA Port and 01-00 = Disk in slot 0 of the shelf. It just makes so much more sense than using the WWID or SN or whatever :slight_smile: And to my knowledge this isn’t implemented in TrueNAS yet… (so much for enterprise)… I guess you could just create your own and re-import the pools, but still…

Also we have 4 GPU’s in this server setup with one VM each assigned one GPU setup for Tdarr (video re-encoding)…
But of cause it could be moved to TrueNAS… just until I find something that is not able to work or works different in TrueNAS compared to Ubuntu… then we would be screwed I guess :wink: But the large data pools are of cause fairly easy to migrate from one to the other…

Just an example of rightsizing from NetApp :slight_smile:
NetApp sells their disks as the vendor reports them, say 8TB disk.
Then NetApp puts their firmware on it, not sure if they set aside more blocks than the vendor does. If you look a but deeper the physical size is 7.501.160MB which is rightsized to 7.486.157MB so 15.003MB is “lost” per disk. So with a shelf with 60 disks, you can do the math :wink:

And NetApp must be doing something right, because even with a RAID Group of the maximum 29 x 20TB disks a rebuild will typically take anywhere from 2-4 days. It can of cause vary dependant on load the priority. But try that with a zRAID3 pool and keep in mind that it has to be filled up, because AFAIK NetApp also rebuilds the empty space.

Anyway, you triggered be to install TrueNAS in a VM just to see if any of my earlier nags have been fixed :wink:

1 Like
18

OK the copy has completed… and I am just about ready to create the pool for the final location… on the 24 x 18TB disks…
My idea is either:

draid2:16d:24c:1s
or
draid3:16d:24c:1s

I will also add a 3way mirror for a special device.
The max. recordsize will be 16M

The 16d should give me 4k x 16d = 64k stripes. My max. record size will me 16MB.

But I just want to make sure I understand this correctly…

When we are writing to the disks, we will write a 64k stripe on either of the examples…
But we have 21 or 20 data “disks”, which doesn’t go nice with the 16d…
Is this because it should not really be thought as disks, but rather what they call “tiles” ? And dRAID then makes sure that whenever we write the 64k stripe, it is written over 16d+2p or 3p ?

Would a draid2:16d:19c:1s be more efficient than draid2:16d:24c:1s because the “numbers add up”, or does it not matter because of the “tiles” ?

19

It is just Debian. I would generally recommend not to tinker with a storage system.

I would also recommend not mixing storage and applications :slight_smile:
For GPU involved taks, I would even recommend bare metal.

For dRAID the stripe width is fixed. So yes, both would write 64k data stripes. The parity level would be different.

Hmm… why 21 or 20?
You have 16 data disks, hence the 16d in both of them. For the dRAID2, there will also be two party, so more like 18 “disks”?

Doesn’t matter.

Exactly. I think they call it “redundancy groups”.

Does not matter because of redundancy groups.

20

OK I am done moving data around for now :slight_smile:
I ended up with the dRAID3:16d:24c:1s + special medatadata device… compression+recordsize=16M…
I learned that zfs send/receive is a painfully slow way to migrate data if you have one large dataset as we did… so we ended up using rclone which is 2-4 times faster :slight_smile: We used rclone mainly because we were unsure if zfs reveive overwrite the dataset settings like the recordsize etc… so by just copying the files over, we made sure that it was working as intended… only downsize it that all folders are set to the current date… I am not sure if that can be fixed, but I doubt it… we used rclone with the “sync” option…

1 Like