Finding a use for SMR drives in Truenas

Hello, First topic.

Looking for some advice and explanations.

I’ve been planning out my first big NAS. First relevant thing I learned was the difference between SMR and CMR. So I now have a stack of SMR drives from everything I’ve had before.

The next relevant thing I learned about was the possibility to “spin-down” disks and it’s benefits, but also that it would be pretty hard to do with Truenas.
I connected some dots and came up with the following.

I want to still use these old SMR drives up. So much so i will jinx it by saying “I’ve never had a hard drive failure in my whole life”. I am swimming in old HDDs…
Sadly all of which I am going to assume are SMR as a precaution.

So i’ve come up with 2 ideas:

  1. I could just make a bunch of mirrored setups (RAID 1) to turn “resilvering” into “recloning” which I think should be faster.

  2. The much more interesting idea…

Can I use the SMR HDDs as an L2 cache and save some of the lifetime of the CMR drives as a bonus?
I understand this is far from optimal, but is it possible?

As long as it doesn’t make the performance worse than having a single external drive would be, I would consider it acceptable for my use case. (It’s just a media library that doesn’t need to be fast, just bulk storage)

And attached to that some things that I’m curious about. Would that indeed allow an array to spin down?
It would allow me to use the SMR drives without any extra risk of data loss? Since it is just a cache of the actual storage.

I am able to actually test this in about a week once the CMR drives arrive. So unless advised against it for actual damages I’m willing to report some results if people can share the commands I need to run for testing.

Feel free to get technical with explanations, I’m looking to learn.

1 Like

I am also a new truenas user with my first box booted up 5 days ago. So keep that in mind I am in no way an authoritative source. That said, if all you care about is bulk storage for media, I don’t think SMR really is going to matter that much. I didn’t have piles of old/spares already, so I did make sure to buy CMR for mine, but based on what I found while researching I would have reused SMRs if I was in a similar situation to you.
I don’t see any reason why you couldn’t use them as L2 cache, but I’m not sure that would really buy you anything. In general the L2’s purpose is to be faster than the bulk storage for files most commonly accessed. In your case 1) it would be slower than your bulk and 2) a media vault/bulk backup box is unlikely to need repeated access to the same files that a cache would benefit. I don’t know the details of how truenas populates the L2 cache, so it might provide the “less wear” benefit you identified, but then it might also do the opposite if truenas decides to shift things in and out of the cache more often than you would just access the bulk drive normally.
I believe in any event, you’re pretty much guaranteed to be better off than external drives. That is also the motivation for me, having one fail finally kicked my ass into gear about a (more) proper storage setup.

I am unfamiliar with the spin-down situation you describe. There is an option to set spin-down of the disks within truenas, but I have not fully explored that yet. I had enabled it and then encountered a problem that may or may not have been related, so I turned it back off for now. I don’t see why spin down would be different for CMR vs SMR though.

I don’t understand the basis behind your resilvering vs recloning comment, I either did not get deep enough into my learning yet to encounter that, or I forgot (or ignored because I determined it would not affect my setup). Your post has given me some things to think about/look up.

I made a thread asking dumb noob questions when I started this journey, and then turned it into a sort of blog. Linked below if you’re interested, but before clicking BE WARNED it is very, very verbose and stream-of-conciousness. But I did that in case 1) anyone bored enough at work read it and chimed in something useful for me to learn and 2) it might help someone else later

1 Like

I haven’t used forums in a LONG time so i’m getting used again to condensing and sorting out relevant data (in my brain).

So to provide some more data from what you’ve said:

From what I understand there might not be storage problems between CMR and SMR. But incredible inefficiencies when having to rebuild from a lost drive in the array. So even with bulk storage in mind over speed. When it comes to redundancy it feels like CMR is the only real option.

The spin-down options would hopefully occur in my setup as a result from having those SMR drives act as cache and if it works like the RAM cache and only gets emptied/rewritten when it gets full the system does not need to access the CMR arrays frequently, even for a media server.

And the resilvering vs recloning point is because of the previous mentioned inefficiencies from having to rebuild an SMR array. Supposedly there is a lot happening to slow that down with SMR drives, but I (blindly) assume in a RAID 1 situation where, the “fixing” of an array would just be recloning the remaining drive to the new drive, that would be faster (since there is no calculating going on, just copying).

CMR vs SMR has to do with the way the data is “packed” onto the physical platters, not the data itself. It’s ENTIRELY possible I’m missing something, but I do not believe there is any data integrity concern between the two, just speed of reading/writing operations because the “shingled” data needs extra steps to unpack. I think of it like physical, hardware compression. Yes this would make operations on it inefficient, but I don’t see why there would be any difference with redundancy. I also don’t think mirroring the SMR drives will bypass that inefficiency. It’s at the physical storage layer, the data still has to get passed as the raw data AFAIK, meaning it will need to be read off the source drive by the system, which will make the drive firmware do its thing to unpack the data, then get written to the destination as the raw data, where THAT drive’s firmware will do its thing to write it in the SMR format. Since it’s about the actual physical layout of the bits, I don’t think there would be a way to just move the “image”. If there was ANY slight difference in how the two different drives interpret/store the end “image”, it would be unreadable.

As for the spin down thing, there isn’t really any reason you need the cache to do this unless you are keeping the NAS under basically constant use. If you aren’t accessing the drives for whatever set period of time you use, they should spin down regardless if they are cached or not. In theory. If you do have some use case that will hit the NAS frequently enough that it wouldn’t spend a lot of time spun down, then yes that could shift the load from the “good” CMR to the “bad” SMR. But I personally don’t think that’s going to be a significant difference in lifetime as long as the CMR drives you bought aren’t the cheapest consumer ones (WD blue, Seagate barracuda), but instead NAS or enterprise-grade (red/black, ironwolf/exos). I found the price difference small, or even better when I bought, at least for the high capacity stuff.

So the potential savings on the CMR’s lifetimes would be insignificant compared to the trouble of setting up an SMR drive as L2 cache, is what I’m getting from that.

As for mirroring not being faster than rebuilding a ZFS array I’m not sure I follow your explanations.

As far as I know, assuming a failed drive, the ZFS pool will actually ‘calculate’ the missing data. There is more going on than just hardware reading and writing, and it is so much that i’ve seen numbers thrown around on search engines that arrays with SMR drives can take 10x as long to rebuild.

If a disk fails in a Raid 1 setup all it needs to do is blindly copy everything over. Technically I could do it with an external cloner. (probably not, but as an example of how easy the operation is that we have external devices for it)
I don’t see how that would not be faster.

I will experiment and learn about spin down once I have the actual array built I suppose. I was not aware truenas actually had configurable settings for it. So I’m at least somewhat excited upon learning something new today.

So with a bunch of left over SMR drives you would just make another array to use them? Probably at least a separate pool for minimum safety?
Compared to making multiple Raid 1 setups with them.

In the end I have at least 4 2TB 3,5" drives, 2 1TB 2.5" drives a single 8TB 3.5" drive and a whole stack (around 6) of 500GB 2.5" ones.
I swear I’ll throw them out eventually, but for now that is too much capacity to just miss out on.

Maybe anyone knows other (novel) ways of potentially using SMR drives responsibly besides L2 cache?

Reviewing and expanding my understanding of SMR drives in a NAS setup it seems that archival storage is relatively safe if you can accept that “just get CMR” isn’t an option.

As media storage I plan to hopefully write once and then only read occasionally, so that fits the use case. Now I just wonder if TrueNAS or ZFS itself throws in wrenches.

Also I assume with the SMR shortcomings there is no difference between having 1 SMR drive in your array vs them all being SMR, or does the rebuilding time multiply even worse? Basically, ‘it’s only as bad as it’s weakest link’ type of scenario?

As it stands I’m considering using the 8TB drive as a L2 cache for the upcoming CMR array, just to see what that does, make another Z1 pool with the 4 2TB drives and very clearly label it for archival use only. And just setup multiple Mirrors for the other drives.

Sell all the SMR drives, get CMR instead. Trust me SMR doesn’t work with zfs, even with mirror.

I would not use a HDD for L2 Cache regardless of the type, SMR especially. L2 Cache is made for SSD’s and is only recommended as an option after you pass 64gb of ram in your system, at least in Truenas.

I would personally just use the SMR drives for cold storage or if for some reason you want to keep using them, your idea for a different pool that is basically never used day to day is fine.

Sell them? If you have no use for them and you don’t trust them surely someone might need them and you’ll recover some of the initial cost.

That is my belief, yes.

I see your point, I think I was on a different wavelength before. I was thinking of the write speed of the drive and the work its firmware has to do to put the data on the platter, not the parity calculations. Rebuilding a zX array is always going to be slower than copying a mirror for the reason you point out, CMR or SMR. I would have thought SMR would be an “additive” effect, adding the time it takes the firmware to do its thing to the whole process, rather than “mulitplying” (i.e. SMR are worse, but they’re the same amount of worse whether you mirror or Z-x), but the others who have chimed in appear to have experience with this and disagree.

FWIW I messed with this a tiny bit this weekend and either the drives didn’t actually spin down, or the power savings from doing so is so small it didn’t even register on the UPS. It’s entirely possible there’s other things truenas does in the background that hit the disks keeping them awake that I need to learn about, but I had powered down the PC and even the network gear so I could read the power usage of just the NAS.

That’s what I would do. If performance isn’t all that important, can still be bulk storage.

I didn’t realize you had such a vast mismash of drives. I don’t know your financial situation, but it’s going to be WAY simpler to just replace all those with a few large, modern CMR drives. If you want to use them just to use them instead of actually needing the space (which I’m fully behind BTW) you’re going to need to set up each different size as its own vdev. Unless you are willing to go no redundancy, that leaves a maximum of 32+ 1 + 5.5 = 9.5 TB total usable space.

I think so, yes. The array is always going to be “waiting” on the data that reads slower from the SMR. I wouldn’t mix CMR and SMR.

Experimentation, I love it! I’m in that stage myself, “fuck it, let’s see what this does”. Let us know how it works if you go through with it, real results always better than “in theory” :smiley:

1 Like

I wanted to suggest btrfs raid1, which I expect would mitigate SMR drawbacks (though SMR drives are not great in general), but apparently Truenas doesn’t support it

Yeah, except the 8TB one I wouldn’t bother. These are all low density drives that take up a lot of space and use a lot of power. The cost (oppurtunity + power use) to keep these is just not worth it.

If you have a use for something like a download drive, I’d use the 8TB one for that. Throw the rest out.

I definitely can see the power savings of spinning my drives down (2x16TB toshiba MG08). It’s about 7W each saved iirc. Probably something is keeping the drives spinning or they don’t respond to the ATA command as they should?

1 Like

This topic was automatically closed 273 days after the last reply. New replies are no longer allowed.