Before you say I shouldn’t do it, just try to help me solve this issue. So I have a USB Enclosure running ZFS in truenas.
I run 24 drives with onboard sata and addin card for sata. I am officially out of pcie lanes and would rather run just one truenas build instead of maintaining another build for 5 drives. So I use a 5 bay usb 3.2 drive enclosure.
Anyways, on data scrubs when plugged into the 3.2 port it eventually (occasionally) throws zfs error (many 40k or so) when using USB 3.0 I see no errors on a scrub. I am guessing it is a usb error on the motherboard. It didn’t throw errors on another computer I ran scrubbing with.
Wendell mentioned he has done this and it can throw errors and disconnect when USB actually has a kernel error. His solution is to just reboot. I would love the added speed of USB 3.2 but what are my options to curtail the USB 3.2 from having issues? Would I be better off running scrubs on 3.0 and the other days not scrubbing on 3.2. I scrub once a month.
I don’t have a bios update available, neither is there an update for truenas scale (running bleeding edge version).
The purpose of ZFS is data integrity, in the enterprise space
data integrity > everything
Bigger drives
You are advised to replace drives after 3 years and I doubt you have 500+ TB of raw storage which is what you would have with modern drives.
The electricity usage alone would offset the drive cost.
I’ve replaced disk shelves of spinning rust with NVME drives for multiple clients and their reduced power usage alone paid for the new server including drives in less than 2 years. The massive performance increase was a freebie.
How is this helpful? Please try to leave helpful comment. Not deter me entirely from using zfs. Plenty of home labbers use truenas now and plenty dont have more drives. My drives are 20TB each and I have 250tb of storage used. I have not had any failed drives yet. Please either be helpful or frig off.
Linus of LMG/LTT has investing in a fork of truenas for mainstream users. And yes it will ikely use ZFS.
Recommending larger drives was an attempt to help you reconcile this mess. If you can afford $6000+ on 24x 20TB drives plus 5x more in a USB enclosure, you can allocate a bit more for a disk shelf or even a server with enough bays to properly house all those drives.
Honestly, you should try some disaster recovery simulations before utilizing the array. And perhaps ZFS is not for you. ZFS is for people that care enough about their data to make concessions including speed and $$$ in the form of hardware costs and utilization.
With data you have: Good, Fast, Cheap
Pick 2
If you want fast over USB, then use a FS without journaling.
do you know what the errors are? Do you have logs? Maybe there are also filesystem errors that precede it.
I’ve had errors in my zfs array from power saving features that don’t work correctly. after turning that off it worked fine.
maybe a long shot, but do the errors maybe happen because of the power supply not handling it. Higher transfer speeds might cause the drives to use more power, more than the enclosure can give. peak power on those large drives is a lot!
For me if 8tb nvme is 1000 dollars and I need significantly more than what I have in terms of hdds. It is not worth it. I have a supplier that sold me 20tb drives for 200bucks, ran stress testing and they haven’t failed after 2 years. I will likely upgrade to higher capacity as I replace the drives. But ssds after 1tb are expensive. As in 1tb is ~60dollars, 1tb hdd is 50dollars and 2tb ssd is 140+ and 2tb hdd is 60, and 4tb ssd is 300ish and 4tb hdd is 80. 8tb ssd is around 1000, and 20tb hdd is 200-400 depending on recertified or new and seagate or wd. SSD prices go up exponentially.
Basically I have a massive media library. Some people on here know I have spent the better part of the last 15 years collecting dvd and blurays. I still am, however in the last 5 years I have focused on ripping them. Which need mass storage such as 16 to 20tb hdds. Ideally it would be ssds, but at present ssds are still extremely expensive.
I chose truenas, because of the app support. Prior to that I ran non zfs raid array in a vm. This was back in 2021.
I have a computer with 3 16tb drives that use about 7W each idle. 7W idle is 30 euro per year with the astronomical prices here. (the whole server runs at about 32W)
16tb drives are 270 euro here. 8tb qvo ssd is 580. so that storage is about 890 euro more expensive, or about 30 years of power use assuming ssd’s don’t use power.
Correct, modern high density drives are cheaper to run than even the most efficient SSD…except when they aren’t.
Raw storage is an order of magnitude cheaper for modern high density HDD’s.
Will do, cause there’s an algorithm and if he’s already running 20TB drives, this does not apply here.
Very few things in this world take more space than media.
Short of a different drive enclosure, I cannot make any more recommendations beyond running at a slower speed or trying a shorter and higher quality USB cable to the enclosure.
Thanks for bringing up power bill and all. It is a consideration. But my electric bill with two gaming pcs and a 6900xt and 4080 are also an issue. But I have all the drives in a single computer beside 5, and I am running 1000watt psu. I will measure power draw, but I don’t think it even maximizes the psu. So yes it is a consideration, but I require large scale storage and I have slowly built up over 5 years my storage.
It 6 or 8 drives I build a new pool.
I just ended up with 5 from my dead synology disk station that I wanted to make use of. Hence a 120 USD USB Enclosure.
So I had an older USB 3.2 10G enclosure. I wound up throwing it out because it kept having these strange cascading errors on the disk. One would throw an error and that would cause the rest to hang and then throw errors one by one.
What model enclosure is it? How long of a cable? Does it appear high-quality? Are you running this on clean-ish power?
P.S. I upgraded to a 8 bay thunderbolt enclosure and it’s awesome. I think thunderbolt is generally a better protocol than USB 3.2 for things like this.
Thunderbolt on X470 is really rare. Just thought I’d let you know what my solution was. I was attaching my disks to an intel NUC.
Cable length sounds good, JBOD mode is the correct mode.
So that device uses a jmicron USB - SATA controller. They’re known to be okay-ish controllers, but it’s a USB 3.1 gen 1 controller on the USB side. Not sure why it’s having issues on a USB 3.2 port.
It’s possible you’re encountering issues with the power saving modes that chip supports. I’d have to do some more digging, but there may be a way to disable power saving mode on that specific USB device.
Found the following blog post as well:
His recommendation, usb-storage.quirks=VVVV:PPPP:u tells the kernel to not bind the UAS driver to that chipset. Not sure if this will work for you, but it’s worth a try.
You’ll need to grab Vendor ID and Product ID from dmesg, and replace VVVV:PPPP with them.
That’s about all the research I can do without starting testing.
Could still be a power thing. I had a 120w picopsu in my nas with 3 16tb drives and especially in intensive writes and reads I got zfs errors. Even though max power use was about 70-,80w. The drives stopped because they didn’t get enought power, maybe it was the 5v rail. It was solved after putting in a 550w power supply.
Cal you try just running the drives off if a regular power supply and check ? Just keep it the same but put a power supply next to it with sata power.
Also make sure the data cables don’t daisy chain too much.
Nutral’s right about power. There’s both device power and USB link power saving to investigate – first confirm those busy disks can keep working with the external rack’s power delivery.
If you have some time, please think about disaster recovery. What this means is to ‘have a plan, not a panic’ when a disk or host device goes bad, and ideally to rehearse the stages of the plan when the heat’s off so that, when it matters, you don’t do it wrong and you don’t lose data. In any case, fingers crossed for you.