Yeah I don’t think there’d be any meaningful performance difference between filesystems with that kind of workload
File systems or hard drives?
During the burn-in ritual of my drives it was interesting to see how performance progressed as the heads flew sequentially across the platters. The drives managed 240 MB/s along the outer edge, which gradually degraded to around 160-150 MB/sec across the inner rim.
It would be roughly 75% big media (video files - hundreds of MB/many GB), maybe 20% small media (1-30 MB, mostly images) and like 5% small files.
So not exactly EZ mode all the way, but fairly close.
I do sometimes work on big files, but I think once I free up space from my scattered current drives, I will use those rather than going over network so playing media over a couple MBbit/s might turn out to be quite accurate - tho multiply that by 2, sometimes even 3. Also, I like to torrent-seed my, uh… linux ISOs. That’s some extra MBits of random IOPS.
Both. Filesystems for different reasons than hard drives (you mentioned).
That’s easy mode. 95% is >1M, a dream. No need to worry about filesystem performance at all. They’ll all be great. And with a use like this, you can expect double/triple the TBW on NAND Flash.
If you’re planning on using snapraid (and I’m assuming you’d add more data disks in the future) you want to avoid working on files stored on those disks as having the array out of sync with deleted or modified files is the most likely way you will get data loss if you suddenly need to restore a disk or file. You can always exclude certain directories from snapraid but it may be better to have a separate ‘active’ storage pool to use for that sort of thing and just use snapraid for static storage.
By spending inordinate amounts of time and money on some other proprietary storage solution that would have less performance, be less reliable, and require more inordinate amounts of money for the ‘snapshot’ licenses (cough 3par, cough IBM storage )
Please let’s not mix the OP use case (media server storage with no requirement for being able to recover from a failure and no backup, and no real experience in how complex his proposed solution is going to be to maintain/manage/ troubleshoot for failure) with 99% of use cases where zfs really shines when you factor in reliability, ease of backups and ease of recovery from failure
And yes, the Truenas people, especially the Truenas CTO, often have a stick through their ass that prevents them from accepting that people is willing to accept running systems with a high risk of data loss because really they do not care if they lose their data versus spending the money to buy and run an ecc backed triple mirrored system…
Having briefly looked at snapraid, what would make me think twice about adopting it:
- It’s a one man show
- It doesn’t even mention XFS in the docs
- 90% of the forum messages are about people struggling to recover data
- File system storage parity is hard
But ymmv, the OP approach is an interesting take and I really hope he’ll achieve what he needs
One last word from a dinosaur, that I keep repeating and getting made fun of: in 20 years working with all kind of Linux systems, the one that most struggled to recover from power loss were XFS based ones, where often the less expensive solution was to reinstall a system, in proportion that almost never happened with ext4, btrfs … to date they still state certain combinations of raid can have data loss so to me it’s a non starter
1 Like
Ok, hold on.
This, totally.
This, not so much.
I would like some safety.
Also, what is CFS?
But yes, otherwise totally no mixing, plz.
Typo, I meant XFS, it’s not in the contemplated filesystems in the FAQ …
And here we could start with another 200 page thread on everybody’s interpretation of safety for a filesystem
There is a reason why when you put ‘NAS storage’ in a topic you get 99% of the people to automatically say ZFS/Unraid/Openmediavault/Synology/Qnap
That is because all of the above, with an adequate amount of storage devices, and following the best practices, give you a reasonable amount of ‘ability to recover from a device failure’
In almost all cases, the ‘safes’ choice at the expense of money is to use mirroring/multiple level parity so that even in the even of multiple failures, barring extremely improbable events, you will be able to recover easily (if slowly in the case of 10+TB drives) from a failure: just pop in a new equal size device, trigger the rebuild, wait for it to complete and off you go.
This comes at the cost of multiple drives, eventually software cost premium in case of synology/qnap/unraid and/or with the pain of the old wheezers ‘telling’ you you need to do things in a certain way
In most cases it’s because we have seen what recovering from a failure using ‘other ways’ entails and it’s always not as straightforward
When you talk about safety everyone and his mother will also mention 3-2-1, that is you need to have three copies of your data, 2 of which on different storage media and one of which offsite, as the minimum amount you need if you really want to be safe
In that context, your proposed solution falls very short of the standard definition of media storage safety
Compound that with the fact that your choice of data parity is a pretty obscure project noone here has used, on top of two file systems that shine when talking about performance (XFS,BTFRS), functionality (BTFRS) but suck at reliability and everyone’s hair is standing up (for people who still have it, you know, old wheezers 
) and waiting for the moment when either a human error
( messing up things when playing with parity ), or assuming the parity covers all your drives while in rality it does not, or the sync process stops working and you don’t notice or any other unknown at this time event happens and you come back here with a ‘can;t access my only copy of my data’ post … and you can start having an idea as to why everyone is more or less saying ‘bite the bullet and buy more drives and use ZFS …’
1 Like
I’ve been using it for 10 years, I’ve recovered probably half a dozen drives and never lost any data. For it’s use case it has a lot of advantages over btrfs or zfs.
2 Likes
I stand corrected
No system is perfect. Drive formats, soft drinks, political parties, cars, football teams, everyone has favorites and others will disagree with you.
A comprehensive layered backup solution is needed regardless of how ‘unsinkable’ your storage array is. 3-2-1 is the minimum, but good IT Departments go much further than that. Our primary production data has six layers of redundancy/backup.
No matter what you have or how good the hardware is, at some point it will fail. Be prepared for that moment, before it happens.
1 Like
Hope you’re having fun in your data center. I’m using 12 year old parts here.
Is it so utterly, completely inconceivable to all of you old farts that a person might lack the financial means to execute on a “proper” backup and data safety strategy?
My deepest apologies if being poor has offended you.
Morning @martixy
My comments were general and not aimed at you. The age of the parts (or their cost) is incidental to the value of your data. My computing background began humbly where my only business backup was a 5.25" floppy.
My point was, invest as much time and effort into your backups as your storage solution.
Have a great day.
Sorry. Being subjected to the same unactionable spiel incessantly wears you down eventually.
Only for metadata, and zeroing the size of a file and wiping-out its contents is acceptable for XFS to ensure consistency.
Enough theory though, let’s test that!
XFS data corruption detection:
= well, no detection at all actually. Fine for unimportant things like replaceable media.
ZFS data corruption detection:
= detected the error, and because the file was small, it packed the contents and metadata into the same block which it stores with copies=2, so it was actually able to return the correct data from a ditto block. magic!
2 Likes
He’s planing on using snapraid which will handle error correction and redundancy, so really it’s not that important how the file system handles it.
I would still pick btrfs for the data drives (or zfs but that will require more tuning) and either btrfs or xfs for the parity disks, really only choosing xfs if there is a meaningful space saving compared to btrfs.
I really love copies=2. It may be wasteful, but on a single disk for only the more important datasets…just amazing. I also have it running on my (mirrored) boot drives where capacity is not a concern. BTRFS DUP is similar.
Only if you want to get in the habit of doing a manual scrub/status/fix before updating files, then doing a sync/scrub/status/fix after every update. Do people using snapraid really do that?
# mount | grep xfs
/dev/sda2 on / type xfs (rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota)
/dev/sdb1 on /mnt/xfs0 type xfs (rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota)
/dev/sdc1 on /mnt/xfs1 type xfs (rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota)
/dev/sdd1 on /mnt/xfs2 type xfs (rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota)
# cat /etc/snapraid.conf
parity /mnt/xfs0/snapraid.parity
content /var/snapraid/snapraid.content
content /mnt/xfs1/snapraid.content
content /mnt/xfs2/snapraid.content
data d1 /mnt/xfs1/
data d2 /mnt/xfs2/
# cat /mnt/xfs2/test.txt
test1234
# snapraid sync
# snapraid scrub
# snapraid status
...
No error detected.
# cat /mnt/xfs2/test.txt
test1234
# hdparm --fibmap /mnt/xfs2/test.txt
/mnt/xfs2/test.txt:
filesystem blocksize 4096, begins at LBA 2048; assuming 512 byte sectors.
byte_offset begin_LBA end_LBA sectors
0 2152 2159 8
# echo -n failfail | dd of=/dev/sdd bs=1 seek=$[2152*512] count=8
# echo 2 > /proc/sys/vm/drop_caches
# cat /mnt/xfs2/test.txt
failfail
# snapraid scrub
Self test...
Loading state from /var/snapraid/snapraid.content...
Using 0 MiB of memory for the file-system.
Initializing...
Using 64 MiB of memory for 64 cached blocks.
Selecting...
Scrubbing...
Nothing to do
Saving state to /var/snapraid/snapraid.content...
Saving state to /mnt/xfs1/snapraid.content...
Saving state to /mnt/xfs2/snapraid.content...
Verifying...
Verified /var/snapraid/snapraid.content in 0 seconds
Verified /mnt/xfs1/snapraid.content in 0 seconds
Verified /mnt/xfs2/snapraid.content in 0 seconds
# snapraid status
No error detected.
# cat /mnt/xfs2/test.txt
failfail
1 Like
By default the scrub command only scrubs the oldest 10% of the array and only data older than some number of days, so it won’t scrub new data, you could use snapraid scrub -p new to scrub new data which has never been scrubbed or snapraid scrub -p full to scrub everything if you were trying to test it.
Snapraid is designed for use with data which is changed infrequently like media libraries, it’s obviously not suitable for data which is modified often. Running sync daily and a partial scrub weekly will work fine for most people.
2 Likes
doesn’t hurt to let the underlying filesystem police it’s self imo. btrfs and zfs can both pick up errors in scrub fine, and will likely do it more efficiently and more thoroughly than snapraid is able to, as well as being able to passively check for errors on read, which is pretty important for a media server imo.
bitrot can just happen without anything writing to a file/through the filesystem, either via direct block device access, such as with rootkits/low-level drm or anticheat, bugs in the kernel or driver, bugs in the filesystem, or just bugs in software that can access the drive as a block device, or just by the device it’s self losing a bit to something, be it a logical error in the drive controller, a physical error in the mechanism, or just emi flipping a random bit.
I would trust a filesystem to better know whether it’s data is inconsistent than some kind of parity check running overtop of it, but I think the best solution is actually to use both.
yeah that’s why I’m suggesting btrfs (or even zfs) for the file system choice rather than xfs or ext4. But I don’t think it’s that important to detect corruption on read for a media server, the chances of bitrot affecting the file you’re actually about to watch is pretty slim and 90% of the time it’s only going to result in a random artefact in the middle of an episode of Rick and Morty. Maybe I’ve just been lucky but I used to have 20 something disks and I think I may have had two or three errors over 10 years, all of which I was able to fix.
2 Likes