I know the initial installation has everything as a dataset. Anything you add, probably isn’t. I know my Kubuntu Laptop had several hundred datasets. It was great (until Canonical killed it).
No management needed. And with child datasets inheriting properties, it’s fairly easy even if you have to tweak stuff.
All RAIDZ are slow.
No. You just split the blocks into smaller pieces that are harder to read later. Any Parity RAID setup sucks at writing. RAIDZ starts fairly fast, but drops off quickly once you fill it with actual data.
If you want IOPS, you need as much vdevs as possible. Mirrors are best, small RAIDZ vdevs are still okish but the wider you go with the vdev, the worse it gets with a given amount of drives.
Which is causing problems as you mentioned earlier.
You move the data from the parent dataset to the newly created child dataset (subfolder so to say). You can mount the child dataset in /path/movies if you like. If it’s shared, you have to update the permissions so the client can browse it like a folder and mount it.
It does. But you won’t use much of what CoW has to offer.
If you don’t see the point in using features, that’s fine. Just don’t complain when things are slow and bad. If you like your 90s storage admin thing, you will also deal with 90s storage problems.
You send snapshots, not datasets. And if you can store files on a tape, you can store snapshots as a file. I don’t think it’s very useful, but I don’t think tapes are useful today either. Too many disadvantages for modern storage. But it’s certainly faster because we don’t deal with files and you get max throughput on the tape drive.
That’s a self-inflicted problem. I never had to put quotas on anything and I can’t see how this helps. If I need more space, I plug in a new drive.
I remember VGA copy in the 80s being able to store a single file on multiple floppies. Can’t the tape drive do this as well? Otherwise you have to manually split chunks of 12TB…
This is all clunky yesteryear to me tbh.
Not that much of a problem with ZFS as long as they don’t sit in the same dataset. Most reads are cached in ARC and replication doesn’t care about small files.
But then you have to move all data between datasets. You start with creating meaningful datasets, not create them afterwards.
It doesn’t have to. Because admins create datasets to suit their needs. Creating like 20-30 datasets over the cause of a year is a trivial amount of work. There is no need to have datasets for every single subfolder because most subfolders aren’t relevant for storage considerations.
You don’t need 10000 datasets, not having to dump every file into C:\ is progress and helps greatly. Create datasets just as you create folders, to seperate things and keep things manageable and easy.
If you want some old-school text file, you can redirect the ls output into a txt file. But this reminds me of people printing their emails. Redundant static data.
ZFS offers in-line compression. Set your backup dataset to high compression (like high GZIP or ZSTD levels). ZFS scales linearly with the amount of cores when compressing/decompressing.
ZSTD is really good and fairly new and I use it on half of my pool, other half being LZ4. LZ4 is ridiculously fast…like GBs/sec per core, which is why I use it for my Flash drives as well.
What you should be concerned with is throughput, not the details of how some tool does or doesn’t manage it (threading). With LTO-8 you only need 360MB/s max. Testing on my old [email protected], unpigz manages over 800MB/s.
Based on what you wrote earlier, when it comes to structuring the ZFS pool (and more specifically, PLANNING for the structure, so that you can manually create the dataset for the downstream benefits of having done so), you would have to know, in advance, what “kinds” of data you are going to be storing, where and how vs. it just being a “catch-all” (data dumpster) for everything, the latter of which, of course, has basically no planning – which is what I’ve deployed.
So you ask “why not both?” (I don’t speak Spanish, so I had to google translate that) – there comes a point of planning to the maximum extent that you possibly can (even though unplanned events still occur) vs. absolutely zero planning went into setting it up.
(i.e. bring both pools online and start dumping data to it, which is what I did)
And it is quite evident that you thought about your layout whereas I didn’t. At all.
When I was in university, my nickname (which still holds true to this day) is “grossly underqualified system administrator”.
The assumption in regards to this is that you assume that other people care as much about sysadmin as you do, when really, I just get it up and running JUST enough for it to work, but otherwise, spend hardly any time, studying it to the level of detail that you might study it.
Conversely though, that would be like me saying “why don’t you know the stuff pertaining to HPC/CAE/FEA/CFD to the level that I have studied these topics?”, if for example, you were trying to develop a product, and wanted to predict whether your product will pass or fail some kind of a design criteria.
The goal wouldn’t be to chastise you for what you don’t know. The goal would be to teach you so that you understand enough about the theory so that if you were to entrust me with running the analyses/simulations for you, that you would know just enough to folllow what I am talking about. You don’t need to know nor fully understand the theory (and/or math) behind the Newton-Rhapson iterative solution process in regards to your question of “did my part pass or fail the simulation/test?”. If you’re interested in the methodology, I can teach you and then we can talk about whether it’s the right methodology to use, or if there is a better, more appropriate solution methodology out there.
But as a “casual” user or consumer of the results – you only really need to know/learn that if you have an intent on either changing it or improving it.
This is no different here.
Are there benefits to me having the same level of knowledge and expertise re: ZFS as you have?
Yes.
But is that an absolute requirement for me to get the system up and running?
No, not necessarily.
Yes, the system might work better, based on your level of understand of how ZFS works, but that doesn’t automatically preclude that just because someone DOESN’T know, that their deployment is going to be automatically trash because they didn’t create as many datasets/zvols as you have.
Here’s the thing with these statements though:
when you execute the zfs create command - this command is used both to create a dataset AND zvol.
The ONLY difference, between the zfs create command used to create a dataset vs. the zfs create -V command that is used to create a zvol is literally the -V (and then the size parameter immediately following the -V flag. That’s it.
Thus, if you didn’t care how much space the volume/dataset takes, WITHOUT specifying said -V flag, you’d be creating a dataset regardless of whether you explicitly wanted to or not. (Source: zfs-create(8))
Stupid question then, based on this example:
What would be the advantages or what would be the use case where you’d want to create a ZFS zvol (block device) and then format that as ext4 vs. having a ZFS formatted zvol/block device?
(The limitation of my imagination can’t figure out what a potential use case for this might be. (i.e. why wouldn’t you just leave it as ZFS-formatted rather than formatted the zvol with ext4?))
Yeah…I’m not so sure about that.
It is my understanding that a VM that has a qcow2 disk image – the contents inside of which aren’t explicitly nor directly visible to another VM unless that VM shares that data and exposes the contents to the “network”.
If you have the following configuration for example:
VM1 ↔ host ↔ VM2
and suppose that you hosted index.html on the host (which was the example that Stefan Hajnoczi put in his presentation and also his talk) – with virtiofs - both VMs can see index.html that resides on the host.
To the point of your statement though, that isn’t the same as having the qcow2 image on the host though. VM1 has a qcow2 disk image file on the host and it can communicate with it, but VM2 has no access to the data nor the contents of VM1.qcow2.
Similarly, VM1 has no access to the data nor the contents that’s contained within VM2.qcow2.
Whereas, with virtiofs enabled on both VMs 1 and 2, both VM1 and VM2 can see index.html that’s on the host. But that also doesn’t automatically grant the ability for VM1 to be able to peer into the contents of VM2.qcow2, even if that file was hosted by the host.
virtiofs (in Stefan’s example) helps to facilitate file/data sharing. qcow2 (and zvols, pursuant to your example) - does NOT facilitate file/data sharing. Thus, they are not the same.
I would invite you to read the presentations that Stefan has put together, to get a better understanding of how virtiofs works, because based on the comments that you’ve made about it so far, my impression is that it isn’t what you think it is.
(qcow2 or raw in datasets or zvols (without iSCSI) does NOT help make it so that VMs can see a common set of data or files, and therefore; they’re not the same.)
I don’t think that I quite follow.
That would only show me the disk transfer rate data/statistics.
I don’t see how that can help.
I literally have 7 dead SSDs that disproves this statement.
They were used as Windows swap drives on my workstation which has 64 GB of RAM.
The average daily write volume was somewhere on the order of 2.29 GB writes/day.
And yet, I’ve killed SEVEN SSDs through this usage pattern.
Once they have exhausted the finite number of write/erase/program cycles of the NAND flash, the SSDs WILL die.
Again, I literally have a stack of dead SSDs which proves this point. And they all died with a FRACTION of the rated TBW.
If you google “SSD death test” where they will purposely push sequential writes to a SSD until it dies, yes, it CAN write more than the rated TBW, but it WILL die. There’s no question about that.
This is how I’ve killed SEVEN SSDs.
You can see from the pictures below how the SSD reporting tool shows an estimated life remaining of 84% and 100% respective, and yet BOTH of these drives has CLEARLY failed.
And they’re both READ failures, not write failures.
(The first time that I killed an Intel 750 series 400 GB NVMe 3.0 x4 SSD – that was a write exhaustion failure. At which point, the SSD put itself into a read-only mode, and IF it would read the data at all, it would do so at approximately 2 MB/s vs. the originally rated speed/specs. But a lot of times, the reads would timeout.)
So - you can argue that it’s quote “flat out wrong” all you want. There’s a stack of dead SSDs on my desk that provides the data and the evidence that forms the basis of my statements.
(And in order to prevent more SSD deaths (because trying to clone a dead drive doesn’t always work), I’ve decommissioned ALL of my SSDs, wherever possible.)
Not necessarily.
TBW is rated under STR conditions, not random I/O (swap usage) conditions.
To the best of my knowledge, there is no counter for how many of the write/erase/program cycles of the NAND flash cells/modules has been consumed based on any sort of given usage pattern.
If there was, I would LOVE to keep a live monitor of how the NAND flash modules/cells are doing when I am using a SSD as swap in Windows, because THAT would tell me how the swap usage pattern is hitting the NAND flash cells/modules (whether it is consuming the finite number of write/erase/program cycles uniformly or not).
But to the best of my knowledge, this sort of a tool only exists at the manufacturers of said SSDs, IF it even exists at all, and it’s NOT publicly available.
i.e. There is no way that I was able to predict that the first death of the Intel 750 Series 400 GB SSD would die after approximately 4 years. I was able to successfully RMA it the first time. The most recent death (as shown in the screenshots above), is actually the RMA’d unit that was sent to me, as a replacement, and I killed it a second time, under the same usage pattern, in about the same amount of time. But again, with a sample size of 1, that does NOT make it a pattern (to be able to do life cycle predictions based on N=1).
(Note, by the way, that you don’t dispute the fact that SSDs WILL die. It isn’t a question of IF, but a question of WHEN.
For $0 cost?
You might have missed the part where I wrote:
Buying replacements for said dead SSDs breaches the second sentence here.
(You might have additional funds allocated for said replacements, and that’s fine and great for you. But this way of thinking of “my way = your way” completely ignores this technical requirement/statement that the system must be and remain in good, operational order, without additional maintenance funds. Using SSDs that have a finite life breaches this statement, and therefore; it is not a solution that complies with this requirement.)
DWPD is assumed based on writes performed at STR. But if you have random I/O, then the datasheets won’t tell you how many random I/Os the drive will be able to take before it will die. And that still belabours my point about how SSDs will die. Like I said – they’re the brake pads of the computer/tech industry – it’s a known, wear item that you will have to replace.
re: risers
I appreciate the reference to parts that you have experience with, that works.
I think that you might have missed the point of this question.
Please refer to Stefan’s presentations re: virtiofs.
It is my understanding that the z_rd_int_n processes isn’t a single application that spawns separate, “compute” or child processes.
(i.e. unlike applications that use distributed, parallel processing via MPI)
Therefore;
a) doesn’t that go back to what I wrote previously about maybe this is why the recommendation or the rule of thumb is that the total number of cores, in the storage server, should equal the total number of drives? (1 core per drive, so that the cores would be able to handle processing the drive I/O requests?)
b) I can see this being more of a problem either:
i) if you had two HBAs and CPU0 was trying to command HBA1 (where the PCIe slot is wired to CPU1) or
ii) all of the I/O that’s being processed by CPU1, through HBA0 (where the PCIe slot is wired to CPU0), is limited by, as we discussed, the puny QPI link that exists between CPU0 and CPU1.
(But being that it’s spinning rust – I was also under the impression that even with the bandwidth and/or latency limitations of the QPI link, that the QPI link itself, would hardly be the bottleneck.)
I mean, in the Supermicro 6048R, you have 24 drives through a SAS 12 Gbps x4 expander (which = 48 Gbps total, split across 24 drives, for 0.5 Gbps each, in terms of theorectical bandwidth/throughput) and then you have 12 drives in the back, that also goes through a x4 expander (for another 48 Gbps).
And then, ignoring for a moment that 8 of my drives are SAS 12 Gbps drives; if we assumed that I was using all SATA 6 Gbps drives - the front 24 bays would nominally be requesting 144 Gbps of bandwidth through a 48 Gbps pipe, whilst the back would be nominally requesting 72 Gbps through another 48 Gbps pipe.
My point is that the puny QPI link, probably isn’t going to be the problem here.
The MegaRAID SAS 12 Gbps HW RAID HBA (where I am running the disks in JBOD mode), has 8x 12 Gbps SAS lanes = 96 Gbps, through a PCIe 3.0 x8 interface, which is only good for a theorectical 64 Gbps max.
Thus, 36 drive bays at a minimum of SATA 6 Gbps = 216 Gbps nominally requested bandwidth, through a straw that can ulimately only deliver a maximum of 64 Gbps.
But that’s also generally NOT an issue, because even if we assumed a STR rate for the HGST spinning rust drives to be between 150 MB/s to 200 MB/s, at the low end, 150 MB/s = 1.2 Gbps. * 36 drives = 43.2 Gbps.
And at the high end, 1.6 Gbps * 36 drives = 57.6 Gbps.
I don’t think that I would be able to hit that with raidz2 vdevs.
Thus, I have little data which shows that NUMA would be an issue, given the maximum throughput that the drives themselves, can do. (And that’s STR. If it is random I/O, it would be significantly less than that.)
(It’s a pity that they don’t rate the QPI link for how many IOPs it can handle.)
If there’s a technical, prescribe methodology/procedure that I can execute, which will collect the data, I can see about running tests on my live, production system, to actually measure and collect data about its performance.
Yes, but if, drawing from my experiences with RDMA over Infiniband – if Infiniband can do remote direct memory access, where it skips the rest of the stack, and can do so remotely, then why wouldn’t/can’t I do it locally and skip the stack, locally???
I mean, a part of the whole point of NFS over RDMA is that you skip the entire networking stack (through the OS, the NIC drivers, the kernel, etc.) and are able to access the data that’s in memory directly, which makes the entire process faster and more efficient, and reduces CPU load, which then, the CPU is free to deal with other stuff that it needs to deal with, rather than having to process the data going through said stack.
If you can do that for NFSoRDMA, then why can’t you do it for a local file system over local DMA?
No, you’re boggled by this because you are thinking how you are thinking rather than asking yourself the question “what does it take for the statement, coming from another person, to be true for that person?”
That’s the position that you’ve taken up in this discussion.
Throughout this entire discussion, you have talked about what you would do, based on how you have done it without ANY regard to the possibility that what and how you are doing things might NOT be the best way, given constraints.
You make your assumptions based on what you know, NOT based on what other people know. And with that, also comes the fact that you also assume that other people need to be or operate at your level in order for them to be effective, which ignores the question “what else does the other person have, going on, that would prevent them from operating at my level?”
This is and has been your ENTIRE approach to this discussion.
It’s based on you and what you would do and how you would think about the problem.
It literally gives zero f’s about the other person/the person who is asking the question.
No, that’s NOT what I wrote.
Read it again.
You’ve CLEARLY never worked with LTO-8 tape, for your homelab.
As a result, you are making assumptions about its theory of operation rather than having any actual “seat time”/real experiences with ACTUALLY using the technology, whereby you would know and recognise its pros and cons, as well as its strengths and limitations.
If you actually HAVE something to offer/contribute in regards to how you would use zfs send and zfs receive on/for LTO tape/LTFS, then you would’ve already copied-and-pasted the commands that you use to accomplish this.
But as it is shown here, you haven’t, which means that you have no idea what you’re talking about in regards to using zfs send and zfs receive to be able to send said dataset to tape.
If I am only using zfs send and zfs receive to go from pool A to pool B, and then I am still using rsync to actually write the data to tape – once rsync hits EOT, it will stop writing/syncing, due to it being out of space. (Ask me how I know that.)
Therefore; if you’ve sent a 30 TB data server from pool A to pool B, this first transfer will be fast, but then you still have to break it up into 12 TB “chunks” so that you can actually write that chunk to tape.
Again, if you have an ACTUAL LTO-8 tape drive and tapes deployed in your homelab, you would know this.
(And you’re using ONLY FOSS to drive/operate said tape backup system.)
(SOR: There can be no cost associated with using the tape drive. (i.e. YoYotta is out.) Which you’re going to ignore this SOR anyways, just like how you’ve already ignored all the other SORs in this discussion here.)
To break up the dataset by assigning 12 TB quotas on all of said datasets (or again, really 11 TB), that will help to facilitate:
a) you can use zfs send/zfs receive to send the dataset from pool A to pool B quickly and then
b) you can write the dataset to tape quickly, because it has already been split into 11 TB “chunks”, so once the zfs receive operation is complete, you can immediately write the data to tape.
You don’t have to send a 30 TB dataset over, and THEN do the splitting, to prepare for writing to tape.
And if, pool B, (because that pool also houses the VM disk images), where there is ever only 12-15 TB of free space left, I can only imagine what would happen if you try to send a 30 TB dataset over using zfs send/zfs receive to pool B that only has 15 TB free space left.
You’re clearly NOT thinking about that.
Nice gaslighting.
I LOVE it when tech/IT people blantantly ignore statements of requirements that are inconvenient for them without ANY regard to the importance of said statement of requirement to the customer/end user. (It’s like they put that requirement into said statement of requirements for shits and giggles.) /s
rsync to tape does not.
Again, I have proof/evidence of that.
Your assumption here is that I am using something OTHER than rsync to write the data to tape, which again, ignores reality.
(If you have a FOSS solution for YoYotta, I’d love you hear it.)
I tried using Xinobris to scan my main pool (155 TB raidz2 total available capacity.)
At the time when Xinorbis crashed, it only scanned 7 TB of the pool, and found ~16k folders.
Extrapolating: (my pool A is about 75% full), 155 * .75 = 116.25
116.25 / 7 = 16.6
16.6*16k = ~265k folders.
If a dataset was to be automatically created per folder, I would have had 265k datasets.
I think that give this, if you ran zfs list, the resulting output to console would be 265k lines long +1 (for the header).
This is why a management tool for the datasets alone, would be necessary, as you try and browse through 265,000 datasets (if it was automatically created per folder).
My thought process in regards to this is that because pool B is only an 8-wide raidz2 vdev (consisting of just the single vdev), that it shouldn’t be that bad.
It might be good for IOPs, but it would not be good for fault/failure protection.
If two drives from the same mirror vdev dies at the same time, then that part of the vdev and its associated data is gone forever. Restore from backup would be the only solution to that.
Whereas with raidz2, and two drives can fail, and the pool will still run, albeit in a degraded state.
The only problem this causes is which system/OS/“infrastructure software stack” supports it and which one doesn’t.
Since it works in Proxmox, I see that I lost very little in terms of what I am missing from TrueNAS Core. Yes, the ZFS admin GUI of TrueNAS is nice. But that’s really about the main thing that it had going for it.
When I was testing all three systems in a side-by-side comparison, I don’t think that I ever got GPU passthrough working correctly in TrueNAS Core, even though they made the GUI ridiculously easy to set it up, but it also didn’t work neither. (Whereas, with Proxmox, the steps it took to get that up and running was more involved, and it didn’t work for any and all GPUs, but once you have a compatible GPU and it you get it up and running, it works quite well, I must say.)
That’s really the only “problem” that I had with virtio-fs.
Operationally speaking, it’s been great. If you have a directory with lots of files (e.g. 2 M files in it), it loads wayyyy faster in Windows Explorer than it does if you try to read the same directory over SMB.
Thank you.
But my question was moreso “what if that mountpoint exists already?”
Will it still let me mount the child dataset to an existing, non-empty mountpoint?
Per my OP - my question was about how do I find out what’s the root cause of what’s causing the slow down and how do I collect the data to be able to identify the root cause?
(how much of the problem can be attributed to my using rsync instead of zfs send/zfs receive? and how much of the problem can be attributed to my not using ZFS datasets?)
(I would HATE to have to create 265k datasets manually in order to help facilitate zfs send/zfs receive.)
Oh no…tapes are still VERY much widely used today, for zero-watt, cold/archival storage of data.
ISOs, if you’re not using it all the time, is an example of something that would be GREAT to store on tape.
It takes a few minutes to be able to get retrieve the ISOs from said tape, but being that you can download the data once, and then just write it to tape, especially older versions of ISOs that may not be available anymore, after a certain amount of time – tape is great for that.
Like anything, it has its pros and cons.
People (home labbers and tech YouTubers) are talking about their Petabyte projects.
Meanwhile, once I back my stuff up onto tape, I can, realistically, SHRINK my homelab even further, because really – the amount of data that changes daily, isn’t that much. Therefore; I don’t really NEED that much in the way of hot/live storage, spinning all the time.
People do it because they think it’s nice (and it has its uses), but let’s be real here – out of all of the files that you have, how often do you use ALL or even MOST of the files that you have in your possession? (vs. how much of it is more of a write-once-read-many times style of data usage pattern?)
Yeah…again. It varies.
I mean, the file-analogous of this would be to write what you were going to write to the tape, into a single tarball archive file, and then send the whole thing over to tape and so that it can write said tarball archive to tape, as a single operation/command.
But if you want to just pull one small text file from it, you’ll have to download the whole thing (or you can try and see if you can extract that single text file from tape directly). But if you want select files from it, then you will have to download and unpack the whole thing, for you to be able to pull out the select files that you want, and then dump the rest.
(Whereas, yeah - it might take me 3 days to write 12 TB to tape because of the presence and existence of small files, but if I want to pull those files back out, I can do that much faster because I won’t have to restore the entire tarball archive file from tape before I can unpack it, to pull what I want out of it.)
Doesn’t work for tapes, if you are sending the data over to tape using rsync.
If the amount of data in a dataset exceeds the capacity of the tape, rsync will ultimately fail, due to “out of space” error.
(Actually, I think that the actual error that rsync will report out is a write failure/error. I don’t think that it actually knows that the destination/target is out of space.)
And so far, I am not aware of any FOSS solutions that can be used to manage tapes. (i.e. no FOSS alternative to YoYotta that I am aware of, for example.)
For VGA copies on floppies - there were tools that came with the OS that can do this (for floppies).
For tapes, there aren’t FOSS solutions that I am aware of, that can automatically span multiple tapes for you.
All of those solutions, that I am aware of, are all paid solutions. And some of them, aren’t cheap. Archiware P5 for example, is $2000.
YoYotta with YoYotta LTO drive, is a $180/year subscription, which means that if you don’t plan on using your drive in a given year – that’s great - you can save some money, but if you use it annually, at any point in the year (or you need to), then you’re paying.
(YoYotta used to be that you just buy the software and that’s it. But like so many software these days, a lot of them have moved to the subscription model instead.)
This drive, for example, comes with YoYottaID LTFS, but I don’t know if that’s the “standalone” version of the software (like they used to offer) or if that’s still the subscription based version of the software.
For FOSS solutions, yes, it’s very manual (and clunky on account of it being manual). But I guess that’s why the paid software solutions are paid software solutions. Someone eles figured out the solution to it, and they want to make money as a result of figuring it out.
I think it might, actually. (sit in the same dataset)
yay!
See…that requires foresight and planning, at the time of system deployment, which I didn’t have.
And rather than dwelling on what I could’ve done different, in the past, it is probably more fruitful of a discussion to talk about what I can do now, GIVEN that I didn’t create said datasets when the system was being deployed.
(i.e. if I am going to have to backup the entire system, so that I can re-deploy it, then my current issue will still be my current issue, if you get what I am saying. i.e. this problem won’t be solved until and/or unless I redeploy the system, which means that the less-than-ideal performance now to paraphrase the song “whatever will be, will be”.)
When the system was initially deployed, the focus was on migrating the data off from the old NAS servers and onto the consolidated Proxmox server (so that the drives can be pulled from the NAS servers, and redeployed as an 8-wide vdev here.
For live storage, yes, I would 100% agree.
But to prepare it to be backed up to tape that only has a 12 TB raw capacity, this doesn’t work.
(because if I use zfs send/zfs receive to transfer a 30 TB dataset, I will still have to chunk it out before writing it to tape.)
If I don’t chunk it before writing it to tape, the write operation (to tape) will fail when it hits EOT.
Automated spanning onto multiple tapes needs to be aware of where/when tape 1 ended (in the overall backup task) and where tape 2 begins.
And given that there isn’t a FOSS solution for this – this is why I have to handle this aspect of an archive spanning multiple tapes, manually - i.e. chunking out the data, to prep for writing to tape.
And the theory behind all of this is that the zfs send and zfs receive will lower the CPU load averages because it is sending and receiving data at the block (device) level rather than at the file(/folder) level.
So it can, potentially help, with a part of the problem.
(It’s too bad that I don’t have a clone of my server, with all of the exact same, identical hardware, so that I would be able to test this out on a dev system, rather than testing it on a live, prod server.)
Run ls | grep on a 112 TB dataset with 15 million files once a week. (I originally wrote repeatedly, but the idea is that you’re going through your dataset when the metadata/the list/index of what’s in your dataset, hasn’t been cached in RAM in a while.)
You’ll see/figure out why I build an index file.
(My old NAS servers used to have a Celeron J3415, an Annapurina Labs ARM processor, and 7th gen Xeon processors. (Xeon L5310, I think.) i.e. they were all very slow processors by today’s standards, which makes ls | grep not particularly fast. Thus, to speed it up, I used to index like maybe once a month, let it chew on that, and then for that month, if I wanted to find a file, it was SIGNIFICANTLY faster to cat | grep the text file than it was to run ls | grep again.)
LTFS ignores ZFS compression.
(The tape backup doesn’t know nor recognise nor cares about ZFS compression. To the tape, it would be as if ZFS’ in-line compression doesn’t exist and/or is disabled.)
Are you decompressing to SSDs or are you decompressing to HDDs?
And to measure the throughput, are you using iotop or is there a utility within pigz that will also report out the throughput as well?
I can test it on my 5.1 TiB pixz archive.
That’ll be an interesting test, I think.
From the pixz github page, it says, quote: “pigz - not indexable”. I’m not sure what that means. Would you happen to know what the author means by that?
(I’m downloading my project file from tape now.)
edit
n/m. I figured out what vasi meant by that pigz is not indexable.
pixz creates an index, if the incoming format of the file to be compressed is a tarball.
That way, you can list the contents of the archive without having to unpack it first.
It would appear that pigz doesn’t support this index feature (of the archive).
For a benchmark, I do RAM to /dev/null to avoid disk speeds skewing results.
I prefer to use the pv command as it’s a user-friendly replacement for cat that shows the current speed and average speed at the end. But dd or even time (and a bit of math) can measure the throughput as well if pv is unavailable.
pv -Ss 4G /dev/urandom > rand # Create 4G incompressible data file
pv rand > /dev/null # Cache to RAM
pv rand | pigz -v > /dev/null # Test compression speed
pv rand | pigz > rand.gz # Write a gz compressed file to disk
rm rand
pv rand.gz > /dev/null # Cache to RAM
pv rand.gz | unpigz > /dev/null # Test decompression speed
Just a mention that, at least on my machine (24 core EPYC), zstd -1 with one thread is faster and compresses my source code and databases only 0.1% larger than pigz using 48t. And zstd -5 -T4 with 4 threads is about 2x faster and compresses better than than pigz with 48 threads.
ZSTD is amazing. It can’t compete if you really want the maximum compressed state, but anything else is so fast with ZSTD. I even use it on BTRFS as inline compression on my daily driver NVMe drives. I keep it at default level 3, but backups run with level 9 and 13.
You should compare zstd -1 against pigz -1 they will be more comparable, though lz4 is the better fit. zstd is a lot like lz4, gzip, and xz rolled into one, depending on the compression level you select.
The compression ratio is going to be highly dependent on the data and not always so favorable. Yes, zstd -1 or lz4 are faster and than pigz, and if you’ve got data that can’t be compressed very well then the output won’t be much larger. But your test data probably isn’t representative of a a full system backup, and you’ll be missing-out on good compression ratios elsewhere… All for NO improvement in speed, since the LTO-8 tape will be the bottleneck anyhow.
In regards to that code snippet though, is I wonder how well that will extrapolate to an archive file that’s currently a total 7.1 TiB?
Here are the results:
(between pigz vs. pixz)
filesize comparison:
4294967296 Jun 14 22:50 rand
4296278034 Jun 14 22:47 rand.gz
4295173152 Jun 14 22:50 rand.pxz
4294973440 Jun 14 22:48 rand.tar
4295173800 Jun 14 23:06 rand.tpxz
4295065618 Jun 14 22:50 rand.zstd
I ended up changing the way that I ran this a little bit in two ways:
I scripted it. I found that if I typed and/or copied-and-pasted the commands one at a time, the results can be SIGNIFICANTLY slower. (e.g. 490MiB/s vs. 650MiB/s)
I brought everything into the time domain because the command to create a pixz archive (via tar) - I don’t know how to make that work with pv, so time would be one method that I can use, to be able to calculate what the data rates were.
Here are the results (with RAM caching):
(time,command,data rate (MiB/s))
22.93 time -p pv -Ss 4G /dev/urandom > rand 178.6306149
2.39 time -p pv rand > /dev/null 1713.807531
7.47 time -p pv rand | pigz -v > /dev/null 548.3266399
7.24 time -p pv rand | pigz > rand.gz 565.7458564
0.64 time -p rm rand 6400
13.34 time -p pv rand.gz > /dev/null 307.0464768
4.93 time -p pv rand.gz | unpigz > /dev/null 830.831643
5.22 time -p pv rand.gz | unpigz > rand 784.6743295
0.98 time -p pv rand > /dev/null 4179.591837
45.21 time -p tar -I pixz -cvf rand.tpxz rand 90.59942491
0.67 time -p rm rand 6113.432836
1.04 time -p pv rand.tpxz > /dev/null 3938.461538
4.77 time -p pixz -d rand.tpxz 858.7002096
0.99 time -p pv rand.tar > /dev/null 4137.373737
8.96 time -p tar -xvf rand.tar 457.1428571
0.99 time -p pv rand > /dev/null 4137.373737
47.97 time -p pixz rand rand.pxz 85.38670002
1.6 time -p pv rand.pxz > /dev/null 2560
20.25 time -p pixz -d rand.pxz rand 202.2716049
16.55 time -p pv -Ss 4G /dev/urandom > rand 247.4924471
14.8 time -p pv rand | pigz > rand.gz 276.7567568
0.58 time -p rm rand 7062.068966
17.09 time -p pv rand.gz | unpigz > rand 239.672323
47.49 time -p tar -I pixz -cvf rand.tpxz rand 86.24973679
0.13 time -p rm rand 31507.69231
20.69 time -p pixz -d rand.tpxz 197.9700338
21.89 time -p tar -xvf rand.tar 187.1174052
43.4 time -p pixz rand rand.pxz 94.37788018
7.44 time -p pixz -d rand.pxz rand 550.5376344
You can see that there is quite a lot of variability in the results.
I’ve ran this muitple times, and the overall trend of the results are generally similar, but the absolute values, with respect to run-to-run variation, can be quite high.
This is running on my server, which is also currently, servicing 7 VMs and 2 containers at the moment. I didn’t stop those services because under the conditions that I will be running my actual data, it would not be on a server that doesn’t have any other services running. So whilst that can be a contributing factor to the run-to-run variation, it also helps to establish the upper and lower bounds of the throughput as well, which might actually better represent that I might see in practice vs. if I had actually shut down the services, only and specifically for these series of tests.
After performing these tests though, two trends emerge:
pigz is generally faster than pixz for both compression and decompression, even without RAM caching.
(I am can’t cache my 7.1 TiB archive file to RAM prior to decompressing it, which is why I ran the tests without the RAM caching as well.)
One of the key advantages that I can think of, in regards to why someone might choose to use pixz over pigz is if they want to extract a specific file and/or a series of files from the archive, pixz is indexable whilst it is my current level of understanding that pigz is not.
Therefore; if you only want to fish out a few files from the archive, you can do so with pixz, whereas; to do the same with pigz, you would have to unpack the whole thing, JUST for those files.
Various use cases would have uses for either tool/application.
For the same test as above, without RAM caching, here are the results for that test
(time,command,data rate (MiB/s))
This is why I end up tending to stick with whatever the default compression ratio is for the tool/application, as I assume that they’ve picked that as the default, hopefully with a fair bit of testing for both data that can be highly compressed vs. data that might not compress as well, if at all.
If I were writing the results directly to tape – yes.
But this is why I tend to prep/stage the data that is to be written, ahead of time, because I have discovered that sometimes, if I were to try and decompress a file directly from tape, the perform would suffer dramatically, because it will read a stream of the archive file for a while, and then it would have to “rewind” and read some more of it and do that a bunch of times. So the net effect data rate can be quite low vs. if I just pull/download to archive file to the “data staging” area, and then unpack it there, I tend to get, overall, better data rate/performance.
And given that I am using rsync to send the data to tape, it is “dumb” in the sense that if won’t know (and probably doesn’t care/doesn’t check) whether there is still available capacity on the tape prior to writing a file, (i.e. no auto-splitting between tapes).
This is why I have to do the splits manually and prepare the “datasets” in 11 TB or 12 TB chunks so that when I issue the command rsync -avrsh --progress=info2 * /tmp/lto8, I usually would’ve already ran the checks to make sure that the data that has been prepared to write to tape, is already in a “chunk” that usually won’t exceed 95% of the tape’s formatted capacity.
The nominal/typical CPU load average, since it’s via “vanilla” NFS (i.e. NOT NFSoRDMA), would be upto 8.00, because of the 8 NFS threads by default.
(It’s really a shame that Debian and by extension, Proxmox, doesn’t support NFSoRDMA because I think that that can help with the CPU load averages, in cases like this. And my LTO-8 tape drive, usually, is only running at around 200 MB/s (1.6 Gbps) over my 100 Gbps Infiniband network, so if the load/demand for data was higher, my Proxmox server would have a hard time serving the data, due in part to NFSoRDMA not being available/supported on Debian.)
mbuffer will greatly improve matters there. Cache a couple gigabytes in RAM, if the buffer gets full (writing tape)/empty (reading tape), tell it not to continue tape read/write until cache is completely full/empty once again. Not to mention getting all the overhead of rsync, NFS (will do network sockets directly) and LTFS entirely out of the equation.
Here’s an example command you can try (with a local tape drive) and see if it doesn’t just work faster than anything you’ve done (the reduced “shoe-shining” will make the drive and tapes last longer, too):
tar -cf - /home | pigz | mbuffer -f -o /dev/nst0 -m 1.9G -P 99 -s 1M -A "echo next tape; read a < /dev/tty"
That’s a bit of a mistake. zstd does NOT use multi-threading by default. You can call it as zstdmt or use zstd -T0 to enable multi-core. That should improve matters.
Also, you shouldn’t just use the default compression level. It ranges from ultra-fast like lz4 to as slow as xz, with compression ratios to match.
In any case, pixz seems too slow to stream to tape, so I would discount it (I haven’t investigated whether it can be streamed/piped), whereas tar + pigz + mbuffer should allow you to stream to tape in realtime.
The syntax of this command would appear that I don’t need to mount the tape in order to be able to write to it. Would that be a true statement? And if so, how would it know where the index begins/ends and where the “data” begins (and ends)?
If I want to download the data from the tape after it has been written – if I am taking LTFS out of the equation altogether, and therefore; not mounting the tape – how would I be able to pull the data off said tape? (I’ve never done it this way, so I would be willing to try it out.)
I use the defaults for this comparison because without knowing the details behind the levels of compression, I cannot guarantee that a -5 with pigz will be the same as a -5 with zstd.
Every time you start/stop writing a file/stream to a tape, it creates a file-mark. mt -f /dev/nst0 status will list which file-mark you are currently on, and allows you to FFW and RWD to the next/previous file mark (fsf and bsf). When you read data back, your tape drive will stop when it hits the file mark. Launch the read command again from there, and you’ll begin reading the next tar file.
It’s common for tape backup software like AMANDA to write a plain text “index” file at the start of the tape with at least the tape ID. With a block-size of 1MB, it can contain as many details about the contents of the tape as you could want, instructions for extracting, etc., without much chance of running out of space. You can keep going back and updating (overwriting) that index file as needed.
Reverse the pipeline, and switch a few options in the commands from write to read…
pigz -11: aborted – was taking in excess of 1490 seconds to compress and it wasn’t even done yet.
Not sure if I am going to test zstd -19 -T0, but as always, whenever I have tried to look this stuff up – it’s always a balance between compression ratio and speed.
Unfortunately, it does not appear that the people who have tested this before, has tested it with pixz, so that makes it difficult to compare (because the dataset that the test is being performed against can make a HUGE difference as well).
edit
The other two problems that I found about pigz are:
It processes the files in the same directory as where they are found, which means that if I tried to dump the compressed results into a single file, it gives me a warning that it will not be able to necessarily recreate the directory/folder structure.
If (1) is true, then it would make calculating the parity for each of the compressed result files more difficult, using par2.
(or it would be more advanced/requires more knowledge than what I currently possess, where it would need to be able to read the files, from where they are stored, and then calculate the parity for each of the compressed file, in the same locations, so that the file, plus its parity information can be all sent to tape)
Speed is one thing. But the ratio also matters (to me, cuz I’m using tape) because a 4% increase in file size CAN (potentially) be the difference between whether something is going to fit on the current tape or if it will spill over to the next tape.
And if the math checks out, that means that if I am writing data to 25 tapes, I’ll need a 26th tape because of this 4% increase in file size (that pigz produces compared to pixz).
That’s not an automatic ‘no’, but that may be important for other people to consider, if other people might be looking at using/deploying some of these solutions.
No formatting needed. You can start writing to brand new blank tapes instantly.
I don’t understand what you are saying here. Unix compression utilities operate on single files, so tar / cpio or other utilities are used to store multiple files and directory structures into a single file to be compressed.
It will very rarely happen that your data aligns exactly with the end of the last tape. 4% larger means there’s only a 4% chance you will need an additional tape in a given set, and on average you will only need 4% more tapes.
Tape is the cheapest storage medium available and trading 4% more tape usage for performing backups 4.3X faster (84.47MB/s pixz vs 360MB/s LTO-8 throughput) is obviously the better trade-off.
Writing more slowly than the maximum speed of the tape and drive greatly increases wear and tear, shortening the life of your very expensive tape drive and the tapes themselves. You could manage this with pixz/xz/zstd if you have enough disk space to hold temporary copies of your backups before writing to tape, on disks which can sustain 360MB/s+ sequential reads, but generally, the better trade-off is slightly faster compression.
You haven’t gotten to the really hard parts, yet. The bigger problem is usually whether your disks perform fast enough that tar can keep your tape drive fed at full speed. With lots of smaller files scattered around the file system, this often becomes a problem. noatime can help. spd_readdir.so (now we’re getting into tape backup esoterica!) helps a lot with ext4/xfs/etc file-systems but is unlikely to help with CoW file systems like ZFS/BTRFS. But perhaps you’ll be lucky.
I am used to the entire archive being a single file, which maintains the file/folder structure, contained within said single-file archive.
Thus, when it comes to calculating the parity for said archive, it acts on the single file.
Stupid question - how would I use par2 to calculate the parity given that it compresses the files where they are located (rather then packing them into a single archive file)?
It is my understanding that the command that you supplied asks for the next tape as part of said command, no?
(Sidebar question: is there a way, using the command that you supplied above, to write to only either 90% or 95% of the tape’s raw capacity?)
Thus, if I understand how the command above works – wouldn’t that write the data as far as the tape can go, therefore; increasing the probability of the extra tape being needed (out of every 25 tapes)?
That will be tested soon.
(The server right now is busy running the pigz -11 test.)
Yeah, my pool B is about 48 TB raw (36 TB raidz2), so it can be temp/scratch space for about a tape and a half, to prep writing to disk.
There is a proposal that I actually back up all of the data on my current Proxmox server, and then once that is done, free up eight of my 10 TB SAS drives again, put them back into my micro HPC headnode (undoing some of the consolidation), so that I would be able to use NFSoRDMA. Probably won’t make much of a difference in the long run, but that’s why it’s a proposal to be assessed and for consideration.
I am not sure if the tape will be able to actually do 360 MB/s given that that’s apparently the rated speed for full height drives, not half-height drives.
I think that whenever I scrub the pool, it can read it at anywhere between 400-500 MB/s, so I think that should be ok.
Will be a part of the test.
I was under the impression that the command that you have provided should help with that by buffering the data, no?
(The system that runs the tape drive has 32 GB of RAM installed, I think, but only a gimped up Core i7-3930K (6-cores originally, but now down to 4-cores, because I think that I killed at least one core, OCing the processor years ago.))
But again, it will all be a part of the test once that gets going.
Yeah…it will depend on how well pigz is going to be able to find and compress/decompress those files in parallel.
The testing will ascertain this (or at least a large part of it).
I don’t recommend par2. LTO-8 tapes already store “two-dimensional interleaved dual error-correction codes.” And they are not prone to bit-rot like hard drives or CD-Rs.
That’s unnecessary. You would not want to use it. It takes longer than xz and doesn’t compress nearly as well. The pigz manual explains this.
I don’t see any reason you’d want to do so, but mbuffer has a -D option that could work.
Your last tape in any set (however many tapes it may be) is always partially empty, never 100% full.
Gotcha (because by default, with both zstd and pigz, by themselves, neither of them will retain the file/folder structure.
Yeah, I read through that, and the TL;DR conclusion appears to be that it may not be possible because par2 needs to know the total file size in order to be able to compute the parity, in its current implementation.
I’ve already had data get corrupted on my tapes before, and was able to successfully recover it using par2, despite the fact that there is supposed to be checksum being performed on the tape itself. But that didn’t throw an error when it was writing the data nor when it was reading the data, and the tpxz archive was tested for consistency/verified prior to writing to tape and it passed those tests.
Yes, but I wanted to see how close it will get to the file size that pixz is able to achieve in the speed vs. compression ratio optimisation.
I read, years ago, that the general recommendation is to not write to the tape past the 90% mark for optimal performance.
This is true, if the number of tapes in a given set = the relative increase in size.
(i.e. In the statements that I’ve made above where I said that with a 4% increase in file size, that you will need an extra tape per 25 tapes, whilst your statement is true, there is also a potential that the driver behind this remark is an implicit assumption that the set of tapes is n=26 in total, where the 26th tape will only be partially full. But, that’s not quite exactly what I said though.
What I said was:
which means that if you took 37 tapes to write all of the data, with 4% increase in file size, it means that you could’ve written the same total volume of data to 36 tapes, with the 36th tape being only about 52% full. So yes, your statement is correct, but that doesn’t change the fact that you would still need an extra tape given said 4% size increase.)
In regards to this method:
would you be able to read only a single file from the backup?
And given this method - what would be a downside to using mbuffer vs. using LTFS?
Or asking the same question in a slightly different way - why would you use LTFS instead of mbuffer?
It sounds like you hit a bug in rsync or your LTFS implementation.
[LTO] tape drives use a strong error correction algorithm
when data is written to the tape it is verified by reading it back using the read heads that are positioned just “behind” the write heads Linear Tape-Open - Wikipedia
LTO tapes certainly do not slow down 5% from the end. They are written from the start to end of the tape, and then in reverse, thousands of times (see: “Total tracks” on LTO-8) to get that 12TB on them. I see no technical reason the last 5% would be any different than the first 5%.
I don’t recall any such issues with DAT tapes, but that was a while back and I can’t distinctly remember watching throughput while I wrote a 100% full DAT tape.
As I said the first time:
a 4% chance you will need an additional tape in a given set, and on average you will only need 4% more tapes.
Tape is the cheapest storage medium available and trading 4% more tape usage for performing backups 4.3X faster (84.47MB/s pixz vs 360MB/s LTO-8 throughput) is obviously the better trade-off.
I suppose it helps in the world of Windows users, where tar and cpio weren’t common/standard utilities included (and zip is unsuitable for tape), so several different proprietary file formats for tape backups were created and compete for dominance.
LTFS serves to make tape seem more user-friendly for the uninitiated. But in exchange it has plenty of pitfalls, such as operations like deleting files on LTFS doesn’t free up space like people would expect.
yay!
You’ll see/figure out why I build an index file.