I upgraded from LTO-4 to LTO-6 and it ended up being a good time (read: Saturday ADHD time sink) to see if I’ve been using Bacula correctly. I used to have 2 Jobs that were pointed at 2 Pools. That way I could have 2 tape batches that I could swap between my house and an external location every so often.
My main array on my NAS is those cool Seagate dual-actuators (back when NewEgg had a ton of refurbished ones). Just 2 of those in a ZFS stripe/mirror (leveraging the dual actuators) so it’s kind of like a 4 drive RAID10 of sorts. It’s fast, but it also does a lot of work (since my NVR and Emby run on it and they might be writing videos and what not).
To get around that, I was spooling to an NVMe. This works great except Bacula is single-threaded per job when it comes to spooling.
So this time, I decided to try to split my big job up into smaller jobs and have them run concurrently. In this way, one job is spooling while the other is despooling. Doing tests it does seem to work well! So far, I’m up to 3 jobs or so as I continue to carve out what was my original huge Jobs.
I was curious if this is about what other folks are doing with a similar setup? Anything I might be missing here?
Update on this. I’ve kept the smaller job sets for now - it does seem easier to manage and I can run more important jobs more often if I want. For instance I don’t need to backup my photos or home videos as often as say my public web server.
While NVMe is fast, I’m also now trying to pivot to using my internal large backup HDD I use for rsync’s as a spooler. Since it’s a single HDD, I dropped back to single concurrency but a much larger spool size (4TB). I was scrambling around this morning looking for old hard drives I swear I have but can’t find to see about making an external JBOD which I could use as a spool for large jobs where I could have higher concurrency.
I also opted to turn off LTO6’s built-in compression and instead use zstd. At a level of 3 this required switching from powersave to the ondemand scheduler since it looks like bacula-fd is fairly single threaded (at least per job) and the CPU wasn’t able to keep up with the I/O without being able to clock up.
Using the large backup drive kinda worked but was the wrong extreme which I should have figured out. There’s a balance between small spools and big spools. I switched back to using 175GB spool sizes with a concurrency of 2 and that seems to be working well. I’m going to try to see if I can use a SATA SSD for this so I can avoid churning through data on my NVMe’s.
I’d still like to try a striped setup of HDDs for spooling just for fun but that I think is ultimately silly.
First, be wary of “new” tapes from certain vendors on eBay and other places. I ordered 10 tapes which were listed as new. I noticed something was up when I was hearing the tape drive having to re-attempt writes and errors with end of tape. sg_read_attr /dev/nst0 is what is used to read the flash chip on the cartridge and that will show a number of good info (notably the number of tape loads). Some of these had over 400 load cycles. eBay had to step in but I was able to do a return/refund though it’s clearly a scam and eBay opted not to really do much about it, which is quite frustrating.
I ended up ordering tapes from NewEgg (making sure they were shipped and sold by NewEgg) and these appear to be new according to their flash chip and did not exhibit the same re-read errors. Shame on shady vendors.
Anyways! With that out of the way, I settled on having a number of Jobs which can run simultaneously. I’m spooling the data to a SATA SSD in 80GB chunks which seems to be a good number for allowing jobs to spool while one job is despooling. This keeps the tape busier. I also opted to use ZSTD3 instead of built-in LTO compression (which is another reason for wanting to spool first instead of directly feeding the tape).
This all seems to work well! I’m using Bacula as that is what I am used to. I did look at BAREOS but opted to stick with Bacula for now. I would like to have Prometheus metrics and may figure out how to do that via console wrapper commands (BAREOS may have helped here) but wanted to stick with MySQL as the backend DB store as I didn’t want to have to stand up PostgreSQL just for tape backups (honestly SQLite is probably sufficient for my needs).
Pretty happy with the upgrade! I was at 5 LTO4 tapes and now I’m at 2 LTO6 for roughly the same amount of data. LTO7 would have gotten me down to 1 but the drive costs are still a bit out of reach at the moment and this still is an overall improvement. Especially since I went down from a big chonky external drive to a nice slim single 5.25” internal.
Overall, minus the sketchy vendors, it’s been a great success!