I’m trying to backup btrfs volume from workstation to NAS basically by doing btrfs send / receive over netcat. But transfer randomly stops at various points and I’m struggling to reliably transfer whole volume. It’s around 10TB of actual send/receive stream but operation freezes typically after few terabytes… And since I’m not aware of resume operation I’m not sure how to tackle this problem.
I use ssh, but so far not for such large volumes, and over a very fast local network.
Do you have any idea why it freezes? Network interruptions?
Perhaps piping through something like mbuffer could work, or going through a vpn connection?
If you have plenty of free space you could btrfs send into a file (perhaps compressed), rsync the file over with —partial so you can resume, and btrfs receive out of the file on the other end?
ps: someone wrote a tool for this GitHub - AmesCornish/buttersink: Buttersink is like rsync for btrfs snapshots
If you want to do this kind of stuff periodically with snapshots btrbk is a nice solution, though it wouldn’t help for resuming a single transfer.
Pps: if there is no obvious error check journals on both ends. With btrbk I had rather hard to debug issues related to selinux at some point (labels existing on source but not destination) causing transfers to fail! Also use the verbose option on both ends to get possible error messages.
2 Likes
It’s 10G LAN between workstation and NAS. I’m not 100% sure but this network works 99% of time and in 1% it just faceplants and my NFS drops to 1 MB/s because potato. Then I unmount nfs, mount again and boom it’s 10G again. I think there’s something fishy going on and since it’s VERY LONG operation on single tcp tunnel made by netcat I believe it just happens to randomly faceplant and clog up.
I considered dumping btrfs send to file like you said, even over NFS since NFS seems to be more robust and then could do restore locally. And there’s ton of space on NAS. It’s one-off operation of volume initialization. From now on I’m rather planning to do incremental smaller backups below 500gb.
1 Like
could torrent it…
Why netcat and not SSH, using port tunneling? If it’s a one-time setup, how about rsync (which would be over SSH)?
I hope that buttersink helps smooth this road-bump.
K3n.
It’s just the initial snapshot. Rsync will do, albeit slowly…every snapshot after than (well, because incremental) won’t be a problem anymore and will fine just fine.
I’d try with mbuffer just in case. I’ve been using that in pipes for multi-TB ZFS before, but never BTRFS streams, although both are serialized and should be similar.
1 Like
I think the root cause of your issues is somewhere else. E.g. the remark about nfs shouldn’t have a place here.
I’m using btrbk for remote backup of btrfs via send/receive functionality (over ssh). Maxes out 10gb link and works very reliably.
The entire matter isn’t trivial. I saw this with S3 put as well. multi-TB stuff over hours and days with varying quality of network, can just hang or terminate the stream. As they are all serialized without back and forth communication (fast and latency doesn’t matter), but this also comes with caveats and limited resume ability out of the box.
That btrbk looks nice judging from the github feature list. I keep that in mind, although my BTRFS stuff is using 10Gbit and that’s done fast in a good network and never had problems. WAN on the other hand is a mess…SSH alone with send/receive can halt abruptly. a lot of piping and mbuffer works for me so far. And additional pipe through zstd to keep transfer well compressed and smaller.
1 Like
Where did WAN come into this discussion? I assume “btrfs send / receive over netcat” is done in a LAN, I hope in 2024 nobody considers unencrypted data transfer in WAN…
I was just talking about WAN. Halted/terminated transfers are far more expected there. OP is using LAN and direct connection at whatever speed between two devices. But at e.g. 1Gbit…10TB take a long time…hours or maybe a day
2 Likes
Turned out that for some reason netcat was not terminating after finishing transfer. And that’s it. It was sending less than actual data because I was sending compressed stream thus I thought it’s incomplete transfer. Btrfs send / receive alone was finishing correctly and when I ran it with more debug (-vvv) it displayed success info at the end… and hanged after that. But it did set received subvolume id and according to devs on #btrfs irc it’s the very last thing performed during send/receive so if it’s done then it means everything is okay.
I have no idea why netcat doesn’t terminate for me after finishing transfer. I believe it should but… dunno.
So is it resolved by using ssh instead?
ssh tunneling doesn’t terminate by design so I would not expect it to terminate in that case so well… in a sense yeah it’d solve the problem xD