I’m running two instances of TrueNAS community edition. One relatively high-powered machine (source), running some containers and virtual machines, and a really basic one (target) that I replicate some important data to periodically; or well, I used to.
I had a replication task (which are based on zfs send) set up on the target machine, to PULL the data when it’s online. I intend to power the target device down whenever it’s not replicating.
This worked fine for several months, until it suddenly did not. I think the problem started when I had a drive in the source’s data vdev go offline unexpectedly. I recovered without issues: brought the drive back online, resilvered, no data was reported as lost at all.
However, since then I’m getting this error within maybe 20 minutes of starting the replication:
[EFAULT] resume token contents: nvlist version: 0 object = 0x3 offset = 0x12d7c0000 bytes = 0x12e13bf7c toguid = 0xe12b27047ecd28d1 toname = bulk/backups/mydataset@auto-2025-08-09_10-30 compressok = 1 rawok = 1 client_loop: send disconnect: Broken pipe cannot receive resume stream: checksum mismatch or incomplete stream. Partially received snapshot is saved. A resuming stream can be generated on the sending system by running: zfs send -t 1-<removed>.
If I deleted the offending snapshot, I will get the same error with another snapshot.
What I’ve tried:
- Multiple (5 or 6) scrubs on the source pool. It comes up clean every time.
- Scrubbing the target pool.
- Nuking the dataset on the target, as well as the replication task and starting over. Same result.
- Cleaning up most snapshots on the source. As mentioned the same problem occurs with a different snapshot.
- Upgrading the target from TrueNAS Scale 24.04.2.5 → 24.10.x → 25.04.2.4 → 25.04.2.5
- Eventually upgrading the source system from 24.04.2.5 → 25.04.2.4 → 25.04.2.5 (later, because I had k8s stuff to convert)
- Probably stuff I forgot to mention
I don’t think I really have enough spare storage to completely recreate the source pool on temporary disks, so I’ve held off on that for now. If ZFS reports everything to be okay, there should be no reason for me to have to do that, right? 
I have no other ideas to try at this point, though. Any suggestions?
PS: I tried asking on the TrueNAS forums before, no responses unfortunately.
Are you / Is TrueNAS doing replication with zfs send -t | zfs receive -s for resumable streams? The resume token contents in your error makes me think so.
If so, then on the receiving side, run zfs receive -A on the affected dataset:
From zfs-receive man-page :
zfs receive -A filesystem|volume
Abort an interrupted zfs receive -s, deleting its saved partially received state.
edit: Hmm, " * Nuking the dataset on the target, as well as the replication task and starting over. Same result." - well that should have taken care of that - I’m not so sure now.
Unfortunately I don’t know the exact commands it’s using under the hood, but that’s part of the reason why I started with a blank dataset.
TrueNAS uses ZFS Send / Receive either over SSH or NETCAT depending on how you set it up. It does use the resume tokens but prior to version 25.10 I found that to be a big buggy.
On the system you are pulling the snapshots from you can run
zfs get -r receive_resume_token poolname/dataset
to see if there are any tokens there and you can clear them using
zfs receive -A poolname/dataset
I have had to do that a few times when replications have broke. Also inside the ZFS replication task is a recreate from scratch option which will rebuild the entire replication, but use with caution because that may take quite a while as it will clear and resend everything.
3 Likes
@lawrencesystems:
The zfs get -r receive_resume_token bulk/backups shows like 476 resume tokens. However, the zfs receive -A does not remove them:
zfs receive -A bulk/backups
'bulk/backups' does not have any resumable receive state to abort
I read elsewhere that I’m supposed to run it on the target instead, but that doesn’t work either. It also didn’t make a lot of sense to me…
There are additional datasets in bulk/backups, so I tried with those names as well, same result though.
If the backup machine can be reinstalled, you could try a different OS with ZFS support, like Proxmox and send snapshots from TrueNAS → Proxmox. That’s how I do my backups.
You’d have to configure PUSH tasks from source server.
This would at least confirm if the issue is specific to TrueNAS or not.
Hi! Just chiming in, @lawrencesystems ‘ command to list which datasets have resume tokens helped a lot, since zfs receive -A only takes the specific dataset for me, not any ones above. After aborting the specific sub-dataset, my replication seems to run again. Thanks!
2 Likes