"cannot receive incremental stream: checksum mismatch or incomplete stream" when using zfs send -I

[Haravikk]

I recently switched a backup script from using zfs send -i with a bookmark, to instead use zfs send -I (capital i) with a snapshot if one exists, to ensure that any intermediate snapshots are also sent.

However I'm now occasionally getting the following error from zfs receive:

cannot receive incremental stream: checksum mismatch or incomplete stream

Nothing else has changed, so the logic is essentially the following (zsh, heavily simplified):

declare -a send_options=()

# If snapshot of the same name exists and has a matching GUID, it is the snapshot for this bookmark and we can use it
if [[ "$(zfs get -Hpo value guid "${dataset}/@${last_name}" 2>/dev/null)" = "$(zfs get -Hpo value guid "${dataset}/#${last_name}" 2>/dev/null)" ]]; then
    send_options+=( '-I' "@${name}" )
else
    send_options+=( '-i' "#${name}" )
fi

zfs send -pw "${send_options[@]}" "${dataset}/@${new_name}" | zfs receive -us "${target}"

Any ideas how I might debug why this change would be causing this problem? I'm running v2.3.0 currently as I tend to be slow with updating, but I don't see anything in the changelogs since then suggesting this has been fixed (assuming it's a bug).

Both the source and target pool are on the same system, so there should be no transmission issues, and both are in good health and showing no sign of errors, so it seems unlikely for this to be either a checksum mismatch or interrupted stream — since I'm using resumable receive I can immediately resume the send using the resume token and it completes without any issue, as does disabling the new behaviour (only use -i) but I'd prefer to fix the problem if I can.

It just seems like switching from -i to -I is causing the problem, but I would expect them to be functionally identical when the incremental stream is only for a pair of snapshots (none in between) so it's a mystery why one works all of the time and the other intermittently fails.

And thus far in my usage it does seem as if the failures are more likely when there are no intermediate snapshots, and the amount of data being sent is relatively small — larger sends, and sends with intermediates, seem to be reliable.

[Haravikk]

So earlier in the week I finally got around to adapting my script to insert a tee between the zfs send and zfs receive, so I can dump some sends to files to see how big they are (or if they're corrupted somehow). However since doing that I've encountered no more errors!

I'm only dumping sends with an estimated size of less than a hundred megabytes (for sanity) but it seems weird to me that adding tee in the middle would work at all — presumably tee adds some amount of buffering that might be helping, but why would that be necessary for zfs send -I but not for zfs send -i?

Now I'm at even more of a loss as to how to debug it. 🤦‍♂️

I'm going to keep an eye on it, maybe I just got "lucky" and I've had a week of flawless sends, but prior to adding tee I'd get at least 2 or 3 a day on an hourly backup.

[Haravikk]

Of course I posted too soon! Less than 24 hours later I get one failure, resulting in a send stream written to file that I can inspect.

I'm not really familiar enough to know what's going on within the file, but what I did notice is that for an estimated 2.04 mb send, I've only got a 999 kb file, which seems to suggest an incomplete stream though I'm unsure how to verify that, plus it could still be disconnecting on the receive side.

Is it worth forcing a buffer in the middle with pv or dd or mbuffer or similar? Could it be some kind of buffering (or lack of) related issue? Still at a loss why -I or -i makes a difference to this case though.

I'm attaching the send output in case anyone can learn more from it (it's an incremental send for an encrypted dataset that isn't important so I'm not too bothered about posting it):
zbackup_isimms_Games_2026-01-26-141059.126384000.send.zip

The command I use to get an estimated transfer is zfs send -Pnvpw -I <last_snapshot_sent> <new_snapshot> and the transfer is then zfs send -pw -I <last_snapshot_sent> <new_snapshot>

[IvanVolosyuk]

I had some kind of corruption which I believe was caused by inconsistent use of -L and untested code doing that. #17829 explained here.

[Haravikk]

Thanks for the response! That does seem to be similar as what I'm seeing, my output from zstreamdump suggests a checksum error of some kind:

BEGIN record
	hdrtype = 2
	features = 0
	magic = 2f5bacbac
	creation_time = 0
	type = 0
	flags = 0x0
	toguid = 0
	fromguid = 0
	toname = zbackup/isimms/Games@backup-2026-01-26-140004
	payloadlen = 796
nvlist version: 0
	fromsnap = backup-2026-01-26-130007
	tosnap = backup-2026-01-26-140004
	not_recursive = 1
	raw = 1
	fss = (embedded nvlist)
	nvlist version: 0
		0xce6222aefdede856 = (embedded nvlist)
		nvlist version: 0
			name = zbackup/isimms/Games
			parentfromsnap = 0x0
			props = (embedded nvlist)
			nvlist version: 0
				recordsize = 0x100000
				keylocation = prompt
			(end props)

			is_encroot = 1
			snaps = (embedded nvlist)
			nvlist version: 0
				backup-2026-01-26-130007 = 0x2b84a4a587d781cc
				backup-2026-01-26-140004 = 0xec4214fd99ce3b34
			(end snaps)

			snapprops = (embedded nvlist)
			nvlist version: 0
				backup-2026-01-26-140004 = (embedded nvlist)
				nvlist version: 0
				(end backup-2026-01-26-140004)

			(end snapprops)

		(end 0xce6222aefdede856)

	(end fss)

END checksum = 22d8a51b91/1064c36bfd78/54ed97b44851f/15c9ca51c507d25
BEGIN record
	hdrtype = 1
	features = 3ca0004
	magic = 2f5bacbac
	creation_time = 69777364
	type = 2
	flags = 0xe
	toguid = ec4214fd99ce3b34
	fromguid = 2b84a4a587d781cc
	toname = zbackup/isimms/Games@backup-2026-01-26-140004
	payloadlen = 1028
nvlist version: 0
	crypt_keydata = (embedded nvlist)
	nvlist version: 0
		DSL_CRYPTO_SUITE = 0x8
		DSL_CRYPTO_GUID = 0xe922eec8d40fbd2
		DSL_CRYPTO_VERSION = 0x1
		DSL_CRYPTO_MASTER_KEY_1 = 0xf8 0x6e 0xac 0x2b 0x9d 0xb8 0xca 0x11 0x4e 0x8c 0x46 0xd9 0x59 0x73 0xee 0x20 0xc8 0xfc 0xca 0xb8 0xc1 0x4 0x12 0x97 0x53 0xb3 0xbc 0xb8 0x85 0x1 0x3c 0x8b
		DSL_CRYPTO_HMAC_KEY_1 = 0x3a 0x4a 0xa6 0x8e 0x7c 0x97 0xe5 0xbf 0x1b 0xf3 0x1c 0xbc 0xf 0x75 0x8e 0x30 0xf7 0xdc 0x5 0xd2 0x71 0xbe 0x3d 0x75 0x50 0x69 0xb6 0x2c 0x83 0x59 0x48 0xaa 0x52 0xaa 0x44 0x68 0xa1 0x72 0x7b 0x96 0x59 0x35 0x4e 0xc6 0x8c 0xdb 0x10 0xd3 0xf2 0xc4 0x8b 0x3 0xfd 0x20 0xf5 0x62 0xdd 0x8 0x8d 0xa8 0x8d 0x43 0x5e 0x6
		DSL_CRYPTO_IV = 0xfc 0x69 0x64 0xff 0x7b 0x16 0x66 0xfa 0x6a 0x78 0xbf 0x2d
		DSL_CRYPTO_MAC = 0x4e 0x25 0x5d 0x76 0x21 0xec 0x95 0xa9 0xa 0xb5 0xcc 0xbc 0xed 0xbe 0x6e 0x15
		portable_mac = 0xdb 0x1d 0x7a 0x14 0x8d 0xaa 0xcd 0x7e 0x4c 0x6 0x3f 0x1f 0xb1 0x56 0x62 0x38 0x9c 0x6f 0x41 0x3f 0x64 0x73 0x5c 0xfd 0x5f 0x23 0x42 0x82 0xbb 0xfa 0xb8 0x9e
		keyformat = 0x3
		pbkdf2iters = 0x55730
		pbkdf2salt = 0x7e887e213cdcb4c
		mdn_checksum = 0x0
		mdn_compress = 0x0
		mdn_nlevels = 0x6
		mdn_blksz = 0x4000
		mdn_indblkshift = 0x11
		mdn_nblkptr = 0x3
		mdn_maxblkid = 0x698
		to_ivset_guid = 0xf478fdb879dada
		from_ivset_guid = 0x831a3239d15c8a
	(end crypt_keydata)

Expected checksum differs from checksum in stream.
Expected checksum = 12bdd7cfadd01/3b773eabed69602e/dd968a6896733bc6/c702fe0a69f0f583
END checksum = 0/0/0/0
invalid checksum
Incorrect checksum in record header.
Expected checksum = 386192bab/dc8dfb835a/1b66d75d910f/24e574f2ae683
SUMMARY:
	Total DRR_BEGIN records = 2 (1824 bytes)
	Total DRR_END records = 2 (0 bytes)
	Total DRR_OBJECT records = 648 (535808 bytes)
	Total DRR_FREEOBJECTS records = 59 (0 bytes)
	Total DRR_WRITE records = 6 (10240 bytes)
	Total DRR_WRITE_BYREF records = 0 (0 bytes)
	Total DRR_WRITE_EMBEDDED records = 0 (0 bytes)
	Total DRR_FREE records = 666 (0 bytes)
	Total DRR_SPILL records = 0 (0 bytes)
	Total records = 1432
	Total payload size = 547872 (0x85c20)
	Total header overhead = 446784 (0x6d140)
	Total stream length = 994968 (0xf2e98)

The key problem seems to be the line END checksum = 0/0/0/0 which is invalid, though it might still indicate an interruption (maybe no checksum was generated because not all data was received)?

I'm not actually using the -L option (that was a mistake in the size estimation command I posted, which doesn't need it) although large blocks is probably being implied by -w since it will be sending raw records that can be larger than 128K.

Skimming back through my logs it does look like the datasets seeing errors are all ones with recordsize=1M, I'll need to look further back to be sure but those with recordsize=128K (the default) do not appear to have been affected.

I don't believe I have ever used -L explicitly, since all of my datasets are encrypted I've always sent them raw (-w) to avoid decryption, but if it's implied by -w then I guess I've technically always been using it.

Still a bit confusing why this would be an issue only when I'm using the -I option though, as it seems like you were encountering it when using -i which has never caused any problems for me. Like you I've had no errors when scrubbing affected datasets (everything seems to verify as correct), I'll need to test a receive from backup to see if I encounter the same crashes that you did.

[no-usernames-left]

Did you really just paste your crypto keys to GH?

[Haravikk]

As I said in the post where I attached the send stream, it's not a dataset that I particularly care about and it's an incremental send and it doesn't really contain anything (just some search indexes being updated) — at worst you'd need access to my machine to do anything with the key, and if you had that you wouldn't need the key anyway.

Plus, unless I've misunderstood how it works completely, it's the encrypted master key, so still requires a passphrase or key file to be used for decryption. One of the benefits of a raw (-w) send is supposed to be that it allows sending to an untrusted source, but it does still include the encrypted key so it can be mounted by someone with the passphrase/key-file either on the target, or after being sent back.

TL;DR
It should be fine actually and also I don't care. 😉
Plus I'd rather anyone that can potentially help has a send stream they can look at for clues.