Skip to content

Implement FIDEDUPERANGE via block cloning#18745

Draft
MorganaFuture wants to merge 6 commits into
openzfs:masterfrom
MorganaFuture:fideduperange
Draft

Implement FIDEDUPERANGE via block cloning#18745
MorganaFuture wants to merge 6 commits into
openzfs:masterfrom
MorganaFuture:fideduperange

Conversation

@MorganaFuture

@MorganaFuture MorganaFuture commented Jul 5, 2026

Copy link
Copy Markdown

Motivation and Context

Closes #11065.

Linux's FIDEDUPERANGE ioctl lets out-of-band dedupe tools (e.g. duperemove,
bees) ask the filesystem to share byte-identical file ranges. On ZFS the ioctl
returned EOPNOTSUPP. This implements it on top of block cloning (BRT), so
matching ranges share storage without the memory cost of the dedup table.

Description

  • Split zfs_clone_range() into a zfs_clone_range_precheck() and a
    zfs_clone_range_locked() helper so the dedupe path can reuse the locked
    clone loop.
  • Add zfs_dedupe_range(): it compares [inoff, inoff+len) and
    [outoff, outoff+len) through dmu_read() and, only if they are
    byte-for-byte identical, clones the source blocks over the destination — all
    under a single RL_READER/RL_WRITER rangelock hold, so the bytes that get
    cloned are exactly the bytes that were compared.
  • Wire zpl_file_range.c: the remap_file_range(REMAP_FILE_DEDUP) path (>=4.20)
    and the pre-4.20 dedupe_file_range fop. -EBADE maps to
    FILE_DEDUPE_RANGE_DIFFERS; REMAP_FILE_CAN_SHORTEN is honored.
  • A dedupe leaves the destination content unchanged, so it does not update
    mtime/ctime or strip setid bits (matching the Linux REMAP_FILE_DEDUP
    convention where file_modified() is skipped).
  • The full requested range is compared before it is aligned down to a block
    boundary, so a differing trailing sub-block reports DIFFERS rather than being
    silently rounded away and reported identical.

How Has This Been Tested?

New block_cloning ZTS cases: whole-file dedupe shares blocks; differing files
fail and change nothing; single-block partial dedupe; feature-disabled; the
destination's mtime/ctime stay unchanged after a dedupe; and a sub-block
request whose bytes differ reports DIFFERS.

Ran the block_cloning group on Linux (Ubuntu, aarch64, --enable-debug) — the
fideduperange tests pass and the existing clone tests are unaffected. BRT
sharing was confirmed via get_same_blocks.

This is a draft — a couple of points I'd like reviewer input on before
polishing:

  • zpl_dedupe_file_range_impl() takes the two inode locks without ordering them
    by address (the existing zpl_clone_file_range_impl() has the same pattern);
    concurrent opposite-direction dedupes could deadlock. I'm happy to add an
    address-ordered two-inode lock helper shared by both paths.
  • A dedupe logs an ordinary TX_CLONE_RANGE, so a crash+replay would restamp
    the destination mtime/ctime (inherited from the flag-less record). Worth a
    dedicated log-record flag?

Types of changes

  • New feature (non-breaking change which adds functionality)
  • Quality assurance (non-breaking change which makes the code more robust against bugs)

Checklist:

  • My code follows the OpenZFS code style requirements.
  • I have updated the documentation accordingly.
  • I have read the contributing document.
  • I have added tests to cover my changes.
  • I have run the ZFS Test Suite with this change applied.
  • All commit messages are properly formatted and contain Signed-off-by.

@MorganaFuture
zfs_vnops: Split zfs_clone_range() into precheck and locked helpers
9db604d 
Factor the offset-independent checks of zfs_clone_range() into
zfs_clone_range_precheck(), and the range-locked clone loop into
zfs_clone_range_locked().  This is a pure refactor with no functional
change: zfs_clone_range() runs the same checks in the same order and
clones identically.

The split lets an upcoming FIDEDUPERANGE implementation reuse both
halves and run a byte comparison under the same range locks, instead
of duplicating the clone machinery.

Signed-off-by: MorganaFuture <103630661+MorganaFuture@users.noreply.github.com>
@MorganaFuture
zfs_vnops: Add zfs_dedupe_range()
5687221 
Add zfs_dedupe_range(), the common back end for the Linux FIDEDUPERANGE
ioctl.  It validates the request with FIDEDUPERANGE semantics (the range
must lie within the source, must not extend the destination, and may be
shortened to the destination EOF and to a block boundary only when the
caller allows it), takes a reader range lock on the source and a writer
range lock on the destination, and compares the two ranges byte for byte
via the DMU.  If they match it clones the source blocks over the
destination with zfs_clone_range_locked() without dropping the locks, so
the blocks that get shared are exactly the bytes that were compared;
concurrent writers, which also take the destination writer lock, cannot
change the data in between.  If the ranges differ nothing is cloned and
the caller is told so.

The comparison reads committed data through dmu_read(), the same data
zfs_clone_range_locked() clones, and dirty blocks are handled by the
existing block-cloning EAGAIN/txg-wait path.

Signed-off-by: MorganaFuture <103630661+MorganaFuture@users.noreply.github.com>
@MorganaFuture
Linux: Implement FIDEDUPERANGE via block cloning
380c206 
Route the FIDEDUPERANGE ioctl to zfs_dedupe_range().  Both the modern
remap_file_range(REMAP_FILE_DEDUP) path and the pre-4.20
dedupe_file_range fop now call a shared zpl_dedupe_file_range_impl(),
which takes the inode locks, invokes zfs_dedupe_range(), and maps the
result: the number of bytes deduped when the ranges match, -EBADE when
they differ (which the VFS reports as FILE_DEDUPE_RANGE_DIFFERS), or
the negative errno on failure.  REMAP_FILE_CAN_SHORTEN is passed
through so the modern interface may shorten the request to the
destination EOF or a block boundary.

This lets out-of-band dedupe tools such as duperemove share identical
blocks on ZFS through block cloning, without the memory cost of the
DDT.

Signed-off-by: MorganaFuture <103630661+MorganaFuture@users.noreply.github.com>
Closes openzfs#11065
@MorganaFuture
ZTS: Add FIDEDUPERANGE tests
7283ce1 
Add block_cloning tests for the FIDEDUPERANGE ioctl: whole-file dedupe
shares every block, differing files are left untouched, a partial
request shares only the matching block, and the ioctl fails when block
cloning is disabled.

Signed-off-by: MorganaFuture <103630661+MorganaFuture@users.noreply.github.com>
@MorganaFuture
zfs_vnops: harden block-clone and FIDEDUPERANGE paths
005608a 
Three issues found while reviewing the FIDEDUPERANGE series:

- zfs_clone_range() and zfs_dedupe_range() cached outzfsvfs->z_log in
  their declarations, before zfs_enter_two().  A concurrent
  suspend/resume (zfs rollback, zfs receive -F) tears down and
  recreates the zilog, so a thread that read the pointer early could
  zil_commit() a stale or freed zilog after it finally entered.  Read
  z_log only once inside the entered region, like
  zfs_clone_range_locked() already does.

- A dedupe does not change file content, yet it reused
  zfs_clone_range_locked() and so bumped the destination's mtime/ctime
  and stripped its setid bits.  That contradicts the Linux
  REMAP_FILE_DEDUP convention, where __generic_remap_file_range_prep()
  skips file_modified(), and breaks tools that rely on dedupe being
  invisible (rsync/backup incrementals, suid binaries).  Pass a dedup
  flag into the locked helper so it leaves the timestamps and setid
  bits alone.

- zfs_dedupe_range() aligned the request down to a block boundary
  before comparing it.  With a large recordsize a sub-block request
  rounds away to nothing and the ranges were then reported identical
  without any bytes being compared, so FIDEDUPERANGE could answer SAME
  for data that differs.  Compare the whole requested range first, then
  align it down for the clone; a differing trailing sub-block now
  correctly reports DIFFERS.

Signed-off-by: MorganaFuture <103630661+MorganaFuture@users.noreply.github.com>
Closes openzfs#11065
@MorganaFuture
ZTS: Cover FIDEDUPERANGE metadata and sub-block behavior
f2e5543 
Add two block_cloning tests for the corrected FIDEDUPERANGE behavior:

- block_cloning_fideduperange_mtime: a successful dedupe shares the
  blocks but leaves the destination's mtime and ctime unchanged.

- block_cloning_fideduperange_subblock: a dedupe request smaller than
  the record size whose bytes differ must fail (DIFFERS) and share
  nothing, rather than being silently rounded away and reported as
  identical.

Also mark block_cloning_disabled_fideduperange.ksh executable; it was
committed 0644 and so would not run under the test harness.

Signed-off-by: MorganaFuture <103630661+MorganaFuture@users.noreply.github.com>
Closes openzfs#11065
@github-actions github-actions Bot added the Status: Work in Progress Not yet ready for general review label Jul 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

Status: Work in Progress Not yet ready for general review

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Support for FIDEDUPERANGE

1 participant

@MorganaFuture