mkfs.btrfs: support reproducible image creation - #1126
Conversation
boryas
left a comment
boryas
left a comment
There was a problem hiding this comment.
Mostly looks good to me. Some little nit picks and a few questions here and there. Would be good to just do a general once over on the string processing stuff to make sure that it follows the same most common conventions in the rest of btrfs-progs.
Thanks!
| struct path_walk_entry *new_entries; | ||
|
|
||
| cap = cap ? cap * 2 : 16; | ||
| new_entries = realloc(entries, |
| } | ||
| entries = new_entries; | ||
| } | ||
| entries[count].name = strdup(de->d_name); |
There was a problem hiding this comment.
I think this is fine cause the dirent.name is null-terminated, but I think it has a bound so you could consider using strndup. no big deal, to me.
| /* Need room for "/" + name + terminator after path_len. */ | ||
| if (path_len + 1 + name_len + 1 > path_buf_size) { | ||
| ret = -ENAMETOOLONG; | ||
| goto cleanup; | ||
| } | ||
| path[path_len] = '/'; | ||
| memcpy(path + path_len + 1, entries[i].name, name_len + 1); | ||
|
|
There was a problem hiding this comment.
common/path-utils.c has a path_cat_out that I think does what you want.
| void reproducible_uuid_generate(const u8 fs_uuid[BTRFS_UUID_SIZE], | ||
| enum reproducible_uuid_role role, u64 key, | ||
| u8 out[BTRFS_UUID_SIZE]) | ||
| { | ||
| u8 input[BTRFS_UUID_SIZE + sizeof(u32) + sizeof(u64)]; | ||
| u8 digest[32]; /* SHA-256 output */ | ||
|
|
||
| if (!reproducible_is_deterministic()) { | ||
| uuid_generate(out); | ||
| return; | ||
| } | ||
|
|
||
| memcpy(input, fs_uuid, BTRFS_UUID_SIZE); | ||
| put_unaligned_le32((u32)role, input + BTRFS_UUID_SIZE); | ||
| put_unaligned_le64(key, input + BTRFS_UUID_SIZE + sizeof(u32)); | ||
|
|
||
| if (hash_sha256(input, sizeof(input), digest) < 0) { | ||
| error("SHA-256 failed deriving deterministic UUID"); | ||
| exit(1); | ||
| } | ||
|
|
||
| memcpy(out, digest, BTRFS_UUID_SIZE); | ||
| /* RFC 4122 v4: set version (high nibble of byte 6) and variant | ||
| * (high two bits of byte 8) so the result identifies as v4. */ | ||
| out[6] = (out[6] & 0x0F) | 0x40; | ||
| out[8] = (out[8] & 0x3F) | 0x80; | ||
| } |
There was a problem hiding this comment.
Two thoughts:
- It was not immediately obvious to me why the interface is so complex. presumably if we are fixing a "determinstic seed" then the random uuid should just be deterministic. Some discussion explaining uuid_generate, or the xfs precedent, or generally just some high level explanation of the spec/expectation/behavior would be helpful. My GUESS is that the issue is that uuid_generate is a library function that is not configurable to have a fixed seed, so we have to sha shuffle the single well known uuid passed in. But it would be nice to make that (or the actual right answer if that is BS) clearer.
- Some quick googling (https://man7.org/linux/man-pages/man3/uuid_generate.3.html) turned up
uuid_generate_sha1which purports to produce a valid v5 uuid from an input uuid using sha. What is the reason to seemingly roll our own rather than use that?
There was a problem hiding this comment.
Your guess is right, uuid_generate can't be seeded, so we derive each UUID from the seeded fs UUID using the role and key to make each one distinct. You're also right that we should just switch to uuid_generate_sha1, no real good reason to roll our own.
| * on a 16K-page host (e.g. aarch64) with the default 16K nodesize | ||
| * produces the same superblock bytes as mkfs on a 4K-page host. | ||
| */ | ||
| if (nodesize > SZ_4K) |
There was a problem hiding this comment.
I'd just take the if out and do this unconditionally. The versions of the kernel that don't support BIG_METADATA are absolutely ancient.
|
Leo, you might want to grab a Docker image of a distro that uses musl, and check that a version of mkfs.btrfs compiled there produces a file with the same hash. Likewise it might be an idea to do the same in a s390x VM, to double-check that you're not making any endian assumptions anywhere. |
e43519d to
78747a3
Compare
Don't necessarily think you need to do these things, btw, this is just me thinking out loud about what it would take to make this absolutely bulletproof. Two more such ideas (again, you don't need to do these):
IIUC the images here are only guaranteed to be the same for the same version of btrfs-progs (and the compression libraries presumably). It's a much harder problem, but if we could guarantee that everything was well-defined and deterministic we could make it so the test was just checking the hash of the image against a precomputed value. |
592c64f to
7c456d8
Compare
78747a3 to
c4dd811
Compare
|
This version looks good to me. However I'd prefer someone from Meta to give it an extra review. As I do not have a use-case here. |
| To create a reproducible image, pin the filesystem UUID and both | ||
| variables, and normalize ownership of the source tree yourself (the one | ||
| image attribute left to the caller, a single :command:`chown`): |
There was a problem hiding this comment.
this bit (and the truncate bit below) feel a bit vague and out of place.
It feels like you want to give a "recipe" for avoiding some gotchas, so I think doing that more totally is best. Alternatively, we can rely on the test as the documentation for the canonical way to do it. I am fine with either but this section of two comments and a code snippet incorporating only one of them doesn't make a ton of sense to me.
| return ret; | ||
| } | ||
|
|
||
| /* Sorted depth-first directory walker. See path-utils.h for the contract. */ |
There was a problem hiding this comment.
not that helpful of a comment IMO.
| const char *slash; | ||
| int ret; | ||
|
|
||
| /* nftw() reports base as the offset of the last path component. */ |
There was a problem hiding this comment.
without a comment anywhere nearby explaining that we are emulating nftw this is sort of weird to me.
| /* | ||
| * Heap-allocate to keep a PATH_MAX buffer off the per-level | ||
| * stack frame. | ||
| */ |
There was a problem hiding this comment.
relatively vacuous comment as well
| /* | ||
| * A byte-reproducible image needs every source of nondeterminism | ||
| * pinned: timestamps (SOURCE_DATE_EPOCH) and UUIDs (DETERMINISTIC_SEED, | ||
| * which also requires -U). With only one set the image still varies, | ||
| * so the extra cost of the sorted -r walk only pays off when both are. | ||
| */ |
There was a problem hiding this comment.
this feels redundant and tautological
| ret = nftw(source_dir, ftw_add_inode, 32, FTW_PHYS); | ||
| /* | ||
| * The sorted walker reads a whole directory into memory before | ||
| * descending, which only pays off when a reproducible image is |
There was a problem hiding this comment.
this is kind of weird wording too, it sort of bakes in the review history. The point is that the reproducible mode is costly, not whether or not sorting "pays off"
I would say it more like: reproducible builds requires sorted traversal, which incurs additional costs. Don't incur the allocation and cpu cost of sorting for regular non-reproducible mkfs.
boryas
left a comment
boryas
left a comment
There was a problem hiding this comment.
LGTM. Left a number of inline doc/comment nits but it looks improved since my last review and I was already pretty happy with it. I like the split sorted/nftw design as well, for what it's worth.
mkfs stamps root-item timestamps with time(NULL), so every run embeds
the current wall clock and no two runs are bit-identical. Honor
SOURCE_DATE_EPOCH when it is set in the environment, the standard
reproducible-builds convention.
Add reproducible_now() in common/reproducible.{c,h}, which parses the
value with parse_u64(). An empty value means unset, per the spec. An
invalid value is rejected with an error and a non-zero exit instead of
falling back, so a typo does not silently produce a non-reproducible
image.
The helper goes in common/ because it is also called from
common/root-tree-utils.c, which is shared with other tools; later
patches add more reproducibility helpers next to it.
Convert the time(NULL) call sites:
- mkfs/common.c btrfs_create_tree_root(): FS_TREE root_item ctime and
otime.
- common/root-tree-utils.c btrfs_make_root_dir(): atime/ctime/mtime/
otime of the top inode of every root it creates (root_tree_dir,
fs_tree, each -u subvolume, the data-reloc tree).
Signed-off-by: Leo Martins <loemra.dev@gmail.com>
mkfs generates several internal UUIDs with uuid_generate(): every
device's dev_item.uuid, the chunk-tree UUID, the FS_TREE subvolume UUID,
and the per-subvolume UUIDs from the uuid-tree rebuild. All are random,
so even with -U and SOURCE_DATE_EPOCH pinned, two runs with identical
inputs differ. uuid_generate() reads the system CSPRNG and cannot be
seeded, so it cannot be made deterministic in place.
When DETERMINISTIC_SEED=1 is set and a fixed UUID is given with -U (the
xfsprogs convention), derive these UUIDs from the fs UUID. A plain copy
of the seed will not do: an image holds several UUIDs that must stay
distinct on disk, so each is derived from a (role, key) pair in the
fs-UUID namespace with libuuid's name-based (v5) generator:
uuid_generate_sha1(out, fs_uuid, le32(role) || le64(key))
role is an enum (chunk tree, subvolume, device) and key distinguishes
instances within a role (subvolume objectid, devid; ignored for the
chunk-tree singleton). The name is serialized little-endian so the
result is the same on any host architecture.
uuid_generate_sha1() has been in libuuid since util-linux 2.31 (2017),
so configure.ac now requires uuid >= 2.31, like the version floors
already set for libzstd, libgcrypt and others.
Two helpers join reproducible_now() in common/reproducible.{c,h}:
reproducible_is_deterministic() checks DETERMINISTIC_SEED == "1", and
reproducible_uuid_generate() derives the UUID in deterministic mode and
otherwise falls back to uuid_generate(). mkfs errors out if
DETERMINISTIC_SEED=1 is set without -U, since there is no seed to derive
from.
Converted call sites:
- mkfs/common.c btrfs_create_tree_root(): FS_TREE subvolume UUID.
- mkfs/common.c make_btrfs(): chunk_tree_uuid and device 1's
dev_item.uuid (used when --device-uuid is not given).
- common/device-scan.c btrfs_add_to_fsid(): per-device UUID for
devices 2..N.
- common/root-tree-utils.c rescan_subvol_uuid(): per-subvolume UUID
during the uuid-tree rebuild.
Only super.fsid is not derived; it is the seed, from -U (or random when
-U is absent). A user-supplied --device-uuid still overrides device 1
as before.
Signed-off-by: Leo Martins <loemra.dev@gmail.com>
Two host attributes leak into the on-disk image, so the same arguments
produce different output on different build machines.
1. nr_global_roots defaulted to sysconf(_SC_NPROCESSORS_ONLN). With
extent-tree-v2 this is written to the super block
(btrfs_set_super_nr_global_roots) and sets how many global roots
are created, so the host CPU count ends up in the image. Default
to 1; --num-global-roots still overrides it. extent-tree-v2 is
experimental, but the leak is real today.
2. BTRFS_FEATURE_INCOMPAT_BIG_METADATA was set from
sysconf(_SC_PAGE_SIZE): with the default 16K nodesize it was on for
4K-page hosts and off for 16K-page hosts (aarch64). The flag is not
a default feature and not -O-toggleable, so this is its only
setter; the kernel sets it on any modern filesystem anyway, and
only kernels older than Linux 3.4 (2012) lack it. Set it
unconditionally, which drops the page-size dependency and also
covers the 4K-nodesize case the old gate skipped.
Neither change is gated by DETERMINISTIC_SEED; both fix cross-host
consistency regardless of reproducibility intent.
Signed-off-by: Leo Martins <loemra.dev@gmail.com>
With `mkfs.btrfs -r`, stat_to_inode_item() copies each source inode's
atime, ctime and mtime from the host, so the image is not reproducible
even when the source tree is otherwise normalized. When
SOURCE_DATE_EPOCH is set, pin ctime and atime to it and clamp mtime to
it. When it is unset or empty, use the host timestamps as before.
The timestamps need different handling:
- ctime cannot be set through the filesystem API; the kernel updates
it after any inode change. mkfs writes the raw inode, so pin it to
the epoch there.
- atime is not stable source metadata. On a relatime filesystem, the
first mkfs can record an old atime and advance it while reading the
source, then a second mkfs records a different value. Pin it to the
epoch so reads cannot change the image.
- mtime remains meaningful source metadata. Preserve values older
than the epoch and clamp newer values, so nothing in the image is
newer than the source date.
Add a regression using a relatime source. The first build advances the
source atime. The test builds again, compares the raw images, and mounts
the result to verify that atime and ctime equal the epoch while
the older mtime is preserved.
Factor the "is SOURCE_DATE_EPOCH set" test into
reproducible_has_source_date() so it reads the same everywhere it is
needed.
uid/gid remain the caller's responsibility (chown -R 0:0): one line of
shell, where a --owner flag would be permanent CLI surface.
Signed-off-by: Leo Martins <loemra.dev@gmail.com>
mkfs.btrfs -r walks the source tree with nftw(), which returns entries in the source filesystem's readdir() order. That order comes from the filesystem, not the tree: ext4 uses a name hash with a per-fs seed, xfs a b+tree key order, tmpfs and btrfs insertion order. Discovery order then decides inode numbers, dir_index values, extent bytenrs, and which hardlink owns the data, so the same tree staged on two different filesystems produces two different images. Add path_sorted_walk() in common/path-utils, which sorts each directory's entries by name (byte-wise strcmp, not the locale-dependent strcoll) before recursing. It keeps nftw()'s callback signature and preorder/FTW_PHYS semantics. ftw_add_inode() still needs a NULL-stat guard: the walker reports a failed lstat() as FTW_NS with a NULL stat where glibc nftw() passes a buffer, so the callback skips it instead of dereferencing. The walker also matches the two nftw() edges the callbacks depend on: an unreadable directory is reported once as FTW_DNR, and errno is checked after the readdir() loop since readdir() returns NULL for both end-of-stream and error. The child path is heap-allocated so a deep tree doesn't overflow the stack. A sorted traversal means reading a whole directory up front plus the qsort and allocations that go with it. A regular mkfs doesn't need that, so keep nftw() as the default and only sort when a reproducible image is actually requested, i.e. when both SOURCE_DATE_EPOCH and DETERMINISTIC_SEED are set (reproducible_is_enabled()). Setting just one can't produce a reproducible image anyway, since the other source of nondeterminism (host time, or random UUIDs) still leaks in. btrfs_mkfs_size_dir() stays on nftw() regardless; it only sums sizes and counts inodes, so walk order doesn't matter there. The same reproducible mode sorts each inode's xattrs by name in add_xattr_item(), the xattr counterpart of sorting directory entries. It only matters when two names collide under btrfs_name_hash() and share a dir_item stored in insertion order, but sorting removes the dependence. Signed-off-by: Leo Martins <loemra.dev@gmail.com>
Add an ENVIRONMENT section to mkfs.btrfs(8) for the two environment variables that make image creation reproducible: SOURCE_DATE_EPOCH (timestamps) and DETERMINISTIC_SEED (UUID derivation, requires -U). It covers the empty-is-unset and invalid-is-an-error handling, the minimal recipe, and that normalizing ownership (chown -R 0:0) of a -r tree is left to the caller. Signed-off-by: Leo Martins <loemra.dev@gmail.com>
Add tests/reproducibility-check.sh, a standalone root-required test that
checks mkfs.btrfs produces byte-identical images from identical inputs.
For a matrix of environment variations it runs mkfs twice and cmp's the
raw images:
- baseline, wall-clock time, TZ, locale, umask and working directory,
for both empty and --rootdir images;
- source uid, source filesystem (the same tree on loop-mounted ext4 vs
xfs), and compression (lzo/zlib/zstd), for --rootdir only.
Each run uses the documented recipe: a fixed -U, SOURCE_DATE_EPOCH and
DETERMINISTIC_SEED=1, and no --device-uuid, so the matrix also confirms
mkfs derives the device UUID from the fs UUID on its own. The source
tree is chown'd to 0:0 first (the one piece left to the caller) and mkfs
normalizes timestamps itself. The ext4 mount's lost+found is stripped so
the ext4 and xfs trees match, and two locales are probed at startup so
the locale case compares two real locales or is skipped.
Root is required for the ownership and loop-mounted source-filesystem
variations. With the preceding patches, all variations pass.
Signed-off-by: Leo Martins <loemra.dev@gmail.com>
c4dd811 to
8c3919a
Compare
|
Hey, v3 is ready now. The main change from v2 is how rootdir atime is handled. v2 preserved source atimes older than SOURCE_DATE_EPOCH and only clamped newer ones, like mtime. That isn't reproducible when reading the source updates atime between builds, so v3 always pins atime to SOURCE_DATE_EPOCH. ctime remains pinned there too, while mtime still preserves older source values and clamps newer ones. I added a btrfs-backed regression for this, tightened the reproducible-image recipe in the docs, and trimmed the comments and commit messages down. |
4 participants
Make
mkfs.btrfsproduce byte-for-byte identical images from identicalinputs (same binary, args, and source tree), for supply-chain verification,
build caching, and meaningful image diffs.
Block placement was already deterministic; this series plugs the remaining
non-determinism sources.
Interface
No new CLI flags — two environment variables (same convention as the
reproducible-builds ecosystem and xfsprogs):
SOURCE_DATE_EPOCH=<unix_ts>— pins all timestamps.DETERMINISTIC_SEED=1— opt into deterministic UUID derivation (requires afixed
-U; errors out if missing instead of silently emitting random UUIDs).The
-UUUID is the seed; all other internal UUIDs derive from it.What's fixed (one per commit)
time(NULL)root-item timestamps →SOURCE_DATE_EPOCH.fs UUID as
SHA256(fs_uuid || role || key), kept unique formulti-device/multi-subvol correctness.
nr_global_roots(CPU count) andBIG_METADATA(page size) madehost-independent.
-rsource fileatime/ctime/mtime→SOURCE_DATE_EPOCH(ctime can't benormalized externally).
-rdirectory order:nftw()(filesystem-dependent) replaced with aname-sorted walker, so the same tree on ext4/xfs/tmpfs yields the same image.
Ownership is left to the caller (
chown) since the value is image-semantic.Testing
Includes
tests/reproducibility-check.sh(root-required). Verifiedbyte-identical across ext4-vs-xfs source, multi-device, all csum types,
nodesize/sectorsize/compression/features, special files/xattrs/ACLs, allocator
perturbation, ASLR, and TZ/locale/umask/cwd.
Caveats
--compressoutput also depends on the compression library version.