btrfs receive doesn't work on bootc containers with composefs enabled

[yorickpeterse]

Originally discussed at bootc-dev/bootc#2170, but reporting here because I think this is more a bug in btrfs-progs, or more precisely it expecting the file system to be in a certain way that won't always be true:

I have two servers running a bootc Fedora installation: "web" and "deskmini". Both use btrfs. On the "deskmini" I have a script that's supposed to create a btrfs snapshot of "web", then download it to "deskmini". If a previous snapshot exists, it sends an incremental snapshot. This is done using ssh 'btrfs send SNAP' | btrfs receive TARGET and ssh 'btrfs send -p PREV SNAP' | btrfs receive TARGET.

It seems that something in bootc (likely the use of composefs/the exotic mount setup) confuses btrfs. Sending an receiving an initial snapshot works fine and produces the same IDs and all that:

# web
$ btrfs subvolume list /root
ID 256 gen 28125 top level 5 path root
ID 257 gen 28087 top level 256 path ostree/deploy/default/var/lib/shost
ID 258 gen 28079 top level 256 path ostree/deploy/default/var/lib/ssh-container
ID 259 gen 28090 top level 256 path ostree/deploy/default/var/lib/letsencrypt
ID 333 gen 28089 top level 256 path ostree/deploy/default/var/lib/snapshots/letsencrypt.a
ID 334 gen 28090 top level 256 path ostree/deploy/default/var/lib/snapshots/letsencrypt.b

# deskmini
$ btrfs subvolume list /root
ID 256 gen 108744 top level 5 path root
ID 261 gen 108744 top level 256 path ostree/deploy/default/var/lib/grafana
ID 262 gen 108744 top level 256 path ostree/deploy/default/var/lib/omada
ID 263 gen 108744 top level 256 path ostree/deploy/default/var/lib/victoria-metrics
ID 264 gen 92546 top level 256 path ostree/deploy/default/var/lib/mosquitto
ID 265 gen 108211 top level 256 path ostree/deploy/default/var/lib/jellyfin

After the initial send+receive:

# deskmini
$ btrfs subvolume list /root
ID 256 gen 108746 top level 5 path root
ID 261 gen 108745 top level 256 path ostree/deploy/default/var/lib/grafana
ID 262 gen 108746 top level 256 path ostree/deploy/default/var/lib/omada
ID 263 gen 108746 top level 256 path ostree/deploy/default/var/lib/victoria-metrics
ID 264 gen 92546 top level 256 path ostree/deploy/default/var/lib/mosquitto
ID 265 gen 108211 top level 256 path ostree/deploy/default/var/lib/jellyfin
ID 293 gen 108746 top level 256 path ostree/deploy/default/var/lib/backups/web/letsencrypt.a

However an incremental send/receive fails:

ssh web 'btrfs send -p /var/lib/snapshots/letsencrypt.a /var/lib/snapshots/letsencrypt.b' | btrfs receive /var/lib/backups/web 
At subvol /var/lib/snapshots/letsencrypt.b
At snapshot letsencrypt.b
It seems that you have changed your default subvolume or you specify other subvolume to
mount btrfs, try to remount this btrfs filesystem with fs tree, and run btrfs receive again!

Some digging around suggests this happens when setting subvol/subvolid, though the btrfs manual is horribly vague on the exact specifics. Both systems use the same setup, minus "deskmini" using luks:

# web
/dev/sda4 on /sysroot/ostree/deploy/default/var type btrfs (rw,relatime,seclabel,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root)
/dev/sda4 on /var type btrfs (rw,relatime,seclabel,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root)

# deskmini
/dev/mapper/luks-4c9af5ab-cf78-4a27-8e0f-3eb5522f554d on /sysroot/ostree/deploy/default/var type btrfs (rw,relatime,seclabel,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root)
/dev/mapper/luks-4c9af5ab-cf78-4a27-8e0f-3eb5522f554d on /var type btrfs (rw,relatime,seclabel,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root)

The output of mount (minus the various lines for Podman's container storage) on the sender is as follows:

sender (bootc)

composefs on / type overlay (ro,relatime,seclabel,lowerdir+=/run/ostree/.private/cfsroot-lower,datadir+=/sysroot/ostree/repo/objects,redirect_dir=on,metacopy=on)
/dev/sda4 on /etc type btrfs (rw,relatime,seclabel,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root)
/dev/sda4 on /sysroot type btrfs (ro,relatime,seclabel,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root)
/dev/sda4 on /sysroot/ostree/deploy/default/var type btrfs (rw,relatime,seclabel,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root)
devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=1862040k,nr_inodes=465510,mode=755,inode64)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel,inode64,usrquota)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=600,ptmxmode=000)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,seclabel,nsdelegate,memory_recursiveprot)
none on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime,seclabel)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,size=778188k,nr_inodes=819200,mode=755,inode64)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,nosuid,noexec,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=41,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=7672)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,nosuid,nodev,relatime,seclabel,pagesize=2M)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime,seclabel)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime,seclabel)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime,seclabel)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
tmpfs on /run/credentials/systemd-journald.service type tmpfs (ro,nosuid,nodev,noexec,relatime,nosymfollow,seclabel,size=1024k,nr_inodes=1024,mode=700,inode64,noswap)
/dev/sda4 on /var type btrfs (rw,relatime,seclabel,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,seclabel,size=1945468k,nr_inodes=1048576,inode64,usrquota)
/dev/sda3 on /boot type ext4 (ro,relatime,seclabel)
/dev/sda2 on /boot/efi type vfat (rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=winnt,errors=remount-ro)
tmpfs on /run/credentials/systemd-resolved.service type tmpfs (ro,nosuid,nodev,noexec,relatime,nosymfollow,seclabel,size=1024k,nr_inodes=1024,mode=700,inode64,noswap)
tmpfs on /run/credentials/getty@tty1.service type tmpfs (ro,nosuid,nodev,noexec,relatime,nosymfollow,seclabel,size=1024k,nr_inodes=1024,mode=700,inode64,noswap)
tmpfs on /run/credentials/serial-getty@ttyS0.service type tmpfs (ro,nosuid,nodev,noexec,relatime,nosymfollow,seclabel,size=1024k,nr_inodes=1024,mode=700,inode64,noswap)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=389092k,nr_inodes=97273,mode=700,inode64)

And for the receiver:

receiver (bootc)

composefs on / type overlay (ro,relatime,seclabel,lowerdir+=/run/ostree/.private/cfsroot-lower,datadir+=/sysroot/ostree/repo/objects,redirect_dir=on,metacopy=on)
/dev/mapper/luks-4c9af5ab-cf78-4a27-8e0f-3eb5522f554d on /etc type btrfs (rw,relatime,seclabel,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root)
/dev/mapper/luks-4c9af5ab-cf78-4a27-8e0f-3eb5522f554d on /sysroot type btrfs (ro,relatime,seclabel,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root)
/dev/mapper/luks-4c9af5ab-cf78-4a27-8e0f-3eb5522f554d on /sysroot/ostree/deploy/default/var type btrfs (rw,relatime,seclabel,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root)
devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=6965204k,nr_inodes=1741301,mode=755,inode64)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel,inode64,usrquota)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=600,ptmxmode=000)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,seclabel,nsdelegate,memory_recursiveprot)
none on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime,seclabel)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,size=2831396k,nr_inodes=819200,mode=755,inode64)
tmpfs on /run/credentials/systemd-cryptsetup@luks\x2d4c9af5ab\x2dcf78\x2d4a27\x2d8e0f\x2d3eb5522f554d.service type tmpfs (ro,nosuid,nodev,noexec,relatime,nosymfollow,seclabel,size=1024k,nr_inodes=1024,mode=700,inode64,noswap)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,nosuid,noexec,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=41,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17412)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime,seclabel)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime,seclabel)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,nosuid,nodev,relatime,seclabel,pagesize=2M)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime,seclabel)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
tmpfs on /run/credentials/systemd-journald.service type tmpfs (ro,nosuid,nodev,noexec,relatime,nosymfollow,seclabel,size=1024k,nr_inodes=1024,mode=700,inode64,noswap)
tmpfs on /run/credentials/systemd-cryptsetup@luks\x2d81a8d327\x2dc628\x2d4eb4\x2d9211\x2dd7e7c32148b4.service type tmpfs (ro,nosuid,nodev,noexec,relatime,nosymfollow,seclabel,size=1024k,nr_inodes=1024,mode=700,inode64,noswap)
tmpfs on /run/credentials/systemd-cryptsetup@luks\x2de3af8483\x2d56d6\x2d4604\x2d99dc\x2d035a7a752df3.service type tmpfs (ro,nosuid,nodev,noexec,relatime,nosymfollow,seclabel,size=1024k,nr_inodes=1024,mode=700,inode64,noswap)
tmpfs on /run/credentials/systemd-cryptsetup@luks\x2d4ce21d9f\x2d8575\x2d4caf\x2d9f8c\x2da6ea6204b369.service type tmpfs (ro,nosuid,nodev,noexec,relatime,nosymfollow,seclabel,size=1024k,nr_inodes=1024,mode=700,inode64,noswap)
/dev/mapper/luks-4c9af5ab-cf78-4a27-8e0f-3eb5522f554d on /var type btrfs (rw,relatime,seclabel,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,seclabel,size=7078488k,nr_inodes=1048576,inode64,usrquota)
/dev/nvme1n1p2 on /boot type ext4 (rw,relatime,seclabel,stripe=8)
tmpfs on /run/credentials/systemd-resolved.service type tmpfs (ro,nosuid,nodev,noexec,relatime,nosymfollow,seclabel,size=1024k,nr_inodes=1024,mode=700,inode64,noswap)
/dev/mapper/luks-81a8d327-c628-4eb4-9211-d7e7c32148b4 on /var/lib/storage/81a8d327 type btrfs (rw,relatime,seclabel,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=5,subvol=/)
/dev/mapper/luks-e3af8483-56d6-4604-99dc-035a7a752df3 on /var/lib/storage/e3af8483 type btrfs (rw,relatime,seclabel,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=5,subvol=/)
/dev/mapper/luks-4ce21d9f-8575-4caf-9f8c-a6ea6204b369 on /var/lib/storage/4ce21d9f type btrfs (rw,relatime,seclabel,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=5,subvol=/)
tmpfs on /run/credentials/getty@tty1.service type tmpfs (ro,nosuid,nodev,noexec,relatime,nosymfollow,seclabel,size=1024k,nr_inodes=1024,mode=700,inode64,noswap)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=1415696k,nr_inodes=353924,mode=700,inode64)

And on a Fedora Workstation installation where btrfs receive does work (mostly to show this isn't an issue on the sender's side):

Fedora Workstation

/dev/mapper/luks-f6f6bf81-bbcb-434c-9903-48c4a469037c on / type btrfs (rw,relatime,seclabel,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root)
devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=7728436k,nr_inodes=1932109,mode=755,inode64)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel,inode64,usrquota)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=600,ptmxmode=000)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,seclabel,nsdelegate,memory_recursiveprot)
none on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime,seclabel)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,size=3128920k,nr_inodes=819200,mode=755,inode64)
tmpfs on /run/credentials/systemd-cryptsetup@luks\x2df6f6bf81\x2dbbcb\x2d434c\x2d9903\x2d48c4a469037c.service type tmpfs (ro,nosuid,nodev,noexec,relatime,nosymfollow,seclabel,size=1024k,nr_inodes=1024,mode=700,inode64,noswap)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,nosuid,noexec,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=42,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13689)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime,seclabel)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime,seclabel)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,nosuid,nodev,relatime,seclabel,pagesize=2M)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime,seclabel)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
tmpfs on /run/credentials/systemd-journald.service type tmpfs (ro,nosuid,nodev,noexec,relatime,nosymfollow,seclabel,size=1024k,nr_inodes=1024,mode=700,inode64,noswap)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,seclabel,nr_inodes=1048576,inode64,usrquota)
/dev/mapper/luks-f6f6bf81-bbcb-434c-9903-48c4a469037c on /home type btrfs (rw,relatime,seclabel,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=257,subvol=/home)
/dev/nvme0n1p2 on /boot type ext4 (rw,relatime,seclabel,stripe=8)
/dev/nvme0n1p1 on /boot/efi type vfat (rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=winnt,errors=remount-ro)
tmpfs on /run/credentials/systemd-resolved.service type tmpfs (ro,nosuid,nodev,noexec,relatime,nosymfollow,seclabel,size=1024k,nr_inodes=1024,mode=700,inode64,noswap)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=1564460k,nr_inodes=391115,mode=700,uid=1000,gid=1000,inode64)
gvfsd-fuse on /run/user/1000/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000)
portal on /run/user/1000/doc type fuse.portal (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000)

Per bootc-dev/bootc#2170 (reply in thread) I played around with trying to get btrfs receive to dump some more information, but it's not clear what exactly this implies:

diff --git a/cmds/receive.c b/cmds/receive.c
index 3e3b989c..daf5d2c8 100644
--- a/cmds/receive.c
+++ b/cmds/receive.c
@@ -370,6 +370,7 @@ static int process_snapshot(const char *path, const u8 *uuid, u64 ctransid,
                                "subvolume or you specify other subvolume to\n"
                                "mount btrfs, try to remount this btrfs filesystem "
                                "with fs tree, and run btrfs receive again!\n");
+               fprintf(stderr, "path: %s\n", parent_subvol->path);
                goto out;
        }
 

At subvol /var/lib/snapshots/letsencrypt.1777258148
At snapshot letsencrypt.1777258148
It seems that you have changed your default subvolume or you specify other subvolume to
mount btrfs, try to remount this btrfs filesystem with fs tree, and run btrfs receive again!
path: ostree/deploy/default/var/lib/backups/web/letsencrypt.1777257910

Further digging through the code and the commit (b5a0977) that introduced it it's not clear to me what exactly the problem is and how to resolve it.

Is there something that would need to be changed by btrfs-progs, or is this perhaps strictly an issue with bootc?

[yorickpeterse]

I think there may be a hint in the following:

On my workstation setup, the output of df -h | grep -v containers is the following:

Filesystem      Size  Used Avail Use% Mounted on
/dev/dm-0       464G   60G  401G  13% /
devtmpfs        7.4G   84K  7.4G   1% /dev
tmpfs           7.5G  176K  7.5G   1% /dev/shm
efivarfs        148K   82K   62K  58% /sys/firmware/efi/efivars
...

Note how the main disk (/dev/dm-0) is mounted on /. Meanwhile on a bootc installation:

Filesystem      Size  Used Avail Use% Mounted on
composefs       5.8M  5.8M     0 100% /
/dev/dm-0       1.9T   10G  1.9T   1% /etc
devtmpfs        6.7G     0  6.7G   0% /dev
tmpfs           6.8G   84K  6.8G   1% /dev/shm
efivarfs        128K   40K   84K  32% /sys/firmware/efi/efivars
...

Here composefs is mounted on / and the disk shows up as mounted on /etc. I'm guessing that btrfs-progs expects the disk to instead be mounted on / instead?

[yorickpeterse]

To add another data point, the difference between bootc setups and regular ones is that the root disk seems to be mounted multiple times but using different subvolumes and mount points:

# bootc setup
mount | grep '/dev/sda4'
/dev/sda4 on /etc type btrfs (rw,relatime,seclabel,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root)
/dev/sda4 on /sysroot type btrfs (ro,relatime,seclabel,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root)
/dev/sda4 on /sysroot/ostree/deploy/default/var type btrfs (rw,relatime,seclabel,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root)
/dev/sda4 on /var type btrfs (rw,relatime,seclabel,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root)
/dev/sda4 on /var/lib/containers/storage/overlay type btrfs (rw,relatime,seclabel,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root)

# My laptop running regular Fedora Workstation
mount | grep '/dev/mapper'
/dev/mapper/luks-f6f6bf81-bbcb-434c-9903-48c4a469037c on / type btrfs (rw,relatime,seclabel,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root)
/dev/mapper/luks-f6f6bf81-bbcb-434c-9903-48c4a469037c on /home type btrfs (rw,relatime,seclabel,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=257,subvol=/home)

Based on the order of entries in mount on the bootc host I'm guessing btrfs-progs just picks the first one which happens to be /etc, when it should instead pick the /var one?