dfc statsfs() every filesystem

[klaernie]

dfc blindly statfs()es every filesystem it find, not just the ones asked about.

This leads to blocking when some automounted filesystems cannot be mounted, but the problematic filesystem is excluded either by type or -l.

In comparison coreutils df only statfs()es the filesystems it will actually print.

kandre@mainframe(pts/11)[master!] ~/git-repos/dfc/build % strace -e statfs,stat bin/dfc -W -f -T -l
stat("/home/kandre/.config/dfc/dfcrc", 0x7ffd438bf890) = -1 ENOENT (No such file or directory)
stat("/home/kandre/.config/dfc/dfcrc/.dfcrc", 0x7ffd438bf890) = -1 ENOENT (No such file or directory)
statfs("/sys", {f_type="SYSFS_MAGIC", f_bsize=4096, f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0, f_fsid={0, 0}, f_namelen=255, f_frsize=4096, f_flags=4142}) = 0
statfs("/proc", {f_type="PROC_SUPER_MAGIC", f_bsize=4096, f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0, f_fsid={0, 0}, f_namelen=255, f_frsize=4096, f_flags=4128}) = 0
statfs("/dev", {f_type="TMPFS_MAGIC", f_bsize=4096, f_blocks=1270410, f_bfree=1270410, f_bavail=1270410, f_files=1270410, f_ffree=1269742, f_fsid={0, 0}, f_namelen=255, f_frsize=4096, f_flags=4130}) = 0
statfs("/dev/pts", {f_type="DEVPTS_SUPER_MAGIC", f_bsize=4096, f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0, f_fsid={0, 0}, f_namelen=255, f_frsize=4096, f_flags=4138}) = 0
statfs("/run", {f_type="TMPFS_MAGIC", f_bsize=4096, f_blocks=254636, f_bfree=248118, f_bavail=248118, f_files=1273178, f_ffree=1272063, f_fsid={0, 0}, f_namelen=255, f_frsize=4096, f_flags=4138}) = 0
statfs("/", {f_type="EXT2_SUPER_MAGIC", f_bsize=4096, f_blocks=9579685, f_bfree=2962987, f_bavail=2958891, f_files=2444624, f_ffree=1935236, f_fsid={-331120939, 823145904}, f_namelen=255, f_frsize=4096, f_flags=4128}) = 0
statfs("/sys/kernel/security", {f_type="SECURITYFS_MAGIC", f_bsize=4096, f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0, f_fsid={0, 0}, f_namelen=255, f_frsize=4096, f_flags=4142}) = 0
statfs("/dev/shm", {f_type="TMPFS_MAGIC", f_bsize=4096, f_blocks=1273178, f_bfree=1195064, f_bavail=1195064, f_files=1273178, f_ffree=1272951, f_fsid={0, 0}, f_namelen=255, f_frsize=4096, f_flags=38}) = 0
statfs("/run/lock", {f_type="TMPFS_MAGIC", f_bsize=4096, f_blocks=1280, f_bfree=1279, f_bavail=1279, f_files=1273178, f_ffree=1273165, f_fsid={0, 0}, f_namelen=255, f_frsize=4096, f_flags=4142}) = 0
statfs("/sys/fs/cgroup", {f_type="TMPFS_MAGIC", f_bsize=4096, f_blocks=1273178, f_bfree=1273178, f_bavail=1273178, f_files=1273178, f_ffree=1273164, f_fsid={0, 0}, f_namelen=255, f_frsize=4096, f_flags=47}) = 0
statfs("/sys/fs/cgroup/systemd", {f_type="CGROUP_SUPER_MAGIC", f_bsize=4096, f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0, f_fsid={0, 0}, f_namelen=255, f_frsize=4096, f_flags=4142}) = 0
statfs("/sys/fs/pstore", {f_type="PSTOREFS_MAGIC", f_bsize=4096, f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0, f_fsid={0, 0}, f_namelen=255, f_frsize=4096, f_flags=4142}) = 0
statfs("/proc/sys/fs/binfmt_misc", {f_type="BINFMTFS_MAGIC", f_bsize=4096, f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0, f_fsid={0, 0}, f_namelen=255, f_frsize=4096, f_flags=4128}) = 0
statfs("/dev/mqueue", {f_type=0x19800202, f_bsize=4096, f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0, f_fsid={0, 0}, f_namelen=255, f_frsize=4096, f_flags=4128}) = 0
statfs("/sys/kernel/debug", {f_type="DEBUGFS_MAGIC", f_bsize=4096, f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0, f_fsid={0, 0}, f_namelen=255, f_frsize=4096, f_flags=4128}) = 0
statfs("/dev/hugepages", {f_type="HUGETLBFS_MAGIC", f_bsize=2097152, f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0, f_fsid={0, 0}, f_namelen=255, f_frsize=2097152, f_flags=4128}) = 0
statfs("/boot/efi", {f_type="MSDOS_SUPER_MAGIC", f_bsize=8192, f_blocks=63965, f_bfree=63924, f_bavail=63924, f_files=0, f_ffree=0, f_fsid={2050, 0}, f_namelen=1530, f_frsize=8192, f_flags=4128}) = 0
statfs("/proc/sys/fs/binfmt_misc", {f_type="BINFMTFS_MAGIC", f_bsize=4096, f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0, f_fsid={0, 0}, f_namelen=255, f_frsize=4096, f_flags=4128}) = 0
statfs("/proc/fs/nfsd", {f_type=0x6e667364, f_bsize=4096, f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0, f_fsid={0, 0}, f_namelen=255, f_frsize=4096, f_flags=4128}) = 0
statfs("/run/user/122", {f_type="TMPFS_MAGIC", f_bsize=4096, f_blocks=254636, f_bfree=254635, f_bavail=254635, f_files=1273178, f_ffree=1273172, f_fsid={0, 0}, f_namelen=255, f_frsize=4096, f_flags=4134}) = 0
statfs("/run/user/1000", {f_type="TMPFS_MAGIC", f_bsize=4096, f_blocks=254636, f_bfree=254631, f_bavail=254631, f_files=1273178, f_ffree=1273160, f_fsid={0, 0}, f_namelen=255, f_frsize=4096, f_flags=4134}) = 0
statfs("/sys/kernel/debug/tracing", 0x7ffd438bf760) = -1 EACCES (Permission denied)
WARNING: /sys/kernel/debug/tracing was skipped because it could not be stated : Permission denied
statfs("/net/debs", {f_type="NFS_SUPER_MAGIC", f_bsize=1048576, f_blocks=18645, f_bfree=14534, f_bavail=13565, f_files=1220608, f_ffree=1163361, f_fsid={0, 0}, f_namelen=255, f_frsize=1048576, f_flags=4144}) = 0
statfs("/net/fynn", 0x7ffd438bf760)     = -1 ENODEV (No such device)
// this last stats() blocked until the automounter gave up
WARNING: /net/fynn was skipped because it could not be stated : No such device
statfs("/net/hive", {f_type="NFS_SUPER_MAGIC", f_bsize=1048576, f_blocks=50269, f_bfree=33155, f_bavail=30763, f_files=3276800, f_ffree=2960926, f_fsid={0, 0}, f_namelen=255, f_frsize=1048576, f_flags=4144}) = 0
statfs("/net/mainframe", {f_type="NFS_SUPER_MAGIC", f_bsize=1048576, f_blocks=37421, f_bfree=11575, f_bavail=11559, f_files=2444624, f_ffree=1935236, f_fsid={0, 0}, f_namelen=255, f_frsize=1048576, f_flags=4144}) = 0
statfs("/net/debs", {f_type="NFS_SUPER_MAGIC", f_bsize=1048576, f_blocks=18645, f_bfree=14534, f_bavail=13565, f_files=1220608, f_ffree=1163361, f_fsid={0, 0}, f_namelen=255, f_frsize=1048576, f_flags=4144}) = 0
statfs("/net/hive", {f_type="NFS_SUPER_MAGIC", f_bsize=1048576, f_blocks=50269, f_bfree=33155, f_bavail=30763, f_files=3276800, f_ffree=2960926, f_fsid={0, 0}, f_namelen=255, f_frsize=1048576, f_flags=4144}) = 0
statfs("/net/mainframe", {f_type="NFS_SUPER_MAGIC", f_bsize=1048576, f_blocks=37421, f_bfree=11575, f_bavail=11559, f_files=2444624, f_ffree=1935236, f_fsid={0, 0}, f_namelen=255, f_frsize=1048576, f_flags=4144}) = 0
FILESYSTEM                           TYPE     (=) USED      FREE (-)  %USED AVAILABLE  TOTAL MOUNTED ON             
udev                                 devtmpfs [--------------------]   0.0%      4.8G   4.8G /dev                   
tmpfs                                tmpfs    [=-------------------]   2.6%    969.2M 994.7M /run                   
/dev/mapper/vg_seven-root            ext4     [==============------]  69.1%     11.3G  36.5G /                      
tmpfs                                tmpfs    [==------------------]   6.1%      4.6G   4.9G /dev/shm               
tmpfs                                tmpfs    [=-------------------]   0.1%      5.0M   5.0M /run/lock              
tmpfs                                tmpfs    [--------------------]   0.0%      4.9G   4.9G /sys/fs/cgroup         
/dev/sda2                            vfat     [=-------------------]   0.1%    499.4M 499.7M /boot/efi              
tmpfs                                tmpfs    [=-------------------]   0.0%    994.7M 994.7M /run/user/122          
tmpfs                                tmpfs    [=-------------------]   0.0%    994.7M 994.7M /run/user/1000         
+++ exited with 0 +++

in comparison to

mainframe(pts/11)[master!] ~/git-repos/dfc/build % strace -e statfs,stat df -Tl
stat("/sys", {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0
stat("/proc", {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0
stat("/dev", {st_mode=S_IFDIR|0755, st_size=4260, ...}) = 0
stat("/dev/pts", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
stat("/run", {st_mode=S_IFDIR|0755, st_size=1240, ...}) = 0
stat("/", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/sys/kernel/security", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
stat("/dev/shm", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=220, ...}) = 0
stat("/run/lock", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=140, ...}) = 0
stat("/sys/fs/cgroup", {st_mode=S_IFDIR|0755, st_size=300, ...}) = 0
stat("/sys/fs/cgroup/systemd", {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0
stat("/sys/fs/pstore", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
stat("/proc/sys/fs/binfmt_misc", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
stat("/dev/mqueue", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=40, ...}) = 0
stat("/sys/kernel/debug", {st_mode=S_IFDIR|0700, st_size=0, ...}) = 0
stat("/dev/hugepages", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
stat("/boot/efi", {st_mode=S_IFDIR|0755, st_size=16384, ...}) = 0
stat("/proc/sys/fs/binfmt_misc", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
stat("/proc/fs/nfsd", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
stat("/run/user/122", {st_mode=S_IFDIR|0700, st_size=80, ...}) = 0
stat("/run/user/1000", {st_mode=S_IFDIR|0700, st_size=140, ...}) = 0
stat("/sys/kernel/debug/tracing", 0x7ffff2f68340) = -1 EACCES (Permission denied)
stat("/net/debs", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/net/fynn", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
stat("/net/hive", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0
stat("/net/mainframe", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
statfs("/dev", {f_type="TMPFS_MAGIC", f_bsize=4096, f_blocks=1270410, f_bfree=1270410, f_bavail=1270410, f_files=1270410, f_ffree=1269742, f_fsid={0, 0}, f_namelen=255, f_frsize=4096, f_flags=4130}) = 0
statfs("/run", {f_type="TMPFS_MAGIC", f_bsize=4096, f_blocks=254636, f_bfree=248118, f_bavail=248118, f_files=1273178, f_ffree=1272063, f_fsid={0, 0}, f_namelen=255, f_frsize=4096, f_flags=4138}) = 0
statfs("/", {f_type="EXT2_SUPER_MAGIC", f_bsize=4096, f_blocks=9579685, f_bfree=2962985, f_bavail=2958889, f_files=2444624, f_ffree=1935236, f_fsid={-331120939, 823145904}, f_namelen=255, f_frsize=4096, f_flags=4128}) = 0
statfs("/sys/kernel/security", {f_type="SECURITYFS_MAGIC", f_bsize=4096, f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0, f_fsid={0, 0}, f_namelen=255, f_frsize=4096, f_flags=4142}) = 0
statfs("/dev/shm", {f_type="TMPFS_MAGIC", f_bsize=4096, f_blocks=1273178, f_bfree=1194580, f_bavail=1194580, f_files=1273178, f_ffree=1272948, f_fsid={0, 0}, f_namelen=255, f_frsize=4096, f_flags=38}) = 0
statfs("/run/lock", {f_type="TMPFS_MAGIC", f_bsize=4096, f_blocks=1280, f_bfree=1279, f_bavail=1279, f_files=1273178, f_ffree=1273165, f_fsid={0, 0}, f_namelen=255, f_frsize=4096, f_flags=4142}) = 0
statfs("/sys/fs/cgroup", {f_type="TMPFS_MAGIC", f_bsize=4096, f_blocks=1273178, f_bfree=1273178, f_bavail=1273178, f_files=1273178, f_ffree=1273164, f_fsid={0, 0}, f_namelen=255, f_frsize=4096, f_flags=47}) = 0
statfs("/sys/fs/cgroup/systemd", {f_type="CGROUP_SUPER_MAGIC", f_bsize=4096, f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0, f_fsid={0, 0}, f_namelen=255, f_frsize=4096, f_flags=4142}) = 0
statfs("/sys/fs/pstore", {f_type="PSTOREFS_MAGIC", f_bsize=4096, f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0, f_fsid={0, 0}, f_namelen=255, f_frsize=4096, f_flags=4142}) = 0
statfs("/proc/sys/fs/binfmt_misc", {f_type="BINFMTFS_MAGIC", f_bsize=4096, f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0, f_fsid={0, 0}, f_namelen=255, f_frsize=4096, f_flags=4128}) = 0
statfs("/dev/hugepages", {f_type="HUGETLBFS_MAGIC", f_bsize=2097152, f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0, f_fsid={0, 0}, f_namelen=255, f_frsize=2097152, f_flags=4128}) = 0
statfs("/boot/efi", {f_type="MSDOS_SUPER_MAGIC", f_bsize=8192, f_blocks=63965, f_bfree=63924, f_bavail=63924, f_files=0, f_ffree=0, f_fsid={2050, 0}, f_namelen=1530, f_frsize=8192, f_flags=4128}) = 0
statfs("/proc/fs/nfsd", {f_type=0x6e667364, f_bsize=4096, f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0, f_fsid={0, 0}, f_namelen=255, f_frsize=4096, f_flags=4128}) = 0
statfs("/run/user/122", {f_type="TMPFS_MAGIC", f_bsize=4096, f_blocks=254636, f_bfree=254635, f_bavail=254635, f_files=1273178, f_ffree=1273172, f_fsid={0, 0}, f_namelen=255, f_frsize=4096, f_flags=4134}) = 0
statfs("/run/user/1000", {f_type="TMPFS_MAGIC", f_bsize=4096, f_blocks=254636, f_bfree=254631, f_bavail=254631, f_files=1273178, f_ffree=1273160, f_fsid={0, 0}, f_namelen=255, f_frsize=4096, f_flags=4134}) = 0
statfs("/sys/kernel/debug/tracing", 0x7ffff2f67c00) = -1 EACCES (Permission denied)
Filesystem                           Type      1K-blocks       Used Available Use% Mounted on
udev                                 devtmpfs    5081640          0   5081640   0% /dev
tmpfs                                tmpfs       1018544      26072    992472   3% /run
/dev/mapper/vg_seven-root            ext4       38318740   26466800  11835556  70% /
tmpfs                                tmpfs       5092712     314392   4778320   7% /dev/shm
tmpfs                                tmpfs          5120          4      5116   1% /run/lock
tmpfs                                tmpfs       5092712          0   5092712   0% /sys/fs/cgroup
/dev/sda2                            vfat         511720        328    511392   1% /boot/efi
tmpfs                                tmpfs       1018544          4   1018540   1% /run/user/122
tmpfs                                tmpfs       1018544         20   1018524   1% /run/user/1000
+++ exited with 0 +++

[rolinh]

I created a branch to attempt to fix this issue (fix-blocking-stat-call).
Note that the issue described here is linux specific since on the BSDs, the MNT_NOWAIT flag is used which prevents calls from blocking.

This 55dec57 fixes the issue for remote file systems. Apart from remote fs, do you see any fs for which this would be a problem?

[klaernie]

Technically this can happen for every filesystem - the underlying layer must just be broken enough. But dfc already behaves correctly, and was just taking a long time.

My biggest thought was about the point, that dfc checks every mount on the system, instead of only the ones it was asked for. With coreutils df it's a bit different - it will only check filesystems that you asked for. So if I run dfc against a single filesystem it should only statfs() this filesystem, and ignore the others.