Recognition of files in a folder works inconsistently between Linux distributions.

[jhojczak]

What happened:
Syft does not recognize binary files on archlinux that are recognized on rockylinux even though the contents of the folder are identical.

I have prepared a script that reproduces this behavior.

The script using 'incus' starts two VMs with different Linux distributions (rockylinux and archlinux) and runs syft from a container inside the VMs to scan the folder. The folder contains the unpacked docker-ce rpm package.
I decided to unpack the rpm before scanning because the purl/cpe generated by syft from the packed package does not allow finding CVEs assigned to docker. Which in most databases are either assigned to the moby project or to the github/docker/docker repository or prul pkg:rpm/docker repository.

What you expected to happen:
Syft should produce the same report from folders containing the same files on both Linux distributions.

Steps to reproduce the issue:

#/bin/bash
set -x
VM_ARCH=arch-tmp
VM_ROCKY=rocky-tmp
incus profile create vm-profile
cat <<EOF >vm-profile.yaml
config:
  security.secureboot: "false"
description: ""
devices:
  agent:
    source: agent:config
    type: disk
  eth0:
    nictype: bridged
    parent: incusbr0
    type: nic
  root:
    path: /
    pool: default
    type: disk
EOF
incus profile edit vm-profile <vm-profile.yaml
incus launch images:archlinux ${VM_ARCH} --vm -p vm-profile
incus launch images:rockylinux/8/cloud ${VM_ROCKY} --vm -p vm-profile
sleep 15

incus exec ${VM_ARCH} -- pacman --noconfirm -Sy docker tree
incus exec ${VM_ARCH} -- systemctl start docker

incus exec ${VM_ROCKY} -- dnf install -y 'dnf-command(config-manager)' curl tree
incus exec ${VM_ROCKY} -- dnf config-manager --add-repo 'https://download.docker.com/linux/centos/docker-ce.repo'
incus exec ${VM_ROCKY} -- dnf install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin
incus exec ${VM_ROCKY} -- systemctl start docker
incus exec ${VM_ROCKY} -- curl -Lo /tmp/docker-ce-23.0.2-1.el8.x86_64.rpm https://download.docker.com/linux/centos/8/x86_64/stable/Packages/docker-ce-23.0.2-1.el8.x86_64.rpm
incus exec ${VM_ROCKY} -- mkdir /tmp/rpm
incus exec --cwd /tmp/rpm ${VM_ROCKY} -- bash -c "rpm2cpio /tmp/docker-ce-23.0.2-1.el8.x86_64.rpm | cpio -idmv"
incus file pull --recursive ${VM_ROCKY}/tmp/rpm ./
incus file push --recursive ./rpm ${VM_ARCH}/tmp
echo "=== Rockylinux ==="
incus exec ${VM_ROCKY} -- tree /tmp/rpm
incus exec ${VM_ROCKY} -- docker run --rm -v /tmp/rpm:/target anchore/syft:v1.2.0 scan dir:/target
echo "=== Archlinux ==="
incus exec ${VM_ARCH} -- tree /tmp/rpm
incus exec ${VM_ARCH} -- docker run --rm -v /tmp/rpm:/target anchore/syft:v1.2.0 scan dir:/target

Anything else we need to know?:
To run the script, you must have 'incus' or lxd installed with the ability to create virtual machines. In the case of lxd, replace the 'incus' command with lxc in the script.
Environment:

• Output of syft version:

Application: syft
Version:    1.2.0
BuildDate:  2024-04-12T18:31:58Z
GitCommit:  dde5d349b1eef740c285255e6a9e3a8f5c9938e1
GitDescription: v1.2.0
Platform:   linux/amd64
GoVersion:  go1.21.9
Compiler:   gc

• Output of cat /etc/os-release:

NAME="Arch Linux"
PRETTY_NAME="Arch Linux"
ID=arch
BUILD_ID=rolling
ANSI_COLOR="38;2;23;147;209"
HOME_URL="https://archlinux.org/"
DOCUMENTATION_URL="https://wiki.archlinux.org/"
SUPPORT_URL="https://bbs.archlinux.org/"
BUG_REPORT_URL="https://gitlab.archlinux.org/groups/archlinux/-/issues"
PRIVACY_POLICY_URL="https://terms.archlinux.org/docs/privacy-policy/"
LOGO=archlinux-logo
=============
NAME="Rocky Linux"
VERSION="8.9 (Green Obsidian)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="8.9"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Rocky Linux 8.9 (Green Obsidian)"
ANSI_COLOR="0;32"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:rocky:rocky:8:GA"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
SUPPORT_END="2029-05-31"
ROCKY_SUPPORT_PRODUCT="Rocky-Linux-8"
ROCKY_SUPPORT_PRODUCT_VERSION="8.9"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.9"

[wagoodman]

I wasn't able to reproduce what you are seeing, specifically doing the equivalent of your script yields the same SBOM for me:

❯ tree
.
├── arch-mount
│   ├── md5sum
│   └── sbom.json
└── rocky-mount
    ├── md5sum
    └── sbom.json

3 directories, 4 files

❯ diff arch-mount/md5sum rocky-mount/md5sum

❯ diff arch-mount/sbom.json rocky-mount/sbom.json

I've attached both (identical) SBOMs to this comment below.

arch-sbom.json
rocky-sbom.json

I didn't use your script exactly since the real steps it appeared you were trying to get across were (using docker instead):

# from rocky
curl -Lo /tmp/docker-ce-23.0.2-1.el8.x86_64.rpm https://download.docker.com/linux/centos/8/x86_64/stable/Packages/docker-ce-23.0.2-1.el8.x86_64.rpm
mkdir /tmp/rpm
rpm2cpio /tmp/docker-ce-23.0.2-1.el8.x86_64.rpm | cpio -idmv
# ... now there is a populated /tmp/rpm dir
syft dir:/tmp/rpm -o json > /volumemount/rocky.sbom.json

# from the host
docker cp alpinectrid:/tmp/rpm ./rpm
docker cp ./rpm rockyctrid:/tmp

# from alpine
syft dir:/tmp/rpm -o json > /volumemount/alpine.sbom.json

The differences from your script and what I did were:

• I used docker instead of lxd/vms
• I used syft 1.3.0 instead of 1.2.0
• I used syft as a binary within the container instead of running syft as a container

One of these differences might be a sensitive factor, so I can try and repeat this again and report back.

[popey]

Thanks for the issue report @jhojczak - I was able to reproduce the same steps and got the same outcome. I appreciate the detailed steps in a script! That made this a lot easier.

This issue was fixed in Syft v1.6.0, likely via PR #2918.

tl;dr This was due to a (now fixed) bug in the way Syft deals with different types of filesystems. The /tmp folder being different in the two distros. On Arch it's a tmpfs. On Rocky it doesn't exist, so your script creates it as a normal folder off the root filesystem.

As such, in Syft up to v1.5.0, on Arch the /target folder is ignored completely, which isn't optimal because that's where you mounted /tmp/rpm - the thing to scan.

---- 8< ----

The long version. i.e. how I figured this out. I did this all on an Ubuntu 24.04 host, with incus installed from the Ubuntu repo.

From verbose logging of syft v1.2.0 (note the -vvv):

incus exec arch_tmp -- docker run --rm -v /tmp/rpm:/target anchore/syft:v1.2.0 scan -vvv dir:/target 2>&1 | tee -a arch-log-verbose-syft-1.2.0.txt

We can see an ignore directive for your /target mount.

$ grep ignoring arch-log-verbose-syft-1.2.0.txt | wc -l
16
$ grep ignoring arch-log-verbose-syft-1.2.0.txt | grep target
[0000] DEBUG ignoring system mountpoint mountpoint=/target
$ echo $?

For Rocky though, things differ. The /target isn't ignored.

$ grep ignoring rocky-log-verbose-syft-1.2.0.txt | wc -l
18
$ grep ignoring rocky-log-verbose-syft-1.2.0.txt | grep target
$ echo $?
1

Here's a very w i d e diff of the logs:

diff --width=$(tput cols) -y --suppress-common-lines arch-log-verbose-syft-1.2.0.txt rocky-log-verbose-syft-1.2.0.txt | more
8 directories, 5 files                                                                                                | 7 directories, 5 files
                                                                                                                      > [0000] DEBUG ignoring system mountpoint mountpoint=/sys/fs/cgroup
[0000] DEBUG ignoring system mountpoint mountpoint=/target                                                            <
                                                                                                                      > [0000] DEBUG ignoring system mountpoint mountpoint=/proc/asound
                                                                                                                      > [0000] DEBUG ignoring system mountpoint mountpoint=/proc/sched_debug
                                                                                                                      > [0000] TRACE indexing filetree path=/target/usr/bin/docker-init
                                                                                                                      > [0000] TRACE indexing filetree path=/target/usr/bin/dockerd
[0000] TRACE searching for paths matching mimetype mimetypes=[application/x-elf application/x-sharedlib application/v | [0000] TRACE searching for paths matching mimetype mimetypes=[application/x-executable application/x-mach-binary appl
[0000] TRACE searching filetree by MIME types types=[application/x-elf application/x-sharedlib application/vnd.micros | [0000] TRACE searching filetree by MIME types types=[application/x-executable application/x-mach-binary application/x
[0000] DEBUG discovered 0 packages cataloger=go-module-binary-cataloger                                               | [0000] TRACE parsing file contents path=/usr/bin/docker-init
                                                                                                                      > [0000] TRACE parsing file contents path=/usr/bin/docker-proxy
                                                                                                                      > [0000] TRACE searching filetree by glob glob=**/go/pkg/mod/github.com/docker/docker/cmd/docker-proxy@(devel)/*
                                                                                                                      > [0000] TRACE parsing file contents path=/usr/bin/dockerd
                                                                                                                      > [0000] TRACE searching filetree by glob glob=**/go/pkg/mod/github.com/docker/docker/cmd/dockerd@(devel)/*
                                                                                                                      > [0000] DEBUG discovered 4 packages cataloger=go-module-binary-cataloger
[0000] TRACE searching filetree by MIME types types=[application/vnd.microsoft.portable-executable application/x-exec | [0000] TRACE searching filetree by MIME types types=[application/x-mach-binary application/x-elf application/x-shared
                                                                                                                      > [0000] TRACE unable to extract SBOM from possible java native-image /usr/bin/docker-init: no symbols found in binary:
                                                                                                                      > [0000] TRACE not a MachO binary error=invalid magic number in record at byte 0x0 filename=/usr/bin/docker-init
                                                                                                                      > [0000] TRACE not a PE binary error=unrecognized PE machine: 0x457f filename=/usr/bin/docker-init
                                                                                                                      > [0000] TRACE unable to extract SBOM from possible java native-image /usr/bin/docker-proxy: one or more symbols are mi
                                                                                                                      > [0000] TRACE not a MachO binary error=invalid magic number in record at byte 0x0 filename=/usr/bin/docker-proxy
                                                                                                                      > [0000] TRACE not a PE binary error=unrecognized PE machine: 0x457f filename=/usr/bin/docker-proxy
                                                                                                                      > [0000] TRACE unable to extract SBOM from possible java native-image /usr/bin/dockerd: one or more symbols are missing
                                                                                                                      > [0000] TRACE not a MachO binary error=invalid magic number in record at byte 0x0 filename=/usr/bin/dockerd
                                                                                                                      > [0000] TRACE not a PE binary error=unrecognized PE machine: 0x457f filename=/usr/bin/dockerd
[0000] TRACE searching filetree by MIME types types=[application/x-sharedlib application/vnd.microsoft.portable-execu | [0000] TRACE searching filetree by MIME types types=[application/x-executable application/x-mach-binary application/x
[0000] DEBUG executable cataloger processed 0 files                                                                   | [0000] TRACE unable to read dynamic symbols from elf file error=no symbol section
[0000] TRACE worker stopped component=eventloop                                                                       | [0000] TRACE unable to read dynamic symbols from elf file error=no symbol section
[0000] TRACE signal exit component=eventloop                                                                          | [0000] TRACE unable to read symbols from elf file error=no symbol section
No packages discovered                                                                                                | [0000] TRACE unable to read symbols from elf file error=no symbol section
                                                                                                                      > [0001] DEBUG executable cataloger processed 3 files
                                                                                                                      > [0001] TRACE worker stopped component=eventloop
                                                                                                                      > [0001] TRACE signal exit component=eventloop
                                                                                                                      > NAME                                       VERSION   TYPE
                                                                                                                      > github.com/docker/docker/cmd/docker-proxy  v23.0.2   go-module
                                                                                                                      > github.com/docker/docker/cmd/dockerd       v23.0.2   go-module
                                                                                                                      > stdlib                                     go1.19.7  go-module  (+1 duplicate)

This likely explains why on Rocky it's producing output for the go packages:

$ tail -n 
NAME                                       VERSION   TYPE
github.com/docker/docker/cmd/docker-proxy  v23.0.2   go-module
github.com/docker/docker/cmd/dockerd       v23.0.2   go-module
stdlib                                     go1.19.7  go-module  (+1 duplicate)

But Arch doesn't:

$ tail -n 4 archlog.txt
[0000] DEBUG executable cataloger processed 0 files
[0000] TRACE worker stopped component=eventloop
[0000] TRACE signal exit component=eventloop
No packages discovered

First thing to check, is this fixed in the latest Syft, v1.10.0?

$ incus exec arch-tmp -- docker run --rm -v /tmp/rpm:/target anchore/syft:v1.10.0 scan dir:/target
NAME                                       VERSION   TYPE
github.com/docker/docker/cmd/docker-proxy  23.0.2    go-module
github.com/docker/docker/cmd/dockerd       23.0.2    go-module
stdlib                                     go1.19.7  go-module  (+1 duplicate)
$ incus exec rocky-tmp -- docker run --rm -v /tmp/rpm:/target anchore/syft:v1.10.0 scan dir:/target
NAME                                       VERSION   TYPE
github.com/docker/docker/cmd/docker-proxy  23.0.2    go-module
github.com/docker/docker/cmd/dockerd       23.0.2    go-module
stdlib                                     go1.19.7  go-module  (+1 duplicate)

Yes. Okay. Which release fixed it? Let's just look at the broken one, Arch.

$ for syftver in v1.2.0 v1.3.0 v1.4.0 v1.4.1 v1.5.0 v1.6.0 v1.7.0 v1.8.0 v1.9.0 v1.10.0; do echo "Syft $syftver"; incus exec arch-tmp -- docker run --rm -v /tmp/rpm:/target anchore/syft:$syftver dir:/target; done
Syft v1.2.0
No packages discovered
Syft v1.3.0
No packages discovered
Syft v1.4.0
No packages discovered
Syft v1.4.1
No packages discovered
Syft v1.5.0
No packages discovered
Syft v1.6.0
NAME                                       VERSION   TYPE
github.com/docker/docker/cmd/docker-proxy  v23.0.2   go-module
github.com/docker/docker/cmd/dockerd       v23.0.2   go-module
stdlib                                     go1.19.7  go-module  (+1 duplicate)
Syft v1.7.0
NAME                                       VERSION   TYPE
github.com/docker/docker/cmd/docker-proxy  v23.0.2   go-module
github.com/docker/docker/cmd/dockerd       v23.0.2   go-module
stdlib                                     go1.19.7  go-module  (+1 duplicate)
Syft v1.8.0
NAME                                       VERSION   TYPE
github.com/docker/docker/cmd/docker-proxy  v23.0.2   go-module
github.com/docker/docker/cmd/dockerd       v23.0.2   go-module
stdlib                                     go1.19.7  go-module  (+1 duplicate)
Syft v1.9.0
NAME                                       VERSION   TYPE
github.com/docker/docker/cmd/docker-proxy  v23.0.2   go-module
github.com/docker/docker/cmd/dockerd       v23.0.2   go-module
stdlib                                     go1.19.7  go-module  (+1 duplicate)
Syft v1.10.0
NAME                                       VERSION   TYPE
github.com/docker/docker/cmd/docker-proxy  23.0.2    go-module
github.com/docker/docker/cmd/dockerd       23.0.2    go-module
stdlib                                     go1.19.7  go-module  (+1 duplicate)

So it's fixed in v1.6.0..

Lets look at the release notes for Syft v1.6.0...

• syft does not find anything in archives if /tmp is a tmpfs #2894 #2918 @willmurphyscode]

Spidey Sense Tingling

The test script uses /tmp/rpm in the incus VM, which is mapped to /target in the docker container for Syft, like this:

incus exec --cwd /tmp/rpm ${VM_ROCKY} -- bash -c "rpm2cpio /tmp/docker-ce-23.0.2-1.el8.x86_64.rpm | cpio -idmv"
incus file pull --recursive ${VM_ROCKY}/tmp/rpm ./
incus file push --recursive ./rpm ${VM_ARCH}/tmp

Indeed the test script runs mkdir /tmp/rpm.

incus exec ${VM_ROCKY} -- mkdir /tmp/rpm`

I suspect there's some difference between /tmp on Arch and Rocky.

$ incus exec rocky-tmp -- mount | grep "/tmp"
$ echo $?
1
$ incus exec arch-tmp -- mount | grep "/tmp"
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,nr_inodes=1048576,inode64)

Ok, I think I'm done here. It was a bug, now isn't, and is yet another lesson for me on the fun with /tmp on different Linux distros.

Spidey-swings out of here

Until next time. Closing this issue

[jhojczak]

@popey @wagoodman Thank you for solving this!:)
@popey Also big thank you for comprehensive explanation :)