Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Building on local image fails with error mounting tmpfs #2244

Open
khaiyichin opened this issue May 22, 2024 · 13 comments
Open

Building on local image fails with error mounting tmpfs #2244

khaiyichin opened this issue May 22, 2024 · 13 comments

Comments

@khaiyichin
Copy link

Version of Apptainer

$ apptainer --version
apptainer version 1.3.1

Expected behavior

Should generate a new image that's based on the first one.

Actual behavior

I built the first image without any issues with sudo, but building the second (that is based on the first) fails with

ERROR:   unpackSIF failed: root filesystem extraction failed: extract command failed: FATAL:   container creation failed: mount hook function failure: mount tmpfs->/var/lib/apptainer/mnt/session error: while mounting tmpfs: can't mount tmpfs filesystem to /var/lib/apptainer/mnt/session: read unix @->@: read: connection reset by peer
: exit status 255
FATAL:   While performing build: packer failed to pack: root filesystem extraction failed: extract command failed: FATAL:   container creation failed: mount hook function failure: mount tmpfs->/var/lib/apptainer/mnt/session error: while mounting tmpfs: can't mount tmpfs filesystem to /var/lib/apptainer/mnt/session: read unix @->@: read: connection reset by peer
: exit status 255

I also ran it with --debug (see below).

Steps to reproduce this behavior

It could be that this is an issue related to my machine; what I did was build one.def and then two.def with sudo apptainer build one.sif one.def; sudo apptainer build two.sif two.def

# one.def
Bootstrap: docker
From: ubuntu:20.04

%labels
    AUTHOR user
    NAME one

%post -c /bin/bash
    export DEBIAN_FRONTEND=noninteractive

    # Terminate on errors
    set -e

# two.def
Bootstrap: localimage
From: one.sif

%labels
    AUTHOR user
    NAME two

%post -c /bin/bash
    
    export DEBIAN_FRONTEND=noninteractive

    # Terminate on errors
    set -e

What OS/distro are you running

$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

How did you install Apptainer

sudo apt install apptainer

Actual behavior when run with --debug

$ sudo apptainer --debug build two.sif two.def 
DEBUG   [U=0,P=11635]      persistentPreRun()            Apptainer version: 1.3.1
DEBUG   [U=0,P=11635]      persistentPreRun()            Parsing configuration file /etc/apptainer/apptainer.conf
DEBUG   [U=0,P=11635]      SetBinaryPath()               Setting binary path to /usr/libexec/apptainer/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
DEBUG   [U=0,P=11635]      SetBinaryPath()               Using that path for all binaries
DEBUG   [U=0,P=11635]      handleConfDir()               /root/.apptainer already exists. Not creating.
DEBUG   [U=0,P=11635]      Init()                        Image format detection
DEBUG   [U=0,P=11635]      Init()                        Check for sandbox image format
DEBUG   [U=0,P=11635]      Init()                        sandbox format initializer returned: not a directory image
DEBUG   [U=0,P=11635]      Init()                        Check for sif image format
DEBUG   [U=0,P=11635]      Init()                        sif format initializer returned: can't read first 2048 bytes: <nil>
DEBUG   [U=0,P=11635]      Init()                        Check for squashfs image format
DEBUG   [U=0,P=11635]      Init()                        squashfs format initializer returned: can't read first 2048 bytes: <nil>
DEBUG   [U=0,P=11635]      Init()                        Check for ext3 image format
DEBUG   [U=0,P=11635]      Init()                        ext3 format initializer returned: can't read first 2048 bytes: <nil>
DEBUG   [U=0,P=11635]      getCacheParentDir()           environment variable APPTAINER_CACHEDIR not set, using default image cache
DEBUG   [U=0,P=11635]      Init()                        Image format detection
DEBUG   [U=0,P=11635]      Init()                        Check for sandbox image format
DEBUG   [U=0,P=11635]      Init()                        sandbox format initializer returned: not a directory image
DEBUG   [U=0,P=11635]      Init()                        Check for sif image format
DEBUG   [U=0,P=11635]      Init()                        sif format initializer returned: can't read first 2048 bytes: <nil>
DEBUG   [U=0,P=11635]      Init()                        Check for squashfs image format
DEBUG   [U=0,P=11635]      Init()                        squashfs format initializer returned: can't read first 2048 bytes: <nil>
DEBUG   [U=0,P=11635]      Init()                        Check for ext3 image format
DEBUG   [U=0,P=11635]      Init()                        ext3 format initializer returned: can't read first 2048 bytes: <nil>
DEBUG   [U=0,P=11635]      newBundle()                   Created temporary directory "/tmp/bundle-temp-4174872336" for the bundle
DEBUG   [U=0,P=11635]      newBundle()                   Created directory "/tmp/build-temp-1166700255/rootfs" for the bundle
DEBUG   [U=0,P=11635]      findOnPath()                  Found "mksquashfs" at "/usr/bin/mksquashfs"
DEBUG   [U=0,P=11635]      ensureGzipComp()              Ensuring gzip compression for mksquashfs
DEBUG   [U=0,P=11635]      findOnPath()                  Found "mksquashfs" at "/usr/bin/mksquashfs"
DEBUG   [U=0,P=11635]      ensureGzipComp()              Gzip compression by default ensured
INFO    [U=0,P=11635]      Full()                        Starting build...
DEBUG   [U=0,P=11635]      Init()                        Image format detection
DEBUG   [U=0,P=11635]      Init()                        Check for sandbox image format
DEBUG   [U=0,P=11635]      Init()                        sandbox format initializer returned: not a directory image
DEBUG   [U=0,P=11635]      Init()                        Check for sif image format
DEBUG   [U=0,P=11635]      Init()                        sif image format detected
DEBUG   [U=0,P=11635]      GetLocalPacker()              Packing from SIF
INFO    [U=0,P=11635]      verifySIF()                   Verifying bootstrap image one.sif
DEBUG   [U=0,P=11635]      GetLocalPacker()              integrity: signature not found for object group 1
DEBUG   [U=0,P=11635]      findOnPath()                  Found "unsquashfs" at "/usr/bin/unsquashfs"
DEBUG   [U=0,P=11635]      extract()                     Trying unsquashfs options: []
DEBUG   [U=0,P=11635]      getLibraryBinds()             Running /lib64/ld-linux-x86-64.so.2 --list /usr/bin/unsquashfs
DEBUG   [U=0,P=11635]      unsquashfsSandboxCmd()        Calling wrapped unsquashfs: apptainer [exec --no-home --no-nv --no-rocm -C --no-init --writable -B /tmp/build-temp-1166700255:/image -B /usr/bin/unsquashfs:/usr/bin/unsquashfs:ro -B /lib/x86_64-linux-gnu/libpthread.so.0:/lib/x86_64-linux-gnu/libpthread.so.0:ro -B /lib/x86_64-linux-gnu/libm.so.6:/lib/x86_64-linux-gnu/libm.so.6:ro -B /lib/x86_64-linux-gnu/libz.so.1:/lib/x86_64-linux-gnu/libz.so.1:ro -B /lib/x86_64-linux-gnu/liblzma.so.5:/lib/x86_64-linux-gnu/liblzma.so.5:ro -B /lib/x86_64-linux-gnu/liblzo2.so.2:/lib/x86_64-linux-gnu/liblzo2.so.2:ro -B /lib/x86_64-linux-gnu/liblz4.so.1:/lib/x86_64-linux-gnu/liblz4.so.1:ro -B /lib/x86_64-linux-gnu/libzstd.so.1:/lib/x86_64-linux-gnu/libzstd.so.1:ro -B /lib/x86_64-linux-gnu/libc.so.6:/lib/x86_64-linux-gnu/libc.so.6:ro -B /lib64/ld-linux-x86-64.so.2:/lib64/ld-linux-x86-64.so.2:ro /tmp/build-temp-1166700255/tmp-rootfs-1835243241 /usr/bin/unsquashfs -f -d /image/rootfs /image/archive-1656700280]
DEBUG   [U=0,P=11635]      unsquashfsSandboxCmd()        LD_LIBRARY_PATH=:/lib/x86_64-linux-gnu:/lib64
DEBUG   [U=0,P=11635]      extract()                     *** BEGIN WRAPPED UNSQUASHFS OUTPUT ***
DEBUG   [U=0,P=11635]      extract()                     DEBUG   [U=0,P=11679]      persistentPreRun()            Apptainer version: 1.3.1
DEBUG   [U=0,P=11679]      persistentPreRun()            Parsing configuration file /etc/apptainer/apptainer.conf
DEBUG   [U=0,P=11679]      SetBinaryPath()               Setting binary path to /usr/libexec/apptainer/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
DEBUG   [U=0,P=11679]      SetBinaryPath()               Using that path for all binaries
DEBUG   [U=0,P=11679]      handleConfDir()               /root/.apptainer already exists. Not creating.
DEBUG   [U=0,P=11679]      setUmask()                    Saving umask 0002 for propagation into container
DEBUG   [U=0,P=11679]      checkEncryptionKey()          Checking for encrypted system partition
DEBUG   [U=0,P=11679]      Init()                        Image format detection
DEBUG   [U=0,P=11679]      Init()                        Check for sandbox image format
DEBUG   [U=0,P=11679]      Init()                        sandbox image format detected
VERBOSE [U=0,P=11679]      setKeyIfNotAlreadyOverridden()Forwarding APPTAINERENV_LD_LIBRARY_PATH as LD_LIBRARY_PATH environment variable
VERBOSE [U=0,P=11679]      SetContainerEnv()             Not forwarding APPTAINER_DEBUG environment variable
VERBOSE [U=0,P=11679]      SetContainerEnv()             Not forwarding APPTAINER_VERBOSE environment variable
VERBOSE [U=0,P=11679]      SetContainerEnv()             Not forwarding APPTAINER_QUIET environment variable
VERBOSE [U=0,P=11679]      SetContainerEnv()             Not forwarding APPTAINER_SILENT environment variable
VERBOSE [U=0,P=11679]      SetContainerEnv()             Not forwarding APPTAINER_MESSAGELEVEL environment variable
VERBOSE [U=0,P=11679]      SetContainerEnv()             Setting HOME=/root
VERBOSE [U=0,P=11679]      SetContainerEnv()             Setting PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
DEBUG   [U=0,P=11679]      InitImageDrivers()            Skipping installing fuseapps image driver because running as root
DEBUG   [U=0,P=11679]      init()                        Use starter binary /usr/libexec/apptainer/bin/starter
VERBOSE [U=0,P=11679]      print()                       Set messagelevel to: 5
VERBOSE [U=0,P=11679]      init()                        Starter initialization
VERBOSE [U=0,P=11679]      is_suid()                     Check if we are running as setuid: 0
DEBUG   [U=0,P=11679]      read_engine_config()          Read engine configuration
DEBUG   [U=0,P=11679]      init()                        Wait completion of stage1
DEBUG   [U=0,P=11691]      set_parent_death_signal()     Set parent death signal to 9
VERBOSE [U=0,P=11691]      init()                        Spawn stage 1
DEBUG   [U=0,P=11691]      func1()                       executablePath is /usr/libexec/apptainer/bin/starter
DEBUG   [U=0,P=11691]      func1()                       starter was not relocated from /usr/libexec
DEBUG   [U=0,P=11691]      func1()                       Install prefix is /usr
DEBUG   [U=0,P=11691]      startup()                     apptainer runtime engine selected
VERBOSE [U=0,P=11691]      startup()                     Execute stage 1
DEBUG   [U=0,P=11691]      StageOne()                    Entering stage 1
DEBUG   [U=0,P=11691]      InitImageDrivers()            Skipping installing fuseapps image driver because running as root
DEBUG   [U=0,P=11691]      prepareRootCaps()             Root full capabilities
DEBUG   [U=0,P=11691]      prepareAutofs()               Found "/proc/sys/fs/binfmt_misc" as autofs mount point
DEBUG   [U=0,P=11691]      prepareAutofs()               Could not keep file descriptor for user bind path /tmp/build-temp-1166700255: no mount point
DEBUG   [U=0,P=11691]      prepareAutofs()               Could not keep file descriptor for user bind path /usr/bin/unsquashfs: no mount point
DEBUG   [U=0,P=11691]      prepareAutofs()               Could not keep file descriptor for user bind path /lib/x86_64-linux-gnu/libpthread.so.0: no mount point
DEBUG   [U=0,P=11691]      prepareAutofs()               Could not keep file descriptor for user bind path /lib/x86_64-linux-gnu/libm.so.6: no mount point
DEBUG   [U=0,P=11691]      prepareAutofs()               Could not keep file descriptor for user bind path /lib/x86_64-linux-gnu/libz.so.1: no mount point
DEBUG   [U=0,P=11691]      prepareAutofs()               Could not keep file descriptor for user bind path /lib/x86_64-linux-gnu/liblzma.so.5: no mount point
DEBUG   [U=0,P=11691]      prepareAutofs()               Could not keep file descriptor for user bind path /lib/x86_64-linux-gnu/liblzo2.so.2: no mount point
DEBUG   [U=0,P=11691]      prepareAutofs()               Could not keep file descriptor for user bind path /lib/x86_64-linux-gnu/liblz4.so.1: no mount point
DEBUG   [U=0,P=11691]      prepareAutofs()               Could not keep file descriptor for user bind path /lib/x86_64-linux-gnu/libzstd.so.1: no mount point
DEBUG   [U=0,P=11691]      prepareAutofs()               Could not keep file descriptor for user bind path /lib/x86_64-linux-gnu/libc.so.6: no mount point
DEBUG   [U=0,P=11691]      prepareAutofs()               Could not keep file descriptor for user bind path /lib64/ld-linux-x86-64.so.2: no mount point
DEBUG   [U=0,P=11691]      prepareAutofs()               Could not keep file descriptor for workdir : no mount point
DEBUG   [U=0,P=11691]      Init()                        Image format detection
DEBUG   [U=0,P=11691]      Init()                        Check for sandbox image format
DEBUG   [U=0,P=11691]      Init()                        sandbox image format detected
DEBUG   [U=0,P=11691]      setSessionLayer()             Not attempting to use overlay or underlay: writable flag requested
DEBUG   [U=0,P=11691]      PrepareConfig()               image driver is 
VERBOSE [U=0,P=11679]      wait_child()                  stage 1 exited with status 0
DEBUG   [U=0,P=11679]      init()                        Applying stage 1 working directory
DEBUG   [U=0,P=11679]      cleanup_fd()                  Close file descriptor 5
DEBUG   [U=0,P=11679]      cleanup_fd()                  Close file descriptor 6
DEBUG   [U=0,P=11679]      cleanup_fd()                  Close file descriptor 7
DEBUG   [U=0,P=11679]      init()                        Set child signal mask
DEBUG   [U=0,P=11679]      init()                        Create socketpair for master communication channel
DEBUG   [U=0,P=11679]      init()                        Create RPC socketpair for communication between stage 2 and RPC server
VERBOSE [U=0,P=11679]      pid_namespace_init()          Create pid namespace
VERBOSE [U=0,P=11679]      init()                        Spawn master process
DEBUG   [U=0,P=1]          set_parent_death_signal()     Set parent death signal to 9
VERBOSE [U=0,P=1]          create_namespace()            Create ipc namespace
VERBOSE [U=0,P=1]          create_namespace()            Create mount namespace
VERBOSE [U=0,P=11679]      enter_namespace()             Entering in mount namespace
DEBUG   [U=0,P=11679]      enter_namespace()             Opening namespace file ns/mnt
VERBOSE [U=0,P=1]          create_namespace()            Create mount namespace
VERBOSE [U=0,P=2]          init()                        Spawn RPC server
DEBUG   [U=0,P=11679]      func1()                       executablePath is /usr/libexec/apptainer/bin/starter
DEBUG   [U=0,P=11679]      func1()                       starter was not relocated from /usr/libexec
DEBUG   [U=0,P=11679]      func1()                       Install prefix is /usr
DEBUG   [U=0,P=11679]      startup()                     apptainer runtime engine selected
VERBOSE [U=0,P=11679]      startup()                     Execute master process
DEBUG   [U=0,P=11679]      InitImageDrivers()            Skipping installing fuseapps image driver because running as root
DEBUG   [U=0,P=11679]      setupSessionLayout()          Using Layer system: none
DEBUG   [U=0,P=11679]      setupDefaultLayout()          Creating default SESSIONDIR layout
DEBUG   [U=0,P=11679]      addRootfsMount()              Mount rootfs in read-write mode
DEBUG   [U=0,P=11679]      addRootfsMount()              Image type is 4098
DEBUG   [U=0,P=11679]      addRootfsMount()              Mounting directory rootfs: /tmp/build-temp-1166700255/tmp-rootfs-1835243241
DEBUG   [U=0,P=11679]      addKernelMount()              Checking configuration file for 'mount proc'
DEBUG   [U=0,P=11679]      addKernelMount()              Adding proc to mount list
VERBOSE [U=0,P=11679]      addKernelMount()              Default mount: /proc:/proc
DEBUG   [U=0,P=11679]      addKernelMount()              Checking configuration file for 'mount sys'
DEBUG   [U=0,P=11679]      addKernelMount()              Adding sysfs to mount list
VERBOSE [U=0,P=11679]      addKernelMount()              Default mount: /sys:/sys
DEBUG   [U=0,P=11679]      addDevMount()                 Checking configuration file for 'mount dev'
DEBUG   [U=0,P=11679]      addDevMount()                 Creating temporary staged /dev
DEBUG   [U=0,P=11679]      addDevMount()                 Creating temporary staged /dev/shm
DEBUG   [U=0,P=11679]      addDevMount()                 Creating temporary staged /dev/mqueue
DEBUG   [U=0,P=11679]      addDevMount()                 Creating temporary staged /dev/pts
DEBUG   [U=0,P=11679]      addDevMount()                 Mounting devpts for staged /dev/pts
DEBUG   [U=0,P=11679]      addSessionDevAt()             Mounting device /dev/tty at /var/lib/apptainer/mnt/session/dev/tty
DEBUG   [U=0,P=11679]      addSessionDevAt()             Mounting device /dev/null at /var/lib/apptainer/mnt/session/dev/null
DEBUG   [U=0,P=11679]      addSessionDevAt()             Mounting device /dev/zero at /var/lib/apptainer/mnt/session/dev/zero
DEBUG   [U=0,P=11679]      addSessionDevAt()             Mounting device /dev/random at /var/lib/apptainer/mnt/session/dev/random
DEBUG   [U=0,P=11679]      addSessionDevAt()             Mounting device /dev/urandom at /var/lib/apptainer/mnt/session/dev/urandom
DEBUG   [U=0,P=11679]      addSessionDevAt()             Adding symlink device /dev/fd to /proc/self/fd at /var/lib/apptainer/mnt/session/dev/fd
DEBUG   [U=0,P=11679]      addSessionDevAt()             Adding symlink device /dev/stdin to /proc/self/fd/0 at /var/lib/apptainer/mnt/session/dev/stdin
DEBUG   [U=0,P=11679]      addSessionDevAt()             Adding symlink device /dev/stdout to /proc/self/fd/1 at /var/lib/apptainer/mnt/session/dev/stdout
DEBUG   [U=0,P=11679]      addSessionDevAt()             Adding symlink device /dev/stderr to /proc/self/fd/2 at /var/lib/apptainer/mnt/session/dev/stderr
DEBUG   [U=0,P=11679]      addHostMount()                Not mounting host file systems per configuration
DEBUG   [U=0,P=11679]      addBindsMount()               Binding /etc/hosts and /etc/localtime only with contain
DEBUG   [U=0,P=11679]      addHomeMount()                Skipping home directory mount by user request.
DEBUG   [U=0,P=11679]      addUserbindsMount()           Adding /tmp/build-temp-1166700255 to mount list
DEBUG   [U=0,P=11679]      addUserbindsMount()           Adding /usr/bin/unsquashfs to mount list
DEBUG   [U=0,P=11679]      addUserbindsMount()           Adding /lib/x86_64-linux-gnu/libpthread.so.0 to mount list
DEBUG   [U=0,P=11679]      addUserbindsMount()           Adding /lib/x86_64-linux-gnu/libm.so.6 to mount list
DEBUG   [U=0,P=11679]      addUserbindsMount()           Adding /lib/x86_64-linux-gnu/libz.so.1 to mount list
DEBUG   [U=0,P=11679]      addUserbindsMount()           Adding /lib/x86_64-linux-gnu/liblzma.so.5 to mount list
DEBUG   [U=0,P=11679]      addUserbindsMount()           Adding /lib/x86_64-linux-gnu/liblzo2.so.2 to mount list
DEBUG   [U=0,P=11679]      addUserbindsMount()           Adding /lib/x86_64-linux-gnu/liblz4.so.1 to mount list
DEBUG   [U=0,P=11679]      addUserbindsMount()           Adding /lib/x86_64-linux-gnu/libzstd.so.1 to mount list
DEBUG   [U=0,P=11679]      addUserbindsMount()           Adding /lib/x86_64-linux-gnu/libc.so.6 to mount list
DEBUG   [U=0,P=11679]      addUserbindsMount()           Adding /lib64/ld-linux-x86-64.so.2 to mount list
DEBUG   [U=0,P=11679]      addTmpMount()                 Checking for 'mount tmp' in configuration file
DEBUG   [U=0,P=11679]      addScratchMount()             Not mounting scratch directory: Not requested
DEBUG   [U=0,P=11679]      addLibsMount()                Checking for 'user bind control' in configuration file
DEBUG   [U=0,P=11679]      addFilesMount()               Checking for 'user bind control' in configuration file
DEBUG   [U=0,P=11679]      addResolvConfMount()          Adding /etc/resolv.conf to mount list
VERBOSE [U=0,P=11679]      addResolvConfMount()          Default mount: /etc/resolv.conf:/etc/resolv.conf
DEBUG   [U=0,P=11679]      addHostnameMount()            Skipping hostname mount, not virtualizing UTS namespace on user request
DEBUG   [U=0,P=11679]      create()                      Mount all
DEBUG   [U=0,P=11679]      mountGeneric()                Mounting tmpfs to /var/lib/apptainer/mnt/session
VERBOSE [U=0,P=1]          wait_child()                  rpc server interrupted by signal number 11
DEBUG   [U=0,P=11679]      CleanupContainer()            Cleanup container
FATAL   [U=0,P=11679]      Master()                      container creation failed: mount hook function failure: mount tmpfs->/var/lib/apptainer/mnt/session error: while mounting tmpfs: can't mount tmpfs filesystem to /var/lib/apptainer/mnt/session: read unix @->@: read: connection reset by peer
DEBUG   [U=0,P=11635]      extract()                     *** END WRAPPED UNSQUASHFS OUTPUT ***
ERROR   [U=0,P=11635]      Pack()                        unpackSIF failed: root filesystem extraction failed: extract command failed: DEBUG   [U=0,P=11679]      persistentPreRun()            Apptainer version: 1.3.1
DEBUG   [U=0,P=11679]      persistentPreRun()            Parsing configuration file /etc/apptainer/apptainer.conf
DEBUG   [U=0,P=11679]      SetBinaryPath()               Setting binary path to /usr/libexec/apptainer/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
DEBUG   [U=0,P=11679]      SetBinaryPath()               Using that path for all binaries
DEBUG   [U=0,P=11679]      handleConfDir()               /root/.apptainer already exists. Not creating.
DEBUG   [U=0,P=11679]      setUmask()                    Saving umask 0002 for propagation into container
DEBUG   [U=0,P=11679]      checkEncryptionKey()          Checking for encrypted system partition
DEBUG   [U=0,P=11679]      Init()                        Image format detection
DEBUG   [U=0,P=11679]      Init()                        Check for sandbox image format
DEBUG   [U=0,P=11679]      Init()                        sandbox image format detected
VERBOSE [U=0,P=11679]      setKeyIfNotAlreadyOverridden()Forwarding APPTAINERENV_LD_LIBRARY_PATH as LD_LIBRARY_PATH environment variable
VERBOSE [U=0,P=11679]      SetContainerEnv()             Not forwarding APPTAINER_DEBUG environment variable
VERBOSE [U=0,P=11679]      SetContainerEnv()             Not forwarding APPTAINER_VERBOSE environment variable
VERBOSE [U=0,P=11679]      SetContainerEnv()             Not forwarding APPTAINER_QUIET environment variable
VERBOSE [U=0,P=11679]      SetContainerEnv()             Not forwarding APPTAINER_SILENT environment variable
VERBOSE [U=0,P=11679]      SetContainerEnv()             Not forwarding APPTAINER_MESSAGELEVEL environment variable
VERBOSE [U=0,P=11679]      SetContainerEnv()             Setting HOME=/root
VERBOSE [U=0,P=11679]      SetContainerEnv()             Setting PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
DEBUG   [U=0,P=11679]      InitImageDrivers()            Skipping installing fuseapps image driver because running as root
DEBUG   [U=0,P=11679]      init()                        Use starter binary /usr/libexec/apptainer/bin/starter
VERBOSE [U=0,P=11679]      print()                       Set messagelevel to: 5
VERBOSE [U=0,P=11679]      init()                        Starter initialization
VERBOSE [U=0,P=11679]      is_suid()                     Check if we are running as setuid: 0
DEBUG   [U=0,P=11679]      read_engine_config()          Read engine configuration
DEBUG   [U=0,P=11679]      init()                        Wait completion of stage1
DEBUG   [U=0,P=11691]      set_parent_death_signal()     Set parent death signal to 9
VERBOSE [U=0,P=11691]      init()                        Spawn stage 1
DEBUG   [U=0,P=11691]      func1()                       executablePath is /usr/libexec/apptainer/bin/starter
DEBUG   [U=0,P=11691]      func1()                       starter was not relocated from /usr/libexec
DEBUG   [U=0,P=11691]      func1()                       Install prefix is /usr
DEBUG   [U=0,P=11691]      startup()                     apptainer runtime engine selected
VERBOSE [U=0,P=11691]      startup()                     Execute stage 1
DEBUG   [U=0,P=11691]      StageOne()                    Entering stage 1
DEBUG   [U=0,P=11691]      InitImageDrivers()            Skipping installing fuseapps image driver because running as root
DEBUG   [U=0,P=11691]      prepareRootCaps()             Root full capabilities
DEBUG   [U=0,P=11691]      prepareAutofs()               Found "/proc/sys/fs/binfmt_misc" as autofs mount point
DEBUG   [U=0,P=11691]      prepareAutofs()               Could not keep file descriptor for user bind path /tmp/build-temp-1166700255: no mount point
DEBUG   [U=0,P=11691]      prepareAutofs()               Could not keep file descriptor for user bind path /usr/bin/unsquashfs: no mount point
DEBUG   [U=0,P=11691]      prepareAutofs()               Could not keep file descriptor for user bind path /lib/x86_64-linux-gnu/libpthread.so.0: no mount point
DEBUG   [U=0,P=11691]      prepareAutofs()               Could not keep file descriptor for user bind path /lib/x86_64-linux-gnu/libm.so.6: no mount point
DEBUG   [U=0,P=11691]      prepareAutofs()               Could not keep file descriptor for user bind path /lib/x86_64-linux-gnu/libz.so.1: no mount point
DEBUG   [U=0,P=11691]      prepareAutofs()               Could not keep file descriptor for user bind path /lib/x86_64-linux-gnu/liblzma.so.5: no mount point
DEBUG   [U=0,P=11691]      prepareAutofs()               Could not keep file descriptor for user bind path /lib/x86_64-linux-gnu/liblzo2.so.2: no mount point
DEBUG   [U=0,P=11691]      prepareAutofs()               Could not keep file descriptor for user bind path /lib/x86_64-linux-gnu/liblz4.so.1: no mount point
DEBUG   [U=0,P=11691]      prepareAutofs()               Could not keep file descriptor for user bind path /lib/x86_64-linux-gnu/libzstd.so.1: no mount point
DEBUG   [U=0,P=11691]      prepareAutofs()               Could not keep file descriptor for user bind path /lib/x86_64-linux-gnu/libc.so.6: no mount point
DEBUG   [U=0,P=11691]      prepareAutofs()               Could not keep file descriptor for user bind path /lib64/ld-linux-x86-64.so.2: no mount point
DEBUG   [U=0,P=11691]      prepareAutofs()               Could not keep file descriptor for workdir : no mount point
DEBUG   [U=0,P=11691]      Init()                        Image format detection
DEBUG   [U=0,P=11691]      Init()                        Check for sandbox image format
DEBUG   [U=0,P=11691]      Init()                        sandbox image format detected
DEBUG   [U=0,P=11691]      setSessionLayer()             Not attempting to use overlay or underlay: writable flag requested
DEBUG   [U=0,P=11691]      PrepareConfig()               image driver is 
VERBOSE [U=0,P=11679]      wait_child()                  stage 1 exited with status 0
DEBUG   [U=0,P=11679]      init()                        Applying stage 1 working directory
DEBUG   [U=0,P=11679]      cleanup_fd()                  Close file descriptor 5
DEBUG   [U=0,P=11679]      cleanup_fd()                  Close file descriptor 6
DEBUG   [U=0,P=11679]      cleanup_fd()                  Close file descriptor 7
DEBUG   [U=0,P=11679]      init()                        Set child signal mask
DEBUG   [U=0,P=11679]      init()                        Create socketpair for master communication channel
DEBUG   [U=0,P=11679]      init()                        Create RPC socketpair for communication between stage 2 and RPC server
VERBOSE [U=0,P=11679]      pid_namespace_init()          Create pid namespace
VERBOSE [U=0,P=11679]      init()                        Spawn master process
DEBUG   [U=0,P=1]          set_parent_death_signal()     Set parent death signal to 9
VERBOSE [U=0,P=1]          create_namespace()            Create ipc namespace
VERBOSE [U=0,P=1]          create_namespace()            Create mount namespace
VERBOSE [U=0,P=11679]      enter_namespace()             Entering in mount namespace
DEBUG   [U=0,P=11679]      enter_namespace()             Opening namespace file ns/mnt
VERBOSE [U=0,P=1]          create_namespace()            Create mount namespace
VERBOSE [U=0,P=2]          init()                        Spawn RPC server
DEBUG   [U=0,P=11679]      func1()                       executablePath is /usr/libexec/apptainer/bin/starter
DEBUG   [U=0,P=11679]      func1()                       starter was not relocated from /usr/libexec
DEBUG   [U=0,P=11679]      func1()                       Install prefix is /usr
DEBUG   [U=0,P=11679]      startup()                     apptainer runtime engine selected
VERBOSE [U=0,P=11679]      startup()                     Execute master process
DEBUG   [U=0,P=11679]      InitImageDrivers()            Skipping installing fuseapps image driver because running as root
DEBUG   [U=0,P=11679]      setupSessionLayout()          Using Layer system: none
DEBUG   [U=0,P=11679]      setupDefaultLayout()          Creating default SESSIONDIR layout
DEBUG   [U=0,P=11679]      addRootfsMount()              Mount rootfs in read-write mode
DEBUG   [U=0,P=11679]      addRootfsMount()              Image type is 4098
DEBUG   [U=0,P=11679]      addRootfsMount()              Mounting directory rootfs: /tmp/build-temp-1166700255/tmp-rootfs-1835243241
DEBUG   [U=0,P=11679]      addKernelMount()              Checking configuration file for 'mount proc'
DEBUG   [U=0,P=11679]      addKernelMount()              Adding proc to mount list
VERBOSE [U=0,P=11679]      addKernelMount()              Default mount: /proc:/proc
DEBUG   [U=0,P=11679]      addKernelMount()              Checking configuration file for 'mount sys'
DEBUG   [U=0,P=11679]      addKernelMount()              Adding sysfs to mount list
VERBOSE [U=0,P=11679]      addKernelMount()              Default mount: /sys:/sys
DEBUG   [U=0,P=11679]      addDevMount()                 Checking configuration file for 'mount dev'
DEBUG   [U=0,P=11679]      addDevMount()                 Creating temporary staged /dev
DEBUG   [U=0,P=11679]      addDevMount()                 Creating temporary staged /dev/shm
DEBUG   [U=0,P=11679]      addDevMount()                 Creating temporary staged /dev/mqueue
DEBUG   [U=0,P=11679]      addDevMount()                 Creating temporary staged /dev/pts
DEBUG   [U=0,P=11679]      addDevMount()                 Mounting devpts for staged /dev/pts
DEBUG   [U=0,P=11679]      addSessionDevAt()             Mounting device /dev/tty at /var/lib/apptainer/mnt/session/dev/tty
DEBUG   [U=0,P=11679]      addSessionDevAt()             Mounting device /dev/null at /var/lib/apptainer/mnt/session/dev/null
DEBUG   [U=0,P=11679]      addSessionDevAt()             Mounting device /dev/zero at /var/lib/apptainer/mnt/session/dev/zero
DEBUG   [U=0,P=11679]      addSessionDevAt()             Mounting device /dev/random at /var/lib/apptainer/mnt/session/dev/random
DEBUG   [U=0,P=11679]      addSessionDevAt()             Mounting device /dev/urandom at /var/lib/apptainer/mnt/session/dev/urandom
DEBUG   [U=0,P=11679]      addSessionDevAt()             Adding symlink device /dev/fd to /proc/self/fd at /var/lib/apptainer/mnt/session/dev/fd
DEBUG   [U=0,P=11679]      addSessionDevAt()             Adding symlink device /dev/stdin to /proc/self/fd/0 at /var/lib/apptainer/mnt/session/dev/stdin
DEBUG   [U=0,P=11679]      addSessionDevAt()             Adding symlink device /dev/stdout to /proc/self/fd/1 at /var/lib/apptainer/mnt/session/dev/stdout
DEBUG   [U=0,P=11679]      addSessionDevAt()             Adding symlink device /dev/stderr to /proc/self/fd/2 at /var/lib/apptainer/mnt/session/dev/stderr
DEBUG   [U=0,P=11679]      addHostMount()                Not mounting host file systems per configuration
DEBUG   [U=0,P=11679]      addBindsMount()               Binding /etc/hosts and /etc/localtime only with contain
DEBUG   [U=0,P=11679]      addHomeMount()                Skipping home directory mount by user request.
DEBUG   [U=0,P=11679]      addUserbindsMount()           Adding /tmp/build-temp-1166700255 to mount list
DEBUG   [U=0,P=11679]      addUserbindsMount()           Adding /usr/bin/unsquashfs to mount list
DEBUG   [U=0,P=11679]      addUserbindsMount()           Adding /lib/x86_64-linux-gnu/libpthread.so.0 to mount list
DEBUG   [U=0,P=11679]      addUserbindsMount()           Adding /lib/x86_64-linux-gnu/libm.so.6 to mount list
DEBUG   [U=0,P=11679]      addUserbindsMount()           Adding /lib/x86_64-linux-gnu/libz.so.1 to mount list
DEBUG   [U=0,P=11679]      addUserbindsMount()           Adding /lib/x86_64-linux-gnu/liblzma.so.5 to mount list
DEBUG   [U=0,P=11679]      addUserbindsMount()           Adding /lib/x86_64-linux-gnu/liblzo2.so.2 to mount list
DEBUG   [U=0,P=11679]      addUserbindsMount()           Adding /lib/x86_64-linux-gnu/liblz4.so.1 to mount list
DEBUG   [U=0,P=11679]      addUserbindsMount()           Adding /lib/x86_64-linux-gnu/libzstd.so.1 to mount list
DEBUG   [U=0,P=11679]      addUserbindsMount()           Adding /lib/x86_64-linux-gnu/libc.so.6 to mount list
DEBUG   [U=0,P=11679]      addUserbindsMount()           Adding /lib64/ld-linux-x86-64.so.2 to mount list
DEBUG   [U=0,P=11679]      addTmpMount()                 Checking for 'mount tmp' in configuration file
DEBUG   [U=0,P=11679]      addScratchMount()             Not mounting scratch directory: Not requested
DEBUG   [U=0,P=11679]      addLibsMount()                Checking for 'user bind control' in configuration file
DEBUG   [U=0,P=11679]      addFilesMount()               Checking for 'user bind control' in configuration file
DEBUG   [U=0,P=11679]      addResolvConfMount()          Adding /etc/resolv.conf to mount list
VERBOSE [U=0,P=11679]      addResolvConfMount()          Default mount: /etc/resolv.conf:/etc/resolv.conf
DEBUG   [U=0,P=11679]      addHostnameMount()            Skipping hostname mount, not virtualizing UTS namespace on user request
DEBUG   [U=0,P=11679]      create()                      Mount all
DEBUG   [U=0,P=11679]      mountGeneric()                Mounting tmpfs to /var/lib/apptainer/mnt/session
VERBOSE [U=0,P=1]          wait_child()                  rpc server interrupted by signal number 11
DEBUG   [U=0,P=11679]      CleanupContainer()            Cleanup container
FATAL   [U=0,P=11679]      Master()                      container creation failed: mount hook function failure: mount tmpfs->/var/lib/apptainer/mnt/session error: while mounting tmpfs: can't mount tmpfs filesystem to /var/lib/apptainer/mnt/session: read unix @->@: read: connection reset by peer
: exit status 255
DEBUG   [U=0,P=11635]      cleanUp()                     Cleaning up "/tmp/build-temp-1166700255/rootfs" and "/tmp/bundle-temp-4174872336"
FATAL   [U=0,P=11635]      runBuildLocal()               While performing build: packer failed to pack: root filesystem extraction failed: extract command failed: DEBUG   [U=0,P=11679]      persistentPreRun()            Apptainer version: 1.3.1
DEBUG   [U=0,P=11679]      persistentPreRun()            Parsing configuration file /etc/apptainer/apptainer.conf
DEBUG   [U=0,P=11679]      SetBinaryPath()               Setting binary path to /usr/libexec/apptainer/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
DEBUG   [U=0,P=11679]      SetBinaryPath()               Using that path for all binaries
DEBUG   [U=0,P=11679]      handleConfDir()               /root/.apptainer already exists. Not creating.
DEBUG   [U=0,P=11679]      setUmask()                    Saving umask 0002 for propagation into container
DEBUG   [U=0,P=11679]      checkEncryptionKey()          Checking for encrypted system partition
DEBUG   [U=0,P=11679]      Init()                        Image format detection
DEBUG   [U=0,P=11679]      Init()                        Check for sandbox image format
DEBUG   [U=0,P=11679]      Init()                        sandbox image format detected
VERBOSE [U=0,P=11679]      setKeyIfNotAlreadyOverridden()Forwarding APPTAINERENV_LD_LIBRARY_PATH as LD_LIBRARY_PATH environment variable
VERBOSE [U=0,P=11679]      SetContainerEnv()             Not forwarding APPTAINER_DEBUG environment variable
VERBOSE [U=0,P=11679]      SetContainerEnv()             Not forwarding APPTAINER_VERBOSE environment variable
VERBOSE [U=0,P=11679]      SetContainerEnv()             Not forwarding APPTAINER_QUIET environment variable
VERBOSE [U=0,P=11679]      SetContainerEnv()             Not forwarding APPTAINER_SILENT environment variable
VERBOSE [U=0,P=11679]      SetContainerEnv()             Not forwarding APPTAINER_MESSAGELEVEL environment variable
VERBOSE [U=0,P=11679]      SetContainerEnv()             Setting HOME=/root
VERBOSE [U=0,P=11679]      SetContainerEnv()             Setting PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
DEBUG   [U=0,P=11679]      InitImageDrivers()            Skipping installing fuseapps image driver because running as root
DEBUG   [U=0,P=11679]      init()                        Use starter binary /usr/libexec/apptainer/bin/starter
VERBOSE [U=0,P=11679]      print()                       Set messagelevel to: 5
VERBOSE [U=0,P=11679]      init()                        Starter initialization
VERBOSE [U=0,P=11679]      is_suid()                     Check if we are running as setuid: 0
DEBUG   [U=0,P=11679]      read_engine_config()          Read engine configuration
DEBUG   [U=0,P=11679]      init()                        Wait completion of stage1
DEBUG   [U=0,P=11691]      set_parent_death_signal()     Set parent death signal to 9
VERBOSE [U=0,P=11691]      init()                        Spawn stage 1
DEBUG   [U=0,P=11691]      func1()                       executablePath is /usr/libexec/apptainer/bin/starter
DEBUG   [U=0,P=11691]      func1()                       starter was not relocated from /usr/libexec
DEBUG   [U=0,P=11691]      func1()                       Install prefix is /usr
DEBUG   [U=0,P=11691]      startup()                     apptainer runtime engine selected
VERBOSE [U=0,P=11691]      startup()                     Execute stage 1
DEBUG   [U=0,P=11691]      StageOne()                    Entering stage 1
DEBUG   [U=0,P=11691]      InitImageDrivers()            Skipping installing fuseapps image driver because running as root
DEBUG   [U=0,P=11691]      prepareRootCaps()             Root full capabilities
DEBUG   [U=0,P=11691]      prepareAutofs()               Found "/proc/sys/fs/binfmt_misc" as autofs mount point
DEBUG   [U=0,P=11691]      prepareAutofs()               Could not keep file descriptor for user bind path /tmp/build-temp-1166700255: no mount point
DEBUG   [U=0,P=11691]      prepareAutofs()               Could not keep file descriptor for user bind path /usr/bin/unsquashfs: no mount point
DEBUG   [U=0,P=11691]      prepareAutofs()               Could not keep file descriptor for user bind path /lib/x86_64-linux-gnu/libpthread.so.0: no mount point
DEBUG   [U=0,P=11691]      prepareAutofs()               Could not keep file descriptor for user bind path /lib/x86_64-linux-gnu/libm.so.6: no mount point
DEBUG   [U=0,P=11691]      prepareAutofs()               Could not keep file descriptor for user bind path /lib/x86_64-linux-gnu/libz.so.1: no mount point
DEBUG   [U=0,P=11691]      prepareAutofs()               Could not keep file descriptor for user bind path /lib/x86_64-linux-gnu/liblzma.so.5: no mount point
DEBUG   [U=0,P=11691]      prepareAutofs()               Could not keep file descriptor for user bind path /lib/x86_64-linux-gnu/liblzo2.so.2: no mount point
DEBUG   [U=0,P=11691]      prepareAutofs()               Could not keep file descriptor for user bind path /lib/x86_64-linux-gnu/liblz4.so.1: no mount point
DEBUG   [U=0,P=11691]      prepareAutofs()               Could not keep file descriptor for user bind path /lib/x86_64-linux-gnu/libzstd.so.1: no mount point
DEBUG   [U=0,P=11691]      prepareAutofs()               Could not keep file descriptor for user bind path /lib/x86_64-linux-gnu/libc.so.6: no mount point
DEBUG   [U=0,P=11691]      prepareAutofs()               Could not keep file descriptor for user bind path /lib64/ld-linux-x86-64.so.2: no mount point
DEBUG   [U=0,P=11691]      prepareAutofs()               Could not keep file descriptor for workdir : no mount point
DEBUG   [U=0,P=11691]      Init()                        Image format detection
DEBUG   [U=0,P=11691]      Init()                        Check for sandbox image format
DEBUG   [U=0,P=11691]      Init()                        sandbox image format detected
DEBUG   [U=0,P=11691]      setSessionLayer()             Not attempting to use overlay or underlay: writable flag requested
DEBUG   [U=0,P=11691]      PrepareConfig()               image driver is 
VERBOSE [U=0,P=11679]      wait_child()                  stage 1 exited with status 0
DEBUG   [U=0,P=11679]      init()                        Applying stage 1 working directory
DEBUG   [U=0,P=11679]      cleanup_fd()                  Close file descriptor 5
DEBUG   [U=0,P=11679]      cleanup_fd()                  Close file descriptor 6
DEBUG   [U=0,P=11679]      cleanup_fd()                  Close file descriptor 7
DEBUG   [U=0,P=11679]      init()                        Set child signal mask
DEBUG   [U=0,P=11679]      init()                        Create socketpair for master communication channel
DEBUG   [U=0,P=11679]      init()                        Create RPC socketpair for communication between stage 2 and RPC server
VERBOSE [U=0,P=11679]      pid_namespace_init()          Create pid namespace
VERBOSE [U=0,P=11679]      init()                        Spawn master process
DEBUG   [U=0,P=1]          set_parent_death_signal()     Set parent death signal to 9
VERBOSE [U=0,P=1]          create_namespace()            Create ipc namespace
VERBOSE [U=0,P=1]          create_namespace()            Create mount namespace
VERBOSE [U=0,P=11679]      enter_namespace()             Entering in mount namespace
DEBUG   [U=0,P=11679]      enter_namespace()             Opening namespace file ns/mnt
VERBOSE [U=0,P=1]          create_namespace()            Create mount namespace
VERBOSE [U=0,P=2]          init()                        Spawn RPC server
DEBUG   [U=0,P=11679]      func1()                       executablePath is /usr/libexec/apptainer/bin/starter
DEBUG   [U=0,P=11679]      func1()                       starter was not relocated from /usr/libexec
DEBUG   [U=0,P=11679]      func1()                       Install prefix is /usr
DEBUG   [U=0,P=11679]      startup()                     apptainer runtime engine selected
VERBOSE [U=0,P=11679]      startup()                     Execute master process
DEBUG   [U=0,P=11679]      InitImageDrivers()            Skipping installing fuseapps image driver because running as root
DEBUG   [U=0,P=11679]      setupSessionLayout()          Using Layer system: none
DEBUG   [U=0,P=11679]      setupDefaultLayout()          Creating default SESSIONDIR layout
DEBUG   [U=0,P=11679]      addRootfsMount()              Mount rootfs in read-write mode
DEBUG   [U=0,P=11679]      addRootfsMount()              Image type is 4098
DEBUG   [U=0,P=11679]      addRootfsMount()              Mounting directory rootfs: /tmp/build-temp-1166700255/tmp-rootfs-1835243241
DEBUG   [U=0,P=11679]      addKernelMount()              Checking configuration file for 'mount proc'
DEBUG   [U=0,P=11679]      addKernelMount()              Adding proc to mount list
VERBOSE [U=0,P=11679]      addKernelMount()              Default mount: /proc:/proc
DEBUG   [U=0,P=11679]      addKernelMount()              Checking configuration file for 'mount sys'
DEBUG   [U=0,P=11679]      addKernelMount()              Adding sysfs to mount list
VERBOSE [U=0,P=11679]      addKernelMount()              Default mount: /sys:/sys
DEBUG   [U=0,P=11679]      addDevMount()                 Checking configuration file for 'mount dev'
DEBUG   [U=0,P=11679]      addDevMount()                 Creating temporary staged /dev
DEBUG   [U=0,P=11679]      addDevMount()                 Creating temporary staged /dev/shm
DEBUG   [U=0,P=11679]      addDevMount()                 Creating temporary staged /dev/mqueue
DEBUG   [U=0,P=11679]      addDevMount()                 Creating temporary staged /dev/pts
DEBUG   [U=0,P=11679]      addDevMount()                 Mounting devpts for staged /dev/pts
DEBUG   [U=0,P=11679]      addSessionDevAt()             Mounting device /dev/tty at /var/lib/apptainer/mnt/session/dev/tty
DEBUG   [U=0,P=11679]      addSessionDevAt()             Mounting device /dev/null at /var/lib/apptainer/mnt/session/dev/null
DEBUG   [U=0,P=11679]      addSessionDevAt()             Mounting device /dev/zero at /var/lib/apptainer/mnt/session/dev/zero
DEBUG   [U=0,P=11679]      addSessionDevAt()             Mounting device /dev/random at /var/lib/apptainer/mnt/session/dev/random
DEBUG   [U=0,P=11679]      addSessionDevAt()             Mounting device /dev/urandom at /var/lib/apptainer/mnt/session/dev/urandom
DEBUG   [U=0,P=11679]      addSessionDevAt()             Adding symlink device /dev/fd to /proc/self/fd at /var/lib/apptainer/mnt/session/dev/fd
DEBUG   [U=0,P=11679]      addSessionDevAt()             Adding symlink device /dev/stdin to /proc/self/fd/0 at /var/lib/apptainer/mnt/session/dev/stdin
DEBUG   [U=0,P=11679]      addSessionDevAt()             Adding symlink device /dev/stdout to /proc/self/fd/1 at /var/lib/apptainer/mnt/session/dev/stdout
DEBUG   [U=0,P=11679]      addSessionDevAt()             Adding symlink device /dev/stderr to /proc/self/fd/2 at /var/lib/apptainer/mnt/session/dev/stderr
DEBUG   [U=0,P=11679]      addHostMount()                Not mounting host file systems per configuration
DEBUG   [U=0,P=11679]      addBindsMount()               Binding /etc/hosts and /etc/localtime only with contain
DEBUG   [U=0,P=11679]      addHomeMount()                Skipping home directory mount by user request.
DEBUG   [U=0,P=11679]      addUserbindsMount()           Adding /tmp/build-temp-1166700255 to mount list
DEBUG   [U=0,P=11679]      addUserbindsMount()           Adding /usr/bin/unsquashfs to mount list
DEBUG   [U=0,P=11679]      addUserbindsMount()           Adding /lib/x86_64-linux-gnu/libpthread.so.0 to mount list
DEBUG   [U=0,P=11679]      addUserbindsMount()           Adding /lib/x86_64-linux-gnu/libm.so.6 to mount list
DEBUG   [U=0,P=11679]      addUserbindsMount()           Adding /lib/x86_64-linux-gnu/libz.so.1 to mount list
DEBUG   [U=0,P=11679]      addUserbindsMount()           Adding /lib/x86_64-linux-gnu/liblzma.so.5 to mount list
DEBUG   [U=0,P=11679]      addUserbindsMount()           Adding /lib/x86_64-linux-gnu/liblzo2.so.2 to mount list
DEBUG   [U=0,P=11679]      addUserbindsMount()           Adding /lib/x86_64-linux-gnu/liblz4.so.1 to mount list
DEBUG   [U=0,P=11679]      addUserbindsMount()           Adding /lib/x86_64-linux-gnu/libzstd.so.1 to mount list
DEBUG   [U=0,P=11679]      addUserbindsMount()           Adding /lib/x86_64-linux-gnu/libc.so.6 to mount list
DEBUG   [U=0,P=11679]      addUserbindsMount()           Adding /lib64/ld-linux-x86-64.so.2 to mount list
DEBUG   [U=0,P=11679]      addTmpMount()                 Checking for 'mount tmp' in configuration file
DEBUG   [U=0,P=11679]      addScratchMount()             Not mounting scratch directory: Not requested
DEBUG   [U=0,P=11679]      addLibsMount()                Checking for 'user bind control' in configuration file
DEBUG   [U=0,P=11679]      addFilesMount()               Checking for 'user bind control' in configuration file
DEBUG   [U=0,P=11679]      addResolvConfMount()          Adding /etc/resolv.conf to mount list
VERBOSE [U=0,P=11679]      addResolvConfMount()          Default mount: /etc/resolv.conf:/etc/resolv.conf
DEBUG   [U=0,P=11679]      addHostnameMount()            Skipping hostname mount, not virtualizing UTS namespace on user request
DEBUG   [U=0,P=11679]      create()                      Mount all
DEBUG   [U=0,P=11679]      mountGeneric()                Mounting tmpfs to /var/lib/apptainer/mnt/session
VERBOSE [U=0,P=1]          wait_child()                  rpc server interrupted by signal number 11
DEBUG   [U=0,P=11679]      CleanupContainer()            Cleanup container
FATAL   [U=0,P=11679]      Master()                      container creation failed: mount hook function failure: mount tmpfs->/var/lib/apptainer/mnt/session error: while mounting tmpfs: can't mount tmpfs filesystem to /var/lib/apptainer/mnt/session: read unix @->@: read: connection reset by peer
: exit status 255
@khaiyichin khaiyichin changed the title Building on local image fails with error mouting tmpfs Building on local image fails with error mounting tmpfs May 22, 2024
@garymansellricardo
Copy link

garymansellricardo commented May 29, 2024 •
edited
Loading

I see this on Ubuntu 20.04 and apptainer-suid 1.3 too, trying to build a sandbox version from an existing sif file (both as a normal user and as root):

root@eespot-hpchb176rsv4-1:/shared/home/grma# apptainer --version
apptainer version 1.3.1

root@eespot-hpchb176rsv4-1:/shared/home/grma# apptainer build --sandbox wps_wrf_4p4p2_py3_latest_azure ./wrf_4p4p2_py3_latest.sif
INFO: Starting build...
INFO: Verifying bootstrap image wrf_4p4p2_py3_latest.sif
ERROR: unpackSIF failed: root filesystem extraction failed: extract command failed: FATAL: container creation failed: mount hook function failure: mount tmpfs->/var/lib/apptainer/mnt/session error: while mounting tmpfs: can't mount tmpfs filesystem to /var/lib/apptainer/mnt/session: read unix @->@: read: connection reset by peer
: exit status 255
FATAL: While performing build: packer failed to pack: root filesystem extraction failed: extract command failed: FATAL: container creation failed: mount hook function failure: mount tmpfs->/var/lib/apptainer/mnt/session error: while mounting tmpfs: can't mount tmpfs filesystem to /var/lib/apptainer/mnt/session: read unix @->@: read: connection reset by peer
: exit status 255

@garymansellricardo
Copy link

I note that this process works on AlmaLinux 8.7 with a slightly newer apptainer-suid package:

[grma@eespot-hpchb120rsv3-1 ~]$ rpm -qa | grep apptainer
apptainer-suid-1.3.1-1.el8.x86_64
apptainer-1.3.1-1.el8.x86_64
[grma@eespot-hpchb120rsv3-1 ~]$

[grma@eespot-hpchb120rsv3-1 ~]$ apptainer build --sandbox wps_wrf_4p4p2_py3_latest_azure ./wrf_4p4p2_py3_latest.sif
INFO: Starting build...
INFO: Verifying bootstrap image wrf_4p4p2_py3_latest.sif
INFO: Creating sandbox directory...
INFO: Build complete: wps_wrf_4p4p2_py3_latest_azure

Is it possible to get this later package distributed to the Ubunta PPA Repo for download and testing?

@DrDaveD
Copy link
Contributor

DrDaveD commented May 29, 2024

@khaiyichin I have not been able to reproduce the problem on my Ubuntu 20.04 VM. What type of filesystem is being used for temporary space? Make sure it is local disk, as many network filesystems don't work properly. I permanently set APPTAINER_TMPDIR to local disk scratch space for builds.

@garymansellricardo What makes you think that the AlmaLinux 8.7 package is slightly newer than the Ubuntu package? Aren't they both 1.3.1? I think your issue is most likely related to the /shared filesystem you're trying to write into, perhaps particularly when running as root. I note that also in the second case you're not running as root. That should be fine, especially if you have /etc/subuid mapping set up, but it is different.

@garymansellricardo
Copy link

garymansellricardo commented May 30, 2024 •
edited
Loading

@DrDaveD - you are probably correct, but please see the two slightly differing apptainer --version outputs for the two OS's below.

I don't think it is anything to do with my /shared nfs filesystem (where my user home dir and bind mount dirs are located) as I am not binding any folders at this point, just extracting the sif container file to a local filesystem. But I will run both tests again as root (who has a local home dir to the node) and copy the sif container to a local scratch disk on the node before extracting which (I think?) will remove this /shared nfs dir from the mix.

See below, I have run the same operation on both Ubuntu 20.04 (which fails) and AlmaLinux 8 (which works). I have included the apptainer version output and mounted filesystem types output too for each node, and am running from /scratch which is a local md (2x nvme disks in a stripe) device with xfs filesystem on both nodes.

Ubuntu 20.04.6 LTS:

root@eespot-hpchb176rsv4-1:~#
root@eespot-hpchb176rsv4-1:~# apptainer --version
apptainer version 1.3.1
root@eespot-hpchb176rsv4-1:~# df -hT
Filesystem             Type      Size  Used Avail Use% Mounted on
/dev/root              ext4       62G   31G   32G  50% /
devtmpfs               devtmpfs  378G     0  378G   0% /dev
tmpfs                  tmpfs     378G     0  378G   0% /dev/shm
tmpfs                  tmpfs      76G  1.8M   76G   1% /run
tmpfs                  tmpfs     5.0M     0  5.0M   0% /run/lock
tmpfs                  tmpfs     378G     0  378G   0% /sys/fs/cgroup
/dev/loop0             squashfs   64M   64M     0 100% /snap/core20/2264
/dev/sda15             vfat      105M  6.1M   99M   6% /boot/efi
/dev/loop1             squashfs   92M   92M     0 100% /snap/lxd/24061
/dev/loop2             squashfs   39M   39M     0 100% /snap/snapd/21465
/dev/sdb1              ext4      472G  212K  448G   1% /mnt
192.168.140.134:/sched nfs4       30G  247M   30G   1% /sched
azu-caesrv-01:/shared  nfs4      4.0T  3.3T  804G  81% /shared
/dev/md0p1             xfs       3.5T   27G  3.5T   1% /scratch
tmpfs                  tmpfs      76G     0   76G   0% /run/user/20003
/dev/loop3             squashfs   39M   39M     0 100% /snap/snapd/21759
/dev/loop4             squashfs   64M   64M     0 100% /snap/core20/2318
root@eespot-hpchb176rsv4-1:~# cd /scratch
root@eespot-hpchb176rsv4-1:/scratch#
root@eespot-hpchb176rsv4-1:/scratch# cp /shared/home/grma/wps_wrf_4p4p2_py3_latest_azure.sif .
root@eespot-hpchb176rsv4-1:/scratch#
root@eespot-hpchb176rsv4-1:/scratch# apptainer build --sandbox wps_wrf_4p4p2_py3_latest_azure ./wps_wrf_4p4p2_py3_latest_azure.sif
INFO:    Starting build...
INFO:    Verifying bootstrap image wps_wrf_4p4p2_py3_latest_azure.sif
ERROR:   unpackSIF failed: root filesystem extraction failed: extract command failed: FATAL:   container creation failed: mount hook function failure: mount tmpfs->/var/lib/apptainer/mnt/session error: while mounting tmpfs: can't mount tmpfs filesystem to /var/lib/apptainer/mnt/session: read unix @->@: read: connection reset by peer
: exit status 255
FATAL:   While performing build: packer failed to pack: root filesystem extraction failed: extract command failed: FATAL:   container creation failed: mount hook function failure: mount tmpfs->/var/lib/apptainer/mnt/session error: while mounting tmpfs: can't mount tmpfs filesystem to /var/lib/apptainer/mnt/session: read unix @->@: read: connection reset by peer
: exit status 255
root@eespot-hpchb176rsv4-1:/scratch#

AlmaLinux 8.7:

[root@eespot-hpchb120rsv3-1 ~]# apptainer --version
apptainer version 1.3.1-1.el8
[root@eespot-hpchb120rsv3-1 ~]# df -hT
Filesystem             Type      Size  Used Avail Use% Mounted on
devtmpfs               devtmpfs  225G     0  225G   0% /dev
tmpfs                  tmpfs     225G     0  225G   0% /dev/shm
tmpfs                  tmpfs     225G   18M  225G   1% /run
tmpfs                  tmpfs     225G     0  225G   0% /sys/fs/cgroup
/dev/sda2              xfs        63G   33G   30G  53% /
/dev/sda1              xfs       994M  209M  786M  21% /boot
/dev/sda15             vfat      495M  5.9M  489M   2% /boot/efi
/dev/sdb1              ext4      472G  212K  448G   1% /mnt
192.168.140.134:/sched nfs4       30G  247M   30G   1% /sched
azu-caesrv-01:/shared  nfs4      4.0T  3.3T  804G  81% /shared
/dev/md0p1             xfs       1.8T   13G  1.8T   1% /scratch
tmpfs                  tmpfs      45G     0   45G   0% /run/user/20003
[root@eespot-hpchb120rsv3-1 ~]#
[root@eespot-hpchb120rsv3-1 ~]# cd /scratch
[root@eespot-hpchb120rsv3-1 scratch]#
[root@eespot-hpchb120rsv3-1 scratch]# cp /shared/home/grma/wps_wrf_4p4p2_py3_latest_azure.sif .
[root@eespot-hpchb120rsv3-1 scratch]# apptainer build --sandbox wps_wrf_4p4p2_py3_latest_azure ./wps_wrf_4p4p2_py3_latest_azure.sif
INFO:    Starting build...
INFO:    Verifying bootstrap image wps_wrf_4p4p2_py3_latest_azure.sif
INFO:    Creating sandbox directory...
INFO:    Build complete: wps_wrf_4p4p2_py3_latest_azure
[root@eespot-hpchb120rsv3-1 scratch]#

@garymansellricardo
Copy link

garymansellricardo commented May 30, 2024 •
edited
Loading

This also works OK on Ubuntu 22.04 - seems to be just Ubuntu 20.04 that is the problem:

root@eespot-hpchb176rsv4-1:~# apptainer --version
apptainer version 1.3.1
root@eespot-hpchb176rsv4-1:~#
root@eespot-hpchb176rsv4-1:~# df -hT
Filesystem             Type   Size  Used Avail Use% Mounted on
/dev/root              ext4    62G   30G   32G  49% /
tmpfs                  tmpfs  378G     0  378G   0% /dev/shm
tmpfs                  tmpfs  152G  1.8M  152G   1% /run
tmpfs                  tmpfs  5.0M     0  5.0M   0% /run/lock
/dev/sda15             vfat   105M  6.1M   99M   6% /boot/efi
/dev/sdb1              ext4   472G  212K  448G   1% /mnt
192.168.140.134:/sched nfs4    30G  247M   30G   1% /sched
azu-caesrv-01:/shared  nfs4   4.0T  3.3T  771G  82% /shared
/dev/md0p1             xfs    3.5T   30G  3.5T   1% /scratch
tmpfs                  tmpfs   76G  4.0K   76G   1% /run/user/20003

root@eespot-hpchb176rsv4-1:~# cd /scratch
root@eespot-hpchb176rsv4-1:/scratch# cp ~grma/wps_wrf_4p4p2_py3_latest_azure.sif .
root@eespot-hpchb176rsv4-1:/scratch# apptainer build --sandbox wps_wrf_4p4p2_py3_latest_azure ./wps_wrf_4p4p2_py3_latest_azure.sif
INFO:    Starting build...
INFO:    Verifying bootstrap image wps_wrf_4p4p2_py3_latest_azure.sif
INFO:    Creating sandbox directory...
INFO:    Build complete: wps_wrf_4p4p2_py3_latest_azure
root@eespot-hpchb176rsv4-1:/scratch#

@garymansellricardo
Copy link

I have gone back to the problem Ubuntu 20.04 VM and have explicitly set the APPTAINER_TMPDIR to a large, local scratch folder and still get the error:

root@eespot-hpchb176rsv4-1:~# cd /scratch/
root@eespot-hpchb176rsv4-1:/scratch# mkdir /scratch/tmp
root@eespot-hpchb176rsv4-1:/scratch# APPTAINER_TMPDIR=/scratch/tmp
root@eespot-hpchb176rsv4-1:/scratch#
root@eespot-hpchb176rsv4-1:/scratch# cp ~grma/wps_wrf_4p4p2_py3_latest_azure.sif .
root@eespot-hpchb176rsv4-1:/scratch# apptainer build --sandbox wps_wrf_4p4p2_py3_latest_azure ./wps_wrf_4p4p2_py3_latest_azure.sif
INFO:    Starting build...
INFO:    Verifying bootstrap image wps_wrf_4p4p2_py3_latest_azure.sif
ERROR:   unpackSIF failed: root filesystem extraction failed: extract command failed: FATAL:   container creation failed: mount hook function failure: mount tmpfs->/var/lib/apptainer/mnt/session error: while mounting tmpfs: can't mount tmpfs filesystem to /var/lib/apptainer/mnt/session: read unix @->@: read: connection reset by peer
: exit status 255
FATAL:   While performing build: packer failed to pack: root filesystem extraction failed: extract command failed: FATAL:   container creation failed: mount hook function failure: mount tmpfs->/var/lib/apptainer/mnt/session error: while mounting tmpfs: can't mount tmpfs filesystem to /var/lib/apptainer/mnt/session: read unix @->@: read: connection reset by peer
: exit status 255
root@eespot-hpchb176rsv4-1:/scratch#


root@eespot-hpchb176rsv4-1:/scratch# df -hT
Filesystem             Type      Size  Used Avail Use% Mounted on
/dev/root              ext4       62G   31G   32G  50% /
devtmpfs               devtmpfs  378G     0  378G   0% /dev
tmpfs                  tmpfs     378G     0  378G   0% /dev/shm
tmpfs                  tmpfs      76G  1.8M   76G   1% /run
tmpfs                  tmpfs     5.0M     0  5.0M   0% /run/lock
tmpfs                  tmpfs     378G     0  378G   0% /sys/fs/cgroup
/dev/loop0             squashfs   64M   64M     0 100% /snap/core20/2264
/dev/sda15             vfat      105M  6.1M   99M   6% /boot/efi
/dev/loop1             squashfs   92M   92M     0 100% /snap/lxd/24061
/dev/loop2             squashfs   39M   39M     0 100% /snap/snapd/21465
/dev/sdb1              ext4      472G  212K  448G   1% /mnt
192.168.140.134:/sched nfs4       30G  247M   30G   1% /sched
azu-caesrv-01:/shared  nfs4      4.0T  3.3T  770G  82% /shared
/dev/md0p1             xfs       3.5T   27G  3.5T   1% /scratch
tmpfs                  tmpfs      76G     0   76G   0% /run/user/20003
/dev/loop3             squashfs   39M   39M     0 100% /snap/snapd/21759
/dev/loop4             squashfs   64M   64M     0 100% /snap/core20/2318
root@eespot-hpchb176rsv4-1:/scratch#

@DrDaveD
Copy link
Contributor

DrDaveD commented May 31, 2024

Thanks for narrowing it down. That extra version information on el8 is just packaging information, not really relevant.

I think it must be something specific about your Ubuntu 20.04 host, because it doesn't happen for me. You don't happen to both be using the same host, or are you?

We haven't seen this exact error before, although we have experienced a case where configuration for other container systems using shared code has caused problems, in #758. That's the only idea I have.

@garymansellricardo
Copy link

I am using the Microsoft Azure Ubuntu 20.04 HPC VM SKU, so it would surprise me if there is something wrong with that image but not the 22.04 image. As I expect they use a similar recipe for building both - they just install additional HPC software packages such as MPI, Nvidia CUDA stuff etc.

@khaiyichin - what Ubuntu 20.04 image are you using?

@garymansellricardo
Copy link

@DrDaveD - I really need to be able to extract containers to a sandbox on Ubuntu 20.04, is there anything I can do to get better logs, or further investigate what the problem could be?

@DrDaveD
Copy link
Contributor

DrDaveD commented Jun 3, 2024

First, make sure it happens with even simple images like the one defined in the description of this PR. Next, if I had root access on the system, I would try using strace to narrow down exactly what it is that apptainer is losing a connection with. That can be very tricky because apptainer ends up calling other processes and strace sometimes gets stuck. Sometimes I have had to replace the starter for example with a script or a binary that invokes strace before invoking the real starter. With a build it ends up re-invoking apptainer with a bunch of parameters and I think it's that apptainer that is experiencing the error, so inserting strace can be even trickier, quite likely requiring compiling a modified version of apptainer. You can see what it is doing with apptainer -d.

If you could give me temporary root access to a system where this can be reproduced I could do the investigation for you.

@khaiyichin
Copy link
Author

I am using the Microsoft Azure Ubuntu 20.04 HPC VM SKU, so it would surprise me if there is something wrong with that image but not the 22.04 image. As I expect they use a similar recipe for building both - they just install additional HPC software packages such as MPI, Nvidia CUDA stuff etc.

@khaiyichin - what Ubuntu 20.04 image are you using?

Sorry for the late reply! I'm not too familiar with a lot of the system-level stuff, but I'm getting this on my Ubuntu 20.04 laptop, so to answer your question @DrDaveD I'm pretty sure the temporary filespace /var/lib/apptainer/mnt/session is on my local filesystem.

@DrDaveD DrDaveD modified the milestones: 1.3.3, 1.3.4 Jul 2, 2024
@DrDaveD
Copy link
Contributor

DrDaveD commented Jul 16, 2024

@garymansellricardo Can you try some of the suggestions I gave?

@garymansellricardo
Copy link

@DrDaveD - Sorry, have been on holida and had some pressing work projects...

I can't seem to re-create the issue now on the (Azure) Microsoft HPC Ubuntu 20.04 marketplace image that I was seeing the issue with before, but I notice that the version now in the repo is 1.3.2 (before 1.3.1) - so perhaps something was done to fix this in the new release?

@khaiyichin - can you perhaps try the 1.3.2 version and confirm it is fixed for you?

@DrDaveD DrDaveD removed this from the 1.3.4 milestone Aug 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@DrDaveD @khaiyichin @garymansellricardo