ERROR : Failed invoking the NEWUSER namespace runtime: Invalid argument ABORT : Retval = 255

[DrrDom]

Version of Singularity:

What version of Singularity are you using? Run:

$ singularity --version
2.4.2-dist

Actual behavior

I modified an existing image (probably created with apptainer) on a local machine using singularity 3.5.3. The image successfully runs on the local machine and one cluster (having apptainer 1.1.5), but it does not run on another cluster with singularity 2.4.2

$ singularity --debug exec ~/singularity_images/1.sif pharmit -h
Enabling debugging
Ending argument loop
Singularity version: 2.4.2-dist
Exec'ing: /apps/software/Singularity/2.4.2-GCC-5.4.0-2.26/libexec/singularity/cli/exec.exec
Evaluating args: '/home/pavlop/singularity_images/1.sif pharmit -h'
VERBOSE [U=1000,P=12382]   message_init()                            Set messagelevel to: 5
VERBOSE [U=1000,P=12382]   singularity_config_parse()                Initialize configuration file: /apps/software/Singularity/2.4.2-GCC-5.4.0-2.26/etc/singularity/singularity.conf
DEBUG   [U=1000,P=12382]   singularity_config_parse()                Starting parse of configuration file /apps/software/Singularity/2.4.2-GCC-5.4.0-2.26/etc/singularity/singularity.conf
VERBOSE [U=1000,P=12382]   singularity_config_parse()                Got config key allow setuid = 'yes'
VERBOSE [U=1000,P=12382]   singularity_config_parse()                Got config key max loop devices = '256'
VERBOSE [U=1000,P=12382]   singularity_config_parse()                Got config key allow pid ns = 'yes'
VERBOSE [U=1000,P=12382]   singularity_config_parse()                Got config key config passwd = 'yes'
VERBOSE [U=1000,P=12382]   singularity_config_parse()                Got config key config group = 'yes'
VERBOSE [U=1000,P=12382]   singularity_config_parse()                Got config key config resolv_conf = 'yes'
VERBOSE [U=1000,P=12382]   singularity_config_parse()                Got config key mount proc = 'yes'
VERBOSE [U=1000,P=12382]   singularity_config_parse()                Got config key mount sys = 'yes'
VERBOSE [U=1000,P=12382]   singularity_config_parse()                Got config key mount dev = 'yes'
VERBOSE [U=1000,P=12382]   singularity_config_parse()                Got config key mount devpts = 'yes'
VERBOSE [U=1000,P=12382]   singularity_config_parse()                Got config key mount home = 'yes'
VERBOSE [U=1000,P=12382]   singularity_config_parse()                Got config key mount tmp = 'yes'
VERBOSE [U=1000,P=12382]   singularity_config_parse()                Got config key mount hostfs = 'no'
VERBOSE [U=1000,P=12382]   singularity_config_parse()                Got config key bind path = '/etc/localtime'
VERBOSE [U=1000,P=12382]   singularity_config_parse()                Got config key bind path = '/etc/hosts'
VERBOSE [U=1000,P=12382]   singularity_config_parse()                Got config key user bind control = 'yes'
VERBOSE [U=1000,P=12382]   singularity_config_parse()                Got config key enable overlay = 'try'
VERBOSE [U=1000,P=12382]   singularity_config_parse()                Got config key mount slave = 'yes'
VERBOSE [U=1000,P=12382]   singularity_config_parse()                Got config key sessiondir max size = '16'
VERBOSE [U=1000,P=12382]   singularity_config_parse()                Got config key allow container squashfs = 'yes'
VERBOSE [U=1000,P=12382]   singularity_config_parse()                Got config key allow container extfs = 'yes'
VERBOSE [U=1000,P=12382]   singularity_config_parse()                Got config key allow container dir = 'yes'
DEBUG   [U=1000,P=12382]   singularity_config_parse()                Finished parsing configuration file '/apps/software/Singularity/2.4.2-GCC-5.4.0-2.26/etc/singularity/singularity.conf'
VERBOSE [U=1000,P=12382]   singularity_registry_init()               Initializing Singularity Registry
VERBOSE [U=1000,P=12382]   singularity_registry_set()                Adding value to registry: 'LIBEXECDIR' = '/apps/software/Singularity/2.4.2-GCC-5.4.0-2.26/libexec'
DEBUG   [U=1000,P=12382]   singularity_registry_set()                Returning singularity_registry_set(libexecdir, /apps/software/Singularity/2.4.2-GCC-5.4.0-2.26/libexec) = 0
VERBOSE [U=1000,P=12382]   singularity_registry_set()                Adding value to registry: 'COMMAND' = 'exec'
DEBUG   [U=1000,P=12382]   singularity_registry_set()                Returning singularity_registry_set(COMMAND, exec) = 0
VERBOSE [U=1000,P=12382]   singularity_registry_set()                Adding value to registry: 'MESSAGELEVEL' = '5'
DEBUG   [U=1000,P=12382]   singularity_registry_set()                Returning singularity_registry_set(MESSAGELEVEL, 5) = 0
VERBOSE [U=1000,P=12382]   singularity_registry_set()                Adding value to registry: 'VERSION' = '2.4.2-dist'
DEBUG   [U=1000,P=12382]   singularity_registry_set()                Returning singularity_registry_set(version, 2.4.2-dist) = 0
VERBOSE [U=1000,P=12382]   singularity_registry_set()                Adding value to registry: 'LOCALSTATEDIR' = '/apps/software/Singularity/2.4.2-GCC-5.4.0-2.26/var'
DEBUG   [U=1000,P=12382]   singularity_registry_set()                Returning singularity_registry_set(localstatedir, /apps/software/Singularity/2.4.2-GCC-5.4.0-2.26/var) = 0
VERBOSE [U=1000,P=12382]   singularity_registry_set()                Adding value to registry: 'SYSCONFDIR' = '/apps/software/Singularity/2.4.2-GCC-5.4.0-2.26/etc'
DEBUG   [U=1000,P=12382]   singularity_registry_set()                Returning singularity_registry_set(sysconfdir, /apps/software/Singularity/2.4.2-GCC-5.4.0-2.26/etc) = 0
VERBOSE [U=1000,P=12382]   singularity_registry_set()                Adding value to registry: 'BINDIR' = '/apps/software/Singularity/2.4.2-GCC-5.4.0-2.26/bin'
DEBUG   [U=1000,P=12382]   singularity_registry_set()                Returning singularity_registry_set(bindir, /apps/software/Singularity/2.4.2-GCC-5.4.0-2.26/bin) = 0
VERBOSE [U=1000,P=12382]   singularity_registry_set()                Adding value to registry: 'IMAGE' = '/home/pavlop/singularity_images/1.sif'
DEBUG   [U=1000,P=12382]   singularity_registry_set()                Returning singularity_registry_set(IMAGE, /home/pavlop/singularity_images/1.sif) = 0
DEBUG   [U=1000,P=12382]   singularity_registry_get()                Returning NULL on 'HOME'
DEBUG   [U=1000,P=12382]   singularity_registry_get()                Returning NULL on 'TARGET_UID'
DEBUG   [U=1000,P=12382]   singularity_registry_get()                Returning NULL on 'TARGET_GID'
DEBUG   [U=1000,P=12382]   singularity_priv_init()                   Initializing user info
DEBUG   [U=1000,P=12382]   singularity_priv_init()                   Set the calling user's username to: pavlop
DEBUG   [U=1000,P=12382]   singularity_priv_init()                   Marking uinfo structure as ready
DEBUG   [U=1000,P=12382]   singularity_priv_init()                   Obtaining home directory
VERBOSE [U=1000,P=12382]   singularity_priv_init()                   Set home (via getpwuid()) to: /home/pavlop
VERBOSE [U=1000,P=12382]   singularity_suid_init()                   Running NON-SUID program workflow
DEBUG   [U=1000,P=12382]   singularity_suid_init()                   Checking program has appropriate permissions
VERBOSE [U=1000,P=12382]   singularity_priv_userns()                 Invoking the user namespace
DEBUG   [U=1000,P=12382]   singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(allow user ns, yes)
DEBUG   [U=1000,P=12382]   singularity_config_get_value_impl()       No configuration entry found for 'allow user ns'; returning default value 'yes'
DEBUG   [U=1000,P=12382]   singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(allow user ns, yes) = 1
DEBUG   [U=1000,P=12382]   singularity_priv_userns()                 Attempting to virtualize the USER namespace
ERROR   [U=1000,P=12382]   singularity_priv_userns()                 Failed invoking the NEWUSER namespace runtime: Invalid argument
ABORT   [U=1000,P=12382]   singularity_priv_userns()                 Retval = 255

1. Is there an issue with installation of singularity? There were several issues and in one of them it was suggested that this is an installation issue - Failed invoking the NEWUSER namespace runtime: Invalid argument #415, ERROR : Failed invoking the NEWUSER namespace runtime: Invalid argument ABORT : Retval = 255s #5479.
   I noticed that it runs in NON-SUID mode.

2. Or is this an incompatibility issue of images created and updated with newer versions?

What OS/distro are you running

$ cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

How did you install Singularity

it was installed on a cluster by an administrator

[github-actions]

New issues are no longer accepted in this repository. If singularity --version says singularity-ce, submit instead to https://github.com/sylabs/singularity, otherwise submit to https://github.com/apptainer/apptainer.