docs/v2-3/faq.html

<!DOCTYPE html>
<head>
    <meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">

<meta name="description" content="   Frequently Asked Questions                                                 General Singularity infoWhy the name “Singularity”?The name “Singularity” for me (Greg) stems back from my past experience working at a company called Linuxcare where the Linux Boot-able Business Card (LNX-BBC) was developed. The BBC, was a Linux rescue disk which paved the way for all live CD bootable distributions using a loop back file system called the “singularity”.This nomenclature represented that all files within the environment were contained within a single file, and for the same reason Singularity emphasizes the same nomenclature. (Thanks LNX-BBC!)Which namespaces are virtualized? Is that select-able?The goal of Singularity is to run an application within a contained environment such as it was not contained. Thus there is a balance between what to separate and what not to separate. At present the virtualized namespaces are process, mount points, and certain parts of the contained file system.When you run your Singularity container, you may find that the process IDs start with 1 (one) and increment from there. You will also find that while the file system is contained starting with ‘/’ (root), you can have access outside the container via your starting path. This means that relative paths will resolve outside the container, and fully qualified paths will resolve inside the container.To achieve this behavior, you will find that several Linux namespaces are separated (PIDS, file systems and descriptors, mounts, and root file system). These can be enabled or disabled by the build and what namespaces the host system supports as well as through environment variables.Can’t you do this with Docker?No, not even close. If that was true, there would be no use for Singularity. But in fact, Singularity has taken off within the scientific computing world! Furthermore, if Docker can be used on traditional HPC resources, it would be; but it is not! There are no HPC centers utilizing Docker on their traditional HPC resources because it is inheritiatnly incompatible with HPC.Singularity has a very different usage model then Docker in that Singularity utilizes complete image files while Docker containers are made up of layers of tar files. With Singularity, you can prefetch an entire container, cache it on shared optimized storage, and run it from there.   Singularity also limits user privileges and access from within the container, making it safe foruser’s to bring their own containers. It doesn’t open up security risks of users being able to control a root owned daemon, and it integrates seemlessly into existing process and resource manager workflows, supports GPU, MPI, architecture independent, among lots of other aspects that Docker does not.In the end, Docker is designed for micro-service network virtulization and emulation of the full isolation requirements in the legacy of full machine level virtualization platforms (e.g. VMWare, Xen, KVM, etc.). Singularity is designed specifically for the scentific, application and environment virtulization. The right tool for the right job. :)If you already have a Docker container you can import/shell/run it directly into Singularity!# Create and importsingularity create hello-world.imgsingularity import hello-world.img docker://hello-world# Shell right insingularity shell docker://hello-world# Or just runsingularity run docker://hello-worldHow is Singularity different from chroot?While Singularity can in fact operate on a chroot directory (thus giving users the ability to securely run within these directories) the feature set of Singularity goes leaps and bounds beyond! For example, chroot can not operate on directories, utilize kernel namespaces, or be used by users (without SUID even when the user namespace is supported). chroot does not support blurring the line between container and host with bind mounts, devices, and does not limit security access for users, or integrate with traditional HPC resources and services.How does Singularity relate/differ from Shifter?NERSC (like most HPC centers) are feeling the pressure from users asking for support for containers, specifically Docker. Due to the architecture of Docker it is very difficult (if not impossible) to properly and securely implement in a multi-tenant HPC environment. Shifter is NERSC’s implementation to provide a Docker compatible front-end interface to their extreme scale HPC resources. It is system/resource specific in that you must import an existing container (from Docker, Singularity, or other), to the host/Shifter implementation.Singularity on the other hand does not leverage the Docker work-flow and targets a different premise - Mobility of Compute. This makes the integration of Singularity non-HPC specific (even though it works very well with HPC) and allows the image to become the primary unit of mobility (you can share and operate directly on Singularity images).Singularity is more of a general purpose mobility of compute solution that is very capable at HPC, Shifter’s primary focus is targeting extreme scale HPC and integration with Cray and the resource manager.How does Singularity relate/differ from FlatpakFlatpak is a packaging subsystem that uses some container technologies to create distribution neutral packages and it is more similar to the initial proof of concept of Singularity. But the use-cases of Singularity dictated that we should support full operating system containers that contain the entire user’s environment.How does Singularity relate/differ from other container systems like OpenVz, LXC/LXD, etc.?Singularity differs from other container systems in several major ways that impact usability on shared systems. For example, most container systems emulate standard systems in that there is the ability and necessity to escalate to root, run on separate IP/network address, run services, and in some cases even virtually boot the container system.Singularity on the other hand focuses on the ability to virtualized only what is necessary to achieve run-time application container and portable environments. For instance, you can not obtain root from within a Singularity container.There are some additional performance and design enhancements which make Singularity also more appropriate in a scheduled HPC environment. The back-end image type is one such feature that negates the need for temporary caching of container images and optimizes meta-data IO (especially on parallel file systems). Another feature is how Singularity interacts with the host operating system to facilitate application work-flows like X11 and MPI.How does Singularity relate/differ from statically compiled binaries?Statically compiled binaries are a good comparison to what Singularity can do for a single program because it will package up all of the dynamic libraries and package them into a single executable (interpreted) format.But because Singularity is actually wrapping operating system files in to a container, you can do much more with it… Such as include other files, scripts, work-flows, pipe lines, data, and multi program processes and package them into a single portable executable format.What Linux distributions are you trying to get on-board?All of them! Help us out by letting them know you want Singularity to be included!Basic Singularity usageDo you need administrator privileges to use Singularity?You generally do not need admin/sudo to use Singularity containers. As of version 2.3, you can create, import, run, export, and shell without it. You do however need admin/root access to install Singularity and to bootstrap a container (build it from a specification file called Singularity). This means that, given that your cluster has Singularity installed, you are empowered to generate your own portable environments.This then defines the work-flow to some extent. If you have a container (whether Singularity or Docker) ready to go, you can run/shell/import without root access. If you want to build and bootstrap, then your Singularity container image must be built and configured on a host where you have root access (this can be a physical system or on a VM or Docker image). And of course once the container image has been configured it can be used on a system where you do not have root access as long as Singularity has been installed there.What if I don’t want to install Singularity on my computer?If you don’t want to build your own images, Singularity Hub will connect to your Github repos with build specification files, and build the containers automatically for you. You can then interact with them easily where Singularity is installed (e.g., on your cluster):singularity shell shub://vanessa/singularity-hello-worldsingularity run shub://vanessa/singularity-hello-worldsingularity create hello-world.img singularity import hello-world.img shub://vanessa/singularity-hello-worldCan you edit/modify a Singularity container once it has been instantiated?Yes, if you call it with the -w/--writable flag. (e.g. ‘singularity shell –writable Container.img’). However, we recommend to not do this as a regular practice, as it induces changes that are not recorded properly. If you need to rebuild the container, you would need to remember those commands.Can multiple applications be packaged into one Singularity Container?Yes! You can even create entire pipe lines and work flows using many applications, binaries, scripts, etc.. The %runscript bootstrap section is where you can define what happens when a Singularity container is run. This definition takes priority over all “other places” that might define a container’s execution. Specifically:  The %runscript section of the bootstrap file takes priority over all  If you bootstrap a docker image, the ENTRYPOINT is used if there is no %runscript  If you add Includecmd: yes to your definition file header, the CMD is used instead  No %runscript, ENTRYPOINT OR CMD means the container’s default execution is /bin/bashHow are external file systems and paths handled in a Singularity Container?Because Singularity is based on container principals, when an application is run from within a Singularity container its default view of the file system is different from how it is on the host system. This is what allows the environment to be portable. This means that root (‘/’) inside the container is different from the host!Singularity automatically tries to resolve directory mounts such that things will just work and be portable with whatever environment you are running on. This means that /tmp and /var/tmp are automatically shared into the container as is /home. Additionally, if you are in a current directory that is not a system directory, Singularity will also try to bind that to your container.There is a caveat in that a directory must already exist within your container to serve as a mount point. If that directory does not exist, Singularity will not create it for you! You must do that. To create custom mounts at runtime, you should use the -B or --bind argument:singularity run --bind /home/vanessa/Desktop:/data container.imgHow does Singularity handle networking?Singularity does no network isolation because it is designed to run like any other application on the system. It has all of the same networking privileges as any program running as that user. Thus, if you run a process in the container directed to localhost, it will show up on your host machine browser as localhost. Take a look at some of our web examples to get a sense for this.Can I import an image from Docker?Yes, you can do this with our import, even without sudo. With import, the Docker layers (the dump of files) plus the environment, and labels are added to the container.$ singularity import container.img docker://ubuntu:latestYou can also do this by way of bootstrapping Docker images, and to do this you would create a definition file, an image, and then bootstrap. First, here is most simplest definition file, “Singularity”:Bootstrap: dockerFrom: ubuntu:latestNow let’s create an image and bootstrap using the Singularity file:$ singularity create ubuntu.img$ singularity bootstrap ubuntu.img SingularityMake sure you are running Singularity &gt; 2.2 to make full use of this feature.Can a Singularity container be multi-threaded?Yes. Singularity imposes no limitations on forks, threads or processes in general.Can a Singularity container be suspended or check-pointed?Yes and maybe respectively. Any Singularity application can be suspended using standard Linux/Unix signals. Check-pointing requires some preloaded libraries to be automatically loaded with the application but because Singularity escapes the hosts library stack, the checkpoint libraries would not be loaded. If however you wanted to make a Singularity container that can be check-pointed, you would need to install the checkpoint libraries into the Singularity container via the specfileAre there any special requirements to use Singularity through a job scheduler?Singularity containers can be run via any job scheduler without any modifications to the scheduler configuration or architecture. This is because Singularity containers are designed to be run like any application on the system, so within your job script just call Singularity as you would any other application!Does Singularity work in multi-tenant HPC cluster environments?Yes! HPC was one of the primary use cases in mind when Singularity was created.Most people that are currently integrating containers on HPC resources do it by creating virtual clusters within the physical host cluster. This precludes the virtual cluster from having access to the host cluster’s high performance fabric, file systems and other investments which make an HPC system high performance.Singularity on the other hand allows one to keep the high performance in High Performance Computing by containerizing applications and supporting a runtime which seamlessly interfaces with the host system and existing environments.Can I run X11 apps through Singularity?Yes. This works exactly as you would expect it to.Can I containerize my MPI application with Singularity and run it properly on an HPC system?Yes! HPC was one of the primary use cases in mind when Singularity was created.While we know for a fact that Singularity can support multiple MPI implementations, we have spent a considerable effort working with Open MPI as well as adding a Singularity module into Open MPI (v2) such that running at extreme scale will be as efficient as possible.note: We have seen no major performance impact from running a job in a Singularity container.Why do we call ‘mpirun’ from outside the container (rather then inside)?With Singularity, the MPI usage model is to call ‘mpirun’ from outside the container, and reference the container from your ‘mpirun’ command. Usage would look like this:$ mpirun -np 20 singularity exec container.img /path/to/contained_mpi_progBy calling ‘mpirun’ outside the container, we solve several very complicated work-flow aspects. For example, if ‘mpirun’ is called from within the container it must have a method for spawning processes on remote nodes. Historically ssh is used for this which means that there must be an sshd running within the container on the remote nodes, and this sshd process must not conflict with the sshd running on that host! It is also possible for the resource manager to launch the job and (in Open MPI’s case) the Orted processes on the remote system, but that then requires resource manager modification and container awareness.In the end, we do not gain anything by calling ‘mpirun’ from within the container except for increasing the complexity levels and possibly loosing out on some added performance benefits (e.g. if a container wasn’t built with the proper OFED as the host).See the Singularity on HPC page for more details.Does Singularity support containers that require GPUs?Yes, Singularity has been tested to run some test and diagnostic code from within a container without modification. There are however potential issues that can come into play when using GPUs, for instance there are version API compatibilities between kernel and user land which will have to be considered.Container portabilityAre Singularity containers kernel dependent?No, never. But sometimes yes.Singularity is using standard container principals and methods so if you are leveraging any kernel version specific or external patches/module functionality (e.g. OFED), then yes there maybe kernel dependencies you will need to consider.Luckily most people that would hit this are people that are using Singularity to inter-operate with an HPC (High Performance Computing) system where there are highly tuned interconnects and file systems you wish to make efficient use of. In this case, See the documentation of MPI with Singularity.There is also some level of glibc forward compatibility that must be taken into consideration for any container system. For example, I can take a Centos-5 container and run it on Centos-7, but I can not take a Centos-7 container and run it on Centos-5.note: If you require kernel dependent features, a container platform is probably not the right solution for you.Can a Singularity container resolve GLIBC version mismatches?Yes. Singularity containers contain their own library stack (including the Glibc version that they require to run).What is the performance trade off when running an application native or through Singularity?So far we have not identified any appreciable regressions of performance (even in parallel applications running across nodes with InfiniBand). There is a small start-up cost to create and tear-down the container, which has been measured to be anywhere from 10 - 20 thousandths of a second.MiscThe following are miscellaneous questions.Are there any special security concerns that Singularity introduces?No and yes.While Singularity containers always run as the user launching them, there are some aspects of the container execution which requires escalation of privileges. This escalation is achieved via a SUID portion of code. Once the container environment has been instantiated, all escalated privileges are dropped completely, before running any programs within the container.Additionally, there are precautions within the container context to mitigate any escalation of privileges. This limits a user’s ability to gain root control once inside the container.You can read more about the Singularity security overview here.TroubleshootingA little bit of help.How to use Singularity with GRSecurity enabled kernelsTo run Singularity on a GRSecurity enabled kernel, you must disable several security features:$ sudo sysctl -w kernel.grsecurity.chroot_caps=0$ sudo sysctl -w kernel.grsecurity.chroot_deny_mount=0$ sudo sysctl -w kernel.grsecurity.chroot_deny_chmod=0$ sudo sysctl -w kernel.grsecurity.chroot_deny_fchdir=0The container isn’t working on a different host!Singularity by default mounts your home directory. While this is great for seamless communication between your host and the container, it can introduce issues if you have software modules installed at $HOME. For example, we had a user run into this issue.Solution 1: Specify the home to mountA first thing to try is to point to some “sanitized home,” which is the purpose of the -H or --home option. For example, here we are creating a home directory under /tmp/homie, and then telling the container to mount it as home:rm -rf /tmp/homie &amp;&amp; mkdir -p /tmp/homie &amp;&amp; \singularity exec -H /tmp/homie analysis.img /bin/bashSolution 2: Specify the executable to useIt may be the issue that there is an executable in your host environment (eg, python) that is being called in preference to the containers. To avoid this, in your runscript (the %runscript section of the bootstrap file) you should specify the path to the executable exactly. This means:%runscript# This specifies the python in the containerexec /usr/bin/python "$@"# This may pick up a different oneexec python "$@"This same idea would be useful if you are issuing the command to the container using exec. Thanks to yarikoptic for the suggestions on this issue.Invalid Argument or Unknown OptionWhen I try mounting my container with the -B or --bind option I receive an unknown option or Invalid argument error.Make sure that you are using the most recent Singularity release to mount your container to the host system, and that the --bind argument is placed after the execution command. An example might look like this:$ singularity run -B $PWD:/data my_container.imgAlso, make sure you are using an up-to-date Singularity to bootstrap your container.  Some features (such as --bind) will not work in earlier versions.Error running Singularity with sudoThis fix solves the following error when Singularity is installed into the default compiled prefix of /usr/local:$ sudo singularity create /tmp/centos.imgsudo: singularity: command not foundThe cause of the problem is that sudo sanitizes the PATH environment variable and does not include /usr/local/bin in the default search path. Considering this program path is by default owned by root, it is reasonable to extend the default sudo PATH to include this directory.To add /usr/local/bin to the default sudo search path, run the program visudo which will edit the sudoers file, and search for the string ‘secure_path’. Once found, append :/usr/local/bin to that line so it looks like this:Defaults    secure_path = /sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin                Previous        Next                 Edit me                                                                                                                                                 Site last generated: Jul 26, 2017                             ">


<meta name="name" content="Frequently Asked Questions">

<meta name="thumbnail" content="http://singularity.lbl.gov/images/logo/logo.svg">


<title>Frequently Asked Questions | Singularity</title>
<link rel="stylesheet" href="assets/css/syntax.css">


<link rel="stylesheet" type="text/css" href="https://maxcdn.bootstrapcdn.com/font-awesome/4.5.0/css/font-awesome.min.css">
<link href="https://fonts.googleapis.com/css?family=Open+Sans" rel="stylesheet">
<!--<link rel="stylesheet" type="text/css" href="css/bootstrap.min.css">-->
<link rel="stylesheet" href="assets/css/modern-business.css">
<link rel="stylesheet" href="assets/css/lavish-bootstrap.css">
<link rel="stylesheet" href="assets/css/customstyles.css">
<link rel="stylesheet" href="assets/css/theme-blue.css">

<link rel="stylesheet" type="text/css" href="assets/css/asciinema-player.css" />

<script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/2.1.4/jquery.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/jquery-cookie/1.4.1/jquery.cookie.min.js"></script>
<script src="assets/js/jquery.navgoco.min.js"></script>


<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.4/js/bootstrap.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/anchor-js/2.0.0/anchor.min.js"></script>
<script src="assets/js/toc.js"></script>
<script src="assets/js/customscripts.js"></script>

<link rel="shortcut icon" href="images/favicon/favicon.ico">

<!-- HTML5 Shim and Respond.js IE8 support of HTML5 elements and media queries -->
<!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
<!--[if lt IE 9]>
<script src="https://oss.maxcdn.com/libs/html5shiv/3.7.0/html5shiv.js"></script>
<script src="https://oss.maxcdn.com/libs/respond.js/1.4.2/respond.min.js"></script>
<![endif]-->

<link rel="alternate" type="application/rss+xml" title="" href="http://localhost:4005feed.xml">

    <script>
        $(document).ready(function() {

            // Initialize navgoco with default options
            $("#mysidebar").navgoco({
                caretHtml: '',
                accordion: true,
                openClass: 'active', // open
                save: false, // leave false or nav highlighting doesn't work right
                cookie: {
                    name: 'navgoco',
                    expires: false,
                    path: '/'
                },
                slide: {
                    duration: 400,
                    easing: 'swing'
                }
            });

            $("#collapseAll").click(function(e) {
                e.preventDefault();
                $("#mysidebar").navgoco('toggle', false);
            });

            $("#expandAll").click(function(e) {
                e.preventDefault();
                $("#mysidebar").navgoco('toggle', true);
            });

        });

    </script>
    <script>
        $(function () {
            $('[data-toggle="tooltip"]').tooltip()
        })
    </script>
    

</head>
<body>
<!-- asciinema player -->
<script src="assets/js/asciinema-player.js"></script>

<!-- Show or hide players on button clicks-->
<script>
$( document ).ready(function() {
    $(".asciinema-button").click(function(){
        var asciinemaVideo = "#asciinema-" + $(this).attr('id');

        if ($(asciinemaVideo).hasClass('hidden')){
            $(asciinemaVideo).removeClass('hidden');
            $(this).text('Hide Tutorial')
        } else {
            $(asciinemaVideo).addClass('hidden');
            $(this).text('Show Tutorial')
        }
        
    });
});
</script>

<!-- Navigation -->
<nav class="navbar navbar-inverse navbar-fixed-top">
    <div class="container topnavlinks">
        <div class="navbar-header">
            <button type="button" class="navbar-toggle" data-toggle="collapse" data-target="#bs-example-navbar-collapse-1">
                <span class="sr-only">Toggle navigation</span>
                <span class="icon-bar"></span>
                <span class="icon-bar"></span>
                <span class="icon-bar"></span>
            </button>
            <a class="navbar-brand" href="index.html">&nbsp;<span class="projectTitle"> Singularity</span></a>
        </div>
        <div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
            <ul class="nav navbar-nav navbar-right">
                <!-- entries without drop-downs appear here -->
                
                
                <li><a href="blog">News</a></li>
                
                
                <!-- entries with drop-downs appear here -->
                <!-- conditional logic to control which topnav appears for the audience defined in the configuration file.-->
                
                
                <li class="dropdown">
                    <a href="#" class="dropdown-toggle" data-toggle="dropdown">Docs<b class="caret"></b></a>
                    <ul class="dropdown-menu">
                        
                        
                        <li><a href="admin-guide">Admin Guide</a></li>
                        
                        
                        <li><a href="user-guide">User Guide</a></li>
                        
                        
                        <li><a href="links">Contributed Content</a></li>
                        
                        
                    </ul>
                </li>
                
                <li class="dropdown">
                    <a href="#" class="dropdown-toggle" data-toggle="dropdown">Quick Links<b class="caret"></b></a>
                    <ul class="dropdown-menu">
                        
                        
                        <li><a href="https://github.com/singularityware/singularity" target="_blank">Github Repo</a></li>
                        
                        
                        <li><a href="https://groups.google.com/a/lbl.gov/forum/#!forum/singularity" target="_blank">Google Group</a></li>
                        
                        
                        <li><a href="http://stackoverflow.com/questions/tagged/singularity" target="_blank">Singularity on Stack Overflow</a></li>
                        
                        
                        <li><a href="https://singularity-hub.org/faq" target="_blank">Singularity Hub</a></li>
                        
                        
                        <li><a href="https://singularity-container.slack.com" target="_blank">Slack</a></li>
                        
                        
                        <li><a href="faq#troubleshooting">Troubleshooting</a></li>
                        
                        
                    </ul>
                </li>
                
                <li class="dropdown">
                    <a href="#" class="dropdown-toggle" data-toggle="dropdown">People<b class="caret"></b></a>
                    <ul class="dropdown-menu">
                        
                        
                        <li><a href="https://github.com/gmkurtzer" target="_blank">Gregory M. Kurtzer</a></li>
                        
                        
                        <li><a href="https://github.com/vsoch" target="_blank">Vanessa Sochat</a></li>
                        
                        
                        <li><a href="https://github.com/bauerm97" target="_blank">Michael Bauer</a></li>
                        
                        
                        <li><a href="https://github.com/bbockelm" target="_blank">Brian Bockelman</a></li>
                        
                        
                        <li><a href="https://github.com/singularityware/singularity/blob/master/AUTHORS" target="_blank">Complete Authors List</a></li>
                        
                        
                    </ul>
                </li>
                
                
                <li><a href="/search"><i class="fa fa-search"></i></li>
                <!-- jekyll search hidden in favor of google
                <li>
                    <div id="search-demo-container">
                        <input type="text" id="search-input" placeholder="search...">
                        <ul id="results-container"></ul>
                    </div>
                    <script src="assets/js/jekyll-search.js" type="text/javascript"></script>
                    <script type="text/javascript">
                            SimpleJekyllSearch.init({
                                searchInput: document.getElementById('search-input'),
                                resultsContainer: document.getElementById('results-container'),
                                dataSource: 'search.json',
                                searchResultTemplate: '<li><a href="{url}" title="Frequently Asked Questions">{title}</a></li>',
                    noResultsText: 'No results found.',
                            limit: 10,
                            fuzzy: true,
                    })
                    </script>
                    end search-->
                </li>
            </ul>
        </div>
        </div>
        <!-- /.container -->
</nav>

<!-- Page Content -->
<div class="container">
    <div class="col-lg-12">&nbsp;</div>
    <!-- Content Row -->
    <div class="row">
        <!-- Sidebar Column -->
        <div class="col-md-3">
          

<div class="shiny"><a href="\"><figure><img src="/images/logo/logo.svg" class="sidebar-logo"/></figure></a></div>


<ul id="mysidebar" class="nav">
    <li class="sidebarTitle">Singularity</li>
    
    
    <li>
        <a href="#">Information</a>
        <ul>
            
            
            <li><a href="blog">News</a></li>
            
            
            <li><a href="about">About Singularity</a></li>
            
            
            <li><a href="presentations">Presentations</a></li>
            
            
            <li><a href="citation-registration">Registry: Citations and Installations</a></li>
            
            
        </ul>
        
        
    <li>
        <a href="#">Download / Installation</a>
        <ul>
            
            
            <li><a href="all-releases">All Releases</a></li>
            
            
            <li><a href="install-linux">Install Singularity on Linux</a></li>
            
            
            <li><a href="install-mac">Install Singularity on Mac</a></li>
            
            
            <li><a href="install-windows">Install Singularity on Windows</a></li>
            
            
        </ul>
        
        
    <li>
        <a href="#">Contributing</a>
        <ul>
            
            
            <li><a href="contributing-support">User Support</a></li>
            
            
            <li><a href="contributing-code">Code Development</a></li>
            
            
            <li><a href="contributing-docs">Writing Documentation</a></li>
            
            
        </ul>
        
        
    <li>
        <a href="#">Getting Help</a>
        <ul>
            
            
            <li class="active"><a href="faq">FAQ</a></li>
            
            
            <li><a href="support">Support</a></li>
            
            
            <li><a href="faq#troubleshooting">Troubleshooting</a></li>
            
            
        </ul>
        
        
    <li>
        <a href="#">Documentation</a>
        <ul>
            
            
            <li><a href="quickstart">Quick Start</a></li>
            
            
            <li><a href="tutorials">Recipes and Tutorials</a></li>
            
            
            <li><a href="links">Contributed Content Links</a></li>
            
            
            <li><a href="user-guide">User Documentation</a></li>
            
            
            <li><a href="admin-guide">Admin Documentation</a></li>
            
            
        </ul>
        
        
        <!-- if you aren't using the accordion, uncomment this block:
           <p class="external">
               <a href="#" id="collapseAll">Collapse All</a> | <a href="#" id="expandAll">Expand All</a>
           </p>
           -->
    </li>
</ul>
</div>

<!-- this highlights the active parent class in the navgoco sidebar. this is critical so that the parent expands when you're viewing a page. This must appear below the sidebar code above. Otherwise, if placed inside customscripts.js, the script runs before the sidebar code runs and the class never gets inserted.-->
<script>$("li.active").parents('li').toggleClass("active");</script>

    <!-- Content Column -->
    <div class="col-md-9">
        <div class="post-header">
   <h1 class="post-title-main">Frequently Asked Questions</h1>
</div>


<div class="post-content">

   
    <!-- Previous and next buttons-->
    <div class="row" style="padding-top:30px; margin-bottom:10px"><div class="col-md-12">
        <a href="#"><button style="float:left" class="hidden previous-button btn btn-circle btn-default"><i class="fa-2x fa fa-angle-double-left"></i></button></a>
        <a href="#"><button style="float:right" class="hidden next-button btn btn-circle btn-default"><i class="fa fa-angle-double-right fa-2x"></i></button></a>
    </div></div>

    <script>
    $(document).ready(function(){

        var next = $("li.active").next().last().find('a').attr('href');
        var previous = $("li.active").prev().last().find('a').attr('href');

        // NEXT BUTTON
        if (typeof next == 'undefined'){
            console.log("disabling next button")
            $(".next-button").addClass("hidden")
        } else if (next == "#") {
            next = $("li.active").next().find("li").first().find('a').attr('href');
            $(".next-button").closest('a').attr('href', next)
            $(".next-button").removeClass('hidden')
        } else {
            $(".next-button").closest('a').attr('href', next)
            $(".next-button").removeClass('hidden')
        }

        // PREVIOUS BUTTON
        if (typeof previous == 'undefined'){
            console.log("disabling previous button")
            $(".previous-button").addClass("hidden")
        } else if (previous == "#") {
            previous = $("li.active").prev().find("li").last().find('a').attr('href')
            $(".previous-button").closest('a').attr('href', previous)
            $(".previous-button").removeClass('hidden')
        } else {
            $(".previous-button").closest('a').attr('href', previous)
            $(".previous-button").removeClass('hidden')
        }

    })
    </script>

    
<!-- this handles the automatic toc. use ## for subheads to auto-generate the on-page minitoc. if you use html tags, you must supply an ID for the heading element in order for it to appear in the minitoc. -->
<script>
$( document ).ready(function() {
  // Handler for .ready() called.

$('#toc').toc({ minimumHeaders: 0, listType: 'ul', showSpeed: 0, headers: 'h2,h3,h4' });

/* this offset helps account for the space taken up by the floating toolbar. */
$('#toc').on('click', 'a', function() {
  var target = $(this.getAttribute('href'))
    , scroll_target = target.offset().top

  $(window).scrollTop(scroll_target - 10);
  return false
})
  
});
</script>

<div id="toc"></div>

    
  <h2 id="general-singularity-info">General Singularity info</h2>

<h3 id="why-the-name-singularity">Why the name “Singularity”?</h3>

<p>The name “Singularity” for me (Greg) stems back from my past experience working at a company called <a href="https://en.wikipedia.org/wiki/Linuxcare" target="_blank">Linuxcare</a> where the Linux Boot-able Business Card (LNX-BBC) was developed. The BBC, was a Linux rescue disk which paved the way for all live CD bootable distributions using a loop back file system called the “singularity”.</p>

<p>This nomenclature represented that all files within the environment were contained within a single file, and for the same reason Singularity emphasizes the same nomenclature. (Thanks LNX-BBC!)</p>

<h3 id="which-namespaces-are-virtualized-is-that-select-able">Which namespaces are virtualized? Is that select-able?</h3>

<p>The goal of Singularity is to run an application within a contained environment such as it was not contained. Thus there is a balance between what to separate and what not to separate. At present the virtualized namespaces are process, mount points, and certain parts of the contained file system.</p>

<p>When you run your Singularity container, you may find that the process IDs start with 1 (one) and increment from there. You will also find that while the file system is contained starting with ‘/’ (root), you can have access outside the container via your starting path. This means that relative paths will resolve outside the container, and fully qualified paths will resolve inside the container.</p>

<p>To achieve this behavior, you will find that several Linux namespaces are separated (PIDS, file systems and descriptors, mounts, and root file system). These can be enabled or disabled by the build and what namespaces the host system supports as well as through environment variables.</p>

<h3 id="cant-you-do-this-with-docker">Can’t you do this with Docker?</h3>

<p>No, not even close. If that was true, there would be no use for Singularity. But in fact, Singularity has taken off within the scientific computing world! Furthermore, if Docker can be used on traditional HPC resources, it would be; but it is not! There are no HPC centers utilizing Docker on their traditional HPC resources because it is inheritiatnly incompatible with HPC.</p>

<p>Singularity has a very different usage model then Docker in that Singularity utilizes complete image files while Docker containers are made up of layers of tar files. With Singularity, you can prefetch an entire container, cache it on shared optimized storage, and run it from there.   Singularity also limits user privileges and access from within the container, making it safe for
user’s to bring their own containers. It doesn’t open up security risks of users being able to control a root owned daemon, and it integrates seemlessly into existing process and resource manager workflows, supports GPU, MPI, architecture independent, among lots of other aspects that Docker does not.</p>

<p>In the end, Docker is designed for micro-service network virtulization and emulation of the full isolation requirements in the legacy of full machine level virtualization platforms (e.g. VMWare, Xen, KVM, etc.). Singularity is designed specifically for the scentific, application and environment virtulization. The right tool for the right job. :)</p>

<p>If you already have a Docker container you can import/shell/run it directly into Singularity!</p>

<div class="language-bash highlighter-rouge"><pre class="highlight"><code><span class="c"># Create and import</span>
singularity create hello-world.img
singularity import hello-world.img docker://hello-world

<span class="c"># Shell right in</span>
singularity shell docker://hello-world

<span class="c"># Or just run</span>
singularity run docker://hello-world
</code></pre>
</div>

<h2 id="how-is-singularity-different-from-chroot">How is Singularity different from chroot?</h2>

<p>While Singularity can in fact operate on a chroot directory (thus giving users the ability to securely run within these directories) the feature set of Singularity goes leaps and bounds beyond! For example, chroot can not operate on directories, utilize kernel namespaces, or be used by users (without SUID even when the user namespace is supported). chroot does not support blurring the line between container and host with bind mounts, devices, and does not limit security access for users, or integrate with traditional HPC resources and services.</p>

<h3 id="how-does-singularity-relatediffer-from-shifter">How does Singularity relate/differ from Shifter?</h3>

<p>NERSC (like most HPC centers) are feeling the pressure from users asking for support for containers, specifically Docker. Due to the architecture of Docker it is very difficult (if not impossible) to properly and securely implement in a multi-tenant HPC environment. Shifter is NERSC’s implementation to provide a Docker compatible front-end interface to their extreme scale HPC resources. It is system/resource specific in that you must import an existing container (from Docker, Singularity, or other), to the host/Shifter implementation.</p>

<p>Singularity on the other hand does not leverage the Docker work-flow and targets a different premise - Mobility of Compute. This makes the integration of Singularity non-HPC specific (even though it works very well with HPC) and allows the image to become the primary unit of mobility (you can share and operate directly on Singularity images).</p>

<p>Singularity is more of a general purpose mobility of compute solution that is very capable at HPC, Shifter’s primary focus is targeting extreme scale HPC and integration with Cray and the resource manager.</p>

<h3 id="how-does-singularity-relatediffer-from-flatpak">How does Singularity relate/differ from Flatpak</h3>

<p>Flatpak is a packaging subsystem that uses some container technologies to create distribution neutral packages and it is more similar to the initial proof of concept of Singularity. But the use-cases of Singularity dictated that we should support full operating system containers that contain the entire user’s environment.</p>

<h3 id="how-does-singularity-relatediffer-from-other-container-systems-like-openvz-lxclxd-etc">How does Singularity relate/differ from other container systems like OpenVz, LXC/LXD, etc.?</h3>

<p>Singularity differs from other container systems in several major ways that impact usability on shared systems. For example, most container systems emulate standard systems in that there is the ability and necessity to escalate to root, run on separate IP/network address, run services, and in some cases even virtually boot the container system.</p>

<p>Singularity on the other hand focuses on the ability to virtualized only what is necessary to achieve run-time application container and portable environments. For instance, you can not obtain root from within a Singularity container.</p>

<p>There are some additional performance and design enhancements which make Singularity also more appropriate in a scheduled HPC environment. The back-end image type is one such feature that negates the need for temporary caching of container images and optimizes meta-data IO (especially on parallel file systems). Another feature is how Singularity interacts with the host operating system to facilitate application work-flows like X11 and MPI.</p>

<h3 id="how-does-singularity-relatediffer-from-statically-compiled-binaries">How does Singularity relate/differ from statically compiled binaries?</h3>

<p>Statically compiled binaries are a good comparison to what Singularity can do for a single program because it will package up all of the dynamic libraries and package them into a single executable (interpreted) format.</p>

<p>But because Singularity is actually wrapping operating system files in to a container, you can do much more with it… Such as include other files, scripts, work-flows, pipe lines, data, and multi program processes and package them into a single portable executable format.</p>

<h3 id="what-linux-distributions-are-you-trying-to-get-on-board">What Linux distributions are you trying to get on-board?</h3>

<p>All of them! Help us out by letting them know you want Singularity to be included!</p>

<h2 id="basic-singularity-usage">Basic Singularity usage</h2>

<h3 id="do-you-need-administrator-privileges-to-use-singularity">Do you need administrator privileges to use Singularity?</h3>

<p>You generally do not need admin/sudo to use Singularity containers. As of version 2.3, you can create, import, run, export, and shell without it. You do however need admin/root access to install Singularity and to bootstrap a container (build it from a specification file called <code class="highlighter-rouge">Singularity</code>). This means that, given that your cluster has Singularity installed, you are empowered to generate your own portable environments.</p>

<p>This then defines the work-flow to some extent. If you have a container (whether Singularity or Docker) ready to go, you can run/shell/import without root access. If you want to build and bootstrap, then your Singularity container image must be built and configured on a host where you have root access (this can be a physical system or on a VM or Docker image). And of course once the container image has been configured it can be used on a system where you do not have root access as long as Singularity has been installed there.</p>

<h3 id="what-if-i-dont-want-to-install-singularity-on-my-computer">What if I don’t want to install Singularity on my computer?</h3>

<p>If you don’t want to build your own images, <a href="https://singularity-hub.org" target="_blank">Singularity Hub</a> will connect to your Github repos with build specification files, and build the containers automatically for you. You can then interact with them easily where Singularity is installed (e.g., on your cluster):</p>

<div class="language-bash highlighter-rouge"><pre class="highlight"><code>singularity shell shub://vanessa/singularity-hello-world
singularity run shub://vanessa/singularity-hello-world
singularity create hello-world.img 
singularity import hello-world.img shub://vanessa/singularity-hello-world
</code></pre>
</div>

<h3 id="can-you-editmodify-a-singularity-container-once-it-has-been-instantiated">Can you edit/modify a Singularity container once it has been instantiated?</h3>

<p>Yes, if you call it with the <code class="highlighter-rouge">-w/--writable</code> flag. (e.g. ‘singularity shell –writable Container.img’). However, we recommend to not do this as a regular practice, as it induces changes that are not recorded properly. If you need to rebuild the container, you would need to remember those commands.</p>

<h3 id="can-multiple-applications-be-packaged-into-one-singularity-container">Can multiple applications be packaged into one Singularity Container?</h3>

<p>Yes! You can even create entire pipe lines and work flows using many applications, binaries, scripts, etc.. The <code class="highlighter-rouge">%runscript</code> bootstrap section is where you can define what happens when a Singularity container is run. This definition takes priority over all “other places” that might define a container’s execution. Specifically:</p>

<ul>
  <li>The <code class="highlighter-rouge">%runscript</code> section of the bootstrap file takes priority over all</li>
  <li>If you bootstrap a docker image, the <code class="highlighter-rouge">ENTRYPOINT</code> is used if there is no <code class="highlighter-rouge">%runscript</code></li>
  <li>If you add <code class="highlighter-rouge">Includecmd: yes</code> to your definition file header, the <code class="highlighter-rouge">CMD</code> is used instead</li>
  <li>No <code class="highlighter-rouge">%runscript</code>, <code class="highlighter-rouge">ENTRYPOINT</code> OR <code class="highlighter-rouge">CMD</code> means the container’s default execution is <code class="highlighter-rouge">/bin/bash</code></li>
</ul>

<h3 id="how-are-external-file-systems-and-paths-handled-in-a-singularity-container">How are external file systems and paths handled in a Singularity Container?</h3>

<p>Because Singularity is based on container principals, when an application is run from within a Singularity container its default view of the file system is different from how it is on the host system. This is what allows the environment to be portable. This means that root (‘/’) inside the container is different from the host!</p>

<p>Singularity automatically tries to resolve directory mounts such that things will just work and be portable with whatever environment you are running on. This means that <code class="highlighter-rouge">/tmp</code> and <code class="highlighter-rouge">/var/tmp</code> are automatically shared into the container as is <code class="highlighter-rouge">/home</code>. Additionally, if you are in a current directory that is not a system directory, Singularity will also try to bind that to your container.</p>

<p>There is a caveat in that a directory <em>must</em> already exist within your container to serve as a mount point. If that directory does not exist, Singularity will not create it for you! You must do that. To create custom mounts at runtime, you should use the <code class="highlighter-rouge">-B</code> or <code class="highlighter-rouge">--bind</code> argument:</p>

<div class="language-bash highlighter-rouge"><pre class="highlight"><code>singularity run --bind /home/vanessa/Desktop:/data container.img
</code></pre>
</div>

<h3 id="how-does-singularity-handle-networking">How does Singularity handle networking?</h3>

<p>Singularity does no network isolation because it is designed to run like any other application on the system. It has all of the same networking privileges as any program running as that user. Thus, if you run a process in the container directed to localhost, it will show up on your host machine browser as localhost. Take a look at some of our <a href="https://github.com/vsoch/singularity-web" target="_blank">web examples</a> to get a sense for this.</p>

<h3 id="can-i-import-an-image-from-docker">Can I import an image from Docker?</h3>

<p>Yes, you can do this with our import, even without sudo. With import, the Docker layers (the dump of files) plus the environment, and labels are added to the container.</p>

<div class="language-bash highlighter-rouge"><pre class="highlight"><code><span class="gp">$ </span>singularity import container.img docker://ubuntu:latest
</code></pre>
</div>

<p>You can also do this by way of bootstrapping Docker images, and to do this you would create a definition file, an image, and then bootstrap. First, here is most simplest definition file, “Singularity”:</p>

<div class="language-bash highlighter-rouge"><pre class="highlight"><code>Bootstrap: docker
From: ubuntu:latest
</code></pre>
</div>

<p>Now let’s create an image and bootstrap using the Singularity file:</p>

<div class="language-bash highlighter-rouge"><pre class="highlight"><code><span class="gp">$ </span>singularity create ubuntu.img
<span class="gp">$ </span>singularity bootstrap ubuntu.img Singularity
</code></pre>
</div>

<p>Make sure you are running Singularity &gt; 2.2 to make full use of this feature.</p>

<h3 id="can-a-singularity-container-be-multi-threaded">Can a Singularity container be multi-threaded?</h3>

<p>Yes. Singularity imposes no limitations on forks, threads or processes in general.</p>

<h3 id="can-a-singularity-container-be-suspended-or-check-pointed">Can a Singularity container be suspended or check-pointed?</h3>

<p>Yes and maybe respectively. Any Singularity application can be suspended using standard Linux/Unix signals. Check-pointing requires some preloaded libraries to be automatically loaded with the application but because Singularity escapes the hosts library stack, the checkpoint libraries would not be loaded. If however you wanted to make a Singularity container that can be check-pointed, you would need to install the checkpoint libraries into the Singularity container via the specfile</p>

<h3 id="are-there-any-special-requirements-to-use-singularity-through-a-job-scheduler">Are there any special requirements to use Singularity through a job scheduler?</h3>

<p>Singularity containers can be run via any job scheduler without any modifications to the scheduler configuration or architecture. This is because Singularity containers are designed to be run like any application on the system, so within your job script just call Singularity as you would any other application!</p>

<h3 id="does-singularity-work-in-multi-tenant-hpc-cluster-environments">Does Singularity work in multi-tenant HPC cluster environments?</h3>

<p>Yes! HPC was one of the primary use cases in mind when Singularity was created.</p>

<p>Most people that are currently integrating containers on HPC resources do it by creating virtual clusters within the physical host cluster. This precludes the virtual cluster from having access to the host cluster’s high performance fabric, file systems and other investments which make an HPC system high performance.</p>

<p>Singularity on the other hand allows one to keep the high performance in High Performance Computing by containerizing applications and supporting a runtime which seamlessly interfaces with the host system and existing environments.</p>

<h3 id="can-i-run-x11-apps-through-singularity">Can I run X11 apps through Singularity?</h3>

<p>Yes. This works exactly as you would expect it to.</p>

<h3 id="can-i-containerize-my-mpi-application-with-singularity-and-run-it-properly-on-an-hpc-system">Can I containerize my MPI application with Singularity and run it properly on an HPC system?</h3>

<p>Yes! HPC was one of the primary use cases in mind when Singularity was created.</p>

<p>While we know for a fact that Singularity can support multiple MPI implementations, we have spent a considerable effort working with Open MPI as well as adding a Singularity module into Open MPI (v2) such that running at extreme scale will be as efficient as possible.</p>

<p>note: We have seen no major performance impact from running a job in a Singularity container.</p>

<h3 id="why-do-we-call-mpirun-from-outside-the-container-rather-then-inside">Why do we call ‘mpirun’ from outside the container (rather then inside)?</h3>

<p>With Singularity, the MPI usage model is to call ‘mpirun’ from outside the container, and reference the container from your ‘mpirun’ command. Usage would look like this:</p>

<div class="language-bash highlighter-rouge"><pre class="highlight"><code><span class="gp">$ </span>mpirun -np 20 singularity <span class="nb">exec </span>container.img /path/to/contained_mpi_prog
</code></pre>
</div>

<p>By calling ‘mpirun’ outside the container, we solve several very complicated work-flow aspects. For example, if ‘mpirun’ is called from within the container it must have a method for spawning processes on remote nodes. Historically ssh is used for this which means that there must be an sshd running within the container on the remote nodes, and this sshd process must not conflict with the sshd running on that host! It is also possible for the resource manager to launch the job and (in Open MPI’s case) the Orted processes on the remote system, but that then requires resource manager modification and container awareness.</p>

<p>In the end, we do not gain anything by calling ‘mpirun’ from within the container except for increasing the complexity levels and possibly loosing out on some added performance benefits (e.g. if a container wasn’t built with the proper OFED as the host).</p>

<p>See the Singularity on HPC page for more details.</p>

<h3 id="does-singularity-support-containers-that-require-gpus">Does Singularity support containers that require GPUs?</h3>

<p>Yes, Singularity has been tested to run some test and diagnostic code from within a container without modification. There are however potential issues that can come into play when using GPUs, for instance there are version API compatibilities between kernel and user land which will have to be considered.</p>

<h2 id="container-portability">Container portability</h2>

<h3 id="are-singularity-containers-kernel-dependent">Are Singularity containers kernel dependent?</h3>

<p>No, never. But sometimes yes.</p>

<p>Singularity is using standard container principals and methods so if you are leveraging any kernel version specific or external patches/module functionality (e.g. OFED), then yes there maybe kernel dependencies you will need to consider.</p>

<p>Luckily most people that would hit this are people that are using Singularity to inter-operate with an HPC (High Performance Computing) system where there are highly tuned interconnects and file systems you wish to make efficient use of. In this case, See the documentation of MPI with Singularity.</p>

<p>There is also some level of glibc forward compatibility that must be taken into consideration for any container system. For example, I can take a Centos-5 container and run it on Centos-7, but I can not take a Centos-7 container and run it on Centos-5.</p>

<p>note: If you require kernel dependent features, a container platform is probably not the right solution for you.</p>

<h3 id="can-a-singularity-container-resolve-glibc-version-mismatches">Can a Singularity container resolve GLIBC version mismatches?</h3>

<p>Yes. Singularity containers contain their own library stack (including the Glibc version that they require to run).</p>

<h3 id="what-is-the-performance-trade-off-when-running-an-application-native-or-through-singularity">What is the performance trade off when running an application native or through Singularity?</h3>

<p>So far we have not identified any appreciable regressions of performance (even in parallel applications running across nodes with InfiniBand). There is a small start-up cost to create and tear-down the container, which has been measured to be anywhere from 10 - 20 thousandths of a second.</p>

<h2 id="misc">Misc</h2>

<p>The following are miscellaneous questions.</p>

<h3 id="are-there-any-special-security-concerns-that-singularity-introduces">Are there any special security concerns that Singularity introduces?</h3>

<p>No and yes.</p>

<p>While Singularity containers always run as the user launching them, there are some aspects of the container execution which requires escalation of privileges. This escalation is achieved via a SUID portion of code. Once the container environment has been instantiated, all escalated privileges are dropped completely, before running any programs within the container.</p>

<p>Additionally, there are precautions within the container context to mitigate any escalation of privileges. This limits a user’s ability to gain root control once inside the container.</p>

<p>You can read more about the Singularity <a href="/docs-security">security overview here</a>.</p>

<h2 id="troubleshooting">Troubleshooting</h2>

<p>A little bit of help.</p>

<h3 id="how-to-use-singularity-with-grsecurity-enabled-kernels">How to use Singularity with GRSecurity enabled kernels</h3>

<p>To run Singularity on a GRSecurity enabled kernel, you must disable several security features:</p>

<div class="language-bash highlighter-rouge"><pre class="highlight"><code><span class="gp">$ </span>sudo sysctl -w kernel.grsecurity.chroot_caps<span class="o">=</span>0
<span class="gp">$ </span>sudo sysctl -w kernel.grsecurity.chroot_deny_mount<span class="o">=</span>0
<span class="gp">$ </span>sudo sysctl -w kernel.grsecurity.chroot_deny_chmod<span class="o">=</span>0
<span class="gp">$ </span>sudo sysctl -w kernel.grsecurity.chroot_deny_fchdir<span class="o">=</span>0
</code></pre>
</div>

<h3 id="the-container-isnt-working-on-a-different-host">The container isn’t working on a different host!</h3>
<p>Singularity by default mounts your home directory. While this is great for seamless communication between your host and the container, it can introduce issues if you have software modules installed at <code class="highlighter-rouge">$HOME</code>. For example, we had a user <a href="https://github.com/singularityware/singularity/issues/476" target="_blank">run into this issue</a>.</p>

<h4 id="solution-1-specify-the-home-to-mount">Solution 1: Specify the home to mount</h4>
<p>A first thing to try is to point to some “sanitized home,” which is the purpose of the <code class="highlighter-rouge">-H</code> or <code class="highlighter-rouge">--home</code> option. For example, here we are creating a home directory under <code class="highlighter-rouge">/tmp/homie</code>, and then telling the container to mount it as home:</p>

<div class="language-bash highlighter-rouge"><pre class="highlight"><code>rm -rf /tmp/homie <span class="o">&amp;&amp;</span> mkdir -p /tmp/homie <span class="o">&amp;&amp;</span> <span class="se">\</span>
singularity <span class="nb">exec</span> -H /tmp/homie analysis.img /bin/bash
</code></pre>
</div>

<h4 id="solution-2-specify-the-executable-to-use">Solution 2: Specify the executable to use</h4>
<p>It may be the issue that there is an executable in your host environment (eg, python) that is being called in preference to the containers. To avoid this, in your runscript (the <code class="highlighter-rouge">%runscript</code> section of the bootstrap file) you should specify the path to the executable exactly. This means:</p>

<div class="language-bash highlighter-rouge"><pre class="highlight"><code>%runscript

<span class="c"># This specifies the python in the container</span>
<span class="nb">exec</span> /usr/bin/python <span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span>

<span class="c"># This may pick up a different one</span>
<span class="nb">exec </span>python <span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span>
</code></pre>
</div>

<p>This same idea would be useful if you are issuing the command to the container using <code class="highlighter-rouge">exec</code>. Thanks to <a href="https://github.com/yarikoptic" target="_blank">yarikoptic</a> for the suggestions on this issue.</p>

<h3 id="invalid-argument-or-unknown-option">Invalid Argument or Unknown Option</h3>
<p>When I try mounting my container with the <code class="highlighter-rouge">-B</code> or <code class="highlighter-rouge">--bind</code> option I receive an <i>unknown option</i> or <i>Invalid argument</i> error.</p>

<p>Make sure that you are using the most recent Singularity release to mount your container to the host system, and that the <code class="highlighter-rouge">--bind</code> argument is placed after the execution command. An example might look like this:</p>

<div class="language-bash highlighter-rouge"><pre class="highlight"><code><span class="gp">$ </span>singularity run -B <span class="nv">$PWD</span>:/data my_container.img
</code></pre>
</div>

<p>Also, make sure you are using an up-to-date Singularity to bootstrap your container.  Some features (such as <code class="highlighter-rouge">--bind</code>) will not work in earlier versions.</p>

<h3 id="error-running-singularity-with-sudo">Error running Singularity with sudo</h3>

<p>This fix solves the following error when Singularity is installed into the default compiled prefix of /usr/local:</p>

<div class="language-bash highlighter-rouge"><pre class="highlight"><code><span class="gp">$ </span>sudo singularity create /tmp/centos.img
sudo: singularity: <span class="nb">command </span>not found
</code></pre>
</div>

<p>The cause of the problem is that <code class="highlighter-rouge">sudo</code> sanitizes the PATH environment variable and does not include /usr/local/bin in the default search path. Considering this program path is by default owned by root, it is reasonable to extend the default sudo PATH to include this directory.</p>

<p>To add /usr/local/bin to the default sudo search path, run the program visudo which will edit the sudoers file, and search for the string ‘secure_path’. Once found, append :/usr/local/bin to that line so it looks like this:</p>

<div class="language-bash highlighter-rouge"><pre class="highlight"><code>Defaults    secure_path <span class="o">=</span> /sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin
</code></pre>
</div>


    <!-- More navigation on the bottom -->
    <div class="row" style="padding-top:30px; margin-bottom:10px"><div class="col-md-12">
        <a href="#"><button style="width:20%; height: 70px; float:left" class="hidden previous-button btn btn-lg btn-default">Previous</button></a>
        <a href="#"><button style="width:20%; height: 70px; float:right" class="hidden next-button btn btn-lg btn-default">Next</button></a>
    </div></div>


    <a style="margin-top:10px;margin-bottom:10px" target="_blank" href="https://github.com/singularityware/singularityware.github.io/blob/master/pages/_pages/docs/faq.md" class="btn btn-default btn-xs githubEditButton" role="button"><i class="fa fa-github fa-lg"></i> Edit me</a>
    

    <div class="tags">
        
    </div>

    
</div>

<hr class="shaded"/>

<footer>
            <div class="row">

               <!-- Social Media links, etc -->
               <div class="col-lg-6 footer">
               
    <a class="no-after social-icon" href="https://twitter.com/SingularityApp">
      <i class="fa fa-4x fa-twitter no-after"></i>
    </a>


    <a class="no-after social-icon" href="https://github.com/singularityware">
      <i class="fa fa-4x fa-github no-after"></i>
    </a>


               </div>

                <div class="col-lg-6 footer">
<p><img src="images/logo/logo.png" alt="Company logo" style="width:40px;padding-bottom:10px"/></p>
 Site last generated: Jul 26, 2017 <br />
                </div>
            </div>
</footer>


    </div>
    <!-- /.row -->
</div>
<!-- /.container -->
    </div>

</body>

<!-- the google_analytics_id gets auto inserted from the config file -->


<script>(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)})(window,document,'script','//www.google-analytics.com/analytics.js','ga');ga('create','UA-84672381-1','auto');ga('require','displayfeatures');ga('send','pageview');</script>


</html>