You are here


Table of Contents


Singularity is an application containerization solution for High-Performance Computing (HPC). The goal of Singularity is to allow for "mobility of computing": an application containerized on one Linux system should be able to run on another system, as it is, and without the need to reconcile software dependencies and Linux version differences between the source and target systems. Currently Singularity is at version 2.3.1. One of the main benefits of Singularity is that containers are executed as user and can have access to local and network filesystems.  More  documentation on Singularity and its use cases can be found following the links listed at the end of this page.

Restrictions / License Information

Singularity is presently distribited under University of California / Lawrence Berkeley National Laboratory license and BSD 3-clause "Revised" License as per links below. At the time of writing, it is free and open source software. The copyright holders of Singularity are as follow:  "Copyright (c) 2017, SingularityWare, LLC. All rights reserved. Copyright (c) 2015-2017, Gregory M. Kurtzer. All rights reserved. Copyright (c) 2016-2017, The Regents of the University of California, through Lawrence Berkeley National Laboratory (subject to receipt of any required approvals from the U.S. Dept. of Energy). All rights reserved.".

Running Instructions on Grex


Grex has Singularity in production since Dec 2016. The current installed version is 2.3.1;  Singularity containers made with previous version 2.2.1 should run on Grex without changes. No overlay FS is possible on Grex, so regular users can execute their containers there, but not yet bootstrap them. Thus, for creating and bootstrapping your own containers, you would need a Linux installation where you have superuser priviliges. WG support staff could in some cases create you containers for specific software as well. (Note that with version 2.3.1, it is possible to pull Docker images directly from Docker Hub and Singularity Hub without bootstrapping locally). So the workflow can be as follows:

  1. Create a container image of sufficent size (4Gb):  sudo singularity create --size 4096 my_container_name.img
  2. Bootstrap the image using a difinition file: sudo singularity bootstrap my_container_name.img my_favorite_distro.def
  3. Often, it is easy to do the last moment configuration (creation of directories, etc.) in a shell session within the container: sudo singularity shell --writable my_container_name.img
  4. Steps 1-3 require administrator or super-user access (sudo), so they usually done by a sysadmin or on a system where you are the sysadmin: Not on Grex! but a virtual machine in ComputeCanada cloud, or your own Linux cluster, or your Linux desktop will do. Once the container is created and bootstrapped, you would transfer it to Grex and  run it as a user on Grex as described below.

Running Singularity containers

Right now, it is very easy to containerize software and workflow that are serial or SMP-parallel, running within a single node. No changes are required to the system or job scripts to run a containerized applicaion, other than adding "singularity exec my_container_name.img" in front of the executable name. Singularity would ensure that the executable in container gets proper operating system and library dependencies.

There are two commands (singularity exec and singularuty run)  to execute a container. The first method is universal ; any application present within a given container can be exececuted. Some containers have an optional %run section in their definition that specifies a command to run. If this %run section was missing on container's bootstrapping, singularity run would default to an interactive shell. By the way, singularity shell would give you an interactive shell within the container. On HPC systems in batch queuing mode, exec should be the preferred execution method.

 singularity exec my_container_name.img echo "hello world"

For Torque jobs, no modification of job scrips is required, other than specifying the container image to execute by singularity.

#! /bin/bash

#PBS -S /bin/bash
#PBS -l nodes=1:ppn=1,walltime=00:15,mem=1gb
#PBS -N sleep_job

singularity exec  my_container_name.img bash -c "sleep 10"
echo "All done!"

It is possible to run MPI parallel programs across the nodes using Hybrid mode, that is, using MPI as installed on Grex to start containerized MPI-programs. But this is still somewhat more advanced topic because MPI implementations in HPC usually are tightly coupled with hardware and kernel drivers (for Infiniband interconnect, Lustre file system etc.) so you might have to take care of matching the MPI versions within and outside of your container, and providing access to system's directories that contain drivers and OS modules within containers. We would advise you to contact Westgrid support ( if you have questions on how to run Singularity containers across the nodes.

Accessing Grex filesystems from the container

One of the strengths of Singularity is the ease of access to network and parallel filesystems without compromising security. The home directory, /home/$USER is mounted automatically. To access other directories and filesystems, they have to be "bound" to a container with corresponding command line option -B source[:target] ; obviously, the target directory should exist, be created first within the container, either when it was built or in a shell session. If the target is omitted, it is assumed to be the same path as the source. The names of the filesystems are specific to the HPC cluster you are running it on.

On Grex, you would likely want to bind the global scratch directory, /global/scratch/$USER, and perhaps the node's local scratch (as defined in $TMPDIR environment varianble) for running Singularity containers within the batch jobs. To have full access to filesystems, one has to bind /global/scratch/$USER and /global/software when executing the container. Note that bind mounts work best for real filesystem sources; automounts and symbolik links present a problem so the real filesystems should be always bound.

Example 1: $HOME is always mounted, no binding needed:

singularity exec python-r-trusty-grex.img ls $HOME

Example 2:

This magic line does bind a user's /global/scratch/$USER and a temporary directory to a singularity image, assuming the image has /global/scratch/$USER directory. Note the /sbb name as the source, because /global/scratch is automounted on Grex!

singularity exec -B /sbb/scratch:/global/scratch python-r-trusty-grex.img ls /global/scratch/$USER

Example 3:

The follwing line binds $TMPDIR (local /scratch directory defined when a script is executed under Torque) in addition to the global scratch. It will print user ID from within container, using Python.

singularity exec -B ${TMPDIR}:/tmp -B /sbb/scratch:/global/scratch \
python-r-trusty-grex.img python -c "import os; print os.getuid();


Other considerations for using Singularity containers.

Singularity images are, technically, Linux sparse files. The nominal size of containers can be quite large; however, only the used, non-zeroed space is actually consumed on disk by the sparse image file which makes great savings of disk space. This is  good, but some care have to be taken into account when moving containers around, in order not to expand the zeroes explicitly. On Linux, cp command has option "--sparse=always" and tar and rsync have option "--sparse" or "-S" as well . These options allow for handling the images efficiently.

Singularity containerizes operating system and software dependencies, but it cannot do much if your code was compiled targeting  a different CPU architecture. For example, executables compiled with Intel compilers using -mavx, -xAVX flags (or -xHost flags on an AVX-enabled CPU host) will not run on Grex because the latest instruction set it supports is SSE4.2. So when containerizing applications, a CPU dispatch should be used to make executables more portable.


Converting from Docker to Singularity containers.

Starting from version 2.3, Singularity allows for automatic import of Docker containers from DockerHub using singularity pull command. The command does not require superuser privileges! It also does not require Docker installed to work. The only caveat is that not every Docker container pulled results in a functional Singularity image: first, there can be minor issues with conversion (nested quoting in scripts that can be fixed manually aftwer pulling with singularity shell ; which, alas, needs sudo again); but second and most importantly Docker, by design, targets mainly services and daemons, while Singularity is for HPC compute processes, so converting a Docker image for a web or database service to Singularity might make little sense.

Having said that, many containers on DockerHub are not services but programming environments, scientific applications and workflows; these can be pulled and used easily on HPC systems that have Singularity, like Grex. An example below would pull and create a small Alpine Linux image with Python2; the image name will be alpine-python2.img:

singularity pull docker://frolvlad/alpine-python2

Or perhaps an Ubuntu image with Miniconda Python (going to miniconda.img)?

singularity pull docker://continuumio/miniconda

A trick (thanks to Stefan Kombrink at singularity mailing list) can be used to create mountpoints / directories within these containers as a user (w/o sudo) using singularity import command (note that the command can be also used for its original purpose, that is, adding your files to the container):

mkdir ./scratch ; tar -cp ./scratch | singularity import miniconda.img

Finally, there exist a small but growing repository of Singularity containers at Sungularity-Hub. Pulling them is as simple as Docker's, except the URL is shub://.

singularity pull shub://singularityhub/neurodebian

Link for the Singularity-Hub  is provided below.






System Grex
Version 2.3.1