Singularity

Singularity is a container system designed for use on High Performance Computing (HPC) systems. It allows users to run both Docker and Singularity containers.

From the Docker website: "A container image is a lightweight, stand-alone, executable package of a piece of software that includes everything needed to run it: code, runtime, system tools, system libraries, settings."

Availability and Restrictions

Versions

Singularity is available on Owens and Pitzer clusters. Only one version is available at any given time. To find out the current version:

singularity version

Check the release page for the changelog: https://github.com/sylabs/singularity/releases

Access

Singularity is available to all OSC users.

Publisher/Vendor/Repository and License Type

SyLabs, Inc., 3-clause BSD License

Usage

Set-up

No setup is required. You can use Singulairty directly on all clusters.

Using Singularity

See HOWTO: Use Docker and Singularity Containers at OSC for information about using Singularity on Owens and Pitzer clusters, including some site-specific caveats.  

Example:  Run a container from the Singularity hub

[owens-login01]$ singularity run shub://singularityhub/hello-world
INFO:    Downloading library image
Tacotacotaco


Known Issues

Reached your pull rate limit

Update: 06/16/2021 
Version: all

You might encounter an error while pulling a large Docker image:

ERROR: toomanyrequests: Too Many Requests.

or

You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limits.

On November 20, 2020, Docker Hub puts rate limits on anonymous and free authenticated pull requests. The rate limits for anonymous and authenticated pulls are 100 per 6 hours and 200 per 6 hours, respectively. Anonymous users have limits enforced via IP. Since all computing nodes at OSC share the same IP, anonymous pull rate limit is shared by all OSC users if you are not authenticated. 

If you encounter this error and want to get rid of it, please consider setting up authenticated access to  Docker Hub: https://sylabs.io/guides/3.7/user-guide/endpoint.html#sec-managing-oci-r....

 

Failed to pull a large Docker image

Update: 05/21/2019 
Version: all

You might encounter an error while pulling a large Docker image:

[owens-login01]$ singularity pull docker://qimme2/core
FATAL: Unable to pull docker://qiime2/core While running mksquashfs: signal: killed

The process could be killed because the image is cached in the home directory which is a slower file system or the image size might exceed a single file size limit.

The solution is to use other file systems like /fs/scratch and $TMPDIR for caches and temp files to build the squashfs filesystem:

[owens-login01]$ sinteractive -n 1 -A PAS1234 
bash-4.2$ export SINGULARITY_CACHEDIR=$TMPDIR
bash-4.2$ export SINGULARITY_TMPDIR=$TMPDIR 
bash-4.2$ singularity pull docker://qiime2/core:2019.1
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
...
INFO:    Creating SIF file...
bash-4.2$ exit

Failed to run a container directly or pull an image from Singularity or Docker hub

Update: 03/08/2019 
Version: all

You might encounter an error while run a container directly from a hub:

[owens-login01]$ singularity run shub://vsoch/hello-world
Progress |===================================| 100.0%
FATAL: container creation failed: mount error: can't mount image /proc/self/fd/13: failed to find loop device: could not attach image file too loop device: No loop devices available

One solution is to remove the Singularity cached images from local cache directory $HOME/.singularity/cache. If the singularity version is prior to 3.0:

[owens-login01]$ cd $HOME/.singularity/cache
[owens-login01 cache]$ rm -rf *

If the singularity version is 3.1 and above:

[owens-login01]$ singulariy cache clean

Alternatively, you can change the Singularity cache directory to different location by setting the variable SINGULARITY_CACHEDIR. For example, in a batch job:

#/bin/bash
#SBATCH --job-name="singularity_test"
#SBSTCH --ntasks=1

export SINGULARITY_CACHEDIR=$TMPDIR
singularity run shub://vsoch/hello-world

Workshop

Further Reading

Supercomputer: