It is now possible to run Docker and Apptainer/Singularity containers on all clusters at OSC. Single-node jobs are currently supported, including GPU jobs; MPI jobs are planned for the future.
From the Docker website: "A container image is a lightweight, stand-alone, executable package of a piece of software that includes everything needed to run it: code, runtime, system tools, system libraries, settings."
As of June 21st, 2022, Singularity is replaced with Apptainer, which is just a renamed open-source project. For more information visit the Apptainer/Singularity page
This document will describe how to run Docker and Apptainer/Singularity containers on OSC clusters. You can use containers from Docker Hub, Sylabs Cloud, or any other source. As examples we will use hello-world and ubuntu from Docker Hub.
If you encounter an error then check the Known Issues on using Apptainer/Singularity or Podman at OSC. If the issue can not be resolved, please contact OSC help.
For Apptainer/Singularity, use apptainer help.
For Podman/Docker, use podman help.
User guides and examples are available at Podman documentation and Apptainer documentation.
No setup is required. You can use Podman or Apptainer/Singularity directly on all clusters.
A container image is a file (e.g. .sif for Apptainer) or image stored in a registry (for Docker/Podman).
You can pull images from hubs: Docker Hub, Sylabs Cloud, or other registries.
Examples:
# Pull the gcc:7.2.0 image from Docker Hub → gcc_7.2.0.sif apptainer pull docker://gcc:7.2.0 # Ubuntu 18.04 → ubuntu_18.04.sif apptainer pull docker://ubuntu:18.04 # Pull from Singularity Hub apptainer pull shub://singularityhub/hello-world
Downloading Apptainer/Singularitycontainers from the hubs is not the only way to get one. You can, for example get a copy from your colleague's computer or directory. If you would like to create your own container you can start from the Build a container section below. If you have any questions, please contact OSC Help.
With Podman/Docker, you pull images to your local image store:
podman pull ubuntu:18.04 podman pull docker.io/library/gcc:7.2.0
Use podman images to list available images in the local registry:
REPOSITORY TAG IMAGE ID CREATED SIZE docker.io/library/ubuntu 18.04 f9a80a55f492 2 years ago 65.5 MB docker.io/library/gcc 7.2.0 81ffb25b1dec 7 years ago 1.73 GB
You can run containers on OSC clusters either interactively or in batch jobs.
We note that the operating system on OSC cluster is Red Hat by running cat /etc/os-release:
NAME="Red Hat Enterprise Linux Server" [..more..]
In the examples below we will often check the operating system to show that we are really inside a container.
If you simply run the container image, it will execute the container’s runscript.
apptainer pull docker://hello-wolrd ./hello-world_latest.sif
You should see the following output:
Hello from Docker! This message shows that your installation appears to be working correctly.
The Apptainer “run” sub-command does the same thing as running a container directly as described above. That is, it executes the container’s runscript.
Example: Run a container from a local file
apptainer run hello-world_latest.sif
Example: Run a container from a hub without explicitly downloading it
apptainer run docker://hello-world
The Apptainer “exec” sub-command lets you execute an arbitrary command within your container instead of just the runscript.
Example: Find out what operating system the ubuntu:18.04 container uses
apptainer pull docker://ubuntu:18.04 apptainer exec ./ubuntu_18.04.sif cat /etc/os-release
You should see the following output:
NAME="Ubuntu" VERSION="18.04.6 LTS (Bionic Beaver)"
The Apptainer “shell” sub-command invokes an interactive shell within a container.
Example: Run an Ubuntu shell.
apptainer shell ubuntu_18.04.sif
You should now see the prompt Apptainer>, indicating that you are logged into the container. You can verify the operating system version by running:
Apptainer> cat /etc/os-release
Output:
NAME="Ubuntu" VERSION="18.04.6 LTS (Bionic Beaver)"
To exit the container, simply type exit.
With Podman or Docker:
Run: runs the container’s default command
podman run hello-world
Exec: execute arbitrary command inside running (or started) container
podman run -t -d --name my_ubuntu ubuntu:18.04 podman exec my_ubuntu cat /etc/os-release
Interactive shell:
podman run -it ubuntu:18.04
When you use a container you run within the container’s environment. The directories available to you by default from the host environment are
/fs/ess/tmp You can review our Available File Systems page for more details about our file system access policy.
If you run the container within a job you will have the usual access to the $PFSDIR environment variable with adding node attribute "pfsdir" in the job request (--gres=pfsdir). You can access most of our file systems from a container without any special treatment.
If using Podman/Docker, you may need to explicitly bind mount host directories into the container. For example:
podman run -it -v $HOME:$HOME -v /fs/ess:/fs/ess $ubuntu:18.04
If you have a GPU-enabled container you can easily run it on Pitzer just by adding the --nv flag to the apptainer exec or run command. The example below comes from the "exec" command section of Apptainer User Guide. It runs a TensorFlow example using a GPU on Pitzer. (Output has been omitted from the example for brevity.)
[pitzer-login01]$ salloc -N 1 --ntasks-per-node=4 -G 1...[p0756]$git clone https://github.com/tensorflow/models.git[p0756]$apptainer exec --nv docker://tensorflow/tensorflow:latest-gpu \ python ./models/tutorials/image/mnist/convolutional.py
In some cases it may be necessary to bind the CUDA_HOME path and add $CUDA_HOME/lib64 to the shared library search path:
[pitzer-login01]$ salloc -N 1 --ntasks-per-node=4 -G 1...[p0756]$module load cuda [p0756]$ export APPTAINER_BINDPATH=$CUDA_HOME [p0756]$ export APPTAINERENV_LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CUDA_HOME/lib64[p0756]$napptainer exec --nv my_container mycmd
To use a GPU in a Docker container, you need to add the GPU device using the --device option.
For example, to request a GPU node with one GPU:
salloc -n 1 -G 1
After obtaining the node, you can test if the GPU device is available in a container by running:
podman run --rm --device nvidia.com/gpu=all nvidia/cuda:11.0.3-base-ubuntu20.04 nvidia-smi
If successful, the nvidia-smi command will display details about the GPU, such as model, memory usage, and driver version.
OSC users can now build an Apptainer/Singularity container image from a definition file using the fakeroot feature:
apptainer build --fakeroot myimage.sif mydef.def
When building an image, it is recommended to change the cache folder (the default location is $HOME/.apptainer/cache) to improve build efficiency and avoid potential file system issues, especially when building a large container image.
Request a compute node with sufficient memory. The following example requests a compute node with 32 GB of memory:
sinteractive -n 8
Change the cache folder to a temporary file system:
export APPTAINER_CACHEDIR=$TMPDIR
Build an image:
apptainer build --fakeroot myimage.sif mydef.def
These steps help optimize performance and prevent file system issues when building large container images.
OSC users can now build a Docker container image from a Dockerfile:
buildah build -f Dockerfile --format docker -t tag_my_container .
The option --format docker ensures that the container format is compatible with the Docker schema for the manifest. The -t flag is used to tag the image, typically in the format name:version.
For example, if you set -t my_container:1.0, you should see the following after listing images with podman images:
REPOSITORY TAG IMAGE ID CREATED SIZE local/my_container 1.0 f9a80a55f492 30 mins ago 65.5 MB
Note that our local registry is saved on a local disk. This means that if you build an image on a compute node, it will not be accessible from other nodes and will be removed once the job ends.
To ensure that your container image can be reused, you should create an account on Docker Hub (or another supported container registry) and tag your image with the registry URL. For example:
podman login podman tag my_container:1.0 docker.io/<username>/my_container:1.0 podman push docker.io/<username>/my_container:1.0