We've been experiencing some instability on the clusters (particularly Cardinal and Ascend). 

Singularity: failed to pull a large Docker image

Category: 
Resolution: 
Resolved

You might encounter an error while pulling a large Docker image:

[pitzer-login01]$ apptainer pull docker://qimme2/core
FATAL: Unable to pull docker://qiime2/core While running mksquashfs: signal: killed

The process could be killed because the image is cached in the home directory which is a slower file system or the image size might exceed a single file size limit.

The solution is to use other file systems like /fs/ess/scratch and $TMPDIR for caches and temp files to build the squashfs filesystem:

[pitzer-login01]$ salloc -n 1 -A PAS1234 
bash-4.2$ export APPTAINER_CACHEDIR=$TMPDIR
bash-4.2$ export APPTAINER_TMPDIR=$TMPDIR 
bash-4.2$ singularity pull docker://qiime2/core:2019.1
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
...
INFO:    Creating SIF file...
bash-4.2$ exit