Apptainer sandbox fails on GPFS due to stale NFS file handles
Users may encounter errors when attempting to run a sandbox on GPFS-mounted directories such as /fs/scratch or /fs/ess. The error output may look like the following:
Users may encounter errors when attempting to run a sandbox on GPFS-mounted directories such as /fs/scratch or /fs/ess. The error output may look like the following:
While using MVAPICH3 builds of Quantum ESPRESSO (QE), users may encounter hangs when running the CP package, which can lead to job timeouts. We recommend switching to the OpenMPI build of any QE version.
Please switch to Intel-OpenMPI version. You can access it via
module load intel/2021.10.0 openmpi/5.0.2 module load quantum-espresso/7.3.1
Note that MVAPICH3 variants will be deprecated on August 19, 2025
MKL module files define some helper environment variables with incorrect paths. This can yield link time errors. All three clusters are affected. We are working to correct the module files. A workaround for users is to redefine the environment variable with the correct path; this requires some computational maturity. We recommend users contact oschelp@osc.edu for assistance. An example error from Cardinal with module intel-oneapi-mkl/2023.2.0 that defined environment variable MKL_LIBS_INT64 follows:
Users may experience their home directory running out of space after executing multiple MATLAB 2024a jobs. This issue is caused by the accumulation of multiple copies of the MathWorks Service Host in $HOME/.MathWorks/ServiceHost.
To address this, we have upgraded MATLAB 2024a to Update 7 on all clusters, as recommended in the following article: Why is the MathWorks Service Host causing issues with my cluster and/or HPC?
Users may encounter the following message and experience NCCL hangs if the first operation is a barrier when running multi-GPU training. We have identified that this issue occurs only on a single Ascend Next Gen (dual-GPU) node where the GPUs are connected via the SMP interconnect across NUMA nodes, rather than through NVLink.
You may encounter the following error message when running a Spark instance using a custom kernel in the Jupyter + Spark app:
You may encounter errors that look similar to these when running STAR 2.7.10b:
STAR: bgzf.c:158: bgzf_open: Assertion `compressBound(0xff00) < 0x10000' failed.
It seems to be related to this issue: https://github.com/alexdobin/STAR/issues/2063
STAR bundles an older version of HTSlib which is incompatible with zlib-ng, a library we build STAR with
Use star/2.7.11b
You may encounter the following warning message when running a Spark instance using the default PySpark kernel in a Jupyter + Spark application:
You might encounter an error while run a container directly from a hub:
[pitzer-login01]$ apptainer run shub://vsoch/hello-world Progress |===================================| 100.0% FATAL: container creation failed: mount error: can't mount image /proc/self/fd/13: failed to find loop device: could not attach image file too loop device: No loop devices available
One solution is to remove the Singularity cached images from local cache directory $HOME/.apptainer/cache
.
You might encounter an error while pulling a large Docker image:
[pitzer-login01]$ apptainer pull docker://qimme2/core FATAL: Unable to pull docker://qiime2/core While running mksquashfs: signal: killed
The process could be killed because the image is cached in the home directory which is a slower file system or the image size might exceed a single file size limit.
The solution is to use other file systems like /fs/ess/scratch
and $TMPDIR
for caches and temp files to build the squashfs filesystem: