We've been experiencing some instability on the clusters (particularly Cardinal and Ascend).
Known issue with an Unresolved Resolution state is an active problem under investigation; a temporary workaround may be available.
A known issue with a Resolved (workaround) Resolution state is an ongoing problem; a permanent workaround is available which may include using different software or hardware.
A known issue with Resolved Resolution state has been corrected.
| Title | Category | Resolution | Description | Posted | Updated |
|---|---|---|---|---|---|
| NCCL hang on Ascend dual-GPU nodes | Ascend, GPU, Software |
Resolved (workaround) |
Users may encounter the following message and experience NCCL hangs when running multi-GPU training: [Global Rank: 1] is using GPU 1 on this node. [Global Rank: 1] waiting for first... Read more |
4 days 2 hours ago | 4 days 2 hours ago |
| STAR error bgzf_open: Assertion failed | Cardinal, Software |
Resolved (workaround) |
You may encounter errors that look similar to these when running STAR 2.7.10b: STAR: bgzf.c:158: bgzf_open: Assertion `compressBound(0xff00) < 0x10000' failed. Cause... Read more |
2 weeks 2 days ago | 4 days 4 hours ago |
| Python version mismatch in Jupyter + Spark instance | Software |
Resolved (workaround) |
You may encounter the following error message when running a Spark instance using a custom kernel in the Jupyter + Spark app: 25/04/25 10:49:01 WARN TaskSetManager: Lost task 0.0 in... Read more |
2 weeks 2 days ago | 2 weeks 2 days ago |
| WARN SparkSession in Jupyter + Spark instance | Software |
Resolved (workaround) |
You may encounter the following warning message when running a Spark instance using the default PySpark kernel in a Jupyter + Spark application: WARN SparkSession: Using an... Read more |
2 weeks 4 days ago | 2 weeks 2 days ago |
| Instability on Clusters after May 13 Downtime |
Unresolved |
We've been experiencing some instability on the clusters (particularly Cardinal and Ascend) following the recent May 13 downtime, especially with parallel job processing. If you notice any unusual... Read more |
2 weeks 2 days ago | 2 weeks 2 days ago | |
| Singularity: failed to run a container directly or pull an image from Singularity or Docker hub | Software |
Resolved (workaround) |
You might encounter an error while run a container directly from a hub: [pitzer-login01]$ apptainer run shub://vsoch/hello-world Progress |===================================| 100.0%... Read more |
2 weeks 4 days ago | 2 weeks 4 days ago |
| Singularity: failed to pull a large Docker image | Software |
Resolved (workaround) |
You might encounter an error while pulling a large Docker image: [pitzer-login01]$ apptainer pull docker://qimme2/core FATAL: Unable to pull docker://qiime2/core While running... Read more |
2 weeks 4 days ago | 2 weeks 4 days ago |
| Singularity: reached your pull rate limit | Software |
Resolved (workaround) |
You might encounter an error while pulling a large Docker image: ERROR: toomanyrequests: Too Many Requests. or You have reached your pull rate limit. You may... Read more |
3 years 11 months ago | 2 weeks 4 days ago |
| MPI-IO issues on home directories with Intel MPI 2019.3 | Pitzer, Software |
Resolved (workaround) |
Certain MPI-IO operations with |
2 weeks 4 days ago | 2 weeks 4 days ago |
| Using mpiexec/mpirun with Intel MPI on Slurm | Software |
Resolved (workaround) |
Intel MPI on Slurm batch system is configured to support PMI process manager. It is recommended to use |
2 weeks 4 days ago | 2 weeks 4 days ago |
