Desmond OpenMPI OPAL ERROR when Slurm is updated to 26.05.5
Slurm 25.05.9 will be upgraded to 26.05.5 in rolling reboots. Users may then experience the following error when running Desmond:
Slurm 25.05.9 will be upgraded to 26.05.5 in rolling reboots. Users may then experience the following error when running Desmond:
We have identified a potential performance regression in point-to-point communication when using NCCL version 2.29 on the Ascend and Pitzer clusters, which results in a 15% to 25% performance drop.
If you experience performance regression on Pitzer or Ascend, please switch to NCCL version 2.23.4.
In some cases, matlab would hang with connecting to parallel pool on Pitzer. This was discovered in the 2026 January downtime. In order to mitigate this we have introduced the "current" parallel profile
Under the "hpc" profile, matlab will generate a new slurm job for parfor loops and the like. This means that one can request a single core matlab OnDemand job, and generate jobs for calculations separate from the OnDemand job
Mathematica was not launching due to firewall issues. We point our mathematica instances at an external Mathematica server, which have stopped working. The issue is now resolved as of Sep 24, 2025
On all clusters the cp2k executables from module cp2k/2023.2 can produce huge output files due to many many repeating errors from MKL, e.g.:
MKL module files define some helper environment variables with incorrect paths. This can yield link time errors. All three clusters are affected. We are working to correct the module files. A workaround for users is to redefine the environment variable with the correct path; this requires some computational maturity. We recommend users contact oschelp@osc.edu for assistance. An example error from Cardinal with module intel-oneapi-mkl/2023.2.0 that defined environment variable MKL_LIBS_INT64 follows:
Certain MPI-IO operations with intelmpi/2019.3 may crash, fail or proceed with errors on the home directory. We do not expect the same issue on our GPFS file system, such as the project space and the scratch space. The problem might be related to the known issue reported by HDF5 group. Please read the section "Problem Reading A Collectively Written Dataset in Parallel" from HDF5 Known Issues for more detail.
2019.3
According to https://github.com/cp2k/cp2k/issues/1830 and user feedback, you may encounter Out-of-Memory (OOM) errors during long molecular dynamics (MD) simulations with CP2K 7.1 on Pitzer and Owens clusters due to a memory leak issue in Intel MPI. If you experience this problem, consider switching to a newer version available on the system.
When running a full-node MPI job with MVAPICH 3.0 , you may encounter the following warning message:
Ascend, Cardinal, and Owens completed. Pitzer is ongoing.