The Slurm upgrades during rolling reboots of Ascend, Owens and Pitzer we performed today (Oct 25 2023) cause all running jobs on the systems requeued around 8:45am. You will not be billed for the consumed resources before the jobs were requeued.
We apologize for the inconvenience this causes you. Please contact oschelp@osc.edu if you have any questions.
Update on Nov 8 2023:
Rolling reboots of all clusters are completed.
Update on Nov 3 2023:
Rolling reboots of Ascend and Pitzer clusters are completed.
Original Post:
We will have rolling reboots of Ascend, Owens and Pitzer clusters including login and compute nodes, starting from 9AM Wednesday October 25, to perform NVIDIA driver and Slurm upgrades.
We will have rolling reboots of Owens and Pitzer clusters including login and compute nodes, starting from 9AM Monday, July 11 2022.
Updates on Feb 25 2022:
This issue is fixed.
Original Post:
Users may see an issue of missing shared library with some mvapich2 modules on Pitzer and Owens. The error is like
<path_to_executable>: error while loading shared libraries: libim_client.so.0: cannot open shared object file: No such file or directory
We are in the process of rebuilding mvapich2 versions that are affected.
OSC will shut down significant portions of the Owens and Pitzer clusters for several hours this afternoon (Thursday, Feb. 10).
When requesting an interactive session in ondemand and requesting gpu resources, users may see an error similar similar to "sbatch: error: Invalid generic resource (gres) specification"
OSC staff are currently looking into this.
This issue has been resolved.
CP2K 6.1 would fail with the following error when running on Pitzer Cascade Lakes (48-core) node:
Program received signal SIGFPE: Floating-point exception - erroneous arithmetic operation. Backtrace for this error:
Thid could be a bug in libxsmm 1.9.0 which is released on Mar 15, 2018 (Cascade Lake is launched in 2019). The bug has been fixed in cp2k/7.1.
We found mpiexec/mpirun from OpenMPI can not be used in an interactive session (launched by sinteractive) after upgrading Pitzer and Owens to Slurm 20.11.4. Please use srun only while you use OpenMPI in an interactive session.
We will have rolling reboot of Pitzer cluster. User will expect a ~10 minute outage of login nodes at about 9AM Feb 05, 2021.
We have found some types of CP2K jobs would fail or have poor performance using cp2k.popt and cp2k.psmp from MVAPICH2 build (gnu/4.8.5 mvapich2/2.3). This version will be removed on December 15th, 2020. Please switch to Intel MPI build (gnu/7.3.0 intelmpi/2018.3).
