Rolling reboots across all OSC clusters beginning 11/04/2026

Date: Wednesday, November 14, 2026

Affected Systems: All Clusters (Pitzer, Ascend, Cardinal)

Description: OSC systems staff will execute a rolling reboot across all active compute clusters (Pitzer, Ascend, and Cardinal) beginning Wednesday, November 14. At the start of the maintenance window, the Slurm database will be offline for approximately 30 minutes to complete database migrations. Following the database migration, schedulers will be restarted, followed by a rolling reboot of login and compute nodes, and final client software updates. Nodes will be systematically set to a drained state prior to rebooting to prevent a full cluster outage and minimize workflow disruption. 

System & Software Updates:

• Slurm upgraded from version 25.05.9 to 26.05.5

• RHEL 9.6 OS updates applied across nodes

• NVIDIA DOCA upgraded from 3.2.2 to 3.2.3

• Apptainer upgraded from 1.4.5 to 1.5.4

• Enabled the tun kernel module to support container networking (e.g., Podman)

Impact: 

• Slurm Database Outage (~30 mins): During the initial Slurm database migration, commands querying accounting or multi-cluster statuses (e.g., sacct, sacctmgr, and commands using the -M flag) will temporarily fail.

• Running Jobs: Active running jobs will not be interrupted by the scheduler or Slurm daemon upgrades.

• Queue & Scheduling: The Slurm batch scheduler will automatically hold pending jobs that cannot complete prior to a node's scheduled reboot. Held jobs will retain their queue position and resume scheduling as nodes return to service. Users should expect temporary increases in queue wait times while capacity is reduced during the rolling reboots.