Reboot of Public-Facing Systems for Security Fix

Due to a critical security vulnerability we need to reboot public-facing systems to deploy a mitigation against the vulnerabilit

[Downtime] Software Environment Changes

Date: 
Tuesday, May 12, 2026 - 7:00am
System(s): 
Ascend
Cardinal
Pitzer

During the upcoming scheduled downtime, the following user-facing changes to the software environment are expected:

  • Apptainer will be upgraded to version 1.4.5.
  • Podman/Docker configuration change: the runroot directory will be set to /tmp to address potential storage insufficiency issues.
  • Enhanced GPU-to-GPU communication over the network on Cardinal and Ascend systems.
    • NCCL upgrade required: If you are using NCCL or any application that depends on it, it is highly recommended to upgrade to version 2.23 or later to avoid potential performance regressions.
    • UCX-related rebuild recommended: If you have MPI implementations or applications built with UCX 1.18 or earlier, it is highly recommended to rebuild them with UCX 1.19 or later to avoid potential performance regressions.