Rolling reboots across all OSC clusters beginning 09/16/2026

OSC systems staff will execute a rolling reboot across all active compute clusters (Pitzer, Ascend, and Cardinal) beginning Wednesday, September 16.

NCCL 2.29 Performance Regression on Ascend and Pitzer Clusters

Category: 
Ascend
Pitzer
Software
Resolution: 
Resolved
Workaround Link: 
Workaround Link
Affected Software: 
NCCL

We have identified a potential performance regression in point-to-point communication when using NCCL version 2.29 on the Ascend and Pitzer clusters, which results in a 15% to 25% performance drop.

Workaround

If you experience performance regression on Pitzer or Ascend, please switch to NCCL version 2.23.4.