After upgrading the operating system to RHEL 9.6 during the scheduled downtime on May 12, 2026, applications utilizing UCX (rc_gda) or GPU-initiated networking may experience failures.
When trying to run starccm on a vis node in order to use GPU for visualization (not for GPU calculation), the typical way to do this would be to load the virtualgl module and use vglrun
module load virtualgl/3.1.1
module load starccm/19.06.009
vglrun starccm+
However, starccm does not pick up libraries from $LD_LIBRARY_PATH by default, so it does not pick up the necessary graphics libraries
We have noticed that GPU memory is not being properly released in some jobs, causing subsequent jobs on the same nodes to run out of memory (OOM). We are currently working on a resolution.
If you encounter an OOM error in a GPU job, you can use the job-dashboard-link.py script to generate a Grafana dashboard link for your job’s resource usage. For example:
job-dashboard-link.py -M cardinal 2502244
Open the generated URL and navigate to GPU Metrics. Check the GPU Memory Usage panel.
Users may encounter the following message and experience NCCL hangs if the first operation is a barrier when running multi-GPU training. We have identified that this issue occurs only on a single Ascend Next Gen (dual-GPU) node where the GPUs are connected via the SMP interconnect across NUMA nodes, rather than through NVLink.
PyTorch can hang on Ascend on dual-GPU nodes
Through internal testing, we have confirmed that the hang issue only occurs on Ascend dual-GPU (nextgen) nodes. We’re still unsure why setting NCCL_P2P_DISABLE is necessary even for single-node setups. However, the performance impact should be minimal. This is because, on dual-GPU nodes, the GPUs are connected via the SMP interconnect across NUMA nodes. As there’s no NVLink on these nodes, GPU communication occurs through shared memory
To get around this set the environment variable NCCL_P2P_DISABLE=1
We have observed that many PyTorch users frequently encounter random timeouts, which result in the termination of their jobs but leave the process running on the node. Consequently, this situation necessitates a reboot of the node.
UPDATE (Mar 15, 2023)
After the downtime on Mar. 14, 2023, OSC enabled a new Slurm option --gres=nsight. DCGM will be disabled on the nodes for the job with the Slurm option, and Nsight will function normally. More information about Nsight is available on the NVHPC page.
==================================
We currently have performance regression of Ruby's GPU nodes. Some of the GPU nodes on Ruby will remain in a power-saving state even after an application starts using them, resulting in performance reduction in some cases. We currently have a reservation on the GPU nodes so that we can do a rolling reboot on them to get them back into a known-good state.
We have opened a bug report with the vendor about this performance regression and how to monitor for it.
We upgraded the drivers for the Nvidia GPUs on all of our clusters during the downtime this week. Unfortunately, we are noticing some subtle problems with the GPUs on Oakley. We will be rolling back to an older driver on that cluster; the GPUs will be unavailable until that work is completed, potentially all weekend.
Update: The driver version has been updated and the issue has been fixed.
In updating the driver version for Oakley's NVIDIA GPUs the NVML libraries that are used in conjunction with torque, our resource manager, were also updated. Unfortunatelly this update has broken the ability to change the GPUs compute mode from its default exclusive thread mode. Requests to change the GPU mode both through a PBS directive and the nvidia-smi command will both fail.
