We are actively investigating intermittent slowness on the shared /fs/scratch filesystem.

OSC systems staff will execute a rolling reboot across all active compute clusters (Pitzer, Ascend, and Cardinal) beginning Thursday, November 05.

Known issues

Unresolved known issues

Known issue with an Unresolved Resolution state is an active problem under investigation; a temporary workaround may be available.

Resolved known issues

A known issue with a Resolved (workaround) Resolution state is an ongoing problem; a permanent workaround is available which may include using different software or hardware.

A known issue with Resolved Resolution state has been corrected.

Known Issues

Titlesort ascending Category Resolution Description Posted Updated
Oakley and Owens queue issue Batch Resolved

We are experiencing a problem with the queuing system on oakley and owens that is delaying or preventing new jobs from running. Our systems staff is investigating.

 

8 years 10 months ago 8 years 10 months ago
Nvidia drivers on Oakley GPU Resolved

We upgraded the drivers for the Nvidia GPUs on all of our clusters during the downtime this week. Unfortunately, we are noticing some subtle problems with the GPUs on Oakley. We will be rolling...

Read more
10 years 1 month ago 8 years 5 months ago
Nsight GPU profiler not working due to DCGM conflict GPU, Infrastructure Resolved

UPDATE (Mar 15, 2023)

After the downtime on Mar. 14, 2023, OSC enabled a new Slurm option --gres=nsight. DCGM will be disabled on the nodes for the...

Read more
3 years 8 months ago 5 months 4 weeks ago
NFS service disruption 6/29/16 filesystem Resolved

OSC experienced errors with NFS services the morning of June 29 between 08:37 and 09:12 that may have caused some jobs to fail, or other unexpected behavior.  The...

Read more
10 years 4 months ago 10 years 4 months ago
NFS outage on Thursday Jan 17 from 7am to 8am filesystem Resolved

Update:

This work has been canceled and will be done during downtime on Feb. 5. 

Original Post:

On Thursday, January 17th from 7 am to 8 am OSC will have a GPFS...

Read more
7 years 10 months ago 7 years 10 months ago
Network card re-seat Network Resolved

At 8AM on Tuesday, July 9th 2013, we will be re-seating a network card in a switch at our operations center. It is possible that a brief (~10 minute) outage may occur. Jobs will pause for the...

Read more
13 years 4 months ago 13 years 4 months ago
Negative Balance Emails client portal Resolved

Negative balance emails continue to be sent once an application is submitted.

To confirm whether or not you have truly submitted an application for additional resources and that you can...

Read more
7 years 6 months ago 7 years 2 months ago
NCCL hang on Ascend dual-GPU nodes Ascend, GPU, Software Resolved
(workaround)

Users may encounter the following message and experience NCCL hangs if the first operation is a barrier when running multi-GPU training. We have identified...

Read more
1 year 5 months ago 1 year 5 months ago
NCCL 2.29 Performance Regression on Ascend and Pitzer Clusters Ascend, Pitzer, Software Resolved
(workaround)

We have identified a potential performance regression in point-to-point communication when using NCCL version 2.29 on the Ascend and Pitzer clusters, which results...

Read more
3 months 2 weeks ago 3 months 2 weeks ago
NAMD 2.11 precompiled binaries do not work Software Resolved

NAMD 2.11 precompiled binaries do not work.  Please use NAMD 2.11 installed from the source and available via module namd/2.11.

The NAMD 2.11 issue involves changes to the command charmrun...

Read more
10 years 9 months ago 7 years 9 months ago

Pages