Known issues

Unresolved known issues

Known issue with an Unresolved Resolution state is an active problem under investigation; a temporary workaround may be available.

Resolved known issues

A known issue with a Resolved (workaround) Resolution state is an ongoing problem; a permanent workaround is available which may include using different software or hardware.

A known issue with Resolved Resolution state has been corrected.

Known Issues

Title Category Resolution Description Posted Updated
Slurm database repair on 01/25/2024 Outage Resolved

We have scheduled a Slurm database repair, which is planned to start at 8:30 am US/Eastern on Thursday, January 25, 2024. During the repair, Slurm database will be offline; running jobs and... Read more

4 months 3 weeks ago 4 months 2 weeks ago
PyTorch jobs timeout and hanging GPU Resolved

We have observed that many PyTorch users frequently encounter random timeouts, which result in the termination of their jobs but leave the process running on the node.... Read more

11 months 1 week ago 5 months 1 week ago
Rolling reboot of Ascend, Owens and Pitzer starting from Oct 25 2023 Owens, Pitzer Resolved

Update on Nov 8 2023:

Rolling reboots of all clusters are completed. 

Update on Nov 3 2023:

Rolling reboots of Ascend and Pitzer clusters... Read more

7 months 3 weeks ago 7 months 1 week ago
Running jobs requeued on all clusters Owens, Pitzer Resolved

The Slurm upgrades during rolling reboots of Ascend, Owens and Pitzer we performed today (Oct 25 2023) cause all running jobs on the systems requeued around 8:45am. You will not be billed for the... Read more

7 months 3 weeks ago 7 months 3 weeks ago
Emergency UPS Maintenance Maintenance Resolved

A UPS in the data center requires some emergency maintenance to be undertaken at 2PM on Oct 11 2023. There is a very small chance that parts of Owens and of the C18 Pitzer nodes may lose power as... Read more

8 months 1 week ago 8 months 4 days ago
MOE license server down Licensing Resolved

The MOE license server is experiencing an unknown issue and potentially down.  We are working to resolve the issue.

9 months 21 hours ago 9 months 19 hours ago
Outbound Emails from MyOSC are Blocked at MS 365 Servers Account Management, client portal Resolved

Outbound emails, including account verification emails, from MyOSC to institutional email addresses utilizing MS 365 are being blocked as phishing attempts at the institution's server.

... Read more

9 months 3 weeks ago 9 months 1 week ago
Intermittent home directory performance issues filesystem Resolved

Users may experience performance issues in home directory. It is recommended to use temporary directory ($TMPDIR, or scratch) or project storage to minimize the impact on... Read more

11 months 3 weeks ago 11 months 4 days ago
Possible performance degradation after August 9th's downtime filesystem Resolved

Updates on May 20 2023:

verbsRDMA is enabled on Pitzer. 

Updates on Dec 14 2022:

verbsRDMA is enabled on Owens during December 13 downtime... Read more

1 year 10 months ago 1 year 3 weeks ago
MPI_THREAD_MULTIPLE is not supported with OpenMPI-HPCX 4.x Owens, Software Resolved
(workaround)

A threading code with MPI where MPI_Init_thread uses MPI_THREAD_MULTIPLE will fail because UCX from HPCX package is built without enabling multi-threading. UCX is the... Read more

1 year 3 months ago 1 year 1 month ago

Pages