Known issue with an Unresolved Resolution state is an active problem under investigation; a temporary workaround may be available.
A known issue with a Resolved (workaround) Resolution state is an ongoing problem; a permanent workaround is available which may include using different software or hardware.
A known issue with Resolved Resolution state has been corrected.
| Title | Category | Resolution | Description | Posted | Updated |
|---|---|---|---|---|---|
| Rolling reboot of Ascend, Owens and Pitzer starting from Oct 25 2023 | Owens, Pitzer |
Resolved |
Update on Nov 8 2023: Rolling reboots of all clusters are completed. Update on Nov 3 2023: Rolling reboots of Ascend and Pitzer clusters... Read more |
1 month 1 week ago | 3 weeks 3 days ago |
| PyTorch jobs timeout and hanging | GPU |
Unresolved |
We have observed that many PyTorch users frequently encounter random timeouts, which result in the termination of their jobs but leave the process running on the node.... Read more |
4 months 3 weeks ago | 1 month 1 week ago |
| Running jobs requeued on all clusters | Owens, Pitzer |
Resolved |
The Slurm upgrades during rolling reboots of Ascend, Owens and Pitzer we performed today (Oct 25 2023) cause all running jobs on the systems requeued around 8:45am. You will not be billed for the... Read more |
1 month 1 week ago | 1 month 1 week ago |
| Emergency UPS Maintenance | Maintenance |
Resolved |
A UPS in the data center requires some emergency maintenance to be undertaken at 2PM on Oct 11 2023. There is a very small chance that parts of Owens and of the C18 Pitzer nodes may lose power as... Read more |
1 month 3 weeks ago | 1 month 2 weeks ago |
| MOE license server down | Licensing |
Resolved |
The MOE license server is experiencing an unknown issue and potentially down. We are working to resolve the issue. |
2 months 1 week ago | 2 months 1 week ago |
| Outbound Emails from MyOSC are Blocked at MS 365 Servers | Account Management, client portal |
Resolved |
Outbound emails, including account verification emails, from MyOSC to institutional email addresses utilizing MS 365 are being blocked as phishing attempts at the institution's server. ... Read more |
3 months 1 week ago | 2 months 3 weeks ago |
| Intermittent home directory performance issues | filesystem |
Resolved |
Users may experience performance issues in home directory. It is recommended to use temporary directory ($TMPDIR, or scratch) or project storage to minimize the impact on... Read more |
5 months 1 week ago | 4 months 2 weeks ago |
| Possible performance degradation after August 9th's downtime | filesystem |
Resolved |
Updates on May 20 2023: verbsRDMA is enabled on Pitzer. Updates on Dec 14 2022: verbsRDMA is enabled on Owens during December 13 downtime... Read more |
1 year 3 months ago | 6 months 1 week ago |
| MPI_THREAD_MULTIPLE is not supported with OpenMPI-HPCX 4.x | Owens, Software |
Resolved (workaround) |
A threading code with MPI where MPI_Init_thread uses MPI_THREAD_MULTIPLE will fail because UCX from HPCX package is built without enabling multi-threading. UCX is the... Read more |
9 months 6 days ago | 6 months 4 weeks ago |
| Possible job failures due to MPI library change on Pitzer after May 20 | Software |
Resolved |
There are changes on MPI libraries on Pitzer after May 20. We will upgrade MOFED from 4.9 to 5.6 and recompile all OpenMPI and Mvapich2 against the newer MOFED version. Users with their own MPI... Read more |
7 months 5 days ago | 7 months 5 days ago |