Problems with home directory servers
We had several auto-reboots with one of our home directory servers. Jobs might be impacted.
/fs/ess and OnDemand not accessible
/fs/ess and OnDemand are not accessible now. We are working on this.
Sorry for the inconvenience. Please contact OSC Help if you have any questions.
Backup failures on ess filesystem
The backups on the /fs/ess filesystem are having issues running. There has not been a successful backup of this filesystem since Sunday, 08 August 2021.
OSC is working with the vendor to resolve this issue as soon as possible.
Problems with GPFS filesystem, and OnDemand is not working
Updates on 12:00pm June 10:
The issue is fixed. All impacted services return to production.
We apologize for the inconvience. If you have any questions, please contact OSC Help.
Original Post:
We have been experiencing some problems with GPFS filesystems starting from 2:34am, Thursday, June 10. Web portals including OnDemand and WebMO are not working. It may also cause unexpected job failures.
Intermittent home directory performance issues
Users may experience performance issues in home directory. It is recommended to use temporary directory ($TMPDIR, or scratch) or project storage to minimize the impact on your jobs.
OSC is currently troubleshooting the cause. Contact oschelp@osc.edu if there are questions.
GPFS errors on compute nodes
We've seen an increase in transient problems that result in compute nodes losing access to the GPFS file systems for ~5 minutes.
Any jobs running on these nodes accessing files on GPFS may have seen errors. GPFS includes /fs/ess, /fs/project and /fs/scratch directories.
If you believe that your job may have been affected by this error, please contact oschelp@osc.edu
Problems with the home directories
We are currently seeing problems with the home directories at OSC's HPC facility.
Security Vulnerability for GPFS
Update: The fix was deployed during May 19 Downtime.
(informational) GPFS maintenance work duplicate known issue
Maintenance work on the GPFS servers is scheduled to be performed today, 28 Feb 2020 at 2:00p.m.
Although there is no direct impact expected to services at OSC, there may be short interruptions to storage services.
Please contact OSC Help at oschelp@osc.edu if you have any questions.
Storage Problems - GPFS services (Project / Scratch)
OSC Project and Scratch file systems have resumed normal operations.