We had several auto-reboots with one of our home directory servers. Jobs might be impacted.
/fs/ess and OnDemand are not accessible now. We are working on this.
Sorry for the inconvenience. Please contact OSC Help if you have any questions.
The backups on the /fs/ess filesystem are having issues running. There has not been a successful backup of this filesystem since Sunday, 08 August 2021.
OSC is working with the vendor to resolve this issue as soon as possible.
Updates on 12:00pm June 10:
The issue is fixed. All impacted services return to production.
We apologize for the inconvience. If you have any questions, please contact OSC Help.
We have been experiencing some problems with GPFS filesystems starting from 2:34am, Thursday, June 10. Web portals including OnDemand and WebMO are not working. It may also cause unexpected job failures.
Users may experience performance issues in home directory. It is recommended to use temporary directory ($TMPDIR, or scratch) or project storage to minimize the impact on your jobs.
OSC is currently troubleshooting the cause. Contact email@example.com if there are questions.
We've seen an increase in transient problems that result in compute nodes losing access to the GPFS file systems for ~5 minutes.
Any jobs running on these nodes accessing files on GPFS may have seen errors. GPFS includes /fs/ess, /fs/project and /fs/scratch directories.
If you believe that your job may have been affected by this error, please contact firstname.lastname@example.org
We are currently seeing problems with the home directories at OSC's HPC facility.
Update: The fix was deployed during May 19 Downtime.
Maintenance work on the GPFS servers is scheduled to be performed today, 28 Feb 2020 at 2:00p.m.
Although there is no direct impact expected to services at OSC, there may be short interruptions to storage services.
Please contact OSC Help at email@example.com if you have any questions.
OSC Project and Scratch file systems have resumed normal operations.