Problems with GPFS filesystem, and OnDemand is not working

Updates on 12:00pm June 10:

The issue is fixed. All impacted services return to production.

We apologize for the inconvience. If you have any questions, please contact OSC Help.

Original Post:

We have been experiencing some problems with GPFS filesystems starting from  2:34am, Thursday, June 10. Web portals including OnDemand and WebMO are not working. It may also cause unexpected job failures. 

GPFS errors on compute nodes

We've seen an increase in transient problems that result in compute nodes losing access to the GPFS file systems for ~5 minutes.

Any jobs running on these nodes accessing files on GPFS may have seen errors. GPFS includes /fs/ess, /fs/project and /fs/scratch directories.

If you believe that your job may have been affected by this error, please contact