Scratch filesystem is down

Updated on 2:30pm Feb 1st:

Scratch filesystem is back. OnDemand is also available now. 

Original Post:

Scratch filesystem is down now. Users can not access OSC OnDemand as well. Please use other methods to connect to OSC systems. We are working to fix this issue now and apologize for any inconvenience. Please contact OSC Help if you have any questions. 

Stale File Handles on GPFS clients

OSC is experiencing some problems with the Project and Scratch filesystems that are resulting in some jobs seeing "stale file handles". We are investigating the problem and will provide updates as we have them. Job scheduling is currently paused on all filesystems.

We are working with vendor engineers to resolve the issue, but you can currently still log in via SSH and access your files in your home directories. OnDemand logins will likely hang, and attempts to access files on Project or Scratch will likely fail.

Issue with GPFS on Owens since April 14, 2017

3:10PM 4/18/2017 Update: Rolling reboots on Owens have started to address this GPFS issue. 

We have had issues with GPFS mounts on Owens Cluster since Friday afternoon, April 14, 2017. The affected nodes have been marked offline to be restarted or rebooted to fix this issue. Jobs may have been negatively impacted by this issue since April 14. If you experience any 'stale file handle' or file not found errors, please let us know.