3:10PM 4/18/2017 Update: Rolling reboots on Owens have started to address this GPFS issue.
We have had issues with GPFS mounts on Owens Cluster since Friday afternoon, April 14, 2017. The affected nodes have been marked offline to be restarted or rebooted to fix this issue. Jobs may have been negatively impacted by this issue since April 14. If you experience any 'stale file handle' or file not found errors, please let us know.
1:00PM 4/6/2017 Update: The Scratch and Project file systems are back to normal service. Scheduling on systems are resumed. We are still investigating the causes to this problem and will keep you updated when we know more.
The Scratch and Project file systems are currently hung. Schedulings on all three clusters (Owens, Ruby, and Oakley) have been paused for investigating this problem. We will update this page when we know more.
Dec 27, 2016 3:46PM Update: Both project and scratch file systems (/fs/project and /fs/scratch ) are back to normal now. Some users' jobs may be affected by this issue. Please contact OSC Help if you have any questions. We really apologize for any inconvenience this may cause you.
11/01/2016 11:52AM Update: This issue has been fixed.
We have become aware of a problem with the Project storage space that gives errors "No space left on device". The cause is that the metadata subsystem is full. This issue might result in failures of users' jobs.
We are working hard to fix this issue as soon as possible and really apologize for any inconveniences this may cause you.
We experienced a brief interruption of both Project and Scratch filesystems at about 5:15PM October 13, 2016. User jobs may have been effected.
We'll update more details later.
We had GPFS hang issue that caused unexpected job failure between 19:37 and 20:00 on 09/08/2016
OSC experienced errors with NFS services the morning of June 29 between 08:37 and 09:12 that may have caused some jobs to fail, or other unexpected behavior. The errors would have resulted in the failure of legacy paths to user home directories through /nfs/[01-18]. There were also mount failures of /nfs/gpfs on several nodes.
Update: Downtime completed at 6:30PM, June 7th.
The June 7th downtime is now slated to be completed at 6:30PM. Previous estimate was 5PM.
All systems and services will continue to be unavailable until that time.
Thank you for your cooperation.
We are currently investigating multiple reports of Globus Online transfers to/from OSC to other sites are failing. Transfers to/from Globus Personal Endpoints do not seem to be affected.
Please let us know if you exprience issues using Globus.
(9/8/15 14:21 Eastern) Project space appears to be back to normal operation. We are running some tests to verify that the problem is fully resolved.
As of early afternoon, Sept. 8, 2015, we are experiencing some problems with access to /nfs/gpfs. We are aware of the problem, and are working to resolve the issue and return the project space service to normal operation as soon as possible.
