1:40PM 4/27/2017 Update: Rolling reboots are completed.
3:10PM 4/18/2017 Update: Rolling reboots on Owens have started to address GPFS errors occured late Friday.
Rolling reboot of Owens, Oakley, and Ruby clusters is scheduled to start from Wednesday morning, April 19, 2017. Highlights of the rolling reboot activities:
1:00PM 4/6/2017 Update: The Scratch and Project file systems are back to normal service. Scheduling on systems are resumed. We are still investigating the causes to this problem and will keep you updated when we know more.
The Scratch and Project file systems are currently hung. Schedulings on all three clusters (Owens, Ruby, and Oakley) have been paused for investigating this problem. We will update this page when we know more.
Starting from Thursday, Feb 2nd, the $PFSDIR directory on scratch (/fs/scratch) won’t be created by job prologue. For example, if you simply use the command cd $PFSDIR, you will get an error indicating that this directory does not exist. The reason we are making this change is to address recent problems with the batch environments on OSC’s clusters. You will have to create the $PFSDIR directory by yourself if you use this directory. Please include the following additional lines in the job script.
If you use bash:
We have noticed some abaqus jobs end up in BatchHold. Once the job is in BatchHold, it will never start. This is because of sharing the abaqus licenses between Oakley and Owens. We have opened a support request with the job scheduler vendor.
Update: Jan 23, 2017 3PM: this issue has been fixed.
There is an issue with OSC OnDemand -> Desktops -> Virtual Desktop Interface (VDI) such that you get "qsub: submit error ..." when you try to launch a VDI session (Oakley, Ruby, or Owens). Please use Desktops -> Interactive HPC instead. We apologize for any inconvenience we may have caused you.
Update Tue Feb 10th 11:30am -- This issue is resolved.
There is a bug in the changes we made to a part of our batch software during the downtime. The bug is affecting some users when they submit jobs to our system.
UPDATE: Regular scheduling on Oakley has been returned to service.
pdsh -j is broken on Oakley. It was broken by updates during the September downtime. We are currently working on resolving the issue.
Users who require a workaround should contact OSC Help for assistance.
Currently, the estimated RU charge reported at the end of a job shows an incorrect value for serial jobs on Oakley of the entire node. Jobs are being charged the correct amount in the official accounting records.
We have temporarily suspended scheduling due to some problems with the parallel scratch file system.
