Symlinks to /fs/project directories were missed for a short period of time both on Tuesday afternoon October 11(right after downtime) and Wedenday morning October 12. It might result in job failures. We have fixed it.
Sorry for the disruption. Please contact OSC Help if you have any questions.
There was an issue with GPFS clients that affected /fs/project and possibly /fs/scratch between around 3:30AM and 8:30AM on Sunday September 4th. Some jobs from clients were also impacted.
The systems are working properly now. Sorry for the inconvenience. Please contact OSC Help if you have any questions.
To prepare for expansion of the IBM Elastic Storage System (ESS), the data transfer is switched from InfiniBand (IB) verbsRDMA to IP over IB (IPoIB) after August 9th's downtime. This may cause some impact to the job performance.
Disabling verbsRDMA is only temporary and we will re-enable it in the next couple months once we're able to upgrade the HPC clusters to a version of the InfiniBand libraries compatible with the ESS upgrades/expansion.
Please contact OSC Help if you see any issue or have any questions.
We have been experiencing inconsistent performance degradation of ESS filesystem (/fs/ess), starting from 12/22/2021.
The systems are working properly now.
Updated 15:30 October 19:
The issue of little space for metadata on scratch storage is resolved. If you have any questions, please contact OSC Help.
Original Post:
The scratch storage /fs/scratch has very little free space for metadata starting from Saturday, October 16. This may result in the failure of file or directory creations on /fs/scratch.
We had several auto-reboots with one of our home directory servers. Jobs might be impacted.
/fs/ess and OnDemand are not accessible now. We are working on this.
Sorry for the inconvenience. Please contact OSC Help if you have any questions.
The backups on the /fs/ess filesystem are having issues running. There has not been a successful backup of this filesystem since Sunday, 08 August 2021.
OSC is working with the vendor to resolve this issue as soon as possible.