Emergency Maintenance: Research Data Storage Scratch Purge
OSC will be conducting an emergency scratch purge on our scratch storage today, starting at 3:00 PM due to low data pool capacity.
filesystem
Intermittent home directory performance issues
Users may experience performance issues in home directory. It is recommended to use temporary directory ($TMPDIR, or scratch) or project storage to minimize the impact on your jobs.
OSC is currently troubleshooting the cause. Contact oschelp@osc.edu if there are questions.
quota exceeded error when using chgrp in /fs/ess directories
Users may receive an error when using the chgrp command on data in /fs/ess/ locations.
$ chgrp -v PEX1234 my-file.txt chgrp: changing group of 'my-file.txt': Disk quota exceeded failed to change group of 'my-file.txt' from PEX0001 to PEX1234
This is a very infrequent issue and OSC have identified a workaround.
Please contact oschelp@osc.edu for assistance with changing group ownership of data.
Symlinks to /fs/project directories were missed
Symlinks to /fs/project directories were missed for a short period of time both on Tuesday afternoon October 11(right after downtime) and Wedenday morning October 12. It might result in job failures. We have fixed it.
Sorry for the disruption. Please contact OSC Help if you have any questions.
GPFS problems with /fs/project and possibly /fs/scratch
There was an issue with GPFS clients that affected /fs/project and possibly /fs/scratch between around 3:30AM and 8:30AM on Sunday September 4th. Some jobs from clients were also impacted.
The systems are working properly now. Sorry for the inconvenience. Please contact OSC Help if you have any questions.
issues accessing /fs/ess/ locations
15 Aug 2022 resolved
Possible performance degradation after August 9th's downtime
Updates on May 20 2023:
verbsRDMA is enabled on Pitzer.
Updates on Dec 14 2022:
verbsRDMA is enabled on Owens during December 13 downtime. verbsRDMA remains disabled on Pitzer.
Original Post:
To prepare for expansion of the IBM Elastic Storage System (ESS), the data transfer is switched from InfiniBand (IB) verbsRDMA to IP over IB (IPoIB) after August 9th's downtime. This may cause some impact to the job performance.
Inconsistent performance degradation of ESS filesystem
We have been experiencing inconsistent performance degradation of ESS filesystem (/fs/ess), starting from 12/22/2021.
Performance degradation of ESS filesystem
The systems are working properly now.
Very little free space for metadata on the scratch storage /fs/scratch
Updated 15:30 October 19:
The issue of little space for metadata on scratch storage is resolved. If you have any questions, please contact OSC Help.
Original Post:
The scratch storage /fs/scratch has very little free space for metadata starting from Saturday, October 16. This may result in the failure of file or directory creations on /fs/scratch.
