[Jupyter+Spark] Worker and executor memory update
Date:
Tuesday, August 26, 2025 - 10:45am
System(s):
The Jupyter+Spark app now configures worker memory based on the job's memory limit and the number of workers.
Gateway
New scratch storage policy implemented
The Ohio Supercomputer Center (OSC) has implemented a new scratch filesystem storage policy. The purge period has changed from 90 days inactivity to 60 days inactivity to help alleviate recent storage stress on our scratch filesystem: https://www.osc.edu/resources/security_accessibility_and_policies
[Jupyter+Spark] Allow for choice of Python
Date:
Thursday, July 3, 2025 - 11:30am
System(s):
The Jupyter+Spark app now allows you to choose the Python version. You can specify a Python version that matches the one used in your Conda environment. This ensures compatibility when setting up the Spark cluster and helps prevent issues related to Python version mismatches.
Pitzer upgrade to RHEL9 is complete
The upgrade of the Pitzer cluster to Red Hat Enterprise Linux 9 is complete. Please refer to the [Pitzer documentation](https://www.osc.edu/resources/technical_support/supercomputers/pitzer) for more information about what has changed.
Update to Scratch File Purge Policy
To help alleviate recent storage stress on our scratch filesystem, we will be changing the purge period for scratch files from 90 days to 60 days. This means that any files not accessed within 60 days will be deleted. We invite you to share your comments or concerns during the public comment period, which will remain open until August 3. Please submit your feedback via https://www.osc.edu/resources/technical_support/supercomputing-policies/proposed_osc_policies_for_public_comments.
Action Requested - Free Up Space on OSC's Scratch Filesystem
The OSC Scratch filesystem (/fs/scratch) is currently under significant space pressure. To address this, OSC is preparing a proposed policy update that would reduce the default data retention period on Scratch.
Under the current policy, we may need to implement a one-time emergency reduction in retention time to free up space. To help avoid this, we are asking all clients to *voluntarily delete any unneeded data* from Scratch as soon as possible.
Your cooperation is greatly appreciated and will help maintain system stability for everyone.
System Downtime August 19 2025
A downtime for OSC HPC systems is scheduled from 7 a.m. to 9 p.m., Tuesday, August 19 2025. The downtime will affect the Pitzer, Cardinal and Ascend Clusters, web portals, the state-wide licenses and HPC file servers. MyOSC (the client portal) will be available during the downtime. In preparation for the downtime, the batch scheduler will not start jobs that cannot be completed before 7 a.m., August 19. Jobs that are not started on clusters will be held until after the downtime and then started once the system is returned to production status.
OneDrive Connector File Transfer Issue with Globus
We are currently seeing an issue with the OneDrive Globus Connector. File transfers to the 'My Files' library in OneDrive are failing if the file size exceeds approximately 10MB. Smaller files are unaffected.
Join the Pitzer OS upgrade Early User Program
OSC is upgrading Pitzer from RHEL7 to RHEL9, bringing updated tools and libraries. RHEL7 will no longer be available after the upgrade. Join the Early User Program (no application needed) to test your workflows between June 23 and July 27, 2025 (tentative). See this page for more information: https://www.osc.edu/resources/technical_support/supercomputers/pitzer/pitzer_os_upgrade_and_early_user_program
Instability on Clusters after May 13 Downtime
We've been experiencing some instability on the clusters (particularly Cardinal and Ascend) following the recent May 13 downtime, especially with parallel job processing. If you notice any unusual behavior with your jobs, please contact OSC Help.
