- Who can get an account?
- Where should a new OSC user begin?
- How do I register for the Armstrong administration portal (for PIs)?
- Do I have to pay for supercomputer use?
- How many supercomputers does OSC have? Which one should I use?
- How do I acknowledge OSC in my publications?
- Can I receive a letter of support from OSC when I apply for outside funding?
- How do I register for a workshop?
- Where can I find documentation?
- My question isn't answered here. Whom can I ask for help?
- Something seems to be wrong with the OSC systems. Should I contact the help desk?
- What are projects and accounts?
- How do I get/renew an account?
- I'm a faculty member. How do I get accounts for my students?
- I'm continuing the research of a student who graduated. Can I use his/her account?
- I'm working closely with another student. Can we share an account?
- How do I change my password?
- I want to use csh instead of bash. How do I change the default shell?
- What is an RU?
- How do I find my account balance?
- How do I get more resources?
- How much will my account be charged for supercomputer usage?
Disk Storage Questions
- What is my disk quota?
- How can I determine the total disk space used by my account?
- How do I get more disk space?
- How can I find my largest directories?
- How can I use tar and gzip to aggregate and compress files?
- Tar is taking too long. Is there a way to compress quicker?
- How do I change the email address OSC uses for me?
- I got an automated email from OSC. Where can I get more information about it?
- What is SSH?
- How does SSH work?
- Can I connect without using an SSH client?
- How can I upload or download files?
- Where can I find SSH and SFTP clients?
- How do I run a graphical application in an ssh session?
Batch Processing Questions
- What is a batch job?
- How do I submit, check the status, and/or delete a batch job?
- Can I be notified by email when my batch job starts or ends?
- Why won't my job run?
- Why is my job being rejected with the error "Group: is not valid"?
- How can I retrieve files from unexpectedly terminated jobs?
- How can I delete all of my jobs on a cluster?
Compiling System Questions
- What languages are available?
- What compiler (vendor) do you recommend?
- Will software built for Glenn run on Oakley?
Parallel Processing Questions
- What is a parallel processing?
- What parallel processing environments are available?
- What is a core?
- I'm not seeing the performance I expected. How can I be sure my code is running in parallel?
- What software applications are available?
- Do you have a newer version of (name your favorite software)?
- How do I get authorized to use a particular software application?
- What math routines are available? Do you have ATLAS and LAPACK?
- Do you have NumPy/SciPy?
- OSC does not have a particular software package I would like to use. How can I request it?
- I have a software package that must be installed as root. What should I do?
- What are modules?
Performance Analysis Questions
- What are MFLOPS/GFLOPS/TFLOPS/PFLOPS?
- How do I find out about my code's performance?
- How can I optimize my code?
Other Common Problems
- What does "CPU time limit exceeded" mean?
- My program or file transfer died for no reason after 20 minutes. What happened?
- Why did my program die with a segmentation fault, address error, or signal 11?
- I created a batch script in a text editor on a Windows system, but when I submit it on an OSC system, almost every line in the script gives an error. Why is that?
- I copied my output file to a Windows system, but it doesn't display correctly. How can I fix it?
Any faculty member or research scientist at an academic institution in Ohio is eligible for an academic account at OSC. These researchers/educators may request accounts for their students and collaborators. Commercial accounts are also available. More information about applying for both academic and commercial accounts at OSC can be found at https://www.osc.edu/supercomputing/support/account.
The first thing you should do as a new OSC user is to check the email account that was given to OSC when you were registered. If you are a PI, this will be the email address you entered in the primary contact information on your Project Request application. When your account is created, you will receive an email giving you information on how to start using the services at OSC. Attached will be a letter you can keep for reference that contains your username and temporary password, as well as some other important information about using the systems at OSC. After carefully reading the email, please first go to my.osc.edu to change your temporary password.
Once you are able to connect to our HPC systems, you should start familiarizing yourself with the software and services available from the OSC, including:
OSC supports the ARMSTRONG portal for administering your users and monitoring your project balance. ARMSTRONG is located at https://armstrong.osc.edu.You do not need to create your ARMSTRONG account if you choose not to use the features. However, if you wish to use ARMSTRONG, you must first register. You must register by clicking on the link in the paragraph to the right of the login information. Your ARMSTRONG userid and password may differ from those of your HPC accounts.
Password and shell changes have been moved to My OSC (my.osc.edu), and other features of Armstrong will be migrated over in the future as well. Please refer to the FAQ for future information regarding the role of My OSC and Armstrong at OSC.
OSC receives funding from the state of Ohio. There is no charge for Ohio academic use.
OSC currently has two HPC clusters, which are named Oakley and Glenn. New users should choose Oakley unless they require software that is available only on Glenn. Long-time users should consider migrating some or all of their work to Oakley.
Any publication of any material, whether copyrighted or not, based on or developed with OSC services, should cite the use of OSC, and the use of the specific services (where applicable). For more information about citing OSC, please visit www.osc.edu/citation.
OSC has a standard letter of support that you can include (electronically or in hard copy) with a proposal for outside funding. This letter does not replace the application process for time on OSC's systems. To receive the letter of support, please send your request to firstname.lastname@example.org. You should provide the following information: name and address of the person/organization to whom the letter should be addressed; name(s) of the principal investigator(s) and the institution(s); title of the proposal; number of years of proposed project; number of RUs requested per year. Please allow at least two working days to process your request.
Hardware information about the systems is available at http://www.osc.edu/supercomputing/hardware.
Contact the OSC Help Desk. Support is available 24x7x365, but more complicated questions will need to wait for regular business hours (Monday - Friday, 9am - 5pm) to be addressed. More information on the OSC supercomputing help desk can be found on our Support Services page.
Information will be coming soon for guidelines on reporting possible system problems.
An eligible principal investigator heads a project. Under a project, authorized users have accounts with credentials that permit users to gain access to the HPC systems. A principal investigator can have more than one project, and authorized users can have more than one account.
For information concerning accounts (i.e., how to apply, who can apply, etc.), see Accounts.
If an eligible principal investigator is new to OSC, he/she can apply for a start-up request and include the contact information for authorized users. Please refer to http://app.osc.edu/cgi-bin/app/startup for the on-line form. If an eligible principal investigator has a current project, he/she can use the ARMSTRONG portal (https://armstrong.osc.edu) (or email OSC Help) to add authorized users. When emailing OSC Help, please include the student's full name, email address, and your project number. Authorized users do not have to be located in Ohio.
Yes, but you must have the account transferred into your name. Have your PI send an email to email@example.com.
No. Each person using the OSC systems must have his/her own account. Sharing files is possible, even with separate accounts. Ask your PI to request an account for you through the ARMSTRONG portal (alternatively, PI's can email student information to OSC Help for this purpose). Contact firstname.lastname@example.org for assistance.
You can change your password through the My OSC portal. Log in at http://my.osc.edu, and click the "change password" button at the bottom left corner of the HPC User Profile box. Please note that your password has certain requirements; these are specified on the "change password" portal. Please wait an hour before attempting to log in with the new password. For security purposes, please note that our password change policy requires a password change every 90 days.
You can change your default shell through the My OSC portal. Log in at my.osc.edu, and use the "Unix Shell" drop-down menu in the HPC User Profile box to change your shell. You will need to log off the HPC system and log back on before the change goes into effect. Please note, that it will take about an hour for the changes to be applied.
An RU is a resource unit. A resource unit is an aggregate measure of the use of CPU, memory, and file storage. For details on charging algorithms, see Charging.
To see usage and balance information from any system, use the ARMSTRONG portal at https://armstrong.osc.edu/.NOTE: Accounting is updated once a day, so the account balance is for the previous day.
To request more resources, the principal investigator must prepare a proposal that will be sent for peer review. After the review process is complete, the Allocations Committee of the Statewide Users Group meets to determine the number of resource units to be awarded. The committee meets on the second Thursday of even-numbered months. PIs should submit their proposals at least three weeks before the meeting date to ensure inclusion on the agenda.
The application form is located at http://app.osc.edu/cgi-bin/app/. A complete application comprises the following items:
- Contact information of new authorized users
- Contact information of at least three recommended reviewers
- Electronic version of PI’s resume
- Publications list (publications, presentations, and articles that acknowledge OSC)
- Justification and performance (quantitative estimation of the number of RUs per run times; the number of runs for the estimated duration of the request; performance reports of the code to be used or explanations about performance; report of any optimization performed on the code
- Proposal text
For details on charging algorithms, see Charging.
Disk Storage Questions
Each user has a quota of 500 gigabytes (GB) of storage and 1,000,000 files. You may also have access to a project directory with a separate quota. See Available File Systems for more information.
Your quota and disk usage are displayed every time you log in. You have limits on both the amount of space you use and the number of files you have. There are separate quotas for your home directory and any project directories you have access to.Note: The quota information displayed at login is updated once a day, so the usage is for the previous day.
You may display your home directory quota information with
A PI may request project space to be shared by all users on a project. Estimate the amount of disk space that you will need and the duration that you will need it. Send requests to email@example.com.
See the Allocations and Accounts section for more information.
To reveal the directories in your account that are taking up the most disk space you can use the
tailcommands. For example, to display the ten largest directories, change to your home directory and then run the command:
du . | sort -n | tail -n 10
gzipcan be used together to produce compressed file archives representing entire directory structures. These allow convenient packaging of entire directory contents. For example, to package a directory structure rooted at
tar -czvf src.tar.gz src/
This archive can then be unpackaged using
tar -xzvf src.tar.gz
where the resulting directory/file structure is identical to what it was initially.
compresscan also be used to create compressed file archives. See the
manpages on these programs for more details.
tarwith the options
zcvfis taking too long you can instead use
pigzin conjunction with tar.
gzipcompression while taking advantage of multiple cores.tar cvf - paths-to-archive | pigz > archive.tgz
pigz defaults to using eight cores, but you can have it use more or less with the -p argument.tar cvf - paths-to-archive | pigz -n 4 > archive.tgzDue to the parallel nature of pigz, if you are using it on a login node you should limit it to using 2 cores. If you would like to use more cores you need to submit either an interactive or batch job to the queue and do the compression from within the job.
pigz does not significantly improve decompression time.
Send your new contact information to firstname.lastname@example.org. Note that changing your email address on ARMSTRONG does not change it for the HPC systems.
See the Knowledge Base.
Linux is an open-source operating system that is similar to UNIX. It is widely used in High Performance Computing.
See the Unix Basics tutorial for more information. There are also many tutorials available on the web.
Secure Shell (SSH) is a program to log into another computer over a network, to execute commands in a remote machine, and to move files from one machine to another. It provides strong authentication and secure communications over insecure channels. SSH provides secure X connections and secure forwarding of arbitrary TCP connections.
SSH works by the exchange and verification of information, using public and private keys, to identify hosts and users. The ssh-keygen command creates a directory ~/.ssh and files that contain your authentication information, The public key is stored in ~/.ssh/identity.pub and the private key is stored in ~/.ssh/identity. Share only your public key. Never share your private key. To further protect your private key you should enter a passphrase to encrypt the key when it is stored in the file system. This will prevent people from using it even if they gain access to your files.
One other important file is ~/.ssh/authorized_keys. Append your public keys to the authorized_keys file and keep the same copy of it on each system where you will make ssh connections.
The OSC OnDemand portal allows you to connect to Oakley and Glenn using your web browser, without having to install any software. You get a login shell and also the ability to transfer files.
Most file transfers are done using sftp (SSH File Transfer Protocol) or scp (Secure CoPy). These utilities are usually provided on Linux/UNIX and Mac platforms. Windows users should read the next section, "Where can I find SSH and SFTP clients".
There are many SSH and SFTP clients available, both commercial and free. See Getting Connected for some suggestions.
Graphics are handled using the X11 protocol. You’ll need to run an X display server on your local system and also set your SSH client to forward (or "tunnel") X11 connections. On most UNIX and Linux systems, the X server will probably be running already. On a Mac or Windows system, there are several choices available, both commercial and free. See our guide to Getting Connected for some suggestions.
Batch Processing Questions
On all OSC systems, batch processing is managed by the Portable Batch System (PBS). PBS batch requests (jobs) are shell scripts that contain the same set of commands that you enter interactively. These requests may also include options for the batch system that provide timing, memory, and processor information. For more information, see our guide to Batch Processing at OSC.
qstat -ato check the status, and
qdelto delete a batch request. For more information, see our Batch-Related Command Summary.
Yes. See the
-moption in our PBS Directives Summary. If you are submitting a large number of jobs, this may not be a good idea.
There are numerous reasons why a job might not run even though there appear to be processors and/or memory available. These include:
- Your account may be at or near the job count or processor count limit for an individual user.
- Your group/project may be at or near the job count or processor count limit for a group.
- The scheduler may be trying to free enough processors to run a large parallel job.
- Your job may need to run longer than the time left until the start of a scheduled downtime.
- You may have requested a scarce resource or node type, either inadvertently or by design.
See our Scheduling Policies and Limits for more information.
If you are a part of a annual allocations project, usually designated by the prefix PAA-, in addition to another project you must explicitly specify which project you want to charge a jobs usage to. This can be done with the
-APBS directive. For example if a user was a part of both projects PAA0999 and PAQ0343 and wanted to charge a job to PAA0999 they would need to add the following line to their script:#PBS -A PAA0999
A batch job that terminates before the script is completed can still copy files from
$TMPDIRto the user's home directory via the
trapcommand. In the batch script, the
trapcommand needs to precede the command causing the TERMination. It could be placed immediately after the PBS header lines. Here is a generic form:
trap "cd $PBS_O_WORKDIR;mkdir $PBS_JOBID;cp -R $TMPDIR/* $PBS_JOBID" TERM
To delete all your jobs on one of the clusters, including those currently running, queued, and in hold, login to the cluster and run the command:qselect -u <username> | xargs qdel
Compiling System Questions
Fortran, C, and C++ are available on all OSC systems. The commands used to invoke the compilers and/or loaders vary from system to system. For more information, see our Compilation Guide.
We have Intel, PGI, and gnu compilers available on both systems. We typically recommend the Intel compilers on Oakley and either PGI or Intel on Glenn. Each compiler vendor supports some options that the other doesn’t, so the choice depends on your individual needs.
Most serial code built on one system will run on the other, although it may run more efficiently if it is built and run on the same system. Parallel (MPI) code typically must be built on the system where it will run.
Parallel Processing Questions
Parallel processing is the simultaneous use of more than one computer (or processor) to solve a problem. There are many different kinds of parallel computers. They are distinguished by the kind of interconnection between processors or nodes (groups of processors) and between processors and memory.
On most systems, both shared-memory and distributed-memory parallel programming models can be used. Versions of OpenMP (for multithreading or shared-memory usage) and MPI (for message-passing or distributed-memory usage) are available. A summary of parallel environments will be coming soon.
A core is a processor. When a single chip contains multiple processors, they are called cores.
We are currently working on a guide for this. Please contact OSC Help for assistance.
See the Software section for more information.
Check the Software section to see what versions are installed. You can also check the installed modules using the
module spidercommand on Oakley or the
module availcommand on Glenn.
Access Agreement forms are available on the Web in pdf format. Where appropriate, they may be found for a given application in the Software section.
See the Software section for information on third-party math libraries (e.g., MKL, ACML, fftw, scalapack, etc). MKL and ACML are highly optimized libraries that include the BLAS and LAPACK plus some other math routines. We recommend them instead of ATLAS. MKL is recommended on Oakley; ACML is recommended on Glenn, although MKL is available with the Intel compilers.
The NumPy and SciPy modules are installed with the python software. See the Python software page.
Please refer to the Software Forms page. You will see a link to Request for Software Form. Download the form, complete the information, and attach the form to an e-mail to email@example.com. The Statewide Users Group will consider the request.
You may install open source software yourself in your home directory. If you have your own license for commercial software, contact the OSC Help desk.
Most packages have a (poorly documented) option to install under a normal user account. Contact the OSC Help desk if you need assistance. We generally do not install user software as root.
Modules are used to manage the environment variable settings associated with software packages in a shell-independent way. On OSC's systems, you will by default have modules in your environment for PBS, MPI, compilers, and a few other pieces of software. For information on using the module system, see our guide to Batch Processing at OSC.
Performance Analysis Questions
MegaFLOPS/GigaFLOPS/TeraFLOPS/PetaFLOPS are millions/billions/trillions/quadrillions of FLoating-point Operations (calculations) Per Second.
A number of performance analysis tools are available on OSC systems. Some are general to all systems and others are specific to a particular system. See our performance analysis guide for more info.
There are several ways to optimize code. Key areas to consider are CPU optimization, I/O optimization, memory optimization, and parallel optimization. See our optimization strategy guide for more info.
Other Common Problems
Programs run on the login nodes are subject to strict CPU time limits. To run an application that takes more time, you need to create a batch request. Your batch request should include an appropriate estimate for the amount of time that your application will need. See our guide to Batch Processing at OSC for more information.
Programs run on the login nodes are subject to strict CPU time limits. Because file transfers use encryption, you may hit this limit when transferring a large file. To run longer programs, use the batch system. To transfer larger files, connect to gridftp01.osc.edu instead of to a login node.
This is most commonly caused by trying to access an array beyond its bounds -- for example, trying to access element 15 of an array with only 10 elements. Unallocated arrays and invalid pointers are other causes. You may wish to debug your program using one of the available tools such as the TotalView Debugger.
I created a batch script in a text editor on a Windows or Mac system, but when I submit it on an OSC system, almost every line in the script gives an error. Why is that?
Windows and Mac have different end-of-line conventions for text files than UNIX and Linux systems do, and most UNIX shells (including the ones interpreting your batch script) don't like seeing the extra character that Windows appends to each line or the alternate character used by Mac. You can use the following commands on the Linux system to convert a text file from Windows or Mac format to UNIX format:
A text file created on Linux/UNIX will usually display correctly in Wordpad but not in Notepad. You can use the following command on the Linux system to convert a text file from UNIX format to Windows format: