Supercomputing FAQ

General Questions

Account Questions

Disk Storage Questions

Email Questions

Linux Questions

SSH Questions

Batch Processing Questions

Compiling System Questions

Parallel Processing Questions

Libraries/Software Questions

Performance Analysis Questions

Other Common Problems

General Questions

Who can get an account?

Any faculty member or research scientist at an academic institution in Ohio is eligible for an academic account at OSC. These researchers/educators may request accounts for their students and collaborators. Commercial accounts are also available.  More information about applying for both academic and commercial accounts at OSC can be found at https://www.osc.edu/supercomputing/support/account

Do I have to pay for supercomputer use?

OSC receives funding from the state of Ohio. There is no charge for Ohio academic use. 

How many supercomputers does OSC have?  Which one should I use?

OSC currently has two HPC clusters, which are named Oakley and Glenn. New users should choose Oakley unless they require software that is available only on Glenn. Long-time users should consider migrating some or all of their work to Oakley.

Where should a new OSC user begin?

The first thing you should do as a new OSC user is register your ARMSTRONG account using the login information in your welcome letter. You will use ARMSTRONG to change your HPC password, connect with other OSC users, add new users to your projects, and more. To register with your HPC information, visit the ARMSTRONG registration page.

 

After registering with ARMSTRONG your should change your High Performance Computing (HPC) password from the default provided; see "How do I change my password?". Your next step will be connecting to the High Performance Computing (HPC) clusters; see Getting Connected.

 

Once you are able to connect to our HPC systems, you should start familiarizing yourself with the software and services available from the OSC, including:

How do I acknowledge OSC in my publications?

An acknowledgment of support from the Ohio Supercomputer Center should appear in a publication of any material, whether copyrighted or not, based on or developed with OSC-supported computing resources.

 

"This work was supported in part by an allocation of computing time from the Ohio Supercomputer Center."

Can I receive a letter of support from OSC when I apply for outside funding?

OSC has a standard letter of support that you can include (electronically or in hard copy) with a proposal for outside funding. This letter does not replace the application process for time on OSC's systems. To receive the letter of support, please send your request to oschelp@osc.edu. You should provide the following information: name and address of the person/organization to whom the letter should be addressed; name(s) of the principal investigator(s) and the institution(s); title of the proposal; number of years of proposed project; number of RUs requested per year. Please allow at least two working days to process your request.

 

When you receive outside funding related to research you conduct on OSC's systems, please log into the ARMSTRONG portal at https://armstrong.osc.edu/ and add that information in the appropriate section. Such information is important to the center and to state officials.

Hardware information about the systems is available at http://www.osc.edu/supercomputing/hardware

How do I register for a workshop?

For a complete schedule of current training offerings, please visit the OSC Training Schedule.  To register or for more information, please email oschelp@osc.edu.  

Where can I find documentation?

For documentation specific to software applications, see Software.  For other available documentation, see Manuals.

My question isn't answered here. Whom can I ask for help?

Contact the OSC Help Desk. Support is available during regular business hours (9am-5pm Monday through Friday). More information on the OSC supercomputing help desk can be found on our Support Services page.

Something seems to be wrong with the OSC systems.  Should I contact the help desk?

Information will be coming soon for guidelines on reporting possible system problems.

Account Questions

What are projects and accounts?

An eligible principal investigator heads a project. Under a project, authorized users have accounts with credentials that permit users to gain access to the HPC systems. A principal investigator can have more than one project, and authorized users can have more than one account. 

How do I get/renew an account?

For information concerning accounts (i.e., how to apply, who can apply, etc.), see Accounts.

I'm a faculty member.  How do I get accounts for my students?

If an eligible principal investigator is new to OSC, he/she can apply for a start-up request and include the contact information for authorized users. Please refer to http://app.osc.edu/cgi-bin/app/startup for the on-line form. If an eligible principal investigator has a current project, he/she can use the ARMSTRONG portal (https://armstrong.osc.edu) to add authorized users. Authorized users do not have to be located in Ohio.

I'm continuing the research of a student who graduated.  Can I use his/her account?

Yes, but you must have the account transferred into your name. Have your PI send an email to oschelp@osc.edu

I'm working closely with another student.  Can we share an account?

No. Each person using the OSC systems must have his/her own account. Sharing files is possible, even with separate accounts. Ask your PI to request an account for you through the ARMSTRONG portal. Contact oschelp@osc.edu for assistance.

How do I change my password?

You can change your password through the ARMSTRONG portal. Log in at https://armstrong.osc.edu/, pull down the HPC menu, and change your password. Please note that your password must have at least 8 characters. Certain special characters are not permitted, and an error message will appear briefly. Please wait an hour before attempting to log in with the new password.

I want to use csh instead of bash.  How do I change the default shell?

You can change your default shell through the ARMSTRONG portal. Log in at https://armstrong.osc.edu/, pull down the HPC menu, and change your shell. You will need to log off the HPC system and log back on before the change goes into effect. Please note, that it will take about an hour for the changes to be applied.

What is an RU?

An RU is a resource unit. A resource unit is an aggregate measure of the use of CPU, memory, and file storage. For details on charging algorithms, see Charging.

How do I find my account balance?

To see usage and balance information from any system, use the ARMSTRONG portal at https://armstrong.osc.edu/.

NOTE:     Accounting is updated once a day, so the account balance is for the previous day.

How do I get more resources?

To request more resources, the principal investigator must prepare a proposal that will be sent for peer review. After the review process is complete, the Allocations Committee of the Statewide Users Group meets to determine the number of resource units to be awarded. The committee meets on the second Thursday of even-numbered months. PIs should submit their proposals at least three weeks before the meeting date to ensure inclusion on the agenda.

 

The application form is located at http://app.osc.edu/cgi-bin/app/. A complete application comprises the following items:

 

  • Contact information of new authorized users
  • Contact information of at least three recommended reviewers
  • Electronic version of PI’s resume
  • Publications list (publications, presentations, and articles that acknowledge OSC)
  • Justification and performance (quantitative estimation of the number of RUs per run times; the number of runs for the estimated duration of the request; performance reports of the code to be used or explanations about performance; report of any optimization performed on the code
  • Proposal text

How much will my account be charged for supercomputer usage?

For details on charging algorithms, see Charging.

Disk Storage Questions

What is my disk quota?

Each user has a quota of 500 gigabytes (GB) of storage and 1,000,000 files. You may also have access to a project directory with a separate quota.   See Available File Systems for more information.

How can I determine the total disk space used by my account?

Your quota and disk usage are displayed every time you log in. You have limits on both the amount of space you use and the number of files you have. There are separate quotas for your home directory and any project directories you have access to.

Note: The quota information displayed at login is updated once a day, so the usage is for the previous day.

You may display your home directory quota information with quota -s.

How do I get more disk space?

Your home directory quota cannot be increased. You should consider deleting, transferring, and/or compressing your files to reduce your usage. 

 

A PI may request project space to be shared by all users on a project. Estimate the amount of disk space that you will need and the duration that you will need it. Send requests to oschelp@osc.edu.

 

See the Allocations and Accounts section for more information. 

How can I find my largest directories?

To reveal the directories in your account that are taking up the most disk space you can use the du, sort and tail commands. For example, to display the ten largest directories, change to your home directory and then run the command:

 

du . | sort -n | tail -n 10

How can I use tar and gzip to aggregate and compress my files?

The commands tar and gzip can be used together to produce compressed file archives representing entire directory structures. These allow convenient packaging of entire directory contents. For example, to package a directory structure rooted at src/ use

 

tar -czvf src.tar.gz src/

 

This archive can then be unpackaged using

 

tar -xzvf src.tar.gz

 

where the resulting directory/file structure is identical to what it was initially.

The programs zip, bzip2 and compress can also be used to create compressed file archives. See the man pages on these programs for more details.

Email Questions

How do I change the email address OSC uses to contact me?

Send your new contact information to oschelp@osc.edu. Note that changing your email address on ARMSTRONG does not change it for the HPC systems.

I got an automated email from OSC.  Where can I get more information about it?

See the Knowledge Base.

Linux Questions

What is Linux?

Linux is an open-source operating system that is similar to UNIX. It is widely used in High Performance Computing.

How can I get started using Linux?

See the Unix Basics tutorial for more information.  There are also many tutorials available on the web.

SSH Questions

What is SSH?

Secure Shell (SSH) is a program to log into another computer over a network, to execute commands in a remote machine, and to move files from one machine to another. It provides strong authentication and secure communications over insecure channels. SSH provides secure X connections and secure forwarding of arbitrary TCP connections.

How does SSH work?

SSH works by the exchange and verification of information, using public and private keys, to identify hosts and users. The ssh-keygen command creates a directory ~/.ssh and files that contain your authentication information, The public key is stored in ~/.ssh/identity.pub and the private key is stored in ~/.ssh/identityShare only your public key. Never share your private key. To further protect your private key you should enter a passphrase to encrypt the key when it is stored in the file system. This will prevent people from using it even if they gain access to your files. 

One other important file is ~/.ssh/authorized_keys. Append your public keys to the authorized_keys file and keep the same copy of it on each system where you will make ssh connections. 

Can I connect without using an SSH client?

The OSC OnDemand portal allows you to connect to Oakley and Glenn using your web browser, without having to install any software. You get a login shell and also the ability to transfer files.  

How can I upload or download files?

Most file transfers are done using sftp (SSH File Transfer Protocol) or scp (Secure CoPy). These utilities are usually provided on Linux/UNIX and Mac platforms. Windows users should read the next section, "Where can I find SSH and SFTP clients".

Where can I find SSH and SFTP clients?

There are many SSH and SFTP clients available, both commercial and free.  See Getting Connected for some suggestions.  

How do I run a graphical application in an SSH session?

Graphics are handled using the X11 protocol. You’ll need to run an X display server on your local system and also set your SSH client to forward (or "tunnel") X11 connections. On most UNIX and Linux systems, the X server will probably be running already. On a Mac or Windows system, there are several choices available, both commercial and free. See our guide to Getting Connected for some suggestions.

Batch Processing Questions

What is a batch request?

On all OSC systems, batch processing is managed by the Portable Batch System (PBS). PBS batch requests (jobs) are shell scripts that contain the same set of commands that you enter interactively. These requests may also include options for the batch system that provide timing, memory, and processor information. For more information, see our guide to Batch Processing at OSC.  

How do I submit, check the status, and/or delete a batch job?

PBS uses qsub to submit, qstat -a to check the status, and qdel to delete a batch request. For more information, see our Batch-Related Command Summary

Can I be notified by email when my batch job starts or ends?

Yes.  See the -m option in our PBS Directives Summary. If you are submitting a large number of jobs, this may not be a good idea.

Why won't my job run?

There are numerous reasons why a job might not run even though there appear to be processors and/or memory available. These include:

  • Your account may be at or near the job count or processor count limit for an individual user.
  • Your group/project may be at or near the job count or processor count limit for a group.
  • The scheduler may be trying to free enough processors to run a large parallel job.
  •  Your job may need to run longer than the time left until the start of a scheduled downtime.
  • You may have requested a scarce resource or node type, either inadvertently or by design.

See our Scheduling Policies and Limits for more information. 

How can I retrieve files from unexpectedly terminated jobs?

A batch job that terminates before the script is completed can still copy files from $TMPDIR to the user's home directory via the trap command. In the batch script, the trap command needs to precede the command causing the TERMination. It could be placed immediately after the PBS header lines. Here is a generic form:

 

trap "cd $PBS_O_WORKDIR;mkdir $PBS_JOBID;cp -R $TMPDIR/* $PBS_JOBID" TERM

Compiling System Questions

What languages are available?

Fortran, C, and C++ are available on all OSC systems. The commands used to invoke the compilers and/or loaders vary from system to system. For more information, see OpenMP.

What compiler (vendor) do you recommend?

We have Intel, PGI, and gnu compilers available on both systems. We typically recommend the Intel compilers on Oakley and either PGI or Intel on Glenn. Each compiler vendor supports some options that the other doesn’t, so the choice depends on your individual needs.

Will software built for Glenn run on Oakley?

Most serial code built on one system will run on the other, although it may run more efficiently if it is built and run on the same system. Parallel (MPI) code typically must be built on the system where it will run.

Parallel Processing Questions

What is parallel processing?

Parallel processing is the simultaneous use of more than one computer (or processor) to solve a problem. There are many different kinds of parallel computers. They are distinguished by the kind of interconnection between processors or nodes (groups of processors) and between processors and memory.

What parallel processing environments are available?

On most systems, both shared-memory and distributed-memory parallel programming models can be used. Versions of OpenMP (for multithreading or shared-memory usage) and MPI (for message-passing or distributed-memory usage) are available.  A summary of parallel environments will be coming soon.

What is a core?

A core is a processor. When a single chip contains multiple processors, they are called cores.

I'm not seeing the performance I expected.  How can I be sure my code is running in parallel?

We are currently working on a guide for this. Please contact OSC Help for assistance.

Libraries/Software Questions

What software applications are available?

See the Software section for more information.

Do you have a newer version of (name your favorite software)?

Check the Software section to see what versions are installed. You can also check the installed modules using the module spider command on Oakley or the module avail command on Glenn.

How do I get authorized to use a particular software application?

Access Agreement forms are available on the Web in pdf format. Where appropriate, they may be found for a given application in the Software section.

What math routines are available?  Do you have ATLAS and LAPACK?

See the Software section for information on third-party math libraries (e.g., MKL, ACML, fftw, scalapack, etc). MKL and ACML are highly optimized libraries that include the BLAS and LAPACK plus some other math routines. We recommend them instead of ATLAS. MKL is recommended on Oakley; ACML is recommended on Glenn, although MKL is available with the Intel compilers.

Do you have NumPy/SciPy?

The NumPy and SciPy modules are installed with the python software. See the Python software page.

OSC does not have a particular software package I would like to use. How can I request it?

Please refer to the Software Forms page. You will see a link to Request for Software Form. Download the form, print, complete the information, scan, and attach the form to e-mail to oschelp@osc.edu. The Statewide Users Group will consider the request.

 

You may install open source software yourself in your home directory. If you have your own license for commercial software, contact the OSC Help desk.

I have a software package that must be installed as root.  What should I do?

Most packages have a (poorly documented) option to install under a normal user account. Contact the OSC Help desk if you need assistance. We generally do not install user software as root.

What are modules?

Modules are used to manage the environment variable settings associated with software packages in a shell-independent way. On OSC's systems, you will by default have modules in your environment for PBS, MPI, compilers, and a few other pieces of software. For information on using the module system, see our guide to Batch Processing at OSC.

Performance Analysis Questions

What are MFLOPS/GFLOPS/TFLOPS/PFLOPS?

MegaFLOPS/GigaFLOPS/TeraFLOPS/PetaFLOPS are millions/billions/trillions/quadrillions of FLoating-point Operations (calculations) Per Second.

How do I find out about my code's performance?

A number of performance analysis tools are available on OSC systems. Some are general to all systems and others are specific to a particular system. See our performance analysis guide for more info.

How can I optimize my code?

There are several ways to optimize code. Key areas to consider are CPU optimization, I/O optimization, memory optimization, and parallel optimization. See our optimization strategy guide for more info.

Other Common Problems

What does "CPU time limit exceeded" mean?

Programs run on the login nodes are subject to strict CPU time limits. To run an application that takes more time, you need to create a batch request. Your batch request should include an appropriate estimate for the amount of time that your application will need. See our guide to Batch Processing at OSC for more information.

My program or file transfer died for no reason after 20 minutes.  What happened?

Programs run on the login nodes are subject to strict CPU time limits. Because file transfers use encryption, you may hit this limit when transferring a large file. To run longer programs, use the batch system. To transfer larger files, connect to gridftp01.osc.edu instead of to a login node.

Why did my program die with a segmentation fault, address error, or signal 11?

This is most commonly caused by trying to access an array beyond its bounds -- for example, trying to access element 15 of an array with only 10 elements. Unallocated arrays and invalid pointers are other causes. You may wish to debug your program using one of the available tools such as the TotalView Debugger.

I created a batch script in a text editor on a Windows or Mac system, but when I submit it on an OSC system, almost every line in the script gives an error. Why is that?

Windows and Mac have different end-of-line conventions for text files than UNIX and Linux systems do, and most UNIX shells (including the ones interpreting your batch script) don't like seeing the extra character that Windows appends to each line or the alternate character used by Mac. You can use the following commands on the Linux system to convert a text file from Windows or Mac format to UNIX format:

 

dos2unix myfile.txt 

 

mac2unix myfile.txt  

I copied my output file to a Windows system, but it doesn't display correctly.  How can I fix it?

A text file created on Linux/UNIX will usually display correctly in Wordpad but not in Notepad. You can use the following command on the Linux system to convert a text file from UNIX format to Windows format:

 

unix2dos myfile.txt