Getting Started

Getting Started icon

Welcome to OSC! If you are new to supercomputing, new to OSC, or simply interested in getting an account (if you don't already have one), we have some resources to help you.

New User Resource Guide

Welcome to OSC! This document provides some useful links to get you started.

How to request an account: For information on getting an account and an allocation with us, please visit

            https://www.osc.edu/supercomputing/support/account

Getting connected: Logging in to OSC resources is primarily accomplished via SSH (or “secure shell”). This is a method of getting a command line interface on our clusters, which run a version of the Linux operating system. For more details on how to connect to our systems, please visit

            http://www.osc.edu/supercomputing/getting-started/getting-connected

If that sounds too intimidating, and you would like to try a zero-client web-browser approach to using OSC, you can log in to our web portal, OnDemand:

            https://ondemand.osc.edu/

For an introduction to the features of OnDemand, check out

            https://www.osc.edu/resources/online_portals/ondemand

Transferring Files: SFTP (Secure File Transfer Protocol) is the primary method of moving files to and from OSC storage systems. SCP (Secure CoPy) is another common method. Small files can be moved to and from OSC systems inside of OnDemand. Moving large files to or from OSC via the login nodes can experience difficulty, since there are compute restrictions on those nodes, so the recommendation is to connect to sftp.osc.edu, or scp.osc.edu. If you have a large amount of data to move, you can schedule transfers via Globus Online.

Submitting Jobs: Instructions to be executed by the node(s) assigned to you when your request for resources can be met by the scheduler – can get very complicated, so it’s best to refer to the documentation for the software you want to use to get started.

            http://www.osc.edu/supercomputing/software/

We also have a more detailed guide on the batch system that we recommend new users read. It provides information about scheduling, job submission and how to write scripts to accomplish various tasks.

            http://www.osc.edu/supercomputing/batch-processing-at-osc

Account Maintenance and Usage: Please use MyOSC to update your password, shell, and check usage on the project. The command OSCusage will also provide usage details.

Training: To get an introduction to High Performance Computing, please read our HPC Basics page.

            https://www.osc.edu/resources/getting_started/hpc_basics

If you want to learn more about using the command line, we have some useful resources on UNIX Basics.

            https://www.osc.edu/documentation/tutorials/unix-basics

For detailed instructions on how to perform some important tasks on our systems, check out our HOWTO articles.

            https://www.osc.edu/resources/getting_started/howto

Need Help? Support can be found by contacting OSC Help weekdays, 9AM to 5PM (1-800-686-6472 or 614-292-1800). 
Fill out a request online.  You can also schedule a consultation (remote or in-person) with our staff. For remote consultations, please include "REMOTE" in the request.

We regularly maintain our webpages to show relevant events at the center (including training) and system notices on our main page (osc.edu). We also provide important information in the “message of the day” (visible when you log in). Furthermore, you can receive notices by following @HPCNotices on Twitter. 

More Useful Links:

FAQ: http://www.osc.edu/supercomputing/faq

Main Supercomputing pages: https://www.osc.edu/services/cluster_computing

New User Presenation Slides: https://www.osc.edu/~kcahill/NewUser

 

Documentation Attachment: 
Supercomputer: 

HPC Basics

Overview

Using the HPC or “supercomputer” is a little different from running programs on your desktop. When you login you’ll be connected to one of the system’s “login nodes”. These nodes serve as a staging area for you to marshal your data and submit jobs to the batch scheduler. Your job will then wait in a queue along with other researchers' jobs. Once the resources it requires become available, the batch scheduler will then run your job on a subset of our hundreds of “compute nodes”. You can see the overall structure in the diagram below.

 

Diagram: Several connected parts illustrating the layout of an OSC cluster. Users connect to one of a few "login nodes", which in turn connect to the "batch system", which runs jobs on a subset of the "compute nodes". The "shared filesystem" is connected to both the login nodes and the compute nodes.

 

HPC Citizenship

An important point about the diagram above is that OSC clusters are a collection of shared, finite resources. When you connect to the login nodes, you are sharing their resources (CPU cycles, memory, disk space, network bandwidth, etc.) with a few dozen other researchers. The same is true of the file servers when you access your home or project directories, and can even be true of the compute nodes.

For most day-to-day activities you should not have to worry about this, and we take precautions to limit the impact that others might have on your experience. That said, there are a few use cases that are worth watching out for:

  • The login nodes should only be used for light computation; any CPU- or memory-intensive operations should be done using the batch system. A good rule of thumb is that if you wouldn't want to run a task on your personal desktop because it would slow down other applications, you shouldn't run it on the login nodes. (See also: Interactive Jobs.)

  • I/O-intensive jobs should copy their files to fast, temporary storage, such as the local storage allocated to jobs or the Scratch parallel filesystem.

  • When running memory-intensive or potentially unstable jobs, we highly recommend requesting whole nodes.  By doing so you prevent other users jobs from being impacted by your job.

  • If you request partial nodes, be sure to consider the amount of memory available per core. (See: HPC Hardware.) If you need more memory, request more cores. It is perfectly acceptable to leave cores idle in this situation; memory is just as valuable a resource as processors.

In general, we just encourage our users to remember that what you do may affect other researchers on the system. If you think something you want to do or try might interfere with the work of others, we highly recommend that you contact us at oschelp@osc.edu.

Getting Connected

There are two ways to connect to our systems. The traditional way will require you to install some software locally on your machine, including an SSH client, SFTP client, and optionally an X Windows server. The alternative is to use our zero-client web portal, OnDemand.

OnDemand Web Portal

OnDemand is our "one stop shop" for access to our High Performance Computing resources. With OnDemand, you can upload and download files, create, edit, submit, and monitor jobs, run GUI applications, and connect via SSH, all via a web broswer, with no client software to install and configure.

You can access OnDemand by pointing a web browser to ondemand.osc.edu. Documentation is available here. Any newer version of a common web brower should be sufficient to connect.

Using Traditional Clients

Required Software

In order to use our systems, you'll need two main pieces of software: an SFTP client and an SSH client.

SFTP ("SSH File Transfer Protocol") clients allow you transfer files between your workstation and our shared filesystem in a secure manner. We recommend the following applications:

  • FileZilla: A high-performance open-source client for Windows, Linux, and OS X. A guide to using FileZilla is available here (external).
  • CyberDuck: A high quality free client for Windows and OS X.
  • sftp: The command-line utility sftp comes pre-installed on OS X and most Linux systems.

SSH ("Secure Shell") clients allow you to open a command-line-based "terminal session" with our clusters. We recommend the following options:

  • PuTTY: A simple, open-source client for Windows.
  • Secure Shell for Google Chrome: A free, HTML5-based SSH client for Google Chrome.
  • ssh: The command-line utility ssh comes pre-installed on OS X and most Linux systems.

A third, optional piece of software you might want to install is an X Windows server, which will be necessary if you want to run graphical, windowed applications like MATLAB. We recommend the following X Windows servers:

  • Xming: Xming offers a free version of their X Windows server for Microsoft Windows systems.
  • X-Win32: StarNet's X-Win32 is a commercial X Windows server for Microsoft Windows systems. They offer a free, thirty-day trial.
  • X11.app/XQuartz: X11.app, an Apple-supported version of the open-source XQuartz project, is freely available for OS X. 

In addition, for Windows users, you can use OSC Connectwhich is a native windows application developed by OSC to provide a launcher for secure file transfer, VNC, terminal, and web based services, as well as preconfigured management of secure tunnel connections. See this page for more information on OSC Connect. 

Connecting via SSH

The primary way you'll interact with the OSC clusters is through the SSH terminal. See our supercomputing environments for the hostnames of our current clusters. You should not need to do anything special beyond entering the hostname.

Once you've established an SSH connection, you will be presented with some informational text about the cluster you've connected to followed by a UNIX command prompt. For a brief discussion of UNIX command prompts and what you can do with them, see the next section of this guide.

Transferring Files

To transfer files, use your preferred SFTP client to connect to:

sftp.osc.edu

You may see warning message including SSH key fingerprint. Verify that the fingerprint in the message matches one of the SSH key fingerprint listed here, then type yes.

Since process times are limited on the login nodes, trying to transfer large files directly to oakley.osc.edu or other login nodes may terminate partway through. The sftp.osc.edu is specially configured to avoid this issue, and so we recommend it for all your file transfers.

Note: The sftp.osc.edu host is not connected to the scheduler, so you cannot submit jobs from this host. Use of this host for any purpose other than file transfer is not permitted.

Firewall Configuration 

See our Firewall and Proxy Settings page for information on how to configure your firewall to allow connection to and from OSC.

Setting up X Windows (Optional)

With an X Windows server you will be able to run graphical applications on our clusters that display on your workstation. To do this, you will need to launch your X Windows server before connecting to our systems. Then, when setting up your SSH connection, you will need to be sure to enable "X11 Forwarding".

For users of the command-line ssh client, you can do this by adding the "-X" option. For example, the below will connect to the Oakley cluster with X11 forwarding:

$ ssh -X username@oakley.osc.edu

If you are connecting with PuTTY, the checkbox to enable X11 forwarding can be found in the connections pane under "Connections → SSH → X11".

For other SSH clients, consult their documentation to determine how to enable X11 forwarding.

NOTE: The X-Windows protocol is not a high-performance one. Depending on your system and Internet connection, X Windows applications may run very slowly, even to the point of being unusable. If you find this to be the case and graphical applications are a necessity for your work, please contact OSC Help to discuss alternatives.
Supercomputer: 
Service: 

Allocations and Accounts

The Ohio Supercomputer Center provides cycles to both academic and industrial clients. The methods for gaining access to the systems are different for each user community.

Primarily, our users are Ohio-based and academic, and the vast majority of our cycles will continue to be consumed by Ohio-based academic users. Those cycles are allocated to academic PIs via an allocations process overseen by the Statewide Users Group, which evaluates applications via a peer review process. There are two basic classes of academic allocations:

  1.  an allocation of a number of resource units (RUs) that never expire
  2.  an annual allocation process that awards an RU allocation for the upcoming calendar year. The annual allocation process is designed for "discovery level" users, who consistently consume large amounts of resources, and allow the Allocations Committee to better allocate the limited Center resources.

Other users (non-Ohio academic or industrial/commercial) interested in using Center resources may purchase cycles at a rate set by the expected total consumption. Expert consulting support is also available.

You may find more information about "RU" calculation and charging here.
 

  Ohio Academic Other
  Rolling Annual Rolling
Duration Unlimited 1 year Unlimited
Allocation Approval SUG SUG OSC Director
Call for Proposals At any time November At any time
  Apply [By invitation] Apply

For users interested in gaining access to larger resources, please contact OSC Help. We can assist you in applying for resources at an NSF or XSEDE site.

Once a project has been created, the PI can create accounts for users by contacting OSC Help to add existing users to that project

I need to add authorized users to my project

Please contact OSC Help. We require PI approval. 

I need additional resources for my existing project

If you already have a project, but are running low on allocated resources, you can click on the links above for your project type to request additional resources.

See our Project Application page for more information. 

I wish to use OSC to support teaching a class

We provide special classroom allocations for this purpose. You may apply here. See our Project Application page for more information. 

I don't think I fit in the above categories

Please contact us.

Managing Your Account

Once your account is set-up, you will receive a welcome letter from us containing your username and instructions for setting up your account.

MyOSC, one of our web portals, provides services to OSC users, including:

  • Password changes and resets for your HPC accounts
  • Changing your UNIX shell
  • Updating your email address
  • Tracking your project(s) balance​

Note: HPC password changes and resets take 3-5 minutes to process. If you attempt to login with your new password, you will receive an error. Too many failed login attempts results in your account being automatically locked for 5 minutes.

 

Are the answers you're looking for not here? Contact OSC Help

Supercomputer: 
Service: 

Managing Your Project

Principal Investigators of OSC projects are responsible for updating their authorized user list, their outside funding sources, and their publications and presentations that cite OSC. All of these tasks can be accomplished by contacting OSC Help with the information. PIs are also responsible for monitoring their project's Resource Unit balance and for submitting a proposal to the Allocations Committee of the Statewide Users Group before the balance goes negative. Projects with negative balances are subject to restrictions, causing submitted jobs to have lower priority in the queue. We have a resource page for your convenience. 

Allocation managment for Ohio-based academic projects

Eligible Principal Investigators must be tenure-track faculty members or permanent research scientists at Ohio colleges or universities. All other researchers, such as postdoctoral researchers, graduate students, and undergraduate students, may be authorized users under a project led by an eligible PI.

Principal Investigators who plan to submit a proposal for additional resource units should have the following information ready when they fill out the application form:

  • names, departments, affiliations, and email addresses of new authorized users
  • names and email address of two or three suggested peer reviewers
  • document with performance reports and justification for request
  • electronic version of resume/CV
  • list of publications and presentations that cite OSC
  • proposal text
  • document with other pertinent information (optional)

Send a query to grant@osc.edu or to oschelp@osc.edu if you require more detailed information concerning the application process.

Once a proposal has been approved for peer review, any restrictions that have been accrued due to a negative balance will be lifted. The Allocations Committee meets every other month (even-numbered months), and awards are made after the meeting.

Account Maintenance and Usage: Please use MyOSC to update your password, shell, and check usage on the project. The command OSCusage will also provide usage details. 

Allocation managment for purchased cycles

Please disregard any RU messages or instructions to apply for additional resources. If you have any billing questions, please consult your contact at OSC or inquire with OSC Help.

 

Supercomputer: 
Service: 

Project Applications

Application if you are based at an Ohio Academic Institution

Statewide academic allocation of computing resources and services is a peer-reviewed account process governed by the Allocations Committee of the Statewide Users Group and subject to the Ohio Supercomputer Center (OSC) Code of Ethics for Academic Users.  Ohio Academic Clients are eligible for highly subsidized access to OSC resources, with fees generally only being charged above certain resource thresholds.  These clients must submit a proposal that is potentially peer-reviewed and/or approved by their host insitution.  More information is available here.

The procedures described enable services to be available to approved projects conducted by eligible principal investigators (PIs) at Ohio academic institutions. In order to be an eligible PI at OSC, you must be eligible to hold PI status at your college, university, or research organization administered by an Ohio academic institution (i.e., be a full-time, permanent academic researcher or tenure-track faculty member at an Ohio college or unviersity). Students, post-doctoral fellows, visiting scientists, and others who wish to use the facilities may be authorized users on projects headed by an eligible PI. Once a PI has received their project information, he/she can contact OSC Help to add new authorized user(s) to a project at any time.

All first-time PIs must submit a signature page (electronically or hard copy). Please send a copy of the signature page to the attention of the Allocations Committee, OSC, 1224 Kinnear Road, Columbus, Ohio, 43212.

OSC's online project application is part of an electronic system that leads you through the process step by step. Before you begin to fill in the application form, especially if you are new to the process, look at the Project Application Help page. From this help page, please note the required materials for each request. Have all information and materials (suggested reviewer names and addresses, proposal text file names, etc.) at hand before you begin the application form. You can save a partially completed project application form for later use. As a reminder, you cannot re-submit application materials if they have already been used. 

Online Project and Account Application Form (NEW OSC PI)
Online Project Application Form (EXISTING PI)
Classroom Project Application Form (use our resources and software in the classroom)

If you need assistance, please contact OSC Help.

Application if you are NOT based at an Ohio Academic Institution

Researchers from business, non-Ohio academic, nonprofit or other organizations who wish to use the OSC's resources should complete the Other Client request form available here. All clients not affiliated with an Ohio academic institution must sign a service agreement, provide a $500 deposit, and pay for resource usage per a standard price list.

Add a new user

Please submit your request to OSC Help or use our online submission form. For reference, we will need the following:

Project:
Full Name:
If not U.S. Citizen, country of citizenship:
Institution:
Department:
Affiliation:
Email address:
Special Requests?

Letter of Commitment for Outside Funding Proposals

OSC will provide a letter of commitment users can include with their account proposals for outside funding, such as from the Department of Energy, National Institutes of Health, National Science Foundation (limited to standard text, per NSF policy), etc. This letter details OSC's commitment to supporting research efforts of its users and the facilities and platforms we provide our users. [Note: This letter does not waive the normal OSC account application process; it merely states that OSC is willing to support such application.] The information users must provide for the letter is:

  • address to which the letter should be directed (e.g. NSF, Department, mailing address)
  • name of funding agency's representative
  • name of the proposal, as well as names and home institutions of the Co-PIs
  • number of RUs per year you would apply for if you were to receive funding
  • number of years of proposed research
  • solicitation number

Send e-mail with your request for the commitment letter to OSC Help or submit online. We will prepare a draft for your approval and then we will send you the final PDF for your proposal submission. Please allow at least two working days for this service.

Letter of Support for Outside Funding Proposals

Letters of support may be subject to strict and specific guidelines, and may not be accepted by your funding agency.

If you need a letter of support, please see above "Letter of Commitment for Outside Funding Proposals". 

Applying at NSF Centers

Researchers requiring additional computing resources should consider applying for allocations at National Science Foundation Centers. For more information, please write to oschelp@osc.edu, and your inquiry will be directed to the appropriate staff member.:

We require you cite OSC in any publications or reports that result from projects supported by our services.

UNIX Basics

OSC HPC resources use an operating system called "Linux", which is a UNIX-based operating system, first released on 5 October 1991. Linux is by a wide margin the most popular operating system choice for supercomputing, with over 90% of the Top 500 list running some variant of it. In fact, many common devices run Linux variant operating systems, including game consoles, tablets, routers, and even Android-based smartphones.

While Linux supports desktop graphical user interface configurations (as does OSC) in most cases, file manipulation will be done via the command line. Since all jobs run in batch will be non-interactive, they by definition will not allow the use of GUIs. Thus, we strongly suggest new users become comfortable with basic command-line operations, so that they can learn to write scripts to submit to the scheduler that will behave as intended. We have provided some tutorials explaining basics from moving about the file system, to extracting archives, to modifying your environment, that are available for self-paced learning.

Linux Command Line Fundamentals

This tutorial teaches you about the linux command line and shows you some useful commands. It also shows you how to get help in linux by using the man and apropos commands.


Linux Tutorial

This tutorial guides you through the process of creating and submitting a batch script on one of our compute clusters. This is a linux tutorial which uses batch scripting as an example, not a tutorial on writing batch scripts. The primary goal is not to teach you about batch scripting, but for you to become familiar with certain linux commands that can be used either in a batch script or at the command line. There are other pages on the OSC web site that go into the details of submitting a job with a batch script.


Linux Shortcuts

This tutorial shows you some handy time-saving shortcuts in linux. Once you have a good understanding of how the command line works, you will want to learn how to work more efficiently.


Tar Tutorial

This tutorial shows you how to download tar (tape archive) files from the internet and how to deal with large directory trees of files.


Service: 

Linux Command Line Fundamentals

 

Description

This tutorial teaches you about the linux command line and shows you some useful commands. It also shows you how to get help in linux by using the man and apropos commands.

For more training and practice using the command line, you can find many great tutorials. Here are a few:

https://www.learnenough.com/command-line-tutorial

https://cvw.cac.cornell.edu/Linux/

http://www.ee.surrey.ac.uk/Teaching/Unix/

https://www.udacity.com/course/linux-command-line-basics--ud595

 

More Advanced:

http://moo.nac.uci.edu/~hjm/How_Programs_Work_On_Linux.html

Prerequisites

None.

Introduction

Unix is an operating system that comes with several application programs. Other examples of operating systems are Microsoft Windows, Apple OS and Google's Android. An operating system is the program running on a computer (or a smartphone) that allows the user to interact with the machine -- to manage files and folders, perform queries and launch applications. In graphical operating systems, like Windows, you interact with the machine mainly with the mouse. You click on icons or make selections from the menus. The Unix that runs on OSC clusters gives you a command line interface. That is, the way you tell the operating system what you want to do is by typing a command at the prompt and hitting return. To create a new folder you type mkdir. To copy a file from one folder to another, you type cp. And to launch an application program, say the editor emacs, you type the name of the application. While this may seem old-fashioned, you will find that once you master some simple concepts and commands you are able to do what you need to do efficiently and that you have enough flexibility to customize the processes that you use on OSC clusters to suit your needs.

Common Tasks on OSC Clusters

What are some common tasks you will perform on OSC clusters? Probably the most common scenario is that you want to run some of the software we have installed on our clusters. You may have your own input files that will be processed by an application program. The application may generate output files which you need to organize. You will probably have to create a job script so that you can execute the application in batch mode. To perform these tasks, you need to develop a few different skills. Another possibility is that you are not just a user of the software installed on our clusters but a developer of your own software -- or maybe you are making some modifications to an application program so you need to be able to build the modified version and run it. In this scenario you need many of the same skills plus some others. This tutorial shows you the basics of working with the Unix command line. Other tutorials go into more depth to help you learn more advanced skills.

The Kernel and the Shell

You can think of Unix as consisting of two parts -- the kernel and the shell. The kernel is the guts of the Unix operating system -- the core software running on a machine that performs the infrastructure tasks like making sure multiple users can work at the same time. You don't need to know anything about the kernel for the purposes of this tutorial. The shell is the program that interprets the commands you enter at the command prompt. There are several different flavors of Unix shells -- Bourne, Korn, Cshell, TCshell and Bash. There are some differences in how you do things in the different shells, but they are not major and they shouldn't show up in this tutorial. However, in the interest of simplicity, this tutorial will assume you are using the Bash shell. This is the default shell for OSC users. Unless you do something to change that, you will be running the Bash shell when you log onto Oakley or Glenn.

The Command Prompt

The first thing you need to do is log onto one of the OSC clusters, Oakley or Glenn. If you do not know how to do this, you can find help at the OSC home page. If you are connecting from a Windows system, you need to download and setup the OSC Starter Kit which you can find here. If you are connecting from a Mac or Linux system, you will use ssh. To get more information about using ssh, go to the OSC home page, hold your cursor over the "Supercomputing" menu in the main blue menu bar and select "FAQ." This should help you get started. Once you are logged in look for the last thing displayed in the terminal window. It should be something like

-bash-3.2$

with a block cursor after it. This is the command prompt -- it's where you will see the commands you type in echoed to the screen. In this tutorial, we will abbreviate the command prompt with just the dollar sign - $. The first thing you will want to know is how to log off. You can log off of the cluster by typing "exit" then typing the <Enter> key at the command prompt:

$ exit <Enter>

For the rest of this tutorial, when commands are shown, the <Enter> will be omitted, but you must always enter <Enter> to tell the shell to execute the command you just typed.

First Simple Commands

So let's try typing a few commands at the prompt (remember to type the <Enter> key after the command):

$ date

$ cal

$ finger

$ who

$ whoami

$ finger -l

That last command is finger followed by a space then a minus sign then the lower case L. Is it obvious what these commands do? Shortly you will learn how to get information about what each command does and how you can make it behave in different ways. You should notice the difference between "finger" and "finger -l" -- these two commands seem to do similar things (they give information about the users who are logged in to the system) but they print the information in different formats. try the two commands again and examine the output. Note that you can use the scroll bar on your terminal window to look at text that has scrolled off the screen.

man

The "man" command is how you find out information about what a command does. Type the following command:

$ man

It's kind of a smart-alecky answer you get back, but at least you learn that "man" is short for "manual" and that the purpose is to print the manual page for a command. Before we start looking at manual pages, you need to know something about the way Unix displays them. It does not just print the manual page and return you to the command prompt -- it puts you into a mode where you are interactively viewing the manual page. At the bottom of the page you should see a colon (:) instead of the usual command prompt (-bash-3.2$). You can move around in the man page by typing things at the colon. To exit the man page, you need to type a "q" followed by <Enter>. So try that first. Type

$ man finger

then at the colon of the man page type

: q

You do not have to type <Enter> after the "q" (this is different from the shell prompt.) You should be back at the shell prompt now. Now let's go through the man page a bit. Once again, type

$ man finger

Now instead of just quitting, let's look at the contents of the man page. The entire man page is probably not displayed in your terminal. To scroll up or down, use the arrow keys or the <Page Up> and <Page Down> keys of the keyboard. The <Enter> and <Space> keys also scroll. Remember that "q" will quit out of the man page and get you back to the shell prompt.

The first thing you see is a section with the heading "NAME" which displays the name of the command and a short summary of what it does. Then there is a section called "SYNOPSIS" which shows the syntax of the command. In this case you should see

SYNOPSIS

     finger [-lmsp] [user ...] [user@host ...]

Remember how "finger" and "finger -l" gave different output? The [-lmsp] tells you that you can use one of those four letters as a command option -- i.e., a way of modifying the way the command works. In the "DESCRIPTION" section of the man page you will see a longer description of the command and an explanation of the options. Anything shown in the command synopsis which is contained within square brackets ([ ]) is optional. That's why it is ok to type "finger" with no options and no user. What about "user" -- what is that? To see what that means, quit out of the man page and type the following at the command prompt:

$ whoami

Let's say your username is osu0000. Then the result of the "whoami" command is osu0000. Now enter the following command (but replace osu0000 with your username):

$ finger osu0000

You should get information about yourself and no other users. You can also enter any of the usernames that are output when you enter the "finger" command by itself. The user names are in the leftmost column of output. Now try

 

$ finger -l osu0000

$ finger -lp osu0000

$ finger -s osu0000 osu0001

For the last command, use your username and the username of some other username that shows up in the output of the "finger" command with no arguments.

Note that a unix command consists of three parts:

  • command
  • option(s)
  • argument(s)

You don't necessarily have to enter an argument (as you saw with the "finger" command) but sometimes a command makes no sense without an argument so you must enter one -- you saw this with the "man" command. Try typing

$ man man

and looking briefly at the output. One thing to notice is the synopsis -- there are a lot of possible options for the "man" command, but the last thing shown in the command synopsis is "name ..." -- notice that "name" is not contained in square brackets. This is because it is not optional -- you must enter at least one name. What happens if you enter two names?

$ man man finger

The first thing that happens is you get the man page for the "man" command. What happens when you quit out of the man page? You should now get the man page for the "finger" command. If you quit out of this one you will be back at the shell prompt.

Combining Commands

You can "pipe" the output of one command to another. First, let's learn about the "more" command:

$ man more

Read the "DESCRIPTION" section -- it says that more is used to page through text that doesn't fit on one screen. It also recommends that the "less" command is more powerful. Ok, so let's learn about the "less" command:

$ man less

You see from the description that "less" also allows you to examine text one screenful at a time. Does this sound familiar? The "man" command actually uses the "less" command to display its output. But you can use the "less" command yourself. If you have a long text file named "foo.txt" you could type

$ less foo.txt

and you would be able to examine the contents of the file one screen at a time. But you can also use "less" to help you look at the output of a command that prints more than one screenful of output. Try this:

$ finger | less

That's "finger" followed by a space followed by the vertical bar (shifted backslash on most keyboards) followed by a space followed by "less" followed by <Enter>. You should now be looking at the output of the "finger" command in an interactive fashion, just as you were looking at man pages. Remember, to scroll use the arrow keys, the <Page Up> and <Page Down> keys, the <Enter> key or the space bar; and to quit, type "q".

Now try the following (but remember to replace "osu0000" with your actual username):

$ finger | grep osu0000

The "grep" command is Unix's command for searching. Here you are telling Unix to search the output of the "finger" command for the text "osu0000" (or whatever your username is.)

If you try to pipe the output of one command to a second command and the second is a command which works with no arguments, you won't get what you expect. Try

$ whoami | finger

You see that it does not give the same output as

$ finger osu0000

(assuming "whoami" returns osu0000.)

In this case what you can do is the following:

$ finger `whoami`

That's "finger" space backquote "whoami" backquote. The backquote key is to the left of the number 1 key on a standard keyboard.

apropos

Enter the following command:

$ man apropos

As you can see, the apropos searches descriptions of commands and finds commands whose descriptions match the keyword you entered as the argument. That means it outputs a list of commands that have something to do with the keyword you entered. Try this

$ apropos

Ok, you need to enter an argument for the "apropos" command.

So try

$ apropos calendar

Now you see that among the results are two commands -- "cal" and "difftime" that have something to do with the keyword "calendar."

Linux Tutorial

 

Description

This tutorial guides you through the process of creating and submitting a batch script on one of our compute clusters. This is a linux tutorial which uses batch scripting as an example, not a tutorial on writing batch scripts. The primary goal is not to teach you about batch scripting, but for you to become familiar with certain linux commands. There are other pages on the OSC web site that go into the details of submitting a job with a batch script.

Prerequisites

Goals

  • Create subdirectories to organize information
  • Create a batch script with a text editor
  • Submit a job
  • Check on the progress of the job
  • Change the permissions of the output files
  • Get familiar with some common unix commands

Step 1 - Organize your directories

When you first log in to our clusters, you are in your home directory. For the purposes of this illustration, we will pretend you are user osu0001 and your project code is PRJ0001, but when you try out commands you must use your own username and project code.

$ pwd
/users/PRJ0001/osu0001
 
Note: you will see your user name and a different number after the /users.
 
It's a good idea to organize your work into separate directories. If you have used Windows or the Mac operating system, you may think of these as folders. Each folder may contain files and subfolders. The subfolders may contain other files and subfolders of their own. In linux we use the term "directory" instead of "folder." Use directories to organize your work.
 
Type the following four lines and take note of the output after each one:
 
$ touch foo1
$ touch foo2
$ ls
$ ls -l
$ ls -lt
$ ls -ltr
 
The "touch" command just creates an empty file with the name you give it.
You probably already know that the ls command shows the contents of the current working directory; that is, the directory you see when you type pwd. But what is the point of the "-l", "-lt" or "-ltr"? You noticed the difference in the output between just the "ls" command and the "ls -l" command.
Most unix commands have options you can specify that change the way the command works. The options can be specified by the "-" (minus sign) followed by a single letter. "ls -ltr" is actually specifying three options to the ls command.
l: I want to see the output in long format -- one file per line with some interesting information about each file
t: sort the display of files by when they were last modified, most-recently modified first
r: reverse the order of display (combined with -t this displays the most-recently modified file last -- it should be BatchTutorial in this case.)
 
I like using "ls -ltr" because I find it convenient to see the most recently modified file at the end of the list.
 
Now try this:
$ mkdir BatchTutorial
$ ls -ltr
 
The "mkdir" command makes a new directory with the name you give it. This is a subfolder of the current working directory. The current working directory is where your current focus is in the hierarchy of directories. The 'pwd' command shows you are in your home directory:
 
$ pwd
/users/PRJ0001/osu0001
 
Now try this:
$ cd BatchTutorial
$ pwd
 
What is the output of 'pwd' now? "cd" is short for "change directory" -- think of it as moving you into a different place in the hierarchy of directories. Now do
$ cd ..
$ pwd
Where are you now?

Step 2 -- Get familiar with some more unix commands

Try the following:

$ echo where am I?
$ echo I am in `pwd`
$ echo my home directory is $HOME
$ echo HOME
$ echo this directory contains `ls -l`

These examples show what the echo command does and how to do some interesting things with it. The `pwd` means the result of issuing the command pwd. HOME is an example of an environment variable. These are strings that stand for other strings. HOME is defined when you log in to a unix system. $HOME means the string the variable HOME stands for. Notice that the result of "echo HOME" does not do the substitution. Also notice that the last example shows things don't always get formatted the way you would like.

Some more commands to try:

$ cal
$ cal > foo3
$ cat foo3
$ whoami
$ date

Using the ">" after a command puts the output of the command into a file with the name you specify. The "cat" command prints the contents of a file to the screen.

Two very important UNIX commands are the cp and mv commands. Assume you have a file called foo3 in your current directory created by the "cal > foo3" command. Suppose you want to make a copy of foo3 called foo4. You would do this with the following command:

$ cp foo3 foo4
$ ls -ltr

Now suppose you want to rename the file 'foo4' to 'foo5'. You do this with:

$ mv foo4 foo5
$ ls -ltr

'mv' is short for 'move' and it is used for renaming files. It can also be used to move a file to a different directory.

$ mkdir CalDir
$ mv foo5 CalDir
$ ls
$ ls CalDir

Notice that if you give a directory with the "ls" command is shows you what is in that directory rather than the current working directory.

Now try the following:

$ ls CalDir
$ cd CalDir
$ ls
$ cd ..
$ cp foo3 CalDir
$ ls CalDir

Notice that you can use the "cp" command to copy a file to a different directory -- the copy will have the same name as the original file. What if you forget to do the mkdir first?

$ cp foo3 FooDir

Now what happens when you do the following:

$ ls FooDir
$ cd FooDir
$ cat CalDir
$ cat FooDir
$ ls -ltr

CalDir is a directory, but FooDir is a regular file. You can tell this by the "d" that shows up in the string of letters when you do the "ls -ltr". That's what happens when you try to cp or mv a file to a directory that doesn't exist -- a file gets created with the target name. You can imagine a scenario in which you run a program and want to copy the resulting files to a directory called Output but you forget to create the directory first -- this is a fairly common mistake.

Step 3 -- Environment Variables

Before we move on to creating a batch script, you need to know more about environment variables. An environment variable is a word that stands for some other text. We have already seen an example of this with the variable HOME. Try this:

$ MY_ENV_VAR="something I would rather not type over and over"
$ echo MY_ENV_VAR
$ echo $MY_ENV_VAR
$ echo "MY_ENV_VAR stands for $MY_ENV_VAR"

You define an environment variable by assigning some text to it with the equals sign. That's what the first line above does. When you use '$' followed by the name of your environment variable in a command line, UNIX makes the substitution. If you forget the '$' the substitution will not be made.

There are some environment variables that come pre-defined when you log in to Glenn or Oakley. Try using 'echo' to see the values of the following variables: HOME, HOSTNAME, SHELL, TERM, PATH.

Now you are ready to use some of this unix knowledge to create and run a script.

Step 4 -- Create and run a script

Before we create a batch script and submit it to a compute node, we will do something a bit simpler. We will create a regular script file that will be run on the login node. A script is just a file that consists of unix commands that will run when you execute the script file. It is a way of gathering together a bunch of commands that you want to execute all at once. You can do some very powerful things with scripting to automate tasks that are tedious to do by hand, but we are just going to create a script that contains a few commands we could easily type in. This is to help you understand what is happening when you submit a batch script to run on a compute node.

Use a text editor to create a file named "tutorial.sh" which contains the following text (note that with emacs or nano you can use the mouse to select text and then paste it into the editor with the middle mouse button):

$ nano tutorial.sh

 

echo ----
echo Job started at `date`
echo ----
echo This job is working on node `hostname`

SH_WORKDIR=`pwd`
echo working directory is $SH_WORKDIR
echo ----
echo The contents of $SH_WORKDIR
ls -ltr
echo
echo ----
echo
echo creating a file in SH_WORKDIR
whoami > whoami-sh-workdir

SH_TMPDIR=${SH_WORKDIR}/sh-temp
mkdir $SH_TMPDIR
cd $SH_TMPDIR
echo ----
echo TMPDIR IS `pwd`
echo ----
echo wait for 12 seconds
sleep 12
echo ----
echo creating a file in SH_TMPDIR
whoami > whoami-sh-tmpdir

# copy the file back to the output subdirectory
cp ${SH_TMPDIR}/whoami-sh-tmpdir ${SH_WORKDIR}/output

cd $SH_WORKDIR

echo ----
echo Job ended at `date`

To run it:

$ chmod u+x tutorial.sh
$ ./tutorial.sh

Look at the output created on the screen and the changes in your directory to see what the script did.

Step 5 -- Create and run a batch job

Use your favorite text editor to create a file called tutorial.pbs in the BatchTutorial directory which has the following contents (remember, you can use the mouse to cut and paste text):

#PBS -l walltime=00:02:00
#PBS -l nodes=1:ppn=1
#PBS -N foobar
#PBS -j oe
#PBS -r n

echo ----
echo Job started at `date`
echo ----
echo This job is working on compute node `cat $PBS_NODEFILE`

cd $PBS_O_WORKDIR
echo show what PBS_O_WORKDIR is
echo PBS_O_WORKDIR IS `pwd`
echo ----
echo The contents of PBS_O_WORKDIR:
ls -ltr
echo
echo ----
echo
echo creating a file in PBS_O_WORKDIR
whoami > whoami-pbs-o-workdir

cd $TMPDIR
echo ----
echo TMPDIR IS `pwd`
echo ----
echo wait for 42 seconds
sleep 42
echo ----
echo creating a file in TMPDIR
whoami > whoami-tmpdir

# copy the file back to the output subdirectory
pbsdcp -g $TMPDIR/whoami-tmpdir $PBS_O_WORKDIR/output

echo ----
echo Job ended at `date`
 
To submit the batch script, type
$ qsub tutorial.pbs
Use qstat -u [username] to check on the progress of your job. If you see something like this
$ qstat -u osu0001

                                                                             Req'd  Req'd   Elap
Job ID             Username    Queue    Jobname          SessID NDS   TSK    Memory Time  S Time
------------------ ----------- -------- ---------------- ------ ----- ------ ------ ----- - -----
458842.oak-batch   osu0001     serial   foobar              --      1      1    --  00:02 Q   --
 
this means the job is in the queue -- it hasn't started yet. That is what the "Q" under the S column means.
 
If you see something like this:
                                                                             Req'd  Req'd   Elap
Job ID             Username    Queue    Jobname          SessID NDS   TSK    Memory Time  S Time
------------------ ----------- -------- ---------------- ------ ----- ------ ------ ----- - -----
458842.oak-batch   osu0001     serial   foobar            26276     1      1    --  00:02 R   --
this means the job is running and has job id 458842.
 
When the output of the qstat command is empty, the job is done.
 
After it is done, there should be a file called "foobar.o458842" in the directory.
Note that your file will end with a different number -- namely the job id number assigned to your job.
Check this with
$ ls -ltr
$ cat foobar.oNNNNNN

Where (NNNNNN is your job id).

The name of this file is determined by two things:
  1. The name you give the job in the script file with the header line #PBS -N foobar
  2. The job id number assigned to the job.

The name of the script file (tutorial.pbs) has nothing to do with the name of the output file.

Examine the contents of the output file foobar.oNNNNNN carefully. You should be able to see the results of some of the commands you put in tutorial.pbs. It also shows you the values of the variables PBS_NODEFILE, PBS_O_WORKDIR and TMPDIR. These variables exist only while your job is running. Try

$ echo $PBS_O_WORKDIR

and you will see it is no longer defined. $PBS_NODEFILE is a file which contains a list of all the nodes your job is running on. Because this script has the line

#PBS -l nodes=1:ppn=1

the contents of $PBS_NODEFILE is the name of a single compute node.

Notice that $TMPDIR is /tmp/pbstmp.NNNNNN (again, NNNNNN is the id number for this job.) Try

$ ls /tmp/pbstmp.NNNNNN

Why doesn't this directory exist? Because it is a directory on the compute node, not on the login node. Each machine in the cluster has its own /tmp directory and they do not contain the same files and subdirectories. The /users directories are shared by all the nodes (login or compute) but each node has its own /tmp directory (as well as other unshared directories.)

Tar Tutorial

 

Prerequisites

Step 1 -- Create a directory to work with and download a "tarball"

Start off with the following:

$ mkdir TarTutorial
$ cd TarTutorial
$ wget http://www.mmm.ucar.edu/wrf/src/WRFDAV3.1.tar.gz
$ ls -ltr

The third command will take a while because it is downloading a file from the internet. The file is call a "tarball" or a "gzipped tarball". TAR is an old unix short name for "tape archive" but a tar file is a file that contains a bunch of other files. If you have to move a bunch of files from one place to another, a good way to do it is to pack them into a tar file, move the tar file where you want it then unpack the files at the destination. A tar file usually has the extension ".tar". What about the ".gz"? This means the tar file has been further compressed with the program gzip -- this makes it a lot smaller.

Step 2 -- Unpack the "tarball" and check out the contents

After step 1 your working directory should be ~/TarTutorial and there should be a file called WRFDAV3.1.tar.gz in it.

Now do this:

$ gunzip WRFDAV3.1.tar.gz
$ ls -ltr

You should now have a file called WRFDAV3.1.tar which should be quite a bit larger in size than WRFDAV3.1.tar.gz -- this is because it has been uncompressed by the "gunzip" command which is the opposite of the "gzip" command.

Now do this:

$ tar -xvf WRFDAV3.1.tar
$ ls -ltr

You should see a lot of filenames go by on the screen and when the first command is done and you issue the ls command you should see two things -- WRFDAV3.1.tar is still there but there is also a directory called WRFDA. You can look at the contents of this directory and navigate around in the directory tree to see what is in there. The options on the "tar" command have the following meanings (you can do a "man tar" to get all the options):

x: extract the contents of the tar file

v: be verbose, i.e. show what is happening on the screen

f: the name of the file which follows the "f" option is the tar file to expand.

Another thing you can do is see how much space is being taken up by the files. Make sure TarTutorial is your working directory then issue the following command:

$ du .

Remember that "." (dot) means the current working directory. The "du" command means "disk usage" -- it shows you how much space is being used by every file and directory in the directory tree. It ends up with the highest level files and directories. You might prefer to do

$ du -h .
$ ls -ltrh

Adding the "-h" option to these commands puts the file sizes in human-readable format -- you should get a size of 66M for the tar file -- that's 66 megabytes -- and "du" should print a size of 77M next to ./WRFDA.

Step 3 -- create your own "tarball"

Now, make your own tar file from the WRFDA directory tree:

$ tar -cf mywrf.tar WRFDA
$ ls -ltrh

You have created a tar from all the files in the WRFDA directory. The options given to the "tar" command have the following meanings:

c: create a tar file

f: give it the name which follows the "f" option

The files WRFDAV3.1.tar and mywrf.tar are identical. Now compress the tar file you made:

$ gzip mywrf.tar
$ ls -ltrh

You should see a file called mywrf.tar.gz which is smaller than WRFDAV3.1.tar.

Step 4 -- Clean up!

You don't want to leave all these files lying around. So delete them

$ rm WRFDAV3.1.tar
$ rm mywrf.tar
$ rm WRFDA

Oops! You can't remove the directory. You need to use the "rmdir" command:

$ rmdir WRFDA

Oh no! That doesn't work on a directory that's not empty. So are you stuck with all those files? Maybe you can do this:

$ cd WRFDA
$ rm *
$ cd ..
$ rmdir WRFDA

That won't work either because there are some subdirectories in WRFDA and "rm *" won't remove them. Do you have to work your way to the all the leaves at the bottom of the directory tree and remove files then come back up and remove directories? No, there is a simpler way:

$ rm -Rf WRFDA

This will get rid of the entire directory tree. The options have the following meanings:

R: recursively remove all files and directories

f: force; i.e., just remove everything without asking for confirmation

I encourage you to do

$ man rm

and check out all the options. Or some of them -- there are quite a few.

 

Unix Shortcuts

 

Description

This tutorial shows you some handy time-saving shortcuts in linux. Once you have a good understanding of how the command line works, you will want to learn how to work more efficiently.

Prerequisites

Linux command line fundamentals.

Goals

  • Save you time when working on a linux system
  • Increase your appreciation of the power of linux

Step 1 -- The Arrow Keys

Note: even if you know how to use the up arrow in linux, you need to enter the commands in this section because they are used in the following sections. So to begin this tutorial, go to your home directory and create a new directory called ShortCuts:

$ cd
$ mkdir Shortcuts
$ cd Shortcuts

(If a directory or file named "Shortcuts" already exists, name it something else.)

Imagine typing in a long linux command and making a typo. This is one of the frustrating things about a command line interface -- you have to retype the command, correcting the typo this time. Or what if you have to type several similar commands -- wouldn't it be nice to have a way to recall a previous command, make a few changes, and enter the new command? This is what the up arrow is for.

Try the following:

$ cd ..
$ cd ShortCuts (type a capital C)

Linux should tell you there is no directory with that name.

Now type the up arrow key -- the previous command you entered shows up on the command line, and you can use the left arrow to move the cursor just after the capital C, hit Backspace, and type a lower case c. Note you can also position the cursor before the capital C and hit Delete to get rid of it.

Once you have changed the capital C to a lower case c you can hit Return to enter the command -- you do not have to move the cursor to the end of the line.

Now hit the up arrow key a few times, then hit the down arrow key and notice what happens. Play around with this until you get a good feel for what is happening.

Linux maintains a history of commands you have entered. Using the up and down arrow keys, you can recall previously-entered commands to the command line, edit them and re-issue them.

Note that in addition to the left and right arrow keys you can use the Home and End keys to move to the beginning or end of the command line. Also, if you hold down the Ctrl key when you type an arrow key, the cursor will move by an entire word instead of a single character -- this is useful is many situations and works in many editors.

Let's use this to create a directory hierarchy and a few files. Start in the Shortcuts directory and enter the following commands, using the arrow keys to simplify your job:

$ mkdir directory1
$ mkdir directory1/directory2
$ mkdir directory1/directory2/directory3
$ cd directory1/directory2/diectoryr3  (remember the Home key and the Ctrl key with left and right arrows)
$ hostname > file1
$ whoami > file2
$ mkdir directory4
$ cal > directory4/file3

Step 2 -- Using the TAB key

Linux has short, cryptic command names to save you typing -- but it is still a command line interface, and that means you interact with the operating system by typing in commands. File names can be long, directory hierarchies can be deep, and this can mean you have to type a lot to specify the file you want or change to current working directory. Not only that, but you have to remember the names of files and directories you type in. The TAB key gives you a way to enter with commands with less typing and less memorization.

Go back to the Shortcuts directory:

$ cd
$ cd Shortcuts

Now enter the following:

$ hostname > file1
$ cal > file2
$ whoami > different-file
$ date > other-file
$ cal > folio5

Now type the following, without hitting the Return key:

$ cat oth <Tab>

What happened? Linux completed the name "other-file" for you! The Tab key is your way of telling Linux to finish the current word you are typing, if possible. Because there is only one file in the directory whose name begins with "oth", when you hit the Tab key Linux is able to complete the name.

Hit Return (if you haven't already) to enter the cat command. Now try

$ cat d <Tab>

As you would expect, Linux completes the name "different-file"

What if you enter

$ cat fi <Tab>

Notice Linux completes as much of the name as possible. You can now enter a "1" or a "2" to finish it off.

But what if you forget what the options are? What if you can't remember if you created "file1" and "file2" or if you created "fileA" and fileB"?

With the comman line showing this:

$ cat file

hit the Tab key twice. Aha! Linux shows you the possible choices for completing the word.

Try

$ cat f <Tab>

The Tab will not add anything -- the command line will still read

$ cat f

Now type the letter o followed by a Tab -- once you add the o there is only one possible completion -- "folio".

Now enter the following:

$ cat directory1/directory2/directory3/directory4/file3

That's kind of a painful to type.

Now type the following without entering Return:

$ ls dir <Tab>

Nice! As you would expect, Linux completes the name of the directory for you. This is because there is only one file in the Shortcuts directory whose name begins with "dir"

Hit Return and Linux will tell you that directory1 contains directory2.

Now type this:

$ ls dir <Tab>

and before you hit return type another d followed by another Tab. Your command line should now look like this:

$ ls directory1/directory2/

If you hit Return, Linux will tell you that directory2 contains directory3.

Now try this:

$ ls dir <Tab>

then type another d followed by <Tab> then another d followed by tab. Don't hit Return yet. Your command line should look like this:

$ ls directory1/directory2/directory3/

Don't hit Return yet. Now type the letter f followed by a Tab. What do you think should happen?

Step 3 -- The Exclamation Point

Hitting the up arrow key is a nice way to recall previously-used commands, but it can get tedious if you are trying to recall a command you entered a while ago -- hitting the same key 30 times is a good way to make yourself feel like an automaton. Fortunately, linux offers a couple of other ways to recall previous commands that can be useful.

Go back to the Shortcuts directory

$ cd ~/Shortcuts

and enter the following:

$ hostname
$ cal
$ date
$ whoami

Now enter this:

$ !c

and hit return.

What happened? Now try

$ !h

and hit return.

The exclamation point ("bang" to Americans, "shriek" to some Englishmen I've worked with) is a way of telling linux you want to recall the last command which matches the text you type after it. So "!c" means recall the last command that starts with the letter c, the "cal" command in this case. You can enter more than one character after the exclamation point in order to distinguish between commands. For example if you enter

$ cd ~/Shortcuts
$ cat file1
$ cal
$ !c

the last command will redo the "cal" command. But if you enter

$ cat file1
$ cal
$ !cat

the last command re-executes the "cat" command.

Step 4 -- Ctrl-r

One problem with using the exclamation point to recall a previous command is that you can feel blind -- you don't get any confirmation about exactly which command you are recalling until it has executed. Sometimes you just aren't sure what you need to type after the exclamation point to get the command you want.

Typing Ctrl-r (that's holding down the Ctrl key and typing a lower case r) is another way to repeat previous commands without having to type the whole command, and it's much more flexible than the bang. The "r" is for "reverse search" and what happens is this. After you type Ctrl-r, start typing the beginning of a previously entered command -- linux will search, in reverse order, for commands that match what you type. To see it in action, type in the following commands (but don't hit <Enter> after the last one):

$ cd ~/Shortcuts
$ cat file1
$ cat folio5
$ cal
$ Ctrl-r cat

You should see the following on your command line:

(reverse-i-search)`cat': cat folio5

Try playing with this now. Type in " fi" (that's a space, an "f" and an "i") -- did the command shown at the prompt change? Now hit backspace four times.

Now enter a right or left arrow key and you will find yourself editing the matching command. This is one you have to play around with a bit before you understand exactly what it is doing. So go ahead and play with it.

Step 5 -- history

Now type

$ history

and hit return.

Cool, huh? You get to see all the commands you have entered (probably a maximum of 1000.) You can also do something like

$ history | grep cal

to get all the commands with the word "cal" in them. You can use the mouse to cut and paste a previous command, or you can recall it by number with the exclamation point:

$ !874

re-executes the command number 874 in your history.

For more information about what you can do to recall previous commands, check out http://www.thegeekstuff.com/2011/08/bash-history-expansion/

Step 6 -- Ctrl-t

I am just including this because to me it is a fun piece of linux trivia. I don't find it particularly useful. Type

$ cat file1

and hit <Return>. Now hit the up arrow key to recall this command and hist the left arrow key twice so the cursor is on the "e" of "file1". Now hit Ctrl-t (again, hold down the control key and type a lower case t.) What just happened? Try hitting Ctrl-t a couple more times. That's right -- it transposes two characters in the command line -- the one the cursor is on and the one to its left. Also, it moves the cursor to the right. Frankly, it takes me more time to think about what is going to happen if I type Ctrl-t than it takes me to delete some characters and retype them in the correct order. But somewhere out there is a linux black belt who gets extra productivity out of this shortcut.

Step 7 -- The alias command

Another nice feature of linux is the alias command. If there is a command you enter a lot you can define a short name for it. For example, we have been typing "cat folio5" a lot in this tutorial. You must be getting sick of typing "cat folio5". So enter the following:

$ alias cf5='cat folio5'

Now type

$ cf5

and hit return. Nice -- you now have a personal shortcut for "cat folio5". I use this for the ssh commands:

$ alias gogl='ssh -Y jeisenl@glenn.osc.edu'

I put this in the .bash_aliases file on my laptop so that it is always available to me.

Classroom Project Resource Guide

Welcome to OSC! 

This document includes some useful information to help your classroom use of our resources be effective.

Application: Please submit an application for a classroom project. We will require your resume/CV as the instructor and PI of the project, a class roster with email addresses, and a class syllabus. 

Access: We suggest that students consider connecting to our OnDemand portal to access the HPC resources. All production supercomputing resources can be accessed via that website, without having to worry about client configuration. We have a guide for new students to help them figure out the basics of using OSC; included are basics on getting connected, HPC system structure, file transfers, and batch systems.

Resources: We currently have three production clusters, Oakley, Ruby, and Owens, with Nvidia GPUs available. All systems have "debug" queues, during typical business hours, that allow small jobs of less than 1 hour to start much quicker than they might otherwise.

If you need to reserve access to particular resources, please contact OSC Help, preferably with at least two weeks lead time, so that we can put in the required reservations to ensure resources are available during lab or class times.

Software: We have a list of supported software, including sample batch scripts, in our documentation. If you have specific needs that we can help with, let OSC Help know.

Account Maintenance: Please use MyOSC to update your password, shell, and check usage on the project.

Homework Submissions?

We can provide you with project space to have the students submit assignments through our systems. Please ask about this service and see our how-to

SupportHelp can be found by contacting OSC Help weekdays, 9AM to 5PM (1-800-686-6472 or 614-292-1800).
Fill out a request online. 

 

We update our webpages to show relevant events at the center (including training) and system notices on our main page (osc.edu). We also provide important information in the “message of the day” (visible when you log in). Furthermore, you can receive notices by following @HPCNotices on Twitter. 
 

Useful Links:

FAQ: http://www.osc.edu/supercomputing/faq

Main Supercomputing pages: http://www.osc.edu/supercomputing/

Supercomputer: 

HOWTO

Our HOWTO collection contains short tutorials that help you step through some of the common (but potentially confusing) tasks users may need to accomplish, that do not quite rise to the level of requiring more structured training materials. Items here may explain a procedure to follow, or present a "best practices" formula that we think may be helpful.

Service: 

HOWTO: Add python packages using the conda package manager

While our Python installations come with many popular packages installed, you may come upon a case where you need an addiditonal package that is not installed.  If the specific package you are looking for is available from anaconda.org (formerlly binstar.org) you can easily install it and required dependencies by using the Conda package manager.

To be able to install a package using the Conda package manager:

  • You must use an Anaconda distribution of Python:
    • On Oakley the following modules:
      • python/2.7.latest*, python/2.7.8, python/3.4.2
    • On Ruby:
      • python/2.7.latest*, python/2.7.8, python/3.4.2
    • On Owens:
      • python/2.7, python/3.5
  • *latest suffix refers to a distribution that will be electively updated
  • Package should be available through anaconda.org and/or using pip (see next section).

If you would like to freeze a distribution so that you can control how often you update Anaconda, please send us a help request at oschelp@osc.edu.

But I use virtualenv and/or pip!

See the comparison to these package management tools here:

https://conda.io/docs/commands.html#conda-vs-pip-vs-virtualenv-commands

Pip installations are supported:

module load python
module list # check which python you just loaded
pip install --upgrade --user PACKAGE # where PACKAGE is a valid package name

If binaries exist in a pip installation they are usually installed in:

$HOME/.local/bin

The one issue with this approach is portability with multiple Python modules.  If you plan to stick with a single Python module, then this should not be an issue.  However, if you commonly switch between Python 2 and Python 3, then be aware of the potential trouble in using the same installation location for all Python versions.

Virtualenv may also work, but is not under active testing at OSC.

Procedure

We will install the yt package to a local directory in this example.

Load proper python module

module load python/2.7.8

Clone python installation to local directory

Three alternative create commands are listed.  These cover the most common cases:

conda create -n local --clone="$PYTHON_HOME"

This will clone the entire python installation to ~/envs/local directory. The process will take serveral minutes.

conda create -n local

This will create a local python installation without any packages. If you need a small number of packages, you may choose this option.

conda create -n local python={version} anaconda

If you like to install a specific version of python, you can specify it with "python" option. For example, you can use "python=2.4" for version 2.4.

To verify that a clone has been created, use the command

conda info -e

For additional conda command documentation see https://conda.io/docs/commands.html

Activate clone environment

For the bash shell:

source activate local

On newer versions of Anaconda on the Owens cluster you may also need to perform the removal of the following packages before trying to install your specific packages:

conda remove conda-build
conda remove conda-env

Install package

conda install yt
  • Replace yt with the name of the package you want to install, as listed by anaconda.org.
If there are errors on this step you will need to resolve them before continuing.

Test python package

Now we will test our installed python package by loading it in python and checking its location to ensure we are using the correct version.

python -c "import yt;print yt.__file__"

Output:

$HOME/.conda/envs/local/lib/python2.7/site-packages/yt/__init__.py
  • Replace both instances of yt  with the name of the package you installed.
Remember, you will need to load the proper version of python before you go to use your newlly installed package.  Packages are only installed to one version of python.

Install your own python modules

If the method using conda above is not working or if you prefer, you can consider installing python modules from the source. Please read HOWTO: install your own python modules.

Further Reading:

Conda Test Drive: https://conda.io/docs/test-drive.html 

Supercomputer: 

HOWTO: Install your own python modules

While we provide a number of Python modules, you may need a module we do not provide. If it is a commonly used module, or one that is particularly difficult to compile, you can contact OSC Help for assistance, but we have provided an example below showing how to build and install your own Python modules, and make them available inside of Python. Note, these instructions use "bash" shell syntax; this is our default shell, but if you are using something else (csh, tcsh, etc), some of the syntax may be different.

Please consider using conda python package manager before you try to build python using the method explained here. We have an instruction on conda here.

Gather your materials

First, you need to collect up what you need in order to do the installation. To keep things tidy, we will do all of our work in $HOME/local/src . You should make this directory now.

mkdir -p $HOME/local/src

Now, we will need to download the source code for the module we want to install. In our example, we will use "NumExpr", a module we already provide in the system version of Python. You can either download the file to your desktop, and then upload it to OSC, or directly download it using the wget utility (if you know the URL for the file).

cd ~/local/src
wget http://numexpr.googlecode.com/files/numexpr-2.0.1.tar.gz

Now, extract the downloaded file. In this case, since it's a "tar.gz" format, we can use tar to decompress and extract the contents.

tar xvfz numexpr-2.0.1.tar.gz

You can delete the downloaded archive now, if you wish, or leave it around should you want to start the installation from scratch.

Build it!

Environment

To build the module, we will want to first create a temporary environment variable to aid in installation. We'll call it "INSTALL_DIR".

export INSTALL_DIR=${HOME}/local/numexpr/2.0.1

I am following, roughly, the convention we use at the system level. This allows us to easily install new versions of software without risking breaking anything that uses older versions. We have specified a folder for the program (numexpr), and for the version (2.0.1). Now, to be consistent with python installations, we're going to create a second temporary environment variable, which will contain the actual installation location.

export TREE=${INSTALL_DIR}/lib/python2.7/site-packages

Now, make the directory tree.

mkdir -p $TREE

Compile

To compile the module, we should switch to the GNU compilers. The system installation of Python was compiled with the GNU compilers, and this will help avoid any unnecessary complications. We will also load the Python module, if it hasn't already been loaded.

module swap intel gnu
module load python

Now, build it. This step may vary a bit, depending on the module you are compiling. You can execute python setup.py --help to see what options are available. Since we are overriding the install path to one that we can write to, and that fits our management plan, we need to use the --prefix option.

python setup.py install --prefix=$INSTALL_DIR

Make it usable

At this point, the module is compiled and installed in ~/local/numexpr/2.0.1/lib/python2.7/site-packages . Occasionally, some files will be installed in ~/local/numexpr/2.0.1/bin as well. To ensure Python can locate these files, we need to modify our environment.

Manual

The most immediate way - but the one that must be repeated every time you wish to use the module - is to manually modify your environment. If files are installed in the "bin" directory, you'll need to add it to your path. As before, these examples are for bash, and may have to be modified for other shells. Also, you will have to modify the directories to match your install location.

export PATH=$PATH:~/local/numexpr/2.0.1/bin

And, for the python libraries:

export PYTHONPATH=$PYTHONPATH:~/local/numexpr/2.0.1/lib/python2.7/site-packages

Hardcode it

We don't really recommend this option, as it is less flexible, and can cause conflicts with system software. But, if you want, you can modify your .bashrc (or similar file, depending on your shell) to set these environment variables automatically. Be extra careful; making a mistake in .bashrc (or similar) can destroy your login environment in a way that will require a system administrator to fix. To do this, you can copy the lines above modifying $PATH and $PYTHONPATH into .bashrc. Remember - test them interactively first! If you destroy your shell interactively, the fix is as simple as logging out and then logging back in. If you break your login environment, you'll have to get our help to fix it.

Make a module (recommended!)

This is the most complicated option, but it is also the most flexible, as you can have multiple versions of this particular software installed, and specify at run-time which one to use. This is incredibly useful if a major feature changes that would break old code, for example. You can see our tutorial on writing modules here, but the important variables to modify are, again, $PATH and $PYTHONPATH . You should specify the complete path to your home directory here, and not rely on any shortcuts like ~ or $HOME .  Below is a modulefile written in Lua:

If you are following the tutorial on writing modules, you will want to place this file in $HOME/local/share/modulefiles/numexpr/2.0.1.lua :

-- This is a Lua modulefile, this file 2.0.1.lua can be located anywhere
-- But if you are following a local modulefile location convention, we place them in
-- $HOME/local/share/modulefiles/
-- For numexpr we place it in $HOME/local/share/modulefiles/numexpr/2.0.1.lua
-- This finds your home directory
local homedir = os.getenv("HOME")
prepend_path("PYTHONPATH", 
pathJoin(homedir, "/local/numexpr/2.0.1/lib/python2.7/site-packages"))
prepend_path(homedir, "local/numexpr/2.0.1/bin"))

 

Once your module is created (again, see the guide), you can use your python module simply by loaded the software module you created.

module use $HOME/local/share/modulefiles/
module load numexpr/2.0.1
Supercomputer: 
Service: 

HOWTO: Connect to OSC services using OSC Connect

 


NOTE:  OSC Ondemand now provides in-browser connectivity via VNC.  OSC Connect is no longer required nor recommended.  We recommend contacting OSC Help if you need assistance using the in-browser capabilities or if there is other functionality that is forcing you to still use OSC Connect.  This page is still available primarily for reference.


To connect to OSC services, a secure tunnel to a session is required. This can be done relatively simply in OSX and Linux by using the SSH functionality built into the system, but Windows users have had to configure and use third party applications like PuTTY or Java to access secure resources at OSC. OSC Connect is a native windows application written in C# and compiled for .NET 2.0, providing preconfigured management of secure tunnel connections for Windows users, as well as providing a launcher for secure file transfer, VNC, terminal, and web based services. 

OSC Connect is supported on Windows versions from Windows XP through Windows 10.

We've created a brief video:

 

 

Getting Started

  1. Download the latest release of OSC Connect from here: Latest Build
    • Click OSCConnect.exe  and download. Use "Save link as" to download this file to a folder of your choice. 
    • See here for all the previous releases. 
  2. Double-Click the OSCConnect.exe icon to run the application. In the current state, OSC Connect is entirely deployed by a single executable file; no further installation is required.   ​
  3. When you run OSC Connect, a temporary folder named "ConnectFiles" with four additional files will be created. These are required for proper functionality of the application. Please ensure that these files are permitted by your IT administrator:
    • plink.exe is the command-line version of PuTTY used by the application to create the secure connection to OSC resources.
    • putty.exe  is the GUI application of PuTTY used to provide terminal emulation remote console connections to OSC resources.
    • vncviewer.exe is the VNC viewer client used to view a remote desktop session.
    • WinSCP.exe  is the SFTP client used for file transfer.
  4. Once your connections to OSC services as well as the OSC Connect app is closed, the temporary folder named "ConnectFiles" will be removed automatically. 

OSC Connect Graphical User Interface

After you double-click the  OSCConnect.exe  icon oscconnect_icon , the application graphical user interface is shown as below:

 interface

  • Network Status: it indicates which OSC cluster you will be connected to. The option can be changed in "Settings". 

  • Settings: it provides several configuration options to modify the behavior of the application. 

    • Connection Settings: use this dropdown to select the default host/cluster. Selecting a server here will change the endpoint for tunneling, sftp connections, console connections, and connectivity checking.

    • System Settings

      • Detect Clipboard Activity: when this option is enabled, the application will detect valid data on the Windows clipboard and populate the application. ( Default: Off )

      • Check for New Versions: when this option is enabled, the application will check for version updates. (Default: on)

    • Automation Settings:

      • Save User Credentials: when this option is enabled, it allows the application to remember the user when the application is reopened. This saves the user credentials to the user settings using DPAPI Encryption. Passwords are decrypted only by current the Windows user account. ( Default: Off )

      • Launch Tunnel On Import: when this option is enabled, the tunnel will automatically connect when the application detects a valid clipboard string and the user credentials have been entered. ( Default: On )

    • VNC Settings

  • ​About: it provides additional information about OSC Connect.

After you provide your OSC Credentials, i.e. your OSC HPC username and password, more functionalities are available as shown below:

gui2

  • SFTP: it uses WinSCP as the SFTP client to securely connect to the OSC file system over SFTP.
  • Console: it uses PuTTY to provides you shell access to OSC cluster.

In addition, Session Type is provided such that advanced users can connect to a running session mannually.

For most users, you can ingore Session Type and are able to connect to a running session through OSC OnDemand using the configuration generated after your session starts. See "Connecting to a Session" for more details. 

Connecting to a Session

The OSC Connect application can be used to connect to a running session launched through OSC OnDemand.

OSC Connect does not have the ability to launch a new session
  1. Navigate to OSC Demand to create a new session under either "Desktops" or "Desktop Apps". 
    • More information on how to use OSC OnDemand can be found here
  2. Once your session is running, you have two options to connect to the session: "noVNC Instructions" and "Native Client Instructions". Click "Native Client Instructions" to use OSC Connect. 
  3. Follow the steps below to connect to the running session:
    • Step 1: Download the latest OSC Connect if you don't already have it. See "Getting Started" for more details. 
    • Step 2: Launch OSC Connect. 
    • Step 3: Click - osc://xxxxx . osc://xxxxx is a custom URI scheme that is registered when you launch the application. Simply click the link to populate the configuration information and connect to your running session. If OSCConnect.exe is not running when you click the URI, the OSC Connect application will be popped up. Enter your OSC HPC username and password and you will be able to connect to the session by clicking the "Connect" button.
    • For advanced users, you can skip step 3 and connect to a running session mannually. Depending on the type of application, you can choose either "iHPC VNC Desktop" to create a VNC tunnel or "Web Browser Session" to create a browser tunnel. 
Be sure to launch OSCConnect.exe at least once before you use it to connect to a running session. The initial launch will add a key to your user registry that initializes the URI scheme.  

FAQ

I've clicked the osc:// link and nothing happened.

Be sure to run OSCConnect.exe at least once. The initial launch will add a key to your user registry that initializes the URI scheme. If you move or rename the OSCConnect.exe file, you will need to run the application again manually to update the path in the handler.

I've received the error "Unable to open helper application. The protocol specified in this address is not valid."

This issue appears in some earlier versions of Internet Explorer when attempting to launch the application from a Temporary location. Download and run the OSCConnect.exe application, being sure to save the file to a non-temporary location.

 

 

Supercomputer: 
Service: 

HOWTO: Identify users on a project account and check status

An eligible principal investigator (PI) heads a project account and can authorize/remove user accounts under the project account (please check our Allocations and Accounts documentation for more details). This document shows you how to identify users on a project account and check the status of each user. 

Identify Users on a Project Account

If you know the project acccount

If you know the project account (projectID), the following command will list all users on the project:

getent group projectID

The returned information is in the format of:

projectID:*:gid: list of user IDs

gid is the group identifier number unique for the project account projectID. 

For example, the command  getent group PZS0530  lists all users on the project account PZS0530 as below:

bash-4.1$ getent group PZS0530
PZS0530:*:2959:yli,osc0539,elton,ananth,osc0413,dhudak,osc0695,ksaantha,changlee,buss,osc0414,osc0478,nsharma,kmanalo,nwadih,bsmith,...

If you don't know the project acccount, but know the user account

If you don't know the project account, but know the user account, userID, use the  groups  command to list all of the groups the userID belongs to:

groups userID

The returned information is in the format of:

userID : list of groups

where the first item after " userID : " is the primary group, i.e. the project account (projectID) which userID is on. Once you know the project account, use the command  getent group projectID  as discussed in the previous session to list all users on this project.

For example, with the userID as xwang, the command groups xwang  returns the information as below:

bash-4.1$ groups xwang
xwang : oscgen lstc gaussian ...

It lists all groups xwang belongs to, and oscgen is the project account which xwang is on. The command  getent group oscgen  lists all user accounts on the project account oscgen:

bash-4.1$ getent group oscgen
oscgen:*:200:jordan,jpu,mlewis,rmonroe,rmarshal,spears,sengupta,jtm,adraghi,lepage,ian,karin,remote,njustice,bedford,hjiang,tom,mudronja,elainep,gisadmin,airani,guilfoos,osc0498,osc0722,ngagnet,mfaerman,justinw,arya,mattm,echong,rahmed,jwright,...

If you don't know either the project acccount or user account

If you don't know either the project account or user account, you can use ldapsearch  command to get the user account based on the user's registration information such as name, email address, etc. 

Use the following command to list all of the user accounts accociated with the name First Last:

ldapsearch -x -LLL "(gecos=First Last)" | grep cn | awk '{print $2}'

Use the following command to list all of the user accounts accociated with the email address email@address:

ldapsearch -x -LLL "(mail=email@address)" | grep cn | awk '{print $2}'

For example, with user's first name as Summer and last name as Wang, the command

ldapsearch -x -LLL "(gecos=Summer Wang)" | grep cn | awk '{print $2}' returns the information as below:

bash-4.1$ ldapsearch -x -LLL "(gecos=Summer Wang)" | grep cn | awk '{print $2}'
xwang
xwangnd
... 

With user's email address as xwang@osc.edu, the command ldapsearch -x -LLL "(mail=xwang@osc.edu)" | grep cn | awk '{print $2}' returns the information as below:

bash-4.1$ ldapsearch -x -LLL "(mail=xwang@osc.edu)" | grep cn | awk '{print $2}'
xwang
xwangnd
...

Once you know the user account userID, follow the discussions in previous session to get all user accounts on the project. Please contact OSC Help if you have any questions. 

Check the Status of a User

Use the  finger  command to check the status of  a user account userID as below:

finger userID

For example, if the userID is xwang, the command  finger xwang  will return:

bash-4.1$ finger xwang
Login: xwang                            Name: Summer Wang
Directory: /users/oscgen/xwang          Shell: /bin/bash
On since Thu Jan 25 15:32 (EST) on pts/49 from localhost:25.0
   6 seconds idle
Mail forwarded to xwang@osc.edu
No mail.
No Plan.
  • The home directory of xwang is  Directory: /users/oscgen/xwang 
  • The shell of xwang is bash ( Shell: /bin/bash ). If the information is Shell:/access/denied , it means this user account has been either archived or resticted. Please contact OSC Help if you'd like to reactivate this user account.
  • xwang@osc.edu is the associated email with the user account xwang; that is, all OSC emails related to the account xwang will be sent to xwang@osc.edu ( Mail forwarded to xwang@osc.edu ). Please contact OSC Help if the email address asscoiated with this user account has been changed to ensure important notifications/messages/remindars from OSC may be received in a timely manner.

Check the Usage and Quota of a User's Home Directory/Project's Project Space

All users see their file system usage staticstics when logging in, like so:

As of 2018-01-25T04:02:23.749853 userid userID on /users/projectID used XGB of quota 500GB and Y files of quota 1000000 files

The information is from the file /users/reporting/storage/quota/*_quota.txt , which is updated twice a day. Some users may see multiple lines asscoiated with userid, as well as information on project space usage and quota, if there is one. The usage and quota of the home diretory of userID is provided by the line including the file server your home directory is on (for more information, please visit Home Directories), while others (generated due to file copy) can be safely ingored. 

You can check any user's home directory or a project's project space usage and quota by running:

grep -h <userID OR projectID > /users/reporting/storage/quota/*_quota.txt 
Supercomputer: 
Service: 

HOWTO: Install Local R Packages

This document shows you the steps to install R packages locally without root access on OSC's clusters. 

R comes with a single library  $R_HOME/library which contains the standard and recommended packages. This is usually in a system location. On Owens, it is   /usr/local/R/3.3.2/lib64/R/library . R also has a default value for a directory where users can install their own R packages. On the Owens cluster, it is ~/R/x86_64-unknown-linux-gnu-library/3.3 if the default R-3.3.2 module is loaded. This directory doesn't exist by default. The first time a user installs an R package, R will ask the user if s/he wants to use the default location and if yes, will create the directory.

A Simple Example

First you need to load the module for R:

module load R

On Owens, the default R module is version 3.3.2 .

Then fire up an R session:

R

To install package lattice, use this command inside R:

> install.packages("lattice", repos="http://cran.r-project.org")

It gives a warning: 

Warning in install.packages("lattice") :
'lib = "/usr/local/R/3.3.2/lib64/R/library"' is not writable
Would you like to create a personal library
~/R/x86_64-unknown-linux-gnu-library/3.3
to install packages into?  (y/n) 

Answer y , and it will create the directory and install the package there.

Setting the Local R Library Path

If you want to use another location rather than the default location, for example, ~/local/R_libs/ ,  you need to create the directory first:

mkdir ~/local/R_libs

Then type the following command inside R:

> install.packages("lattice", repos="http://cran.r-project.org", lib="~/local/R_libs/")

It is a bit of burden having to type the long string of library path every time. To avoid doing that, you can create a file .Renviron in your home directory, and add the following line to the file:

export R_LIBS=~/local/R_libs/ 

Whenever R is started, the directory ~/local/R_libs/ is added to the list of places to look for R packages and so:

> install.packages("lattice", repos="http://cran.r-project.org")

will have the same effect as the previous install.packages() command. 

To see the directories where R searches for libraries, use the command:

>.libPaths();

Setting The Repository

When you install an R package, you are asked which repository R should use. To set the repository and avoid having to specify this at every package install, create a file .Rprofile in your home directory. This is the start up code for R. Add the following line to the file:

cat(".Rprofile: Setting R repository:")
repo = getOption("repos") 
# set up the server from which you will download the package.
repo["CRAN"] = "http://cran.case.edu" 
options(repos = repo)
rm(repo)

Now you only need to do 

> install.packages("lattice")

That will download the package lattice from  http://cran.case.edu and install it in ~/local/R_libs  .

Updating Packages

> update.packages() inside an R session is the simplest way to ensure that all the packages in your local R library are up to date. It downloads the list of available packages and their current versions, compares it with those installed and offers to fetch and install any that have later versions on the repositories.

Note that attempting to update centrally installed packages will fail. You will have to use install.packages() to install a newer copy in your home directory instead. For example, if you are installing Bioconductor, the installation routine will try to update a bunch of centrally installed packages. To clone those into your personal library, tell biocLite() not to update any of the packages, and copy the list from the output into your own install.packages() command. See the example below (the c function creates an array of strings).

> install.packages(c('Rmpi', 'assertthat', 'backports'))

Removing packages

> remove.packages("lattice") inside an R session to remove package lattice . An even easier way is just to go into the directory ~/local/R_libs and remove the directory lattice from there.

 

Installing Packages from GitHub

Users can install R packages directly from Github using devtools package as follows

module load R/3.4.2
R

           

>library(devtools)
>install_github("author/package")

 

References

Add-on packages in R installation guide (http://cran.r-project.org/doc/manuals/R-admin.pdf)

Supercomputer: 
Service: 

HOWTO: Install a MATLAB toolbox

If you need to use a MATLAB toolbox that is not provided through our installations. You can follow these instructions, and if you have any difficulties you can contact OSC Help for assistance.

A reminder: It is your responsibility to verify that your use of software packages on OSC’s systems including any 3rd party toolboxes (whether installed by OSC staff or by yourself)  complies with the packages’ license terms.

Gather your materials

First, we recommend making a new directory within your home directory in order to keep everything organized. You can use the unix command to make a new directory: "mkdir"

Now you can download the toolbox either to your desktop, and then upload it to OSC, or directly download it using the "wget" utility (if you know the URL for the file).

Now you can extract the downloaded file.

Adding the path

There are two methods on how to add the MATLAB toolbox path.

Method 1: Load up the Matlab GUI and click on "Set Path" and "Add folder"

Method 2: Use the "addpath" fuction in your script. More information on the function can be found here: https://www.mathworks.com/help/matlab/ref/addpath.html

Running the toolbox

Please refer to the instructions given alongside the toolbox. They should contain instructions on how to run the toolbox.

Supercomputer: 
Service: 
Technologies: 
Fields of Science: 

HOWTO: Install your own Perl modules

While we provide a number of Perl modules, you may need a module we do not provide. If it is a commonly used module, or one that is particularly difficult to compile, you can contact OSC Help for assistance, but we have provided an example below showing how to build and install your own Perl modules. Note, these instructions use "bash" shell syntax; this is our default shell, but if you are using something else (csh, tcsh, etc), some of the syntax may be different.

CPAN Minus

CPAN, the Comprehensive Perl Achive Network, is the primary source for publishing and fetching the latest modules and libraries for the Perl programming language. The default method for installing Perl modules using the "CPAN Shell", provides users with a great deal of power and flexibility but at the cost of a complex configuration and inelegant default setup.

 

Setting Up CPAN Minus

To use CPAN Minus, we must first load it, if it hasn't already been loaded. Note that this is not necessary if you loaded a version of Perl with the module load command.

module load cpanminus

Next, in order to use cpanminus, you will need to run the following command only ONCE:

perl -I $CPANMINUS_INC -Mlocal::lib

 

Using CPAN Minus

In most cases, using CPAN Minus to install modules is as simple as issuing a command in the following form:

cpanm [Module::Name]

For example, below are three examples of installing perl modules:

cpanm Math::CDF
cpanm SET::IntervalTree
cpanm DB_File

 

Testing Perl Modules

To test a perl module import, here are some examples below:

perl -e "require Math::CDF"
perl -e "require Set::IntervallTree"
perl -e "require DB_File"

The modules are installed correctly if no output is printed.

What Local Modules are Installed in my Account?

To show the local modules you have installed in your user account:

perldoc perllocal

Reseting Module Collection

If you should ever want to start over with your perl module collection, delete the following folders:

rm -r ~/perl5 
rm -r ~/.cpanm

 

Supercomputer: 
Service: 

HOWTO: Locally Installing Software

Sometimes the best way to get access to a piece of software on the HPC systems is to install it yourself as a "local install". This document will walk you through the OSC-recommended procedure for maintaining local installs in your home directory or project space.

NOTE: Throughout this document we'll assume you're installing into your home directory, but you can follow the steps below in any directory for which you have read/write permissions.
This document assumes you are familiar with the process of building software using "configure" or via editing makefiles, and only provides best practices for installing in your home directory.

Getting Started

Before installing your software, you should first prepare a place for it to live. We recommend the following directory structure, which you should create in the top-level of your home directory:

    local
    |-- src
    |-- share
        `-- lmodfiles

This structure is how OSC organizes the software we provide. Each directory serves a specific purpose:

  • local - Gathers all the files related to your local installs into one directory, rather than cluttering your home directory. Applications will be installed into this directory with the format "appname/version". This allows you to easily store multiple versions of a particular software install if necessary.
  • local/src - Stores the installers -- generally source directories -- for your software. Also, stores the compressed archives ("tarballs") of your installers; useful if you want to reinstall later using different build options.
  • local/share/lmodfiles - The standard place to store module files, which will allow you to dynamically add or remove locally installed applications from your environment.

You can create this structure with one command.

NOTE: Ensure $HOME is the full path of your home directory. You can identify this from the command line with the command echo $HOME.

After navigating to where you want to create the directory structure, run:

    mkdir -p $HOME/local/src $HOME/local/share/lmodfiles

Installing Software

Now that you have your directory structure created, you can install your software. For demonstration purposes, we will install a local copy of Git.

First, we need to get the source code onto the HPC filesystem. The easiest thing to do is find a download link, copy it, and use the wget tool to download it on the HPC. We'll download this into $HOME/local/src:

    cd $HOME/local/src
    wget https://github.com/git/git/archive/v2.9.0.tar.gz

Now extract the tar file:

    tar zxvf v2.9.0.tar.gz

Next, we'll go into the source directory and build the program. Consult your application's documentation to determine how to install into $HOME/local/"software_name"/"version". Replace "software_name" with the software's name and "version" with the version you are installing, as demonstrated below. In this case, we'll use the configure tool's --prefix option to specify the install location.

You'll also want to specify a few variables to help make your application more compatible with our systems. We recommend specifying that you wish to use the Intel compilers and that you want to link the Intel libraries statically. This will prevent you from having to have the Intel module loaded in order to use your program. To accomplish this, add CC=icc CFLAGS=-static-intel to the end of your invocation of configure. If your application does not use configure, you can generally still set these variables somewhere in its Makefile or build script.

Then, we can build Git using the following commands:

    cd git-2.9.0
    autoconf # this creates the configure file
    ./configure --prefix=$HOME/local/git/2.9.0 CC=icc CFLAGS=-static-intel
    make && make install

Your application should now be fully installed. However, before you can use it you will need to add the installation's directories to your path. To do this, you will need to create a module.

Creating a Module

Modules allow you to dynamically alter your environment to define environment variables and bring executables, libraries, and other features into your shell's search paths.

We will be useing the filename 2.9.0.lua ("version".lua). A simple Lua module for our Git installation would be:

-- Local Variables
local name = "git"
local version = "2.9.0"

-- Locate Home Directory
local homedir = os.getenv("HOME")
local root = pathJoin(homedir, "local", name, version)

-- Set Basic Paths
prepend_path("PATH", pathJoin(root, "bin"))
prepend_path("MANPATH", pathJoin(root, "share/man"))

NOTE: For future module files, copy our sample modulefile from ~support/doc/modules/sample_module.lua. This module file follows the recommended design patterns laid out above and includes samples of many common module operations

Oakley, Ruby, and Owens use a Lua based module system. However, there is another module system based in TCL that will not be discussed in this HOWTO.                                                                         NOTE: TCL is cross-compatible and is converted to Lua when loaded. More documentation is available at https://www.tacc.utexas.edu/research-development/tacc-projects/lmod/ or by executing module help.

Initializing Modules

Any module file you create should be saved into your local lmodfiles directory ($HOME/local/share/lmodfiles). To prepare for future software installations, create a subdirectory within lmodfiles named after your software and add one module file to that directory for each version of the software installed.

In the case of our Git example, you should create the directory $HOME/local/share/lmodfiles/git and create a module file within that directory named 2.9.0.lua.

To make this module usable, you need to tell lmod where to look for it. You can do this by issuing the command module use $HOME/local/share/lmodfiles in our example. You can see this change by performing module avail. This will allow you to load your software using either module load git or module load git/2.9.0

NOTE: module use$HOME/local/share/lmodfiles and module load "software_name" needs to be entered into the command line every time you enter a new session on the system.

If you install another version later on (lets say version 2.9.1) and want to created a module file for it, you need to make sure you call it 2.9.1.lua. When loading Git, lmod will automatically load the newer version. If you need to go back to an older version, you can do so by specifying the version you want: module load git/2.9.0.

To make sure you have the correct module file loaded, type which git which should give the output "~/local/git/2.9.0/bin/git" (NOTE: ~ is equivalent to $HOME). 

To make sure the software was installed correctly and that the module is working, type git --version which shuold give the output "git version 2.9.0".

Further Reading

For more information about modules, be sure to read the webpage indicated at the end of module help. If you have any questions about modules or local installations, feel free to contact the OSC Help Desk and oschelp@osc.edu.

Supercomputer: 
Service: 

HOWTO: Reduce Disk Space Usage

This HOWTO will demonstrate how to lower ones' disk space usage. The following procedures can be applied to all of OSC's file systems.

We recommend users regularly check their data usage and clean out old data that is no longer needed.

Users who need assistance lowering their data usage can contact OSC Help.

Preventing Excessive Data Usage Before It Starts

Users should ensure that their jobs are written in such a way that temporary data is not saved to permanent file systems, such as the project space file system or their home directory.

If your job copies data from the scratch file system or its node's local disk ($TMPDIR) back to a permanent file system, such as the project space file system ( /fs/project/PXX####/) or a home directory ( /users/PXX####/xxx####/), you should ensure you are only copying the files you will need later.

Identifying Old and Large Data

The following commands will help you identify old data using the find command.

find commands may produce an excessive amount of output. To terminate the command while it is running, click CTRL + C.

Find all files in a directory that have not been accessed in the past 100 days:

This command will recursively search the users home directory and give a detailed listing of all files not accessed in the past 100 days.

The last access time atime is updated when a file is opened by any operation, including grep, cat, head, sort, etc.

find ~ -atime +100 -exec ls -l {} \;
  • To search a different directory replace ~ with the path you wish to search. A period . can be used to search the current directory.
  • To view files not accessed over a different time span, replace 100 with your desired number of days.
  • To view the total size in bytess of all the files found by find, you can add | awk '{s+=$5} END {print "Total SIZE (bytes): " s}' to the end of the command:
find ~ -atime +100 -exec ls -l {} \;| awk '{s+=$5} END {print "Total SIZE (bytes): " s}'

Find all files in a directory that have not been modified in the past 100 days:

This command will recursively search the users home directory and give a detailed listing of all files not modified in the past 100 days.

The last modified time mtime is updated when a file's contents are updated or saved. Viewing a file will not update the last modified time.

find ~ -mtime +100 -exec ls -l {} \; 
  • To search a different directory replace ~ with the path you wish to search. A period . can be used to search the current directory.
  • To view files not modified over a different time span, replace 100 with your desired number of days.
  • To view the total size in bytes of all the files found by find, you can add | awk '{s+=$5} END {print "Total SIZE (bytes): " s}' to the end of the command:
find ~ -mtime +100 -exec ls -l {} \;| awk '{s+=$5} END {print "Total SIZE (bytes): " s}'

List files larger than a specified size:

Adding the -size <size> option and argument to the find command allows you to only view files larger than a certain size. This option and argument can be added to any other find command.

For example, to view all files in a users home directory that are larger than 1GB:

find ~ -size 1G -exec ls -l {} \;

Deleting Identified Data

CAUTION: Be careful when deleting files. Be sure your command will do what you want before running it. Extra caution should be used when deleting files from a file system that is not backed up, such as the scratch file system.

If you no longer need the old data, you can delete it using the rm command.

If you need to delete a whole directory tree (a directory and all of its subcontents, including other directories), you can use the rm -R command.

For example, the following command will delete the data directory in a users home directory:

rm -R ~/data

If you would like to be prompted for conformation before deleting every file, use the -i option.

rm -Ri ~/data 

Enter y or n when prompted. Simply pressing the enter button will default to n.

Deleting files found by find

The rm command can be combined with any find command to delete the files found. The syntax for doing so is:

find <location> <other find options> -exec rm -i {} \;

Where <other find options> can include one or more of the options -atime <time>, -mtime <time>, and -size <size>.

The following command would find all files in the ~/data directory 1G or larger that have not been accessed in the past 100 days, and then prompt for confirmation to delete each file:

find ~/data -atime +100 -size 1G -exec rm -i {} \;

If you are absolutely sure the files identified by find are okay to delete you can remove the -i option to rm and you will not be prompted. Extreme caution should be used when doing so!

Archiving Data

If you still need the data but do not plan on needing the data in the immediate future, contact OSC Help to discuss moving the data to a archive file system. Requests for data to be moved to the archive file system should be larger than 1TB.

Compressing

If you need the data but do not access the data frequently, you should compress the data using tar or gzip.

Moving Data to a Local File System

If you have the space available locally you can transfer your data there using sftp or Globus.

Globus is recommended for large transfers.

The OnDemand File application should not be used for transfers larger than 1GB.

Supercomputer: 
Service: 

HOWTO: Submit multiple jobs using parameters

Often users want to submit a large amount of jobs all at once with each using different parameters for each job.  These parameters could be anything, including the path of a data file or different input values for a program.  This how-to will show you how you can do this using a simple python script, a CSV file, and a template script.  You will need to adapt this advice for your own situation.

 

Consider the following batch script:

#PBS -l nodes=1:ppn=12
#PBS -l walltime=80:00:00
#PBS -n week42_data8

# Copy input data to the nodes fast local disk 
cp ~/week42/data/source1/data8.out $TMPDIR
cd $TMPDIR

# Run the analysis 
full_analysis data8.in data8.out

# Copy results to proper folder
cp  data8.out ~/week42/results

Lets say you need to submit 100 of these jobs on a weekly basis.  Each job uses a different data file as input.  You recieve data from two different sources, and thus your data is located within two different folders.  All of the jobs from one week need to store their results in a single weekly results folder.  The output file name is based upon the input file name.

Creating a Template Script

As you can see, this job follows a general template.  There are three main parameters that change in each job:

  1. The week 
    • Used as part of the job name
    • Used to find the proper data file to copy to the nodes local disk
    • Used to copy the results to the correct folder
  2. The data source
    • Used to find the proper data file to copy to the nodes local disk
  3. The data file's name
    • Used as part of the job name
    • Used to find the proper data file to copy to the nodes local disk
    • Used to specify both the input and output file to the program full_analysis
    • Used to copy the results to the correct folder

If we replace these parameters with variables, prefixed by the dollar sign $and surrounded by curly braces { }, we get the following template script:

#PBS -l nodes=1:ppn=12
#PBS -l walltime=80:00:00
#PBS -n ${WEEK}_${DATA}

# Copy input data to the nodes fast local disk 
cp ~/${WEEK}/data/${SOURCE}/${DATA}.out $TMPDIR
cd $TMPDIR

# Run the analysis 
full_analysis ${DATA}.in ${DATA}.out

# Copy results to proper folder
cp  ${DATA}.out ~/${WEEK}/results

Automating Job Submission

We can now use qsub's -v option to pass parameters to our template script.  The format for passing parameters is:

qsub -v par_name=par_value[,par_name=par_value...] script.sh

Submitting a 100 jobs using qsub -v option manually does not make our task much easier than modifying and submitting each job one by one.  To complete our task we need to automate the submission of our jobs.  We will do this by using a python script that submits our jobs using parameters it reads from a CSV file.  

Note that python was chosen for this task for its general ease of use and understandability -- if you feel more comfortable using another scritping language feel free to interpret/translate this python code for your own use.

Here is the script that submits the jobs using the parameters:

#!/usr/bin/env python
import csv, subprocess

parameter_file_full_path = "/nfs/12/user0123/week42/job_params.csv"

with open(parameter_file_full_path, "rb") as csvfile:
    reader = csv.reader(csvfile)
    for job in reader:
        qsub_command = """qsub -v WEEK={0},SOURCE={1},DATA={2} template_1.pbs""".format(*job)

        #print qsub_command # Uncomment this line when testing to view the qsub command

        # Comment the following 3 lines when testing to prevent jobs from being submitted
        exit_status = subprocess.call(qsub_command, shell=True)
        if exit_status is 1:  # Check to make sure the job submitted
            print "Job {0} failed to submit".format(qsub_command)
print "Done submitting jobs!"

This script will open the CSV file specified by the variable parameter_file_full_path and step through the file line by line, submitting a job for each line using the lines values.  If the qsub command returns a non-zero exit code, usually indicating it was not submitted, we will print this out to the display.  The jobs will be submitted using the general format:

qsub -v WEEK=WEEK_VALUE,SOURCE=SOURCE_VALUE,DATA=DATA_VALUE template_1.pbs

Where WEEK_VALUE ,SOURCE_VALUE, and DATA_VALUE are the first, second, and third comma separated values in the CSV file's current row, and template_1.pbs is the name of our template script.

Creating a CSV File

We now need to create a CSV file with parameters for each job.  This can be done with a regular text editor or using a spreadsheet editor such as excel.  By default you should use commas as your delimiter.  

Here is our CSV file with parameters:

week42,source1,data1
week42,source1,data2
week42,source1,data3
...
week42,source2,data98
week42,source2,data99
week42,source2,data100

The submit script would read in the first row of this CSV file and form and  submit the following qsub command:

qsub -v WEEK=week42,SOURCE=source1,DATA=data1 template_1.pbs

Submitting Jobs

Once all the above is done all you need to do to submit your jobs is make sure the CSV file is populated with the proper parameters and run the automatic submission script.  

Before submitting a large amount of jobs for the first time using this method it is recommended you test our your implimentation with a small amount of jobs.

HOWTO: Transfer files using Globus Connect

Globus Connect Logo

Globus Connect is a reliable, high-performance file transfer platform allowing users to transfer large amounts of data seamlessly between systems.  It aims to make transfers a "click-and-forget" process by setting up configuration details in the background and automating fault recovery.  

Globus can be used for both file transfers between OSC and:

  • A computing institution with Globus installed (check with your site provider for availability) or
  • A personal computer (known as a personal endpoint)

Users transferring between OSC and another computing institution with​ Globus installed do not need to install Globus Connect Personal, and can skip to Usage.

More on how Globus works can be found on the Globus "How It Works" page.

If you are looking to transfer smaller sized files you can utilize OnDemand's file transfer capabilities, or use a SFTP client to connect to  sftp.osc.edu . Our general recommendation is that for small files - measured in MB to several hundred MB - to use OnDemand or SFTP. You can continue to use SFTP and get reasonable performance up to file sizes of several GB. For transfers of several GB or larger, you should consider using Globus Online.

Install Globus Connect Personal

To use Globus to transfer from a personal computer, you will need to install the Globus Connect Personal client on your computer following the steps below. Those transferring between OSC and another computing institution can skip to Usage.

  1. Sign up for a free Globus account
  2. Download the Globus Connect Personal Client 
    • Click "Manage Endpoints" under the "Manage Transfers/Data" menu
    • Click "add Globus Connect" on the top-right of the page
    • Choose a unique name for your endpoint and generate the setup key
    • Download the Globus Connect client for your operating system
  3. Install Globus Connect Personal Client
    • Windows
      1. Run the Installer
      2. Copy-Paste the setup key to complete the installation
    • Mac​​
      1. Mount your drives
      2. Copy the Globus Client to your application Folder
      3. Start The Globus Client, and enter the provided setup key
    • Linux
      1. Un-tar the .tgz file with the command tar -zxvf
      2. Run globusconnect , found within the unzipped directory
      3. Copy-Paste the setup key when prompted
  4. (Optional) Changing directories accessible to Globus

By default Globus will only add certain default folders to the list of files and directories accessible by Globus. To change/add/remove files and directories from this list:

Windows

  1. Start Globus Connect Personal
  2. Go to "Options"
  • Add directories/files using the  "+" button
  • Remove directories/files using the "-" button
  • Revert to the default accessible directories/files using the "Reset to Defaults" button
  • Any changes you make are not made permanent until you press the "Save" button

​​Mac

  1. Start Globus Connect Personal
  2. Go to Preferences -> Access
  • Add directories/files using the  "+" button
  • Remove directories/files using the "-" button
  • Revert to the default accessible directories/files using the "Reset to Defaults" button
  • Any changes you make are not made permanent until you press the "Save" button

​Linux

  1. Configure the   ~/.globusonline/lta/config-paths  file. This file is a plain text file, with each line corresponding to the configuration of a particular directory path you wish to make accessible. Each line consists of 3 comma separated fields as below
    <path1>,<sharing flag>,<R/W flag>
    <path2>,<sharing flag>,<R/W flag>
    <path3>,<sharing flag>,<R/W flag>
    ...
    

    Path: an absolute directory/path to be permitted. A leading tilde "~" can be used to represent the home directory of the user that runs globusconnectpersonal.

    Sharing Flag: it controls sharing, with a value of "1" allowing sharing for the path and a value of "0" disallowing sharing for the path.

    R/W Flag: it determines whether the path will be accessible read-only or read/write, with a "1" permitting read/write access and a "0" marking the path as read-only. 

  2. After making a change to the ~/.globusonline/lta/config-paths file, you must stop and restart Globus Connect Personal before the changes will take effect as below
$ ./globusconnectpersonal -stop
$ ./globusconnectpersonal -start &

Usage

  1. Login to Globus and navigate to the "Transfer Files" under the "Manage Data" menu
  2. Enter your end point in one of the boxes
    • If transferring to a computer with Globus Connect Personal installed, this will be the unique name chosen during installation. It is also listed under "Administered by Me"
  3. Enter osc#gcs (OSC Globus Connect Server) as the other endpoint
    • Enter your OSC username and password for the authetication
  4. You can now transfer files and directories both ways by selecting them and pressing the arrow indicating which way you'd like to transfer
If you are doing a large transfer you should transfer to/from the parallel file system or project space for best performance.
Once a transfer has begun, you do not need to keep the Globus webpage up, but you will need to make sure the Globus Connect Personal Client is running on your computer until it has completed.  If the transfer is interrupted for any reason, Globus will attempt to re-initiate the transfer automatically.

Further Reading

HOWTO: Add InCommon Authentication to Globus

(OPTIONAL) Adding InCommon Authentication 

Adding InCommon authentication to your Globus account allows you to login to Globus Online using your university credentials.  Using this process you can store your Globus username password for safe keeping, and instead use your university username and password to login.  If your already logged in to your university authentication system, logging in to Globus can be as simple as two clicks away.

To use this feature, your university needs to be a InCommon participant.  Some Ohio universities active in InCommon include: Ohio State University, Case Western University, Columbus State Community College, Miami University, Ohio Northern University, Ohio University, University of Findlay, University of Dayton, and many more.  

For a complete list, visit https://incommon.org/participants/ .

To add InCommon Authentication:
  1. Login to Globus Online
  2. Go to "Manage Identities" under your username
  3. Click "Add External Identity"
    • Choose a name for the Identity Settings. 
    • Choose InCommon / CILogon from the drop down menu
  4. On the next page, choose your University / Identity Provider
  • Click "Remember this selection"
  • Click "Log on"
  • You may be prompted to login to your university authentication system if you are not already

When you go to login next, click "alternative login" and then "InCommon / CILogon".  Select your university on the next page, and login using your university credentials.  Globus will remember this preference, and automatically prompt you to login using your university authentication next time.

HOWTO: Use Docker and Singularity Containers at OSC

It is now possible to run Docker and Singularity containers on the Owens cluster at OSC. Single-node jobs are currently supported, including GPU jobs; MPI jobs are planned for the future.

From the Docker website:  "A container image is a lightweight, stand-alone, executable package of a piece of software that includes everything needed to run it: code, runtime, system tools, system libraries, settings."

This document will describe how to run Docker and Singularity containers on Owens. You can use containers from Docker Hub, Singularity Hub, or any other source. As examples we will use vsoch/singularity-hello-world and ubuntu from Docker Hub.

If you want to create your own container or modify a container, you will have to do it on a system where you have root access. Or, you can use Singularity Hub to create a container from a recipe. But, You cannot build or update a container on Owens or any other OSC system.

Getting help

The most up-to-date help on Singularity comes from the command itself.

singularity help
singularity help exec

The Singularity web site has documentation including a User Guide and examples.

Setting up your environment for Singularity usage

To load the Singularity environment on Owens, run the following command:

module load singularity

Accessing a container

A Singularity container is a single file with a .img or .simg extension.

You can simply download ("pull") a container from a hub. Popular hubs are Docker Hub and Singularity Hub. You can go there and search if they have a container that meets your needs. Docker Hub has more containers and may be more up to date but supports a much wider community than just HPC. Singularity Hub is for HPC, but the number of available containers are fewer. Additionally there are domain and vendor repositories such as biocontainers and NVIDIA HPC containers that may have relevant containers.

Examples of “pulling” a container from the both hubs:

Docker Hub:  Pull from the 7.2.0 branch of the gcc repository on Docker Hub. The 7.2.0 is called a tag.

singularity pull docker://gcc:7.2.0

Filename:  gcc-7.2.0.img

When you pull from docker, there will be a message:

WARNING: pull for Docker Hub is not guaranteed to produce the
WARNING: same image on repeated pull. Use Singularity Registry
WARNING: (shub://) to pull exactly equivalent images.

As we mentioned, containers from Singularity Hub are more likely to be compatible with our container environment. But, since Docker Hub has more updated versions, it is more likely to have a container that you are looking for.

Singularity Hub:  Pull the vsoch/singularity-hello-world container from the Singularity hub. Since no tag is specified it pulls from the master branch of the repository.

singularity pull shub://vsoch/hello-world

Filename:  vsoch-hello-world-master.img

Example:  Pull an Ubuntu container from Docker Hub.

singularity pull docker://ubuntu

Filename:  ubuntu.img

Downloading containers from the hubs is not the only way to get one. You can, for example get a copy from your colleague's computer or directory. If you would like to create your own container you can start from the user guide below. If you have any questions, please contact OSC Help.

Running a container

There are four ways to run a container under Singularity.

You can do this either in a batch job or on a login node. (Don’t run on a login node if the container will be performing heavy computation, of course.)

We note that the operating system on Owens is Red Hat:

[owens-login01]$ cat /etc/os-release
NAME="Red Hat Enterprise Linux Server"
VERSION="7.3 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="7.3"
PRETTY_NAME="Red Hat Enterprise Linux"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.3:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.3
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.3"

In the examples below we will often check the operating system to show that we are really inside a container.

Run container like a native command

If you simply run the container image it will execute the container’s runscript.

Example:  Run an Ubuntu shell. (See also the “shell” sub-command below.)

[owens-login01]$ ./ubuntu.img
[owens-login01]$ cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04.3 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.3 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial
[owens-login01]$ exit
exit
[owens-login01]$

Example:  Run vsoch/hello-world

Note that this container returns you to your native OS after you run it.

[owens-login01]$ ./vsoch-hello-world-master.img
RaawwWWWWWRRRR!!
[owens-login01]$

Use the “run” sub-command

The Singularity “run” sub-command does the same thing as running a container directly as described above. That is, it executes the container’s runscript.

Example:  Run a container from a local file

[owens-login01]$ singularity run vsoch-hello-world-master.img
RaawwWWWWWRRRR!!
[owens-login01]$

Example:  Run a container from a hub without explicitly downloading it

[owens-login01]$ singularity run shub://vsoch/hello-world
Progress |===================================| 100.0%
RaawwWWWWWRRRR!!
[owens-login01]$

Use the “exec” sub-command

The Singularity “exec” sub-command lets you execute an arbitrary command within your container instead of just the runscript.

Example:  Find out what operating system the vsoch/hello-world container uses

[owens-login01]$ singularity exec vsoch-hello-world-master.img cat /etc/os-release
NAME="Ubuntu"
VERSION="14.04.5 LTS, Trusty Tahr"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 14.04.5 LTS"
VERSION_ID="14.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
[owens-login01]$ 

Use the “shell” sub-command

The Singularity “shell” sub-command invokes an interactive shell within a container.

Example:  Examine the contents of the vsoch/hello-world container

Note:  A container’s runscript is the file /singularity within the container.

[owens-login01]$ singularity shell vsoch-hello-world-master.img
Singularity: Invoking an interactive shell within container...
Singularity vsoch-hello-world-master.img:~/singularity> ls /
bin   dev  home  lib64       media  opt   rawr.sh  run   singularity  sys  users  var
boot  etc  lib   lost+found  mnt    proc  root     sbin  srv          tmp  usr
Singularity vsoch-hello-world-master.img:~/singularity> cat /singularity
#!/bin/sh
exec /bin/bash /rawr.sh
Singularity vsoch-hello-world-master.img:~/singularity> cat /rawr.sh
#!/bin/bash
echo "RaawwWWWWWRRRR!!"
Singularity vsoch-hello-world-master.img:~/singularity> exit
exit
[owens-login01]$

Example:  Run an Ubuntu shell. Note the “Singularity” prompt within the shell.

[owens-login01]$ singularity shell ubuntu.img
Singularity: Invoking an interactive shell within container...
Singularity ubuntu.img:~/singularity> cat /singularity
#!/bin/sh
exec /bin/bash "$@"Singularity ubuntu.img:~/singularity> exit
exit
[owens-login01]$

File system access

When you use a container you run within the container’s environment.  The directories available to you by default from the host environment (Owens) are

  • your home directory
  • working directory (directory you were in when you ran the container)
  • /fs/project
  • /fs/scratch
  • /tmp

You can review our Available File Systems page for more details about our file system access policy.

If you run the container within a job you will have the usual access to the $PFSDIR environment variable with adding node attribute "pfsdir" in the job request (nodes=XX:ppn=XX:pfsdir). You can access most of our file systems from a container without any special treatment, however, if you need TMPDIR you must specify it on the command line as in this example:

SINGULARITYENV_TMPDIR=$TMPDIR singularity shell ubuntu.img
Note: Most environment variables are passed into the container by default, but for obscure reasons $TMPDIR is not. The above syntax passes it in explicitly.

GPU usage within a container

If you have a GPU-enabled container you can easily run it on Owens just by adding the --nv flag to the singularity exec or run command.  The example below comes from the exec command section of Singularity User Guide.  It runs a TensorFlow example using a GPU on Owens.  (Output has been omitted from the example for brevity.)

[owens-login01]$ qsub -I -l nodes=1:ppn=28:gpus=1
...
[o0756]$ cd $PBS_O_WORKDIR
[o0756]$ module use /usr/local/share/lmodfiles/project/osc
[o0756]$ module load singularity
[o0756]$

 git clone https://github.com/tensorflow/models.git

[o0756]$

 singularity exec --nv docker://tensorflow/tensorflow:latest-gpu \
python ./models/tutorials/image/mnist/convolutional.py

References

 

Supercomputer: 

HOWTO: Use NFSv4 ACL

This document shows you how to use the NFSv4 ACL permissions system. An ACL is a list of permissions associated with a file or directory. These permissions allow you to restrict access to a certian file or directory by user or group. NFSv4 ACLs provide more specific options than typical POSIX read/write/execute permissions used in most systems.

Understanding NFSv4 ACL

This is an example of an NFSv4 ACL

A::user@nfsdomain.org:rxtncy
A::alice@nfsdomain.org:rxtncy
A::alice@nfsdomain.org:rxtncy
A::alice@nfsdomain.org:rxtncy

The following sections will break down this example from left to right and provide more usage options

 

ACE Type

The 'A' in the example is known as the ace type. The 'A' denotes "Allow" meaning this ACL is allowing the user or group to perform actions requiring permissions. Anything that is not explicitly allowed is denied by default.

Note: 'D' can denote a Deny ACE. While this is a valid option, this ACE type is not reccomended since any permission that is not explicity granted is automatically denied meaning Deny ACE's can be redundant and complicated.

 

ACE Flags

The above example could have a distinction known as a flag shown below

A:d:user@osc.edu:rxtncy

The 'd' used above is called an inheritence flag. This makes it so the ACL set on this directory will be automatically established on any new subdirectories. Inheritence flags only work on directories and not files. Multiple inheritence flags can be used in combonation or omitted entirely. Examples of inheritence flags are listed below:

Flag Name Function
d directory-inherit New subdirectories will have the same ACE
f file-inherit New files will have the same ACE minus the inheritence flags 
n no-propogate inherit New subdirectories will inherit the ACE minus the inheritence flags
i inherit-only New files and subdirectories will have this ACE but the ACE for the directory with the flag is null

 

ACE Principal

The 'user@nfsdomain.org' is a principal. The principle denotes the people the ACL is allowing access to. Principals can be the following:

  • A named user
  • Special principals
    • OWNER@
    • GROUP@
    • EVERYONE@
  • A group
    • Note: When the principal is a group, you need to add a group flag, 'g', as shown in the below example
    • A:g:group@osc.edu:rxtncy
      

 

ACE Permissions

The 'rxtncy' are the permissions the ACE is allowing. Permissions can be used in combonation with each other. A list of permissions and what they do can be found below:

Permission Function
r read-data (files) / list-directory (directories)
w write-data (files) / create-file (directories)
a append-data (files) / create-subdirectory (directories)
x execute (files) / change-directory (directories
d delete the file/directory
D delete-child : remove a file or subdirectory from the given directory (directories only)
t read the attributes of the file/directory
T write the attribute of the file/directory
n read the named attributes of the file/directory
N write the named attributes of the file/directory
c read the file/directory ACL
C write the file/directory ACL
o change ownership of the file/directory

 

Note: Aliases such as 'R', 'W', and 'X' can be used as permissions. These work simlarly to POSIX Read/Write/Execute. More detail can be found below.

Alias Name Expansion
R Read rntcy
W Write watTNcCy (with D added to directory ACE's
X Execute xtcy

 

Using NFSv4 ACL

This section will show you how to set, modify, and view ACLs

 

Set and Modify ACLs

To set an ACE use this command:

nfs4_setfacl [OPTIONS] COMMAND file

To modify an ACE, use this command:

nfs4_editfacl [OPTIONS] file

Where file is the name of your file or directory. More information on Options and Commands can be found below.

 

Options

Options can be used in combination or ommitted entirely. A list of options is shown below:

Option Name Function
-R recursive Applies ACE to a directory's files and subdirectories
-L logical Used with -R, follows symbolic links
-P physical Used with -R, skips symbolic links

 

Commands

Commands are only used when first setting an ACE. Commands and their uses are listed below.

 

View ACLs

To view ACLs, use the following command:

nfs4_getfacl file

Where file is your file or directory

 

Supercomputer: 
Service: 

HOWTO: Use VNC in a batch job

SSHing directly to a compute node at OSC - even if that node has been assigned to you in a current batch job - and starting VNC is an "unsafe" thing to do. When your batch job ends (and the node is assigned to other users), stray processes will be left behind and negatively impact other users. However, it is possible to use VNC on compute nodes safely.

You can use OnDemand, which is a much easier way to access desktops. If your work is not a very large, very intensive computation (for example, you do not expect to saturate all of the cores on a machine for a significant portion of the time you have the application you require open - e.g., you are using the GUI to set up a problem for a longer non-interactive compute job), you can choose one VDI under "Virtual Desktop Interface" from "Desktops" menu. Otherwise, please use "Interactive HPC" from Desktops" menu.

The examples below are for Oakley. If you use other systems, please see this page for supported versions of TurboVNC on our systems.

Starting your VNC server

Step one is to create your VNC server inside a batch job.

Option 1: Interactive

The preferred method is to start an interactive job, requesting an entire node, and then once your job starts, you can start the VNC server.

qsub -I -l nodes=1:ppn=12:gpus=2:vis

This command requests an entire GPU node, and tells the batch system you wish to use the GPUs for visualization. This will ensure that the X11 server can access the GPU for acceleration. In this example, I have not specified a duration, which will then default to 1 hour.

module load virtualgl
module load turbovnc/2.0

Then start your VNC server. (The first time you run this command, it may ask you for a password - this is to secure your VNC session from unauthorized connections. Set it to whatever password you desire. We recommend a strong password.)

vncserver

The output of this command is important: it tells you where to point your client to access your desktop. Specifically, we need both the host name (before the :), and the screen (after the :).

New 'X' desktop is n0302.ten.osc.edu:1

Option 2: Batch

This option is less optimal, because it is slightly more difficult to get the hostname and screen. However, by submitting a non-interactive batch job, you can go away and have the system email you when your desktop is ready to be connected to, and more importantly if your SSH connection to OSC is somewhat unstable and intermittent, you do not run the risk of being disconnected during your interactive session and having your VNC server terminated. In general, it is recommended you only use this option if running via an interactive session is not feasible.

In order to start an VNC session non-interactively, you can submit the following script to the scheduler using qsub (adjusting your walltime to what you need):

#PBS -l nodes=1:ppn=12:gpus=2:vis
#PBS -l walltime=00:15:00
#PBS -m b
#PBS -N VNCjob
#PBS -j oe
module load virtualgl
module load turbovnc/2.0
vncserver
sleep 100
vncpid=`pgrep -s 0 Xvnc`
while [ -e /proc/$vncpid ]; do sleep 0.1; done

This script will send you an email when your job has started, which includes the hostname.

PBS Job Id: 9200155.oak-batch.osc.edu
Job Name:   VNCjob
Exec host:  n0311/0-11
Begun execution

The screen is virtually always "1", unless someone else started a VNC server on that node outside of the batch system. You can verify the output of the vncserver command by using qpeek on a login node:

qpeek jobid

Where "jobid" is the batch system job number, for example, "9200155".

 

Connecting to your VNC server

Because the compute nodes of our clusters are not directly accessible, you must log in to one of the login nodes and allow your VNC client to "tunnel" through SSH to the compute node. The specific method of doing so may vary depending on your client software.

Linux/MacOS

Option 1: Manually create an SSH tunnel 

I will be providing the basic command line syntax, which works on Linux and MacOS. You would issue this in a new terminal window on your local machine, creating a new connection to Oakley.

ssh -L 5901:n0302.ten.osc.edu:5901 guilfoos@oakley.osc.edu

Open your VNC client, and connect to "localhost:1" - this will tunnel to the correct node on Oakley.

Option 2: Use your VNC software to tunnel 

This example uses Chicken of the VNC, a MacOS VNC client.

The default window that comes up for Chicken requires the host to connect to, the screen (or port) number, and optionally allows you to specify a host to tunnel through via SSH. This screenshot shows a proper configuration for the output of vncserver shown above. Substitute your host, screen, and username as appropriate.

When you click [Connect], you will be prompted for your HPC password (to establish the tunnel, provided you did not input it into the "password" box on this dialog), and then (if you set one), for your VNC password. If your passwords are correct, the desktop will display in your client.

 

Windows

This example shows how to create a SSH tunnel through your ssh client.  We will be using Putty in this example, but these steps are applicable to most SSH clients.

First, make sure you have x11 forwarding enabled in your SSH client.

Next, open up the port forwarding/tunnels settings and enter the hostname and port you got earlier in the destination field.  You will need to add 5900 to the port number when specifiying it here.  Some clients may have separate boxes for the desination hostname and port.  

For source port, pick a number between 11-99 and add 5900 to it.  This number between 11-99 will be the port you connect to in your VNC client.

Make sure to add the forwaded port, and save the changes you've made before exiting the configutations window.

PuTTY Tunnel Configuration Settings

Now start a SSH session to the respective cluster your vncserver is running on.  The port forwarding will automatically happen in the background.  Closing this SSH session will close the forwarded port; leave the session open as long as you want to use VNC.

Now start a VNC client.  TurboVNC has been tested with our systems and is recommended.  Enter localhost:[port], replacing [port] with the port between 11-99 you chose earlier.

New TurboVNC Connection

If you've set up a VNC password you will be prompted for it now.  A desktop display should pop up now if everything is configured correctly.

How to Kill a VNC session?

Occasionally you may make a mistake and start a VNC server on a login node or somewhere else you did not want to.  In this case it is important to know how to properly kill your VNC server so no processes are left behind.

The command syntax to kill a VNC session is:

vncserver -kill :[screen]

In the example above, screen would be 1.

You need to make sure you are on the same node you spawned the VNC server on when running this command.

Supercomputer: 
Service: 
Fields of Science: 

HOWTO: Use an Externally Hosted License

Many software packages require a license.  These licenses are usually made available via a license server, which allows software to check out necessary licenses.  In this document external refers to a license server that is not hosted inside OSC.

If you have such a software license server set up using a license manager, such as FlexNet, this guide will instruct you on the necessary steps to connect to and use the licenses at OSC.

Users who wish to host their software licenses inside OSC should consult OSC Help.

 

You are responsible for ensuring you are following your software license terms.  Please ensure your terms allow you to use the license at OSC before beginning this process!

Introduction

Broadly speaking, there are two different ways in which the external license server's network may be configured.  These differ by whether the license server is directly externally reachable or if it sits behind a private internal network with a port forwarding firewall.  

If your license server sits behind a private internal network with a port forwarding firewall you will need to take additional steps to allow the connection from our systems to the license server to be properly routed. 

License Server is Directly Externally Reachable

Diagram showing network setup of license server that is directly externally reachable

 

License Server is Behind Port Forwarding Firewall

Diagram showing network setup of license server that is behind a port forwarding firewall

 

Unsure?

If you are unsure about which category your situation falls under contact your local IT administrator.

Configure Remote Firewall

In order for connections from OSC to reach the license server, the license server's firewall will need to be configured.  All outbound network traffic from all of OSC's compute nodes are routed through a network address translation host (NAT), or two backup servers.

The license server should be configured to allow connections from the following IP addresses to the SERVER:PORT where the license server is running:

  • nat.osc.edu (192.157.5.13)
  • 192.148.248.35
  • 192.148.248.186

A typical FlexNet-based license server uses two ports: one is server port and the other is demon port, and the firewall should be configured for the both ports. A typical license file looks, for example,

SERVER licXXX.osc.edu 0050XXXXX5C 28000
VENDOR {license name} port=28001

In this example, "28000" is the server port, and "28001" is the demon port. The demon port is not mandatory if you use it on a local network, however it becomes necessary if you want to use it outside of your local network. So, please make sure you declared the demon port in the license file and configured the firewall for the port.

Confirm Configuration

The firewall settings should be verified by attempting to connect to the license server from the compute environment using telenet.

Get on to a compute node by requesting a short, small, interactive job and test the connection using telenet:

telnet <License Server IP Address> <Port#>

(Recommended) Restrict Access to IPs/Usernames

It is also recommended to restrict accessibility using the remote license server's access control mechanisms, such as limiting access to particular usernames in the options.dat file used with FlexNet-based license servers.

For FlexNet tools, you can add the following line to your options.dat file, one for each user.

INCLUDEALL USER <OSC username>

If you have a large number of users to give access to you may want to define a group using GROUP within the options.dat file and give access to that whole group using INCLUDEALL GROUP <group name> .

Users who use other license managers should consult the license manager's documentation.

Modify Job Environment to Point at License Server

The software must now be told to contact the license server for it's licenses.  The exact method of doing so can vary between each software package, but most use an environment variable that specifies the license server IP address and port number to use.

For example LS DYNA uses the environment variable LSTC_LICENSE and LSTC_LICENSE_SERVER to know where to look for the license.  The following lines would be added to a job script to tell LS-DYNA to use licenses from port 2345 on server 1.2.3.4:

setenv LSTC_LICENSE network
setenv LSTC_LICENSE_SERVER 2345@1.2.3.4

License Server is Behind Port Forwarding Firewall

If the license server is behind a port forwarding firewall, and has a different IP address from the IP address of the firewall, additional steps must be taken to allow connections to be properly routed within the license server's internal network.  

  1. Use the license server's fully qualified domain name in the environment variables instead of the IP address.
  2. Contact OSC Help to have the firewall IP address mapped to the fully qualified domain name.

 

Software Specific Details

The following outlines details particular to a specific software package.  

ANSYS

Uses the following environment variables:


ANSYSLI_SERVERS=<port>@<IP>
ANSYSLMD_LICENSE=<port>@<IP>

 

HOWTO: Use ulimit command to set soft limits

This document shows you how to set soft limits using ulimit command.

The ulimit command sets or reports user process resource limits. The default limits are defined and applied when a new user is added to the system. Limits are categorized as either soft or hard. With the ulimit command, you can change your soft limits for the current shell environment, up to the maximum set by the hard limits. You must have root user authority to change resource hard limits.

Syntax

ulimit [-H] [-S] [-a] [-c] [-d] [-f] [-m] [-n] [-s] [-t] [Limit]
flags description

-a

Lists all of the current resource limits

-c

Specifies the size of core dumps, in number of 512-byte blocks

-d

Specifies the size of the data area, in number of K bytes

-f

Sets the file size limit in blocks when the Limit parameter is used, or reports the file size limit if no parameter is specified. The -f flag is the default

-H

Specifies that the hard limit for the given resource is set. If you have root user authority, you can increase the hard limit. Anyone can decrease it

-m

Specifies the size of physical memory, in number of K bytes

-n

Specifies the limit on the number of file descriptors a process may have

-s

Specifies the stack size, in number of K bytes

-S

Specifies that the soft limit for the given resource is set. A soft limit can be increased up to the value of the hard limit. If neither the -H nor -S flags are specified, the limit applies to both

-t

Specifies the number of seconds to be used by each process

The limit for a specified resource is set when the Limit parameter is specified. The value of the Limit parameter can be a number in the unit specified with each resource, or the value "unlimited". For example, to set the file size limit to 51,200 bytes, use:

ulimit -f 100

To set the size of core dumps to unlimited, use:

ulimit –c unlimited

How to change ulimit for a MPI program

ulimit command affects the current shell environment. When a MPI program is started, it does not spawn in the current shell. You have to use mpiexec to start a wrapper script that sets the limit if you want to set the limit for each process. Below is how you set the limit for each shell (We use  ulimit –c unlimited to allow unlimited core dumps, as an example): 

  1. Prepare your batch job script named "myjob" as below (Here, we request 5-hour 2-node on Oakley cluster):
#PBS -l nodes=2:ppn=12
#PBS -l walltime=5:00:00
#PBS ...

#!/bin/bash

cd $PBS_O_WORKDIR
...
mpiexec ./test1
...
  1. Prepare the wrapper script named "test1" as below:
#!/bin/bash
ulimit –c unlimited
.....(your own program)
  1. qsub myjob
Supercomputer: 
Service: 

HOWTO: submit homework to repository at OSC

This page outlines a way a professor can set up a file submission system at OSC for his/her classroom project.

Usage for Professor

There was a bug causing a permission issue if the professor (that owns more than one project account, and the primary group is NOT the classroom project) used the tool between 7/17/2018 and 9/26/2018. This bug has been fixed. Please contact OSC Help if you encounter any permission issues.

After connecting to OSC system, professor runs submit_prepare as

$ /fs/scratch/xwang/bin/submit_prepare

Follow the instructions. It will create a designated directory where students submit their assignments, as well as generate submit for students used to submit homework to OSC, both of which are located in the directory specified by the professor.

Note:

The PI can also enforce deadline by simply changing the permission of the submission directory or renaming the submission directory at the deadline.

(Only works on Owens): One way is to use at command following the steps below:

  • Use at command to specify the deadline:
at [TIME]

where TIME is formatted HH:MM AM/PM MM/DD/YY. For example:

at 2:30 PM 08/21/2017
  • After running this command, run:
$ chmod 700 [DIRECTORY]

where DIRECTORY is the assignment folder to be closed off.

  • Enter [ctrl+D] to submit this command.

The permission of DIRECTORY will be changed to 700 at 2:30PM, August 21, 2018. After that, the student will get an error message when he/she tries to submit an assignment to this directory.

Usage for Students

If the student owns more than one project account, and the primary group is NOT the classroom project account, please change the primary group to the classroom project (C_PROJECT) during a login session before you run the submit command: newgrp C_PROJECT

A student should create one directory which includes all the files he/she wants to submit before running this script to submit his/her assignment. Also, the previous submission of the same assignment from the student will be replaced by the new submission. 

To submit the assignment, the student runs submit after connecting to OSC system as

$ /path/to/directory/from/professor/submit

Follow the instructions. It will allow students to submit an assignment to the designated directory specified by professor and send a confirmation email, or return an error message.

Supercomputer: 
Service: 

Citation

The Ohio Supercomputer Center provides High Performance Computing resources and expertise to academic researchers across the State of Ohio. Any paper citing this document has utilized OSC to conduct research on our production services. OSC is a member of the Ohio Technology Consortium, a division of the Ohio Department of Higher Education.

Hardware

OSC services can be cited by visiting the documentation for the service in question and finding the "Citation" page (located in the menu on the left of the screen).

HPC systems currently in production use can be found here: https://www.osc.edu/supercomputing/hpc

Decommissioned HPC systems can be found here: https://www.osc.edu/supercomputing/hpc/decommissioned.

Logos

Please refer to our branding webpage

Citing OSC

We prefer that you cite OSC when using our services, using the following information, taking into account the appropriate citation style guidelines. For your convenience, we have included the citation information in BibTeX and EndNote formats.

Ohio Supercomputer Center. 1987. Ohio Supercomputer Center. Columbus OH: Ohio Supercomputer Center. http://osc.edu/ark:/19495/f5s1ph73.

BibTeX:

@misc{OhioSupercomputerCenter1987,
ark = {ark:/19495/f5s1ph73},
howpublished = {\url{http://osc.edu/ark:/19495/f5s1ph73}},
year  = {1987},
author = {Ohio Supercomputer Center},
title = {Ohio Supercomputer Center}
}

EndNote:

%0 Generic
%T Ohio Supercomputer Center
%A Ohio Supercomputer Center
%R ark:/19495/f5s1ph73
%U http://osc.edu/ark:/19495/f5s1ph73
%D 1987

Here is an .ris file to better suit your needs. Please change the import option to .ris

Documentation Attachment: 

OSC User Code of Ethics

The Ohio Supercomputer Center (OSC) exists to provide state-of-the-art computing services to universities and colleges; to provide supercomputer services to Ohio scientists and engineers; to stimulate unique uses of supercomputers in Ohio; to attract students, faculty, resources and industry; to catalyze inter-institutional supercomputer research and development projects; to serve as the model for other state-sponsored technology initiatives.

OSC serves a large number and variety of users including students, faculty, staff members, and commercial clients throughout the state of Ohio.  Ethical and legal standards, in particular, that apply to the use of computing facilities are not unique to the computing field. Rather, they derive directly from standards of common sense and common decency that apply to the use of any public resource.  Indeed, OSC depends upon the spirit of mutual respect and cooperative attitudes.

This statement on conditions of use is published in that spirit. The purpose of this statement is to promote the responsible, ethical, and secure use of OSC resources for the protection of all users.

Authorized Use

As a condition of use of OSC facilities, the user agrees:

  1. To respect the privacy of other users; for example, users shall not intentionally seek information on, obtain copies of, or modify files, tapes, or passwords belonging to other users unless explicitly authorized to do so by those users.
  2. To respect the legal protection provided by copyrights and licenses to programs and data; for example, users shall not make copies of a licensed computer program to avoid paying additional license fees.
  3. To respect the intended usage for which access to computing resources was granted; for example, users shall use accounts authorized for their use by the principal investigator responsible for these accounts only for the purposes specified by the principal investigator and shall not use any other user's account.
  4. To respect the integrity of computing systems; for example, users shall not intentionally develop or use programs that harass other users or infiltrate a computer or computing systems or damage or alter the software components of a computing system.
  5. To respect the financial structure of computing systems; for example, users shall not intentionally develop or use any unauthorized mechanisms to alter or avoid charges levied by OSC for computing services.
  6. To not enable other institutions or users to avoid licensing restrictions or fees by simply allowing them to use their account.
  7. To abide by software specific licensing terms and restrictions, as outlined in the specific software page or agreement.

In addition, users are expected to report to OSC information that they may obtain concerning instances in which the above conditions have been or are being violated.

Violations of the following conditions are certainly unethical and are possibly a criminal offense: unauthorized use of another user's account; tampering with other users' files, tapes, or passwords, harassment of other users; unauthorized alteration of computer charges; and unauthorized copying or distribution of copyrighted or licensed software or data. Therefore, when OSC becomes aware of possible violations of these conditions, it will initiate an investigation. At the same time, in order to prevent further possible unauthorized activity, OSC may suspend the authorization of computing services to the individual or account in question. In accordance with established practices, confirmation of the unauthorized use of the facilities by an individual may result in disciplinary review, expulsion from his/her university, termination of employment, and/or legal action.

Users of computing resources should be aware that although OSC provides and preserves the security of files, account numbers, and passwords, security can be breached through actions or causes beyond reasonable control. Users are urged, therefore, to safeguard their data, to take full advantage of file security mechanisms built into the computing systems, and to change account passwords frequently.

Appropriate Use

Computing resources shall be used in a manner consistent with the instructional and/or research objectives of the community, in general, and consistent with the objectives of the specified project for which such use was authorized. All uses inconsistent with these objectives are considered to be inappropriate use and may jeopardize further authorization.

Beyond the allocation of computing resources, OSC normally cannot and does not judge the value or appropriateness of any user's computing. However, the use of computing resources for playing games for purely recreational purposes, the production of output that is unrelated to the objectives of the account, and, in general, the use of computers simply to use computing resources are examples of questionable use of these resources.

When possible inappropriate use of computing resources is encountered, OSC shall notify the principal investigator responsible. The principal investigator is expected either to take action or to indicate that such use should be considered appropriate.

Should possible inappropriate use continue after notification of the principal investigator, or should unresolvable differences of opinion persist, these shall be brought to the attention of OSC staff for recommendations on further action.  Upon the recommendation of OSC staff, the Director may impose limitations on continued use of computing resources.

Responsible Use

Users are expected to use computing resources in a responsible and efficient manner consistent with the goals of the account for which the resources were approved. OSC will provide guidance to users in their efforts to achieve efficient and productive use of these resources. Novice users may not be aware of efficient and effective techniques; such users may not know how to optimize program execution; nor may such optimization necessarily lead to improved cost benefits for these users. Those who use large amounts of computing resources in production runs should attempt to optimize their programs to avoid the case where large inefficient programs deny resources to other users.

Programming, especially in an interactive environment, involves people, computers, and systems. Efficient use of certain resources, such as computers, may lead to inefficient use of other resources, such as people. Indeed, the benefits attributed to good personal or interactive computing systems are that they speed total program development and thus lower attendant development costs even though they may require more total computer resources. Even with this understanding, however, users are expected to refrain from engaging in deliberately wasteful practices, for example, performing endless unnecessary computations.

OSC Responsibilities

OSC has a responsibility to provide service in the most efficient manner that best meets the needs of the total user community. At certain times the process of carrying out these responsibilities may require special actions or intervention by the staff. At all other times, OSC staff members have no special rights above and beyond those of other users. OSC shall make every effort to ensure that persons in positions of trust do not misuse computing resources or take advantage of their positions to access information not required in the performance of their duties.

OSC prefers not to act as a disciplinary agency or to engage in policing activities. However, in cases of unauthorized, inappropriate, or irresponsible behavior the Center does reserve the right to take action, commencing with an investigation of the possible abuse. In this connection, OSC, with all due regard for the rights of privacy and other rights of users', shall have the authority to examine files, passwords, accounting information, printouts, tapes, or other material that may aid the investigation. Examination of users files must be authorized by the Director of OSC or his designee. Users, when requested, are expected to cooperate in such investigations. Failure to do so may be grounds for cancellation of access privileges.


OSC User Code of Ethics © 2018

OSCusage

Introduction

OSCusage is command developed at OSC for use on OSC's systems.  It allows for a user to see information on their project's current RU balance, including which users and jobs incurred what charges.

Note:  a new prototype OSCusage command is available at /opt/osc/libexec/OSCusage which clients are encouraged to try out and provide feedback on.

Availability

Oakley ruby owens
X X X

 

Usage

OSCusage takes the following options and parameters.

usage: OSCusage [-qvn] -[r|h] [MM/DD/YYYY] [MM/DD/YYYY]
or [-i]
Note: the year requires all 4 digits.
OSCusage can only provide information on a user's primary project, visible with the command id -gn .  If you are a part of other projects, visible under id -Gn , you will be unable to pull information on those projects.  Contact OSC Help for assistance in getting such information.

Today's Usage

Running OSCusage with no options or parameters specified will provide RU usage information per a user for the current day.

$ OSCusage
Usage Statistics for project PAS0435
for 5/3/2016 to 5/3/2016
PI: Dr. Judith Gardiner
RU Balance: -17.88635
Username  Dates                            RUs  Status
osu525    5/3/2016 to   5/3/2016     0.00000  ACTIVE
*osu391    5/3/2016 to   5/3/2016     0.00000  ACTIVE
osu6396    5/3/2016 to   5/3/2016     0.00000  ACTIVE
osu6445    5/3/2016 to   5/3/2016     0.00000  PASSWORD
osu8968    5/3/2016 to   5/3/2016     0.00000  ACTIVE
===============   PAS0435 TOTAL    0.00000

Users on Project

Specifying -i provides information on the accounts under the project.

$ OSCusage -i
Information for project PAS0435
PI: Dr. Judith Gardiner
RU Balance: -17.88635
Username    Status  Full Name
*osu391    ACTIVE  Dr. Judith Gardiner
osu6396    ACTIVE  Dr. Judith Gardiner
osu8968    ACTIVE  Koh Schooley


RU Usage in Timeframe

If you specify a timeframe you can get RU utilization information specifically for jobs that completed within that period.

$ OSCusage 01/01/2016 05/01/2016
Usage Statistics for project PAS0435
for 01/01/2016 to 05/01/2016
PI: Dr. Judith Gardiner
RU Balance: -17.88635
Username  Dates                            RUs  Status
osu525  01/01/2016 to 05/01/2016     0.00000  ACTIVE
*osu391  01/01/2016 to 05/01/2016     0.00000  ACTIVE
osu6396  01/01/2016 to 05/01/2016     0.00000  ACTIVE
osu6445  01/01/2016 to 05/01/2016     0.00000  PASSWORD
osu8968  01/01/2016 to 05/01/2016     0.15752  ACTIVE
===============   PAS0435 TOTAL    0.15752

Show only my usage

Specify -q to show only the current user's usage.

$ OSCusage -q 01/01/2016 05/01/2016
Usage Statistics for project PAS0435
for 01/01/2016 to 05/01/2016
PI: Dr. Judith Gardiner
RU Balance: -17.88635
Username  Dates                            RUs  Status
osu8968  01/01/2016 to 05/01/2016     0.15752  ACTIVE
===============   PAS0435 TOTAL    0.15752

Detailed Charges Breakdown

Specify -v to get detailed information on what job's incurred what charges.

$ OSCusage -v 01/01/2016 05/01/2016
Usage Statistics for project PAS0435
for 01/01/2016 to 05/01/2016
PI: Dr. Judith Gardiner
RU Balance: -17.88635
Start                 RUs     Charge           Status
Username  Date  Date & Time          Used       Type     Queue           Job
-------------            osu525 TOTAL    0.00000
-------------           *osu391 TOTAL    0.00000
-------------           osu6396 TOTAL    0.00000
-------------           osu6445 TOTAL    0.00000
osu8968 03/30  03/29 15:14:31    0.00277   CPU-Ruby    serial  E      STDIN
osu8968 03/22  03/21 11:38:14    0.00005 CPU-Oakley    serial  E   login_en
osu8968 03/22  03/21 11:48:25    0.00005 CPU-Oakley    serial  E   test_env
osu8968 03/10  03/09 12:30:36    0.00066 CPU-Oakley    serial  E   test_pat
osu8968 02/20  02/19 17:14:23    0.01266 CPU-Oakley  parallel  E   initial_
osu8968 02/20  02/19 17:16:24    0.14133 CPU-Oakley  parallel  E   initial_
-------------           osu8968 TOTAL    0.15752
===============   PAS0435 TOTAL    0.15752

View Usage in Hours

Specify -h to view the usage information in core hours, as opposed to RUs.

$ OSCusage -h 01/01/2016 05/01/2016
Usage Statistics for project PAS0435
for 01/01/2016 to 05/01/2016
PI: Dr. Judith Gardiner
RU Balance: -17.88635
Note: Listing is in HOURS, as opposed to RUs.
Username  Dates                            HRS  Status
osu525  01/01/2016 to 05/01/2016     0.00000  ACTIVE
*osu391  01/01/2016 to 05/01/2016     0.00000  ACTIVE
osu6396  01/01/2016 to 05/01/2016     0.00000  ACTIVE
osu6445  01/01/2016 to 05/01/2016     0.00000  PASSWORD
osu8968  01/01/2016 to 05/01/2016     1.57520  ACTIVE
===============   PAS0435 TOTAL    1.57520

Usage on Secondary Group Projects

Specifying -A  provides usage information for all of the secondary group projects authorized under your account, including the primary project. This is mainly for Annual Allocations PIs. 

$ OSCusage -A
$ OSCusage -A 01/01/2016 05/01/2016

No examples can be shown for this command. Please contact OSC Help with questions. 

Supercomputer: 
Service: 

Supercomputing FAQ

General Questions

Account Questions

Disk Storage Questions

Email Questions

Linux Questions

SSH Questions

Batch Processing Questions

Compiling System Questions

Parallel Processing Questions

Libraries/Software Questions

Performance Analysis Questions

Other Common Problems

General Questions

Who can get an account?

Any faculty member or research scientist at an academic institution in Ohio is eligible for an academic account at OSC. These researchers/educators may request accounts for their students and collaborators. Commercial accounts are also available.  More information about applying for both academic and commercial accounts at OSC can be found at https://www.osc.edu/supercomputing/support/account

Where should a new OSC user begin?

The first thing you should do as a new OSC user is to check the email account that was given to OSC when you were registered.  If you are a PI, this will be the email address you entered in the primary contact information on your Project Request application. When your account is created, you will receive an email giving you information on how to start using the services at OSC.  Attached will be a letter you can keep for reference that contains your username and temporary password, as well as some other important information about using the systems at OSC.  After carefully reading the email, please first go to my.osc.edu to change your temporary password.

Once you are able to connect to our HPC systems, you should start familiarizing yourself with the software and services available from the OSC, including:

Do I have to pay for supercomputer use?

OSC receives funding from the state of Ohio. There is no charge for Ohio academic use. 

How many supercomputers does OSC have?  Which one should I use?

OSC currently has three HPC clusters: Owens Cluster, a 23,500+ core Dell cluster with Intel Xeon processors, Ruby Cluster, a 4800 core Intel Xeon machine from HP, and Oakley Cluster, a 8,300+ core HP Intel Xeon machine. New users have access to Owens and Oakley clusters. Ruby is currently unavailable for general access. To learn more, click here.

How do I cite OSC in my publications?

Any publication of any material, whether copyrighted or not, based on or developed with OSC services, should cite the use of OSC, and the use of the specific services (where applicable). For more information about citing OSC, please visit www.osc.edu/citation.

How do I submit my publications and funding information to OSC?

Please submit a list of publications, noting which cite OSC, as well as a list of current funding with your application materials for additional resources. Alternatively, please email the lists to OSC Help

Can I receive a letter of support from OSC when I apply for outside funding?

OSC has a standard letter of support that you can include (electronically or in hard copy) with a proposal for outside funding. This letter does not replace the application process for time on OSC's systems. To receive the letter of support, please send your request to oschelp@osc.edu. You should provide the following information: name and address of the person/organization to whom the letter should be addressed; name(s) of the principal investigator(s) and the institution(s); title of the proposal; number of years of proposed project; number of RUs requested per year. Please allow at least two working days to process your request.

Hardware information about the systems is available at http://www.osc.edu/supercomputing/hardware

How do I register for a workshop?

For a complete schedule of current training offerings, please visit the OSC Training Schedule.  To register or for more information, please email oschelp@osc.edu.  

Where can I find documentation?

For documentation specific to software applications, see Software.  For other available documentation, see Manuals.

My question isn't answered here. Whom can I ask for help?

Contact the OSC Help Desk. Support is available 24x7x365, but more complicated questions will need to wait for regular business hours (Monday - Friday, 9am - 5pm) to be addressed. More information on the OSC supercomputing help desk can be found on our Support Services page.

Something seems to be wrong with the OSC systems.  Should I contact the help desk?

Information will be coming soon for guidelines on reporting possible system problems.

Where can I find logos for my presentations, posters, etc.?

Please see our citation webpage.

Account Questions

What are projects and accounts?

An eligible principal investigator heads a project. Under a project, authorized users have accounts with credentials that permit users to gain access to the HPC systems. A principal investigator can have more than one project, and authorized users can have more than one account. 

How do I get/renew an account?

For information concerning accounts (i.e., how to apply, who can apply, etc.), see Accounts.

I'm a faculty member.  How do I get accounts for my students?

If an eligible principal investigator is new to OSC, he/she can apply for a start-up request and include the contact information for authorized users. Please refer to http://app.osc.edu/cgi-bin/app/startup for the on-line form. If an eligible principal investigator has a current project, he/she can email OSC Help to add the user. Authorized users do not have to be located in Ohio.

I'm continuing the research of a student who graduated.  Can I use his/her account?

Yes, but you must have the account transferred into your name. Have your PI send an email to oschelp@osc.edu

I'm working closely with another student.  Can we share an account?

No. Each person using the OSC systems must have his/her own account. Sharing files is possible, even with separate accounts. Ask your PI to request an account for you by emailing OSC Help.

How do I change my password?

You can change your password through the My OSC portal. Log in at http://my.osc.edu, and click the "change password" button at the bottom left corner of the HPC User Profile box. Please note that your password has certain requirements; these are specified on the "change password" portal. Please wait an hour before attempting to log in with the new password. For security purposes, please note that our password change policy requires a password change every 90 days.

I want to use csh instead of bash.  How do I change the default shell?

You can change your default shell through the My OSC portal. Log in at my.osc.edu, and use the "Unix Shell" drop-down menu in the HPC User Profile box to change your shell. You will need to log off the HPC system and log back on before the change goes into effect. Please note, that it will take about an hour for the changes to be applied.

What is an RU?

An RU is a resource unit. A resource unit is an aggregate measure of the use of CPU, memory, and file storage. For details on charging algorithms, see Charging.

How do I find my account balance?

To see usage and balance information from any system, refer to the OSCusage page.

NOTE: Accounting is updated once a day, so the account balance is for the previous day.

How do I get more resources?

To request more resources, the principal investigator must prepare a proposal that will be sent for peer review. After the review process is complete, the Allocations Committee of the Statewide Users Group meets to determine the number of resource units to be awarded. The committee meets on the second Thursday of even-numbered months. PIs should submit their proposals at least three weeks before the meeting date to ensure inclusion on the agenda.

The application form is located at http://app.osc.edu/cgi-bin/app/. A complete application comprises the following items:

  • Contact information of new authorized users
  • Contact information of at least three recommended reviewers
  • Electronic version of PI’s resume
  • Publications list (publications, presentations, and articles that cite OSC)
  • Justification and performance (quantitative estimation of the number of RUs per run times; the number of runs for the estimated duration of the request; performance reports of the code to be used or explanations about performance; report of any optimization performed on the code
  • Proposal text

How much will my account be charged for supercomputer usage?

For details on charging algorithms, see Charging.

Disk Storage Questions

What is my disk quota?

Each user has a quota of 500 gigabytes (GB) of storage and 1,000,000 files. You may also have access to a project directory with a separate quota.  See Available File Systems for more information.

How can I determine the total disk space used by my account?

Your quota and disk usage are displayed every time you log in. You have limits on both the amount of space you use and the number of files you have. There are separate quotas for your home directory and any project directories you have access to.

Note: The quota information displayed at login is updated twice a day, so the information may not reflect the curent usage.

You may display your home directory quota information with quota -s .

How do I get more disk space?

Your home directory quota cannot be increased. You should consider deleting, transferring, and/or compressing your files to reduce your usage. 

A PI may request project space to be shared by all users on a project. Estimate the amount of disk space that you will need and the duration that you will need it. Send requests to oschelp@osc.edu.

See the Allocations and Accounts section for more information. 

How can I find my largest directories?

To reveal the directories in your account that are taking up the most disk space you can use the du , sort and tail commands. For example, to display the ten largest directories, change to your home directory and then run the command:

du . | sort -n | tail -n 10

Why do I receive "no space left" error when writing data to my home directory?

If you receive the error "No space left on device" when you try to write data to your home directory, it indicates the disk is full. First, check your home directory quota. Each user has 500 GB quota of storage and the quota information is shown when you login to our systems. If your disk quota is full, consider reducing your disk space usage. If your disk quota isn't full (usage less than 500GB), it is very likely that your disk is filled up with 'snapshot' files, which are invisible to users and used to track fine-grained changes to your files for recovering lost/deleted files. In this case, please contact OSC Help for further assitance. To avoid this situation in future, consider running jobs that do a lot of disk I/O in the temporary filesystem ($TMPDIR or $PFSDIR) and copy the final output back at the end of the run. See Available File Systems for more information.

How can I use tar and gzip to aggregate and compress my files?

The commands tar and gzip can be used together to produce compressed file archives representing entire directory structures. These allow convenient packaging of entire directory contents. For example, to package a directory structure rooted at src/ use

tar -czvf src.tar.gz src/

This archive can then be unpackaged using

tar -xzvf src.tar.gz

where the resulting directory/file structure is identical to what it was initially.

The programs zip , bzip2 and compress can also be used to create compressed file archives. See the man pages on these programs for more details.

Tar is taking too long.  Is there a way to compress quicker?

If using tar with the options zcvf is taking too long you can instead use pigz  in conjunction with tar.   pigz does gzip compression while taking advantage of multiple cores.

tar cvf - paths-to-archive | pigz > archive.tgz

pigz defaults to using eight cores, but you can have it use more or less with the -p argument.

tar cvf - paths-to-archive | pigz -n 4 > archive.tgz
Due to the parallel nature of pigz, if you are using it on a login node you should limit it to using 2 cores.  If you would like to use more cores you need to submit either an interactive or batch job to the queue and do the compression from within the job.

Note: pigz does not significantly improve decompression time.  

Email Questions

How do I change the email address OSC uses to contact me?

Please update your email on my.osc.edu, or send your new contact information to oschelp@osc.edu

I got an automated email from OSC.  Where can I get more information about it?

See the Knowledge Base.

Linux Questions

What is Linux?

Linux is an open-source operating system that is similar to UNIX. It is widely used in High Performance Computing.

How can I get started using Linux?

See the Unix Basics tutorial for more information.  There are also many tutorials available on the web.

SSH Questions

What is SSH?

Secure Shell (SSH) is a program to log into another computer over a network, to execute commands in a remote machine, and to move files from one machine to another. It provides strong authentication and secure communications over insecure channels. SSH provides secure X connections and secure forwarding of arbitrary TCP connections.

How does SSH work?

SSH works by the exchange and verification of information, using public and private keys, to identify hosts and users. The ssh-keygen command creates a directory ~/.ssh and files that contain your authentication information. The public key is stored in ~/.ssh/id_rsa.pub and the private key is stored in  ~/.ssh/id_rsa. Share only your public key. Never share your private key. To further protect your private key you should enter a passphrase to encrypt the key when it is stored in the file system. This will prevent people from using it even if they gain access to your files. 

One other important file is ~/.ssh/authorized_keys. Append your public keys to the authorized_keys file and keep the same copy of it on each system where you will make ssh connections. 

In addition, on Owens the default SSH client config enables hashing of a user’s known_hosts file.  So if SSH is used on Owens the remote system’s SSH key is added to ~/.ssh/known_hosts in a hashed format which can’t be unhashed.  If the remote server’s SSH key changes, special steps must be taken to remove the SSH key entry:

ssh-keygen -R <hostname>

Can I connect without using an SSH client?

The OSC OnDemand portal allows you to connect to our systems using your web browser, without having to install any software. You get a login shell and also the ability to transfer files.  

How can I upload or download files?

Most file transfers are done using sftp (SSH File Transfer Protocol) or scp (Secure CoPy). These utilities are usually provided on Linux/UNIX and Mac platforms. Windows users should read the next section, "Where can I find SSH and SFTP clients".

Where can I find SSH and SFTP clients?

There are many SSH and SFTP clients available, both commercial and free.  See Getting Connected for some suggestions.  

How do I run a graphical application in an SSH session?

Graphics are handled using the X11 protocol. You’ll need to run an X display server on your local system and also set your SSH client to forward (or "tunnel") X11 connections. On most UNIX and Linux systems, the X server will probably be running already. On a Mac or Windows system, there are several choices available, both commercial and free. See our guide to Getting Connected for some suggestions.

Batch Processing Questions

What is a batch request?

On all OSC systems, batch processing is managed by the Portable Batch System (PBS). PBS batch requests (jobs) are shell scripts that contain the same set of commands that you enter interactively. These requests may also include options for the batch system that provide timing, memory, and processor information. For more information, see our guide to Batch Processing at OSC.  

How do I submit, check the status, and/or delete a batch job?

PBS uses qsub to submit, qstat to check the status, and qdel to delete a batch request. For more information, see our Batch-Related Command Summary

Can I be notified by email when my batch job starts or ends?

Yes.  See the -m option in our PBS Directives Summary. If you are submitting a large number of jobs, this may not be a good idea.

Why won't my job run?

There are numerous reasons why a job might not run even though there appear to be processors and/or memory available. These include:

  • Your account may be at or near the job count or processor count limit for an individual user.
  • Your group/project may be at or near the job count or processor count limit for a group.
  • The scheduler may be trying to free enough processors to run a large parallel job.
  •  Your job may need to run longer than the time left until the start of a scheduled downtime.
  • You may have requested a scarce resource or node type, either inadvertently or by design.

See our Scheduling Policies and Limits for more information. 

Why is my job being rejected with the error "Group: is not valid"?

If you are a part of a annual allocations project, usually designated by the prefix PAA-, in addition to another project you must explicitly specify which project you want to charge a jobs usage to.  This can be done with the -A PBS directive.  For example if a user was a part of both projects PAA0999 and PAQ0343 and wanted to charge a job to PAA0999 they would need to add the following line to their script:

#PBS -A PAA0999

How can I retrieve files from unexpectedly terminated jobs?

A batch job that terminates before the script is completed can still copy files from $TMPDIR to the user's home directory via the trap command ( trap commands do not work in csh and tcsh shell batch scripts). In the batch script, the trap command needs to precede the command causing the TERMination. It could be placed immediately after the PBS header lines. Here is a generic form:

trap "cd $PBS_O_WORKDIR;mkdir $PBS_JOBID;cp -R $TMPDIR/* $PBS_JOBID;exit" TERM

If a command in a batch script is killed for excessive memory usage (see Out-of-Memory (OOM) or Excessive Memory Usage for details) then the trap command may not be executed.  However, normal shell scripting can handle this situation:  the exit status of a command that may possibly cause an OOM can be checked and appropriate action taken.  Here is a Bourne shell example:

bla_bla_big_memory_using_command_that_may_cause_an_OOM
if [ $? -ne 0 ]; then
    cd $PBS_O_WORKDIR;mkdir $PBS_JOBID;cp -R $TMPDIR/* $PBS_JOBID
    exit
fi

Finally, if a node your job is running on crashes then the trap command may not be executed. It may be possible to recover your files from batch-managed directories in this case. Contact OSC Help for assistance.

How can I delete all of my jobs on a cluster?

To delete all your jobs on one of the clusters, including those currently running, queued, and in hold, login to the cluster and run the command:

qselect -u <username> | xargs qdel

How can I determine the number of cores in use by me or my group?

To determine the number of cores (processors) in use by you account on a particular system run:

showq -u $USER | grep "local jobs"

To determine the number of cores (processors) in use by your primary project on a particular system run:

showq -w acct=$GROUP | grep "local jobs"

Genrerally, to see the number of cores in use by a particular project on a particular system run:

showq -w acct=<project> | grep "local jobs"

replacing <project> with the project ID.

How to request GPU nodes for visualization?

By default, we don't start an X server on gpu nodes because it impacts computational performance. Add vis in your GPU request such that the batch system uses the GPUs for visualization. For example, on Owens, it should be

nodes=1:ppn=28:gpus=1:vis

Compiling System Questions

What languages are available?

Fortran, C, and C++ are available on all OSC systems. The commands used to invoke the compilers and/or loaders vary from system to system. For more information, see our Compilation Guide.

What compiler (vendor) do you recommend?

We have Intel, PGI, and gnu compilers available on all systems. Each compiler vendor supports some options that the other doesn’t, so the choice depends on your individual needs. For more information, see our Compilation Guide.

Will software built for one system run on another system?

Most serial code built on one system will run on another system, although it may run more efficiently if it is built and run on the same system. Parallel (MPI) code typically must be built on the system where it will run.

Parallel Processing Questions

What is parallel processing?

Parallel processing is the simultaneous use of more than one computer (or processor) to solve a problem. There are many different kinds of parallel computers. They are distinguished by the kind of interconnection between processors or nodes (groups of processors) and between processors and memory.

What parallel processing environments are available?

On most systems, both shared-memory and distributed-memory parallel programming models can be used. Versions of OpenMP (for multithreading or shared-memory usage) and MPI (for message-passing or distributed-memory usage) are available.  A summary of parallel environments will be coming soon.

What is a core?

A core is a processor. When a single chip contains multiple processors, they are called cores.

I'm not seeing the performance I expected.  How can I be sure my code is running in parallel?

We are currently working on a guide for this. Please contact OSC Help for assistance.

Libraries/Software Questions

What software applications are available?

See the Software section for more information.

Do you have a newer version of (name your favorite software)?

Check the Software section to see what versions are installed. You can also check the installed modules using the module spider  or module avail  command.

How do I get authorized to use a particular software application?

Please contact OSC Help for assistance.

What math routines are available?  Do you have ATLAS and LAPACK?

See the Software section for information on third-party math libraries (e.g., MKL, ACML, fftw, scalapack, etc). MKL and ACML are highly optimized libraries that include the BLAS and LAPACK plus some other math routines. 

Do you have NumPy/SciPy?

The NumPy and SciPy modules are installed with the python software. See the Python software page.

OSC does not have a particular software package I would like to use. How can I request it?

Please refer to the Software Forms page. You will see a link to Request for Software Form. Download the form, complete the information, and attach the form to an e-mail to oschelp@osc.edu. The Statewide Users Group will consider the request.

You may install open source software yourself in your home directory. If you have your own license for commercial software, contact the OSC Help desk.

I have a software package that must be installed as root.  What should I do?

Most packages have a (poorly documented) option to install under a normal user account. Contact the OSC Help desk if you need assistance. We generally do not install user software as root.

What are modules?

Modules are used to manage the environment variable settings associated with software packages in a shell-independent way. On OSC's systems, you will by default have modules in your environment for PBS, MPI, compilers, and a few other pieces of software. For information on using the module system, see our guide to Batch Processing at OSC.

Performance Analysis Questions

What are MFLOPS/GFLOPS/TFLOPS/PFLOPS?

MegaFLOPS/GigaFLOPS/TeraFLOPS/PetaFLOPS are millions/billions/trillions/quadrillions of FLoating-point Operations (calculations) Per Second.

How do I find out about my code's performance?

A number of performance analysis tools are available on OSC systems. Some are general to all systems and others are specific to a particular system. See our performance analysis guide for more info.

How can I optimize my code?

There are several ways to optimize code. Key areas to consider are CPU optimization, I/O optimization, memory optimization, and parallel optimization. See our optimization strategy guide for more info.

Other Common Problems

What does "CPU time limit exceeded" mean?

Programs run on the login nodes are subject to strict CPU time limits. To run an application that takes more time, you need to create a batch request. Your batch request should include an appropriate estimate for the amount of time that your application will need. See our guide to Batch Processing at OSC for more information.

My program or file transfer died for no reason after 20 minutes.  What happened?

Programs run on the login nodes are subject to strict CPU time limits. Because file transfers use encryption, you may hit this limit when transferring a large file. To run longer programs, use the batch system. To transfer larger files, connect to gridftp01.osc.edu instead of to a login node.

Why did my program die with a segmentation fault, address error, or signal 11?

This is most commonly caused by trying to access an array beyond its bounds -- for example, trying to access element 15 of an array with only 10 elements. Unallocated arrays and invalid pointers are other causes. You may wish to debug your program using one of the available tools such as the TotalView Debugger.

I created a batch script in a text editor on a Windows or Mac system, but when I submit it on an OSC system, almost every line in the script gives an error. Why is that?

Windows and Mac have different end-of-line conventions for text files than UNIX and Linux systems do, and most UNIX shells (including the ones interpreting your batch script) don't like seeing the extra character that Windows appends to each line or the alternate character used by Mac. You can use the following commands on the Linux system to convert a text file from Windows or Mac format to UNIX format:

dos2unix myfile.txt 

mac2unix myfile.txt  

I copied my output file to a Windows system, but it doesn't display correctly.  How can I fix it?

A text file created on Linux/UNIX will usually display correctly in Wordpad but not in Notepad. You can use the following command on the Linux system to convert a text file from UNIX format to Windows format:

unix2dos myfile.txt  

What IP ranges do I need to allow in my firewall to use OSC services?

See our knowledge base article on the topic.

Supercomputing Policies

OSC-1, OSC Data Lifecycle Management Policy - Data storage space is a limited resource, and in an effort to keep the resource available to the largest amount of active users, the following policy and accompanying procedures and resources have been developed to reduce system management overhead and the impact on other users of OSC systems.

OSC-2, OSC Media Inventory Management - The purpose of this policy and accompanying procedures and resources is to help ensure the protection of the media containing the data from accidental or intentional unauthorized access, damage, alteration or disclosure while preserving the ability of authorized users to access and use the data.

OSC-3, OSC Information Security Framework - This policy and its supporting sub-policies provide a foundation for the security of OSC information technology systems. The requirements put forth in this policy and its supporting sub-policies are designed to ensure that due diligence is exercised in the protection of information, systems and services. This policy describes fundamental practices of information security that are to be applied by OSC to ensure that protective measures are implemented and maintained.

OSC-4, OSC Malicious Code Security - This policy is to implement and operate a malicious code security program. The program should help to ensure that adequate protective measures are in place against introduction of malicious code into OSC information systems and that computer system users are able to maintain a high degree of malicious code awareness.

OSC-5, OSC Remote Access Security - This policy is to establish practices wherever a remote access capability is provided to OSC systems so that inherent vulnerabilities in such services may be compensated.

OSC-6, OSC Security Education and Awareness - This policy requires OSC to provide information technology security education and awareness to employees, contractors, temporary personnel and other agents of OSC who use and administer computer and telecommunications systems.

OSC-7, OSC Security Incident Response - This policy defines adequate security response for identified security incidents.

OSC-8, OSC Password PIN Security - We have implemented a new password change policy. This portion of our policies page is currently under construction. The Password PIN Security policy is dated, but established minimum requirements regarding the proper selection, use and management of passwords and personal identification numbers (PINs); references in this policy to passwords also apply to PINs, except where explicitly noted.

OSC-9, OSC Portable Security Computing - This policy addresses information technology (IT) security concerns with portable computing devices and provides direction for their use, management and control. This policy includes security concerns with the physical device itself, as well as its applications and data.

OSC-10, OSC Security Notifications - This OSC policy identifies the methods used to inform users of their duty, limitations on use, legal requirements and personal privacy expectations associated with the use of OSC and university computers, networks or telecommunications systems.

OSC-11, OSC User Management Policy - This policy establishes the information and qualifications required to establish an account to use OSC resources. This policy will also define the basic levels of support that users of OSC IT environments can expect.

OSC-12, OSC Intrusion Prevention and Detection - The purpose of this state policy is to establish an intrusion prevention and detection capability that is designed to prevent, monitor and identify system intrusions or misuse.

OSC-13, OSC IT Business Continuity Planning - This document provides guidance in the development and implementation of a comprehensive information technology business continuity plan that, in the event of a business disruption, will help enable the continuation of critical processes and the delivery of essential services at an acceptable level.

OSC-14, OSC Virtual Machine Lifecycle Management - Virtual Machines at OSC are a resource that must be maintained and protected. This document provides guidance in the hosting and maintaining of all systems and virtual environment infrastructure that require support from a limited resource. The following policy and accompanying procedures and resources have been developed to reduce system management overhead and the impact on other users of OSC systems.

 

Proposed OSC Policies for Public Comments

This page lists all proposed OSC policies for public comments. Your comments help inform our policies and are encouraged. We will provide response to comments on this webpage after the public comment period closes. Please submit your comments via our online form by the deadline. 

Currently Open for Public Comment:

Scratch Storage Policy

OSC provides a scratch file system as high-performance, high-capacity, shared space. It is temporary without backup. No customer should rely upon the retention of data placed on scratch storage. The purpose of this policy is to (1) keep sufficient scratch space available at all times, (2) maintain good performance of the scratch file system, (3) protect users from misuse of scratch which will result in data loss and (4) reduce manual management of scratch data by OSC staff. 

View this PDF for the proposed policy. 

We need to receive your comments via online form by Friday, April 28, 2017

 

Comment Form 

Supercomputer: 
Service: 

Supercomputing Terms

(alphabetical listing)

account application

The account application comprises the application form, the proposal text, the resume, the performance report, the publications list, and other materials that support a request for use of OSC resources.

Allocations Committee

A committee, comprising volunteers from the Statewide Users Group, that meets bimonthly to make decisions regarding allocations in general and account applications in particular.

authorized users

Authorized users include the principal investigator and secondary investigators who are part of the research team on a project. For classroom accounts, authorized users are the registered students and teaching assistants.

authorized users, adding new ones to existing project

To add a new authorized user to a project, the principal investigator should send the following information about the new user to account@osc.edu:

  • Dr., Mr., Ms, Mrs., etc.
    first name
    middle name/initial
    last name
    position title
    institution
    mailing address
    work telephone number
    fax number
    home telephone number
    e-mail address
    project number
    machines on which user should have accounts
    your fax number, so we can fax you the login account letters

    balance

    To determine your project balance, please log on to any machine and use the following command: OSCusage To maintain a positive balance, please submit follow-up proposals (either standard or major) before your balance reaches a negative level.

    classroom account

    A project that allows students to learn high performance computing or to apply high performance computing to a particular subject. Each student receives a userid. The project is good for one school term. The instructor should submit by e-mail a brief final report about the class. The award is 5000 RUs. Classroom accounts need only administrative review.

    major account

    A account application for 30,000 resource units. Users request this level when their work is rather constant with a significant use of resources.

    NSF Field of Science number

    Refer to the list of National Science Foundation's Field of Science numbers. You may find that more than one applies to your work; put up to three numbers in the blanks.

    performance report

    A report generated by performance tools. Required for major account applications. Please see http://www.osc.edu/supercomputing/computing for machine-specific information.

    principal investigator

    • A full-time, permanent researcher or faculty member of an Ohio college or university.
      • Responsibilities of a principal investigator
        • Central contact and administrator of the project
        • Responsible for keeping records of the resources consumed by authorized users and monitoring account balance
        • Notify account@osc.edu of changes/additions in authorized users
        • Forward to OSC, Allocations Committee, copies of papers, preprints, and technical reports written from results computed on OSC's resources
        • Ensure ethical use of OSC's resources by research team
        • Submitting new account applications in a timely manner

    project

    A project contains one or more proposals, which may or may not be related. Each project has a number consisting of a three- or four-letter prefix and four numbers. Principal investigators may have more than one project, but they should be aware that userids are not transferable from one project to another. Each project contains unique userids. All of the proposals under one project share a common pool of resource units.

    proposal

    The text of a account application that provides the information noted in the Account Application Process Overview.

    publications

    The publications section of the account application form should list publications and presentations that cite OSC (prior to August 2015: acknowledge OSC). New users will not have entries for this section.

    resource unit

    An RU is a resource unit and an mRU is a milli resource unit (i.e., 1 RU = 1000 mRUs). A resource unit is an aggregate measure of the use of CPU, memory, and file storage. For details on charging algorithms, see Charges for Academic Use. Maintaining a positive balance is the PI's responsibility. Use the OSCusage command to monitor your account balance on a daily basis. Principal investigators should submit follow-up account applications before their projects reach a negative balance.

    review process

    Standard and major accounts must undergo a review process. Principal investigators provide the names of recommended reviewers, and OSC administrators identify other potential reviewers. These reviewers review the account application and submit their recommendations to the Allocations Committee.

    reviewers, recommended

    Principal investigators who request standard or major allocations must supply the names of recommended reviewers.

    secondary investigators

    These are authorized users other than the principal investigator. The PI is responsible for keeping OSC updated on changes in authorized users.

    sequence number

    Each proposal under a project has a sequence number for use by OSC's database.

    signature page

    The signature page for first-time principal investigators should be signed by the PI and sent to OSC Help.

    special account

    A special account is one that requires more than 30,000 resource units. Large computing runs and high-consumption software packages sometimes necessitate larger requests. For such a request, please send e-mail to account@osc.edu. Principal investigators who want to request more than 30,000 RUs may be asked to prepare a special petition to the Allocations Committee.

    standard account

    A standard account is a request for 10,000 resource units. It is a peer-reviewed proposal.

    start-up account

    New principal investigators who have never had accounts on OSC's high-performance computers may complete a brief on-line request form for a start-up account. This account receives a one-time 5000 allocation, installed automatically without review. PIs are to monitor their project balances and submit follow-up account applications before their account reaches a negative balance.

    Statewide Users Group

    The Statewide Users Group comprises representatives from Ohio's colleges and universities. The members meet every other month and serve as an advisory body to OSC.

    support from other sources

    If your research is supported by monetary accounts from funding agencies, the Center appreciates learning of this. Such data helps the Center determine its role in Ohio's research activities.

    type of account

    The Center categorizes accounts as classroom (5000 RUs), standard (10,000 RUs), major (30,000 RUs), and special (more than 30,000 RUs). Start-up accounts for first-time users of OSC receive an automatic, one-time 5000 RU allocation.

    userid

    Unique login name of a user on a particular project. If a user is on more than one project, that user will have more than one userid.

    userid, changing owner

    It is possible to change the owner of a userid. The request must come from the principal investigator. Send e-mail to account@osc.edu, giving the userid, the current owner, and the following information about the new owner:

    Dr., Mr., Ms, Mrs., etc.
    first name
    middle name/initial
    last name
    position title
    institution
    mailing address
    work telephone number
    fax number
    home telephone number
    e-mail address
    project number
    machines on which user should have accounts
    your fax number, so we can fax you the login account letters