Application if you are employed by an Ohio Academic Institution

Use of computing resources and services at OSC is subject to the Ohio Supercomputer Center (OSC) Code of Ethics for Academic Users.  Ohio Academic Clients are eligible for highly subsidized access to OSC resources, with fees only accruing after a credit provided is exhausted. Clients from an Ohio academic institution that expect to use more than the credit should consult with their institution on the proper guidance for requesting approval to be charged for usage. See the academic fee structure FAQ page for more information.

Eligible principal investigators (PIs) at Ohio academic institutions are able to request projects at OSC, but should also consult with their institution before incurring charges. In order to be an eligible PI at OSC, you must be eligible to hold PI status at your college, university, or research organization administered by an Ohio academic institution (i.e., be a full-time, permanent academic researcher or tenure-track faculty member at an Ohio college or university). Students, post-doctoral fellows, visiting scientists, and others who wish to use the facilities may be authorized users on projects headed by an eligible PI. Once a PI has received their project information, he/she can manage users for the project. Principal Investigators of OSC projects are responsible for updating their authorized user list, their outside funding sources, and their publications and presentations that cite OSC. All of these tasks can be accomplished by contacting using the client portal. Please review the documentation for more information. PIs are also responsible for monitoring their project's budget (balance) and for requesting a new budget (balance) before going negative, as projects with negative balances are restricted.

OSC's online project requests through our Client Portal is part of an electronic system that leads you through the process step by step. Before you begin to fill in the application form, especially if you are new to the process, look at the academic fee structure page.  You can save a partially completed project request for later use.

If you need assistance, please contact OSC Help.

Application if you are NOT employed by an Ohio Academic Institution

Researchers from businesses, non-Ohio academic, nonprofits, hospitals or other organizations (which do not need to be based in Ohio) who wish to use the OSC's resources should complete the Other Client request form available here. All clients not affiliated with and approved by an Ohio academic institution must sign a service agreement, provide a $500 deposit, and pay for resource usage per a standard price list.

Letter of Commitment for Outside Funding Proposals

OSC will provide a letter of commitment users can include with their account proposals for outside funding, such as from the Department of Energy, National Institutes of Health, National Science Foundation (limited to standard text, per NSF policy), etc. This letter details OSC's commitment to supporting research efforts of its users and the facilities and platforms we provide our users. [Note: This letter does not waive the normal OSC budget process; it merely states that OSC is willing to support such research.] The information users must provide for the letter is:

• address to which the letter should be directed (e.g. NSF, Department, mailing address)
• name of funding agency's representative
• name of the proposal, as well as names and home institutions of the Co-PIs
• budget amount per year you would apply for if you were to receive funding
• number of years of proposed research
• solicitation number

Send e-mail with your request for the commitment letter to OSC Help or submit online. We will prepare a draft for your approval and then we will send you the final PDF for your proposal submission. Please allow at least two working days for this service.

Letter of Support for Outside Funding Proposals

Letters of support may be subject to strict and specific guidelines, and may not be accepted by your funding agency.

If you need a letter of support, please see above "Letter of Commitment for Outside Funding Proposals". 

Applying at NSF Centers

Researchers requiring additional computing resources should consider applying for allocations at National Science Foundation Centers. For more information, please write to oschelp@osc.edu, and your inquiry will be directed to the appropriate staff member.:

We require you cite OSC in any publications or reports that result from projects supported by our services.

Description

This tutorial teaches you about the linux command line and shows you some useful commands. It also shows you how to get help in linux by using the man and apropos commands.

For more training and practice using the command line, you can find many great tutorials. Here are a few:

https://www.learnenough.com/command-line-tutorial

https://cvw.cac.cornell.edu/Linux/

http://www.ee.surrey.ac.uk/Teaching/Unix/

https://www.udacity.com/course/linux-command-line-basics--ud595

More Advanced:

http://moo.nac.uci.edu/~hjm/How_Programs_Work_On_Linux.html

Prerequisites

None.

Introduction

Unix is an operating system that comes with several application programs. Other examples of operating systems are Microsoft Windows, Apple OS and Google's Android. An operating system is the program running on a computer (or a smartphone) that allows the user to interact with the machine -- to manage files and folders, perform queries and launch applications. In graphical operating systems, like Windows, you interact with the machine mainly with the mouse. You click on icons or make selections from the menus. The Unix that runs on OSC clusters gives you a command line interface. That is, the way you tell the operating system what you want to do is by typing a command at the prompt and hitting return. To create a new folder you type mkdir. To copy a file from one folder to another, you type cp. And to launch an application program, say the editor emacs, you type the name of the application. While this may seem old-fashioned, you will find that once you master some simple concepts and commands you are able to do what you need to do efficiently and that you have enough flexibility to customize the processes that you use on OSC clusters to suit your needs.

Common Tasks on OSC Clusters

What are some common tasks you will perform on OSC clusters? Probably the most common scenario is that you want to run some of the software we have installed on our clusters. You may have your own input files that will be processed by an application program. The application may generate output files which you need to organize. You will probably have to create a job script so that you can execute the application in batch mode. To perform these tasks, you need to develop a few different skills. Another possibility is that you are not just a user of the software installed on our clusters but a developer of your own software -- or maybe you are making some modifications to an application program so you need to be able to build the modified version and run it. In this scenario you need many of the same skills plus some others. This tutorial shows you the basics of working with the Unix command line. Other tutorials go into more depth to help you learn more advanced skills.

The Kernel and the Shell

You can think of Unix as consisting of two parts -- the kernel and the shell. The kernel is the guts of the Unix operating system -- the core software running on a machine that performs the infrastructure tasks like making sure multiple users can work at the same time. You don't need to know anything about the kernel for the purposes of this tutorial. The shell is the program that interprets the commands you enter at the command prompt. There are several different flavors of Unix shells -- Bourne, Korn, Cshell, TCshell and Bash. There are some differences in how you do things in the different shells, but they are not major and they shouldn't show up in this tutorial. However, in the interest of simplicity, this tutorial will assume you are using the Bash shell. This is the default shell for OSC users. Unless you do something to change that, you will be running the Bash shell when you log onto Owens or Pitzer.

The Command Prompt

The first thing you need to do is log onto one of the OSC clusters, Owens or Pitzer. If you do not know how to do this, you can find help at the OSC home page. If you are connecting from a Windows system, you need to download and setup the OSC Starter Kit which you can find here. If you are connecting from a Mac or Linux system, you will use ssh. To get more information about using ssh, go to the OSC home page, hold your cursor over the "Supercomputing" menu in the main blue menu bar and select "FAQ." This should help you get started. Once you are logged in look for the last thing displayed in the terminal window. It should be something like

-bash-3.2$

with a block cursor after it. This is the command prompt -- it's where you will see the commands you type in echoed to the screen. In this tutorial, we will abbreviate the command prompt with just the dollar sign - $. The first thing you will want to know is how to log off. You can log off of the cluster by typing "exit" then typing the <Enter> key at the command prompt:

$ exit <Enter>

For the rest of this tutorial, when commands are shown, the <Enter> will be omitted, but you must always enter <Enter> to tell the shell to execute the command you just typed.

First Simple Commands

So let's try typing a few commands at the prompt (remember to type the <Enter> key after the command):

$ date

$ cal

$ finger

$ who

$ whoami

$ finger -l

That last command is finger followed by a space then a minus sign then the lower case L. Is it obvious what these commands do? Shortly you will learn how to get information about what each command does and how you can make it behave in different ways. You should notice the difference between "finger" and "finger -l" -- these two commands seem to do similar things (they give information about the users who are logged in to the system) but they print the information in different formats. try the two commands again and examine the output. Note that you can use the scroll bar on your terminal window to look at text that has scrolled off the screen.

man

The "man" command is how you find out information about what a command does. Type the following command:

$ man

It's kind of a smart-alecky answer you get back, but at least you learn that "man" is short for "manual" and that the purpose is to print the manual page for a command. Before we start looking at manual pages, you need to know something about the way Unix displays them. It does not just print the manual page and return you to the command prompt -- it puts you into a mode where you are interactively viewing the manual page. At the bottom of the page you should see a colon (:) instead of the usual command prompt (-bash-3.2$). You can move around in the man page by typing things at the colon. To exit the man page, you need to type a "q" followed by <Enter>. So try that first. Type

$ man finger

then at the colon of the man page type

: q

You do not have to type <Enter> after the "q" (this is different from the shell prompt.) You should be back at the shell prompt now. Now let's go through the man page a bit. Once again, type

$ man finger

Now instead of just quitting, let's look at the contents of the man page. The entire man page is probably not displayed in your terminal. To scroll up or down, use the arrow keys or the <Page Up> and <Page Down> keys of the keyboard. The <Enter> and <Space> keys also scroll. Remember that "q" will quit out of the man page and get you back to the shell prompt.

The first thing you see is a section with the heading "NAME" which displays the name of the command and a short summary of what it does. Then there is a section called "SYNOPSIS" which shows the syntax of the command. In this case you should see

SYNOPSIS

     finger [-lmsp] [user ...] [user@host ...]

Remember how "finger" and "finger -l" gave different output? The [-lmsp] tells you that you can use one of those four letters as a command option -- i.e., a way of modifying the way the command works. In the "DESCRIPTION" section of the man page you will see a longer description of the command and an explanation of the options. Anything shown in the command synopsis which is contained within square brackets ([ ]) is optional. That's why it is ok to type "finger" with no options and no user. What about "user" -- what is that? To see what that means, quit out of the man page and type the following at the command prompt:

$ whoami

Let's say your username is osu0000. Then the result of the "whoami" command is osu0000. Now enter the following command (but replace osu0000 with your username):

$ finger osu0000

You should get information about yourself and no other users. You can also enter any of the usernames that are output when you enter the "finger" command by itself. The user names are in the leftmost column of output. Now try

$ finger -l osu0000

$ finger -lp osu0000

$ finger -s osu0000 osu0001

For the last command, use your username and the username of some other username that shows up in the output of the "finger" command with no arguments.

Note that a unix command consists of three parts:

• command
• option(s)
• argument(s)

You don't necessarily have to enter an argument (as you saw with the "finger" command) but sometimes a command makes no sense without an argument so you must enter one -- you saw this with the "man" command. Try typing

$ man man

and looking briefly at the output. One thing to notice is the synopsis -- there are a lot of possible options for the "man" command, but the last thing shown in the command synopsis is "name ..." -- notice that "name" is not contained in square brackets. This is because it is not optional -- you must enter at least one name. What happens if you enter two names?

$ man man finger

The first thing that happens is you get the man page for the "man" command. What happens when you quit out of the man page? You should now get the man page for the "finger" command. If you quit out of this one you will be back at the shell prompt.

Combining Commands

You can "pipe" the output of one command to another. First, let's learn about the "more" command:

$ man more

Read the "DESCRIPTION" section -- it says that more is used to page through text that doesn't fit on one screen. It also recommends that the "less" command is more powerful. Ok, so let's learn about the "less" command:

$ man less

You see from the description that "less" also allows you to examine text one screenful at a time. Does this sound familiar? The "man" command actually uses the "less" command to display its output. But you can use the "less" command yourself. If you have a long text file named "foo.txt" you could type

$ less foo.txt

and you would be able to examine the contents of the file one screen at a time. But you can also use "less" to help you look at the output of a command that prints more than one screenful of output. Try this:

$ finger | less

That's "finger" followed by a space followed by the vertical bar (shifted backslash on most keyboards) followed by a space followed by "less" followed by <Enter>. You should now be looking at the output of the "finger" command in an interactive fashion, just as you were looking at man pages. Remember, to scroll use the arrow keys, the <Page Up> and <Page Down> keys, the <Enter> key or the space bar; and to quit, type "q".

Now try the following (but remember to replace "osu0000" with your actual username):

$ finger | grep osu0000

The "grep" command is Unix's command for searching. Here you are telling Unix to search the output of the "finger" command for the text "osu0000" (or whatever your username is.)

If you try to pipe the output of one command to a second command and the second is a command which works with no arguments, you won't get what you expect. Try

$ whoami | finger

You see that it does not give the same output as

$ finger osu0000

(assuming "whoami" returns osu0000.)

In this case what you can do is the following:

$ finger `whoami`

That's "finger" space backquote "whoami" backquote. The backquote key is to the left of the number 1 key on a standard keyboard.

apropos

Enter the following command:

$ man apropos

As you can see, the apropos searches descriptions of commands and finds commands whose descriptions match the keyword you entered as the argument. That means it outputs a list of commands that have something to do with the keyword you entered. Try this

$ apropos

Ok, you need to enter an argument for the "apropos" command.

So try

$ apropos calendar

Now you see that among the results are two commands -- "cal" and "difftime" that have something to do with the keyword "calendar."

Description

This tutorial guides you through the process of creating and submitting a batch script on one of our compute clusters. This is a linux tutorial which uses batch scripting as an example, not a tutorial on writing batch scripts. The primary goal is not to teach you about batch scripting, but for you to become familiar with certain linux commands. There are other pages on the OSC web site that go into the details of submitting a job with a batch script.

Prerequisites

• Familiarity with a text editor (emacs, nano, vim)
• Basic understanding of the unix command line
• Linux Command Line Fundamentals tutorial

Goals

• Create subdirectories to organize information
• Create a batch script with a text editor
• Submit a job
• Check on the progress of the job
• Change the permissions of the output files
• Get familiar with some common unix commands

Step 1 - Organize your directories

When you first log in to our clusters, you are in your home directory. For the purposes of this illustration, we will pretend you are user osu0001 and your project code is PRJ0001, but when you try out commands you must use your own username and project code.

$ pwd
/users/PRJ0001/osu0001

$ touch foo1
$ touch foo2
$ ls
$ ls -l
$ ls -lt
$ ls -ltr

$ mkdir BatchTutorial
$ ls -ltr

$ pwd
/users/PRJ0001/osu0001

$ cd BatchTutorial
$ pwd

$ cd ..
$ pwd

Step 2 -- Get familiar with some more unix commands

Try the following:

$ echo where am I?
$ echo I am in `pwd`
$ echo my home directory is $HOME
$ echo HOME
$ echo this directory contains `ls -l`

These examples show what the echo command does and how to do some interesting things with it. The `pwd` means the result of issuing the command pwd. HOME is an example of an environment variable. These are strings that stand for other strings. HOME is defined when you log in to a unix system. $HOME means the string the variable HOME stands for. Notice that the result of "echo HOME" does not do the substitution. Also notice that the last example shows things don't always get formatted the way you would like.

Some more commands to try:

$ cal
$ cal > foo3
$ cat foo3
$ whoami
$ date

Using the ">" after a command puts the output of the command into a file with the name you specify. The "cat" command prints the contents of a file to the screen.

Two very important UNIX commands are the cp and mv commands. Assume you have a file called foo3 in your current directory created by the "cal > foo3" command. Suppose you want to make a copy of foo3 called foo4. You would do this with the following command:

$ cp foo3 foo4
$ ls -ltr

Now suppose you want to rename the file 'foo4' to 'foo5'. You do this with:

$ mv foo4 foo5
$ ls -ltr

'mv' is short for 'move' and it is used for renaming files. It can also be used to move a file to a different directory.

$ mkdir CalDir
$ mv foo5 CalDir
$ ls
$ ls CalDir

Notice that if you give a directory with the "ls" command is shows you what is in that directory rather than the current working directory.

Now try the following:

$ ls CalDir
$ cd CalDir
$ ls
$ cd ..
$ cp foo3 CalDir
$ ls CalDir

Notice that you can use the "cp" command to copy a file to a different directory -- the copy will have the same name as the original file. What if you forget to do the mkdir first?

$ cp foo3 FooDir

Now what happens when you do the following:

$ ls FooDir
$ cd FooDir
$ cat CalDir
$ cat FooDir
$ ls -ltr

CalDir is a directory, but FooDir is a regular file. You can tell this by the "d" that shows up in the string of letters when you do the "ls -ltr". That's what happens when you try to cp or mv a file to a directory that doesn't exist -- a file gets created with the target name. You can imagine a scenario in which you run a program and want to copy the resulting files to a directory called Output but you forget to create the directory first -- this is a fairly common mistake.

Step 3 -- Environment Variables

Before we move on to creating a batch script, you need to know more about environment variables. An environment variable is a word that stands for some other text. We have already seen an example of this with the variable HOME. Try this:

$ MY_ENV_VAR="something I would rather not type over and over"
$ echo MY_ENV_VAR
$ echo $MY_ENV_VAR
$ echo "MY_ENV_VAR stands for $MY_ENV_VAR"

You define an environment variable by assigning some text to it with the equals sign. That's what the first line above does. When you use '$' followed by the name of your environment variable in a command line, UNIX makes the substitution. If you forget the '$' the substitution will not be made.

There are some environment variables that come pre-defined when you log in. Try using 'echo' to see the values of the following variables: HOME, HOSTNAME, SHELL, TERM, PATH.

Now you are ready to use some of this unix knowledge to create and run a script.

Step 4 -- Create and run a script

Before we create a batch script and submit it to a compute node, we will do something a bit simpler. We will create a regular script file that will be run on the login node. A script is just a file that consists of unix commands that will run when you execute the script file. It is a way of gathering together a bunch of commands that you want to execute all at once. You can do some very powerful things with scripting to automate tasks that are tedious to do by hand, but we are just going to create a script that contains a few commands we could easily type in. This is to help you understand what is happening when you submit a batch script to run on a compute node.

Use a text editor to create a file named "tutorial.sh" which contains the following text (note that with emacs or nano you can use the mouse to select text and then paste it into the editor with the middle mouse button):

$ nano tutorial.sh

echo ----
echo Job started at `date`
echo ----
echo This job is working on node `hostname`

SH_WORKDIR=`pwd`
echo working directory is $SH_WORKDIR
echo ----
echo The contents of $SH_WORKDIR
ls -ltr
echo
echo ----
echo
echo creating a file in SH_WORKDIR
whoami > whoami-sh-workdir

SH_TMPDIR=${SH_WORKDIR}/sh-temp
mkdir $SH_TMPDIR
cd $SH_TMPDIR
echo ----
echo TMPDIR IS `pwd`
echo ----
echo wait for 12 seconds
sleep 12
echo ----
echo creating a file in SH_TMPDIR
whoami > whoami-sh-tmpdir

# copy the file back to the output subdirectory
cp ${SH_TMPDIR}/whoami-sh-tmpdir ${SH_WORKDIR}/output

cd $SH_WORKDIR

echo ----
echo Job ended at `date`

To run it:

$ chmod u+x tutorial.sh
$ ./tutorial.sh

Look at the output created on the screen and the changes in your directory to see what the script did.

Step 5 -- Create and run a batch job

Use your favorite text editor to create a file called tutorial.pbs in the BatchTutorial directory which has the following contents (remember, you can use the mouse to cut and paste text):

#PBS -l walltime=00:02:00
#PBS -l nodes=1:ppn=1
#PBS -N foobar
#PBS -j oe
#PBS -r n

echo ----
echo Job started at `date`
echo ----
echo This job is working on compute node `cat $PBS_NODEFILE`

cd $PBS_O_WORKDIR
echo show what PBS_O_WORKDIR is
echo PBS_O_WORKDIR IS `pwd`
echo ----
echo The contents of PBS_O_WORKDIR:
ls -ltr
echo
echo ----
echo
echo creating a file in PBS_O_WORKDIR
whoami > whoami-pbs-o-workdir

cd $TMPDIR
echo ----
echo TMPDIR IS `pwd`
echo ----
echo wait for 42 seconds
sleep 42
echo ----
echo creating a file in TMPDIR
whoami > whoami-tmpdir

# copy the file back to the output subdirectory
pbsdcp -g $TMPDIR/whoami-tmpdir $PBS_O_WORKDIR/output

echo ----
echo Job ended at `date`

$ qsub tutorial.pbs

$ qstat -u osu0001

                                                                             Req'd  Req'd   Elap
Job ID             Username    Queue    Jobname          SessID NDS   TSK    Memory Time  S Time
------------------ ----------- -------- ---------------- ------ ----- ------ ------ ----- - -----
458842.oak-batch   osu0001     serial   foobar              --      1      1    --  00:02 Q   --

If you see something like this:
                                                                             Req'd  Req'd   Elap
Job ID             Username    Queue    Jobname          SessID NDS   TSK    Memory Time  S Time
------------------ ----------- -------- ---------------- ------ ----- ------ ------ ----- - -----
458842.oak-batch   osu0001     serial   foobar            26276     1      1    --  00:02 R   --

$ ls -ltr
$ cat foobar.oNNNNNN

1. The name you give the job in the script file with the header line #PBS -N foobar
2. The job id number assigned to the job.

The name of the script file (tutorial.pbs) has nothing to do with the name of the output file.

Examine the contents of the output file foobar.oNNNNNN carefully. You should be able to see the results of some of the commands you put in tutorial.pbs. It also shows you the values of the variables PBS_NODEFILE, PBS_O_WORKDIR and TMPDIR. These variables exist only while your job is running. Try

$ echo $PBS_O_WORKDIR

and you will see it is no longer defined. $PBS_NODEFILE is a file which contains a list of all the nodes your job is running on. Because this script has the line

#PBS -l nodes=1:ppn=1

the contents of $PBS_NODEFILE is the name of a single compute node.

Notice that $TMPDIR is /tmp/pbstmp.NNNNNN (again, NNNNNN is the id number for this job.) Try

$ ls /tmp/pbstmp.NNNNNN

Why doesn't this directory exist? Because it is a directory on the compute node, not on the login node. Each machine in the cluster has its own /tmp directory and they do not contain the same files and subdirectories. The /users directories are shared by all the nodes (login or compute) but each node has its own /tmp directory (as well as other unshared directories.)

Prerequisites

Step 1 -- Create a directory to work with and download a "tarball"

Start off with the following:

$ mkdir TarTutorial
$ cd TarTutorial
$ wget http://www.mmm.ucar.edu/wrf/src/WRFDAV3.1.tar.gz
$ ls -ltr

The third command will take a while because it is downloading a file from the internet. The file is call a "tarball" or a "gzipped tarball". TAR is an old unix short name for "tape archive" but a tar file is a file that contains a bunch of other files. If you have to move a bunch of files from one place to another, a good way to do it is to pack them into a tar file, move the tar file where you want it then unpack the files at the destination. A tar file usually has the extension ".tar". What about the ".gz"? This means the tar file has been further compressed with the program gzip -- this makes it a lot smaller.

Step 2 -- Unpack the "tarball" and check out the contents

After step 1 your working directory should be ~/TarTutorial and there should be a file called WRFDAV3.1.tar.gz in it.

Now do this:

$ gunzip WRFDAV3.1.tar.gz
$ ls -ltr

You should now have a file called WRFDAV3.1.tar which should be quite a bit larger in size than WRFDAV3.1.tar.gz -- this is because it has been uncompressed by the "gunzip" command which is the opposite of the "gzip" command.

Now do this:

$ tar -xvf WRFDAV3.1.tar
$ ls -ltr

You should see a lot of filenames go by on the screen and when the first command is done and you issue the ls command you should see two things -- WRFDAV3.1.tar is still there but there is also a directory called WRFDA. You can look at the contents of this directory and navigate around in the directory tree to see what is in there. The options on the "tar" command have the following meanings (you can do a "man tar" to get all the options):

x: extract the contents of the tar file

v: be verbose, i.e. show what is happening on the screen

f: the name of the file which follows the "f" option is the tar file to expand.

Another thing you can do is see how much space is being taken up by the files. Make sure TarTutorial is your working directory then issue the following command:

$ du .

Remember that "." (dot) means the current working directory. The "du" command means "disk usage" -- it shows you how much space is being used by every file and directory in the directory tree. It ends up with the highest level files and directories. You might prefer to do

$ du -h .
$ ls -ltrh

Adding the "-h" option to these commands puts the file sizes in human-readable format -- you should get a size of 66M for the tar file -- that's 66 megabytes -- and "du" should print a size of 77M next to ./WRFDA.

Step 3 -- create your own "tarball"

Now, make your own tar file from the WRFDA directory tree:

$ tar -cf mywrf.tar WRFDA
$ ls -ltrh

You have created a tar from all the files in the WRFDA directory. The options given to the "tar" command have the following meanings:

c: create a tar file

f: give it the name which follows the "f" option

The files WRFDAV3.1.tar and mywrf.tar are identical. Now compress the tar file you made:

$ gzip mywrf.tar
$ ls -ltrh

You should see a file called mywrf.tar.gz which is smaller than WRFDAV3.1.tar.

Step 4 -- Clean up!

You don't want to leave all these files lying around. So delete them

$ rm WRFDAV3.1.tar
$ rm mywrf.tar
$ rm WRFDA

Oops! You can't remove the directory. You need to use the "rmdir" command:

$ rmdir WRFDA

Oh no! That doesn't work on a directory that's not empty. So are you stuck with all those files? Maybe you can do this:

$ cd WRFDA
$ rm *
$ cd ..
$ rmdir WRFDA

That won't work either because there are some subdirectories in WRFDA and "rm *" won't remove them. Do you have to work your way to the all the leaves at the bottom of the directory tree and remove files then come back up and remove directories? No, there is a simpler way:

$ rm -Rf WRFDA

This will get rid of the entire directory tree. The options have the following meanings:

R: recursively remove all files and directories

f: force; i.e., just remove everything without asking for confirmation

I encourage you to do

$ man rm

and check out all the options. Or some of them -- there are quite a few.

Description

This tutorial shows you some handy time-saving shortcuts in linux. Once you have a good understanding of how the command line works, you will want to learn how to work more efficiently.

Prerequisites

Linux command line fundamentals.

Goals

• Save you time when working on a linux system
• Increase your appreciation of the power of linux

Step 1 -- The Arrow Keys

Note: even if you know how to use the up arrow in linux, you need to enter the commands in this section because they are used in the following sections. So to begin this tutorial, go to your home directory and create a new directory called ShortCuts:

$ cd
$ mkdir Shortcuts
$ cd Shortcuts

(If a directory or file named "Shortcuts" already exists, name it something else.)

Imagine typing in a long linux command and making a typo. This is one of the frustrating things about a command line interface -- you have to retype the command, correcting the typo this time. Or what if you have to type several similar commands -- wouldn't it be nice to have a way to recall a previous command, make a few changes, and enter the new command? This is what the up arrow is for.

Try the following:

$ cd ..
$ cd ShortCuts (type a capital C)

Linux should tell you there is no directory with that name.

Now type the up arrow key -- the previous command you entered shows up on the command line, and you can use the left arrow to move the cursor just after the capital C, hit Backspace, and type a lower case c. Note you can also position the cursor before the capital C and hit Delete to get rid of it.

Once you have changed the capital C to a lower case c you can hit Return to enter the command -- you do not have to move the cursor to the end of the line.

Now hit the up arrow key a few times, then hit the down arrow key and notice what happens. Play around with this until you get a good feel for what is happening.

Linux maintains a history of commands you have entered. Using the up and down arrow keys, you can recall previously-entered commands to the command line, edit them and re-issue them.

Note that in addition to the left and right arrow keys you can use the Home and End keys to move to the beginning or end of the command line. Also, if you hold down the Ctrl key when you type an arrow key, the cursor will move by an entire word instead of a single character -- this is useful is many situations and works in many editors.

Let's use this to create a directory hierarchy and a few files. Start in the Shortcuts directory and enter the following commands, using the arrow keys to simplify your job:

$ mkdir directory1
$ mkdir directory1/directory2
$ mkdir directory1/directory2/directory3
$ cd directory1/directory2/diectoryr3  (remember the Home key and the Ctrl key with left and right arrows)
$ hostname > file1
$ whoami > file2
$ mkdir directory4
$ cal > directory4/file3

Step 2 -- Using the TAB key

Linux has short, cryptic command names to save you typing -- but it is still a command line interface, and that means you interact with the operating system by typing in commands. File names can be long, directory hierarchies can be deep, and this can mean you have to type a lot to specify the file you want or change to current working directory. Not only that, but you have to remember the names of files and directories you type in. The TAB key gives you a way to enter with commands with less typing and less memorization.

Go back to the Shortcuts directory:

$ cd
$ cd Shortcuts

Now enter the following:

$ hostname > file1
$ cal > file2
$ whoami > different-file
$ date > other-file
$ cal > folio5

Now type the following, without hitting the Return key:

$ cat oth <Tab>

What happened? Linux completed the name "other-file" for you! The Tab key is your way of telling Linux to finish the current word you are typing, if possible. Because there is only one file in the directory whose name begins with "oth", when you hit the Tab key Linux is able to complete the name.

Hit Return (if you haven't already) to enter the cat command. Now try

$ cat d <Tab>

As you would expect, Linux completes the name "different-file"

What if you enter

$ cat fi <Tab>

Notice Linux completes as much of the name as possible. You can now enter a "1" or a "2" to finish it off.

But what if you forget what the options are? What if you can't remember if you created "file1" and "file2" or if you created "fileA" and fileB"?

With the comman line showing this:

$ cat file

hit the Tab key twice. Aha! Linux shows you the possible choices for completing the word.

Try

$ cat f <Tab>

The Tab will not add anything -- the command line will still read

$ cat f

Now type the letter o followed by a Tab -- once you add the o there is only one possible completion -- "folio".

Now enter the following:

$ cat directory1/directory2/directory3/directory4/file3

That's kind of a painful to type.

Now type the following without entering Return:

$ ls dir <Tab>

Nice! As you would expect, Linux completes the name of the directory for you. This is because there is only one file in the Shortcuts directory whose name begins with "dir"

Hit Return and Linux will tell you that directory1 contains directory2.

Now type this:

$ ls dir <Tab>

and before you hit return type another d followed by another Tab. Your command line should now look like this:

$ ls directory1/directory2/

If you hit Return, Linux will tell you that directory2 contains directory3.

Now try this:

$ ls dir <Tab>

then type another d followed by <Tab> then another d followed by tab. Don't hit Return yet. Your command line should look like this:

$ ls directory1/directory2/directory3/

Don't hit Return yet. Now type the letter f followed by a Tab. What do you think should happen?

Step 3 -- The Exclamation Point

Hitting the up arrow key is a nice way to recall previously-used commands, but it can get tedious if you are trying to recall a command you entered a while ago -- hitting the same key 30 times is a good way to make yourself feel like an automaton. Fortunately, linux offers a couple of other ways to recall previous commands that can be useful.

Go back to the Shortcuts directory

$ cd ~/Shortcuts

and enter the following:

$ hostname
$ cal
$ date
$ whoami

Now enter this:

$ !c

and hit return.

What happened? Now try

$ !h

and hit return.

The exclamation point ("bang" to Americans, "shriek" to some Englishmen I've worked with) is a way of telling linux you want to recall the last command which matches the text you type after it. So "!c" means recall the last command that starts with the letter c, the "cal" command in this case. You can enter more than one character after the exclamation point in order to distinguish between commands. For example if you enter

$ cd ~/Shortcuts
$ cat file1
$ cal
$ !c

the last command will redo the "cal" command. But if you enter

$ cat file1
$ cal
$ !cat

the last command re-executes the "cat" command.

Step 4 -- Ctrl-r

One problem with using the exclamation point to recall a previous command is that you can feel blind -- you don't get any confirmation about exactly which command you are recalling until it has executed. Sometimes you just aren't sure what you need to type after the exclamation point to get the command you want.

Typing Ctrl-r (that's holding down the Ctrl key and typing a lower case r) is another way to repeat previous commands without having to type the whole command, and it's much more flexible than the bang. The "r" is for "reverse search" and what happens is this. After you type Ctrl-r, start typing the beginning of a previously entered command -- linux will search, in reverse order, for commands that match what you type. To see it in action, type in the following commands (but don't hit <Enter> after the last one):

$ cd ~/Shortcuts
$ cat file1
$ cat folio5
$ cal
$ Ctrl-r cat

You should see the following on your command line:

(reverse-i-search)`cat': cat folio5

Try playing with this now. Type in " fi" (that's a space, an "f" and an "i") -- did the command shown at the prompt change? Now hit backspace four times.

Now enter a right or left arrow key and you will find yourself editing the matching command. This is one you have to play around with a bit before you understand exactly what it is doing. So go ahead and play with it.

Step 5 -- history

Now type

$ history

and hit return.

Cool, huh? You get to see all the commands you have entered (probably a maximum of 1000.) You can also do something like

$ history | grep cal

to get all the commands with the word "cal" in them. You can use the mouse to cut and paste a previous command, or you can recall it by number with the exclamation point:

$ !874

re-executes the command number 874 in your history.

For more information about what you can do to recall previous commands, check out http://www.thegeekstuff.com/2011/08/bash-history-expansion/

Step 6 -- Ctrl-t

I am just including this because to me it is a fun piece of linux trivia. I don't find it particularly useful. Type

$ cat file1

and hit <Return>. Now hit the up arrow key to recall this command and hist the left arrow key twice so the cursor is on the "e" of "file1". Now hit Ctrl-t (again, hold down the control key and type a lower case t.) What just happened? Try hitting Ctrl-t a couple more times. That's right -- it transposes two characters in the command line -- the one the cursor is on and the one to its left. Also, it moves the cursor to the right. Frankly, it takes me more time to think about what is going to happen if I type Ctrl-t than it takes me to delete some characters and retype them in the correct order. But somewhere out there is a linux black belt who gets extra productivity out of this shortcut.

Step 7 -- The alias command

Another nice feature of linux is the alias command. If there is a command you enter a lot you can define a short name for it. For example, we have been typing "cat folio5" a lot in this tutorial. You must be getting sick of typing "cat folio5". So enter the following:

$ alias cf5='cat folio5'

Now type

$ cf5

and hit return. Nice -- you now have a personal shortcut for "cat folio5". I use this for the ssh commands:

$ alias gogl='ssh -Y jeisenl@pitzer.osc.edu'

I put this in the .bash_aliases file on my laptop so that it is always available to me.

OSC provides an isolated and custom Jupyter environment for each classroom project that requires Jupyter Notebook or JupyterLab. The interface can be accessed at class.osc.edu. To set up this interface, please apply for a classroom project that is unique for the course. More details on the classroom project can be found in our classroom project guide. The custom Jupyter environment for the course will be tied to this project ID. Please inform us if you have additional requirements for storage. Once we get the information, we will provide you a course ID (which is commonly the course ID provided by PI + school code, e.g. PHYS280_WIT) and add your course to the interface.

Before testing the interface, please follow the instructions below to create a Jupyter environment for the course and send us with the full path of the environment. Once we update the interface, you can login to class.osc.edu and start testing the course. After login, you will see several Apps listed. Pick Jupyter App and that will take you to the Jupyter Job submission page. Please pick your course from the drop-down menu under the Class materials and the number of hours and cores needed.

Set up a Jupyter environment for class

A Jupyter environment for class is created using Python 3 venv module, and it is shared with students. The environment created is an isolated Python environment, meaning that system Python packages and user's local packages become inaccessible when you activate this environment. It is easy for PI to manage packages dedicated for class. 

Using project space

If project space is used, e.g. /fs/ess/projectID/courseID/jupyter where you want to install Jupyter environment, no permission change is required because students would be added into the project.

Create a virtual environment

Login to Owens or Pitzer at OSC (see Getting Connected). In a terminal, assuming that the course ID is  PHYSIC6820 and project ID is PAS1234, run the following commands to create a virtual environment in project space

~support/classroom/tools/create_class_venv --prompt "PHYSIC6820" /fs/ess/PAS1234/PHYSIC6820/jupyter 

Initiate the environment

After creating the virtual environment, activate the environment:

source /fs/ess/PAS1234/PHYSIC6820/jupyter/bin/activate # use activate.csh if you are in a C shell

Then install Jupyter Notebook and python packges for class, e.g. numpy:

(PHYSIC6820)$ pip install jupyterlab jupyter-console qtconsole ipywidgets
(PHYSIC6820)$ pip install numpy
(PHYSIC6820)$ deactivate

The command deactivate is not required if you continue testing and using packages installed in the Jupyter environment. 

We use /fs/ess/projectID/courseID/jupyter as default Jupyter environment for classroom setup. If you set up a different location, you must update us with the full path of the Jupyter environment in order to complete the interface for classroom.  Once the information is recevied, we will update the interfrace, and you can login to class.osc.edu to test your Jupyter classroom.

Install more packages

PI can install more package anytime in a terminal 

source /fs/ess/PAS1234/PHYSIC6820/jupyter/bin/activate # use activate.csh if you use C shell
(PHYSIC6820)$ pip install pkg1 pkg2 ...
(PHYSIC6820)$ deactivate

or in a Jupyter notebook

Enable PDF exporter

To export a notebook as PDF file, it needs complete Tex environment and Pandoc binary (not pandoc python package). The former is taken care of by our system module. For the latter PI needs to install pandoc in a class Jupyter environment:

%%bash
cd $TMPDIR
wget https://github.com/jgm/pandoc/releases/download/2.10.1/pandoc-2.10.1-linux-amd64.tar.gz
tar xf pandoc-2.10.1-linux-amd64.tar.gz -C $VIRTUAL_ENV --strip=1

An extra step might be required to fix the permission issue

%%bash
chmod 755 $VIRTUAL_ENV/jupyter/share/jupyter/nbconvert/templates/html

Enable local package access (optional)

By default this Jupyter environment is an isolated Python environment. Anyone launches python from this environment can only access packages installed inside unless PYTHONPATH is used. PI can change it by setting include-system-site-packages = true in /PATH/TO/JUPYTER/VENV/pyvenv.cfg. This will allows students to access packages in home directory ~/.local/lib/pythonx.x/site-packages ,and install packages via pip install –user.

Workspace

When a class session starts, we create a classroom workspace under PI's and students' home space: $HOME/osc_classes/courseID, and launch Jupyter at the workspace. The root /  will appear in the landing page (Files) but everything can be found in $HOME/osc_classes/courseID on OSC clusters.

Shared Access

Share class material

PI can share class material from home or project space with students. For example, you have a directory called modules in project space /fs/ess/projectID/courseID/modules  (source) that contains the class material. When a student launch a Jupyter session, the diretory will be copied to the student's worksapce $HOME/osc_classes/courseID (destinatation). The student will see the directory modules at the landing page:

PI can add files to the material source directory. New files will be copied to the destination everytime when a new Jupyter session starts. 

Once you decide the location for the material, please update us with the full path of the material directory.

Access student workspace

PI and TAs can access a student's workspace with limited permissions. First, PI sends us a request with the information including PI's and TAs' OSC accounts. After a student launches a class session, you can access known files and directories in the student's workspace. For example, you cannot explore the student's workspace

ls /users/PZS1234/student1/osc_classes/courseID
ls: cannot open directory /users/PZS1234/student1/osc_classes/courseID: Permission denied

but you can access a known file or directory in the workspace

ls /users/PZS1234/student1/osc_classes/courseID/homework

OSC provides an isolated and custom R environment for each classroom project that requires Rstudio. The interface can be accessed at class.osc.edu. Before using this interface, please apply for a classroom project account that is unique for the course.  More details on the classroom project can be found here. The custom R environment for the course will be tied to this project ID. Please inform us if you have additional requirements for the class. Once we get the information, we will provide you a course_ID and add your course to the server with the class module created using the course_ID. After login to the class.osc.edu server, you will see several Apps listed. Pick Rstudio server and that will take you to the Rstudio Job submission page. Please pick your course from the drop-down menu under the Class materials and the number of hours needed. 

Clicking on the Launch will submit the Rstudio job to the scheduler and you will see Connect to Rstudio server option when the resource is ready for the job. Each Rstudio launch will run on 1 core on Owens machine with 4GB of memory. 

Rstudio will open up in a new tab with a custom and isolated environment that is set through a container-based solution. This will create a folder under $HOME/osc_classes/course_ID for each user. Please note that inside the Rstudio, you won't be able to access any files other than class materials. However, you can access the class directory outside of Rstudio to upload or download files.

You can quit a Rstudio session by clicking on File from the top tabs then on the Quit. This will only quit the session, but the resource you requested is still held until walltime limit is reached. To release the resource, please click on DELETE in the Rstudio launch page.

Shared Access

PI can store and share materials like data, scripts, etc, and R packages with the class. We will set up a  project space for the project ID of the course. This project space will be created under /fs/ess/project_ID. Once the project space is ready, please create a folder with the course_ID  and then two subfolders 1) Rpkgs 2) materials, e.g.

mkdir -p /fs/ess/project_ID/course_ID/Rpkgs
mkdir -p /fs/ess/project_ID/course_ID/materials

Shared R packages

Once the class module is ready, PI can access the course at class.osc.edu under the Rstudio job submission page. PI can launch the course environment and install R packages for the class. 

After launching Rstudio, please run the .libPaths() as follows

> .libPaths()
[1] "/users/PZS0680/soottikkal/osc_classes/OSCWORKSHOP/R" "/fs/ess/PZS0687/OSCWORKSHOP/Rpkgs"                  
[3] "/usr/local/R/gnu/9.1/3.6.3/site/pkgs"                "/usr/local/R/gnu/9.1/3.6.3/lib64/R/library" 

Here you will see four R library paths. The last two are system R library paths and are accessible for all OSC users. OSC installs a number of popular R packages at the site location. You can check available packages with library() command. The first path is a personal R library of each user in the course environment and is not shared with students. The second lib path is accessible to all students of the course(Eg: /fs/ess/PZS0687/OSCWORKSHOP/Rpkgs). PI should install R packages in this library to share with the class. As a precaution, it is a good idea to eliminate PI's personal R library from .libPaths() before R package installation as follows. Please note that this step is needed to be done only when preparing course materials by PI.

> .libPaths(.libPaths()[-1])
> .libPaths()
[1] "/fs/ess/PZS0687/OSCWORKSHOP/Rpkgs"          "/usr/local/R/gnu/9.1/3.6.3/site/pkgs"      
[3] "/usr/local/R/gnu/9.1/3.6.3/lib64/R/library"

Now there is only one writable R library path such that all packages will be installed into this library path and shared for all users.

PI can install all packages required for the class using install.packages() function. Once the installation is complete, students will have access to all those packages. 

Please note that students can also install their own packages. Those packages will be installed into their personable library in the class environment i.e., the first path listed under .libPaths()

Shared materials

PI can share materials like data, scripts, and rmd files stored in /fs/ess/project_ID/course_ID/materials with students. When a student launch a Rstduio session, the directory will be copied to the student's workspace $HOME/osc_classes/courseID (destination). Please inform us if you want to use a source directory other than  /fs/ess/project_ID/course_ID/materials. The student will see the directory materials on the landing page. PI can add files to the material source directory. New files will be copied to the destination every time when a new Rstudio session starts.  But If PI modifies exciting files, the changes won't be copied as the files were copied before. Therefore we recommend renaming the file after the update so that it will be copied. 

There are several different ways to copy materials manually from a directory to students' workspace. T

1. On class.osc.edu server, click on Files from the top tabs, then on $HOME directory. From the top right, click on Go to and enter the storage path (Eg: /fs/ess/PZS0687/) in the box and press OK. This will open up storage path and users can copy files. Open the class folder from the $HOME tree shown on left and paste files there. All files copied to  $HOME/osc_classes/course_ID will appear in the Rstudio FIle browser.
   
   
2. On class.osc.edu server, Click on Clusters from the top tabs, then on Owens Shell Access. This will open up a terminal on Owens where students can enter Unix command for copying. Eg: 
   
   cp -r /fs/ess/PZS0687/OSCWORKSHOP/materials $HOME/osc_classes/course_ID
   
   Please note that $HOME/osc_classes/course_ID will be created only after launching Rstudio instance at least once.
3. Students can also upload material directly to Rstudio using the upload tab located in the File browser of Rstudio from their local computer. This assumes they have already downloaded materials to their computers.
   
   

Checklist for PIs

• Apply for a classroom project ID that is unique to the course
• Add yourself, PI, to the project as an authorized user.
• Inform us about additional requirements such as R version or other software
• Once the class module is ready, create class materials under the storage path, and install R packages in the class environment.
• Make a reservation on OSC cluster for the class schedule.
• Invite students to the project at my.osc.edu to give them access to the project ID.

Please reach out to oschelp@osc.edu if you have any questions. 
 

Using ngbrader in Jupyter

Install nbgrader

You can install nbgrader in a notebook:

1. Launch a Juypter session from class.osc.edu
2. Open a new notebook 
3. To Install nbgrader, run:

!pip install nbgrader
!jupyter nbextension install --sys-prefix --py nbgrader --overwrite 
!jupyter nbextension enable --sys-prefix --py nbgrader 
!jupyter serverextension enable --sys-prefix --py nbgrader

To check the installed extensions, run

!jupyter nbextension list

There are six enabled extensions

Configure nbgrader

In order to upload and collect assignments, nbgrader requires a exchange directory with write permissions for everyone. For example, to create a directory in project space, run:

%%bash
mkdir -p /fs/ess/projectID/courseID/exchange
chmod a+wx /fs/ess/projectID/courseID/exchange

Then get your cousre ID for configuratin. In a notebook, run:

%%bash
echo $OSC_CLASS_ID

Finally create the nbgrader configuration at the root of the workspace. In a notebook, run 

%%file nbgrader_config.py
c = get_config()
c.CourseDirectory.course_id = "courseID"     # it must be the value of $OSC_CLASS_ID
c.Exchange.root = "/fs/ess/projectID/courseID/exchange"
c.Exchange.timezone = 'EST'

Once the file is created, you can launch a new Jupyter session then start creating assignments. For using nbgrader, please refer the nbgrader documents.

Access assignments

To let students access the assignments, students need to have the following configuration file in the root of their workspace:

%%file nbgrader_config.py
c = get_config()
c.Exchange.root = "/fs/ess/projectID/courseID/exchange"

The XDMoD tool at xdmod.osc.edu can be used to get an overview of how accurate the requested time of jobs are with the elapsed time of jobs.

One way of specifying a time request is:

#SBATCH --time=xx:xx:xx

The elapsed time is how long the job ran for before completing. This can be obtained using the  sacct command.

$ sacct -u <username> --format=jobid,account,elapsed

It is important to understand that the requested time is used when scheduling a submitted job. If a job requests a time that is much more than the expected elapsed time, then it may take longer to start because the resources need to be allocated for the time that the job requests even if the job only uses a small portion of that requested time.

This allows one to view the requested time accuracy for an individual job, but XDMoD can be used to do this for jobs submitted in over a time range.

First, login to xdmod.osc.edu, see this page for more instructions.

https://www.osc.edu/supercomputing/knowledge-base/xdmod_tool

Then, navigate to the Metric Explorer tab.

Look for the Metric Catalog on the left side of the page and expand the SUPREMM options. Select Wall Hours: Requested: Per Job and group by None.

This will now show the average time requested.

The actual time data can be added by navigating to Add Data -> SUPREMM -> Wall Hours: Per Job.

This will open a new window titled Data Series Definition, to change some parameters before showing the new data. In order to easily distinguish between elapsed and requested time, change the Display Type to Bar, then click add to view the new data.

Now there is a line which shows the average requested time of jobs, and bars which depict the average elapsed time of jobs. Essentialy, the closer the bar is to the line, without intersecting the line, the more accurate the time predicition. If the bar intersects the line, then it may indicate the there was not enough time requested for a job to complete, but remember that these values are averages.

One can also view more detailed information about these jobs by clicking a data point and using the Show raw data option.

While our Python installations come with many popular packages installed, you may come upon a case in which you need an additional package that is not installed. If the specific package you are looking for is available from anaconda.org (formerlly binstar.org), you can easily install it and required dependencies by using the conda package manager.

To be able to install a package using the conda package manager:

• You must use an Anaconda distribution of Python:
  • Load an appropriate Python module, for example module load python/3.6-conda5.2 on Pitzer.
  • Run conda --version to check that a version of Anaconda was successfully loaded.
• Package should be available through anaconda.org and/or using pip (see next section).

If you would like to freeze a distribution so that you can control how often you update Anaconda, please send us a help request at oschelp@osc.edu.

Procedure

The following steps are an example of how to set up a Python environment and install packages to a local directory using conda. We use the name  local for the environment, but you may use any other name.

Load proper Python module

module load python/3.6-conda5.2

Create Python installation to local directory

Three alternative create commands are listed. These cover the most common cases.

Create New Environment

The following will create a minimal Python installation without any extraneous packages:

conda create -n local

Clone base environment

If you want to clone the full base Python environment from the system, you may use the following create command:

conda create -n local --clone base

Create New Environment with specific packages

You can augment the command above by listing specific packages you would like installed into the environment. For example, the following will create a minimal Python installation with only the specified packages (in this case, numpy and babel):

conda create -n local numpy babel

By default, conda will install the newest versions of the packages it can find. Specific versions can be specified by adding =<version> after the package name. For example, the following will create a Python installation with Python version 2.7 and NumPy version 1.16:

conda create -n local python=2.7 numpy=1.16

To verify that a clone has been created, use the command

conda info -e

For additional conda command documentation see https://docs.conda.io/projects/conda/en/latest/commands.html#conda-general-commands

Activate environment

Before the created environment can be used, it must be activated.

For the bash shell:

source activate local

On newer versions of Anaconda on the Owens cluster you may also need to perform the removal of the following packages before trying to install your specific packages:

conda remove conda-build
conda remove conda-env

Install packages

To install additional packages, use the conda install command. For example, to install the yt package:

conda install yt

By default, conda will install the newest version if the package that it can find. Specific versions can be specified by adding =<version> after the package name. For example, to install version 1.16 of the NumPy package:

conda install numpy=1.16

Test Python package

Now we will test our installed Python package by loading it in Python and checking its location to ensure we are using the correct version. For example, to test that NumPy is installed correctly, run

python -c "from __future__ import print_function; import numpy; print(numpy.__file__)"

and verify that the output generally matches

$HOME/.conda/envs/local/lib/python3.6/site-packages/numpy/__init__.py

To test installations of other packages, replace all instances of numpy with the name of the package you installed.

Install your own Python packages

If the method using conda above is not working, or if you prefer, you can consider installing Python packages from the source. Please read HOWTO: install your own Python packages.

But I use virtualenv and/or pip!

See the comparison to these package management tools here:

https://docs.conda.io/projects/conda/en/latest/commands.html#conda-vs-pip-vs-virtualenv-commands

Use pip only without conda package manager

pip installations are supported:

module load python
module list                            # check which python you just loaded
pip install --user --upgrade PACKAGE   # where PACKAGE is a valid package name

Note the default installation prefix is set to the system path where OSC users cannot install the package. With the option --user, the prefix is set to $HOME/.local where lib, bin, and other top-level folders for the installed packages are placed. Finally, the option --upgrade will upgrade the existing packages to the newest available version.

The one issue with this approach is portability with multiple Python modules. If you plan to stick with a single Python module, then this should not be an issue. However, if you commonly switch between different Python versions, then be aware of the potential trouble in using the same installation location for all Python versions.

Use pip in a Python virtual environment (Python 3 only)

In some cases where the conda package installations have no source from conda channels or have dependency issues, you may consider using pip in a Python virtual environment.

To create a virtual environment:

module reset
python3 -m venv --without-pip $HOME/venv/mytest --prompt "local"
source $HOME/venv/mytest/bin/activate
(local) curl https://bootstrap.pypa.io/get-pip.py |python     # get the newest version of pip
(local) deactivate

where we use the path $HOME/venv/mytest and the name local for the environment, but you may use any other path and name. 

To activate and deactivate the virtual environment:

source $HOME/venv/mytest/bin/activate
(local) deactivate 

To install packages:

source $HOME/venv/mytest/bin/activate
(local) pip install PACKAGE 

You don't need the --user option within the virtual environment.  

Further Reading

Conda Test Drive: https://conda.io/docs/test-drive.html 

This page outlines ways to generate and view performance data for your program using tools available at OSC.

Intel Tools

This section describes how to use performance tools from Intel. Make sure that you have an Intel module loaded to use these tools.

Intel VTune

Intel VTune is a tool to generate profile data for your application. Generating profile data with Intel VTune typically involves three steps:

1. Prepare the executable for profiling.

You need executables with debugging information to view source code line detail: re-compile your code with a -g  option added among the other appropriate compiler options. For example:

mpicc wave.c -o wave -g -O3

2. Run your code to produce the profile data.

Profiles are normally generated in a batch job. To generate a VTune profile for an MPI program:

mpiexec <mpi args> amplxe-cl <vtune args> <program> <program args>

where <mpi args> represents arguments to be passed to mpiexec, <program> is the executable to be run, <vtune args> represents arguments to be passed to the VTune executable amplxe-cl, and <program args> represents arguments passed to your program.

For example, if you normally run your program with mpiexec -n 12 wave_c, you would use

mpiexec -n 12 amplxe-cl -collect hotspots -result-dir r001hs wave_c

To profile a non-MPI program:

amplxe-cl <vtune args> <program> <program args>

The profile data is saved in a .map file in your current directory.

As a result of this step, a subdirectory that contains the profile data files is created in your current directory. The  subdirectory name is based on the -result-dir argument and the node id, for example, r001hs.o0674.ten.osc.edu.

3. Analyze your profile data.

You can open the profile data using the VTune GUI in interactive mode. For example:

amplxe-gui r001hs.o0674.ten.osc.edu

One should use an OnDemand VDI (Virtual Desktop Interface) or have X11 forwarding enabled (see Setting up X Windows). Note that X11 forwarding can be distractingly slow for interactive applications.

Intel ITAC

Intel Trace Analyzer and Collector (ITAC) is a tool to generate trace data for your application. Generating trace data with Intel ITAC typically involves three steps:

1. Prepare the executable for tracing.

You need to compile your executbale with -tcollect  option added among the other appropriate compiler options to insert instrumentation probes calling the ITAC API. For example:

mpicc wave.c -o wave -tcollect -O3

2. Run your code to produce the trace data.

mpiexec -trace <mpi args> <program> <program args>

For example, if you normally run your program with mpiexec -n 12 wave_c, you would use

mpiexec -trace -n 12 wave_c

As a result of this step, .anc, .f, .msg, .dcl, .stf, and .proc files will be generated in your current directory.

3. Analyze the trace data files using Trace Analyzer

You will need to use traceanalyzer to view the trace data. To open Trace Analyzer:

traceanalyzer /path/to/<stf file>

where the base name of the .stf file will be the name of your executable.

One should use an OnDemand VDI (Virtual Desktop Interface) or have X11 forwarding enabled (see Setting up X Windows) to view the trace data. Note that X11 forwarding can be distractingly slow for interactive applications.

Intel APS

Intel's Application Performance Snapshot (APS) is a tool that provides a summary of your application's performance . Profiling HPC software with Intel APS typically involves four steps:

1. Prepare the executable for profiling.

Regular executables can be profiled with Intel APS. but source code line detail will not be available. You need executables with debugging information to view source code line detail: re-compile your code with a -g  option added among the other approriate compiler options. For example:

mpicc wave.c -o wave -tcollect -O3

2. Run your code to produce the profile data directory.

Profiles are normally generated in a batch job. To generate profile data for an MPI program:

mpiexec -trace <mpi args> <program> <program args>

where <mpi args> represents arguments to be passed to mpiexec, <program> is the executable to be run and <program args> represents arguments passed to your program.

For example, if you normally run your program with mpiexec -n 12 wave_c, you would use

mpiexec -n 12 wave_c

To profile a non-MPI program:

aps <program> <program args>

The profile data is saved in a subdirectory in your current directory. The directory name is based on the date and time, for example, aps_result_YYYYMMDD/. 

3. Generate the profile file from the directory.

To generate the html profile file from the result subdirectory:

aps --report=./aps_result_YYYYMMDD

to create the file aps_report_YYYYMMDD_HHMMSS.html.

4. Analyze the profile data file.

You can open the profile data file using a web browswer on your local desktop computer. This option typically offers the best performance.

ARM Tools

This section describes how to use performance tools from ARM.

ARM MAP

Instructions for how to use MAP is available here.

ARM DDT

Instructions for how to use DDT is available here.

ARM Performance Reports

Instructions for how to use Performance Reports is available here.

Other Tools

This section describes how to use other performance tools.

HPC Toolkit

Rice University's HPC Toolkit is a collection of performance tools. Instructions for how to use it at OSC is available here.

TAU Commander

TAU Commander is a user interface for University of Oregon's TAU Performance System. Instructions for how to use it at OSC is available here.

To connect to OSC services, a secure tunnel to a session is required. This can be done relatively simply in OSX and Linux by using the SSH functionality built into the system, but Windows users have had to configure and use third party applications like PuTTY or Java to access secure resources at OSC. OSC Connect is a native windows application written in C# and compiled for .NET 2.0, providing preconfigured management of secure tunnel connections for Windows users, as well as providing a launcher for secure file transfer, VNC, terminal, and web based services. 

OSC Connect is supported on Windows versions from Windows XP through Windows 10.

We've created a brief video:

Getting Started

1. Download the latest release of OSC Connect from here: Latest Build
   • Click OSCConnect.exe  and download. Use "Save link as" to download this file to a folder of your choice. 
   • See here for all the previous releases. 
2. Double-Click the OSCConnect.exe icon to run the application. In the current state, OSC Connect is entirely deployed by a single executable file; no further installation is required.   ​
3. When you run OSC Connect, a temporary folder named "ConnectFiles" with four additional files will be created. These are required for proper functionality of the application. Please ensure that these files are permitted by your IT administrator:
   • plink.exe is the command-line version of PuTTY used by the application to create the secure connection to OSC resources.
   • putty.exe  is the GUI application of PuTTY used to provide terminal emulation remote console connections to OSC resources.
   • vncviewer.exe is the VNC viewer client used to view a remote desktop session.
   • WinSCP.exe  is the SFTP client used for file transfer.

4. Once your connections to OSC services as well as the OSC Connect app is closed, the temporary folder named "ConnectFiles" will be removed automatically. 

OSC Connect Graphical User Interface

After you double-click the  OSCConnect.exe  icon  , the application graphical user interface is shown as below:

• Network Status: it indicates which OSC cluster you will be connected to. The option can be changed in "Settings". 

• Settings: it provides several configuration options to modify the behavior of the application. 

  • Connection Settings: use this dropdown to select the default host/cluster. Selecting a server here will change the endpoint for tunneling, sftp connections, console connections, and connectivity checking.

  • System Settings: 

    • Detect Clipboard Activity: when this option is enabled, the application will detect valid data on the Windows clipboard and populate the application. ( Default: Off )

    • Check for New Versions: when this option is enabled, the application will check for version updates. (Default: on)

  • Automation Settings:

    • Save User Credentials: when this option is enabled, it allows the application to remember the user when the application is reopened. This saves the user credentials to the user settings using DPAPI Encryption. Passwords are decrypted only by current the Windows user account. ( Default: Off )

    • Launch Tunnel On Import: when this option is enabled, the tunnel will automatically connect when the application detects a valid clipboard string and the user credentials have been entered. ( Default: On )

  • VNC Settings

• ​About: it provides additional information about OSC Connect.

After you provide your OSC Credentials, i.e. your OSC HPC username and password, more functionalities are available as shown below:

• SFTP: it uses WinSCP as the SFTP client to securely connect to the OSC file system over SFTP.
• Console: it uses PuTTY to provides you shell access to OSC cluster.

In addition, Session Type is provided such that advanced users can connect to a running session mannually.

Connecting to a Session

The OSC Connect application can be used to connect to a running session launched through OSC OnDemand.

1. Navigate to OSC Demand to create a new session under either "Desktops" or "Desktop Apps". 
   • More information on how to use OSC OnDemand can be found here
2. Once your session is running, you have two options to connect to the session: "noVNC Instructions" and "Native Client Instructions". Click "Native Client Instructions" to use OSC Connect. 
3. Follow the steps below to connect to the running session:
   • Step 1: Download the latest OSC Connect if you don't already have it. See "Getting Started" for more details. 
   • Step 2: Launch OSC Connect. 
   • Step 3: Click - osc://xxxxx . osc://xxxxx is a custom URI scheme that is registered when you launch the application. Simply click the link to populate the configuration information and connect to your running session. If OSCConnect.exe is not running when you click the URI, the OSC Connect application will be popped up. Enter your OSC HPC username and password and you will be able to connect to the session by clicking the "Connect" button.
   • For advanced users, you can skip step 3 and connect to a running session mannually. Depending on the type of application, you can choose either "iHPC VNC Desktop" to create a VNC tunnel or "Web Browser Session" to create a browser tunnel. 

FAQ

I've clicked the osc:// link and nothing happened.

Be sure to run OSCConnect.exe at least once. The initial launch will add a key to your user registry that initializes the URI scheme. If you move or rename the OSCConnect.exe file, you will need to run the application again manually to update the path in the handler.

I've received the error "Unable to open helper application. The protocol specified in this address is not valid."

This issue appears in some earlier versions of Internet Explorer when attempting to launch the application from a Temporary location. Download and run the OSCConnect.exe application, being sure to save the file to a non-temporary location.

This article focuses on debugging strategies for C/C++ codes, but many are applicable to other languages as well.

Rubber Duck Debugging

This approach is a great starting point. Say you have written some code, and it does not do what you expect it to do. You have stared at it for a few minutes, but you cannot seem to spot the problem.

Try explaining what the problem is to a rubber duck. Then, walk the rubber duck through your code, line by line, telling it what it does. Don’t have a rubber duck? Any inanimate object will do (or even an animate one if you can grab a friend).

It sounds silly, but rubber duck debugging helps you to get out of your head, and hopefully look at your code from a new perspective. Saying what your code does (or is supposed to do) out loud has a good chance of revealing where your understanding might not be as good as you think it is.

Printf() Debugging

You’ve written a whole bunch of new code. It takes some inputs, chugs along for a while, and then creates some outputs. Somewhere along this process, something goes wrong. You know this because the output is not at all what you expected. Unfortunately, you have no idea where things are going wrong in the code.

This might be a good time to try out printf() debugging. It’s as simple as its name implies: simply add (more) printf() statements to your code. You’ve likely seen this being used. It’s the name given to the infamous ‘printf(“here”);’ calls used to verify that a particular codepath is indeed taken.

Consider printing out arguments and return values to key functions. Or, the results or summary statistics from large calculations. These values can be used as “sanity checks” to ensure that up until that point in the code, everything is going as expected.

Assertion calls, such as "assert(...)", can also be used for a similar purpose. However, often the positive feedback you get from print statements is helpful in when you’re debugging. Seeing a valid result printed in standard out or a log file tells you positively that at least something is working correctly.

Debuggers

Debuggers are tools that can be used to interactively (or with scripts) debug your code. A fairly common debugger for C and C++ codes is gdb. Many guides exist online for using gdb with your code.

OSC systems also provide the ARM DDT debugger. This debugger is designed for use with HPC codes and is arguably easier to use than gdb. It can be used to debug MPI programs as well.

Debuggers allow you to interact with the program while it is running. You can do things like read and write variable values, or check to see if/when certain functions are called.

Testing

Okay, this one isn’t exactly a debugging strategy. It’s a method to catch bugs early, and even prevent the addition of bugs. Writing a test suite for your code that’s easy to run (and ideally fast) lets you test new changes to ensure they don’t break existing functionality.

There are lots of different philosophies on testing software. Too many to cover here. Here’s two concepts that are worth looking into: unit testing and system testing.

The idea behind unit testing is writing tests for small “units” of code. These are often functions or classes. If you know that the small pieces that make up your code work, then you’ll have more confidence in the overall assembled program. There’s an added architecture benefit here too. Writing code that is testable in the first place often results in code that’s broken up into separate logical pieces (google “separation of concerns”). This makes your code more modular and less “spaghetti-like”. Your code will be easier to modify and understand.

The second concept – system testing – involves writing tests that run your entire program. These often take longer than unit tests, but have the added benefit that they’ll let you know whether or not your entire program still works after introducing a new change.

When writing tests (both system and unit tests), it’s often helpful to include a couple different inputs. Occasionally a program may work just fine for one input, but fail horribly with another input.

Minimal, Reproducible Example

Maybe your code takes a couple hours (or longer…) to run. There’s a bug in it, but every time you try to fix it, you have to wait a few hours to see if the fix worked. This is driving you crazy.

A possible approach to make your life easier is to try to make a Minimal, Reproducible Example (see this stackoverflow page for information).

Try to extract just the code that fails, from your program, and also its inputs. Wrap this up into a separate program. This allows you to run just the code that failed, hopefully greatly reducing the time it takes to test out fixes to the problem.

Once you have this example, can you make it smaller? Maybe take out some code that’s not needed to reproduce the bug, or shrink the input even further? Doing this might help you solve the problem.

Tools and other resources

• Compiler warnings – compilers are your friend. Chances are your compiler has a flag that can be used to enable more warnings than are on by default. GNU tools have “-Wall” and “-Wextra”. These can be used to instruct the compiler to tell you about places in the code where bugs may exist.
• The Practice of Programming by Brian Kernighan and Rob Pike contains a very good chapter on debugging C and C++ programs.
• Valgrind is a tool that can be used for many types of debugging including looking for memory corruptions and leaks. However, it slows down your code a very sizeable amount. This might not be feasible for HPC codes
• ASAN (address sanitizer) is another tool that can be used for memory debugging. It is less featureful than Valgrind, but runs much quicker, and so will likely work with your HPC code.

The Owens and Pitzer clusters have access to the DDN Infinite Memory Engine (IME), a fast data tier between the compute nodes and the /fs/scratch file system. IME is a Solid State Disk (SSD) that can act as a cache and burst buffer to improve the performance of the scratch file system.

While some jobs will benefit from using IME, others will not. You need to understand the workflow of IME well, since sometimes it is not very intuitive. Files must be explicitly imported, synchronized and/or purged.

When to Use IME

Benefits of Using IME

The most obvious reason for using IME is to speed up the I/O in your job. Jobs with heavy I/O may benefit from IME.

Another reason to use IME is to reduce the I/O load on the scratch file system. A job that writes a large amount of data to /fs/scratch in a short period of time may overload the servers and degrade performance for other users. Even if IME doesn’t improve performance for that job, it may have a system-wide benefit.

Limitations of IME

IME should not be used if many small files are being written or if files are opened and closed frequently. Metadata-intensive operations such as these are likely to be slower on IME than on the underlying file system.

We recommand to test properly before using IME for production. If data is not managed properly (see below) it’s possible to lose data or end up with corrupted files.

The IME system at OSC is still somewhat new and may not be as stable as the standard file systems.

IME Concepts

IME is a user-managed cache available on Owens and Pitzer scratch file systems. Data in /fs/scratch is not automatically loaded into IME. Similarly, data written to IME is not automatically written to /fs/scratch . Here we define some terminology as a means of introducing important concepts.

resident

File data is resident in IME if it is present in the IME layer. It may or may not also exist in /fs/scratch . Data is made resident either by explicitly importing it with the IME command-line utility ime-ctl or by directly writing data to the IME.

clean

Data resident in IME is clean if it has not been modified since it was last imported or synchronized. Clean data matches the corresponding file data in /fs/scratch unless the file was modified in /fs/scratch . A file typically consists of many fragments, or chunks, some of which may be clean and others dirty.

dirty

Data resident in IME is dirty if it has been written or modified and not synchronized. Dirty data are not visible on /fs/scratch and may also not be visible from other IME clients.

import

A file in /fs/scratch may be imported into IME using the command-line utility ime-ctl . The file is then resident in IME; the file in /fs/scratch is unchanged. Reading a file through the IME interface without explicitly importing it does not import it, or make it resident in IME, and is typically slower than reading it directly from /fs/scratch .

synchronize

Synchronization is the process of writing dirty file data from IME to /fs/scratch . The user must explicitly synchronize an IME-resident file, using the command-line utility ime-ctl , to make it visible on /fs/scratch . There is no automatic synchronization. Exception: If IME gets too full, the system will synchronize and purge older resident data to make space for new data.

purge

File data resident in IME remains resident until it is purged. Files should be purged when they are no longer needed in IME, typically at the end of a job. Because IME is small (40TB) compared to the size of /fs/scratch , all users are asked to purge their data from IME promptly unless there is a reason for keeping it resident. Purging a file from IME does not affect the corresponding file in /fs/scratch . Unsynchronized data will be lost if it is purged. The command-line utility ime-ctl is used to purge data. The system may purge resident data if IME gets too full; dirty data is synchronized before purging.

FUSE

The FUSE (Filesystem in Userspace) interface is a POSIX mount point at /ime/scratch . The /ime/scratch directory is available only through batch jobs that explicitly request it (see below); it is not mounted on the login nodes. All files in /fs/scratch are visible in /ime/scratch , even when they are not resident in IME.

Relationship between IME and /fs/scratch

The interaction between IME and /fs/scratch is complex and sometimes not intuitive. Here are a few points to keep in mind.

• All files and data on /fs/scratch are visible from IME even if they are not resident in IME.
• All files in IME are visible from /fs/scratch although data is not available unless it is synchronized. File size may not be correct, showing as 0 for unsynchronized files created and written in IME.
• File permissions are the same in /ime/scratch and /fs/scratch . If a read-only file is created in IME, it can't be synchronized because the file in scratch can't be written. If a read-only file is imported into IME, it can't be purged from IME.
• If a file is changed in /fs/scratch while it is resident in IME, results are unpredictable. Data loss or corruption is likely. Purge the file from IME before modifying in /fs/scratch .
• Although files are not automatically synchronized or purged, the system may take these actions without warning if it becomes necessary to free up space in IME. Dirty data is synchronized before it is purged in this situation.
• Recall that IME is much smaller than /fs/scratch with capacity of 40TB.

How to Access IME

IME is available at OSC only through batch jobs, not on the login nodes.

There are three ways to access IME. We consider them in order of increasing programming effort and performance.

/ime/scratch FUSE filesystem

The easiest way to access IME is to use the directory /ime/scratch , a POSIX mount point known as the FUSE (Filesystem in Userspace) interface.

The /ime/scratch directory is mounted on a compute node only when a user job explicitly requests it by adding the :ime specification to a node request:

#SBATCH --nodes=1 --ntasks=40 --gres=ime

All files in /fs/scratch will appear under /ime/scratch , but that doesn't mean the data is resident in the IME. Accessing files on /fs/scratch through the /ime/scratch directory is slower than accessing them directly.

See the Use Cases below for examples on how to access IME through the FUSE interface.

$IMEDIR environment variable

Every job that requests IME has an environment variable $IMEDIR pointing to the IME counterpart to the $PFSDIR job-temporary scratch directory. That is, if $PFSDIR is /fs/scratch/MYGROUP/myname/jobid , then $IMEDIR is /ime/scratch/MYGROUP/myname/jobid . Since $PFSDIR is deleted at the end of the job, $IMEDIR goes away also.

MPI-IO API (or libraries built on this API)

Some of our MPI installations are built using an IME-specific version of ROMIO to provide high performance MPI-IO operations on IME. You must build and run your application using one of the special MPI modules to take advantage of this feature. This is not currently available at OSC.

IME Command Line Utility

The ime-ctl command is used to import, synchronize, and purge IME file data as well as check status. It works through the FUSE interface, /ime/scratch . You can include these commands in a batch job with the :ime specification on the nodes= line.

To manage your IME file data outside of a batch job you should use an interactive batch job:

salloc -t 1:00:00 --ntasks-per-node=1 --gres=ime srun --pty /bin/bash

Following are some useful ime-ctl options. In all cases the file or directory name may be specified as either an absolute or a relative path and must be located in the /ime/scratch directory.

Add -R to make an operation recursive: ime-ctl -p -R mydirectory

help

ime-ctl -h

Displays help and usage message.

import

ime-ctl -i /ime/scratch/filename

Import the entire file, first purging any clean data already resident.

ime-ctl -i -K /ime/scratch/filename

Import the file data but keep existing clean data during the import operation.

Note: The import operation by default is nonblocking. Add -b to block on import completion.

ime-ctl -i -b /ime/scratch/filename

synchronize

ime-ctl -r /ime/scratch/filename

Synchronize the file data from IME to /fs/scratch , keeping the data in IME.

Note: The synchronize operation by default is nonblocking. Add -b to block on synchronize completion.

ime-ctl -r -b /ime/scratch/filename

purge

ime-ctl -p /ime/scratch/filename

Purge the file data from IME. Unsynchronized data will be lost. The file on /fs/scratch is not affected.

show fragment status

ime-ctl -s /ime/scratch/filename

Show fragment status of file data resident in IME. Fragments can be in one of four states: dirty, pending (in the process of synchronizing), clean, deletable (in the process of being purged).

Use Cases

This section gives detailed command sequences for some common situations where IME may be used. The examples strongly depend on the software it uses. So, you need to undertand the I/O structure of the software.

Temporary files

Temporary files are those that are written and read back in but discarded at the end of a job. In this example we use the FUSE interface with the job-specific $IMEDIR directory. Files are written and read but not synchronized. At the end of the job $IMEDIR is automatically removed. This example was tested on Owens.

#!/bin/bash
# Serial job that uses $IMEDIR for temporary files (read/write)
#SBATCH --job-name=temporary_to_ime
#SBATCH --nodes=1 --ntasks=28 --gres=ime
#SBATCH --time=1:00:00
#SBATCH --account={MYACCT} # put your primery account group, like PAS1234
module load bwa

# Change to job-specific IME temporary directory
# $IMEDIR is automatically deleted at end of job
cp upstream1000.fa $IMEDIR
cd $IMEDIR

# Run program, assuming temporary files will be written to current directory
bwa index upstream1000.fa

# check status
ls -l
ime-ctl -s $IMEDIR/upstream1000.fa.bwt
# Note: No need to purge temporary files because directory will be deleted at end of job

ime-ctl -s will display the file's status, for example

File: `/ime/scratch/{your file location}/4239XXX.owens-batch.ten.osc.edu/upstream1000.fa.bwt'
Number of bytes:
Dirty: 43886080
Clean: 0
Syncing: 0

As you see, the file is completely "Dirty", so you know that it is only on IME side.

Output files to be kept

In this example an output file is written to IME. At the end of the job the file is synchronized and then purged from IME, remaining on the scratch file system.

#!/bin/bash
# Serial job that writes output files to IME
#SBATCH --job-name=output_to_ime
#SBATCH --nodes=1 --ntasks=28 --gres=ime
#SBATCH --time=1:00:00 
#SBATCH --account={MYACCT} # put your primery account group, like PAS1234

module load bwa

# create working directory under IME, and copy input files.
# The input files can be read from regular file system as well.
export IME_WORKDIR=/ime/scratch/{your file location}
mkdir $IME_WORKDIR
cp upstream1000.fa $IME_WORKDIR
cd $IME_WORKDIR

# Run program, assuming output will be written to current directory (IME directory)
bwa index upstream1000.fa

# Wait for synchronization to complete (blocking)
ime-ctl -b -R -r $IME_WORKDIR

# checking status
ls -l
ime-ctl -s $IME_WORKDIR/upstream1000.fa.bwt

Large read-only files used by multiple jobs

Some workflows involve large input files that are used by many jobs but never modified. This example keeps the input file resident in IME, re-importing it as necessary. If the file is ever changed it must be manually purged or completely reloaded; this is not part of the job workflow.

#!/bin/bash
#Job with large read-only input file used by multiple jobs
#SBATCH --job-name=read_only_to_ime
#SBATCH --nodes=1 --ntasks=28 --gres=ime
#SBATCH --time=1:00:00
#SBATCH --account={MYACCT} # put your primery account group, like PAS1234

module load bwa

# We assume the input file is located in /fs/scratch, then we import to IME
export IME_WORKDIR=/ime/scratch/{your file location}
export INPUTFILE=/fs/scratch/{your file location}/upstream1000.fa

# Get IME FUSE path for input file (changes /fs to /ime)
export IME_INPUTFILE=$(echo $INPUTFILE | sed 's/^\/fs/\/ime/')

# import to IME
ime-ctl -b -i $IME_INPUTFILE

cd $IME_WORKDIR

# Run program
bwa index upstream1000.fa

# check status
ls -l
ime-ctl -s $IME_WORKDIR/upstream1000.fa.bwt

Checkpoint files

Checkpoint files are written by a program to allow restart in case the program is terminated before completion. If the program completes the checkpoint file is discarded. Checkpoint files should not be written to $IMEDIR because they need to persist beyond the end of the job. In this example the checkpoint files are left in IME if the script does not complete and must be manually recovered or purged, or used directly from IME by a subsequent job.

#!/bin/bash
# Job that writes checkpoint files to IME
#SBATCH --job-name=checkpoint_to_ime
#SBATCH --nodes=1 --ntasks=28 --gres=ime
#SBATCH --time=1:00:00
#SBATCH --account={MYACCT} # put your primery account group, like PAS1234

module load qchem/5.1.1-openmp
module list

export CKPTDIR=/fs/scratch/{your file location}/ckptdir
mkdir $CKPTDIR

# Get IME FUSE path for checkpoint directory (changes /fs to /ime)
export IME_CKPTDIR=$(echo $CKPTDIR | sed 's/^\/fs/\/ime/')

# Set checkpoint path in Q-Chem
export QCSCRATCH=$IME_CKPTDIR

# Run program, writing checkpoint files to $IME_CKPTDIR
qchem -save -nt $SLURM_NTASKS HF_water.in HF_water.out HF_water

# If program completed successfully delete checkpoint files
retVal=$?
if [ $retVal -eq 0 ]; then
    rm -r $CKPTDIR
fi
exit $retVal

# Note: If program did not complete successfully or job was killed, checkpoint
#   files will remain in IME and can be synchronized and purged manually.
#   Or they can be used directly from IME by a subsequent job.

An eligible principal investigator (PI) heads a project account and can authorize/remove user accounts under the project account (please check our Allocations and Accounts documentation for more details). This document shows you how to identify users on a project account and check the status of each user. 

Identify Users on a Project Account

If you know the project acccount

If the project account (projectID) is known, the OSCgetent command will list all users on the project:

$ OSCgetent group projectID

The returned information is in the format of:

projectID:*:gid: list of user IDs

gid is the group identifier number unique for the project account projectID. 

For example, the command OSCgetent group PZS0712 lists all users on the project account PZS0712 as below:

$ OSCgetent group PZS0712
PZS0712:*:5513:amarcum,guilfoos,hhamblin,kcahill,xwang

Multiple groups can also be queried at once.

For Example, the command OSCgetent group PZS0712 PZS0726 lists all users on both PZS0712 and PZS0726:

PZS0712:*:5513:amarcum,guilfoos,hhamblin,kcahill,xwang
PZS0726:*:6129:amarcum,kkappel

Details on a project can also be obtained along with the user list using the OSCfinger command.

$ OSCfinger -g projectID 

This returns:

Group: projectID                                  GID: XXXX
Status: 'active/restricted/etc'                   Type: XX
Principal Investigator: 'PI email'                Admins: NA
Members: 'list of users'
Category: NA
Institution: 'affliated institution'
Description: 'short description'
---

If you don't know the project acccount, but know the username

If the project account is not known, but the username is known, use the  OSCfinger command to list all of the groups the user belongs to:

OSCfinger username

The returned information is in the format of:

Login: username                                   Name: First Last
Directory: home directory path                    Shell: /bin/bash
E-mail: user's email address
Primary Group: user's primary project
Groups: list of projects and other groups user is in
Password Changed: date password was last changed  Password Expires: date password expires
Login Disabled: TRUE/FALSE                             Password Expired: TRUE/FALSE
Current Logins:
Displays if user is currently logged in and from where/when

For example, with the username as amarcum, the command OSCfinger amarcum returns the information as below:

$ OSCfinger amarcum
Login: amarcum                                    Name: Antonio Marcum
Directory: /users/PZS0712/amarcum                 Shell: /bin/bash
E-mail: amarcum@osc.edu
Primary Group: PZS0712
Groups: sts,ruby,l2supprt,oscall,clntstf,oscstaff,clntall,PZS0712,PZS0726
Password Changed: May 12 2019 15:47 (calculated)  Password Expires: Aug 11 2019 12:05 AM
Login Disabled: FALSE                             Password Expired: FALSE
Current Logins:
On since Mar 07 2019 12:12 on pts/14 from pitzer-login01.hpc.osc.edu
----

If you don't know either the project account or user account

If the project account or username is not known, use the OSCfinger -e command with the '-e' flag to get the user account based on the user's name.

Use the following command to list all of the user accounts associated with a First and Last name:

$ OSCfinger -e 'First Last'

For example, with user's first name as Summer and last name as Wang, the command

OSCfinger -e 'Summer Wang' returns the information as below:

$ OSCfinger -e 'Summer Wang'
Login: xwang                                      Name: Summer Wang
Directory: /users/oscgen/xwang                    Shell: /bin/bash
E-mail: xwang@osc.edu
Primary Group: PZS0712
Groups: amber,abaqus,GaussC,comsol,foampro,sts,awsmdev,awesim,ruby,matlab,aasheats,mars,ansysflu,wrigley,lgfuel,l2supprt,fsl,oscall,clntstf,oscstaff,singadm,clntall,dhgremot,fsurfer,PZS0530,PCON0003,PZS0680,PMIU0149,PZS0712,PAS1448
Password Changed: Jan 08 2019 11:41               Password Expires: Jul 08 2019 12:05 AM
Login Disabled: FALSE                             Password Expired: FALSE
---

Once you know the user account username, follow the discussions in the previous section identify users on a project to get all user accounts on the project. Please contact OSC Help if you have any questions. 

Check the Status of a User

Use the OSCfinger command to check the status of a user account as below:

OSCfinger username

For example, if the username is xwang, the command  OSCfinger xwang will return:

$ OSCfinger xwang
Login: xwang                                      Name: Summer Wang
Directory: /users/oscgen/xwang                    Shell: /bin/bash
E-mail: xwang@osc.edu
Primary Group: PZS0712
Groups: amber,abaqus,GaussC,comsol,foampro,sts,awsmdev,awesim,ruby,matlab,aasheats,mars,ansysflu,wrigley,lgfuel,l2supprt,fsl,oscall,clntstf,oscstaff,singadm,clntall,dhgremot,fsurfer,PZS0530,PCON0003,PZS0680,PMIU0149,PZS0712,PAS1448
Password Changed: Jan 08 2019 11:41               Password Expires: Jul 08 2019 12:05 AM
Login Disabled: FALSE                             Password Expired: FALSE
---

• The home directory of xwang is   Directory: /users/oscgen/xwang
• The shell of xwang is bash (Shell: /bin/bash). If the information is Shell:/access/denied, it means this user account has been either archived or restricted. Please contact OSC Help if you'd like to reactivate this user account.
• xwang@osc.edu is the associated email with the user account xwang; that is, all OSC emails related to the account xwang will be sent to xwang@osc.edu (Mail forwarded to xwang@osc.edu). Please contact OSC Help if the email address associated with this user account has been changed to ensure important notifications/messages/reminders from OSC may be received in a timely manner.

Check the Usage and Quota of a User's Home Directory/Project Space

All users see their file system usage statistics when logging in, like so:

As of 2018-01-25T04:02:23.749853 userid userID on /users/projectID used XGB of quota 500GB and Y files of quota 1000000 files

The information is from the file /users/reporting/storage/quota/*_quota.txt , which is updated twice a day. Some users may see multiple lines associated with a username, as well as information on project space usage and quota, if there is one. The usage and quota of the home directory of a username is provided by the line including the file server your home directory is on (for more information, please visit Home Directories), while others (generated due to file copy) can be safely ignored. 

You can check any user's home directory or a project's project space usage and quota by running:

grep -h 'userID' OR 'projectID' /users/reporting/storage/quota/*_quota.txt

Here is an example of project PZS0712:

$ grep -h PZS0712 /users/reporting/storage/quota/*_quota.txt
As of 2019-03-07T13:55:01.000000 project/group PZS0712 on /fs/project used 262 GiB of quota 2048 GiB and 166987 files of quota 200000 files
As of 2019-03-07T13:55:01.000000 userid xwang on /fs/project/PZS0712 used 0 GiB of quota 0 GiB and 21 files of quota 0 files
As of 2019-03-07T13:55:01.000000 userid dheisterberg on /fs/project/PZS0712 used 262 GiB of quota 0 GiB and 166961 files of quota 0 files
As of 2019-03-07T13:55:01.000000 userid amarcum on /fs/project/PZS0712 used 0 GiB of quota 0 GiB and 2 files of quota 0 files
As of 2019-03-07T13:55:01.000000 userid root on /fs/project/PZS0712 used 0 GiB of quota 0 GiB and 2 files of quota 0 files
As of 2019-03-07T13:55:01.000000 userid guilfoos on /fs/project/PZS0712 used 0 GiB of quota 0 GiB and 1 files of quota 0 files
As of 2019-03-07T13:51:23.000000 userid amarcum on /users/PZS0712 used 399.86 MiB of quota 500 GiB and 8710 files of quota 1000000 files

Here is an example for username amarcum:

$ grep -h amarcum /users/reporting/storage/quota/*_quota.txt
As of 2019-03-07T13:55:01.000000 userid amarcum on /fs/project/PZS0712 used 0 GiB of quota 0 GiB and 2 files of quota 0 files
As of 2019-03-07T13:56:39.000000 userid amarcum on /users/PZS0645 used 4.00 KiB of quota 500 GiB and 1 files of quota 1000000 files
As of 2019-03-07T13:56:39.000000 userid amarcum on /users/PZS0712 used 399.86 MiB of quota 500 GiB and 8710 files of quota 1000000 files

Check the RU Usage for Projects and Users

The OSCusage commnad can provide detailed information about computational usage for a given project and user.

See the OSCusage command page for details.

Free Academic Ansys Desktop License (2020 R1 version)

Ansys, Inc. is generously providing a temporary free trial license for home use as a COVID-19 response for OSC academic users. You can download the installer and license to your desktop following the instructions below. The license will expire on Dec 31, 2020. (The original expiry date was May 31, 2020, but it has been extended to Dec 31, 2020)

This license is for OSC academic users only. 

Notice: This temporary license key is not to be distributed, downloaded or made accessible to any other individual. It may only be used by you, the recipient, solely in connection with your university class or authorized university research. This temporary license key is not to be distributed, downloaded or made accessible to anyone in China or Russia.​

Download

Please click here (login required) to download the temporary Ansys license file and Ansys 2020R1 installer for Windows and Linux. You will need an OSC account and password to access. You don't need to download all the binary files. You can select only files that are related to your study. 

If you downloaded and used Ansys with the license expiring on May 31 or Sep 30, 2020, you can download the new license file from here (login required). Just replacing the license file would be enough to use Ansys until Dec 31, 2020.   

You can find supported platform information from here.

Installation

Windows:

1. Extract downloaded files
2. Install software by running setup.exe from the extracted directory.
   • Select Install Ansys Products
   • Follow the instructions
   • For Enter License Server Specification, click on Skip this step and configure later, follow instructions in step 3.
3. Setting the temporary license file (academic.lic)
   1. Locate the downloaded license file. By default, your browser will write the file to your Downloads directory. The directions assume you are leaving it in the Downloads directory (%USERPROFILE%\Downloads\academic.lic).

   2. Should you choose to save the file to a different directory, please make note of that new file location so you can replace the text in quotes in the setx commands below with the new file location

   3. Open a command prompt: press the Windows key + R at the same time, enter cmd, and press OK.
   4. In the command prompt, run the following commands to set appropriate environment variables to point to the license file: 
      • setx ANSYSLMD_LICENSE_FILE "%USERPROFILE%\Downloads\academic.lic"
      • setx DYNARDO_LICENSE_FILE "%USERPROFILE%\Downloads\academic.lic"
      • setx ESTERELD_LICENSE_FILE "%USERPROFILE%\Downloads\academic.lic"

Linux:

1. Extract downloaded files. For example,

   tar xvf STRUCTURES_2020R1_LINX64.tar
   unzip ELECTRONICS_2020R1_LINX64.zip
2. Install software by running ./INSTALL from the extracted directory.
   • Select Install Ansys Products
   • Follow the instructions
   • For Enter License Server Specification, click on Skip this step and configure later, follow instructions in step 3.
3. Setting the temporary license file (academic.lic)
   1. Locate the downloaded license file. 
   2. Set up the environment variables for the license file: 
      1. For csh:

         setenv ANSYSLMD_LICENSE_FILE "{path to}/academic.lic"
         setenv DYNARDO_LICENSE_FILE "{path to}/academic.lic"
         setenv ESTERELD_LICENSE_FILE "{path to}/academic.lic"

      2. For bash:

         export ANSYSLMD_LICENSE_FILE="{path to}/academic.lic"
         export DYNARDO_LICENSE_FILE="{path to}/academic.lic"
         export ESTERELD_LICENSE_FILE="{path to}/academic.lic"

Support

If you have any questions, please send an email to OSC Help.

If you need to use a MATLAB toolbox that is not provided through our installations. You can follow these instructions, and if you have any difficulties you can contact OSC Help for assistance.

Gather your materials

First, we recommend making a new directory within your home directory in order to keep everything organized. You can use the unix command to make a new directory: "mkdir"

Now you can download the toolbox either to your desktop, and then upload it to OSC, or directly download it using the "wget" utility (if you know the URL for the file).

Now you can extract the downloaded file.

Adding the path

There are two methods on how to add the MATLAB toolbox path.

Method 1: Load up the Matlab GUI and click on "Set Path" and "Add folder"

Method 2: Use the "addpath" fuction in your script. More information on the function can be found here: https://www.mathworks.com/help/matlab/ref/addpath.html

Running the toolbox

Please refer to the instructions given alongside the toolbox. They should contain instructions on how to run the toolbox.

While we provide a number of Perl modules, you may need a module we do not provide. If it is a commonly used module, or one that is particularly difficult to compile, you can contact OSC Help for assistance, but we have provided an example below showing how to build and install your own Perl modules. Note, these instructions use "bash" shell syntax; this is our default shell, but if you are using something else (csh, tcsh, etc), some of the syntax may be different.

CPAN Minus

CPAN, the Comprehensive Perl Achive Network, is the primary source for publishing and fetching the latest modules and libraries for the Perl programming language. The default method for installing Perl modules using the "CPAN Shell", provides users with a great deal of power and flexibility but at the cost of a complex configuration and inelegant default setup.

Setting Up CPAN Minus

To use CPAN Minus, we must first load it, if it hasn't already been loaded. Note that this is not necessary if you loaded a version of Perl with the module load command.

module load cpanminus

Next, in order to use cpanminus, you will need to run the following command only ONCE:

perl -I $CPANMINUS_INC -Mlocal::lib

Using CPAN Minus

In most cases, using CPAN Minus to install modules is as simple as issuing a command in the following form:

cpanm [Module::Name]

For example, below are three examples of installing perl modules:

cpanm Math::CDF
cpanm SET::IntervalTree
cpanm DB_File

Testing Perl Modules

To test a perl module import, here are some examples below:

perl -e "require Math::CDF"
perl -e "require Set::IntervallTree"
perl -e "require DB_File"

The modules are installed correctly if no output is printed.

What Local Modules are Installed in my Account?

To show the local modules you have installed in your user account:

perldoc perllocal

Reseting Module Collection

If you should ever want to start over with your perl module collection, delete the following folders:

rm -r ~/perl5 
rm -r ~/.cpanm

Sometimes the best way to get access to a piece of software on the HPC systems is to install it yourself as a "local install". This document will walk you through the OSC-recommended procedure for maintaining local installs in your home directory or project space. This majority of this document describes the process of "manually" building and installing your software. We also show a partially automated approach through the use of a bash script in the Install Script section below.

Getting Started

Before installing your software, you should first prepare a place for it to live. We recommend the following directory structure, which you should create in the top-level of your home directory:

    local
    |-- src
    |-- share
        `-- lmodfiles

This structure is how OSC organizes the software we provide. Each directory serves a specific purpose:

• local - Gathers all the files related to your local installs into one directory, rather than cluttering your home directory. Applications will be installed into this directory with the format "appname/version". This allows you to easily store multiple versions of a particular software install if necessary.
• local/src - Stores the installers -- generally source directories -- for your software. Also, stores the compressed archives ("tarballs") of your installers; useful if you want to reinstall later using different build options.
• local/share/lmodfiles - The standard place to store module files, which will allow you to dynamically add or remove locally installed applications from your environment.

You can create this structure with one command.

NOTE: Ensure $HOME is the full path of your home directory. You can identify this from the command line with the command echo $HOME.

After navigating to where you want to create the directory structure, run:

    mkdir -p $HOME/local/src $HOME/local/share/lmodfiles

Installing Software

Now that you have your directory structure created, you can install your software. For demonstration purposes, we will install a local copy of Git.

First, we need to get the source code onto the HPC filesystem. The easiest thing to do is find a download link, copy it, and use the wget tool to download it on the HPC. We'll download this into $HOME/local/src:

    cd $HOME/local/src
    wget https://github.com/git/git/archive/v2.9.0.tar.gz

Now extract the tar file:

    tar zxvf v2.9.0.tar.gz

Next, we'll go into the source directory and build the program. Consult your application's documentation to determine how to install into $HOME/local/"software_name"/"version". Replace "software_name" with the software's name and "version" with the version you are installing, as demonstrated below. In this case, we'll use the configure tool's --prefix option to specify the install location.

You'll also want to specify a few variables to help make your application more compatible with our systems. We recommend specifying that you wish to use the Intel compilers and that you want to link the Intel libraries statically. This will prevent you from having to have the Intel module loaded in order to use your program. To accomplish this, add CC=icc CFLAGS=-static-intel to the end of your invocation of configure. If your application does not use configure, you can generally still set these variables somewhere in its Makefile or build script.

Then, we can build Git using the following commands:

    cd git-2.9.0
    autoconf # this creates the configure file
    ./configure --prefix=$HOME/local/git/2.9.0 CC=icc CFLAGS=-static-intel
    make && make install

Your application should now be fully installed. However, before you can use it you will need to add the installation's directories to your path. To do this, you will need to create a module.

Creating a Module

Modules allow you to dynamically alter your environment to define environment variables and bring executables, libraries, and other features into your shell's search paths.

We will be using the filename 2.9.0.lua ("version".lua). A simple Lua module for our Git installation would be:

-- Local Variables
local name = "git"
local version = "2.9.0"

-- Locate Home Directory
local homedir = os.getenv("HOME")
local root = pathJoin(homedir, "local", name, version)

-- Set Basic Paths
prepend_path("PATH", pathJoin(root, "bin"))
prepend_path("LD_LIBRARY_PATH", root .. "/lib")
prepend_path("LIBRARY_PATH", root .. "/lib")
prepend_path("INCLUDE", root .. "/include")
prepend_path("CPATH", root .. "/include")
prepend_path("PKG_CONFIG_PATH", root .. "/lib/pkgconfig")
prepend_path("MANPATH", root .. "/share/man")

NOTE: For future module files, copy our sample modulefile from ~support/doc/modules/sample_module.lua. This module file follows the recommended design patterns laid out above and includes samples of many common module operations

Initializing Modules

Any module file you create should be saved into your local lmodfiles directory ($HOME/local/share/lmodfiles). To prepare for future software installations, create a subdirectory within lmodfiles named after your software and add one module file to that directory for each version of the software installed.

In the case of our Git example, you should create the directory $HOME/local/share/lmodfiles/git and create a module file within that directory named 2.9.0.lua.

To make this module usable, you need to tell lmod where to look for it. You can do this by issuing the command module use $HOME/local/share/lmodfiles in our example. You can see this change by performing module avail. This will allow you to load your software using either module load git or module load git/2.9.0. 

If you install another version later on (lets say version 2.9.1) and want to created a module file for it, you need to make sure you call it 2.9.1.lua. When loading Git, lmod will automatically load the newer version. If you need to go back to an older version, you can do so by specifying the version you want: module load git/2.9.0.

To make sure you have the correct module file loaded, type which git which should give the output "~/local/git/2.9.0/bin/git" (NOTE: ~ is equivalent to $HOME). 

To make sure the software was installed correctly and that the module is working, type git --version which shuold give the output "git version 2.9.0".

Automating With Install Script

Simplified versions of the scripts used to manage the central OSC software installations are provided at ~support/share/install-script. The idea is that you provide the minimal commands needed to obtain, compile, and install the software (usually some variation on wget, tar, ./configure, make, and make install) in a script, which then sources an OSC-maintained template that provides all of the "boilerplate" commands to create and manage a directory structure similar to that outlined in the Getting Started section above. You can copy an example install script from ~support/share/install-script/install-osc_sample.sh and follow the notes in that script, as well as in ~support/share/install-script/README.md, to modify it to install software of your choosing.

Further Reading

For more information about modules, be sure to read the webpage indicated at the end of module help. If you have any questions about modules or local installations, feel free to contact the OSC Help Desk and oschelp@osc.edu.

An ACL (access control list) is a list of permissions associated with a file or directory. These permissions allow you to restrict access to a certain file or directory by user or group. 

OSC supports NFSv4 ACL on our home directory and POSIX ACL on our project and scratch file systems. Please see the how to use NFSv4 ACL for home directory ACL management and how to use POSIX ACL for managing ACLs in project and scratch file systems. 

This HOWTO will demonstrate how to lower ones' disk space usage. The following procedures can be applied to all of OSC's file systems.

We recommend users regularly check their data usage and clean out old data that is no longer needed.

Users who need assistance lowering their data usage can contact OSC Help.

Preventing Excessive Data Usage Before It Starts

Users should ensure that their jobs are written in such a way that temporary data is not saved to permanent file systems, such as the project space file system or their home directory.

If your job copies data from the scratch file system or its node's local disk ($TMPDIR) back to a permanent file system, such as the project space file system or a home directory ( /users/PXX####/xxx####/), you should ensure you are only copying the files you will need later.

Identifying Old and Large Data

The following commands will help you identify old data using the find command.

Find all files in a directory that have not been accessed in the past 100 days:

This command will recursively search the users home directory and give a detailed listing of all files not accessed in the past 100 days.

The last access time atime is updated when a file is opened by any operation, including grep, cat, head, sort, etc.

find ~ -atime +100 -exec ls -l {} \;

• To search a different directory replace ~ with the path you wish to search. A period . can be used to search the current directory.
• To view files not accessed over a different time span, replace 100 with your desired number of days.
• To view the total size in bytes of all the files found by find, you can add | awk '{s+=$5} END {print "Total SIZE (bytes): " s}' to the end of the command:

find ~ -atime +100 -exec ls -l {} \;| awk '{s+=$5} END {print "Total SIZE (bytes): " s}'

Find all files in a directory that have not been modified in the past 100 days:

This command will recursively search the users home directory and give a detailed listing of all files not modified in the past 100 days.

The last modified time mtime is updated when a file's contents are updated or saved. Viewing a file will not update the last modified time.

find ~ -mtime +100 -exec ls -l {} \; 

• To search a different directory replace ~ with the path you wish to search. A period . can be used to search the current directory.
• To view files not modified over a different time span, replace 100 with your desired number of days.
• To view the total size in bytes of all the files found by find, you can add | awk '{s+=$5} END {print "Total SIZE (bytes): " s}' to the end of the command:

find ~ -mtime +100 -exec ls -l {} \;| awk '{s+=$5} END {print "Total SIZE (bytes): " s}'

List files larger than a specified size:

Adding the -size <size> option and argument to the find command allows you to only view files larger than a certain size. This option and argument can be added to any other find command.

For example, to view all files in a users home directory that are larger than 1GB:

find ~ -size 1G -exec ls -l {} \;

Deleting Identified Data

If you no longer need the old data, you can delete it using the rm command.

If you need to delete a whole directory tree (a directory and all of its subcontents, including other directories), you can use the rm -R command.

For example, the following command will delete the data directory in a users home directory:

rm -R ~/data

If you would like to be prompted for confirmation before deleting every file, use the -i option.

rm -Ri ~/data 

Enter y or n when prompted. Simply pressing the enter button will default to n.

Deleting files found by find

The rm command can be combined with any find command to delete the files found. The syntax for doing so is:

find <location> <other find options> -exec rm -i {} \;

Where <other find options> can include one or more of the options -atime <time>, -mtime <time>, and -size <size>.

The following command would find all files in the ~/data directory 1G or larger that have not been accessed in the past 100 days, and then prompt for confirmation to delete each file:

find ~/data -atime +100 -size 1G -exec rm -i {} \;

If you are absolutely sure the files identified by find are okay to delete you can remove the -i option to rm and you will not be prompted. Extreme caution should be used when doing so!

Archiving Data

If you still need the data but do not plan on needing the data in the immediate future, contact OSC Help to discuss moving the data to an archive file system. Requests for data to be moved to the archive file system should be larger than 1TB.

Compressing

If you need the data but do not access the data frequently, you should compress the data using tar or gzip.

Moving Data to a Local File System

If you have the space available locally you can transfer your data there using sftp or Globus.

Globus is recommended for large transfers.

The OnDemand File application should not be used for transfers larger than 1GB.

This page outlines a way a professor can set up a file submission system at OSC for his/her classroom project.

Usage for Professor

After connecting to OSC system, professor runs submit_prepare as

$ /users/PZS0645/support/bin/submit_prepare

Follow the instruction and provided the needed information (name of the assignment, TA username if appropriate, a size limit if not the default 1000MB per student, and whether or not you want the email notification of a submit). It will create a designated directory where students submit their assignments, as well as generate submit for students used to submit homework to OSC, both of which are located in the directory specified by the professor.

Note:

The PI can also enforce the deadline by simply changing the permission of the submission directory or renaming the submission directory at the deadline.

(Only works on Owens): One way is to use at command following the steps below:

• Use at command to specify the deadline:

at [TIME]

where TIME is formatted HH:MM AM/PM MM/DD/YY. For example:

at 2:30 PM 08/21/2017

• After running this command, run:

$ chmod 700 [DIRECTORY]

where DIRECTORY is the assignment folder to be closed off.

• Enter [ctrl+D] to submit this command.

The permission of DIRECTORY will be changed to 700 at 2:30PM, August 21, 2018. After that, the student will get an error message when he/she tries to submit an assignment to this directory.

Usage for Students

A student should create one directory which includes all the files he/she wants to submit before running this script to submit his/her assignment. Also, the previous submission of the same assignment from the student will be replaced by the new submission. 

To submit the assignment, the student runs submit after connecting to OSC system as

$ /path/to/directory/from/professor/submit

Follow the instructions. It will allow students to submit an assignment to the designated directory specified by the professor and send a confirmation email, or return an error message.

Often users want to submit a large number of jobs all at once, with each using different parameters for each job. These parameters could be anything, including the path of a data file or different input values for a program. This how-to will show you how you can do this using a simple python script, a CSV file, and a template script. You will need to adapt this advice for your own situation.

Consider the following batch script:

#!/bin/bash
#SBATCH --ntasks-per-node=2
#SBATCH --time=1:00:00
#SBATCH --job-name=week42_data8

# Copy input data to the nodes fast local disk
cp ~/week42/data/source1/data8.in $TMPDIR

cd $TMPDIR

# Run the analysis
full_analysis data8.in data8.out

# Copy results to proper folder
cp  data8.out ~/week42/results

Let's say you need to submit 100 of these jobs on a weekly basis. Each job uses a different data file as input. You recieve data from two different sources, and so your data is located within two different folders. All of the jobs from one week need to store their results in a single weekly results folder. The output file name is based upon the input file name.

Creating a Template Script

As you can see, this job follows a general template.  There are three main parameters that change in each job:

1. The week 
   • Used as part of the job name
   • Used to find the proper data file to copy to the nodes local disk
   • Used to copy the results to the correct folder
2. The data source
   • Used to find the proper data file to copy to the nodes local disk
3. The data file's name
   • Used as part of the job name
   • Used to find the proper data file to copy to the nodes local disk
   • Used to specify both the input and output file to the program full_analysis
   • Used to copy the results to the correct folder

If we replace these parameters with variables, prefixed by the dollar sign $and surrounded by curly braces { }, we get the following template script:

#!/bin/bash
#SBATCH --ntasks-per-node=2
#SBATCH --time=1:00:00

# Copy input data to the nodes fast local disk 
cp ~/${WEEK}/data/${SOURCE}/${DATA}.in $TMPDIR
cd $TMPDIR

# Run the analysis 
full_analysis ${DATA}.in ${DATA}.out

# Copy results to proper folder
cp  ${DATA}.out ~/${WEEK}/results

Automating Job Submission

We can now use the sbatch --exportoption to pass parameters to our template script. The format for passing parameters is:

sbatch --job-name=name --export=var_name=value[,var_name=value...]

Submitting 100 jobs using the sbatch --export option manually does not make our task much easier than modifying and submitting each job one by one. To complete our task we need to automate the submission of our jobs. We will do this by using a python script that submits our jobs using parameters it reads from a CSV file.  

Note that python was chosen for this task for its general ease of use and understandability -- if you feel more comfortable using another scripting language feel free to interpret/translate this python code for your own use.

Here is the script that submits the jobs using the parameters:

#!/usr/bin/python3
## file: submit_jobs.py
import argparse
import os
import csv, subprocess

def create_dirs(n_dir):
    if not os.path.isdir(n_dir + "job_logs"):
        os.mkdir(n_dir + "job_logs/")
    if not os.path.isdir(n_dir + "results"):
        os.mkdir(n_dir + "results/")
            

def main():
    parser = argparse.ArgumentParser(
        description="Automatically submit jobs using a csv file")

    parser.add_argument('jobscript',help="job script to use")
    parser.add_argument('parameters',help="csv parameter file to use")
    parser.add_argument('account',help="project account to charge jobs to")
    parser.add_argument('-t','--test',action='store_false',help="test script without submitting jobs")
    args = parser.parse_args()

    with open(args.parameters,mode='r',newline='',encoding='utf-8-sig') as csvfile:
        reader = csv.reader(csvfile)
        created_dirs = []
        for job in reader:
            n_dir = os.getenv("HOME") + "/{0}/".format(*job)
            if dir not in created_dirs:
                create_dirs(n_dir)
                created_dirs.append(n_dir)
            
            submit_command = (
                "sbatch " +
                "-A {} ".format(args.account) +
                "-o ~/{0}/job_logs/{0}_{1}_{2}.job_log ".format(*job) +
                "--job-name={0}_{1}_{2} "
                "--export=WEEK={0},SOURCE={1},DATA={2} ".format(*job) + args.jobscript)
            if not args.test:
                print(submit_command)
            else:
                exit_status = subprocess.call(submit_command,shell=True)
                # Check to make sure the job submitted
                if exit_status is 1:
                    print("Job {0} failed to submit".format(submit_command))

    print("Done submitting jobs")

if __name__ == "__main__":
    main()

This script will open the CSV file specified by the variable parameter_file_full_path and step through the file line by line, submitting a job for each line using the lines values. If the submit command returns a non-zero exit code, usually indicating it was not submitted, we will print this out to the display. The jobs will be submitted using the general format:

sbatch -A <project-account> -o ~/x/job_logs/x_y_z.job_log --job-name=x_y_z --export=WEEK=x,SOURCE=y,DATA=z job.sh

Where x, y and z are determined by the values in the CSV parameter file. Below we relate x to week, y to source and z to data.

Creating a CSV File

We now need to create a CSV file with parameters for each job. This can be done with a regular text editor or using a spreadsheet editor such as Excel. By default you should use commas as your delimiter.  

Here is our CSV file with parameters:

week42,source1,data1
week42,source1,data2
week42,source1,data3
...
week42,source2,data98
week42,source2,data99
week42,source2,data100

The submit script would read in the first row of this CSV file and form and execute the command:

sbatch -A <project-account> -o ~/week42/job_logs/week42_source1_data1.job_log --job-name=week42_source1_data1 --export=WEEK=week42,SOURCE=source1,DATA=data1 job.sh

Submitting Jobs

Once all the above is done, all you need to do to submit your jobs is to make sure the CSV file is populated with the proper parameters and run the automatic submission script with the right flags.

Try using submit_jobs.py --help for an explanation:

$ python3 submit_jobs.py --help
usage: submit_jobs.py [-h] [-t] jobscript parameters account

Automatically submit jobs using a csv file

positional arguments:
  jobscript   job script to use
  parameters  csv parameter file to use
  account     project account to charge jobs to

optional arguments:
  -h, --help  show this help message and exit
  -t, --test  test script without submitting jobs

Table of Contents

Introduction

Setup

Debugging

Hardware

Performance Measurement

Timing

Profiling

Help From the Compiler

Compiler Optimization Reports

Memory Optimizations

Vectorization/Streaming

OpenMP

MPI

GPU Accelerated Computing

Summary

Introduction

This tutorial goes over techniques to tune the performance of your application. Keep in mind that correctness of results, code readability/maintainability, and portability to future systems are more important than performance. Some factors that can affect performance are:

• Effective use of processor features for a high degree of internal concurrency in a single core
• Mmory access patterns (memory access is slow compared to computation)
• Use of an appropriate file system for file I/O
• Scalability of algorithms
• Compiler optimizations
• Explicit parallelism

We will be using code based on the HPCCD miniapp from Mantevo. It performs the Conjugate Gradient (CG) on a 3D chimney domain. CG is an iterative algorithm to numerically approximate the solution to a system of linear equations.

Run code with:

srun -n <numprocs> ./test_HPCCG nx ny nz

where nx, ny, nz are the number of nodes in the x, y, and z dimension on each processor.

Setup

First start an interactive Pitzer Desktop session with OnDemand.

You need to load intel 19.0.5 and mvapich2 2.3.3:

module load intel/19.0.5 mvapich2/2.3.3

Then clone the repository:

git clone https://code.osu.edu/khuvis.1/performance2021_handson.git

Debugging

Debuggers let you execute your program one line at a time, inspect variable values, stop your programming at a particular line, and open a core file after the program crashes.

For debugging, use the -g flag and remove optimzation or set to -O0. For example:

icc -g -o mycode.c

gcc -g -O0 -o mycode mycode.c

To see compiler warnings and diagnostic options:

icc -help diag

man gcc

ARM DDT

ARM DDT is a commercial debugger produced by ARM. It can be loaded on all OSC clusters:

module load arm-ddt

To run a non-MPI program from the command line:

ddt --offline --no-mpi ./mycode [args]

To run an MPI program from the command line:

ddt --offline -np num.procs ./mycode [args]

Hands On

Compile and run the code:

make
srun -n 2 ./test_HPCCG 150 150 150

You should have received the following error message at the end of the program output:

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 308893 RUNNING AT p0200
=   EXIT CODE: 11
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
YOUR APPPLICATIN TERMINATED WITH EXIT STRING: Segmentation fault (signal 11)
This typically referes to a problem with your application.
Please see tthe FAQ page for debugging suggestions

Set compiler flags -O0 -g  to CPP_OPT_FLAGS in Makefile. Then recompile and run with ARM DDT:

make clean; make
module load arm-ddt
ddt -np 2 ./test_HPCCG 150 150 150

Solution

for(size_t i=0; i<=children.size(); i++) {

for(size_t i=0; i<children.size(); i++) {

Hardware

On Pitzer, there are 40 cores per node (20 cores per socket and 2 sockets per node). There is support for AVX512, vector length 8 double or 16 single precision values and fused multiply-add. (There is hardware support for 4 thread per core, but it is currently not enabled on OSC systems.)

There are three cache levels on Pitzer, and the statistics are shown in the table below:

Never do heavy I/O in your home directory. Home directories are for long-term storage, not scratch files.

One option for I/O intensive jobs is to use the local disk on a compute node. Stage files to and from your home directory into $TMPDIR using the pbsdcp command (e.g. pbsdcp file1 file2 $TMPDIR), and execute the program in $TMPDIR.

Another option is to use the scratch file system ($PFSDIR). This is faster than other file systems, good for parallel jobs, and may be faster than local disk.

For more information about OSC's file system, click here.

For example batch scripts showing the use of $TMPDIR and $PFSDIR, click here.

For more information about Pitzer, click here.

Performance Measurement

FLOPS stands for "floating point operations per second." Pitzer has a theoretical maximum of 720 teraflops. With the LINPACK benchmark of solving a dense system of linear equations, 543 teraflops. With the STREAM benchmark, which measures sustainable memory bandwidth and the corresponding computation rate for vector kernels, copy: 299095.01 MB/s, scale: 298741.01 MB/s, add: 331719.18 MB/s, and traid: 331712.19 MB/s. Application performance is typically much less than peak/sustained performance since applications usually do not take full advantage of all hardware features.

Timing

You can time a program using the /usr/bin/time command. It gives results for user time (CPU time spent running your program), system time (CPU time spent by your program in system calls), and elapsed time (wallclock). It also shows % CPU, which is (user + system) / elapsed, as well as memory, pagefault, swap, and I/O statistics.

/usr/bin/time j3
5415.03user 13.75system 1:30:29elapsed 99%CPU \
(0avgtext+0avgdata 0maxresident)k \
0inputs+0outputs (255major+509333minor)pagefaults 0 swaps

You can also time portions of your code:

Profiling

A profiler can show you whether code is compute-bound, memory-bound, or communication bound. Also, it shows how well the code uses available resources and how much time is spent in different parts of your code. OSC has the following profiling tools: ARM Performance Reports, ARM MAP, Intel VTune, Intel Trace Analyzer and Collector (ITAC), Intel Advisor, TAU Commander, and HPCToolkit.

For profiling, use the -g flag and specify the same optimization level that you normally would normally use with -On. For example:

icc -g -O3 -o mycode mycode.c

Look for

• Hot spots (where most of the time is spent)
• Excessive number of calls to short functions (use inlining!)
• Memory usage (swapping and thrashing are not allowed at OSC)
• % CPU (low CPU utilization may mean excessive I/O delays).

ARM Performance Reports

ARM PR works on precompiled binaries, so the -g flag is not needed. It gives a summary of your code's performance that you can view with a browser.

For a non-MPI program:

module load arm-pr
perf-report --no-mpi ./mycode [args]

For an MPI program:

module load arm-pr
perf-report --np num_procs ./mycode [args]

ARM MAP

Interpreting this profile requires some expertise. It gives details about your code's performance. You can view and explore the resulting profile using an ARM client.

For a non-MPI program:

module load arm-map
map --no-mpi ./mycode [args]

For an MPI program:

module load arm-pr
map --np num_procs ./mycode [args]

For more information about ARM Tools, view OSC resources or visit ARM's website.

Intel Trace Analyzer and Collector (ITAC)

ITAC is a graphical tool for profiling MPI code (Intel MPI).

To use:

module load intelmpi # then compile (-g) code
mpiexec -trace ./mycode

View and explore the results using a GUI with traceanalyzer:

traceanalyzer <mycode>.stf

Help From the Compiler

HPC software is traditionally written in Fortran or C/C++. OSC supports several compiler families. Intel (icc, icpc, ifort) usually gives fastest code on Intel architecture). Portland Group (PGI - pgcc, pgc++, pgf90) is good for GPU programming, OpenACC. GNU (gcc, g++, gfortran) is open source and universally available.

Compiler options are easy to use and let you control aspects of the optimization. Keep in mind that different compilers have different values for options.  For all compilers, any highly optimized builds, such as those employing the options herein, should be thoroughly validated for correctness.

Some examples of optimization include:

• Function inlining (eliminating function calls)
• Interprocedural optimization/analysis (ipo/ipa)
• Loop transformations (unrolling, interchange, splitting, tiling)
• Vectorization (operate on arrays of operands)
• Automatic parallization of loops (very conservative multithreading)

Compiler flags to try first are:

• General optimization flags (-O2, -O3, -fast)
• Fast math
• Interprocedural optimization/analysis

Faster operations are sometimes less accurate. For Intel compilers, fast math is default with -O2 and -O3. If you have a problem, use -fp-model precise. For GNU compilers, precise math is default with -O2 and -O3. If you want faster performance, use -ffast-math.

Inlining is replacing a subroutine or function call with the actual body of the subprogram. It eliminates overhead of calling the subprogram and allows for more loop optimizations. Inlining for one source file is typically automatic with -O2 and -O3.

Optimization Compiler Options

Options for Intel compilers are shown below. Don't use -fast for MPI programs with Intel compilers. Use the same compiler command to link for -ipo with separate compilation. Many other optimization options can be found in the man pages. The recommended options are -O3 -xHost. An example is ifort -O3 program.f90.

Options for PGI compilers are shown below. Use the same compiler command to link for -Mipa with separate compilation. Many other optimization options can be found in the man pages. The recommended option is -fast. An example is pgf90 -fast program.f90.

Options for GNU compilers are shown below. Use the same compiler command to link for -Mipa with separate compilation. Many other optimization options can be found in the man pages. The recommended options are -O3 -ffast-math. An example is gfortran -O3 program.f90.

Hands On

Compile and run with different compiler options:

time srun -n 2 ./test_HPCCG 150 150 150

Using the optimal compiler flags, get an overview of the bottlenecks in the code with the ARM performance report:

module load arm-pr
perf-report -np 2 ./test_HPCCG 150 150 150

Solution

Compiler Optimization Reports

Compiler optimization reports let you understand how well the compiler is doing at optimizing your code and what parts of your code need work. They are generated at compile time and describe what optimizations were applied at various points in the source code. The report may tell you why optimizations could not be performed.

For Intel compilers, -qopt-report and outputs to a file.

For Portland Group compilers, -Minfo and outputs to stderr.

For GNU compilers, -fopt-info and ouputs to stderr by default.

A sample output is:

LOOP BEGIN at laplace-good.f(10,7)
   remark #15542: loop was not vectorized: inner loop was already vectorized

   LOOP BEGIN at laplace-good.f(11,10)
   <Peeled loop for vectorization>
   LOOP END

   LOOP BEGIN at laplace-good.f(11,10)
      remark #15300: LOOP WAS VECTORIZED
   LOOP END

   LOOP BEGIN at laplace-good.f(11,10)
   <Remainder loop for vectorization>
      remark #15301: REMAINDER LOOP WAS VECTORIZED
   LOOP END

   LOOP BEGIN at laplace-good.f(11,10)
   <Remainder loop for vectorization>
   LOOP END
LOOP END

Hands On

Add the compiler flag -qopt-report=5 and recompile to view an optimization report.

Vectorization/Streaming

Code is structured to operate on arrays of operands. Vector instructions are built into the processor. On Pitzer, the vector length is 16 single or 8 double precision. The following is a vectorizable loop:

do i = 1,N
  a(i) = b(i) + x(1) * c(i)
end do

Some things that can inhibit vectorization are:

• Loops being in the wrong order (usually fixed by compiler)
• Loops over derived types
• Function calls (can sometimes be fixed by inlining)
• Too many conditionals
• Indexed array accesses

Hands On

Use ARM MAP to identify the most expensive parts of the code.

module load arm-map
map -np 2 ./test_HPCCG 150 150 150

Check the optimization report previously generated by the compiler (with -qopt-report=5) to see if any of the loops in the regions of the code are not being vectorized. Modify the code to enable vectorization and rerun the code.

Solution

for (int j=0; j< cur_nnz; j++)
  y[i] += cur_vals[j]*x[cur_inds[j]];

for (int j=0; j< cur_nnz; j++)
  sum += cur_vals[j]*x[cur_inds[j]];
y[i] = sum;

Memory Optimizations

Memory access is often the most important factor in your code's performance. Loops that work with arrays should use a stride of one whenever possible. C and C++ are row-major (store elements consecutively by row in 2D arrays), so the first array index should be the outermost loop and the last array index should be the innermost loop. Fortran is column-major, so the reverse is true. You can get factor of 3 or 4 speedup just by using unit stride. Avoid using arrays of derived data types, structs, or classes. For example, use structs of arrays instead of arrays of structures.

Efficient cache usage is important. Cache lines are 8 words (64 bytes) of consecutive memory. The entire cache line is loaded when a piece of data is fetched.

The code below is a good example. 2 cache lines are used for every 8 loop iterations, and it is unit stride:

real*8 a(N), b(N)
do i = 1,N
  a(i) = a(i) + b(i)
end do

! 2 cache lines:
! a(1), a(2), a(3) ... a(8)
! b(1), b(2), b(3) ... b(8)

The code below is a bad example. 1 cache line is loaded for each loop iteration, and it is not unit stride:

TYPE :: node
  real*8 a, b, c, d, w, x, y, z
END TYPE node
TYPE(node) :: s(N)
do i = 1, N
  s(i)%a = s(i)%a + s(i)%b
end do

! cache line:
! a(1), b(1), c(1), d(1), w(1), x(1), y(1), z(1)

Hands On

Look again at the most expensive parts of the code using ARM MAP:

module load arm-map
map -np 2 ./test_HPCCG 150 150 150

Look for any inefficient memory access patterns. Modify the code to improve memory access patterns and rerun the code. Do these changes improve performance?

Solution

for (int ix=0; ix<nx; ix++) {
  for (int iy=0; iy<ny; iy++) {
    for (int iz=0; iz<nz; iz++) {
      int curlocalrow = iz*nx*ny+iy*nx+ix;
      int currow = start_row+iz*nx*ny+iy*nx+ix;
      int nnzrow = 0;
      (*A)->ptr_to_vals_in_row[curlocalrow] = curvalptr;
      (*A)->ptr_to_inds_in_row[curlocalrow] = curindptr;
      .
      .
      .
    }
  }
}

for (int iz=0; iz<nz; iz++) {
  for (int iy=0; iy<ny; iy++) {
    for (int ix=0; ix<nx; ix++) {
      .
      .
      .
    }
  }
}

OpenMP

OpenMP is a shared-memory, threaded parallel programming model. It is a portable standard with a set of compiler directives and a library of support functions. It is supported in compilers by Intel, Portland Group, GNU, and Cray.

The following are parallel loop execution examples in Fortran and C. The inner loop vectorizes while the outer loop executes on multiple threads:

PROGRAM omploop
INTEGER, PARAMETER :: N = 1000
INTEGER i, j
REAL, DIMENSION(N, N) :: a, b, c, x
... ! Initialize arrays
!$OMP PARALLEL DO
do j = 1, N
  do i = 1, N
    a(i, j) = b(i, j) + x(i, j) * c(i, j)
  end do
end do
!$OMP END PARALLEL DO
END PROGRAM omploop

int main() {
  int N = 1000;
  float *a, *b, *c, *x;
... // Allocate and initialize arrays
#pragma omp parallel for
  for (int i = 0; i < N; i++) {
    for (int j = 0; j < N; j++) {
      a[i*N+j] = b[i*N+j] + x[i*N+j] * c[i*N+j]
    }
  }
}

You can add an option to compile a program with OpenMP.

For Intel compilers, add the -qopenmp option. For example, ifort -qopenmp ompex.f90 -o ompex.

For GNU compilers, add the -fopenmp option. For example, gcc -fopenmp ompex.c -o ompex.

For Portland group compilers, add the -mp option. For example, pgf90 -mp ompex.f90 -o ompex.

To run an OpenMP program, requires multiple processors through SLURM (--N 1 -n 40) and set the OMP_NUM_THREADS environment variable (default is use all available cores). For the best performance, run at most one thread per core.

An example script is:

#!/bin/bash
#SBATCH -J omploop
#SBATCH -N 1
#SBATCH -n 40
#SBATCH -t 1:00

export OMP_NUM_THREADS=40
/usr/bin/time ./omploop

For more information, visit http://www.openmp.org, OpenMP Application Program Interface, and self-paced turorials. OSC will host an XSEDE OpenMP workshop on November 5, 2019.

MPI

MPI stands for message passing interface for when multiple processes run on one or more nodes. MPI has functions for point-to-point communication (e.g. MPI_Send, MPI_Recv). It also provides a number of functions for typical collective communication patterns, including MPI_Bcast (broadcasts value from root process to all other processes), MPI_Reduce (reduces values on all processes to a single value on a root process), MPI_Allreduce (reduces value on all processes to a single value and distributes the result back to all processes), MPI_Gather (gathers together values from a group of processes to a root process), and MPI_Alltoall (sends data from all processes to all processes).

A simple MPI program is:

#include <mpi.h>
#include <stdio.h>

int main(int argc, char *argv[]) {
  int rank, size;
  MPI_INIT(&argc, &argv);
  MPI_Comm_rank(MPI_COMM_WORLD, &rank);
  MPI_COMM_size(MPI_COMM_WORLD, &size);
  printf("Hello from node %d of %d\n", rank size);
  MPI_Finalize();
  return(0);
}

MPI implementations available at OSC are mvapich2, Intel MPI (only for Intel compilers), and OpenMPI.

MPI programs can be compiled with MPI compiler wrappers (mpicc, mpicxx, mpif90). They accept the same arguments as the compilers they wrap. For example, mpicc -o hello hello.c.

MPI programs must run in batch only. Debugging runs may be done with interactive batch jobs. srun automatically determines exectuion nodes from PBS:

#!/bin/bash
#SBATCH -J mpi_hello
#SBATCH -N 2
#SBATCH --ntasks-per-node=40
#SBATCH -t 1:00

cd $PBS_O_WORKDIR
srun ./hello

For more information about MPI, visit MPI Forum and MPI: A Message-Passing Interface Standard. OSC will host an XSEDE MPI workshop on September 3-4, 2019. Self-paced tutorials are available here.

Hands On

Use ITAC to get a timeline of the run of the code.

module load intelmpi
LD_PRELOAD=libVT.so \
mpiexec -trace -np 40 ./test_HPCCG 150 150 150
traceanalyzer <stf_file>

Look at the Event Timeline (under Charts). Do you see any communication patterns that could be replaced by a single MPI command?

Solution

  MPI_Barrier(MPI_COMM_WORLD);
  if(rank == 0) {
    for(int dst_rank=1; dst_rank < size; dst_rank++) {
      MPI_Send(&global_result, 1, MPI_DOUBLE, dst_rank, 1, MPI_COMM_WORLD);
    }
  }
  if(rank != 0) MPI_Recv(&global_result, 1, MPI_DOUBLE, 0, 1, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
  MPI_Barrier(MPI_COMM_WORLD);

MPI_Bcast(&global_result, 1, MPI_DOUBLE, 0, MPI_COMM_WORLD);

This article discusses memory tuning strategies for VASP.

Data Distribution

Typically the first approach for memory sensitive VASP issues is to tweak the data distribution (via NCORE or NPAR). The information at https://www.vasp.at/wiki/index.php/NPAR covers a variety of machines. OSC has fast communications via Infiniband. 

Performance and memory consumption are dependent on the simulation model.  So we recommend a series of benchmarks varying the number of nodes and NCORE. The recommended initial value for NCORE is the ppn value used in the PBS directive. Of course, if this benchmarking is intractable then one must reexamine the model. For general points see:  https://www.vasp.at/wiki/index.php/Memory_requirements and https://www.vasp.at/wiki/index.php/Not_enough_memory  And of course one should start small and incrementally improve or scale up ones model.

Rationalization

Using the key parameters with respect to memory scaling listed at the VASP memory requirements page one can rationalize VASP memory usage.  The general approach is to study working calculations and then apply that understanding to scaled up or failing calculations.  This might help one identify if a calculation is close to a node's memory limit and happens to cross over the limit for reasons that might be out of ones control, in which case one might need to switch to higher memory nodes.

Here is an example of rationalizing memory consumption.  Extract from a simulation output the key parameters:

Dimension of arrays:
k-points NKPTS = 18 k-points in BZ NKDIM = 18 number of bands NBANDS= 1344
total plane-waves NPLWV = 752640
...
dimension x,y,z NGXF= 160 NGYF= 168 NGZF= 224
support grid NGXF= 320 NGYF= 336 NGZF= 448

This yields 273 GB of memory according to
https://www.vasp.at/wiki/index.php/Memory_requirements

This estimate should be compared to actual memory reports.  See for example XDModD.  Note that most application software has an overhead in the ballpack of ten to twenty percent.  In addition, disk caching can consume significant memory.  Thus, one must adjust the memory estimate upwards.  It can then be comapred to the available memory per cluster and per cluster node type.

Miscellaneous

• In the INCAR input file NWRITE=3 is for verbose output and NWRITE=4 is for debugging output.
• OSC does not have a VASP license and our staff has limited experience with it.  So investigate alternate forms of help:  ask within your research group and  post on the VASP mailing list.
• Valgrind is a tool that can be used for many types of debugging including looking for memory corruptions and leaks. However, it slows down your code a very sizeable amount. This might not be feasible for HPC codes
• ASAN (address sanitizer) is another tool that can be used for memory debugging. It is less featureful than Valgrind, but runs much quicker, and so will likely work with your HPC code.

Address Sanitizer is a tool developed by Google detect memory access error such as use-after-free and memory leaks. It is built into GCC versions >= 4.8 and can be used on both C and C++ codes. Address Sanitizer uses runtime instrumentation to track memory allocations, which mean you must build your code with Address Sanitizer to take advantage of it's features.

There is extensive documentation on the AddressSanitizer Github Wiki.

Memory leaks can increase the total memory used by your program. It's important to properly free memory when it's no longer required. For small programs, loosing a few bytes here and there may not seem like a big deal. However, for long running programs that use gigabytes of memory, avoiding memory leaks becomes increasingly vital. If your program fails to free the memory it uses when it no longer needs it, it can run out of memory, resulting in early termination of the application. AddressSanitizer can help detect these memory leaks.

Additionally, AddressSanitizer can detect use-after-free bugs. A use-after-free bug occurs when a program tries to read or write to memory that has already been freed. This is undefined behavior and can lead to corrupted data, incorrect results, and even program crashes.

Building With Address Sanitzer

We need to use gcc to build our code, so we'll load the gcc module:

module load gnu/9.1.0

The "-fsanitize=address" flag is used to tell the compiler to add AddressSanitizer.

Additionally, due to some environmental configuration settings on OSC systems, we must also statically link against Asan. This is done using the "-static-libasan" flag.

It's helpful to compile the code with debug symbols. AddressSanitizer will print line numbers if debug symbols are present. To do this, add the "-g" flag. Additionally, the "-fno-omit-frame-pointer" flag may be helpful if you find that your stack traces do not look quite correct.

In one command, this looks like:

gcc main.c -o main -fsanitize=address -static-libasan -g

Or, splitting into separate compiling and linking stages:

gcc -c main.c -fsanitize=address -g
gcc main.o -o main -fsanitize=address -static-libasan

Notice that both the compilation and linking steps require the "-fsanitize-address" flag, but only the linking step requires "-static-libasan". If your build system is more complex, it might make sense to put these flags in CFLAGS and LDFLAGS environment variables.

And that's it!

Examples

No Leak

First, let's look at a program that has no memory leaks (noleak.c):

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

int main(int argc, const char *argv[]) {
    char *s = malloc(100);
    strcpy(s, "Hello world!");
    printf("string is: %s\n", s);
    free(s);
    return 0; 
}

To build this we run:

gcc noleak.c -o noleak -fsanitize=address -static-libasan -g

And, the output we get after running it:

string is: Hello world!

That looks correct! Since there are no memory leaks in this program, AddressSanitizer did not print anything. But, what happens if there are leaks?

Missing free

Let's look at the above program again, but this time, remove the free call (leak.c):

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

int main(int argc, const char *argv[]) {
    char *s = malloc(100);
    strcpy(s, "Hello world!");
    printf("string is: %s\n", s);
    return 0;
}

Then, to build:

gcc leak.c -o leak -fsanitize=address -static-libasan

And the output:

string is: Hello world!

=================================================================
==235624==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 100 byte(s) in 1 object(s) allocated from:
    #0 0x4eaaa8 in __interceptor_malloc ../../.././libsanitizer/asan/asan_malloc_linux.cc:144
    #1 0x5283dd in main /users/PZS0710/edanish/test/asan/leak.c:6
    #2 0x2b0c29909544 in __libc_start_main (/lib64/libc.so.6+0x22544)

SUMMARY: AddressSanitizer: 100 byte(s) leaked in 1 allocation(s).

This is a leak report from AddressSanitizer. It detected that 100 bytes were allocated, but never freed. Looking at the stack trace that it provides, we can see that the memory was allocated on line 6 in leak.c

Use After Free

Say we found the above leak in our code, and we wanted to fix it. We need to add a call to free. But, what if we add it in the wrong spot?

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

int main(int argc, const char *argv[]) {
    char *s = malloc(100);
    free(s);
    strcpy(s, "Hello world!");
    printf("string is: %s\n", s);
    return 0;
}

The above (uaf.c) is clearly wrong. Albiet a contrived example, the allocated memory, pointed to by "s", was written to and read from after it was freed.

To Build:

gcc uaf.c -o uaf -fsanitize=address -static-libasan

Building it and running it, we get the following report from AddressSanitizer:

=================================================================
==244157==ERROR: AddressSanitizer: heap-use-after-free on address 0x60b0000000f0 at pc 0x00000047a560 bp 0x7ffcdf0d59f0 sp 0x7ffcdf0d51a0
WRITE of size 13 at 0x60b0000000f0 thread T0
    #0 0x47a55f in __interceptor_memcpy ../../.././libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:790
    #1 0x528403 in main /users/PZS0710/edanish/test/asan/uaf.c:8
    #2 0x2b47dd204544 in __libc_start_main (/lib64/libc.so.6+0x22544)
    #3 0x405f5c  (/users/PZS0710/edanish/test/asan/uaf+0x405f5c)

0x60b0000000f0 is located 0 bytes inside of 100-byte region [0x60b0000000f0,0x60b000000154)
freed by thread T0 here:
    #0 0x4ea6f7 in __interceptor_free ../../.././libsanitizer/asan/asan_malloc_linux.cc:122
    #1 0x5283ed in main /users/PZS0710/edanish/test/asan/uaf.c:7
    #2 0x2b47dd204544 in __libc_start_main (/lib64/libc.so.6+0x22544)

previously allocated by thread T0 here:
    #0 0x4eaaa8 in __interceptor_malloc ../../.././libsanitizer/asan/asan_malloc_linux.cc:144
    #1 0x5283dd in main /users/PZS0710/edanish/test/asan/uaf.c:6
    #2 0x2b47dd204544 in __libc_start_main (/lib64/libc.so.6+0x22544)

SUMMARY: AddressSanitizer: heap-use-after-free ../../.././libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:790 in __interceptor_memcpy
Shadow bytes around the buggy address:
  0x0c167fff7fc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c167fff7fd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c167fff7fe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c167fff7ff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c167fff8000: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
=>0x0c167fff8010: fd fd fd fd fd fa fa fa fa fa fa fa fa fa[fd]fd
  0x0c167fff8020: fd fd fd fd fd fd fd fd fd fd fd fa fa fa fa fa
  0x0c167fff8030: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c167fff8040: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c167fff8050: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c167fff8060: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==244157==ABORTING

This is a bit intimidating. It looks like there's alot going on here, but it's not as bad as it looks. Starting at the top, we see what AddressSanitizer detected. In this case, a "WRITE" of 13 bytes (from our strcpy). Immediately below that, we get a stack trace of where the write occured. This tells us that the write occured on line 8 in uaf.c in the function called "main".

Next, AddressSanitizer reports where the memory was located. We can ignore this for now, but depending on your use case, it could be helpful information.

Two key pieces of information follow. AddressSanitizer tells us where the memory was freed (the "freed by thread T0 here" section), giving us another stack trace indicating the memory was freed on line 7. Then, it reports where it was originally allocated ("previously allocated by thread T0 here:"), line 6 in uaf.c.

This is likely enough information to start to debug the issue. The rest of the report provides details about how the memory is laid out, and exactly which addresses were accessed/written to. You probably won't need to pay too much attention to this section. It's a bit "down in the weeds" for most use cases.

Heap Overflow

AddresssSanitizer can also detect heap overflows. Consider the following code (overflow.c):

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
int main(int argc, const char *argv[]) {
    // whoops, forgot c strings are null-terminated
    // and not enough memory was allocated for the copy
    char *s = malloc(12);
    strcpy(s, "Hello world!");
    printf("string is: %s\n", s);
    free(s);
    return 0;
}

The "Hello world!" string is 13 characters long including the null terminator, but we've only allocated 12 bytes, so the strcpy above will overflow the buffer that was allocated. To build this:

gcc overflow.c -o overflow -fsanitize=address -static-libasan -g -Wall

Then, running it, we get the following report from AddressSanitizer:

==168232==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60200000003c at pc 0x000000423454 bp 0x7ffdd58700e0 sp 0x7ffdd586f890
WRITE of size 13 at 0x60200000003c thread T0
    #0 0x423453 in __interceptor_memcpy /apps_src/gnu/8.4.0/src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:737
    #1 0x5097c9 in main /users/PZS0710/edanish/test/asan/overflow.c:8
    #2 0x2ad93cbd7544 in __libc_start_main (/lib64/libc.so.6+0x22544)
    #3 0x405d7b  (/users/PZS0710/edanish/test/asan/overflow+0x405d7b)

0x60200000003c is located 0 bytes to the right of 12-byte region [0x602000000030,0x60200000003c)
allocated by thread T0 here:
    #0 0x4cd5d0 in __interceptor_malloc /apps_src/gnu/8.4.0/src/libsanitizer/asan/asan_malloc_linux.cc:86
    #1 0x5097af in main /users/PZS0710/edanish/test/asan/overflow.c:7
    #2 0x2ad93cbd7544 in __libc_start_main (/lib64/libc.so.6+0x22544)

SUMMARY: AddressSanitizer: heap-buffer-overflow /apps_src/gnu/8.4.0/src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:737 in __interceptor_memcpy
Shadow bytes around the buggy address:
  0x0c047fff7fb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c047fff7fc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c047fff7fd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c047fff7fe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c047fff7ff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c047fff8000: fa fa 00 fa fa fa 00[04]fa fa fa fa fa fa fa fa
  0x0c047fff8010: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff8020: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff8030: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff8040: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff8050: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==168232==ABORTING

This is similar to the use-after-free report we looked at above. It tells us that a heap buffer overflow occured, then goes on to report where the write happened and where the memory was originally allocated. Again, the rest of this report describes the layout of the heap, and probably isn't too important for your use case.

C++ Delete Mismatch

AddressSanitizer can be used on C++ codes as well. Consider the following (bad_delete.cxx):

#include <iostream>
#include <cstring>

int main(int argc, const char *argv[]) {
    char *cstr = new char[100];
    strcpy(cstr, "Hello World");
    std::cout << cstr << std::endl;

    delete cstr;
    return 0;
}

What's the problem here? The memory pointed to by "cstr" was allocated with new[]. An array allocation must be deleted with the delete[] operator, not "delete".

To build this code, just use g++ instead of gcc:

g++ bad_delete.cxx -o bad_delete -fsanitize=address -static-libasan -g

And running it, we get the following output:

Hello World
=================================================================
==257438==ERROR: AddressSanitizer: alloc-dealloc-mismatch (operator new [] vs operator delete) on 0x60b000000040
    #0 0x4d0a78 in operator delete(void*, unsigned long) /apps_src/gnu/8.4.0/src/libsanitizer/asan/asan_new_delete.cc:151
    #1 0x509ea8 in main /users/PZS0710/edanish/test/asan/bad_delete.cxx:9
    #2 0x2b8232878544 in __libc_start_main (/lib64/libc.so.6+0x22544)
    #3 0x40642b  (/users/PZS0710/edanish/test/asan/bad_delete+0x40642b)

0x60b000000040 is located 0 bytes inside of 100-byte region [0x60b000000040,0x60b0000000a4)
allocated by thread T0 here:
    #0 0x4cf840 in operator new[](unsigned long) /apps_src/gnu/8.4.0/src/libsanitizer/asan/asan_new_delete.cc:93
    #1 0x509e5f in main /users/PZS0710/edanish/test/asan/bad_delete.cxx:5
    #2 0x2b8232878544 in __libc_start_main (/lib64/libc.so.6+0x22544)

SUMMARY: AddressSanitizer: alloc-dealloc-mismatch /apps_src/gnu/8.4.0/src/libsanitizer/asan/asan_new_delete.cc:151 in operator delete(void*, unsigned long)
==257438==HINT: if you don't care about these errors you may set ASAN_OPTIONS=alloc_dealloc_mismatch=0
==257438==ABORTING

This is similar to the other AddressSanitizer outputs we've looked at. This time, it tells us there's a mismatch between new and delete. It prints a stack trace for where the delete occured (line 9) and also a stack trace for where to allocation occured (line 5).

Performance

The documentation states:

This tool is very fast. The average slowdown of the instrumented program is ~2x

AddressSanitizer is much faster than tools that do similar analysis such as valgrind. This allows for usage on HPC codes.

However, if you find that AddressSanitizer is too slow for your code, there are compiler flags that can be used to disable it for specific functions. This way, you can use address sanitizer on cooler parts of your code, while manually auditing the hot paths.

The compiler directive to skip analyzing functions is:

__attribute__((no_sanitize_address)

It is possible to utilize Cron and the OSCusage command to send regular usage reports via email

Cron

It is easy to create Cron jobs on the Owens and Pitzer clusters at OSC. Cron is a Linux utility which allows the user to schedule a command or script to run automatically at a specific date and time. A cron job is the task that is scheduled.

Shell scripts run as a cron job are usually used to update and modify files or databases; however, they can perform other tasks, for example a cron job can send an email notification.

Getting Help

In order to use what cron has to offer, here is a list of the command name and options that can be used

Usage: 
crontab [options] file 
crontab [options] 
crontab -n [hostname] 
Options: 
-u  define user 
-e edit user's crontab 
-l list user's crontab 
-r delete user's crontab 
-i prompt before deleting 
-n  set host in cluster to run users' crontabs 
-c get host in cluster to run users' crontabs 
-s selinux context 
-x  enable debugging 

Also, if this is your first time using cron, you will be asked to choose an editor for setting your cron job. Choose whatever you find to be easiest for you.

Running a Cron Job

To check for any running cron jobs on the server, use the command (As shown above)

crontab -l 

and to create and edit your cron job use the following command,

crontab -e 

Now, in order to write you first cron job, you need to be familiar with the formatting system that cron follows.

Linux Crontab Format

The formatting system has 6 fields, each field from 1-5 is used to define the date and time of the execution. The 6th field is used for the command or script to be executed. The format is the following,

MIN HOUR DOM MON DOW CMD 

where,

figure 1: Cron’s formatting syntax

Getting Notified by Email Using a Cron Job

You can get an email notification using a cron job as mentioned earlier. The following is an example of a cron job that runs every minute and sends an email notification every minute,

* * * * * {cmd} | mail -s "title of the email notification" {your email} 

A user can also set up email notifications regarding usage by using the OSCusage cmd,

12 15 * * * /opt/osc/bin/OSCusage | mail -s "OSC usage on $(date)" {your email} 2> /path/to/file/for/stdout/and/stderr 2>&1 

This cron job will run every day at (15:12 or 3:12 PM).

Using OSCusage

The OSCusage command offers many options, the following is a list that pertains to that,

$ /opt/osc/bin/OSCusage --help 
usage: OSCusage.py [-h] [-u USER] 
[-s {opt,pitzer,glenn,bale,oak,oakley,owens,ruby}] [-A] 
[-P PROJECT] [-q] [-H] [-r] [-n] [-v] 
[start_date] [end_date] 

positional arguments: 
start_date start date (default: 2020-04-23) 
end_date end date (default: 2020-04-24) 

optional arguments: 
-h, --help show this help message and exit 
-u USER, --user USER username to run as. Be sure to include -P or -A. (default: kalattar) 
-s {opt,pitzer,glenn,bale,oak,oakley,owens,ruby}, --system {opt,pitzer,glenn,bale,oak,oakle 
-A Show all 
-P PROJECT, --project PROJECT project to query (default: PZS0715) 
-q show user data 
-H show hours 
-r show raw 
-n show job ID 
-v do not summarize 

As it can be seen, one could for example use OSCusage to receive information regarding another user’s usage with the -u option and write a cron script that is set up with email notification.

Some other usage examples,

 OSCusage 2018-01-24 

where the command specifies the usage’s start time. The end time could also be specified with,

OSCusage 2018-01-24 2018-01-25 

Terminating a Cron Job

To terminate a cron job, you need to first determine the process id,

ps aux | grep crontab 

and then use,

kill {PID}

A user can also just clear out the cron script with,

crontab -e 

It is now possible to run Docker and Singularity containers on the Owens and Pitzer clusters at OSC. Single-node jobs are currently supported, including GPU jobs; MPI jobs are planned for the future.

From the Docker website:  "A container image is a lightweight, stand-alone, executable package of a piece of software that includes everything needed to run it: code, runtime, system tools, system libraries, settings."

This document will describe how to run Docker and Singularity containers on the Owens and Pitzer. You can use containers from Docker Hub, Sylabs Cloud, Singularity Hub, or any other source. As examples we will use hello-world from Singularity Hub and ubuntu from Docker Hub.

Getting help

The most up-to-date help on Singularity comes from the command itself.

singularity help

User guides and examples can be found at Sylabs.io.

Setting up your environment for Singularity usage

No setup is required. You can use Singulairty directly on all clusters.

Accessing a container

A Singularity container is a single file with a .sif  extension.

You can simply download ("pull") a container from a hub. Popular hubs are Docker Hub and Singularity Hub. You can go there and search if they have a container that meets your needs. Docker Hub has more containers and may be more up to date but supports a much wider community than just HPC. Singularity Hub is for HPC, but the number of available containers are fewer. Additionally there are domain and vendor repositories such as biocontainers and NVIDIA HPC containers that may have relevant containers.

Pull a container from hubs

Docker Hub

Pull from the 7.2.0 branch of the gcc repository on Docker Hub. The 7.2.0 is called a tag.

singularity pull docker://gcc:7.2.0

Filename:  gcc_7.2.0.sif

Pull an Ubuntu container from Docker Hub.

singularity pull docker://ubuntu:18.04

Filename:  ubuntu_18.04.sif

Singularity Hub

Pull the singularityhub/hello-world ontainer from the Singularity hub. Since no tag is specified it pulls from the master branch of the repository.

singularity pull shub://singularityhub/hello-world

Filename:  hello-world_latest.sif

Downloading containers from the hubs is not the only way to get one. You can, for example get a copy from your colleague's computer or directory. If you would like to create your own container you can start from the user guide below. If you have any questions, please contact OSC Help.

Running a container

There are four ways to run a container under Singularity.

You can do this either in a batch job or on a login node. (Don’t run on a login node if the container will be performing heavy computation, of course.)

We note that the operating system on Owens is Red Hat:

[owens-login01]$ cat /etc/os-release
NAME="Red Hat Enterprise Linux Server"
VERSION="7.5 (Maipo)"
ID="rhel"
[..more..]

In the examples below we will often check the operating system to show that we are really inside a container.

Run container like a native command

If you simply run the container image it will execute the container’s runscript.

Example:  Run singularityhub/hello-world

Note that this container returns you to your native OS after you run it.

[owens-login01]$ ./hello-world_latest.sif
Tacotacotaco

Use the “run” sub-command

The Singularity “run” sub-command does the same thing as running a container directly as described above. That is, it executes the container’s runscript.

Example:  Run a container from a local file

[owens-login01]$ singularity run hello-world_latest.sif
Tacotacotaco

Example:  Run a container from a hub without explicitly downloading it

[owens-login01]$ singularity run shub://singularityhub/hello-world
INFO: Downloading shub image
Progress |===================================| 100.0%
Tacotacotaco

Use the “exec” sub-command

The Singularity “exec” sub-command lets you execute an arbitrary command within your container instead of just the runscript.

Example:  Find out what operating system the singularityhub/hello-world container uses

[owens-login01]$ singularity exec hello-world_latest.sif cat /etc/os-release
NAME="Ubuntu"
VERSION="14.04.5 LTS, Trusty Tahr"
ID=ubuntu
[..more..] 

Use the “shell” sub-command

The Singularity “shell” sub-command invokes an interactive shell within a container.

Example:  Run an Ubuntu shell. Note the “Singularity” prompt within the shell.

[owens-login01 ~]$ singularity shell ubuntu_18.04.sif
Singularity ubuntu_18.04.sif:~> cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04 LTS (Bionic Beaver)"
ID=ubuntu
[.. more ..] 
Singularity ubuntu_18.04.sif:~> exit
exit

File system access

When you use a container you run within the container’s environment.  The directories available to you by default from the host environment are

• your home directory
• working directory (directory you were in when you ran the container)
• /fs/project
• /fs/scratch
• /tmp

You can review our Available File Systems page for more details about our file system access policy.

If you run the container within a job you will have the usual access to the $PFSDIR environment variable with adding node attribute "pfsdir" in the job request (nodes=XX:ppn=XX:pfsdir). You can access most of our file systems from a container without any special treatment.

GPU usage within a container

If you have a GPU-enabled container you can easily run it on Owens or Pitzer just by adding the --nv flag to the singularity exec or run command.  The example below comes from the "exec" command section of Singularity User Guide.  It runs a TensorFlow example using a GPU on Owens.  (Output has been omitted from the example for brevity.)

[owens-login01]$ sinteractive -n 28 -g 1
...
[o0756]$ git clone https://github.com/tensorflow/models.git
[o0756]$ singularity exec --nv docker://tensorflow/tensorflow:latest-gpu \
python ./models/tutorials/image/mnist/convolutional.py

In some cases it may be necessary to bind the CUDA_HOME path and add $CUDA_HOME/lib64 to the shared library search path:

[owens-login01]$ sinteractive -n 28 -g 1
...
[o0756]$ module load cuda
[o0756]$ export SINGULARITY_BINDPATH=$CUDA_HOME
[o0756]$ export SINGULARITYENV_LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CUDA_HOME/lib64
[o0756]$ singularity exec --nv my_container mycmd

Build a container

If you want to create or modify a container, you need root-like privilege on any OSC system. Alternatively you can create a container on a hub or a local computer then pull/upload it to OSC system.  

References

• https://www.sylabs.io/docs/
• https://github.com/ArangoGutierrez/Singularity-tutorial
• https://hpc.nih.gov/apps/singularity.html
• https://www.docker.com/

Globus is a cloud-based service designed to let users move, share, and discover research data via a single interface, regardless of its location or number of files or size.

Globus was developed and is maintained at the University of Chicago and is used extensively at supercomputer centers and major research facilities.

Globus is available as a free service that any user can access. More on how Globus works can be found on the Globus "How It Works" page.

Further Reading

• https://www.globus.org/

SSHing directly to a compute node at OSC - even if that node has been assigned to you in a current batch job - and starting VNC is an "unsafe" thing to do. When your batch job ends (and the node is assigned to other users), stray processes will be left behind and negatively impact other users. However, it is possible to use VNC on compute nodes safely.

The examples below are for Pitzer. If you use other systems, please see this page for supported versions of TurboVNC on our systems.

Starting your VNC server

Step one is to create your VNC server inside a batch job.

Option 1: Interactive

The preferred method is to start an interactive job, requesting an gpu node, and then once your job starts, you can start the VNC server.

salloc --nodes=1 --ntasks-per-node=40 --gpus-per-node=1 --gres=vis --constraint=40core srun --pty /bin/bash

This command requests an entire GPU node, and tells the batch system you wish to use the GPUs for visualization. This will ensure that the X11 server can access the GPU for acceleration. In this example, I have not specified a duration, which will then default to 1 hour.

module load virtualgl
module load turbovnc

Then start your VNC server. (The first time you run this command, it may ask you for a password - this is to secure your VNC session from unauthorized connections. Set it to whatever password you desire. We recommend a strong password.)

vncserver

The output of this command is important: it tells you where to point your client to access your desktop. Specifically, we need both the host name (before the :), and the screen (after the :).

New 'X' desktop is p0302.ten.osc.edu:1

Connecting to your VNC server

Because the compute nodes of our clusters are not directly accessible, you must log in to one of the login nodes and allow your VNC client to "tunnel" through SSH to the compute node. The specific method of doing so may vary depending on your client software.

The port assigned to the vncserver will be needed. It is usually 5900 + <display_number>. e.g.

New 'X' desktop is p0302.ten.osc.edu:1

would use port 5901.

Linux/MacOS

Option 1: Manually create an SSH tunnel 

I will be providing the basic command line syntax, which works on Linux and MacOS. You would issue this in a new terminal window on your local machine, creating a new connection to Pitzer.

ssh -L <port>:<node_hostname>.ten.osc.edu:<port> <username>@pitzer.osc.edu

The above command establishes a proper ssh connection for the vnc client to use for tunneling to the node.

Open your VNC client, and connect to localhost:<screen_number>, which will tunnel to the correct node on Pitzer.

Option 2: Use your VNC software to tunnel 

This example uses Chicken of the VNC, a MacOS VNC client. It is a vncserver started on host n0302 with port 5901 and display 1.

The default window that comes up for Chicken requires the host to connect to, the screen (or port) number, and optionally allows you to specify a host to tunnel through via SSH. This screenshot shows a proper configuration for the output of vncserver shown above. Substitute your host, screen, and username as appropriate.

When you click [Connect], you will be prompted for your HPC password (to establish the tunnel, provided you did not input it into the "password" box on this dialog), and then (if you set one), for your VNC password. If your passwords are correct, the desktop will display in your client.

Windows

This example shows how to create a SSH tunnel through your ssh client.  We will be using Putty in this example, but these steps are applicable to most SSH clients.

First, make sure you have x11 forwarding enabled in your SSH client.

Next, open up the port forwarding/tunnels settings and enter the hostname and port you got earlier in the destination field.  You will need to add 5900 to the port number when specifiying it here.  Some clients may have separate boxes for the desination hostname and port.  

For source port, pick a number between 11-99 and add 5900 to it.  This number between 11-99 will be the port you connect to in your VNC client.

Make sure to add the forwaded port, and save the changes you've made before exiting the configutations window.

Now start a SSH session to the respective cluster your vncserver is running on.  The port forwarding will automatically happen in the background.  Closing this SSH session will close the forwarded port; leave the session open as long as you want to use VNC.

Now start a VNC client.  TurboVNC has been tested with our systems and is recommended.  Enter localhost:[port], replacing [port] with the port between 11-99 you chose earlier.

If you've set up a VNC password you will be prompted for it now.  A desktop display should pop up now if everything is configured correctly.

How to Kill a VNC session?

Occasionally you may make a mistake and start a VNC server on a login node or somewhere else you did not want to.  In this case it is important to know how to properly kill your VNC server so no processes are left behind.

The command syntax to kill a VNC session is:

vncserver -kill :[screen]

In the example above, screen would be 1.

You need to make sure you are on the same node you spawned the VNC server on when running this command.

Conda environments must be installed on Jupyter notebooks prior to use. Whereas older versions of Conda automatically installed a Jupyter kernel, users must now manually perform the installation process.

To perform the installation, users should load the preferred version of Python, activate the Conda environment and run the following commands.

module load python             # change to the proper version of python
MYENV='useful-project-name'    # replace with the name of conda env
source activate "$MYENV"
python -m ipykernel install --user --name "$MYENV" --display-name "Python ($MYENV)"

If the envirnoment is rebuilt or renamed, users may want to erase any custom jupyter kernel installations.

rm -rf ~/.local/share/jupyter/kernels/$(MYENV)

Many software packages require a license.  These licenses are usually made available via a license server, which allows software to check out necessary licenses.  In this document external refers to a license server that is not hosted inside OSC.

If you have such a software license server set up using a license manager, such as FlexNet, this guide will instruct you on the necessary steps to connect to and use the licenses at OSC.

Users who wish to host their software licenses inside OSC should consult OSC Help.

Introduction

Broadly speaking, there are two different ways in which the external license server's network may be configured.  These differ by whether the license server is directly externally reachable or if it sits behind a private internal network with a port forwarding firewall.  

If your license server sits behind a private internal network with a port forwarding firewall you will need to take additional steps to allow the connection from our systems to the license server to be properly routed. 

License Server is Directly Externally Reachable

License Server is Behind Port Forwarding Firewall

Unsure?

If you are unsure about which category your situation falls under contact your local IT administrator.

Configure Remote Firewall

In order for connections from OSC to reach the license server, the license server's firewall will need to be configured.  All outbound network traffic from all of OSC's compute nodes are routed through a network address translation host (NAT), or two backup servers.

The license server should be configured to allow connections from the following IP addresses to the SERVER:PORT where the license server is running:

• nat.osc.edu (192.157.5.13)
• 192.148.248.35
• 192.148.248.186

A typical FlexNet-based license server uses two ports: one is server port and the other is daemon port, and the firewall should be configured for the both ports. A typical license file looks, for example,

SERVER licXXX.osc.edu 0050XXXXX5C 28000
VENDOR {license name} port=28001

In this example, "28000" is the server port, and "28001" is the daemon port. The daemon port is not mandatory if you use it on a local network, however it becomes necessary if you want to use it outside of your local network. So, please make sure you declared the daemon port in the license file and configured the firewall for the port.

Confirm Configuration

The firewall settings should be verified by attempting to connect to the license server from the compute environment using telenet.

Get on to a compute node by requesting a short, small, interactive job and test the connection using telenet:

telnet <License Server IP Address> <Port#>

(Recommended) Restrict Access to IPs/Usernames

It is also recommended to restrict accessibility using the remote license server's access control mechanisms, such as limiting access to particular usernames in the options.dat file used with FlexNet-based license servers.

For FlexNet tools, you can add the following line to your options.dat file, one for each user.

INCLUDEALL USER <OSC username>

If you have a large number of users to give access to you may want to define a group using GROUP within the options.dat file and give access to that whole group using INCLUDEALL GROUP <group name> .

Users who use other license managers should consult the license manager's documentation.

Modify Job Environment to Point at License Server

The software must now be told to contact the license server for it's licenses.  The exact method of doing so can vary between each software package, but most use an environment variable that specifies the license server IP address and port number to use.

For example LS DYNA uses the environment variable LSTC_LICENSE and LSTC_LICENSE_SERVER to know where to look for the license.  The following lines would be added to a job script to tell LS-DYNA to use licenses from port 2345 on server 1.2.3.4, if you use bash:

export LSTC_LICENSE=network
export LSTC_LICENSE_SERVER=2345@1.2.3.4

or, if you use csh:

setenv LSTC_LICENSE network
setenv LSTC_LICENSE_SERVER 2345@1.2.3.4

License Server is Behind Port Forwarding Firewall

If the license server is behind a port forwarding firewall, and has a different IP address from the IP address of the firewall, additional steps must be taken to allow connections to be properly routed within the license server's internal network.  

1. Use the license server's fully qualified domain name in SERVER line in the license file instead of the IP address.
2. Contact your IT team to have the firewall IP address mapped to the fully qualified domain name.

Software Specific Details

The following outlines details particular to a specific software package.  

ANSYSLI_SERVERS=<port>@<IP>
ANSYSLMD_LICENSE_FILE=<port>@<IP>

This document shows you how to set soft limits using the ulimit command.

The ulimit command sets or reports user process resource limits. The default limits are defined and applied when a new user is added to the system. Limits are categorized as either soft or hard. With the ulimit command, you can change your soft limits for the current shell environment, up to the maximum set by the hard limits. You must have root user authority to change resource hard limits.

Syntax

ulimit [-HSTabcdefilmnpqrstuvx [Limit]]

The limit for a specified resource is set when the Limit parameter is specified. The value of the Limit parameter can be a number in the unit specified with each resource, or the value "unlimited." For example, to set the file size limit to 51,200 bytes, use:

ulimit -f 100

To set the size of core dumps to unlimited, use:

ulimit –c unlimited

How to change ulimit for a MPI program

The ulimit command affects the current shell environment. When a MPI program is started, it does not spawn in the current shell. You have to use srun to start a wrapper script that sets the limit if you want to set the limit for each process. Below is how you set the limit for each shell (We use  ulimit –c unlimited to allow unlimited core dumps, as an example): 

1. Prepare your batch job script named "myjob" as below (Here, we request a job with 5-hour 2-cores):

#!/bin/bash
#SBATCH --ntasks=2
#SBATCH --time=5:00:00
#SBATCH ...


...
srun ./test1
...

2. Prepare the wrapper script named "test1" as below:

#!/bin/bash
ulimit –c unlimited
.....(your own program)

3. sbatch myjob

Introduction

OSCfinger is a command developed at OSC for use on OSC's systems and is similar to the standard finger command. It allows various account information to be viewed.

Availability

Usage

OSCfinger takes the following options and parameters.

$ OSCfinger -h
usage: OSCfinger.py [-h] [-e] [-g] USER

positional arguments:
  USER

optional arguments:
  -h, --help   show this help message and exit
  -e           Extend search to include gecos/full name (user) or
               category/institution (group)
  -g, --group  Query group instead of users

Query user:
    OSCfinger foobar

Query by first or last name:
    OSCfinger -e Foo
    OSCfinger -e Bar

Query group:
    OSCfinger -g PZS0001

Query group by category or insitituion:
    OSCfinger -e -g OSC    

View information by username

The OSCfinger command can be used to view account information given a username.

$ OSCfinger jsmith
Login: xxx                                   Name: John Smith
Directory: xxx                               Shell: /bin/bash
E-mail: xxx
Primary Group: PPP1234
Groups: 

Project Information by Project ID

The OSCfinger command can also reveal details about a project using the -g flag.

$ OSCfinger -g PPP1234
Group: PPP1234                                    GID: 1234
Status: ACTIVE                                    Type: Academic
Principal Investigator: xxx                       Admins: NA
Members: xxx
Category: NA
Institution: OHIO SUPERCOMPUTER CENTER
Description: xxx
---

Search for a user via first and/or last name

If the username is not known, a lookup can be initiated using the -e flag.

This example is shown using the lookup for a first and last name.

$ OSCfinger -e "John Smith"
Login: jsmith                                     Name: John Smith
Directory: xxx                                    Shell: /bin/bash
E-mail: NA
Primary Group: PPP1234
Groups: xxx
Password Changed: Jul 04 1776 15:47 (calculated)  Password Expires: Aug 21 1778 12:05 AM
Login Disabled: FALSE                             Password Expired: FALSE
---

One can also lookup users with only the last name:

$ OSCfinger -e smith
Login: jsmith                                      Name: John Smith
Directory: xxx                                    Shell: /bin/bash
E-mail: NA
Primary Group: PPP1234
Groups:
---
Login: asmith                                     Name: Anne Smith
Directory: xxx                                    Shell: /bin/bash
E-mail: xxx
Primary Group: xxx
Groups: 
---

Only the first name can also be used, but many accounts are likely to be returned.

$ OSCfinger -e John
Login: jsmith                                     Name: John Smith
Directory: xxx                                    Shell: /bin/bash
E-mail: xxx
Primary Group: PPP1234
Groups:
---
Login: xxx                                        Name: John XXX
Directory: xxx                                    Shell: /bin/bash
E-mail: xxx
Primary Group: xxx
Groups:
---
Login: xxx                                        Name: John XXX
Directory: xxx                                    Shell: /bin/ksh
E-mail: xxx
Primary Group: xxx
Groups:
---
...(more accounts below)...

Slurm usage

While in a slurm environment, the OSCfinger command shows some additional information:

$ OSCfinger jsmith 
Login: xxx Name: John Smith 
Directory: xxx Shell: /bin/bash 
E-mail: xxx 
Primary Group: PPP1234 
Groups:
SLURM Enabled: TRUE
SLURM Clusters: pitzer
SLURM Accounts: PPP1234, PPP4321
SLURM Default Account: PPPP1234

It's important to note that the default account in slurm will be used if an account is not specified at job submission.

Introduction

OSCgetent is a command developed at OSC for use on OSC's systems and is similar to the standard getent command. It lets one view group information.

Availability

Usage

OSCgetent takes the following options and parameters.

$ OSCgetent -h
usage: OSCgetent.py [-h] {group} [name [name ...]]

positional arguments:
  {group}
  name

optional arguments:
  -h, --help  show this help message and exit

Query group:
    OSCgetent.py group PZS0708

Query multiple groups:
    OSCgetent.py group PZS0708 PZS0709
    

View group information

The OSCgetent command can be used to view group(s) members:

$ OSCgetent group PZS0712
PZS0712:*:5513:amarcum,amarcumtest,amarcumtest2,guilfoos,hhamblin,kcahill,xwang 

View information on multiple groups

$ OSCgetent group PZS0712 PZS0708
PZS0708:*:5509:djohnson,ewahl,kearley,kyriacou,linli,soottikkal,tdockendorf,troy
PZS0712:*:5513:amarcum,amarcumtest,amarcumtest2,guilfoos,hhamblin,kcahill,xwang

Introduction

OSCprojects is a command developed at OSC for use on OSC's systems and is used to view your logged in accounts project information.

Availability

Usage

OSCprojects does not take any arguments or options:

$ OSCprojects
OSC projects for user amarcumtest2:

Project         Status          Members
-------         ------          -------
PZS0712         ACTIVE          amarcumtest2,amarcumtest,guilfoos,amarcum,xwang
PZS0726         ACTIVE          amarcumtest2,xwangtest,amarcum

This command returns the current users projects, whether those projects are active/restricted and the current members of the projects.

Introduction

OSCusage is command developed at OSC for use on OSC's systems.  It allows for a user to see information on their project's usage, including different users and their jobs.

Availability

Usage

OSCusage takes the following options and parameters.

$ OSCusage --help
usage: OSCusage.py [-h] [-u USER]
                   [-s {opt,pitzer,glenn,bale,oak,oakley,owens,ruby}] [-A]
                   [-P PROJECT] [-q] [-H] [-r] [-n] [-v]
                   [start_date] [end_date]

positional arguments:
  start_date            start date (default: 2021-03-16)
  end_date              end date (default: 2021-03-17)

optional arguments:
  -h, --help            show this help message and exit
  -u USER, --user USER  username to run as. Be sure to include -P or -A.
                        (default: amarcum)
  -s {opt,pitzer,glenn,bale,oak,oakley,owens,ruby}, --system {opt,pitzer,glenn,bale,oak,oakley,owens,ruby}
  -A                    Show all
  -P PROJECT, --project PROJECT
                        project to query (default: PZS0712)
  -q                    show user data
  -H                    show hours
  -r                    show raw
  -n                    show job ID
  -v                    do not summarize

Usage Examples:

    Specify start time:
        OSCusage 2018-01-24

    Specify start and end time:
        OSCusage 2018-01-24 2018-01-25

Today's Usage

Running OSCusage with no options or parameters specified will provide the usage information in Dollars for the current day.

$ OSCusage
----------------  ------------------------------------
                  Usage Statistics for project PZS0712
Time              2021-03-16 to 2021-03-17
PI                guilfoos@osc.edu
Remaining Budget  -1.15
----------------  ------------------------------------

User          Jobs    Dollars    Status
------------  ------  ---------  --------
amarcum       0       0.0        ACTIVE
amarcumtest   0       0.0        ACTIVE
amarcumtest2  0       0.0        ACTIVE
guilfoos      0       0.0        ACTIVE
hhamblin      0       0.0        ACTIVE
kcahill       0       0.0        ACTIVE
wouma         0       0.0        ACTIVE
xwang         12      0.0        ACTIVE
--            --      --
TOTAL         12      0.0

Usage in Timeframe

If you specify a timeframe you can get utilization information specifically for jobs that completed within that period.

$ OSCusage 2020-01-01 2020-07-01 -H
----------------  ------------------------------------
                  Usage Statistics for project PZS0712
Time              2020-01-01 to 2020-07-01
PI                Brian Guilfoos <guilfoos@osc.edu>
Remaining Budget  -1.15
----------------  ------------------------------------

User          Jobs    core-hours    Status
------------  ------  ------------  ----------
amarcum       86      260.3887      ACTIVE
amarcumtest   0       0.0           ACTIVE
amarcumtest2  0       0.0           RESTRICTED
guilfoos      9       29.187        ACTIVE
hhamblin      1       1.01          ACTIVE
kcahill       7       40.5812       ACTIVE
wouma         63      841.2503      ACTIVE
xwang         253     8148.2638     ACTIVE
--            --      --
TOTAL         419     9320.681

Show only a single user's usage

Specify -q to show only the current user's usage. This stacks with -u to specify which user you want to see.

$ OSCusage -u xwang -q 2020-01-01 2020-07-01 -H
----  -------------------------------
      Usage Statistics for user xwang
Time  2020-01-01 to 2020-07-01
----  -------------------------------

User    Jobs    core-hours    Status
------  ------  ------------  --------
xwang   253     8148.2638     -
--      --      --
TOTAL   253     8148.2638

Show a particular project

By default, the tool shows your default (first) project. You can use -P to specify which charge code to report on.

$ OSCusage -P PZS0200 -H
----------------  ------------------------------------
                  Usage Statistics for project PZS0200
Time              2020-09-13 to 2020-09-14
PI                David Hudak <dhudak@osc.edu>
Remaining Budget  0
----------------  ------------------------------------

User        Jobs    core-hours    Status
----------  ------  ------------  ----------
adraghi     0       0.0           ARCHIVED
airani      0       0.0           ARCHIVED
alingg      0       0.0           ARCHIVED

You can show all of your charge codes/projects at once, by using -A .

Select a particular cluster

By default, all charges are shown in the output. However, you can filter to show a particular system with -s .

$ OSCusage -s pitzer -H
----------------  ------------------------------------
                  Usage Statistics for project PZS0712
Time              2021-03-16 to 2021-03-17
PI                guilfoos@osc.edu
Remaining Budget  -1.15
----------------  ------------------------------------

User          Jobs    core-hours    Status
------------  ------  ------------  --------
amarcum       0       0.0           ACTIVE
amarcumtest   0       0.0           ACTIVE
amarcumtest2  0       0.0           ACTIVE
guilfoos      0       0.0           ACTIVE
hhamblin      0       0.0           ACTIVE
kcahill       0       0.0           ACTIVE
wouma         0       0.0           ACTIVE
xwang         0       0.0           ACTIVE
--            --      --
TOTAL         0       0.0

Changing the units reported

The report can show usage dollars. You can elect to get usage in core-hours using -H or raw seconds using -r

$ OSCusage 2020-01-01 2020-07-01 -r
----------------  ------------------------------------
                  Usage Statistics for project PZS0712
Time              2020-01-01 to 2020-07-01
PI                Brian Guilfoos <guilfoos@osc.edu>
Remaining Budget  -1.15
----------------  ------------------------------------

User          Jobs    raw_used    Status
------------  ------  ----------  ----------
amarcum       86      937397.0    ACTIVE
amarcumtest   0       0.0         ACTIVE
amarcumtest2  0       0.0         RESTRICTED
guilfoos      9       105073.0    ACTIVE
hhamblin      1       3636.0      ACTIVE
kcahill       7       146092.0    ACTIVE
wouma         63      3028500.0   ACTIVE
xwang         253     29333749.0  ACTIVE
--            --      --
TOTAL         419     33554447.0


Detailed Charges Breakdown

Specify -v to get detailed information jobs.

You can add the -n option to the -v option to add the job ID to the report output. OSCHelp will need the job ID to answer any questions about a particular job record.

Please contact OSC Help with questions. 

Introduction

checkgpu is a command developed at OSC for use on OSC’s systems to report information regarding the usage of OSC’s GPU nodes. It reports various information on the usage and availability of the GPU nodes.

Availability

Usage

checkgpu takes the following options and parameters (viewable by passing  -h/--help)

$ checkgpu -h
usage: checkgpu.py [-h] [-j] [-r] [-q] [-n] [-u] [-a] [-v]

optional arguments:
  -h, --help     show this help message and exit
  -j, --jobs     Check status of gpu jobs
  -r, --run      Check status of running gpu jobs (use with --jobs)
  -q, --queued   Check status of queued gpu jobs (use with --jobs)
  -n, --node     Check status of gpu nodes
  -u, --used     Check status of used gpu nodes (use with --node)
  -a, --avail    Check status of available gpu nodes (use with --node)
  -v, --verbose  Display full job report

View information on GPU jobs

By using the flag -j/--jobs, checkgpu can report summaries for the total number of jobs using GPU nodes

$ checkgpu -j
*** SUMMARY ***
===================================================
Summary of Running GPU Jobs:

152 GPU jobs running using 152 GPU nodes. Among them:
 * 146 GPU jobs running using 146 GPU nodes from non-condo and non-debugging jobs
 * 0 GPU jobs running using 0 GPU nodes from debugging jobs
 * 6 GPU Jobs running using 6 GPU nodes from PCONXXXX
===================================================
Summary of Queued GPU Jobs:

485 GPU Jobs queued requesting 492 GPU nodes. Among them:
 * 382 GPU Jobs queued requesting 389 GPU nodes from non-condo and non-debugging jobs
 * 0 GPU Jobs queued requesting 0 debug GPU nodes
 * 103 GPU Jobs queued requesting 103 GPU nodes from PCONXXXX
===================================================

View information on GPU jobs (run-only)

By using the flag -r/--run in tandem with -j, checkgpu can report summaries for the total number of running jobs using GPU nodes

$ checkgpu -j --run
===================================================
Summary of Running GPU Jobs:
152 GPU jobs running using 152 GPU nodes. Among them:
 * 146 GPU jobs running using 146 GPU nodes from non-condo and non-debugging jobs
 * 0 GPU jobs running using 0 GPU nodes from debugging jobs
 * 6 GPU Jobs running using 6 GPU nodes from PCONXXXX

View information on GPU jobs (queue-only)

By using the flag-q/--queued in tandem with -j, checkgpu can report summaries for the total number of queued jobs using GPU nodes

$ checkgpu -j --queued
===================================================
Summary of Queued GPU jobs:
484 GPU Jobs queued requesting 491 GPU nodes. Among them:
 * 382 GPU Jobs queued requesting 389 GPU nodes from non-condo and non-debugging jobs
 * 0 GPU Jobs queued requesting 0 debug GPU nodes
 * 102 GPU Jobs queued requesting 102 GPU nodes from PCONXXXX

View information on GPU nodes

By using the flag -n/--node, checkgpu can report summaries for the usage of OSC’s GPU nodes

$ checkgpu --node
***USED NODES***
================================================================
================================================================
Summary of the current GPU nodes:
153 total jobs on gpu nodes
 * 153 GPU jobs using 153 GPU nodes. Among them:
  ** 147 GPU nodes used by non-condo and non-debug jobs
  ** 0 GPU nodes used by debug jobs
  ** 6 GPU nodes used by condo group PCONXXXX
 * 0 non-GPU jobs using 0 GPU nodes. Among them:
  ** 0 GPU nodes used by non-condo and non-debug non-gpu jobs
  ** 0 GPU nodes used by debug non-gpu jobs
  ** 0 GPU nodes used by non-gpu condo jobs

***AVAILABLE NODES***
================================================================
================================================================
Summary of the available GPU nodes:
144 available gpu nodes. Among them: 
 * 4 fully available gpu nodes.
 * 140 partly available gpu nodes.
 * 0 open gpus on partly available nodes.

View information on GPU nodes (used-only)

By using the flag -u/--used in tandem with -n, checkgpu can report summaries for the usage of used GPU nodes

$ checkgpu -n -u
================================================================
Summary of the current GPU nodes:
152 total jobs on gpu nodes
 * 152 GPU jobs using 152 GPU nodes. Among them:
  ** 147 GPU nodes used by non-condo and non-debug jobs
  ** 0 GPU nodes used by debug jobs
  ** 5 GPU nodes used by condo group PCONXXXX
 * 0 non-GPU jobs using 0 GPU nodes. Among them:
  ** 0 GPU nodes used by non-condo and non-debug non-gpu jobs
  ** 0 GPU nodes used by debug non-gpu jobs
  ** 0 GPU nodes used by non-gpu condo jobs

View information on GPU nodes (available-only)

By using the flag -a/--avail in tandem with -n, checkgpu can report summaries for the availability of GPU nodes

$ chekgpu -n -a
================================================================
Summary of the available GPU nodes:
144 available gpu nodes. Among them: 
 * 4 fully available gpu nodes.
 * 140 partly available gpu nodes.
 * 0 open gpus on partly available nodes.

View full job summary

By using the flag -v/--verbose in tandem with any of the above command options, checkgpu will report full summaries on the usage of GPU nodes

$ checkgpu --jobs --run -v
============================================
Job Details:
List of non-condo and non-debug GPU jobs
XXXXXXX.owens-batch.ten.osc.edu
o0678-gpu/0
XXXXXXX.owens-batch.ten.osc.edu
o0673-gpu/0
XXXXXXX.owens-batch.ten.osc.edu
o0695-gpu/0
XXXXXXX.owens-batch.ten.osc.edu
o0805-gpu/0
XXXXXXX.owens-batch.ten.osc.edu
o0763-gpu/0
...

List of debug GPU jobs

List of GPU jobs from PCON0005
XXXXXXX.owens-batch.ten.osc.edu
o0732-gpu/0
XXXXXXX.owens-batch.ten.osc.edu
o0779-gpu/0
XXXXXXX.owens-batch.ten.osc.edu
o0745-gpu/0
XXXXXXX.owens-batch.ten.osc.edu
o0712-gpu/0
XXXXXXX.owens-batch.ten.osc.edu
o0663-gpu/0
XXXXXXX.owens-batch.ten.osc.edu
o0750-gpu/0

===================================================
Summary of Running GPU Jobs:
153 GPU jobs running using 153 GPU nodes. Among them:
 * 147 GPU jobs running using 147 GPU nodes from non-condo and non-debugging jobs
 * 0 GPU jobs running using 0 GPU nodes from debugging jobs
 * 6 GPU Jobs running using 6 GPU nodes from PCON0005